| Date | Commit message (Collapse) |
|---|
|
getpid() isn't cached by glibc nowadays and system calls are
more expensive due to CPU vulnerability mitigations. To
ensure we switch to the new semantics properly, introduce
a new `on_destroy' function to simplify callers.
Furthermore, most OnDestroy correctness is often tied to the
process which creates it, so make the new API default to
guarded against running in subprocesses.
For cases which require running in all children, a new
PublicInbox::OnDestroy::all call is provided.
|
|
The musl strftime(3) implementation on AlpineLinux 3.19.0
doesn't support `%k' and `%k' isn't in POSIX, either. So we
fall back to using the `sprintf' perlop in the user-facing UI
since leading zeroes require needless overhead for my eyes and
brain to parse in the time.
|
|
This is a major step in solving the problem of having to
manually associate hundreds/thousands of coderepos with
hundreds/thousands of public-inboxes to power solver
(and more).
|
|
We must not attempt to read request bodies from the HTTP client
while processing a long request since that drains pipelined
requests. The NNTP/IMAP/POP3 event_step callbacks follow the
same behavior when {long_cb} is present from ->long_response.
This bug has little real-world consequence since HTTP/1.1
pipelining is not widely-used, especially when behind varnish
or other reverse proxies.
I found this bug while randomly strace-ing an active -netd
process to see the kind of traffic it was seeing.
|
|
This fixes two major problems with the use of tie for filehandles:
* no way to do fcntl, stat, etc. calls directly on the tied handle,
forcing callers to use the `tied' perlop to access the underlying
IO::Handle
* needing separate classes to handle blocking and non-blocking I/O
As a result, Git->cleanup_if_unlinked, InputPipe->consume,
and Qspawn->_yield_start have fewer bizzare bits and we
can call `$io->blocking(0)' directly instead of
`(tied *$io)->{fh}->blocking(0)'
Having a PublicInbox::IO class will also allow us to support
custom read buffering which allows inspecting the current state.
|
|
This will open the door for us to drop `tie' usage from
ProcessIO completely in favor of OO method dispatch. While
OO method dispatches (e.g. `$fh->close') are slower than normal
subroutine calls, it hardly matters in this case since process
teardown is a fairly rare operation and we continue to use
`close($fh)' for Maildir writes.
|
|
Found while hunting for popen_rd calls.
|
|
This ensures reused processes get a clean start and
avoids surprises as we develop more code around the
DS event loop.
|
|
We need some pipes in our parent process to capture the output
of lsof(1), so give us some more padding for temporary FDs.
|
|
The `:epoll' tag has been gone for a few weeks, and EPOLLIN
isn't used in this file anywhere.
Fixes: 3005c1bc5d05 (ds: use object-oriented API for epoll)
|
|
It's not worth the code and memory to have a setter method we
never use outside of tests.
|
|
HTTP::Tiny is part of the Perl standard library since Perl 5.14
while Net::HTTP has never been (unlike Net::NNTP or Net::POP3).
For the test which forces server-side buffering, we'll just use
regular socket handle.
|
|
We can't assume git output is UTF-8, and we'll always have
legacy data in git coderepos. So attempt to display some
some garbled text rather than nothing at all if Perl croaks
on it.
sox commit c38987e8d20505621b8d872863afa7d233ed1096
(Added raw inverse-bit u-law and A-law support. Updated *.txt files., 2001-12-13)
is an example of a commit which caused problems for me.
|
|
We can just use the non-wantarray form of popen_rd to
save us some extra error checking.
|
|
require_bsd and require_mods(':fcntl_lock') are now
supported in TestCommon to make it easier to maintain
than a big list of regexps.
getsockopt for SO_ACCEPTFILTER seems to always succeed,
even if the retrieved struct is all zeroes.
|
|
While .onion URLs don't commonly use TLS, using Tor to access
non-.onion URLs is possible and TLS is advisable in that case.
TLS + SOCKS support is also useful for non-Tor SOCKS proxies
(e.g. "ssh -D"), but 127.0.0.1:9050 (Tor) is probably the most
standardized address.
While we're in the area: switch to v5.12, use autodie, and
ensure all necessary modules are present.
|
|
The export tag for :epoll no longer exists and it seems like
it hasn't been needed in a while, anyways. We can also drop a
useless EPOLLIN import, as well.
Fixes: 3005c1bc5d05 (ds: use object-oriented API for epoll)
|
|
This allows us to get rid of some duplication in our Makefile
|
|
This non-parallelized target is useful for noticing core dumps
and preventing them from being clobbered as we run the test
suite. It will also notice leftover temporary files and
directories.
This make target was used on OpenBSD 7.3 to develop at least two
recent fixes:
e281363ba937 (lei: ensure we run DESTROY|END at daemon exit w/ kqueue)
759885e60e59 (lei: ensure --stdin sets %ENV and $current_lei)
I considered using a per-test TMPDIR for this to enable
parallelization, but on-filesystem UNIX sockets won't work
with excessively long path names.
|
|
I'm not sure if `?' or `=' are allowed characters in POP3
mailbox names. In fact, I can't find any information on
valid characters allowed in RFC 1081 nor RFC 1939.
In any case, it works fine with mpop, Claws-Mail, and
Thunderbird.
Tested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The Xapian SWIG bindings are favored by Xapian upstream for
ease-of-maintenance compared to the XS version. While Debian
lags on this front, the SWIG bindings are widely available
on all *BSDs.
|
|
This allows us to avoid repeatedly using memory-intensive
anonymous subs in CodeSearchIdx where the callback is assigned
frequently. Anonymous subs are known to leak memory in old
Perls (e.g. 5.16.3 in enterprise distros) and still expensive in
newer Perls. So favor the (\&subroutine, @args) form which
allows us to eliminate anonymous subs going forward.
Only CodeSearchIdx takes advantage of the new API at the moment,
since it's the biggest repeat user of post-loop callback
changes.
Getting rid of the subroutine and relying on a global `our'
variable also has two advantages:
1) Perl warnings can detect typos at compile-time, whereas the
(now gone) method could only detect errors at run-time.
2) `our' variable assignment can be `local'-ized to a scope
|
|
File::Path already accounts for the existence of directories,
handles races from redundant mkdir(2), and croaks on
unrecoverable errors. So there's no point in doing any
of that on our end.
Furthermore, avoiding the overhead of loading File::Path doesn't
seem worth it to save 20-60ms given the overhead of loading
our other code. Instead, try to reduce optree overhead on
our code, instead, since File::Path gets used in a bunch of
places.
We'll also favor the newer make_path for multi-directory
invocations to avoid bloating our own optree to create an
arrayref, but mkpath is one fewer subroutine call within
File::Path itself, right now.
|
|
Back in the old days, git didn't store commit encodings
and allowed messages in various encodings to enter history.
Assuming such a commit is UTF-8 trips up s/// operations
on buffers read with the `:utf8' PerlIO layer. So clear
Perl's internal UTF-8 flag if we end up with something
which isn't valid UTF-8
An example is commit 7eb93c89651c47c8095d476251f2e4314656b292
in git.git ([PATCH] Simplify git script, 2005-09-07)
|
|
We'll rely on defined(wantarray) to implicitly skip subtests,
and memoize these to reduce syscalls, since tests should
be short-lived enough to not be affected by new installations or
removals of git/xapian-compact/curl/etc...
|
|
Uppercase characters aren't allowed for labels due to Xapian
boolean limitations, so we need to use lowercase labels.
Fixes: 27015c3365fd0690 (lei_input: disallow uppercase characters for labels, 2021-10-31)
|
|
On my x86-64 machine, OpenSSL SHA-256 is nearly twice as fast as
the Digest::SHA implementation from Perl, most likely due to an
optimized assembly implementation. SHA-1 is a few percent
faster, too.
|
|
Loading cgitrc (and associated projects.list) can get users
out of defining as many individual coderepos.
xt/solver.t needs a use of `$_' replaced since that
gets clobbered while parsing cgitrc.
|
|
While SHA-256 isn't supported for inboxes, yet
xt/git-http-backend.t now runs properly against a SHA-256 code
repository
|
|
This emphasizes annotated tags as their own object type in the
web UI while being able to link to the existing show_commit()
linkification and dfblob: search.
|
|
Solver tests can never succeed without coderepos configured,
since that's the whole point of solver. And improve the
original skip message to note that it's about the `git'
public-inbox, not `git' itself.
|
|
This allows us to consolidate our checks for
Plack::Test::ExternalServer and enforce our redirect-disabled
LWP::UserAgent.
|
|
perf-obfuscate was close enough to perf-msgview that it only
required setting the `obfuscate' field of the inbox.
Then update perf-msgview to account for upcoming internal
changes. The current use of {obuf} and concat ops results in
excessive scratchpad space and I may be able to even get
speedups by avoiding concat ops.
|
|
There'll be a number of upcoming changes to HTML rendering
of messages to hopefully reduce memory usage and speedups
by writing out to the gzip buffer earlier.
Update the tests now so it'll be easier to test before
and after results.
|
|
I'm making some tweaks to solver and will probably find extra
output useful, and also update to v5.12 while we're at it.
|
|
This ought to be a good stress test to ensure our POP3
implementation works against the POP3 client I've found.
|
|
This ensures memory usage is reasonable when DEFLATE and TLS are
enabled. It's also our only coverage for NNTP COMPRESS since
Net::NNTP has yet to implement compression support:
https://rt.cpan.org/Public/Bug/Display.html?id=129967
|
|
Fixes: 23af251dd607c4e7 (imap+nntp: share COMPRESS implementation, 2022-07-23)
|
|
Their code was nearly identical to begin with, so save some
memory in -netd and disk space for all of our tarball/distro
users, at least.
And I seem to have used multiple inheritance successfully, here,
maybe...
|
|
I just did a double-take and nearly thought authentication
was broken while reading LeiConvert.pm. Add a comment in
LeiConvert.pm to clarify things, too.
|
|
In case an IMAP folder is deleted, just set an error and
ignore it rather than creating an empty folder which we
attempt to export keywords to for non-existent messages.
|
|
require_mods covers it, and we're not testing Plack itself.
|
|
Some code paths may use maximum size checks, so ensure
any checks are waited on, too.
|
|
This helped me quickly reproduce a bug in Encode[1] and
will help me determine performance implications of workarounds
for the aforementioned bug.
[1] https://rt.cpan.org/Public/Bug/Display.html?id=139622
|
|
Since signalfd is often combined with our event loop, give it a
convenient API and reduce the code duplication required to use it.
EventLoop is replaced with ::event_loop to allow consistent
parameter passing and avoid needlessly passing the package name
on stack.
We also avoid exporting SFD_NONBLOCK since it's the only flag we
support. There's no sense in having the memory overhead of a
constant function when it's in cold code.
|
|
It can be useful to test with some of these, but we can't enable
them universally for all servers (and debug + compress is gross)
|
|
I'm not sure what caused it, but I've noticed two missing
messages that failed from "lei up" on an https:// external;
and I've also seen some duplicates in the past (which I
think I fixed...).
|
|
Untested at the moment(*), but we were inadvertantly truncating
mailbox names with '/' due to our work-in-progress handling of
"/;UID=$NUM" parameter.
(*) strangely, my dovecot instance doesn't allow '/' by default,
so the change to xt/net_writer-imap.t is untested.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210914175025.eq7s2shkc323itaf@meerkat.local/
|
|
Since "lei up" is expected to be a heavily-used command,
better support for IMAP seems like a reasonable idea.
This is inefficient since we waste an IMAP(S) TCP connection
since it dies when an auth-only LeiUp worker process dies, but
it's better than not working at all, right now.
|
|
That's the minimum, at least...
|