Date | Commit message (Collapse) |
|
One syscall is better than two for atomicity in Maildirs. This
means there's no window where another process can see both the
old and new file at the same time (link && unlink), nor a window
where we might inadvertantly clobber an existing file if we were
to do `stat && rename'.
|
|
We need a transaction across two SQL statements so readers
(which don't use flock) will see the result as atomic.
This may help against some occasional test failures I'm seeing
from t/lei-auto-watch.t and t/lei-watch.t, or make the problem
more apparent.
|
|
Error reporting for recv_cmd4 methods is a bit wonky.
|
|
Whether an MUA uses rename(2) or link(2)+unlink(2) combination
should not matter to us. We should be able to handle both
cases.
|
|
When a file goes away, try to make sure we don't waste
time trying to access or store it.
|
|
IMAP and NNTP client performance absolutely sucks compared to what
the read-only daemons are capable of...
|
|
The top-level daemon process already blocks all signals,
so there's no reason to block them around fork() calls.
|
|
This may make it less likely for watch-dependent tests to get
stuck. Unfortunately, due to the synchronous API of
Mail::IMAPClient, ->idle is still susceptible to missing
signals.
|
|
Allow checking for keyword changes if we have an known OID,
even if the blob isn't currently reachable.
|
|
No need to pass extra arrayref args, here.
|
|
We don't want to lose the failure message in case note-event
fails.
|
|
The lei/store process should only exit from EOF on the
socket, so make sure we note any unintended signals
|
|
I got one mysterious test failure here, once, and can't seem
to reproduce it...
|
|
While it doesn't matter to us, the Maildir spec specifies
characters are to be sorted in alphabetical order.
|
|
Maybe these will help track down some failures and make
diagnosing bugs easier. "lei export-kw" should also become
optional, even, so allow disabling it easily in the test.
|
|
Malicious clients may attempt HTTP request smuggling this way.
This doesn't affect our current code as we only look for exact
matches, but it could affect other servers behind a
to-be-implemented reverse proxy built around our -httpd.
This doesn't affect users behind varnish at all, nor the
HTTPS/HTTP reverse proxy I use (I don't know about nginx), but
could be passed through by other reverse proxies.
This change is only needed for HTTP::Parser::XS which most users
probably use. Users of the pure Perl parser (via
PLACK_HTTP_PARSER_PP=1) already hit 400 errors in this case,
so this makes the common XS case consistent with the pure Perl
case.
cf. https://www.mozilla.org/en-US/security/advisories/mfsa2006-33/
|
|
It could prove useful for diagnosing bugs (either on our
end or an MUA's), or storage device failures.
|
|
While inspect is intended for debugging, the Unix epoch in
seconds requires extra steps for human consumption; just
steal what we used for "lei q -f json" output.
|
|
This is necessary for in case an inspect command is run
in a parallel with other commands.
|
|
While we're at it, start dropping copyright years
since it seems acceptable to not have them:
https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
Copyright years are also a noisy to update every year (maybe,
just maybe, we'll make it to 2022...)
|
|
Our graceful shutdown doesn't time out clients.
|
|
These can be used to temporarily disable using certain
externals in case of temporary network failure or mount point
unavailability.
|
|
Some error messages already include "\n" (w/ file+line info),
so don't add another one. (`warn' will automatically add its
caller location unless there's a final "\n").
|
|
We can still continue with some local externals, maybe;
but the error needs to be propagated to the calling process
for scripting purposes.
|
|
This allows "lei up" to continue processing unrelated externals
if on output fails.
|
|
This will help distinguish between mail outputs and external
public-inboxes.
|
|
This might speed up non-daemon-using tests.
|
|
Running "make test" on this project doesn't pass unless you've got an
existing PublicInbox.pm in your @INC, presumably nobody's set this up
on a fresh machine in a while.
This Makefile.PL trickery seems to do it, I've validated this with
this ad-hoc test of committing blib/ and Makefile to the repository:
git clean -dxf; perl Makefile.PL && make -j8 all && git add -f blib Makefile.PL Makefile && git commit -m"now"
Running that in interactive rebase before/after shows that only the
PublicInbox.pm file was added to blib/lib/. We use $(INST_LIB) instead
of a hardcoded 'blib/lib' now, but it's what ExtUtils::MakeMaker
recommends, so it's probably for the better.
As far as I can tell this broke with 1fae720d (build: generate
PublicInbox.pm with $VERSION, 2021-04-01), but I have not tested
that. See also 1fae720d (build: generate PublicInbox.pm with $VERSION,
2021-04-01) which made the PublicInbox.pm a generated file.
|
|
For odd messages with reused Message-IDs, the second message
showing up in a mirror (via git-fetch + -index) should never
clobber an entry with a different blob in over.
This is noticeable only if the messages arrive in-between
indexing runs.
Fixes: 4441a38481ed ("v2: index forwards (via `git log --reverse')")
|
|
There seems to be a bug in v2 inbox reindexing somewhere...
|
|
Check for graceful termination at every message since it's
a fairly inexpensive check.
|
|
I'm not sure if this is a bug or not (or it could be
an old bug in the v2 indexing code).
|
|
Ensure the num highwater mark of the target inbox is stable
before using it. Otherwise we may end up repeating work
done to index a message.
|
|
Since this is intended for use on the command-line,
include TZ offset in time and try to shorten the
message a bit so it wraps less on a terminal.
|
|
Caching the value doesn't seem necessary from a performance
perspective, and it adds a caveat for read-only users which
may lead to bugs in future code.
|
|
Our previous workaround didn't actually work around the leak in
<https://rt.cpan.org/Public/Bug/Display.html?id=139622> since
croak()-via-Perl was still invoked before the SV reference
count could be decremented.
Put in a proper workaround which saves warnings onto a temporary
variable and only croak after ->decode or ->encode returns; not
inside those methods.
|
|
|
|
Sigfd->event_step needs priority over script/lei clients,
LeiSelfSocket, and everything else.
|
|
Sigfd->event_step needs priority over InputPipe (and everything
else). We keep Edge Triggering here but use ->requeue instead
of looping inside event_step. This was necessary because
InputPipe can be used with regular files which can't be
monitored with epoll.
We'll also rid of the vestigial lei-oneshot support while we're
at it.
|
|
Sigfd->event_step needs priority over PktOp (and everything else).
We'll also add ECONNRESET checking, here, since it could see
bidirectional use in the future.
This is unlikely to have any sort of performance difference
since this is only for small, occasional packets, but the code
reduction is nice.
|
|
Sigfd->event_step needs priority over WQWorkers (and everything
else). Do that by running once per event_loop iteration rather
than looping inside event_step. This lowers throughput since it
requires more syscalls, but that's the price of fairness.
|
|
Running tests over a non-interactive ssh session fails,
otherwise.
|
|
At this point all of the current lei commands, aside from -help and
-sucks, should be covered.
|
|
Most the lei-related entries in txt2pre and Makefile.PL are in
alphabetical order. Reorder the few that aren't.
While at it, reflow the Makefile.PL entries in preparation for the
entries that will be added in the next commit.
|
|
We can't attempt to unref messages beyond the highwater mark of
an inbox. This bugfix was found by commit c485036d0b1ce7ed
(extindex: guard against buggy unrefs, 2021-10-14), which
actually did its intended job and guarded against a buggy unref.
|
|
We'll save ourselves some code here and let the kernel do more
work, instead.
|
|
Some yak-shaving while I try to track down other bugs...
|
|
We don't need to flood the terminal with "W: $oid is (!= blob)\n"
messages when somebody nukes a git cat-file process from under
us.
|
|
This will make our code more flexible in case it gets used in
non-lei things.
|
|
Seeing the same warning over and over again gets annoying.
|