about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-10-22lei: use RENAME_NOREPLACE on Linux 3.15+
One syscall is better than two for atomicity in Maildirs. This means there's no window where another process can see both the old and new file at the same time (link && unlink), nor a window where we might inadvertantly clobber an existing file if we were to do `stat && rename'.
2021-10-22lei_mail_sync: mv_src: use transaction, check UNIQUE
We need a transaction across two SQL statements so readers (which don't use flock) will see the result as atomic. This may help against some occasional test failures I'm seeing from t/lei-auto-watch.t and t/lei-watch.t, or make the problem more apparent.
2021-10-22lei: no Perl FileHandle for `undef' w/ ECONNRESET
Error reporting for recv_cmd4 methods is a bit wonky.
2021-10-22dir_idle: treat IN_MOVED_FROM as a gone event
Whether an MUA uses rename(2) or link(2)+unlink(2) combination should not matter to us. We should be able to handle both cases.
2021-10-22lei note-event: clear_src on ENOENT
When a file goes away, try to make sure we don't waste time trying to access or store it.
2021-10-22doc: lei-overview: add CAVEATS section
IMAP and NNTP client performance absolutely sucks compared to what the read-only daemons are capable of...
2021-10-22watch: remove redundant signal mask manipulation
The top-level daemon process already blocks all signals, so there's no reason to block them around fork() calls.
2021-10-22watch: check for {quit} before IDLE
This may make it less likely for watch-dependent tests to get stuck. Unfortunately, due to the synchronous API of Mail::IMAPClient, ->idle is still susceptible to missing signals.
2021-10-22lei_search: try harder to associate "lei index"-ed messages
Allow checking for keyword changes if we have an known OID, even if the blob isn't currently reachable.
2021-10-22lei note-event: wq_io_do => wq_do
No need to pass extra arrayref args, here.
2021-10-22lei note-event: drop unnecessary eval guard
We don't want to lose the failure message in case note-event fails.
2021-10-22lei/store: check for any unexpected process death
The lei/store process should only exit from EOF on the socket, so make sure we note any unintended signals
2021-10-22t/lei-p2q: extra diagnostics
I got one mysterious test failure here, once, and can't seem to reproduce it...
2021-10-22t/lei-import-maildir: rename fix (SR -> RS)
While it doesn't matter to us, the Maildir spec specifies characters are to be sorted in alphabetical order.
2021-10-22t/lei-{auto-watch,export-kw}: extra diagnostics on failure
Maybe these will help track down some failures and make diagnosing bugs easier. "lei export-kw" should also become optional, even, so allow disabling it easily in the test.
2021-10-20httpd: reject requests with spaces in header names
Malicious clients may attempt HTTP request smuggling this way. This doesn't affect our current code as we only look for exact matches, but it could affect other servers behind a to-be-implemented reverse proxy built around our -httpd. This doesn't affect users behind varnish at all, nor the HTTPS/HTTP reverse proxy I use (I don't know about nginx), but could be passed through by other reverse proxies. This change is only needed for HTTP::Parser::XS which most users probably use. Users of the pure Perl parser (via PLACK_HTTP_PARSER_PP=1) already hit 400 errors in this case, so this makes the common XS case consistent with the pure Perl case. cf. https://www.mozilla.org/en-US/security/advisories/mfsa2006-33/
2021-10-19lei_mail_sync: show non-matching SHA
It could prove useful for diagnosing bugs (either on our end or an MUA's), or storage device failures.
2021-10-19lei inspect: show ISO8601 {rt} and {dt}, too
While inspect is intended for debugging, the Unix epoch in seconds requires extra steps for human consumption; just steal what we used for "lei q -f json" output.
2021-10-19lei inspect: add atfork hook
This is necessary for in case an inspect command is run in a parallel with other commands.
2021-10-19doc: lei: describe lei-daemon-kill and upgrades
While we're at it, start dropping copyright years since it seems acceptable to not have them: https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/ Copyright years are also a noisy to update every year (maybe, just maybe, we'll make it to 2022...)
2021-10-19lei: remove unused ->busy time arg
Our graceful shutdown doesn't time out clients.
2021-10-19lei up: support --exclude=, --no-(external|remote|local)
These can be used to temporarily disable using certain externals in case of temporary network failure or mount point unavailability.
2021-10-19lei: conditionally add "\n" to error messages
Some error messages already include "\n" (w/ file+line info), so don't add another one. (`warn' will automatically add its caller location unless there's a final "\n").
2021-10-19lei up: propagate redispatch_all failure via exit code
We can still continue with some local externals, maybe; but the error needs to be propagated to the calling process for scripting purposes.
2021-10-19lei: use die for external and query handling
This allows "lei up" to continue processing unrelated externals if on output fails.
2021-10-19lei up: prefix `remote' and `local' with `o_'
This will help distinguish between mail outputs and external public-inboxes.
2021-10-19test_common: lazy-require AutoReap
This might speed up non-daemon-using tests.
2021-10-19Makefile.PL: drop generated lib/PublicInbox.pm in blib/
Running "make test" on this project doesn't pass unless you've got an existing PublicInbox.pm in your @INC, presumably nobody's set this up on a fresh machine in a while. This Makefile.PL trickery seems to do it, I've validated this with this ad-hoc test of committing blib/ and Makefile to the repository: git clean -dxf; perl Makefile.PL && make -j8 all && git add -f blib Makefile.PL Makefile && git commit -m"now" Running that in interactive rebase before/after shows that only the PublicInbox.pm file was added to blib/lib/. We use $(INST_LIB) instead of a hardcoded 'blib/lib' now, but it's what ExtUtils::MakeMaker recommends, so it's probably for the better. As far as I can tell this broke with 1fae720d (build: generate PublicInbox.pm with $VERSION, 2021-04-01), but I have not tested that. See also 1fae720d (build: generate PublicInbox.pm with $VERSION, 2021-04-01) which made the PublicInbox.pm a generated file.
2021-10-18v2: mirrors don't clobber msgs w/ reused Message-IDs
For odd messages with reused Message-IDs, the second message showing up in a mirror (via git-fetch + -index) should never clobber an entry with a different blob in over. This is noticeable only if the messages arrive in-between indexing runs. Fixes: 4441a38481ed ("v2: index forwards (via `git log --reverse')")
2021-10-18extindex: show mismatches for messages deleted from inbox
There seems to be a bug in v2 inbox reindexing somewhere...
2021-10-17extindex: better locations for {quit} checks
Check for graceful termination at every message since it's a fairly inexpensive check.
2021-10-17extindex: guard against false mismatch unrefs
I'm not sure if this is a bug or not (or it could be an old bug in the v2 indexing code).
2021-10-17extindex: retry sync_inbox before reindex
Ensure the num highwater mark of the target inbox is stable before using it. Otherwise we may end up repeating work done to index a message.
2021-10-17extindex: use localtime to display lock time
Since this is intended for use on the command-line, include TZ offset in time and try to shorten the message a bit so it wraps less on a terminal.
2021-10-17msgmap: do not cache num_highwater
Caching the value doesn't seem necessary from a performance perspective, and it adds a caveat for read-only users which may lead to bugs in future code.
2021-10-16eml: fix leak workaround
Our previous workaround didn't actually work around the leak in <https://rt.cpan.org/Public/Bug/Display.html?id=139622> since croak()-via-Perl was still invoked before the SV reference count could be decremented. Put in a proper workaround which saves warnings onto a temporary variable and only croak after ->decode or ->encode returns; not inside those methods.
2021-10-16MANIFEST: regenerate with: git ls-files >MANIFEST
2021-10-16lei sockets: favor level-triggered epoll for fairness
Sigfd->event_step needs priority over script/lei clients, LeiSelfSocket, and everything else.
2021-10-16input_pipe: do not loop in ->event_step for fairness
Sigfd->event_step needs priority over InputPipe (and everything else). We keep Edge Triggering here but use ->requeue instead of looping inside event_step. This was necessary because InputPipe can be used with regular files which can't be monitored with epoll. We'll also rid of the vestigial lei-oneshot support while we're at it.
2021-10-16pkt_op: favor level-triggered epoll for fairness
Sigfd->event_step needs priority over PktOp (and everything else). We'll also add ECONNRESET checking, here, since it could see bidirectional use in the future. This is unlikely to have any sort of performance difference since this is only for small, occasional packets, but the code reduction is nice.
2021-10-16wqworker: favor level-triggered epoll for fairness
Sigfd->event_step needs priority over WQWorkers (and everything else). Do that by running once per event_loop iteration rather than looping inside event_step. This lowers throughput since it requires more syscalls, but that's the price of fairness.
2021-10-16t/lei*: set EDITOR for dumb terminals
Running tests over a non-interactive ssh session fails, otherwise.
2021-10-16doc: lei: add manpages for remaining commands
At this point all of the current lei commands, aside from -help and -sucks, should be covered.
2021-10-16doc: lei: restore alphabetical order to some listings
Most the lei-related entries in txt2pre and Makefile.PL are in alphabetical order. Reorder the few that aren't. While at it, reflow the Makefile.PL entries in preparation for the entries that will be added in the next commit.
2021-10-16extindex: avoid triggering a buggy unref
We can't attempt to unref messages beyond the highwater mark of an inbox. This bugfix was found by commit c485036d0b1ce7ed (extindex: guard against buggy unrefs, 2021-10-14), which actually did its intended job and guarded against a buggy unref.
2021-10-16httpd/async: switch to level-triggered epoll
We'll save ourselves some code here and let the kernel do more work, instead.
2021-10-16inbox + search: use 5.10.1 and do some golfing
Some yak-shaving while I try to track down other bugs...
2021-10-16lei_to_mail: quiet down abort messages
We don't need to flood the terminal with "W: $oid is (!= blob)\n" messages when somebody nukes a git cat-file process from under us.
2021-10-16lei_overview: die rather than lei->fail
This will make our code more flexible in case it gets used in non-lei things.
2021-10-16extindex: prune invalid alternate entries on --gc
Seeing the same warning over and over again gets annoying.