about summary refs log tree commit homepage
path: root/t
DateCommit message (Collapse)
2022-08-10pop3: remove untouched rows on QUIT/disconnect
Some POP3 clients may connect and never retrieve messages nor trigger deletes. In that case, save some storage by removing unused rows from the `deletes' and `users' tables.
2022-08-03daemon: reload TLS certs and keys on SIGHUP
This allows new TLS certificates to be loaded for new clients without having to timeout nor drop existing clients with established connections made with the old certs. This should benefit users with admins who expire certificates frequently (as encouraged by Let's Encrypt).
2022-08-02daemon: support per-listener env, .psgi, out, err
This allows memory savings by allowing multiple, completely unrelated-PSGI apps to run within the same process as IMAP, NNTP, and POP3.
2022-07-28www: drop --subject from "git send-email" instructions
Apparently, --subject doesn't work[1] with "git send-email" in this context. So drop the CLI arg and add a note to tell the user to set a "Subject:" line in their response body, instead. [1] I'm not sure if --subject ever worked as I thought it would, or if it's a regression. In either case, there are current versions of git where it doesn't, so just tell users to use the currently supported method. Link: https://80x24.org/lore/git/CAC4O8c-Tf11CpwuRudyrpXv5bGshuyEenV9kKrs0zRWER-+yHA@mail.gmail.com/
2022-07-23nntp: inline CRLF in all response lines
This brings NNTP closer to POP3 and IMAP implementations to allow CoW avoidance on constants.
2022-07-21pop3: drop File::FcntlLock requirement for FreeBSD and Linux
I know Linux has a stable ABI for this, and FreeBSD seems to, too (*BSDs don't have stable syscall numbers, though). I suspect this is safe enough for all *BSDs. This is stricter than the MboxLock one since we use exact byte ranges with these locks.
2022-07-20pop3: advertise STLS in CAPA if appropriate
This is documented in RFC 2595, and POP3 clients may rely on seeing "STLS" in CAPA output to initiate TLS negotiation.
2022-07-20pop3: TOP requests do not expire messages
RFC 2449 only documents "EXPIRE 0" behavior for RETR requests which fetch the whole message. TOP requests only fetch the headers and top $N lines of the body, so it's probably harmful for deletions to be triggered in those cases.
2022-07-20public-inbox-pop3d - a mostly read-only POP3 server
Old account expiry has not been implemented, but it seems to work well with both mpop(1) and getmail(1). The strictness of mpop was particularly helpful in ironing out bugs in our implementation of (dreaded) message sequence numbers. "EXPIRE 0" (RFC 2449) can theoretically save numerous "DELE" commands, but that's untested by real-world clients. mpop supports PIPELINING which is effective in hiding latency, and the core networking functionality is already well-tested from our NNTP and IMAP implementations. Configuration requires "publicinbox.pop3state" to point to a directory writable by the otherwise read-only daemon. See public-inbox-pop3d(1) manpage for more usage details.
2022-07-08imap: STATUS: count messages properly
This only affects the rarely-used STATUS command, our message count was consistely zero due to misusing ->imap_exists. Noticed while implementing POP3 server.
2022-06-21search: do not index base-85 binary patches
Base-85 binary patches generated by git lead to many false positives, so skip over gibberish words which may occur in them. To avoid regressions in search results, continue to allow searching for exact size matches (via "literal $SIZE") and the phrase "GIT binary patch" for the mere presence of a binary patch.
2022-06-21search: support "patchid:" prefix (git patch-id --stable)
This allows easy searching via patch-id from a git commit. Currently, abbreviations are not supported, and it seems needless to support them since AFAIK (git) doesn't generate nor resolve abbreviated patch-ids anywhere.
2022-06-11t/spawn: Find invalid PID to try to join its process group
In the container used to build packages of the GNU Guix distribution, PID 1 runs as the same user as the test so this spawn that should fail actually succeeds. Fix the problem by going through different PIDs and picking one that either doesn't exist or we aren't allowed to signal.
2022-06-09view: do not escape first `@' in mailto: URLs
It's probably not a perfect match for RFC 6068 atm, but perfect is the enemy of good. Reported-by: Moritz Poldrack <moritz@poldrack.dev> Link: https://public-inbox.org/meta/CKJSWGSZFKMX.3VUSIYE955Z9X@Archetype/
2022-05-05public-inbox-netd: a multi-protocol superserver
Since we'll be adding POP3 support as our 4th network protocol; asking admins to run yet another daemon on top of existing -httpd, -nntpd, -imapd is a maintenance burden and a waste of memory. The goal of public-inbox-netd is to be able to replace all existing read-only daemons with a single process to save memory and reduce administrative overhead; hopefully encouraging more users to self-host their own mirrors. It's barely-tested at the moment. Eventually, multiple PI_CONFIG and HOME directories will be supported, as are per-listener .psgi config files.
2022-04-18lei: wire up pure Perl sendmsg/recvmsg for Linux users
This enables lei-daemon to work without Inline::C nor Socket::MsgHdr installed. Prior to this, only the `lei' client was using the pure Perl implementation. Either C implementation is still marginally faster, however.
2022-04-02lei_mail_sync: ensure URLs and folder names are stored as binary
Apparently leaving {sqlite_unicode} unset isn't enough, and there's subtle differences where BLOBs are stored differently than TEXT when dealing with binary data. We also want to avoid odd cases where SQLite will attempt to treat a number-like value as an integer. This should avoid problems in case non-UTF-8 URLs and pathnames are used. They'll automatically be upgraded if not, but downgrades to older lei would cause duplicates to appear.
2022-03-23syscall: implement sendmsg+recvmsg in pure Perl
Socket::MsgHdr is only packaged for Debian and derivatives at the moment, and Inline::C pulling in gcc/clang is a huge amount of disk space and bandwidth for some users. This enables disk space and/or bandwidth-limited users to use lei. Only Linux guarantees a stable ABI and syscall numbers, but that's the majority of our userbase. FreeBSD users will still have to use Inline::C (or get Socket::MsgHdr packaged). x86, x32, and x86-64 are all currently supported, more to be added.
2022-03-14t/lei-sigpipe.t: ensure SIGPIPE is not ignored instead of not blocked
Ignoring a signal is different than blocking a signal, and the "IgnoreSIGPIPE" option of systemd ignores. [ew: note systemd behavior] Acked-by: Eric Wong <e@80x24.org>
2022-03-08index|extindex: support --dangerous flag
This enables Xapian::DB_DANGEROUS to support in-place updates. This can speed up the initial index and reduce I/O at the cost of preventing concurrent readers and being unsafe in the face of any abnormal terminations. This is more dangerous than --no-fsync. --no-fsync is only unsafe in the event of a power loss or kernel crash; --dangerous is unsafe even on SIGKILL.
2022-03-01t/lei-sigpipe: ensure SIGPIPE is unblocked for this test
Tests run under systemd (and similar) have SIGPIPE blocked by default. This was causing this SIGPIPE test to get stuck when run by automated builders used by Nix. Thanks to Julien Moutinho and Dominique Martinet for tracking down this failure. Reported-by: Julien Moutinho <julm+public-inbox@sourcephile.fr> Reported-by: Dominique Martinet <asmadeus@codewreck.org> Link: https://public-inbox.org/meta/20220227080422.gyqowrxomzu6gyin@sourcephile.fr/
2022-02-18t/lei-sigpipe: attempt to improve diagnostics for stuck test
This may help diagnose a difficult-to-reproduce test failure on NixOS. Link: https://public-inbox/meta/20211209013743.okzgim7bbrpahks7@sourcephile.fr/
2022-02-14sharedkv: avoid ambiguity for numeric-like string keys
While we only store URLs and binary SHA-1/SHA-256 values in skv at the moment, we may store potentially ambiguous keys/values in the future. It's possible to store "02" and have it treated as `2' unless explicitly binding parameters as SQL_BLOB. This behavior was independent of the sqlite_unicode parameter as evidenced by the new tests. I only noticed this bug while hacking on another project using DBD::SQLite, and not while hacking on public-inbox itself.
2022-02-14sharedkv: remove unused subs
Some features didn't get used, and they're just getting in the way of upcoming bugfixes.
2022-02-14t/lei-*watch: disable flaky tests by default for now
Properly fixing these tests is too difficult for me at the moment, so just disable these tests for now. A proper fix and fleshing out support for inotify will hopefully happen at some point.
2022-01-31rewrite Linux nodatacow use in pure Perl w/o system
btrfs is Linux-only at the moment (and likely to remain that way for practical purposes). So rely on Linux ABI stability and use the `syscall' and `ioctl' perlops rather than relying on Inline::C. Inline::C (and gcc||clang) are monstrous dependencies which we can't expect users to have. This makes supporting new architectures more difficult, but new architectures come along rarely and this reduces the burden for the majority of Linux users on popular architectures (while still avoiding the distribution of pre-built binaries). Link: https://public-inbox.org/meta/YbCPWGaJEkV6eWfo@codewreck.org/
2022-01-23t/eml.t: ignore newer Email::MIME behavior
Once again, our message parser class matches the more tolerant behavior of older Email::MIME releases in order to handle ancient messages. This fixes <https://bugs.debian.org/1002219>, but dropping Email::MIME entirely from the test suite may be prudent in the future.
2021-11-22t/lei-mirror: skip lei comparisons if lei missing
We can't compare created_at times with lei if lei tests are skipped due to Inline::C or Socket::MsgHdr unavailability. Reported-by: Jörg Rödel <joro@8bytes.org> Link: https://public-inbox.org/meta/YZebmAxlFJy4lqAw@8bytes.org/
2021-11-10t/lei-watch: test with with higher sleep
0.1s may not be enough for a task switch and inotify wakeup, so try doubling it and see if it fixes test reliability, for now. A future change may be to implement a watcher/tracer for inotify -> lei/store events. Link: https://public-inbox.org/meta/20211104134327.zrf5jijfz7dsvb7l@meerkat.local/
2021-11-10lei q: disallow "\n" in argv[] elements
I don't expect this to be hit in real-world use via normal interactive shells. However, somebody could accidentally add "\n" in languages (e.g. Perl, C) where it's easy to pass "\n" in argv[].
2021-11-09lei q|up: fix saved searches for single-phrase search
`"' (double-quote) needs to be quoted for stdin searches. We also need to differentiate between "lei q --stdin" usage when calling "lei up", do it by setting an internal "rawstr" knob to ensure we can parse the config properly regardless of whether the initial search used --stdin or not.
2021-11-02t/lei-refresh-mail-sync: speed up test on FreeBSD 12
And improve reliability while we're at it. It seems closing a TCP listen socket on FreeBSD 12.2 doesn't cause connect()-ing clients to fail. This happens regardless of whether a socket is IPv4 or IPv6 This non-failure was causing tests to timeout slowly on the client side instead of failing immediately. We now fork a new process which does nothing but accept() + shutdown() to emulate a dead server. Reliability improves on all OSes since there's never a point in time when another process can bind the socket.
2021-11-02init: respect umask when creating description
I noticed a description for a new inbox had st_mode=0600.
2021-10-31lei_input: disallow uppercase characters for labels
Xapian boolean terms rely on upper-case prefixes, so the terms themselves need to be all lowercase.
2021-10-30lei_xsearch: quiet error message on SIG{PIPE,TERM}
SIGPIPE and SIGTERM are common and user-induced, so they're not worth warning on. Add the value of "$?", though, since it can help users notice other errors (e.g. SIGSEGV).
2021-10-28lei rm: move generic input_maildir_cb to LeiInput parent class
It's not much of a savings, right now, but maybe it can be in the future. I wanted to eliminate the "lei convert" one, too, but convert needs to preserve keywords which isn't possible with the generic fallback, so new tests were written for convert, instead.
2021-10-26t/lei-watch: add diagnostics for failure
I just got a difficult-to-reproduce failure, here; so there's still some issues with the up-to-dateness of the inotify watcher.
2021-10-26eml: keep body if no headers are found
This easily allows us to treat "git diff" output as header-less "messages" for commands such as "lei p2q".
2021-10-25t/index-git-times: support non-master default branch
2021-10-25www: $MSGID/raw: set charset in HTTP response
By using the charset specified in the message, web browsers are more likely to display the raw text properly for human readers. Inspired by a patch by Thomas Weißschuh: https://public-inbox.org/meta/20211024214337.161779-3-thomas@t-8ch.de/ Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-24t/git: support non-master default branch
2021-10-24t/watch_maildir: support non-master default branch
2021-10-24thread: avoid Perl5 internal scratchpad target cache
The use of array-returning built-ins such as `grep' inside arrayref declarations appears to result in permanently allocated scratchpad space for caching according to my malloc inspector. Thread skeletons get discarded every response, but multiple skeletons can exist in memory at once, so do what we can to prevent long-lived allocations from being made, here. In other words, replacing constructs such as: my $foo = [ grep(...) ]; with: my @foo = grep(...); Seems to ensure the mortality of the underlying array.
2021-10-24lei export-kw: skip read-only IMAP folders
Since we want to store IMAP flags asynchronously and not wait for results, we can't check for IMAP errors this way and end up wasting bandwidth on public-inbox-imapd. Now, we just check PERMANENTFLAGS up front to ensure a folder can handle IMAP flag storage before proceeding.
2021-10-23git: simplify local_nick, avoid "foo.git.git"
We need to use a non-greedy regexp to avoid capturing the ".git" suffix in the pathname before blindly appending our own.
2021-10-23t/v2index-late-dedupe: don't read user's ~/.public-inbox/config
Otherwise things can get noisy if bad entries exist in that file, because they do.
2021-10-22lei forget-search: support --prune=<local|remote>
Instead of: lei forget-search $OUTPUT && rm -r $OUTPUT we'll also allow a user to do: rm -r $OUTPUT && lei forget-search --prune This gives users flexibility to choose whatever flow is most natural to them.
2021-10-22wwwatomstream: call gmtime with scalar
When the gmtime() calls were moved from feed_entry() and atom_header() into feed_updated() in c447bbbd, @_ rather than a scalar was passed to gmtime(). As a result, feed <updated> values end up as "1970-01-01T00:00:00Z". Switch back to using a scalar argument to restore the correct timestamps. Fixes: c447bbbddb4ac8e1 ("wwwatomstream: simplify feed_update callers")
2021-10-22lei: use RENAME_NOREPLACE on Linux 3.15+
One syscall is better than two for atomicity in Maildirs. This means there's no window where another process can see both the old and new file at the same time (link && unlink), nor a window where we might inadvertantly clobber an existing file if we were to do `stat && rename'.
2021-10-22t/lei-p2q: extra diagnostics
I got one mysterious test failure here, once, and can't seem to reproduce it...