about summary refs log tree commit homepage
DateCommit message (Collapse)
2022-08-02daemon: share FDs for identical log paths
We rely on the %logs hash for SIGUSR1 log reopening. Without this sharing, some FDs would be hidden inside its respective {HTTP,IMAP,POP3}D object and not reopened on USR2
2022-08-02daemon: allow listening on well-known ports based on protocol
This allows admins to use "-l nntp://0.0.0.0/" to bind on port 119 without specifying ":119" on the CLI.
2022-08-02daemon: add diagnostics about inherited/bound listeners
These are helpful for diagnosing configuration problems, as well as a bug (to be fixed in the following commit).
2022-08-02daemon: require absolute cert/key paths with --daemonize
This is preparation for supporting loading new certs on SIGHUP.
2022-08-02daemon: support per-listener env, .psgi, out, err
This allows memory savings by allowing multiple, completely unrelated-PSGI apps to run within the same process as IMAP, NNTP, and POP3.
2022-08-02httpd: make internals slightly more generic
This brings the HTTP server closer to the IMAP/NNTP/POP3 implementations and eliminates package-wide globals in PublicInbox::HTTPD. The end goal is to be able to host completely different PSGI applications on different listen ports.
2022-07-30solver: avoid deprecation warnings in git 2.36.0+
git deprecated core.fsyncObjectFiles in favor of core.fsync with 2.36.0+, while GIT_TEST_FSYNC was added in 2.35.0. So use the environment variable since it's been supported slightly longer than the new configuration knob.
2022-07-30tests: maintainer test for using mpop
This ought to be a good stress test to ensure our POP3 implementation works against the POP3 client I've found.
2022-07-30doc|www: flesh out POP3 documentation for servers and users
Hopefully it makes sense to new users deploying or using POP3...
2022-07-29doc: httpd: document GIT_HTTP_MAX_REQUEST_BUFFER
We've always shared this environment with git-http-backend(1) (but don't (yet) support http.maxRequestBuffer anywhere)
2022-07-28www: drop --subject from "git send-email" instructions
Apparently, --subject doesn't work[1] with "git send-email" in this context. So drop the CLI arg and add a note to tell the user to set a "Subject:" line in their response body, instead. [1] I'm not sure if --subject ever worked as I thought it would, or if it's a regression. In either case, there are current versions of git where it doesn't, so just tell users to use the currently supported method. Link: https://80x24.org/lore/git/CAC4O8c-Tf11CpwuRudyrpXv5bGshuyEenV9kKrs0zRWER-+yHA@mail.gmail.com/
2022-07-24add xt/mem-nntpd-tls maintainer test
This ensures memory usage is reasonable when DEFLATE and TLS are enabled. It's also our only coverage for NNTP COMPRESS since Net::NNTP has yet to implement compression support: https://rt.cpan.org/Public/Bug/Display.html?id=129967
2022-07-24dsdeflate: shorten scope of initial buffer
There's no need to keep the initial buffer alive in package-wide scope once it's replaced by `$next' in ->write or ->zflush.
2022-07-24xt/mem-imapd-tls: update aliases to DSdeflate subs
Fixes: 23af251dd607c4e7 (imap+nntp: share COMPRESS implementation, 2022-07-23)
2022-07-23nntp: use substr to check for trailing CRLF
Regexps consume more CPU cycles and memory, and aren't necessary here since we just converted the entire buffer to CRLF.
2022-07-23pop3: reduce memory use while generating the mailbox cache
While the cache itself is relatively compact for 50K messages, generating it was inefficient due to our schema and Over.pm APIs being designed for NNTP. While we won't change our schema for now, we can choose better DBI APIs to use and limit our ephemeral memory use. This amounts to a 60% reduction in memory usage and a 5-10% speedup against org.kernel.vger.git.0: { echo 'USER '$(uuidgen)'@org.kernel.vger.git.0' echo PASS anonymous echo STAT echo QUIT } | nc $HOST $PORT
2022-07-23imap+nntp: share COMPRESS implementation
Their code was nearly identical to begin with, so save some memory in -netd and disk space for all of our tarball/distro users, at least. And I seem to have used multiple inheritance successfully, here, maybe...
2022-07-23nntp: resolve inboxes immediately on group listings
This prevents potential races between SIGHUP config reloads while gigantic group listings are streaming, allowing us to avoid many invalidation checks. This also reduces send(2) syscalls and avoid Perl internal pad allocations in a few places where it's not beneficial. There might be a slight (0.5%) speedup, but I'm not sure if that's down to system noise, power/thermal management, or other users on my VM.
2022-07-23ds: share long_step between NNTP and IMAP
It's not actually used by our POP3 code at the moment, but it may be soon to reduce memory usage when loading 50K smsg objects into memory.
2022-07-23nntp: inline CRLF in all response lines
This brings NNTP closer to POP3 and IMAP implementations to allow CoW avoidance on constants.
2022-07-23nntp: listgroup_range_i: remove useless `map' op
No need to iterate through the array twice; and this even seems a hair faster than what I got with commit 726d6e71aee5d974 (nntp: small speed up for multi-line responses, 2020-12-04)
2022-07-23ds: move requeue_once
It's the same subroutine everywhere.
2022-07-23ds: move no-op ->zflush to common base class
More deduplication, and POP3 never needed it.
2022-07-23ds: support greeting protocols
We can share some common code between IMAP, NNTP, and POP3 without too much trouble, so cut down our LoC.
2022-07-23nntp: remove more() wrapper
Using PublicInbox::DS->msg_more directly can avoid unnecessary CoW memory traffic since there's no appending "\r\n".
2022-07-23nntp: start adding CRLF to responses natively
With IMAP and POP3, I've started to embed CRLF into constant response codes to avoid triggering CoW and extra memory traffic in Perl. The end goal is to enable more code sharing between IMAP, NNTP, and POP3 inside one -netd process.
2022-07-23nntp: pass regexp to split() callers
Current implementations of Perl5 don't have optimizations for single-character field separators.
2022-07-21pop3: drop File::FcntlLock requirement for FreeBSD and Linux
I know Linux has a stable ABI for this, and FreeBSD seems to, too (*BSDs don't have stable syscall numbers, though). I suspect this is safe enough for all *BSDs. This is stricter than the MboxLock one since we use exact byte ranges with these locks.
2022-07-21www: note "x=m" and "t=1" (mis)use for GET requests
We require "x=m" (requests for mboxes) to be POST requests to avoid unnecessary traffic from crawlers. "t=1" only collapses threads in the summary view, which isn't normally accessible from <form> elements. This also fixes the missing "[summary|nested]" element when "x=m" is used.
2022-07-20gcf2: avoid excessive checks for unlinked files
We were misusing the timer and not expiring it before checking for unlinked files. Now, we check for unlinked files every 60s, instead.
2022-07-20pop3: advertise STLS in CAPA if appropriate
This is documented in RFC 2595, and POP3 clients may rely on seeing "STLS" in CAPA output to initiate TLS negotiation.
2022-07-20netd: setup TLS bits for well-known STARTTLS ports
Unfortunately, I can't think of an easy way to test this in our test suite since binding these ports are privileged and are often in use, anyways.
2022-07-20pop3: TOP requests do not expire messages
RFC 2449 only documents "EXPIRE 0" behavior for RETR requests which fetch the whole message. TOP requests only fetch the headers and top $N lines of the body, so it's probably harmful for deletions to be triggered in those cases.
2022-07-20pop3: implement IN-USE from RESP-CODES (RFC 2449)
This may help clients communicate to users if they're making parallel connections or if we have server bugs.
2022-07-20public-inbox-pop3d - a mostly read-only POP3 server
Old account expiry has not been implemented, but it seems to work well with both mpop(1) and getmail(1). The strictness of mpop was particularly helpful in ironing out bugs in our implementation of (dreaded) message sequence numbers. "EXPIRE 0" (RFC 2449) can theoretically save numerous "DELE" commands, but that's untested by real-world clients. mpop supports PIPELINING which is effective in hiding latency, and the core networking functionality is already well-tested from our NNTP and IMAP implementations. Configuration requires "publicinbox.pop3state" to point to a directory writable by the otherwise read-only daemon. See public-inbox-pop3d(1) manpage for more usage details.
2022-07-20netd: load modules for well-known ports
When inheriting well-known ports from systemd (or similar), we can auto-load the proper *D.pm file based on the port number without requiring command-line args. load_mod also gets fixed to use its argument, instead of implicit $1 since that won't work for our well-known.
2022-07-20lei note-event: inline note_event_arm_done
This was a single-caller sub since 47d4e53734820b4e (lei_mail_sync: rely on flock(2), avoid IPC, 2021-09-18) and unlikely to be used further, so inline it and save a few KB of memory.
2022-07-20lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
Enqueuing "note-event" requests from the DS event loop must not wait on workers being able to drain the queue quickly enough. Thus we make the SOCK_SEQPACKET writes nonblocking and rely on the lei-daemon event loop to enqueue writes. This is a unique problem for "note-event" since it reuses workers in between commands, while most lei commands currently fork off new workers.
2022-07-19searchidx: skip "delta $N" sections for base-85
I don't deal with binary patches ever, so I failed to notice binary deltas are supported in addition to the more common literals. A quick check of apply.c in git.git confirms "delta" and "literal" are the only binary patch classes we can expect.
2022-07-09test_common: avoid uninitialized warning on readlink
Of course, waiting for inotify to become active can't rely on inotify, so we need to do a busy loop here, instead...
2022-07-08imap: STATUS: count messages properly
This only affects the rarely-used STATUS command, our message count was consistely zero due to misusing ->imap_exists. Noticed while implementing POP3 server.
2022-07-07lei: track seen messages to note duplicates
This may help track down deduplication or other bugs in lei which lead to occasionally missing messages. Link: https://public-inbox.org/meta/CAL_JsqJH8xx_2NyZffNsRXbGXiv3kjmCETvKXt3Yfb0uToLm9Q@mail.gmail.com/
2022-07-07lei_xsearch: simplify lei/store import check
There's no need to check for two fields when one will suffice.
2022-07-01tree-wide: Fix typo accomodate
This was pointed out by the Debian package linter "lintian".
2022-07-01tree-wide: Fix typo likelyhood
This was pointed out by the Debian package linter "lintian".
2022-06-22searchthread: delete children early while ordering
This allows us to free up some memory sooner rather than later in case ordersub is expensive.
2022-06-22searchthread: remove + inline single-use cast sub
No point in wasting several kilobytes of memory for a single-use one-line sub.
2022-06-22doc: lei-q: regenerate for patchid: help
2022-06-21search: add help for patchid: prefix
Noticed-by: Kyle Meyer <kyle@kyleam.com>
2022-06-21search: do not index base-85 binary patches
Base-85 binary patches generated by git lead to many false positives, so skip over gibberish words which may occur in them. To avoid regressions in search results, continue to allow searching for exact size matches (via "literal $SIZE") and the phrase "GIT binary patch" for the mere presence of a binary patch.