| Date | Commit message (Collapse) |
|---|
|
lei-daemon (the top-level daemon process) should not have
synchronous waits, and this was causing a deadlock with
interrupted commands. There may still be a bug lurking in
lei/store despite this fix, though. I originally thought commit
fd261b9e65674505 (lei_store_err: use level-trigger for error pipe, 2022-08-15)
was sufficient, but at least this change is needed, as well.
|
|
We can calculate the total size of the mailbox while generating
the cache, which allows us to iterate the cache again to
calculate the size of the mailbox slice. While we're in the
area, simplify the loop and avoid needlessly updating the `$beg'
variable.
This adds a small amount of constant time overhead to DELE,
however that is amortized across multiple requests for fairness.
|
|
This fixes deadlocks from errors inside lei/store when multiple
errors are spewed.
|
|
mark_dele already works on the cache offset, so there's no need
to make further adjustments to the offset (as we do with POP3
sequence numbers).
|
|
Setting the $if_active parameter of ->prepare_cached to `1'
seemed to be the best option many years ago, so it's probably
the best option going forward when caching prepared statements.
Fixes: cab36ebd00ca72f8 ("pop3: remove untouched rows on QUIT/disconnect")
|
|
Tested on gcc92.fsffrance.org from cfarm.
|
|
This fixes t/v2index-late-dupe.t to be skipped properly on
systems without DBD::SQLite
|
|
...as we've already done for the simpler case of mbox locking in lei.
I've just confirmed NetBSD and OpenBSD share the same "struct flock"
with FreeBSD, and assume DragonflyBSD is the same. sizeof(pid_t) == 4
in all places I've checked, and it's unlikely we'll need 64-bit
pid_t any time soon...
|
|
curl only supports "pop3://" and "pop3s://", despite RFC 2384
existing for "pop://". AFAIK, there's no RFCs for "pop3://"
and "pop3s://", but please let us know if there are.
In any case, real-world cases like curl are more relevant.
|
|
We use "Newsgroup" (singular) when there's only one NNTP
server address configured.
|
|
warn/carp usage is unavoidable given Perl itself and standard
libraries, so just rely on localized $SIG{__WARN__} from
60d262483a4d6ddf (daemon: use per-listener SIG{__WARN__} callbacks, 2022-08-08)
for all error reporting.
While we're in the area, make some of the error handling more
consistent between IMAP/NNTP/POP3.
|
|
While the ';' requires escaping on the command-line, the
presence of ";AUTH=ANONYMOUS" communicates clearly that
anonymous access is supported in accordance to RFC 4505.
|
|
NNTP and IMAP can also exist in the same area, so clarify
that the username + password is only for POP3
|
|
This will make it easier to link to these sections in
3rd-party documentation.
|
|
Some POP3 clients may connect and never retrieve messages nor
trigger deletes. In that case, save some storage by removing
unused rows from the `deletes' and `users' tables.
|
|
It was never useful for IMAP, and NNTP hasn't needed it since:
1bf653ad139bf7bb (nntp+www: drop List-* and Archived-At headers, 2020-12-10)
|
|
Since IMAP mailbox lists are tied to the PublicInbox::Config
object, we can share them the same way the config object is
shared when an -imapd or -netd instance has multiple listeners.
This ought to reduce memory use and startup time when binding
multiple sockets which share a common config file.
|
|
This avoids dangling {''} entries in $xnetd and
%tls_opt hashes. Furthermore, we can safely undef
%tls_opt once it's associated with each $xnetd object.
|
|
This allows "-l $ADDRESS?err=/path/to/err.log to isolate normal
warn() (and carp()) messages for a particular listen address to
track down errors more easily.
|
|
This ensures the "bound $URL" diagnostic message at startup
always shows the URL scheme handled if not relying on socket
inheritance.
This also avoids duplicate/unused data structures when binding
sockets ourselves, as bound socket names can expand from short
names to longer names (e.g. "0:119" => "0.0.0.0:119").
|
|
...by deprioritizing clients using a username + password.
As IMAP provides AUTH=ANONYMOUS for designating anonymous
access, we'll rely on it as a heuristic for favoring "good"
clients. Clients using a username + password seem to (more
often than not) be malicious and looking for info which doesn't
belong in public inboxes.
This copies the technique used by WWW + -httpd to deprioritize
expensive mbox.gz downloads.
|
|
Looking at IMAP traffic on public-inbox.org, it seems there is a
fair amount of traffic coming from malicious clients assuming
the IMAP server is compromised and searching for private
information. Since AUTH=ANONYMOUS clients are more likely to
be legitimate clients looking for publicly-archived mail,
give them priority.
|
|
This improves fairness while having no measurable performance
impact for a single uncached IMAP client (mutt) opening a folder
for the first time.
I noticed this problem with the public-inbox.org IMAP server where
a few IMAP clients were unfairly monopolizing the -netd process.
|
|
This means all Inbox, Git, Over, Msgmap, Search objects also get
deduplicated if they belong to the same config file, reducing
memory and FD usage. This helps save memory and improve cache
hit rates in -netd setups where NNTP, IMAP, HTTP, and POP3
servers run in the same process.
InboxIdle was the only bit which needed adjustment, but there
may be other bugs lurking despite all tests passing.
|
|
->zflush must return a string to its caller, not undef.
Additionally, {http_out} may be deleted on ->write if ->close
recurses.
This should fix the following errors:
Use of uninitialized value $_[1] in string eq at PublicInbox/HTTP.pm line 211.
E: Can't call method "close" on an undefined value at GzipFilter.pm line 167.
Fixes: a6d50dc1098c01a1 (www: gzip_filter: gracefully handle socket ->write failures, 2022-08-03)
|
|
We can rely on auto-vivification to avoid an intermediate
array for the map result.
|
|
We can bless objects while doing the initial insertion to avoid
extra the extra map iteration and temporary array(s). Fewer ops
means memory savings for the likely case of ->over users, too.
|
|
We can reduce ops and temporary objects here by folding the
stringification into the `for' loop and push directly into the
{mailboxlist} array; relying on autovivification to turn it into
a noop for the initial population.
|
|
We can rely on //g and autovivification, here.
|
|
We can use DBI's selectcol_arrayref directly (as we do in other
places) to avoid unnecessary arrays and ops on our end.
|
|
There's no need to create two intermediate arrays when we can
modify the existing arrayref.
|
|
This may save a few bytes with many connected clients.
Noticed while working on the JMAP endpoint.
|
|
We must not clobber already-parsed per-listener options when
handling inherited sockets which are well-known. Unfortunately,
this isn't easy to test in a non-intrusive way for regular
users.
|
|
ConfigIter was still too slow despite being fair. The addition of
ART_MIN in ALL->misc means it can be used as a startup/reload cache
for -imapd, too.
This results in a ~3x faster startup for -imapd with 50K inboxes.
|
|
By taking advantage of the new ART_MIN/ART_MAX value in MiscIdx,
we can avoid the overhead of opening per-inbox msgmap DB files.
The result gives us a ~40 speedup with 50K newgroups.
|
|
This will be used to speed up NNTP group listings and IMAP startup
with thousands of inboxes.
|
|
While PublicInbox::NNTP doesn't use it, config sharing inside
public-inbox-netd will mean inbox objects also get shared.
|
|
This allows new TLS certificates to be loaded for new clients
without having to timeout nor drop existing clients with
established connections made with the old certs. This should
benefit users with admins who expire certificates frequently (as
encouraged by Let's Encrypt).
|
|
->zflush can take a buffer arg, so there's no need to
make a separate call to ->translate in some cases.
|
|
->zflush is already for GzipFilter in PublicInbox::WWW,
while we use DEFLATE for NNTP and IMAP. This ought to
make the code easier-to-follow.
|
|
A few things I noticed while reviewing and evaluating
the PSGI code for JMAP support.
|
|
Socket ->write failures are expected and common for TCP traffic,
especially if it's facing unreliable remote connections. So
just bail out silently if our {gz} field was already clobbered
during the small bit of recursion we hit on ->write failures
from async responses.
This ought to fix some GzipFilter::zflush errors (via $forward
->close from PublicInbox::HTTP) I've been noticing on
deployments running -netd. I'm still unsure as to why I hadn't
seen them before, but it might've only been ignorance on my
part...
Link: https://public-inbox.org/meta/20220802065436.GA13935@dcvr/
|
|
We rely on the %logs hash for SIGUSR1 log reopening. Without this sharing,
some FDs would be hidden inside its respective {HTTP,IMAP,POP3}D
object and not reopened on USR2
|
|
This allows admins to use "-l nntp://0.0.0.0/" to bind on port 119
without specifying ":119" on the CLI.
|
|
These are helpful for diagnosing configuration problems,
as well as a bug (to be fixed in the following commit).
|
|
This is preparation for supporting loading new certs on SIGHUP.
|
|
This allows memory savings by allowing multiple, completely
unrelated-PSGI apps to run within the same process as IMAP,
NNTP, and POP3.
|
|
This brings the HTTP server closer to the IMAP/NNTP/POP3
implementations and eliminates package-wide globals in
PublicInbox::HTTPD. The end goal is to be able to host
completely different PSGI applications on different listen
ports.
|
|
git deprecated core.fsyncObjectFiles in favor of core.fsync
with 2.36.0+, while GIT_TEST_FSYNC was added in 2.35.0. So
use the environment variable since it's been supported slightly
longer than the new configuration knob.
|
|
Hopefully it makes sense to new users deploying or using POP3...
|