| Date | Commit message (Collapse) |
|---|
|
This matches PublicInbox::IMAP::event_step and will allow us to
handle blob retrievals from git asynchronously without falling
over on pipelined requests.
|
|
Doing a ref($obj) string comparison ties us to IO::Socket::SSL
(and OpenSSL) In the future, we may support GnuTLS or other TLS
implementations. This was already done in the IMAP code.
|
|
We need to clear the UID-offset-to-MSN mapping when
leaving mailboxes via EXAMINE/SELECT/CLOSE.
Furthermore, uo2m_last_uid() needs to account for tiny mailboxes
where the scalar representation of {uo2m} may be evaluated to
`false' in a boolean context.
|
|
I accidentally dropped "TEXT" handling while porting
the IMAP search query parser to Parse::RecDescent.
This reinstates it and adds a test to prevent future
regression, and the additional test fixes a counting
error for non-Xapian-enabled systems.
|
|
We no longer pass an arrayref to search_common() or
parse_query(), so handle the CHARSET directive in
the Parse::RecDescent-generated parser directly.
|
|
For properly parsing IMAP search requests, it's easier to use a
recursive descent parser generator to deal with subqueries and
the "OR" statement.
Parse::RecDescent was chosen since it's mature, well-known,
widely available and already used by our optional dependencies:
Inline::C and Mail::IMAPClient. While it's possible to build
Xapian queries without using the Xapian string query parser;
this iteration of the IMAP parser still builds a string which is
passed to Xapian's query parser for ease-of-diagnostics.
Since this is a recursive descent parser dealing with untrusted
inputs, subqueries have a nesting limit of 10. I expect that is
more than adequate for real-world use.
|
|
Since we support MSNs properly, now, it seems acceptable
to support regular SEARCH requests in case there are any
clients which still use non-UID SEARCH.
|
|
|
|
stop_idle was a noop when the client issues a "DONE"
continuation or just disconnects. This would not have
led to a long term memory leak since FDs get closed and
reused, anyways, and all of our InboxIdle mappings are
keyed by FD.
|
|
Since IMAP IDLE users aren't expected to issue any commands, we
can terminate their connections immediately on graceful
shutdown.
Furthermore, we need to drop the inotify FD from the epoll set
to avoid warnings during global destruction. Embarassingly,
this required fixing wacky test ordering from 2a717d13f10fcdc6
("nntpd+imapd: detect replaced over.sqlite3")
|
|
"DONE" is a continuation and not a normal IMAP command, so
ensure it can't be called like a normal IMAP command which
has a tag.
|
|
Mail::IMAPClient understandably stumbles into a warning
by our bogus test request. Just silence it on our end
since it's not normal operation for Mail::IMAPClient.
|
|
->has_capability on Mail::IMAPClient 3.37 (tested on CentOS 7.x)
only returned boolean values, and not the value of the capability.
|
|
Linux::Inotify2 1.x lacked ->on_overflow and ->broadcast
methods. Just don't use them for now. We may eventually
provide a pure Perl alternative which doesn't require closures,
XS, or the common::sense dependency.
Overflowing the inotify queue seems difficult to trigger at
the moment: /proc/sys/fs/inotify/max_queued_events defaults
to 16384 on a my CentOS 7.x VM with 2GB RAM.
|
|
IMAP requires either the Email::Address::XS or Mail::Address
package (part of perl-MailTools RPM or libmailtools-perl deb);
and Email::Address::XS is not officially packaged for some older
distros, most notably CentOS 7.x.
|
|
Lots of big changes coming Thanks to The Linux Foundation for
sponsoring me to hack on this in 2020 :)
|
|
For v1 inboxes (and possibly v2 in the future, for VACUUM),
public-inbox-compact replaces over.sqlite3 with a new file.
This currently doesn't need an extra inotify watch descriptor
(or FD for kevent) at the moment, so it can coexist nicely for
systems w/o IO::KQueue or Linux::Inotify2.
|
|
Since we limit our mailboxes slices to 50K and can guarantee a
contiguous UID space for those mailboxes, we can store a mapping
of "UID offsets" (not full UIDs) to Message Sequence Numbers as
an array of 16-bit unsigned integers in a 100K scalar.
For UID-only FETCH responses, we can momentarily unpack the
compact 100K representation to a ~1.6M Perl array of IV/UV
elements for a slight speedup.
Furthermore, we can (ab)use hash key deduplication in Perl5 to
deduplicate this 100K scalar across all clients with the same
mailbox slice open.
Technically we can increase our slice size to 64K w/o increasing
our storage overhead, but I suspect humans are more accustomed
to slices easily divisible by 10.
|
|
This finally seems to make mutt header caching behave properly.
We expect to be able to safely load 50K IV/UVs in memory without
OOM, since that's "only" 1.6 MB that won't live beyond a single
event loop iteration. So create a simple array which can
quickly map MSNs in requests to UIDs and not leave out messages.
MSNs in the FETCH response will NOT be correct, since it's
inefficient to implement properly and mutt doesn't seem to
care.
Since the conversion code is easily shared, "UID SEARCH" can
allow the same MSN => UID mapping non-UID "FETCH" does.
|
|
The IMAP code already limits the range to UID_SLICE (50K),
so that's about 1.6MB of of IVs for an ephemeral allocation
that won't live beyond one iteration of the event loop.
|
|
Supporting MSNs in long-lived connections beyond the lifetime of
a single request/response cycle is not scalable to a C10K
scenario. It's probably not needed, since most clients seem to
use UIDs.
A somewhat efficient implementation I can come up uses
pack("S*" ...) (AKA "uint16_t mapping[50000]") has an overhead
of 100K per-client socket on a mailbox with 50K messages. The
100K is a contiguous scalar, so it could be swapped out for
idle clients on most architectures if THP is disabled.
An alternative could be to use a tempfile as an allocator
partitioned into 100K chunks (or SQLite); but I'll only do that
if somebody presents a compelling case to support MSN SEARCH.
|
|
The sort was unstable on my test instance anyways, and
clients don't seem to mind. So stop wasting CPU cycles.
|
|
Note some of our limitations for potential hackers.
We'll be renaming "UID_BLOCK" to "UID_SLICE", since "block" is
overused term and "slice" isn't used in our codebase. Also,
document how "slice" and "epochs" are similar concepts for
different clients.
|
|
Simple queries work, more complex queries involving parentheses,
"OR", "NOT" don't work, yet.
Tested with "=b", "=B", and "=H" search and limits in mutt
on both v1 and v2 with multiple Xapian shards.
|
|
We can share a bit of code with FETCH to refill UID
ranges which hit the SQLite overview.
|
|
We can get exact values for EXISTS, UIDNEXT using SQLite
rather than calculating off $ibx->mm->max ourselves.
Furthermore, $ibx->mm is less useful than $ibx->over for IMAP
(and for our read-only daemons in general) so do not depend on
$ibx->mm outside of startup/reload to save FDs and reduce kernel
page cache footprint.
|
|
This appears to significantly improve header caching behavior
with mutt. With the current public-inbox.org/git mirror(*),
mutt will only re-FETCH the last ~300 or so messages in the
final "inbox.comp.version-control.git.7" mailbox, instead of
~49,000 messages every time.
It's not perfect, but a 500ms query is better than a >10s query
and mutt itself spends as much time loading its header cache.
(*) there are many gaps in NNTP article numbers (UIDs) due to
spam removal from public-inbox-learn.
|
|
Since headers are big and include a lot of lines MUAs don't
care about, we can skip the CRLF_HDR ops and just do the
CRLF conversion in partial_hdr_get and partial_hdr_not.
This is another 10-15% speedup for mutt w/o header caching.
|
|
This speeds up requests from mutt for HEADER.FIELDS by around 10%
since we don't waste time doing CRLF conversion on large message
bodies that get discarded, anyways.
|
|
Ensure {uid_base} is always set, so we don't need to add `//'
checks everywhere. Furthermore, this fixes a hard-to-test bug
where the STATUS command would inadvertantly clobber {uid_base}.
|
|
The performance problem with mutt not using header caches isn't
fixed, yet, but mutt header caching seems to depend on MSNs
(message sequence numbers). We'll switch to storing the 0-based
{uid_base} instead of the 1-based {uid_min} since it simplifies
most of our code.
|
|
RFC 2683 section 3.2.1.5 recommends it:
> For its part, a server should allow for a command line of at least
> 8000 octets. This provides plenty of leeway for accepting reasonable
> length commands from clients. The server should send a BAD response
> to a command that does not end within the server's maximum accepted
> command length.
To conserve memory, we won't bother reading the entire line
before sending the BAD response and disconnecting them.
|
|
While selecting a mailbox is done case-insensitively, "INBOX" is
special for the LIST command, according to RFC 3501 6.3.8:
> The special name INBOX is included in the output from LIST, if
> INBOX is supported by this server for this user and if the
> uppercase string "INBOX" matches the interpreted reference and
> mailbox name arguments with wildcards as described above. The
> criteria for omitting INBOX is whether SELECT INBOX will
> return failure; it is not relevant whether the user's real
> INBOX resides on this or some other server.
Thus, the existing news.public-inbox.org convention of naming
newsgroups starting with "inbox." needs to be special-cased to
not confuse clients.
While we're at it, do not create ".0" for dummy newsgroups if
they're selected, either.
|
|
It seems required based on my reading of RFC 3501 for
the non-UID "FETCH" command.
|
|
Since we started indexing the CRLF-adjusted size of messages,
we can take an order-of-magnitude speedup for certain MUAs
which fetch this attribute without needing much else.
Admins are encouraged to --reindex existing inboxes for IMAP
support, anyways. It won't be fatal if it's not reindexed, but
some client bugs and warnings can be fixed and they'll be able
to support more of IMAP.
|
|
NNTP and IMAP both require CRLF conversions on the wire.
They're also the only components which care about
$smsg->{bytes}, so store the CRLF-adjusted value in over.sqlite3
and Xapian DBs..
This will allow us to optimize RFC822.SIZE fetch item in IMAP
without triggering size mismatch errors in some clients' default
configurations (e.g. Mail::IMAPClient), but not most others.
It could also fix hypothetical problems with NNTP clients that
report discrepancies between overview and article data.
|
|
We can cleanup some of our v1 code slightly and let git do
I/O+decoding in parallel. This gives a slight 2-4%
re-indexing performance boost even on an SSD.
|
|
This is one boolean attribute not worth wasting space for.
With 20000 sockets, this reduces RSS by around 5% at a glance,
and locked hashes doesn't do us much good when clients
use compression, anyways.
|
|
RFC 3501 section 5.4 requires this to be >= 30 minutes,
10x higher than what is recommended for NNTP. Fortunately
our design is reasonably memory-efficient despite being Perl.
|
|
We should not waste memory for IDLE unless it's used on the most
recent inbox slice. We also need to keep the IDLE connection
alive regardless of $PublicInbox::DS::EXPTIME.
|
|
We can speed up this common mutt request by another 2-3x by not
loading the entire smsg from SQLite, just the UID.
|
|
We can avoid loading the entire message from git when mutt makes
a "UID FETCH" request for "(UID FLAGS)". This speeds mutt up by
more than an order-of-magnitude in informal measurements.
|
|
This is just a hair faster and cacheable in the future, if we
need it. Most notably, this avoids doing PublicInbox::Eml->new
for simple "RFC822", "BODY[]", and "RFC822.SIZE" requests.
|
|
Dummy messages make for bad user experience with MUAs which
still use sequence numbers. Not being able to fetch a message
doesn't seem fatal in mutt, so just ignore (sometimes large)
gaps.
|
|
We'll need to support searching UID ranges for IMAP,
so make sure it's indexed, too.
|
|
Searching for messages smaller than a certain size is allowed by
offlineimap(1), mbsync(1), and possibly other tools. Maybe
public-inbox-watch will support it, too.
I don't see a reason to expose searching by size via WWW search
right now (but maybe in the future, I could be convinced to).
Note: we only store the byte-size of the message in git,
this is typically LF-only and we won't have the correct
size after CRLF conversion for NNTP or IMAP.
|
|
This speeds up xt/imapd-validate.t by around 10% when used with
an abandoned patch to remove ->query_xover. We may also depend
on this further if we abandon storing doc_data in Xapian to save
disk space.
|
|
Since it seems somewhat common for IMAP clients to limit
searches by sent Date: or INTERNALDATE, we can rely on
the NNTP/WWW-optimized overview DB.
For other queries, we'll have to depend on the Xapian DB.
|
|
We won't support searching across mailboxes, just yet;
but maybe in the future.
|
|
None of the new cases are wired up, yet, but existing cases
still work.
|