about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2020-06-28imaptracker: use ~/.local/share/public-inbox/imap.sqlite3
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/. Existing users of ~/.public-inbox/imap.sqlite3 will remain supported, but the preference for new data is to use ~/.local/share and other paths standardized by XDG. Cc: "Eric W. Biederman" <ebiederm@xmission.com>
2020-06-28IMAPTracker: Add a helper to track our place in reading imap mailboxes
This removes the need to delete from an imap mailbox when downloading it's messages. [ew: minor style changes] Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2020-06-28inbox: warn on ->on_inbox_unlock exception
Otherwise, we may never know what went wrong.
2020-06-28inboxwritable: ensure ssoma.lock exists on init
This will allow us to use InboxIdle on empty/unindexed v1 inboxes.
2020-06-28eml: header_str_set: correctly encode UTF-8 headers
Apparently, using $1 from an octet string still results in a multi-byte string. Thus we need to perform utf8::encode after the regexp character match to ensure wide characters don't get passed to encode_base64. This fixes a bug in which caused -watch to crash when using PublicInbox::Filter::SubjectTag to remove "[list prefix]" tags from Subject: lines. I only found this bug because the proposed -watch updates for NNTP/IMAP support introduced a possible bug which triggered a full rescan of old archives: https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
2020-06-27linkify: support imap, imaps, news, and snews URIs
Since we'll have an IMAP server released soon, maybe imaps:// and imap:// URLs can become popular. news:// is defined with nntp:// in RFC 5538, and we can at least support the news:// form in rendered HTML. snews:// may appear in old mail archives, too, so we'll attempt to support it in case clients do.
2020-06-27imap: EXAMINE: avoid potential race conditions
We need to rely on num_highwater for UIDNEXT since the highest `num' stored in over.sqlite3 may be rolled back if the most recent messages were spam. We also need to load the uo2m immediately on EXAMINE to ensure EXISTS responses are always consistent with regard to future updates.
2020-06-27imap: always send EXISTS on uo2m_extend
Clients which are NOT in an IDLE state still need to be notified of message existence. Unlike the EXPUNGE response, untagged EXISTS responses seem to be allowed at any time according to RFC 3501. We'll also perform uo2m_extend on the NOOP command, since NOOP is the recommended command for message polling.
2020-06-25ds: flush_write: early return on closed socket
This quiets warnings from IMAP::fetch_blob (called via long_response) failing to access `$self->{ibx}->git' because ->{ibx} gets deleted by IMAP::close.
2020-06-25git_async_cat: remove circular reference
While this circular reference was carefully managed to not leak memory; it was still triggering a warning at -imapd/-nntpd shutdown due to the EPOLL_CTL_DEL op failing after the $Epoll FD gets closed. So remove the circular reference by providing a ref to `undef', instead.
2020-06-25lock: reduce inotify wakeups
We can reduce the amount of platform-specific code by always relying on IN_MODIFY/NOTE_WRITE notifications from lock release. This reduces the number of times our read-only daemons will need to wake up when -watch sees no-op message changes (e.g. replied, seen, recent flag changes).
2020-06-23testcommon: DS->Reset when using fork-only subprocess
This fixes a bug on FreeBSD 11 here -nntpd + TEST_RUN_MODE=2 (default) was occasionally causing failures in t/v2writable.t due to the kqueue descriptor being auto-closed by the OS on fork.
2020-06-23init: add --skip-artnum parameter
For archivists with only newer mail archives, this option allows reserving reserve NNTP article numbers for yet-to-be-archived old messages. Indexers will need to be updated to support this feature in future commits. -V1 inboxes will now be initialized with SQLite and Xapian support if this option is used, or if --indexlevel= is specified.
2020-06-23imap: refill_xap: remove needless loop
There's no need to loop when the first iteration guarantees a `return'.
2020-06-21testcommon: require IO::Socket::SSL >= 2.007
Net::NNTP does not support older IO::Socket::SSL. 1.94 on CentOS 7.x fails HTTPS and IMAPS tests, too. cf. https://rt.cpan.org/Ticket/Display.html?id=100529
2020-06-21nntp: support slow blob retrievals
Having `git cat-file' as a separate process naturally lends itself to asynchronous dispatch. Our event loop for -nntpd no longer blocks on slow git storage. Pipelining in -imapd was tricky and bugs were exposed by mbsync(1). Update t/nntpd.t to support pipelining ARTICLE requests to ensure we don't have the same problems -imapd did during development.
2020-06-21nntp: event_step: prepare for async git reads
This matches PublicInbox::IMAP::event_step and will allow us to handle blob retrievals from git asynchronously without falling over on pipelined requests.
2020-06-21daemon: use ->can to check for IO::Socket::SSL
Doing a ref($obj) string comparison ties us to IO::Socket::SSL (and OpenSSL) In the future, we may support GnuTLS or other TLS implementations. This was already done in the IMAP code.
2020-06-16imap: fix UID-offset-to-MSN mapping bugs
We need to clear the UID-offset-to-MSN mapping when leaving mailboxes via EXAMINE/SELECT/CLOSE. Furthermore, uo2m_last_uid() needs to account for tiny mailboxes where the scalar representation of {uo2m} may be evaluated to `false' in a boolean context.
2020-06-16imap: *SEARCH: reinstate "TEXT" search-key
I accidentally dropped "TEXT" handling while porting the IMAP search query parser to Parse::RecDescent. This reinstates it and adds a test to prevent future regression, and the additional test fixes a counting error for non-Xapian-enabled systems.
2020-06-16imap: *SEARCH: fix CHARSET handling
We no longer pass an arrayref to search_common() or parse_query(), so handle the CHARSET directive in the Parse::RecDescent-generated parser directly.
2020-06-16imap: *SEARCH: use Parse::RecDescent
For properly parsing IMAP search requests, it's easier to use a recursive descent parser generator to deal with subqueries and the "OR" statement. Parse::RecDescent was chosen since it's mature, well-known, widely available and already used by our optional dependencies: Inline::C and Mail::IMAPClient. While it's possible to build Xapian queries without using the Xapian string query parser; this iteration of the IMAP parser still builds a string which is passed to Xapian's query parser for ease-of-diagnostics. Since this is a recursive descent parser dealing with untrusted inputs, subqueries have a nesting limit of 10. I expect that is more than adequate for real-world use.
2020-06-16imap: reinstate non-UID SEARCH
Since we support MSNs properly, now, it seems acceptable to support regular SEARCH requests in case there are any clients which still use non-UID SEARCH.
2020-06-16imap: stop_idle: fix parameter parsing :x
stop_idle was a noop when the client issues a "DONE" continuation or just disconnects. This would not have led to a long term memory leak since FDs get closed and reused, anyways, and all of our InboxIdle mappings are keyed by FD.
2020-06-16imap: improve IDLE handling at graceful shutdown
Since IMAP IDLE users aren't expected to issue any commands, we can terminate their connections immediately on graceful shutdown. Furthermore, we need to drop the inotify FD from the epoll set to avoid warnings during global destruction. Embarassingly, this required fixing wacky test ordering from 2a717d13f10fcdc6 ("nntpd+imapd: detect replaced over.sqlite3")
2020-06-16imap: clarify "DONE" usage with IDLE
"DONE" is a continuation and not a normal IMAP command, so ensure it can't be called like a normal IMAP command which has a tag.
2020-06-15inboxidle: support Linux::Inotify2 1.x
Linux::Inotify2 1.x lacked ->on_overflow and ->broadcast methods. Just don't use them for now. We may eventually provide a pure Perl alternative which doesn't require closures, XS, or the common::sense dependency. Overflowing the inotify queue seems difficult to trigger at the moment: /proc/sys/fs/inotify/max_queued_events defaults to 16384 on a my CentOS 7.x VM with 2GB RAM.
2020-06-15testcommon: allow OR-ing module dependencies
IMAP requires either the Email::Address::XS or Mail::Address package (part of perl-MailTools RPM or libmailtools-perl deb); and Email::Address::XS is not officially packaged for some older distros, most notably CentOS 7.x.
2020-06-13nntpd+imapd: detect replaced over.sqlite3
For v1 inboxes (and possibly v2 in the future, for VACUUM), public-inbox-compact replaces over.sqlite3 with a new file. This currently doesn't need an extra inotify watch descriptor (or FD for kevent) at the moment, so it can coexist nicely for systems w/o IO::KQueue or Linux::Inotify2.
2020-06-13imap: introduce memory-efficient uo2m mapping
Since we limit our mailboxes slices to 50K and can guarantee a contiguous UID space for those mailboxes, we can store a mapping of "UID offsets" (not full UIDs) to Message Sequence Numbers as an array of 16-bit unsigned integers in a 100K scalar. For UID-only FETCH responses, we can momentarily unpack the compact 100K representation to a ~1.6M Perl array of IV/UV elements for a slight speedup. Furthermore, we can (ab)use hash key deduplication in Perl5 to deduplicate this 100K scalar across all clients with the same mailbox slice open. Technically we can increase our slice size to 64K w/o increasing our storage overhead, but I suspect humans are more accustomed to slices easily divisible by 10.
2020-06-13imap: FETCH: proper MSN => UID mapping for requests
This finally seems to make mutt header caching behave properly. We expect to be able to safely load 50K IV/UVs in memory without OOM, since that's "only" 1.6 MB that won't live beyond a single event loop iteration. So create a simple array which can quickly map MSNs in requests to UIDs and not leave out messages. MSNs in the FETCH response will NOT be correct, since it's inefficient to implement properly and mutt doesn't seem to care. Since the conversion code is easily shared, "UID SEARCH" can allow the same MSN => UID mapping non-UID "FETCH" does.
2020-06-13over: uid_range: remove LIMIT
The IMAP code already limits the range to UID_SLICE (50K), so that's about 1.6MB of of IVs for an ephemeral allocation that won't live beyond one iteration of the event loop.
2020-06-13imap: remove non-UID SEARCH for now
Supporting MSNs in long-lived connections beyond the lifetime of a single request/response cycle is not scalable to a C10K scenario. It's probably not needed, since most clients seem to use UIDs. A somewhat efficient implementation I can come up uses pack("S*" ...) (AKA "uint16_t mapping[50000]") has an overhead of 100K per-client socket on a mailbox with 50K messages. The 100K is a contiguous scalar, so it could be swapped out for idle clients on most architectures if THP is disabled. An alternative could be to use a tempfile as an allocator partitioned into 100K chunks (or SQLite); but I'll only do that if somebody presents a compelling case to support MSN SEARCH.
2020-06-13imapd: don't bother sorting LIST output
The sort was unstable on my test instance anyways, and clients don't seem to mind. So stop wasting CPU cycles.
2020-06-13imap: misc cleanups and notes
Note some of our limitations for potential hackers. We'll be renaming "UID_BLOCK" to "UID_SLICE", since "block" is overused term and "slice" isn't used in our codebase. Also, document how "slice" and "epochs" are similar concepts for different clients.
2020-06-13imap: wire up Xapian, MSN SEARCH and multi sequence-sets
Simple queries work, more complex queries involving parentheses, "OR", "NOT" don't work, yet. Tested with "=b", "=B", and "=H" search and limits in mutt on both v1 and v2 with multiple Xapian shards.
2020-06-13imap: UID SEARCH: support multiple ranges
We can share a bit of code with FETCH to refill UID ranges which hit the SQLite overview.
2020-06-13imap: STATUS/EXAMINE: rely on SQLite overview
We can get exact values for EXISTS, UIDNEXT using SQLite rather than calculating off $ibx->mm->max ourselves. Furthermore, $ibx->mm is less useful than $ibx->over for IMAP (and for our read-only daemons in general) so do not depend on $ibx->mm outside of startup/reload to save FDs and reduce kernel page cache footprint.
2020-06-13imap: FETCH: try to make fake MSNs sequentially
This appears to significantly improve header caching behavior with mutt. With the current public-inbox.org/git mirror(*), mutt will only re-FETCH the last ~300 or so messages in the final "inbox.comp.version-control.git.7" mailbox, instead of ~49,000 messages every time. It's not perfect, but a 500ms query is better than a >10s query and mutt itself spends as much time loading its header cache. (*) there are many gaps in NNTP article numbers (UIDs) due to spam removal from public-inbox-learn.
2020-06-13imap: further speed up HEADER.FIELDS FETCH requests
Since headers are big and include a lot of lines MUAs don't care about, we can skip the CRLF_HDR ops and just do the CRLF conversion in partial_hdr_get and partial_hdr_not. This is another 10-15% speedup for mutt w/o header caching.
2020-06-13imap: FETCH: more granular CRLF conversion
This speeds up requests from mutt for HEADER.FIELDS by around 10% since we don't waste time doing CRLF conversion on large message bodies that get discarded, anyways.
2020-06-13imap: cleanup ->{uid_base} usage
Ensure {uid_base} is always set, so we don't need to add `//' checks everywhere. Furthermore, this fixes a hard-to-test bug where the STATUS command would inadvertantly clobber {uid_base}.
2020-06-13imap: reinstate some message sequence number support
The performance problem with mutt not using header caches isn't fixed, yet, but mutt header caching seems to depend on MSNs (message sequence numbers). We'll switch to storing the 0-based {uid_base} instead of the 1-based {uid_min} since it simplifies most of our code.
2020-06-13imap: support 8000 octet lines
RFC 2683 section 3.2.1.5 recommends it: > For its part, a server should allow for a command line of at least > 8000 octets. This provides plenty of leeway for accepting reasonable > length commands from clients. The server should send a BAD response > to a command that does not end within the server's maximum accepted > command length. To conserve memory, we won't bother reading the entire line before sending the BAD response and disconnecting them.
2020-06-13imap: LIST shows "INBOX" in all caps
While selecting a mailbox is done case-insensitively, "INBOX" is special for the LIST command, according to RFC 3501 6.3.8: > The special name INBOX is included in the output from LIST, if > INBOX is supported by this server for this user and if the > uppercase string "INBOX" matches the interpreted reference and > mailbox name arguments with wildcards as described above. The > criteria for omitting INBOX is whether SELECT INBOX will > return failure; it is not relevant whether the user's real > INBOX resides on this or some other server. Thus, the existing news.public-inbox.org convention of naming newsgroups starting with "inbox." needs to be special-cased to not confuse clients. While we're at it, do not create ".0" for dummy newsgroups if they're selected, either.
2020-06-13imap: UID FETCH requires at least one data item
It seems required based on my reading of RFC 3501 for the non-UID "FETCH" command.
2020-06-13imap: rely on smsg->{bytes} for RFC822.SIZE
Since we started indexing the CRLF-adjusted size of messages, we can take an order-of-magnitude speedup for certain MUAs which fetch this attribute without needing much else. Admins are encouraged to --reindex existing inboxes for IMAP support, anyways. It won't be fatal if it's not reindexed, but some client bugs and warnings can be fixed and they'll be able to support more of IMAP.
2020-06-13index: account for CRLF conversion when storing bytes
NNTP and IMAP both require CRLF conversions on the wire. They're also the only components which care about $smsg->{bytes}, so store the CRLF-adjusted value in over.sqlite3 and Xapian DBs.. This will allow us to optimize RFC822.SIZE fetch item in IMAP without triggering size mismatch errors in some clients' default configurations (e.g. Mail::IMAPClient), but not most others. It could also fix hypothetical problems with NNTP clients that report discrepancies between overview and article data.
2020-06-13searchidx: v1 (re)-index uses git asynchronously
We can cleanup some of our v1 code slightly and let git do I/O+decoding in parallel. This gives a slight 2-4% re-indexing performance boost even on an SSD.
2020-06-13imap: split ->logged_in attribute into a separate class
This is one boolean attribute not worth wasting space for. With 20000 sockets, this reduces RSS by around 5% at a glance, and locked hashes doesn't do us much good when clients use compression, anyways.