From: Eric Wong <email@example.com> To: firstname.lastname@example.org Subject: IMAP server notes, maybe JMAP? Date: Tue, 9 Jun 2020 11:34:42 +0000 Message-ID: <20200609113442.GA16856@dcvr> (raw) OK, so I almost have something that won't kill clients or trigger OOMs on the server. I think I'll have to implement MSN (message sequence numbers) properly for some clients, cheaply. I know there's also interest in getting search usable via an HTTP(S) API, so maybe JMAP is worth looking into since it seems like an easier-to-implement take on IMAP; and both have search. We already use JSON for manifest.js.gz...  https://lwn.net/Articles/680057/ - (a look at JMAP). GraphQL has also come up privately, but at a glance it seems to suffer some of the same problems with IMAP in offering excessive granularity and cache-unfriendliness. But anyways, I'm close to having a pretty good read-only IMAP server. JMAP cannot be any harder than IMAP right? :) Anyways, the IMAP server is based on the existing NNTP server code; but also provides a fresh place to test new ideas for improving scalability and performance: * It's fair to other clients if using slow blob storage, and I've gotten to be fairly happy with the API. Slow Xapian storage isn't accounted for, yet, but will be. Aggressive pipelining in mbsync(1) helped me get async bugs ironed out. * LIST wildcard matching/globbing tested with 100K inboxes, other pieces still need work. * Incremental config reloading on SIGHUP works, but could be more granular. * IMAP IDLE supported for EXISTS (but not EXPUNGE for spam). * Inotify (or EVFILT_VNODE) used for IMAP IDLE notifications. More inotify/EVFILT_VNODE usage to come for manifest.js.gz updates, git-cat-file restarts, DB reopens, etc. I've used inotify a bunch in the past, but not in Perl. This is my first time with EVFILT_VNODE, but inotify seems more capable. * --reindex is recommended for the RFC822.SIZE fetch attribute to account for CRLF conversion. Actually, we've been wrong about NNTP :bytes reporting for years, too, but IMAP clients (at least Mail::IMAPClient) actually complain. * Common cases of Xapian search work, but also require --reindex. Reindex is doable while serving, results just won't show until reindexed. * COMPRESS, STARTTLS, TLS are all inherited from -nntpd * PublicInbox::Eml can descend in to message/* subparts properly, which makes it possible to implement stuff like BODYSTRUCTURE properly. And who uses BODYSTRUCTURE? And a note: * Don't create 100K public-inboxes in the worktree itself, MakeMaker looks inside all directories when doing "perl Makefile.PL" :x TODO: * Queries involving OR, NOT, and parentheses don't work, yet, since Xapian's default query parser works differently than the prefix (Polish) notation of IMAP. * IMAP extensions are worth looking into, especially ones around search. Some of them look interesting w.r.t. search, along with overlap with JMAP. * There's also thread-related stuff and that may be able to implement "Search based on data in follow-ups" https://email@example.com/ * An "All Mail" mailbox could be cool for IMAP/JMAP search (it's planned for HTTP, anyways) I hit numerous bugs in 3rd-party libraries while working on the server, all of them involving around compression: * Python imaplib2 (via offlineimap) - compress timeout: https://bugs.debian.org/961713 * Mail::IMAPClient - compress reference cycle: https://rt.cpan.org/Ticket/Display.html?id=132654 - compress read starvation: https://rt.cpan.org/Ticket/Display.html?id=132720 * Compress::Raw::Zlib - inflate appending to OOK scalars https://rt.cpan.org/Ticket/Display.html?id=132734 mbsync(1) helped me expose this bug Both Mail::IMAPClient and Mail::IMAPTalk do weird sleeps with select(2) and SSL; maybe as a holdover from the days when IO::Socket::SSL lacked SSL_WANT_READ/SSL_WANT_WRITE? headaches for dealing with MUAs: * Inboxes are split into 50K slices to avoid overloading MUAs. The word "slice" was not previously used in our codebase. Conceptually, it's like v2 "epochs". Epochs are to deal with the limitations of git clients, slices are to deal with the limitations of IMAP clients. * HEADER.FIELDS retrieval still requires taking the whole blob from git. Clients can request any header(s) they wish; unlike NNTP, where the server defines the overview. I've managed to speed this up significantly with a little pure-Perl opcode compiler, though :) * MSNs (message sequence numbers) seem required to get decent performance from mutt and maybe other MUAs. Showing fake dummy messages for removed spam is poor UX, too...
next reply index Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-06-09 11:34 Eric Wong [this message] 2020-06-15 6:21 ` Parse::RecDescent dependency (was: IMAP server notes, maybe JMAP?) Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200609113442.GA16856@dcvr \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself Archives are clonable: git clone --mirror http://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta Example config snippet for mirrors Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/public-inbox.git