From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 4E5F21F55B for ; Wed, 10 Jun 2020 07:05:19 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/82] public-inbox-imapd: read-only IMAP server Date: Wed, 10 Jun 2020 07:03:57 +0000 Message-Id: <20200610070519.18252-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: So I finally wrote my first IMAP server! And I'm actually fairly satisfied with how it's turning out to support a bunch of other performance + scalability work I've wanted to do. Some previous notes here: https://public-inbox.org/meta/20200609113442.GA16856@dcvr/ I finally seem to have gotten it to play nicely with mutt header caching, so it's fit for public consumption :) imaps://news.public-inbox.org/INBOX.comp.mail.public-inbox.meta.0 You can use any username+password, and AUTH=ANONYMOUS also works if your client does that. It doesn't support UTF-7 (mailbox names) or advertise UTF-8 in CAPABILITIES, yet; I still have RFCs to read :P And there's a bunch of new things which could use some testing from non-mutt/mbsync/offlineimap users. Maybe you'll find some client-side bugs like I did :P v1 reindexing also gets a little bit of parallelism :) Anyways, I'll probably be porting some of the scalability and slow-storage work to older parts of the code before fiddling with more IMAP extensions. Eric Wong (82): doc: add some IMAP standards nntpd: restrict allowed newsgroup names preliminary imap server implementation inboxidle: new class to detect inbox changes imap: support IDLE msgmap: split ->max into its own method imap: delay InboxIdle start, support refresh imap: implement STATUS command imap: use Text::ParseWords::parse_line to handle quoted words imap: support LIST command t/imapd: support FakeInotify and KQNotify imap: support fetch for BODYSTRUCTURE and BODY eml: each_part: single part $idx is 1 imap: allow fetch of partial of BODY[...] and headers imap: always include `resp-text' in responses imap: split out unit tests and benchmarks imap: fix multi-message partial header fetches imap: simplify partial fetch structure imap: support sequence number FETCH imap: do not include ".PEEK" in responses imap: support the CLOSE command imap: speed up HEADER.FIELDS[.NOT] range fetches git: async: flatten the inflight array git: do our own read buffering for cat-file imap: use git-cat-file asynchronously git: idle rbuf for async imap: support LSUB command imap: FETCH: support comma-delimited ranges add imapd compression test testcommon: tcp_(server|connect): BAIL_OUT on failure *deflate: drop invalid comment about rbuf imap: fix pipelining with async git git: cat_async: provide requested OID + "missing" on missing blobs git: move async_cat reference to PublicInbox::Git git: async: automatic retry on alternates change imapclient: wrapper for Mail::IMAPClient xt: add imapd-validate and imapd-mbsync-oimap imap: support out-of-bounds ranges xt/perf-imap-list: time refresh_inboxlist imap: case-insensitive mailbox name comparisons imap: break giant inboxes into sub-inboxes of 50K messages imap: start introducing iterative config reloading imap: require ".$UID_MIN-$UID_END" suffix imapd: ensure LIST is sorted alphabetically, for now imap: omit $UID_END from mailbox name, use index t/config.t: always compare against git bool behavior xt/*: show some tunable parameters imap: STATUS and LIST are case-insensitive, too imap: EXAMINE/STATUS: return correct counts imap: avoid uninitialized warnings on incomplete commands imap: start parsing out queries for SQLite and Xapian imap: SEARCH: clamp results to the 50K UID range imap: allow UID range search on timestamps over: get_art: use dbh->prepare_cached search: index byte size of a message for IMAP search search: index UID for IMAP search, too imap: remove dummies from sequence number FETCH imap: compile UID FETCH to opcodes imap: UID FETCH: optimize for smsg-only case imap: UID FETCH: optimize (UID FLAGS) harder imap: IDLE: avoid extraneous wakeups, keep-alive imap: 30 minute auto-logout timer imap: split ->logged_in attribute into a separate class searchidx: v1 (re)-index uses git asynchronously index: account for CRLF conversion when storing bytes imap: rely on smsg->{bytes} for RFC822.SIZE imap: UID FETCH requires at least one data item imap: LIST shows "INBOX" in all caps imap: support 8000 octet lines imap: reinstate some message sequence number support imap: cleanup ->{uid_base} usage imap: FETCH: more granular CRLF conversion imap: further speed up HEADER.FIELDS FETCH requests imap: FETCH: try to make fake MSNs sequentially imap: STATUS/EXAMINE: rely on SQLite overview imap: UID SEARCH: support multiple ranges imap: wire up Xapian search, msn SEARCH and multiple ranges imap: misc cleanups and notes imapd: don't bother sorting LIST output imap: drop non-UID SEARCH for now over: uid_range: remove LIMIT imap: FETCH: proper MSN => UID mapping for requests Documentation/public-inbox-imapd.pod | 91 ++ Documentation/standards.perl | 10 + MANIFEST | 18 + lib/PublicInbox/Config.pm | 18 + lib/PublicInbox/Daemon.pm | 24 +- lib/PublicInbox/DummyInbox.pm | 22 + lib/PublicInbox/Eml.pm | 9 +- lib/PublicInbox/FakeInotify.pm | 59 ++ lib/PublicInbox/Git.pm | 163 +-- lib/PublicInbox/GitAsyncCat.pm | 51 + lib/PublicInbox/IMAP.pm | 1397 ++++++++++++++++++++++++++ lib/PublicInbox/IMAPClient.pm | 119 +++ lib/PublicInbox/IMAPD.pm | 114 +++ lib/PublicInbox/IMAPdeflate.pm | 126 +++ lib/PublicInbox/Import.pm | 2 +- lib/PublicInbox/In2Tie.pm | 17 + lib/PublicInbox/Inbox.pm | 33 +- lib/PublicInbox/InboxIdle.pm | 79 ++ lib/PublicInbox/KQNotify.pm | 66 ++ lib/PublicInbox/Lock.pm | 7 + lib/PublicInbox/MsgIter.pm | 2 +- lib/PublicInbox/Msgmap.pm | 20 +- lib/PublicInbox/NNTPD.pm | 12 +- lib/PublicInbox/NNTPdeflate.pm | 1 - lib/PublicInbox/Over.pm | 50 +- lib/PublicInbox/Search.pm | 32 +- lib/PublicInbox/SearchIdx.pm | 89 +- lib/PublicInbox/SearchIdxShard.pm | 11 +- lib/PublicInbox/Smsg.pm | 8 +- lib/PublicInbox/TestCommon.pm | 7 +- lib/PublicInbox/V2Writable.pm | 10 +- script/public-inbox-imapd | 14 + t/config.t | 15 +- t/eml.t | 2 +- t/git.t | 40 +- t/imap.t | 133 +++ t/imapd-tls.t | 204 ++++ t/imapd.t | 398 ++++++++ t/import.t | 5 +- t/inbox_idle.t | 72 ++ t/nntpd.t | 5 +- t/over.t | 3 + t/search.t | 19 + xt/cmp-msgstr.t | 1 - xt/cmp-msgview.t | 1 - xt/eml_check_limits.t | 6 +- xt/git_async_cmp.t | 2 +- xt/imapd-mbsync-oimap.t | 132 +++ xt/imapd-validate.t | 177 ++++ xt/mem-msgview.t | 1 + xt/msgtime_cmp.t | 1 - xt/perf-msgview.t | 1 - 52 files changed, 3718 insertions(+), 181 deletions(-) create mode 100644 Documentation/public-inbox-imapd.pod create mode 100644 lib/PublicInbox/DummyInbox.pm create mode 100644 lib/PublicInbox/FakeInotify.pm create mode 100644 lib/PublicInbox/GitAsyncCat.pm create mode 100644 lib/PublicInbox/IMAP.pm create mode 100644 lib/PublicInbox/IMAPClient.pm create mode 100644 lib/PublicInbox/IMAPD.pm create mode 100644 lib/PublicInbox/IMAPdeflate.pm create mode 100644 lib/PublicInbox/In2Tie.pm create mode 100644 lib/PublicInbox/InboxIdle.pm create mode 100644 lib/PublicInbox/KQNotify.pm create mode 100644 script/public-inbox-imapd create mode 100644 t/imap.t create mode 100644 t/imapd-tls.t create mode 100644 t/imapd.t create mode 100644 t/inbox_idle.t create mode 100644 xt/imapd-mbsync-oimap.t create mode 100644 xt/imapd-validate.t