From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 59DF91F5AE for ; Sat, 27 Jun 2020 10:04:00 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/34] watch: add IMAP and NNTP support Date: Sat, 27 Jun 2020 10:03:26 +0000 Message-Id: <20200627100400.9871-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Some fairly major changes to -watch. Filesys::Notify::Simple is no longer used, and -watch now uses inotify, signalfd or kevent like the read-only daemons. Credentials are handled via Net::Netrc (Perl standard library) or "git-credential", so we do no password storage on our own. NNTP (and non-IDLE IMAP) may allow more parallelization in the future. One significant project-wide change is getting rid of "use fields". It gets in my way more than it helps, and it's probably alien to a fair amount of Perl hackers. AFAIK, it's never really been popular outside of Danga::Socket-based projects. Eric W. Biederman (1): IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric Wong (33): inboxwritable: ensure ssoma.lock exists on init inbox: warn on ->on_inbox_unlock exception imaptracker: use ~/.local/share/public-inbox/imap.sqlite3 watchmaildir: hoist out compile_watchheaders watchmaildir: fix check for spam vs ham inbox conflicts URI IMAP support watch: preliminary IMAP support kqnotify|fake_inotify: detect Maildir write ops watch: remove Filesys::Notify::Simple dependency watch: use signalfd for Maildir watching ds: remove fields.pm usage watch: wire up IMAP IDLE reapers to DS watch: support IMAP polling config: support ->urlmatch method for -watch watch: stop importers before forking watch: use UID SEARCH to avoid empty UID FETCH ds: add_timer: allow passing arg to callback. imaptracker: add {url} field to reduce args imaptracker: drop {dbname} field watch: avoid long transaction when writing to IMAPTracker watch: support imap.fetchBatchSize parameter watch: imap: be quieter about disconnecting on quit watch: support multiple watch: directives per-inbox watch: remove {mdir} array watch: just use ->urlmatch testcommon: $ENV{TAIL} supports non-@ARGV redirects watch: add NNTP support watch: show user-specified URL consistently. watch: enable autoflush for STDOUT and STDERR watch: use our own "git credential" wrapper watch: support ~/.netrc via Net::Netrc imaptracker: use flock(2) around writes watch: simplify internal structures Documentation/public-inbox-watch.pod | 3 +- INSTALL | 8 - MANIFEST | 11 + Makefile.PL | 4 - ci/deps.perl | 1 - lib/PublicInbox/Config.pm | 21 +- lib/PublicInbox/DS.pm | 29 +- lib/PublicInbox/Daemon.pm | 19 +- lib/PublicInbox/DirIdle.pm | 49 ++ lib/PublicInbox/FakeInotify.pm | 56 +- lib/PublicInbox/GitAsyncCat.pm | 4 +- lib/PublicInbox/GitCredential.pm | 55 ++ lib/PublicInbox/HTTP.pm | 23 +- lib/PublicInbox/HTTPD/Async.pm | 22 +- lib/PublicInbox/IMAP.pm | 19 +- lib/PublicInbox/IMAPTracker.pm | 82 +++ lib/PublicInbox/In2Tie.pm | 13 + lib/PublicInbox/Inbox.pm | 1 + lib/PublicInbox/InboxIdle.pm | 20 +- lib/PublicInbox/InboxWritable.pm | 3 + lib/PublicInbox/KQNotify.pm | 38 +- lib/PublicInbox/Listener.pm | 8 +- lib/PublicInbox/NNTP.pm | 12 +- lib/PublicInbox/NNTPdeflate.pm | 5 +- lib/PublicInbox/ParentPipe.pm | 8 +- lib/PublicInbox/Sigfd.pm | 21 +- lib/PublicInbox/TestCommon.pm | 40 +- lib/PublicInbox/URIimap.pm | 113 +++ lib/PublicInbox/WatchMaildir.pm | 998 +++++++++++++++++++++++---- script/public-inbox-watch | 33 +- t/config.t | 18 + t/dir_idle.t | 6 + t/fake_inotify.t | 45 ++ t/imap_tracker.t | 54 ++ t/imapd.t | 74 ++ t/kqnotify.t | 41 ++ t/nntpd.t | 52 ++ t/uri_imap.t | 65 ++ t/watch_filter_rubylang.t | 2 +- t/watch_imap.t | 21 + t/watch_maildir.t | 96 ++- t/watch_maildir_v2.t | 4 +- t/watch_multiple_headers.t | 2 +- t/watch_nntp.t | 17 + xt/mem-imapd-tls.t | 18 +- 45 files changed, 1944 insertions(+), 290 deletions(-) create mode 100644 lib/PublicInbox/DirIdle.pm create mode 100644 lib/PublicInbox/GitCredential.pm create mode 100644 lib/PublicInbox/IMAPTracker.pm create mode 100644 lib/PublicInbox/URIimap.pm create mode 100644 t/dir_idle.t create mode 100644 t/fake_inotify.t create mode 100644 t/imap_tracker.t create mode 100644 t/kqnotify.t create mode 100644 t/uri_imap.t create mode 100644 t/watch_imap.t create mode 100644 t/watch_nntp.t