From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 8B25E1F42D for ; Mon, 19 Mar 2018 08:14:59 +0000 (UTC) From: "Eric Wong (Contractor, The Linux Foundation)" To: meta@public-inbox.org Subject: [PATCH 00/27] v2 public-inbox-watch support Date: Mon, 19 Mar 2018 08:14:32 +0000 Message-Id: <20180319081459.10645-1-e@80x24.org> List-Id: barrier support in v2writable makes checking duplicates during large imports faster as it no longer requires tearing down subprocesses. This also makes checking deletes a bit faster (we check deletes to prevent blindly writing the same deleted files over and over again) On the subject of subprocesses for v2, they're now optional (per-invocation), so -init obviously won't need to spawn subprocesses, and normal -watch usage shouldn't, either, as mail doesn't tend to arrive quickly. public-inbox-watch seems to be working well enough on my desktop with my Maildir for LKML (synched via offlineimap). Eric Wong (Contractor, The Linux Foundation) (27): content_id: use Sender header if From is not available v2writable: support "barrier" operation to avoid reforking use string ref for Email::Simple->new v2writable: remove unnecessary idx_init call searchidx: do not delete documents while iterating search: allow ->reopen to be chainable v2writable: implement remove correctly skeleton: barrier init requires a lock import: (v2) delete writes the blob into history in subdir import: (v2): write deletes to a separate '_' subdirectory import: implement barrier operation for v1 repos mid: mid_mime uses v2-compatible mids function watchmaildir: use content_digest to generate Message-Id import: force Message-ID generation for v1 here import: switch to URL-safe Base64 for Message-IDs v2writable: test for idempotent removals import: enable locking under v2 index: s/GIT_DIR/REPO_DIR/ Lock: new base class for writable lockers t/watch_maildir: note the reason for FIFO creation v2writable: ensure ->done is idempotent watchmaildir: support v2 repositories searchidxpart: s/barrier/remote_barrier/ v2writable: allow disabling parallelization scripts/import_vger_from_mbox: filter out same headers as MDA v2writable: add DEBUG_DIFF env support v2writable: remove "resent" message for duplicate Message-IDs Documentation/public-inbox-index.pod | 8 +- MANIFEST | 2 + lib/PublicInbox/ContentId.pm | 16 +++- lib/PublicInbox/Import.pm | 132 ++++++++++++++++++----------- lib/PublicInbox/Lock.pm | 31 +++++++ lib/PublicInbox/MID.pm | 3 +- lib/PublicInbox/Msgmap.pm | 8 ++ lib/PublicInbox/Search.pm | 1 + lib/PublicInbox/SearchIdx.pm | 159 +++++++++++++++++++++++------------ lib/PublicInbox/SearchIdxPart.pm | 62 +++++++++----- lib/PublicInbox/SearchIdxSkeleton.pm | 112 +++++++++++++++++------- lib/PublicInbox/SearchMsg.pm | 4 +- lib/PublicInbox/V2Writable.pm | 136 ++++++++++++++++++++++++------ lib/PublicInbox/WatchMaildir.pm | 54 ++++-------- script/public-inbox-index | 12 +-- script/public-inbox-init | 3 +- scripts/import_vger_from_mbox | 2 + t/v2writable.t | 51 +++++++++-- t/watch_maildir.t | 3 +- t/watch_maildir_v2.t | 125 +++++++++++++++++++++++++++ 20 files changed, 683 insertions(+), 241 deletions(-) create mode 100644 lib/PublicInbox/Lock.pm create mode 100644 t/watch_maildir_v2.t -- EW