From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id E515B1F404 for ; Wed, 28 Feb 2018 23:42:07 +0000 (UTC) From: "Eric Wong (Contractor, The Linux Foundation)" To: meta@public-inbox.org Subject: [PATCH v2 0/21] UI bits and v2 import fixes Date: Wed, 28 Feb 2018 23:41:41 +0000 Message-Id: <20180228234202.8839-1-e@80x24.org> List-Id: The most important fix was fixing a long-standing bug (also in v1) with "searchidx: do not modify Xapian DB while iterating". It turns out my initial v2 import was missing a bit of data and got threading wrong when Subjects were mismatched (every patch series). Unfortunately that means the import times I initially reported were too optimistic and real import times may take 30-40% longer :< (More optimizations are planned, however) Fortunately, old optimizations made to avoid git tree lookups during the v1 era has made v2 UI work much easier and I was able to spot some errors and bugs quickly in the PSGI interface. For sorting, relying on the Date: header seems unreliable as kernel developers seem more prone to having bad clocks than other lists I've imported. I'll probably switch the internal timestamps to use the Received: date as a result. Eric Wong (Contractor, The Linux Foundation) (21): v2writable: warn on duplicate Message-IDs v2/ui: some hacky things to get the PSGI UI to show up v2/ui: retry DB reopens in a few more places v2writable: cleanup unused pipes in partitions searchidxpart: binmode use PublicInbox::MIME consistently searchidxpart: chomp line before splitting searchidx*: name child subprocesses searchidx: get rid of pointless index_blob wrapper view: remove X-PI-TS reference searchidxthread: load doc data for references searchidxpart: force integers into add_message search: reopen skeleton DB as well searchidx: index values in the threader search: use different Enquire object for skeleton queries rename SearchIdxThread to SearchIdxSkeleton v2writable: commit to skeleton via remote partitions searchidxskeleton: extra error checking searchidx: do not modify Xapian DB while iterating search: query_xover uses skeleton DB iff available v2/ui: get nntpd and init tests running on v2 MANIFEST | 2 +- lib/PublicInbox/Import.pm | 7 ++ lib/PublicInbox/Inbox.pm | 23 +++++-- lib/PublicInbox/MIME.pm | 2 + lib/PublicInbox/Search.pm | 74 ++++++++++++++++++++-- lib/PublicInbox/SearchIdx.pm | 72 +++++++++++---------- lib/PublicInbox/SearchIdxPart.pm | 27 +++++--- .../{SearchIdxThread.pm => SearchIdxSkeleton.pm} | 50 +++++++++------ lib/PublicInbox/V2Writable.pm | 50 +++++++++------ lib/PublicInbox/View.pm | 3 +- lib/PublicInbox/WatchMaildir.pm | 2 - lib/PublicInbox/WwwAttach.pm | 3 +- script/public-inbox-init | 48 ++++++++++---- script/public-inbox-learn | 2 - script/public-inbox-mda | 4 +- scripts/import_slrnspool | 4 +- scripts/import_vger_from_mbox | 3 +- t/init.t | 15 +++++ t/nntpd.t | 36 +++++++++-- 19 files changed, 302 insertions(+), 125 deletions(-) rename lib/PublicInbox/{SearchIdxThread.pm => SearchIdxSkeleton.pm} (63%)