From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 0F8B21F66F for ; Sat, 7 Nov 2020 10:57:01 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/10] extindex: another round of updates Date: Sat, 7 Nov 2020 10:56:50 +0000 Message-Id: <20201107105700.12586-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: A major user-visible change is renaming -eindex to -extindex, because rhyming with "reindex" is probably confusing (and I'm easily confused :x). PATCH 10/10 finally starts making sense performance-wise, still testing... I've long thought the default 1m batch-size is too small for 64-bit machines, so maybe that'll change, too. But it took me way too long to figure out why indexBatchSize was seemed to have no effect in my PI_CONFIG :< My Internet access has been terribly unreliable, lately, too; so relying on mosh/ssh access to work on more powerful machines aint too pleasant. Overall extindex it seems to be working somewhat OK for incremental updates the past few weeks, but could still benefit from speedups to work better on HW I have locally. Will have to retest SQLite cache_size and mmap_size pragmas, too. Eric Wong (10): extsearch: rename -eindex to -extindex extsearchidx: avoid needless alternates rewrite in ALL.git searchidxshard: reduce syscalls when writing ->eidx_key searchidxshard: further improve {current_info} readability v2writable: less expensive checkpoint for extindex extsearchidx: quiet warning for unindexed `d' messages extsearch: canonicalize topdir v2writable: more accurate {current_info} warnings/progress extindex: SIGUSR1 supports checkpoint extindex: fix --batch-size support MANIFEST | 2 +- lib/PublicInbox/Config.pm | 2 +- lib/PublicInbox/ExtSearch.pm | 2 + lib/PublicInbox/ExtSearchIdx.pm | 37 +++++++++++++------ lib/PublicInbox/SearchIdxShard.pm | 9 ++--- lib/PublicInbox/V2Writable.pm | 37 ++++++++++++++----- ...lic-inbox-eindex => public-inbox-extindex} | 8 +++- t/extsearch.t | 6 +-- 8 files changed, 68 insertions(+), 35 deletions(-) rename script/{public-inbox-eindex => public-inbox-extindex} (84%)