From: Eric Wong <e@80x24.org> To: meta@public-inbox.org Subject: [PATCH 00/10] extindex: another round of updates Date: Sat, 7 Nov 2020 10:56:50 +0000 Message-ID: <20201107105700.12586-1-e@80x24.org> (raw) A major user-visible change is renaming -eindex to -extindex, because rhyming with "reindex" is probably confusing (and I'm easily confused :x). PATCH 10/10 finally starts making sense performance-wise, still testing... I've long thought the default 1m batch-size is too small for 64-bit machines, so maybe that'll change, too. But it took me way too long to figure out why indexBatchSize was seemed to have no effect in my PI_CONFIG :< My Internet access has been terribly unreliable, lately, too; so relying on mosh/ssh access to work on more powerful machines aint too pleasant. Overall extindex it seems to be working somewhat OK for incremental updates the past few weeks, but could still benefit from speedups to work better on HW I have locally. Will have to retest SQLite cache_size and mmap_size pragmas, too. Eric Wong (10): extsearch: rename -eindex to -extindex extsearchidx: avoid needless alternates rewrite in ALL.git searchidxshard: reduce syscalls when writing ->eidx_key searchidxshard: further improve {current_info} readability v2writable: less expensive checkpoint for extindex extsearchidx: quiet warning for unindexed `d' messages extsearch: canonicalize topdir v2writable: more accurate {current_info} warnings/progress extindex: SIGUSR1 supports checkpoint extindex: fix --batch-size support MANIFEST | 2 +- lib/PublicInbox/Config.pm | 2 +- lib/PublicInbox/ExtSearch.pm | 2 + lib/PublicInbox/ExtSearchIdx.pm | 37 +++++++++++++------ lib/PublicInbox/SearchIdxShard.pm | 9 ++--- lib/PublicInbox/V2Writable.pm | 37 ++++++++++++++----- ...lic-inbox-eindex => public-inbox-extindex} | 8 +++- t/extsearch.t | 6 +-- 8 files changed, 68 insertions(+), 35 deletions(-) rename script/{public-inbox-eindex => public-inbox-extindex} (84%)
next reply other threads:[~2020-11-07 10:57 UTC|newest] Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-07 10:56 Eric Wong [this message] 2020-11-07 10:56 ` [PATCH 01/10] extsearch: rename -eindex to -extindex Eric Wong 2020-11-07 10:56 ` [PATCH 02/10] extsearchidx: avoid needless alternates rewrite in ALL.git Eric Wong 2020-11-07 10:56 ` [PATCH 03/10] searchidxshard: reduce syscalls when writing ->eidx_key Eric Wong 2020-11-07 10:56 ` [PATCH 04/10] searchidxshard: further improve {current_info} readability Eric Wong 2020-11-07 10:56 ` [PATCH 05/10] v2writable: less expensive checkpoint for extindex Eric Wong 2020-11-07 10:56 ` [PATCH 06/10] extsearchidx: quiet warning for unindexed `d' messages Eric Wong 2020-11-07 10:56 ` [PATCH 07/10] extsearch: canonicalize topdir Eric Wong 2020-11-07 10:56 ` [PATCH 08/10] v2writable: more accurate {current_info} warnings/progress Eric Wong 2020-11-07 10:56 ` [PATCH 09/10] extindex: SIGUSR1 supports checkpoint Eric Wong 2020-11-07 10:57 ` [PATCH 10/10] extindex: fix --batch-size support Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201107105700.12586-1-e@80x24.org \ --to=e@80x24.org \ --cc=meta@public-inbox.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself This inbox may be cloned and mirrored by anyone: git clone --mirror https://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \ meta@public-inbox.org public-inbox-index meta Example config snippet for mirrors. Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ code repositories for the project(s) associated with this inbox: https://80x24.org/public-inbox.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git