* [PATCH 09/10] extindex: SIGUSR1 supports checkpoint
2020-11-07 10:56 7% [PATCH 00/10] extindex: another round of updates Eric Wong
@ 2020-11-07 10:56 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-11-07 10:56 UTC (permalink / raw)
To: meta
Matching the behavior of git-fast-import(1), we'll allow a user
to send SIGUSR1 to checkpoint over.sqlite3 and Xapian.
---
lib/PublicInbox/ExtSearchIdx.pm | 23 ++++++++++++-----------
script/public-inbox-extindex | 1 +
2 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index 50342802..7aaf8291 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -284,16 +284,9 @@ sub last_commits {
}
sub _sync_inbox ($$$) {
- my ($self, $opt, $ibx) = @_;
- my $sync = {
- need_checkpoint => \(my $bool = 0),
- reindex => $opt->{reindex},
- -opt => $opt,
- self => $self,
- ibx => $ibx,
- nr => \(my $nr = 0),
- -regen_fmt => "%u/?\n",
- };
+ my ($self, $sync, $ibx) = @_;
+ $sync->{ibx} = $ibx;
+ $sync->{nr} = \(my $nr = 0);
my $v = $ibx->version;
my $ekey = $ibx->eidx_key;
if ($v == 2) {
@@ -324,10 +317,18 @@ sub eidx_sync { # main entry point
local $SIG{__WARN__} = sub {
$warn_cb->($self->{current_info}, ': ', @_);
};
+ my $sync = {
+ need_checkpoint => \(my $need_checkpoint = 0),
+ reindex => $opt->{reindex},
+ -opt => $opt,
+ self => $self,
+ -regen_fmt => "%u/?\n",
+ };
+ local $SIG{USR1} = sub { $need_checkpoint = 1 };
# don't use $_ here, it'll get clobbered by reindex_checkpoint
for my $ibx (@{$self->{ibx_list}}) {
- _sync_inbox($self, $opt, $ibx);
+ _sync_inbox($self, $sync, $ibx);
}
$self->{oidx}->rethread_done($opt);
diff --git a/script/public-inbox-extindex b/script/public-inbox-extindex
index bb1e174a..864a2732 100644
--- a/script/public-inbox-extindex
+++ b/script/public-inbox-extindex
@@ -33,6 +33,7 @@ die "--jobs must be >= 0\n" if defined $opt->{jobs} && $opt->{jobs} < 0;
# require lazily to speed up --help
my $eidx_dir = shift(@ARGV) // die "E: $help";
+local $SIG{USR1} = 'IGNORE'; # to be overridden in eidx_sync
require PublicInbox::Admin;
my $cfg = PublicInbox::Config->new;
my @ibxs = PublicInbox::Admin::resolve_inboxes(\@ARGV, $opt, $cfg);
^ permalink raw reply related [relevance 7%]
* [PATCH 00/10] extindex: another round of updates
@ 2020-11-07 10:56 7% Eric Wong
2020-11-07 10:56 7% ` [PATCH 09/10] extindex: SIGUSR1 supports checkpoint Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-11-07 10:56 UTC (permalink / raw)
To: meta
A major user-visible change is renaming -eindex to -extindex,
because rhyming with "reindex" is probably confusing (and I'm
easily confused :x).
PATCH 10/10 finally starts making sense performance-wise, still
testing... I've long thought the default 1m batch-size is too
small for 64-bit machines, so maybe that'll change, too.
But it took me way too long to figure out why indexBatchSize was
seemed to have no effect in my PI_CONFIG :<
My Internet access has been terribly unreliable, lately, too; so
relying on mosh/ssh access to work on more powerful machines
aint too pleasant.
Overall extindex it seems to be working somewhat OK for
incremental updates the past few weeks, but could still benefit
from speedups to work better on HW I have locally.
Will have to retest SQLite cache_size and mmap_size pragmas, too.
Eric Wong (10):
extsearch: rename -eindex to -extindex
extsearchidx: avoid needless alternates rewrite in ALL.git
searchidxshard: reduce syscalls when writing ->eidx_key
searchidxshard: further improve {current_info} readability
v2writable: less expensive checkpoint for extindex
extsearchidx: quiet warning for unindexed `d' messages
extsearch: canonicalize topdir
v2writable: more accurate {current_info} warnings/progress
extindex: SIGUSR1 supports checkpoint
extindex: fix --batch-size support
MANIFEST | 2 +-
lib/PublicInbox/Config.pm | 2 +-
lib/PublicInbox/ExtSearch.pm | 2 +
lib/PublicInbox/ExtSearchIdx.pm | 37 +++++++++++++------
lib/PublicInbox/SearchIdxShard.pm | 9 ++---
lib/PublicInbox/V2Writable.pm | 37 ++++++++++++++-----
...lic-inbox-eindex => public-inbox-extindex} | 8 +++-
t/extsearch.t | 6 +--
8 files changed, 68 insertions(+), 35 deletions(-)
rename script/{public-inbox-eindex => public-inbox-extindex} (84%)
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-11-07 10:56 7% [PATCH 00/10] extindex: another round of updates Eric Wong
2020-11-07 10:56 7% ` [PATCH 09/10] extindex: SIGUSR1 supports checkpoint Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).