* [PATCH v2 3/4] lei/store: stop shard workers + cat-file on idle
2024-04-16 20:56 7% ` [PATCH 3/4] lei/store: stop shard workers + cat-file on idle Eric Wong
@ 2024-04-17 9:34 6% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2024-04-17 9:34 UTC (permalink / raw)
To: meta
Schedule a timer to stop shard workers and the git-cat-file
process after a `barrier' command. This allows us to save some
memory again when the lei-daemon is idle but preserves the fork
overhead reduction when issuing many commands in parallel or in
quick succession.
---
v2 fixes an incorrect call to add_uniq_timer. Sometimes I wish Perl
could have more static type||arg checking, but it's probably still
better than other scripting languages...
Interdiff against v1:
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index a054f649..b2da2bc3 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -574,7 +574,7 @@ sub set_xvmd {
sub check_done {
my ($self) = @_;
$self->git->_active ?
- add_uniq_timer("$self-check_done", \&check_done, $self) :
+ add_uniq_timer("$self-check_done", 5, \&check_done, $self) :
done($self);
}
lib/PublicInbox/LeiStore.pm | 46 ++++++++++++++++++-------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 162c915f..b2da2bc3 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -571,21 +571,11 @@ sub set_xvmd {
sto_export_kw($self, $smsg->{num}, $vmd);
}
-sub barrier {
+sub check_done {
my ($self) = @_;
- my ($errfh, $lei_sock) = @$self{0, 1}; # via sto_barrier_request
- my @err;
- if ($self->{im}) {
- eval { $self->{im}->barrier };
- push(@err, "E: import barrier: $@\n") if $@;
- }
- delete $self->{lms};
- eval { $self->{priv_eidx}->barrier };
- push(@err, "E: priv_eidx barrier: $@\n") if $@;
- print { $errfh // \*STDERR } @err;
- send($lei_sock, 'child_error 256', 0) if @err && $lei_sock;
- xchg_stderr($self);
- die @err if @err;
+ $self->git->_active ?
+ add_uniq_timer("$self-check_done", 5, \&check_done, $self) :
+ done($self);
}
sub xchg_stderr {
@@ -602,23 +592,33 @@ sub xchg_stderr {
undef;
}
-sub done {
- my ($self) = @_;
- my ($errfh, $lei_sock) = @$self{0, 1};
+sub _commit ($$) {
+ my ($self, $cmd) = @_; # cmd is 'done' or 'barrier'
+ my ($errfh, $lei_sock) = @$self{0, 1}; # via sto_barrier_request
my @err;
- if (my $im = delete($self->{im})) {
- eval { $im->done };
- push(@err, "E: import done: $@\n") if $@;
+ if ($self->{im}) {
+ eval { $self->{im}->$cmd };
+ push(@err, "E: import $cmd: $@\n") if $@;
}
delete $self->{lms};
- eval { $self->{priv_eidx}->done }; # V2Writable::done
- push(@err, "E: priv_eidx done: $@\n") if $@;
- print { $errfh // *STDERR{GLOB} } @err;
+ eval { $self->{priv_eidx}->$cmd };
+ push(@err, "E: priv_eidx $cmd: $@\n") if $@;
+ print { $errfh // \*STDERR } @err;
send($lei_sock, 'child_error 256', 0) if @err && $lei_sock;
xchg_stderr($self);
die @err if @err;
+ # $lei_sock goes out-of-scope and script/lei can terminate
+}
+
+sub barrier {
+ my ($self) = @_;
+ _commit $self, 'barrier';
+ add_uniq_timer("$self-check_done", 5, \&check_done, $self);
+ undef;
}
+sub done { _commit $_[0], 'done' }
+
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
^ permalink raw reply related [relevance 6%]
* [PATCH 3/4] lei/store: stop shard workers + cat-file on idle
2024-04-16 20:56 6% [PATCH 0/4] lei parallelism fixes Eric Wong
@ 2024-04-16 20:56 7% ` Eric Wong
2024-04-17 9:34 6% ` [PATCH v2 " Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2024-04-16 20:56 UTC (permalink / raw)
To: meta
Schedule a timer to stop shard workers and the git-cat-file
process after a `barrier' command. This allows us to save some
memory again when the lei-daemon is idle but preserves the fork
overhead reduction when issuing many commands in parallel or in
quick succession.
---
lib/PublicInbox/LeiStore.pm | 46 ++++++++++++++++++-------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 162c915f..a054f649 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -571,21 +571,11 @@ sub set_xvmd {
sto_export_kw($self, $smsg->{num}, $vmd);
}
-sub barrier {
+sub check_done {
my ($self) = @_;
- my ($errfh, $lei_sock) = @$self{0, 1}; # via sto_barrier_request
- my @err;
- if ($self->{im}) {
- eval { $self->{im}->barrier };
- push(@err, "E: import barrier: $@\n") if $@;
- }
- delete $self->{lms};
- eval { $self->{priv_eidx}->barrier };
- push(@err, "E: priv_eidx barrier: $@\n") if $@;
- print { $errfh // \*STDERR } @err;
- send($lei_sock, 'child_error 256', 0) if @err && $lei_sock;
- xchg_stderr($self);
- die @err if @err;
+ $self->git->_active ?
+ add_uniq_timer("$self-check_done", \&check_done, $self) :
+ done($self);
}
sub xchg_stderr {
@@ -602,23 +592,33 @@ sub xchg_stderr {
undef;
}
-sub done {
- my ($self) = @_;
- my ($errfh, $lei_sock) = @$self{0, 1};
+sub _commit ($$) {
+ my ($self, $cmd) = @_; # cmd is 'done' or 'barrier'
+ my ($errfh, $lei_sock) = @$self{0, 1}; # via sto_barrier_request
my @err;
- if (my $im = delete($self->{im})) {
- eval { $im->done };
- push(@err, "E: import done: $@\n") if $@;
+ if ($self->{im}) {
+ eval { $self->{im}->$cmd };
+ push(@err, "E: import $cmd: $@\n") if $@;
}
delete $self->{lms};
- eval { $self->{priv_eidx}->done }; # V2Writable::done
- push(@err, "E: priv_eidx done: $@\n") if $@;
- print { $errfh // *STDERR{GLOB} } @err;
+ eval { $self->{priv_eidx}->$cmd };
+ push(@err, "E: priv_eidx $cmd: $@\n") if $@;
+ print { $errfh // \*STDERR } @err;
send($lei_sock, 'child_error 256', 0) if @err && $lei_sock;
xchg_stderr($self);
die @err if @err;
+ # $lei_sock goes out-of-scope and script/lei can terminate
+}
+
+sub barrier {
+ my ($self) = @_;
+ _commit $self, 'barrier';
+ add_uniq_timer("$self-check_done", 5, \&check_done, $self);
+ undef;
}
+sub done { _commit $_[0], 'done' }
+
sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] lei parallelism fixes
@ 2024-04-16 20:56 6% Eric Wong
2024-04-16 20:56 7% ` [PATCH 3/4] lei/store: stop shard workers + cat-file on idle Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2024-04-16 20:56 UTC (permalink / raw)
To: meta
This series allows `lei reindex' to run in parallel with other
lei commands which write to lei/store.
Eric Wong (4):
v2 + lei/store: always wait for fast-import checkpoint
lei: use ->barrier to commit to lei/store
lei/store: stop shard workers + cat-file on idle
lei: use async barrier for --import-before
lib/PublicInbox/EOFpipe.pm | 7 ++--
lib/PublicInbox/ExtSearchIdx.pm | 1 +
lib/PublicInbox/LEI.pm | 6 ++--
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiRemote.pm | 4 +--
lib/PublicInbox/LeiStore.pm | 46 ++++++++++++++++-----------
lib/PublicInbox/LeiToMail.pm | 28 ++++++++++++----
lib/PublicInbox/LeiXSearch.pm | 17 ++++++----
lib/PublicInbox/V2Writable.pm | 8 +----
t/lei-store-fail.t | 2 +-
11 files changed, 74 insertions(+), 49 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2024-04-16 20:56 6% [PATCH 0/4] lei parallelism fixes Eric Wong
2024-04-16 20:56 7% ` [PATCH 3/4] lei/store: stop shard workers + cat-file on idle Eric Wong
2024-04-17 9:34 6% ` [PATCH v2 " Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).