user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [PATCH 0/7] lei bugfixes and other fixes
@ 2021-09-22  2:24 Eric Wong
  2021-09-22  2:24 ` [PATCH 1/7] ipc: do not add "0" to $0 of solo workers Eric Wong
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

How did I forget 2/7 for all these months? :<

6/7 took a while for me to figure out, I only noticed it
because of background processes running after MUA exit
(from "lei up --mua=... MFOLDER"), but I got 4/7 and 7/7
fixes out of it.

Eric Wong (7):
  ipc: do not add "0" to $0 of solo workers
  treewide: fix %SIG localization, harder
  script/lei: describe purpose of sleep loop
  lei: dclose: do not close unnecessarily
  inbox: do not waste hash slot on httpbackend_limiter
  lei up: avoid excessively parallel --all
  lei: drop redundant WQ EOF callbacks

 lib/PublicInbox/Admin.pm        |  2 +-
 lib/PublicInbox/ExtSearchIdx.pm |  4 +-
 lib/PublicInbox/IPC.pm          |  7 +--
 lib/PublicInbox/Inbox.pm        | 10 ++--
 lib/PublicInbox/LEI.pm          |  3 +-
 lib/PublicInbox/LeiImportKw.pm  |  8 +--
 lib/PublicInbox/LeiMirror.pm    | 10 +---
 lib/PublicInbox/LeiNoteEvent.pm |  8 +--
 lib/PublicInbox/LeiPmdir.pm     |  8 +--
 lib/PublicInbox/LeiUp.pm        | 86 +++++++++++++++++++++------------
 lib/PublicInbox/TestCommon.pm   |  3 +-
 lib/PublicInbox/Watch.pm        |  4 +-
 lib/PublicInbox/Xapcmd.pm       |  8 +--
 script/lei                      |  2 +-
 script/public-inbox-clone       |  2 +-
 t/run.perl                      |  1 +
 16 files changed, 86 insertions(+), 80 deletions(-)

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 1/7] ipc: do not add "0" to $0 of solo workers
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 2/7] treewide: fix %SIG localization, harder Eric Wong
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

It's needless noise and misleads users reading "ps" into
thinking there's more workers when there's only one.
---
 lib/PublicInbox/IPC.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index add5f3df..1c699d76 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -353,7 +353,8 @@ sub _wq_worker_start ($$$$) {
 				keys %{delete($self->{-wq_workers}) // {}};
 		$SIG{$_} = 'IGNORE' for (qw(PIPE));
 		$SIG{$_} = 'DEFAULT' for (qw(TTOU TTIN TERM QUIT INT CHLD));
-		local $0 = "$self->{-wq_ident} $self->{-wq_worker_nr}";
+		local $0 = $one ? $self->{-wq_ident} :
+			"$self->{-wq_ident} $self->{-wq_worker_nr}";
 		# ensure we properly exit even if warn() dies:
 		my $end = PublicInbox::OnDestroy->new($$, sub { exit(!!$@) });
 		eval {

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 2/7] treewide: fix %SIG localization, harder
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
  2021-09-22  2:24 ` [PATCH 1/7] ipc: do not add "0" to $0 of solo workers Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 3/7] script/lei: describe purpose of sleep loop Eric Wong
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

This fixes the occasional t/lei-sigpipe.t infinite loop
under "make check-run".

Link: http://nntp.perl.org/group/perl.perl5.porters/258784
  <CAHhgV8hPbcmkzWizp6Vijw921M5BOXixj4+zTh3nRS9vRBYk8w@mail.gmail.com>
Followup-to: b552bb9150775fe4 ("daemon+watch: fix localization of %SIG for non-signalfd users")
---
 lib/PublicInbox/Admin.pm        | 2 +-
 lib/PublicInbox/ExtSearchIdx.pm | 4 ++--
 lib/PublicInbox/IPC.pm          | 4 ++--
 lib/PublicInbox/TestCommon.pm   | 3 ++-
 lib/PublicInbox/Watch.pm        | 4 +++-
 lib/PublicInbox/Xapcmd.pm       | 8 ++++----
 t/run.perl                      | 1 +
 7 files changed, 15 insertions(+), 11 deletions(-)

diff --git a/lib/PublicInbox/Admin.pm b/lib/PublicInbox/Admin.pm
index 9ff59bca..20964f9c 100644
--- a/lib/PublicInbox/Admin.pm
+++ b/lib/PublicInbox/Admin.pm
@@ -274,7 +274,7 @@ sub index_inbox {
 	if (my $pr = $opt->{-progress}) {
 		$pr->("indexing $ibx->{inboxdir} ...\n");
 	}
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	setup_signals(\&index_terminate, $ibx);
 	my $idx = { current_info => $ibx->{inboxdir} };
 	local $SIG{__WARN__} = sub {
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index e0ba6c32..6b29789a 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -1272,7 +1272,7 @@ sub event_step { # PublicInbox::DS::requeue callback
 
 sub eidx_watch { # public-inbox-extindex --watch main loop
 	my ($self, $opt) = @_;
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	for my $sig (qw(HUP USR1 TSTP QUIT INT TERM)) {
 		$SIG{$sig} = sub { warn "SIG$sig ignored while scanning\n" };
 	}
@@ -1307,7 +1307,7 @@ sub eidx_watch { # public-inbox-extindex --watch main loop
 	$sig->{QUIT} = $sig->{INT} = $sig->{TERM} = $quit;
 	my $sigfd = PublicInbox::Sigfd->new($sig,
 					$PublicInbox::Syscall::SFD_NONBLOCK);
-	%SIG = (%SIG, %$sig) if !$sigfd;
+	@SIG{keys %$sig} = values(%$sig) if !$sigfd;
 	local $self->{-watch_sync} = $sync; # for ->on_inbox_unlock
 	if (!$sigfd) {
 		# wake up every second to accept signals if we don't
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 1c699d76..3e29def8 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -115,7 +115,7 @@ sub ipc_worker_spawn {
 			$fields //= {};
 			local @$self{keys %$fields} = values(%$fields);
 			my $on_destroy = $self->ipc_atfork_child;
-			local %SIG = %SIG;
+			local @SIG{keys %SIG} = values %SIG;
 			PublicInbox::DS::sig_setmask($sigset);
 			ipc_worker_loop($self, $r_req, $w_res);
 		};
@@ -361,7 +361,7 @@ sub _wq_worker_start ($$$$) {
 			$fields //= {};
 			local @$self{keys %$fields} = values(%$fields);
 			my $on_destroy = $self->ipc_atfork_child;
-			local %SIG = %SIG;
+			local @SIG{keys %SIG} = values %SIG;
 			PublicInbox::DS::sig_setmask($oldset);
 			wq_worker_loop($self, $bcast2);
 		};
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 9e152394..92a7db36 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -317,7 +317,8 @@ sub run_script ($;$$) {
 		# note: "local *STDIN = *STDIN;" and so forth did not work in
 		# old versions of perl
 		local %ENV = $env ? (%ENV, %$env) : %ENV;
-		local %SIG = %SIG;
+		local @SIG{keys %SIG} = map { undef } values %SIG;
+		local $SIG{FPE} = 'IGNORE'; # Perl default
 		local $0 = join(' ', @$cmd);
 		my $orig_io = _prepare_redirects($fhref);
 		my $cwdfh = $lei_cwdfh;
diff --git a/lib/PublicInbox/Watch.pm b/lib/PublicInbox/Watch.pm
index 387eb6d2..0523ad03 100644
--- a/lib/PublicInbox/Watch.pm
+++ b/lib/PublicInbox/Watch.pm
@@ -384,7 +384,9 @@ sub watch_atfork_child ($) {
 	delete $self->{poll_pids};
 	delete $self->{opendirs};
 	PublicInbox::DS->Reset;
-	%SIG = (%SIG, %{$self->{sig}}, CHLD => 'DEFAULT');
+	my $sig = delete $self->{sig};
+	$sig->{CHLD} = 'DEFAULT';
+	@SIG{keys %$sig} = values %$sig;
 	PublicInbox::DS::sig_setmask($self->{oldset});
 }
 
diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm
index 588e7b94..b962fa84 100644
--- a/lib/PublicInbox/Xapcmd.pm
+++ b/lib/PublicInbox/Xapcmd.pm
@@ -149,7 +149,7 @@ sub process_queue {
 
 	# run in parallel:
 	my %pids;
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	setup_signals(\&kill_pids, \%pids);
 	while (@$queue) {
 		while (scalar(keys(%pids)) < $max && scalar(@$queue)) {
@@ -285,7 +285,7 @@ sub run {
 		PublicInbox::SearchIdx::load_xapian_writable();
 	}
 
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	setup_signals();
 	$ibx->with_umask(\&_run, $ibx, $cb, $opt);
 }
@@ -343,7 +343,7 @@ sub compact ($$) { # cb_spawn callback
 	$pr->("$pfx `".join(' ', @$cmd)."'\n") if $pr;
 	push @$cmd, $src, $dst;
 	my ($rd, $pid);
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	setup_signals(\&kill_compact, \$pid);
 	($rd, $pid) = popen_rd($cmd, undef, $rdr);
 	while (<$rd>) {
@@ -428,7 +428,7 @@ sub cpdb ($$) { # cb_spawn callback
 	}
 
 	my ($tmp, $ft);
-	local %SIG = %SIG;
+	local @SIG{keys %SIG} = values %SIG;
 	if ($opt->{compact}) {
 		my ($dir) = ($new =~ m!(.*?/)[^/]+/*\z!);
 		same_fs_or_die($dir, $new);
diff --git a/t/run.perl b/t/run.perl
index e5ee0ade..0fe6d08b 100755
--- a/t/run.perl
+++ b/t/run.perl
@@ -168,6 +168,7 @@ my $start_worker = sub {
 	my $pid = fork // DIE "fork: $!";
 	if ($pid == 0) {
 		close $wr if $wr;
+		$SIG{USR1} = undef; # undo parent $SIG{USR1}
 		$worker = $$;
 		while (1) {
 			my $r = sysread($rd, my $buf, UINT_SIZE);

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 3/7] script/lei: describe purpose of sleep loop
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
  2021-09-22  2:24 ` [PATCH 1/7] ipc: do not add "0" to $0 of solo workers Eric Wong
  2021-09-22  2:24 ` [PATCH 2/7] treewide: fix %SIG localization, harder Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 4/7] lei: dclose: do not close unnecessarily Eric Wong
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

It looks dumb, but I'm not about to take a runtime penalty to
use signalfd|EVFILT_SIGNAL, here, either.
---
 script/lei | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/script/lei b/script/lei
index 399296ba..bc437798 100755
--- a/script/lei
+++ b/script/lei
@@ -137,6 +137,6 @@ while (1) {
 $sigchld->();
 if (my $sig = ($x_it_code & 127)) {
 	kill $sig, $$;
-	sleep(1) while 1;
+	sleep(1) while 1; # no self-pipe/signalfd, here, so we loop
 }
 exit($x_it_code >> 8);

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 4/7] lei: dclose: do not close unnecessarily
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
                   ` (2 preceding siblings ...)
  2021-09-22  2:24 ` [PATCH 3/7] script/lei: describe purpose of sleep loop Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 5/7] inbox: do not waste hash slot on httpbackend_limiter Eric Wong
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

The bit about reap_compress is no longer true since
LeiXSearch->query_done triggers it, instead.  I only noticed
this while working on "lei up".
---
 lib/PublicInbox/LEI.pm | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 29293e6c..a1cab55a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1110,7 +1110,6 @@ sub dclose {
 	my ($self) = @_;
 	delete $self->{-progress};
 	_drop_wq($self) if $self->{failed};
-	close(delete $self->{1}) if $self->{1}; # may reap_compress
 	$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
 }
 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 5/7] inbox: do not waste hash slot on httpbackend_limiter
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
                   ` (3 preceding siblings ...)
  2021-09-22  2:24 ` [PATCH 4/7] lei: dclose: do not close unnecessarily Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 6/7] lei up: avoid excessively parallel --all Eric Wong
  2021-09-22  2:24 ` [PATCH 7/7] lei: drop redundant WQ EOF callbacks Eric Wong
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

A few dozen bytes saved here can add up when we have thousands
of inboxes.  It also makes Data::Dumper debug output a bit cleaner.
---
 lib/PublicInbox/Inbox.pm | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 6cd20ec0..20f8c884 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -126,11 +126,12 @@ sub version {
 
 sub git_epoch {
 	my ($self, $epoch) = @_; # v2-only, callers always supply $epoch
-	$self->{"$epoch.git"} ||= do {
+	$self->{"$epoch.git"} //= do {
 		my $git_dir = "$self->{inboxdir}/git/$epoch.git";
 		return unless -d $git_dir;
 		my $g = PublicInbox::Git->new($git_dir);
-		$g->{-httpbackend_limiter} = $self->{-httpbackend_limiter};
+		my $lim = $self->{-httpbackend_limiter};
+		$g->{-httpbackend_limiter} = $lim if $lim;
 		# caller must manually cleanup when done
 		$g;
 	};
@@ -138,11 +139,12 @@ sub git_epoch {
 
 sub git {
 	my ($self) = @_;
-	$self->{git} ||= do {
+	$self->{git} //= do {
 		my $git_dir = $self->{inboxdir};
 		$git_dir .= '/all.git' if $self->version == 2;
 		my $g = PublicInbox::Git->new($git_dir);
-		$g->{-httpbackend_limiter} = $self->{-httpbackend_limiter};
+		my $lim = $self->{-httpbackend_limiter};
+		$g->{-httpbackend_limiter} = $lim if $lim;
 		_cleanup_later($self);
 		$g;
 	};

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 6/7] lei up: avoid excessively parallel --all
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
                   ` (4 preceding siblings ...)
  2021-09-22  2:24 ` [PATCH 5/7] inbox: do not waste hash slot on httpbackend_limiter Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  2021-09-22  2:24 ` [PATCH 7/7] lei: drop redundant WQ EOF callbacks Eric Wong
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

We shouldn't dispatch all outputs right away since they
can be expensive CPU-wise.  Instead, rely on DESTROY to
trigger further redispatches.

This also fixes a circular reference bug for the single-output
case that could lead to a leftover script/lei after MUA exit.

I'm not sure how --jobs/-j should work when the actual xsearch
and lei2mail has it's own parallelism ("--jobs=$X,$M"), but
it's better than having thousands of subtasks running.

Fixes: b34a267efff7b831 ("lei up: fix --mua with single output")
---
 lib/PublicInbox/LEI.pm   |  2 +-
 lib/PublicInbox/LeiUp.pm | 86 +++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 32 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a1cab55a..1305dfb8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1384,7 +1384,7 @@ sub fchdir {
 sub wq_eof { # EOF callback for main daemon
 	my ($lei) = @_;
 	my $wq1 = delete $lei->{wq1} // return $lei->fail; # already failed
-	$wq1->wq_wait_old(\&wq_done_wait, $lei);
+	$wq1->wq_wait_old($wq1->can('_wq_done_wait') // \&wq_done_wait, $lei);
 }
 
 sub watch_state_ok ($) {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 89cf0112..377a720e 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -36,7 +36,7 @@ sub up1 ($$) {
 		$lei->{opt}->{$k} //= $v;
 	}
 	my $o = $lei->{opt}->{output} // '';
-	return $lei->fail("lei.q.output unset in $f") if $o eq '';
+	return $lei->fail("lei.q.output unset in $f (out=$out)") if $o eq '';
 	$lss->translate_dedupe($lei) or return;
 	$lei->{lss} = $lss; # for LeiOverview->new and query_remote_mboxrd
 	my $lxs = $lei->lxs_prepare or return;
@@ -44,39 +44,30 @@ sub up1 ($$) {
 	$lei->_start_query;
 }
 
-sub up1_redispatch {
-	my ($lei, $out, $op_p) = @_;
-	my $l;
-	if (defined($lei->{opt}->{mua})) { # single output
-		$l = $lei;
-	} else { # multiple outputs
-		$l = bless { %$lei }, ref($lei);
-		$l->{opt} = { %{$l->{opt}} }; # deep copy
-		delete $l->{opt}->{all};
-		delete $l->{sock}; # do not close
-		# make close($l->{1}) happy in lei->dclose
-		open my $fh, '>&', $l->{1} or
-			return $l->child_error(0, "dup: $!");
-		$l->{1} = $fh;
-		$l->qerr("# updating $out");
-	}
-	$l->{''} = $op_p; # daemon only ($l => $lei => script/lei)
-	eval { $l->dispatch('up', $out) };
-	$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
-}
-
 sub redispatch_all ($$) {
 	my ($self, $lei) = @_;
+	my $upq = [ (@{$self->{local} // []}, @{$self->{remote} // []}) ];
+	return up1($lei, $upq->[0]) if @$upq == 1; # just one, may start MUA
+
+	# FIXME: this is also used per-query, see lei->_start_query
+	my $j = $lei->{opt}->{jobs} || do {
+		my $n = $self->detect_nproc // 1;
+		$n > 4 ? 4 : $n;
+	};
+	$j = ($j =~ /\A([0-9]+)/) ? $1 + 0 : 1; # may be --jobs=$x,$m on CLI
 	# re-dispatch into our event loop w/o creating an extra fork-level
+	# $upq will be drained via DESTROY as each query finishes
 	$lei->{fmsg} = PublicInbox::LeiFinmsg->new($lei);
 	my ($op_c, $op_p) = PublicInbox::PktOp->pair;
-	for my $o (@{$self->{local} // []}, @{$self->{remote} // []}) {
-		PublicInbox::DS::requeue(sub {
-			up1_redispatch($lei, $o, $op_p);
-		});
+	# call lei->dclose when upq is done processing:
+	$op_c->{ops} = { '' => [ $lei->can('dclose'), $lei ] };
+	my @first_batch = splice(@$upq, 0, $j); # initial parallelism
+	$lei->{-upq} = $upq;
+	$lei->event_step_init; # wait for client disconnects
+	for my $out (@first_batch) {
+		PublicInbox::DS::requeue(
+			PublicInbox::LeiUp1::nxt($lei, $out, $op_p));
 	}
-	$lei->event_step_init;
-	$lei->pkt_ops($op_c->{ops} = { '' => [$lei->can('dclose'), $lei] });
 }
 
 sub lei_up {
@@ -98,7 +89,7 @@ sub lei_up {
 		} else {
 			$lei->fail("only --all=$all not understood");
 		}
-	} elsif ($lei->{lse}) {
+	} elsif ($lei->{lse}) { # redispatched
 		scalar(@outs) == 1 or die "BUG: lse set w/ >1 out[@outs]";
 		return up1($lei, $outs[0]);
 	} else {
@@ -131,16 +122,49 @@ sub net_merge_all_done {
 	my ($self, $lei) = @_;
 	$lei->{net} = delete($self->{-net_new}) if $self->{-net_new};
 	$self->wq_close(1);
-	redispatch_all($self, $lei);
+	eval { redispatch_all($self, $lei) };
+	warn "E: $@" if $@;
 }
 
-sub _complete_up {
+sub _complete_up { # lei__complete hook
 	my ($lei, @argv) = @_;
 	my $match_cb = $lei->complete_url_prepare(\@argv);
 	map { $match_cb->($_) } PublicInbox::LeiSavedSearch::list($lei);
 }
 
+sub _wq_done_wait { # dwaitpid callback
+	my ($arg, $pid) = @_;
+	my ($wq, $lei) = @$arg;
+	$lei->child_error($?, 'auth failure') if $?
+}
+
 no warnings 'once';
 *ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
 
+package PublicInbox::LeiUp1; # for redispatch_all
+use strict;
+use v5.10.1;
+
+sub nxt ($$$) {
+	my ($lei, $out, $op_p) = @_;
+	bless { lei => $lei, out => $out, op_p => $op_p }, __PACKAGE__;
+}
+
+sub event_step { # runs via PublicInbox::DS::requeue
+	my ($self) = @_;
+	my $lei = $self->{lei}; # the original, from lei_up
+	my $l = bless { %$lei }, ref($lei); # per-output copy
+	delete($l->{sock}) or return; # client disconnected if {sock} is gone
+	$l->{opt} = { %{$l->{opt}} }; # deep copy
+	delete $l->{opt}->{all};
+	$l->qerr("# updating $self->{out}");
+	$l->{up_op_p} = $self->{op_p}; # ($l => $lei => script/lei)
+	eval { $l->dispatch('up', $self->{out}) };
+	$lei->child_error(0, $@) if $@ || $l->{failed}; # lei->fail()
+
+	# onto the next:
+	my $out = shift(@{$lei->{-upq}}) or return;
+	PublicInbox::DS::requeue(nxt($lei, $out, $self->{op_p}));
+}
+
 1;

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 7/7] lei: drop redundant WQ EOF callbacks
  2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
                   ` (5 preceding siblings ...)
  2021-09-22  2:24 ` [PATCH 6/7] lei up: avoid excessively parallel --all Eric Wong
@ 2021-09-22  2:24 ` Eric Wong
  6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2021-09-22  2:24 UTC (permalink / raw)
  To: meta

Redundant code is noise and therefore confusing :<
---
 lib/PublicInbox/LeiImportKw.pm  |  8 +-------
 lib/PublicInbox/LeiMirror.pm    | 10 ++--------
 lib/PublicInbox/LeiNoteEvent.pm |  8 +-------
 lib/PublicInbox/LeiPmdir.pm     |  8 +-------
 script/public-inbox-clone       |  2 +-
 5 files changed, 6 insertions(+), 30 deletions(-)

diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 21c93515..c35c5c26 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -46,17 +46,11 @@ sub ck_update_kw { # via wq_io_do
 	$self->{sto}->wq_do('set_eml_vmd', undef, { kw => $kw }, \@docids);
 }
 
-sub ikw_done_wait {
-	my ($arg, $pid) = @_;
-	my ($self, $lei) = @$arg;
-	$lei->can('wq_done_wait')->($arg, $pid);
-}
-
 sub _lei_wq_eof { # EOF callback for main lei daemon
 	my ($lei) = @_;
 	my $ikw = delete $lei->{ikw} or return $lei->fail;
 	$lei->sto_done_request($ikw->{lei_sock});
-	$ikw->wq_wait_old(\&ikw_done_wait, $lei);
+	$ikw->wq_wait_old($lei->can('wq_done_wait'), $lei);
 }
 
 1;
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index d9c13f05..6bfa4b6f 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -11,7 +11,7 @@ use PublicInbox::Spawn qw(popen_rd spawn run_die);
 use File::Temp ();
 use Fcntl qw(SEEK_SET O_CREAT O_EXCL O_WRONLY);
 
-sub do_finish_mirror { # dwaitpid callback
+sub _wq_done_wait { # dwaitpid callback (via wq_eof)
 	my ($arg, $pid) = @_;
 	my ($mrr, $lei) = @$arg;
 	my $f = "$mrr->{dst}/mirror.done";
@@ -28,12 +28,6 @@ sub do_finish_mirror { # dwaitpid callback
 	$lei->dclose;
 }
 
-sub _lei_wq_eof { # EOF callback for main daemon
-	my ($lei) = @_;
-	my $mrr = delete $lei->{wq1} or return $lei->fail;
-	$mrr->wq_wait_old(\&do_finish_mirror, $lei);
-}
-
 # for old installations without manifest.js.gz
 sub try_scrape {
 	my ($self) = @_;
@@ -176,7 +170,7 @@ sub index_cloned_inbox {
 		PublicInbox::Admin::progress_prepare($opt, $lei->{2});
 		PublicInbox::Admin::index_inbox($ibx, undef, $opt);
 	}
-	open my $x, '>', "$self->{dst}/mirror.done"; # for do_finish_mirror
+	open my $x, '>', "$self->{dst}/mirror.done"; # for _wq_done_wait
 }
 
 sub run_reap {
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 43d5ed0f..d2429485 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -107,17 +107,11 @@ sub ipc_atfork_child {
 	$self->SUPER::ipc_atfork_child;
 }
 
-sub lne_done_wait {
-	my ($arg, $pid) = @_;
-	my ($self, $lei) = @$arg;
-	$lei->can('wq_done_wait')->($arg, $pid);
-}
-
 sub _lei_wq_eof { # EOF callback for main lei daemon
 	my ($lei) = @_;
 	my $lne = delete $lei->{lne} or return $lei->fail;
 	$lei->sto_done_request;
-	$lne->wq_wait_old(\&lne_done_wait, $lei);
+	$lne->wq_wait_old($lei->can('wq_done_wait'), $lei);
 }
 
 1;
diff --git a/lib/PublicInbox/LeiPmdir.pm b/lib/PublicInbox/LeiPmdir.pm
index 23bccb4f..2d3b9755 100644
--- a/lib/PublicInbox/LeiPmdir.pm
+++ b/lib/PublicInbox/LeiPmdir.pm
@@ -47,17 +47,11 @@ sub mdir_iter { # via wq_io_do
 	$self->{ipt}->pmdir_cb($f, $fl, @args);
 }
 
-sub pmd_done_wait {
-	my ($arg, $pid) = @_;
-	my ($self, $lei) = @$arg;
-	$lei->can('wq_done_wait')->($arg, $pid);
-}
-
 sub _lei_wq_eof { # EOF callback for main lei daemon
 	my ($lei) = @_;
 	my $pmd = delete $lei->{pmd} or return $lei->fail;
 	$lei->sto_done_request($pmd->{lei_sock});
-	$pmd->wq_wait_old(\&pmd_done_wait, $lei);
+	$pmd->wq_wait_old($lei->can('wq_done_wait'), $lei);
 }
 
 1;
diff --git a/script/public-inbox-clone b/script/public-inbox-clone
index 2b18969f..0efde1a8 100755
--- a/script/public-inbox-clone
+++ b/script/public-inbox-clone
@@ -54,5 +54,5 @@ my $mrr = bless {
 	dst => $dst,
 }, 'PublicInbox::LeiMirror';
 $mrr->do_mirror;
-$mrr->can('do_finish_mirror')->([$mrr, $lei], $$);
+$mrr->can('_wq_done_wait')->([$mrr, $lei], $$);
 exit(($lei->{child_error} // 0) >> 8);

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-09-22  2:24 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-22  2:24 [PATCH 0/7] lei bugfixes and other fixes Eric Wong
2021-09-22  2:24 ` [PATCH 1/7] ipc: do not add "0" to $0 of solo workers Eric Wong
2021-09-22  2:24 ` [PATCH 2/7] treewide: fix %SIG localization, harder Eric Wong
2021-09-22  2:24 ` [PATCH 3/7] script/lei: describe purpose of sleep loop Eric Wong
2021-09-22  2:24 ` [PATCH 4/7] lei: dclose: do not close unnecessarily Eric Wong
2021-09-22  2:24 ` [PATCH 5/7] inbox: do not waste hash slot on httpbackend_limiter Eric Wong
2021-09-22  2:24 ` [PATCH 6/7] lei up: avoid excessively parallel --all Eric Wong
2021-09-22  2:24 ` [PATCH 7/7] lei: drop redundant WQ EOF callbacks Eric Wong

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).