* [PATCH 08/11] ds: avoid excessive queueing when reaping PIDs
2020-08-31 4:41 6% [PATCH 00/11] watch: fix contention w/ Maildir & NNTP Eric Wong
@ 2020-08-31 4:41 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-08-31 4:41 UTC (permalink / raw)
To: meta
We should not enqueue reap_pids() to run more than once per
EventLoop iteration. We'll start reformatting reap_pids
to tabs, too, since we're no longer Danga::Socket.
We should also be able to remove timer usage for reaping
down-the-line once we stop abusing dwaitpid() in -watch.
---
lib/PublicInbox/DS.pm | 43 ++++++++++++++++++++++++++-----------------
1 file changed, 26 insertions(+), 17 deletions(-)
diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index a3f2e76c..b252ea3c 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -40,7 +40,7 @@ my $wait_pids; # list of [ pid, callback, callback_arg ]
my $later_queue; # list of callbacks to run at some later interval
my $EXPMAP; # fd -> idle_time
our $EXPTIME = 180; # 3 minutes
-my ($later_timer, $reap_timer, $exp_timer);
+my ($later_timer, $reap_armed, $reap_timer, $exp_timer);
my $ToClose; # sockets to close when event loop is done
our (
%DescriptorMap, # fd (num) -> PublicInbox::DS object
@@ -68,7 +68,7 @@ Reset all state
=cut
sub Reset {
%DescriptorMap = ();
- $in_loop = $wait_pids = $later_queue = undef;
+ $in_loop = $wait_pids = $later_queue = $reap_armed = undef;
$EXPMAP = {};
$nextq = $ToClose = $reap_timer = $later_timer = $exp_timer = undef;
$LoopTimeout = -1; # no timeout by default
@@ -225,24 +225,33 @@ sub RunTimers {
# and other things. So we scan the $wait_pids list, which is hopefully
# not too big. We keep $wait_pids small by not calling dwaitpid()
# until we've hit EOF when reading the stdout of the child.
+
sub reap_pids {
- my $tmp = $wait_pids or return;
- $wait_pids = $reap_timer = undef;
- foreach my $ary (@$tmp) {
- my ($pid, $cb, $arg) = @$ary;
- my $ret = waitpid($pid, WNOHANG);
- if ($ret == 0) {
- push @$wait_pids, $ary; # autovivifies @$wait_pids
- } elsif ($cb) {
- eval { $cb->($arg, $pid) };
- }
- }
- # we may not be done, yet, and could've missed/masked a SIGCHLD:
- $reap_timer = add_timer(1, \&reap_pids) if $wait_pids;
+ $reap_armed = undef;
+ my $tmp = $wait_pids or return;
+ $wait_pids = undef;
+ foreach my $ary (@$tmp) {
+ my ($pid, $cb, $arg) = @$ary;
+ my $ret = waitpid($pid, WNOHANG);
+ if ($ret == 0) {
+ push @$wait_pids, $ary; # autovivifies @$wait_pids
+ } elsif ($cb) {
+ eval { $cb->($arg, $pid) };
+ }
+ }
+ # we may not be done, yet, and could've missed/masked a SIGCHLD:
+ if ($wait_pids && !$reap_armed) {
+ $reap_timer //= add_timer(1, \&reap_pids_timed);
+ }
+}
+
+sub reap_pids_timed {
+ $reap_timer = undef;
+ goto \&reap_pids;
}
# reentrant SIGCHLD handler (since reap_pids is not reentrant)
-sub enqueue_reap ($) { push @$nextq, \&reap_pids }; # autovivifies
+sub enqueue_reap { $reap_armed //= requeue(\&reap_pids) }
sub in_loop () { $in_loop }
@@ -627,7 +636,7 @@ sub dwaitpid ($$$) {
push @$wait_pids, [ @_ ]; # [ $pid, $cb, $arg ]
# We could've just missed our SIGCHLD, cover it, here:
- requeue(\&reap_pids);
+ goto &enqueue_reap; # tail recursion
}
sub _run_later () {
^ permalink raw reply related [relevance 7%]
* [PATCH 00/11] watch: fix contention w/ Maildir & NNTP
@ 2020-08-31 4:41 6% Eric Wong
2020-08-31 4:41 7% ` [PATCH 08/11] ds: avoid excessive queueing when reaping PIDs Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-08-31 4:41 UTC (permalink / raw)
To: meta
Here's a bunch of fixes to improve watch performance when
both Maildirs and NNTP are being watched (possibly on the same
inbox, or if `watchspam' is configured for spam removals).
Wakeups are reduced, and inbox.lock contention is minimized by
using read-only ->over to check for `watchspam' removals.
These affect IMAP, too; but I've been mainly using NNTP.
Eric Wong (11):
watch: limit batch size of NNTP and IMAP workers, too
watchmaildir: use v5.10.1, drop warnings
rename WatchMaildir => Watch
watch: log signal activities to STDERR
watch: avoid unnecessary spawning on spam removals
watch: block signals before fork on non-signalfd/kevent systems
watch: comments and tiny cleanups
ds: avoid excessive queueing when reaping PIDs
watch: use EOFpipe to reduce dwaitpid wakeups
ds: avoid unnecessary timer for waitpid
replace ParentPipe with EOFpipe
MANIFEST | 4 +-
lib/PublicInbox/DS.pm | 38 +++---
lib/PublicInbox/Daemon.pm | 6 +-
lib/PublicInbox/EOFpipe.pm | 24 ++++
lib/PublicInbox/Import.pm | 3 +
lib/PublicInbox/ParentPipe.pm | 23 ----
lib/PublicInbox/V2Writable.pm | 3 +
lib/PublicInbox/{WatchMaildir.pm => Watch.pm} | 111 +++++++++++++-----
script/public-inbox-watch | 34 ++++--
t/imapd.t | 2 +-
t/nntpd.t | 2 +-
t/watch_filter_rubylang.t | 4 +-
t/watch_imap.t | 4 +-
t/watch_maildir.t | 18 +--
t/watch_maildir_v2.t | 22 ++--
t/watch_multiple_headers.t | 4 +-
t/watch_nntp.t | 4 +-
17 files changed, 190 insertions(+), 116 deletions(-)
create mode 100644 lib/PublicInbox/EOFpipe.pm
delete mode 100644 lib/PublicInbox/ParentPipe.pm
rename lib/PublicInbox/{WatchMaildir.pm => Watch.pm} (92%)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-08-31 4:41 6% [PATCH 00/11] watch: fix contention w/ Maildir & NNTP Eric Wong
2020-08-31 4:41 7% ` [PATCH 08/11] ds: avoid excessive queueing when reaping PIDs Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).