user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 08/11] ds: avoid excessive queueing when reaping PIDs
Date: Mon, 31 Aug 2020 04:41:37 +0000
Message-ID: <20200831044140.17027-9-e@80x24.org> (raw)
In-Reply-To: <20200831044140.17027-1-e@80x24.org>

We should not enqueue reap_pids() to run more than once per
EventLoop iteration.  We'll start reformatting reap_pids
to tabs, too, since we're no longer Danga::Socket.

We should also be able to remove timer usage for reaping
down-the-line once we stop abusing dwaitpid() in -watch.
---
 lib/PublicInbox/DS.pm | 43 ++++++++++++++++++++++++++-----------------
 1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index a3f2e76c..b252ea3c 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -40,7 +40,7 @@ my $wait_pids; # list of [ pid, callback, callback_arg ]
 my $later_queue; # list of callbacks to run at some later interval
 my $EXPMAP; # fd -> idle_time
 our $EXPTIME = 180; # 3 minutes
-my ($later_timer, $reap_timer, $exp_timer);
+my ($later_timer, $reap_armed, $reap_timer, $exp_timer);
 my $ToClose; # sockets to close when event loop is done
 our (
      %DescriptorMap,             # fd (num) -> PublicInbox::DS object
@@ -68,7 +68,7 @@ Reset all state
 =cut
 sub Reset {
     %DescriptorMap = ();
-    $in_loop = $wait_pids = $later_queue = undef;
+    $in_loop = $wait_pids = $later_queue = $reap_armed = undef;
     $EXPMAP = {};
     $nextq = $ToClose = $reap_timer = $later_timer = $exp_timer = undef;
     $LoopTimeout = -1;  # no timeout by default
@@ -225,24 +225,33 @@ sub RunTimers {
 # and other things.  So we scan the $wait_pids list, which is hopefully
 # not too big.  We keep $wait_pids small by not calling dwaitpid()
 # until we've hit EOF when reading the stdout of the child.
+
 sub reap_pids {
-    my $tmp = $wait_pids or return;
-    $wait_pids = $reap_timer = undef;
-    foreach my $ary (@$tmp) {
-        my ($pid, $cb, $arg) = @$ary;
-        my $ret = waitpid($pid, WNOHANG);
-        if ($ret == 0) {
-            push @$wait_pids, $ary; # autovivifies @$wait_pids
-        } elsif ($cb) {
-            eval { $cb->($arg, $pid) };
-        }
-    }
-    # we may not be done, yet, and could've missed/masked a SIGCHLD:
-    $reap_timer = add_timer(1, \&reap_pids) if $wait_pids;
+	$reap_armed = undef;
+	my $tmp = $wait_pids or return;
+	$wait_pids = undef;
+	foreach my $ary (@$tmp) {
+		my ($pid, $cb, $arg) = @$ary;
+		my $ret = waitpid($pid, WNOHANG);
+		if ($ret == 0) {
+			push @$wait_pids, $ary; # autovivifies @$wait_pids
+		} elsif ($cb) {
+			eval { $cb->($arg, $pid) };
+		}
+	}
+	# we may not be done, yet, and could've missed/masked a SIGCHLD:
+	if ($wait_pids && !$reap_armed) {
+		$reap_timer //= add_timer(1, \&reap_pids_timed);
+	}
+}
+
+sub reap_pids_timed {
+	$reap_timer = undef;
+	goto \&reap_pids;
 }
 
 # reentrant SIGCHLD handler (since reap_pids is not reentrant)
-sub enqueue_reap ($) { push @$nextq, \&reap_pids }; # autovivifies
+sub enqueue_reap { $reap_armed //= requeue(\&reap_pids) }
 
 sub in_loop () { $in_loop }
 
@@ -627,7 +636,7 @@ sub dwaitpid ($$$) {
 	push @$wait_pids, [ @_ ]; # [ $pid, $cb, $arg ]
 
 	# We could've just missed our SIGCHLD, cover it, here:
-	requeue(\&reap_pids);
+	goto &enqueue_reap; # tail recursion
 }
 
 sub _run_later () {

  parent reply	other threads:[~2020-08-31  4:41 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-31  4:41 [PATCH 00/11] watch: fix contention w/ Maildir & NNTP Eric Wong
2020-08-31  4:41 ` [PATCH 01/11] watch: limit batch size of NNTP and IMAP workers, too Eric Wong
2020-08-31  4:41 ` [PATCH 02/11] watchmaildir: use v5.10.1, drop warnings Eric Wong
2020-08-31  4:41 ` [PATCH 03/11] rename WatchMaildir => Watch Eric Wong
2020-08-31  4:41 ` [PATCH 04/11] watch: log signal activities to STDERR Eric Wong
2020-08-31  4:41 ` [PATCH 05/11] watch: avoid unnecessary spawning on spam removals Eric Wong
2020-08-31  4:41 ` [PATCH 06/11] watch: block signals before fork on non-signalfd/kevent systems Eric Wong
2020-08-31  4:41 ` [PATCH 07/11] watch: comments and tiny cleanups Eric Wong
2020-08-31  4:41 ` Eric Wong [this message]
2020-08-31  4:41 ` [PATCH 09/11] watch: use EOFpipe to reduce dwaitpid wakeups Eric Wong
2020-08-31  4:41 ` [PATCH 10/11] ds: avoid unnecessary timer for waitpid Eric Wong
2020-08-31  4:41 ` [PATCH 11/11] replace ParentPipe with EOFpipe Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200831044140.17027-9-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git