user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 1/9] ds: improve DS->Reset fork-safety
Date: Wed, 24 Mar 2021 14:23:27 +0500	[thread overview]
Message-ID: <20210324092335.12345-2-e@80x24.org> (raw)
In-Reply-To: <20210324092335.12345-1-e@80x24.org>

None of these fixes affect current public-inbox-* code, or even
normal uses of lei.  However, lei users wanting to switch
between $HOME directories or use alternate store paths may
notice strange behavior and this fixes some of it.

We'll also loop to account for DESTROY callbacks inserting into
container objects and retry appropriately.
---
 lib/PublicInbox/DS.pm | 76 ++++++++++++++++++++++++++-----------------
 1 file changed, 46 insertions(+), 30 deletions(-)

diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index 15ece4df..c881c895 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -35,9 +35,10 @@ use Errno qw(EAGAIN EINVAL);
 use Carp qw(carp croak);
 our @EXPORT_OK = qw(now msg_more dwaitpid add_timer);
 
+our %Stack;
 my $nextq; # queue for next_tick
 my $wait_pids; # list of [ pid, callback, callback_arg ]
-my $later_queue; # list of callbacks to run at some later interval
+my $later_q; # list of callbacks to run at some later interval
 my $EXPMAP; # fd -> idle_time
 our $EXPTIME = 180; # 3 minutes
 my ($later_timer, $reap_armed, $exp_timer);
@@ -65,19 +66,27 @@ Reset();
 Reset all state
 
 =cut
+our %state;
 sub Reset {
-    $in_loop = undef; # first in case DESTROY callbacks use this
-    %DescriptorMap = ();
-    $wait_pids = $later_queue = $reap_armed = undef;
-    $EXPMAP = {};
-    $nextq = $ToClose = $later_timer = $exp_timer = undef;
-    $LoopTimeout = -1;  # no timeout by default
-    @Timers = ();
-
-    $PostLoopCallback = undef;
-
-    $_io = undef; # closes real $Epoll FD
-    $Epoll = undef; # may call DSKQXS::DESTROY
+	do {
+		$in_loop = undef; # first in case DESTROY callbacks use this
+		%DescriptorMap = ();
+		@Timers = ();
+		$PostLoopCallback = undef;
+
+		# we may be iterating inside one of these on our stack
+		my @q = delete @Stack{keys %Stack};
+		for my $q (@q) { @$q = () }
+		$EXPMAP = {};
+		$wait_pids = $later_q = $nextq = $ToClose = undef;
+		$_io = undef; # closes real $Epoll FD
+		$Epoll = undef; # may call DSKQXS::DESTROY
+	} while (@Timers || keys(%Stack) || $nextq || $wait_pids ||
+		$later_q || $ToClose || keys(%DescriptorMap) ||
+		$PostLoopCallback);
+
+	$reap_armed = $later_timer = $exp_timer = undef;
+	$LoopTimeout = -1;  # no timeout by default
 }
 
 =head2 C<< CLASS->SetLoopTimeout( $timeout ) >>
@@ -160,17 +169,19 @@ C<PostLoopCallback> below for how to exit the loop.
 sub now () { clock_gettime(CLOCK_MONOTONIC) }
 
 sub next_tick () {
-    my $q = $nextq or return;
-    $nextq = undef;
-    for my $obj (@$q) {
-        # we avoid "ref" on blessed refs to workaround a Perl 5.16.3 leak:
-        # https://rt.perl.org/Public/Bug/Display.html?id=114340
-        if (blessed($obj)) {
-            $obj->event_step;
-        } else {
-            $obj->();
-        }
-    }
+	my $q = $nextq or return;
+	$nextq = undef;
+	$Stack{cur_runq} = $q;
+	for my $obj (@$q) {
+		# avoid "ref" on blessed refs to workaround a Perl 5.16.3 leak:
+		# https://rt.perl.org/Public/Bug/Display.html?id=114340
+		if (blessed($obj)) {
+			$obj->event_step;
+		} else {
+			$obj->();
+		}
+	}
+	delete $Stack{cur_runq};
 }
 
 # runs timers and returns milliseconds for next one, or next event loop
@@ -221,6 +232,7 @@ sub reap_pids {
 	$reap_armed = undef;
 	my $tmp = $wait_pids or return;
 	$wait_pids = undef;
+	$Stack{reap_runq} = $tmp;
 	my $oldset = block_signals();
 	foreach my $ary (@$tmp) {
 		my ($pid, $cb, $arg) = @$ary;
@@ -237,6 +249,7 @@ sub reap_pids {
 		}
 	}
 	sig_setmask($oldset);
+	delete $Stack{reap_runq};
 }
 
 # reentrant SIGCHLD handler (since reap_pids is not reentrant)
@@ -271,7 +284,6 @@ sub EventLoop {
     $Epoll //= _InitPoller();
     local $in_loop = 1;
     my @events;
-    my $obj; # guard stack-not-refcounted w/ Carp + @DB::args
     do {
         my $timeout = RunTimers();
 
@@ -282,7 +294,9 @@ sub EventLoop {
             # that ones in the front triggered unregister-interest actions.  if we
             # can't find the %sock entry, it's because we're no longer interested
             # in that event.
-            $obj = $DescriptorMap{$fd};
+
+	    # guard stack-not-refcounted w/ Carp + @DB::args
+            my $obj = $DescriptorMap{$fd};
             $obj->event_step;
         }
     } while (PostEventLoop());
@@ -646,13 +660,15 @@ sub dwaitpid ($;$$) {
 }
 
 sub _run_later () {
-	my $run = $later_queue or return;
-	$later_timer = $later_queue = undef;
-	$_->() for @$run;
+	my $q = $later_q or return;
+	$later_timer = $later_q = undef;
+	$Stack{later_q} = $q;
+	$_->() for @$q;
+	delete $Stack{later_q};
 }
 
 sub later ($) {
-	push @$later_queue, $_[0]; # autovivifies @$later_queue
+	push @$later_q, $_[0]; # autovivifies @$later_q
 	$later_timer //= add_timer(60, \&_run_later);
 }
 

  reply	other threads:[~2021-03-24  9:23 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-24  9:23 [PATCH 0/9] lei: various corner case leak fixes Eric Wong
2021-03-24  9:23 ` Eric Wong [this message]
2021-03-24 23:01   ` [SQUASH] [PATCH 1/9] ds: improve DS->Reset fork-safety Eric Wong
2021-03-24  9:23 ` [PATCH 2/9] mbox_lock: dotlock: chdir for relative lock paths Eric Wong
2021-03-24  9:23 ` [PATCH 3/9] lei: drop circular reference in lei_store process Eric Wong
2021-03-24  9:23 ` [PATCH 4/9] lei: update {3} after -C chdirs Eric Wong
2021-03-24  9:23 ` [PATCH 5/9] lei: clean up pkt_op consumer on exception, too Eric Wong
2021-03-24  9:23 ` [PATCH 6/9] lei_store: give process a better name Eric Wong
2021-03-24  9:23 ` [PATCH 7/9] v2writable: cleanup SQLite handles on --xapian-only Eric Wong
2021-03-24  9:23 ` [PATCH 8/9] lei_mirror: fix circular reference Eric Wong
2021-03-24  9:23 ` [PATCH 9/9] lei-daemon: do not leak FDs on bogus requests Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210324092335.12345-2-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    --subject='Re: [PATCH 1/9] ds: improve DS->Reset fork-safety' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).