user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* Re: [PATCH 10/17] daemon: avoid race when quitting workers
  2019-11-24  0:22  6%   ` [PATCH 10/17] daemon: avoid race when quitting workers Eric Wong
@ 2019-11-25  8:59  7%     ` Eric Wong
  0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2019-11-25  8:59 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> While the master process has a self-pipe to avoid missing
> signals, worker processes lack that aside from a pipe to
> detect master death.
> 
> That pipe doesn't exist when there's no master process,
> so it's possible DS::close never finishes because it
> never woke up from epoll_wait.  So create a pipe on
> the worker_quit signal and force it into epoll/kevent
> so it wakes up right away.

Nope.  A self-pipe in any pure-Perl form is insufficient,
since Perl_csighandler() can't wake up processes from any
sleeping syscalls (no-internal self-pipe/eventfd like
other VMs).

Fortunately, Perl exposes sigprocmask, so signalfd should be
usable on Linux.  EVFILT_SIGNAL also exists for kevent...

IO::Poll users might be out-of-luck (anybody try GNU/Hurd?)

^ permalink raw reply	[relevance 7%]

* [PATCH 10/17] daemon: avoid race when quitting workers
  2019-11-24  0:22  6% ` [PATCH 00/17] test fixes and cleanups Eric Wong
@ 2019-11-24  0:22  6%   ` Eric Wong
  2019-11-25  8:59  7%     ` Eric Wong
  0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2019-11-24  0:22 UTC (permalink / raw)
  To: meta

While the master process has a self-pipe to avoid missing
signals, worker processes lack that aside from a pipe to
detect master death.

That pipe doesn't exist when there's no master process,
so it's possible DS::close never finishes because it
never woke up from epoll_wait.  So create a pipe on
the worker_quit signal and force it into epoll/kevent
so it wakes up right away.
---
 lib/PublicInbox/Daemon.pm | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/Daemon.pm b/lib/PublicInbox/Daemon.pm
index 90f11137..0e3b95d2 100644
--- a/lib/PublicInbox/Daemon.pm
+++ b/lib/PublicInbox/Daemon.pm
@@ -252,6 +252,11 @@ sub daemonize () {
 	}
 }
 
+sub shrink_pipes {
+	if ($^O eq 'linux') { # 1031: F_SETPIPE_SZ, 4096: page size
+		fcntl($_, 1031, 4096) for @_;
+	}
+}
 
 sub worker_quit {
 	# killing again terminates immediately:
@@ -260,6 +265,17 @@ sub worker_quit {
 	$_->close foreach @listeners; # call PublicInbox::DS::close
 	@listeners = ();
 
+	# create a lazy self-pipe which kicks us out of the EventLoop
+	# so DS::PostEventLoop can fire
+	if (pipe(my ($r, $w))) {
+		shrink_pipes($w);
+
+		# shrink_pipes == noop
+		PublicInbox::ParentPipe->new($r, *shrink_pipes);
+		close $w; # wake up from the event loop
+	} else {
+		warn "E: pipe failed ($!), quit unreliable\n";
+	}
 	my $proc_name;
 	my $warn = 0;
 	# drop idle connections and try to quit gracefully
@@ -468,10 +484,7 @@ sub unlink_pid_file_safe_ish ($$) {
 sub master_loop {
 	pipe(my ($p0, $p1)) or die "failed to create parent-pipe: $!";
 	pipe(my ($r, $w)) or die "failed to create self-pipe: $!";
-
-	if ($^O eq 'linux') { # 1031: F_SETPIPE_SZ = 1031
-		fcntl($_, 1031, 4096) for ($w, $p1);
-	}
+	shrink_pipes($w, $p1);
 
 	IO::Handle::blocking($w, 0);
 	my $set_workers = $worker_processes;

^ permalink raw reply related	[relevance 6%]

* [PATCH 00/17] test fixes and cleanups
  @ 2019-11-24  0:22  6% ` Eric Wong
  2019-11-24  0:22  6%   ` [PATCH 10/17] daemon: avoid race when quitting workers Eric Wong
  0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2019-11-24  0:22 UTC (permalink / raw)
  To: meta

There's some fixes for race conditions around daemon
startup and shutdown and resurrects start_script for
slightly improved test performance.

And slowly eliminating all END{} block usages

Eric Wong (17):
  tests: disable daemon workers in a few more places
  tests: use strict everywhere
  t/v1-add-remove-add: quiet down "git init"
  t/xcpdb-reshard: test xcpdb --compact
  t/httpd-corner: wait for worker process death
  t/nntpd-tls: sometimes SSL_connect succeeds quickly
  .gitignore: ignore local prove(1) files
  daemon: use sigprocmask to block signals at startup
  daemon: use sigprocmask when respawning workers
  daemon: avoid race when quitting workers
  t/common: start_script replaces spawn_listener
  t/nntpd-validate: get rid of threads dependency
  xapcmd: replace Xtmpdirs with File::Temp->newdir
  tests: use File::Temp->newdir instead of tempdir()
  tests: quiet down commit graph
  t/perf-*.t: use $ENV{GIANT_INBOX_DIR} consistently
  tests: move giant inbox/git dependent tests to xt/

 .gitignore                   |   2 +
 MANIFEST                     |  11 ++-
 lib/PublicInbox/Daemon.pm    |  35 +++++--
 lib/PublicInbox/Xapcmd.pm    |  73 +++++---------
 t/.gitconfig                 |   4 +
 t/admin.t                    |   4 +-
 t/altid.t                    |   4 +-
 t/altid_v2.t                 |   3 +-
 t/cgi.t                      |   3 +-
 t/common.perl                | 184 ++++++++++++++++++++++++++---------
 t/config.t                   |   4 +-
 t/convert-compact.t          |   3 +-
 t/edit.t                     |   3 +-
 t/emergency.t                |   4 +-
 t/feed.t                     |   3 +-
 t/filter_rubylang.t          |   5 +-
 t/git.t                      |   6 +-
 t/html_index.t               |   4 +-
 t/httpd-corner.psgi          |   2 +-
 t/httpd-corner.t             |  70 +++++++------
 t/httpd-https.t              |  31 ++----
 t/httpd-unix.t               |  51 +++++-----
 t/httpd.t                    |  16 ++-
 t/import.t                   |   4 +-
 t/indexlevels-mirror.t       |   3 +-
 t/init.t                     |   3 +-
 t/mda.t                      |   3 +-
 t/mda_filter_rubylang.t      |   3 +-
 t/mid.t                      |   1 +
 t/msgmap.t                   |   4 +-
 t/nntpd-tls.t                |  42 +++-----
 t/nntpd.t                    |  19 ++--
 t/nulsubject.t               |   4 +-
 t/over.t                     |   4 +-
 t/plack.t                    |   4 +-
 t/psgi_attach.t              |   4 +-
 t/psgi_bad_mids.t            |   4 +-
 t/psgi_mount.t               |   4 +-
 t/psgi_multipart_not.t       |   4 +-
 t/psgi_scan_all.t            |   4 +-
 t/psgi_search.t              |   4 +-
 t/psgi_text.t                |   4 +-
 t/psgi_v2.t                  |   3 +-
 t/purge.t                    |   4 +-
 t/qspawn.t                   |   1 +
 t/replace.t                  |   6 +-
 t/search-thr-index.t         |   4 +-
 t/search.t                   |   4 +-
 t/solver_git.t               |   3 +-
 t/spamcheck_spamc.t          |   4 +-
 t/v1-add-remove-add.t        |   6 +-
 t/v1reindex.t                |   3 +-
 t/v2-add-remove-add.t        |   3 +-
 t/v2mda.t                    |   3 +-
 t/v2mirror.t                 |  26 ++---
 t/v2reindex.t                |   3 +-
 t/v2writable.t               |  13 ++-
 t/watch_filter_rubylang.t    |   3 +-
 t/watch_maildir.t            |   5 +-
 t/watch_maildir_v2.t         |  10 +-
 t/www_listing.t              |  11 +--
 t/xcpdb-reshard.t            |   7 +-
 {t => xt}/git-http-backend.t |  19 ++--
 {t => xt}/nntpd-validate.t   |  57 +++++------
 {t => xt}/perf-msgview.t     |   6 +-
 {t => xt}/perf-nntpd.t       |  34 +++----
 {t => xt}/perf-threading.t   |   8 +-
 67 files changed, 462 insertions(+), 431 deletions(-)
 create mode 100644 t/.gitconfig
 rename {t => xt}/git-http-backend.t (87%)
 rename {t => xt}/nntpd-validate.t (85%)
 rename {t => xt}/perf-msgview.t (85%)
 rename {t => xt}/perf-nntpd.t (79%)
 rename {t => xt}/perf-threading.t (72%)


^ permalink raw reply	[relevance 6%]

Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-11-16 11:43     [PATCH 29/29] t/common: start_script replaces spawn_listener Eric Wong
2019-11-24  0:22  6% ` [PATCH 00/17] test fixes and cleanups Eric Wong
2019-11-24  0:22  6%   ` [PATCH 10/17] daemon: avoid race when quitting workers Eric Wong
2019-11-25  8:59  7%     ` Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).