* Re: [PATCH 5/5] watch: make waitpid() synchronous for Maildir scans
2020-06-29 10:34 7% ` [PATCH 5/5] watch: make waitpid() synchronous for Maildir scans Eric Wong
@ 2020-06-29 10:37 7% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-06-29 10:37 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong <e@yhbt.net> wrote:
> + local $PublicInbox::DS::in_loop = 0; # waitpid() synchronlusly
> + local $PublicInbox::DS::in_loop = 0; # waitpid() synchronlusly
"synchronously" :x
^ permalink raw reply [relevance 7%]
* [PATCH 5/5] watch: make waitpid() synchronous for Maildir scans
2020-06-29 10:34 6% ` [PATCH 0/5] watch: Maildir fixes Eric Wong
@ 2020-06-29 10:34 7% ` Eric Wong
2020-06-29 10:37 7% ` Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2020-06-29 10:34 UTC (permalink / raw)
To: meta
Maildir scanning still happens in the main process. Scanning
dozens of Maildirs is still time-consuming and monopolizes the
event loop during WatchMaildir::event_step. This can cause
cause zombies to accumulate before Sigfd::event_step triggers
DS::reap_pids.
---
lib/PublicInbox/WatchMaildir.pm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/WatchMaildir.pm b/lib/PublicInbox/WatchMaildir.pm
index 288f64d1e6c..d147962994e 100644
--- a/lib/PublicInbox/WatchMaildir.pm
+++ b/lib/PublicInbox/WatchMaildir.pm
@@ -123,9 +123,9 @@ sub new {
sub _done_for_now {
my ($self) = @_;
- my $importers = $self->{importers};
- foreach my $im (values %$importers) {
- $im->done;
+ local $PublicInbox::DS::in_loop = 0; # waitpid() synchronlusly
+ for (values %{$self->{importers}}) {
+ $_->done if $_; # $_ may be undef during cleanup
}
}
@@ -936,6 +936,7 @@ sub fs_scan_step {
my ($self) = @_;
return if $self->{quit};
my $op = shift @{$self->{ops}};
+ local $PublicInbox::DS::in_loop = 0; # waitpid() synchronlusly
# continue existing scan
my $max = 10;
^ permalink raw reply related [relevance 7%]
* [PATCH 0/5] watch: Maildir fixes
@ 2020-06-29 10:34 6% ` Eric Wong
2020-06-29 10:34 7% ` [PATCH 5/5] watch: make waitpid() synchronous for Maildir scans Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2020-06-29 10:34 UTC (permalink / raw)
To: meta
The introduction of NNTP and IMAP support in -watch introduced
some annoying-but-non-fatal regressions for existing Maildir
users, in one case due to a race fix.
https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
It turned out -watch could missing full scans at startup due
to the old alarm(1)-triggered scan firing too soon, and
switching to signalfd(2)[*] exposed performance problems fixed
by [PATCH 1/5] in this series.
PATCH 2/5 was just a minor debugging change while I was fixing
commit 52e37476e8e62e8e54104d9a2abcf2a86d1d1a32
("eml: header_str_set: correctly encode UTF-8 headers")
3 & 4 may not be strictly necessary at the moment, but they
could be in the future if subprocesses (spamc, git) we spawn
ever depend on SIGCHLD. More signals may be unblocked in
subprocesses in the future, perhaps only SIGINT (from Ctrl-C)
could remain masked.
Finally, 5/5 fixes the zombies introduced which kicked off
the investigation into this series.
Eric Wong (5):
watch: check for duplicates in ->over before spamcheck
watch: show path for warnings from spam messages
watch: ensure SIGCHLD works in forked children
spawn: unblock SIGCHLD in subprocess
watch: make waitpid() synchronous for Maildir scans
lib/PublicInbox/Import.pm | 2 +-
lib/PublicInbox/Spawn.pm | 11 +++++++---
lib/PublicInbox/SpawnPP.pm | 5 +++++
lib/PublicInbox/V2Writable.pm | 2 +-
lib/PublicInbox/WatchMaildir.pm | 37 +++++++++++++++++++++++++--------
t/spawn.t | 27 ++++++++++++++++++++++++
6 files changed, 70 insertions(+), 14 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-06-27 10:03 [PATCH 00/34] watch: add IMAP and NNTP support Eric Wong
2020-06-29 10:34 6% ` [PATCH 0/5] watch: Maildir fixes Eric Wong
2020-06-29 10:34 7% ` [PATCH 5/5] watch: make waitpid() synchronous for Maildir scans Eric Wong
2020-06-29 10:37 7% ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).