* [PATCH 0/3] improve error handling for rare cases
@ 2020-01-09 11:14 Eric Wong
2020-01-09 11:14 ` [PATCH 1/3] listener: EPOLL_CTL_ADD errors are non fatal Eric Wong
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Eric Wong @ 2020-01-09 11:14 UTC (permalink / raw)
To: meta
I tend to configure high RLIMIT_NOFILE limits, and the default
/proc/sys/fs/epoll/max_user_watches is difficult to hit, so
I've never hit the errors these patches supposedly fix...
But, I've noticed there's a places were we could die() without
an eval{} to trap it. These patches fix some of those.
On a side note, we still lack sufficient testing for rare error
conditions. These errors are not easy to reproduce, especially
EPOLL_CTL_ADD failures.
Also, the IO::KQueue->EV_SET calls actually make calls to
kqueue(2) directly (working more like epoll_ctl), so those could
die() inside the emulated epoll_ctl... Another day
Eric Wong (3):
listener: EPOLL_CTL_ADD errors are non fatal
http: log response_write errors
qspawn: catch transient errors on pipe, EPOLL_CTL_ADD
lib/PublicInbox/HTTP.pm | 5 ++++-
lib/PublicInbox/Listener.pm | 3 ++-
lib/PublicInbox/Qspawn.pm | 29 ++++++++++++++++++-----------
3 files changed, 24 insertions(+), 13 deletions(-)
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/3] listener: EPOLL_CTL_ADD errors are non fatal
2020-01-09 11:14 [PATCH 0/3] improve error handling for rare cases Eric Wong
@ 2020-01-09 11:14 ` Eric Wong
2020-01-09 11:14 ` [PATCH 2/3] http: log response_write errors Eric Wong
2020-01-09 11:14 ` [PATCH 3/3] qspawn: catch transient errors on pipe, EPOLL_CTL_ADD Eric Wong
2 siblings, 0 replies; 4+ messages in thread
From: Eric Wong @ 2020-01-09 11:14 UTC (permalink / raw)
To: meta
EPOLL_CTL_ADD may fail with transient ENOMEM or ENOSPC errors,
so don't tear down the process when that happens.
---
lib/PublicInbox/Listener.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/Listener.pm b/lib/PublicInbox/Listener.pm
index 928d9301..cdfd34a2 100644
--- a/lib/PublicInbox/Listener.pm
+++ b/lib/PublicInbox/Listener.pm
@@ -40,7 +40,8 @@ sub event_step {
# on high-traffic sites.
if (my $addr = accept(my $c, $sock)) {
IO::Handle::blocking($c, 0); # no accept4 :<
- $self->{post_accept}->($c, $addr, $sock);
+ eval { $self->{post_accept}->($c, $addr, $sock) };
+ warn "E: $@\n" if $@;
$self->requeue;
} elsif ($! == EAGAIN || $! == ECONNABORTED || $! == EPERM) {
# EAGAIN is common and likely
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/3] http: log response_write errors
2020-01-09 11:14 [PATCH 0/3] improve error handling for rare cases Eric Wong
2020-01-09 11:14 ` [PATCH 1/3] listener: EPOLL_CTL_ADD errors are non fatal Eric Wong
@ 2020-01-09 11:14 ` Eric Wong
2020-01-09 11:14 ` [PATCH 3/3] qspawn: catch transient errors on pipe, EPOLL_CTL_ADD Eric Wong
2 siblings, 0 replies; 4+ messages in thread
From: Eric Wong @ 2020-01-09 11:14 UTC (permalink / raw)
To: meta
Application-supplied callbacks may error out, try to log them
so the PSGI app developer can figure out what went wrong.
---
lib/PublicInbox/HTTP.pm | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/HTTP.pm b/lib/PublicInbox/HTTP.pm
index 1346901a..a6ec1d0d 100644
--- a/lib/PublicInbox/HTTP.pm
+++ b/lib/PublicInbox/HTTP.pm
@@ -174,7 +174,10 @@ sub app_dispatch {
response_write($self, $env, $res);
}
};
- $self->close if $@;
+ if ($@) {
+ err($self, "response_write error: $@");
+ $self->close;
+ }
}
sub response_header_write {
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 3/3] qspawn: catch transient errors on pipe, EPOLL_CTL_ADD
2020-01-09 11:14 [PATCH 0/3] improve error handling for rare cases Eric Wong
2020-01-09 11:14 ` [PATCH 1/3] listener: EPOLL_CTL_ADD errors are non fatal Eric Wong
2020-01-09 11:14 ` [PATCH 2/3] http: log response_write errors Eric Wong
@ 2020-01-09 11:14 ` Eric Wong
2 siblings, 0 replies; 4+ messages in thread
From: Eric Wong @ 2020-01-09 11:14 UTC (permalink / raw)
To: meta
popen_rd dies on pipe()/pipe2() failure due to FD exhaustion.
EPOLL_CTL_ADD (via PublicInbox::HTTPD::Async->new) may also fail
due to memory exhaustion or exceeding the value of
/proc/sys/fs/epoll/max_user_watches
---
lib/PublicInbox/Qspawn.pm | 29 ++++++++++++++++++-----------
1 file changed, 18 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 65bb178a..31a1583d 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -45,25 +45,28 @@ sub new ($$$;) {
sub _do_spawn {
my ($self, $start_cb, $limiter) = @_;
my $err;
- my ($cmd, $cmd_env, $opts) = @{$self->{args}};
- my %opts = %{$opts || {}};
+ my ($cmd, $cmd_env, $opt) = @{$self->{args}};
+ my %o = %{$opt || {}};
$self->{limiter} = $limiter;
foreach my $k (PublicInbox::Spawn::RLIMITS()) {
if (defined(my $rlimit = $limiter->{$k})) {
- $opts{$k} = $rlimit;
+ $o{$k} = $rlimit;
}
}
+ eval {
+ # popen_rd may die on EMFILE, ENFILE
+ ($self->{rpipe}, $self->{pid}) = popen_rd($cmd, $cmd_env, \%o);
+ $self->{args} = $o{quiet} ? undef : $cmd;
- ($self->{rpipe}, $self->{pid}) = popen_rd($cmd, $cmd_env, \%opts);
+ die "E: $!" unless defined($self->{pid});
- $self->{args} = $opts{quiet} ? undef : $cmd;
-
- if (defined $self->{pid}) {
$limiter->{running}++;
- } else {
- $self->{err} = $!;
+ $start_cb->($self); # EPOLL_CTL_ADD may ENOSPC/ENOMEM
+ };
+ if ($@) {
+ $self->{err} = $@;
+ finish($self);
}
- $start_cb->($self);
}
sub child_err ($) {
@@ -105,7 +108,11 @@ sub waitpid_err ($$) {
}
if ($err) {
- $self->{err} = $err;
+ if ($self->{err}) {
+ $self->{err} .= "; $err";
+ } else {
+ $self->{err} = $err;
+ }
if ($env && $self->{args}) {
log_err($env, join(' ', @{$self->{args}}) . ": $err");
}
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-01-09 11:14 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-09 11:14 [PATCH 0/3] improve error handling for rare cases Eric Wong
2020-01-09 11:14 ` [PATCH 1/3] listener: EPOLL_CTL_ADD errors are non fatal Eric Wong
2020-01-09 11:14 ` [PATCH 2/3] http: log response_write errors Eric Wong
2020-01-09 11:14 ` [PATCH 3/3] qspawn: catch transient errors on pipe, EPOLL_CTL_ADD Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).