* [PATCH 2/2] nntp: correctly log long response errors
@ 2020-01-08 10:44 5% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-01-08 10:44 UTC (permalink / raw)
To: meta
We cannot safely call "fileno(undef)" without bringing down the
entire -nntpd process :x. To ensure no logging regression, we
now stash the FD for the duration of the long response to ensure
the error can be matched to the original command in logs.
Fixes: 207b89615a1a0c06 ("nntp: remove cyclic refs from long_response")
---
lib/PublicInbox/NNTP.pm | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 95c9082d..9f0dfaaa 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -596,12 +596,11 @@ sub get_range ($$) {
sub long_step {
my ($self) = @_;
# wbuf is unset or empty, here; {long} may add to it
- my ($cb, $t0, @args) = @{$self->{long_cb}};
+ my ($fd, $cb, $t0, @args) = @{$self->{long_cb}};
my $more = eval { $cb->($self, @args) };
if ($@ || !$self->{sock}) { # something bad happened...
delete $self->{long_cb};
my $elapsed = now() - $t0;
- my $fd = fileno($self->{sock});
if ($@) {
err($self,
"%s during long response[$fd] - %0.6f",
@@ -638,11 +637,11 @@ sub long_step {
sub long_response ($$;@) {
my ($self, $cb, @args) = @_; # cb returns true if more, false if done
- $self->{sock} or return;
+ my $sock = $self->{sock} or return;
# make sure we disable reading during a long response,
# clients should not be sending us stuff and making us do more
# work while we are stream a response to them
- $self->{long_cb} = [ $cb, now(), @args ];
+ $self->{long_cb} = [ fileno($sock), $cb, now(), @args ];
long_step($self); # kick off!
undef;
}
^ permalink raw reply related [relevance 5%]
* [PATCH 2/8] nntp: remove cyclic refs from long_response
2019-12-21 7:59 7% [PATCH 0/8] nntp: eliminate response-duration cyclic refs Eric Wong
@ 2019-12-21 8:00 5% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2019-12-21 8:00 UTC (permalink / raw)
To: meta
Leftover cyclic references are a source of memory leaks. While
our code is AFAIK unaffected by such leaks at the moment,
eliminating a potential source of bugs will make maintenance
easier.
We make the long_response API cycle-free by stashing the
callback into the NNTP object. However, callers will need
to be updated to get rid of the circular reference to $self.
We do that be replacing anonymous subs with name subroutine
references, such as xref_range_i replacing the formerly
anonymous sub inside hdr_xref.
---
lib/PublicInbox/NNTP.pm | 117 +++++++++++++++++++++-------------------
1 file changed, 61 insertions(+), 56 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index cd5d7bba..33c557f5 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -6,7 +6,7 @@ package PublicInbox::NNTP;
use strict;
use warnings;
use base qw(PublicInbox::DS);
-use fields qw(nntpd article ng);
+use fields qw(nntpd article ng long_cb);
use PublicInbox::Search;
use PublicInbox::Msgmap;
use PublicInbox::MID qw(mid_escape);
@@ -596,53 +596,57 @@ sub get_range ($$) {
[ \$beg, $end ];
}
-sub long_response ($$) {
- my ($self, $cb) = @_; # cb returns true if more, false if done
+sub long_step {
+ my ($self) = @_;
+ # wbuf is unset or empty, here; {long} may add to it
+ my ($cb, $t0, @args) = @{$self->{long_cb}};
+ my $more = eval { $cb->($self, @args) };
+ if ($@ || !$self->{sock}) { # something bad happened...
+ delete $self->{long_cb};
+ my $elapsed = now() - $t0;
+ my $fd = fileno($self->{sock});
+ if ($@) {
+ err($self,
+ "%s during long response[$fd] - %0.6f",
+ $@, $elapsed);
+ }
+ out($self, " deferred[$fd] aborted - %0.6f", $elapsed);
+ $self->close;
+ } elsif ($more) { # $self->{wbuf}:
+ $self->update_idle_time;
+
+ # COMPRESS users all share the same DEFLATE context.
+ # Flush it here to ensure clients don't see
+ # each other's data
+ $self->zflush;
+
+ # no recursion, schedule another call ASAP
+ # but only after all pending writes are done
+ my $wbuf = $self->{wbuf} ||= [];
+ push @$wbuf, \&long_step;
+
+ # wbuf may be populated by $cb, no need to rearm if so:
+ $self->requeue if scalar(@$wbuf) == 1;
+ } else { # all done!
+ delete $self->{long_cb};
+ res($self, '.');
+ my $elapsed = now() - $t0;
+ my $fd = fileno($self->{sock});
+ out($self, " deferred[$fd] done - %0.6f", $elapsed);
+ my $wbuf = $self->{wbuf};
+ $self->requeue unless $wbuf && @$wbuf;
+ }
+}
- my $fd = fileno($self->{sock});
- defined $fd or return;
+sub long_response ($$;@) {
+ my ($self, $cb, @args) = @_; # cb returns true if more, false if done
+
+ $self->{sock} or return;
# make sure we disable reading during a long response,
# clients should not be sending us stuff and making us do more
# work while we are stream a response to them
- my $t0 = now();
- my $long_cb; # DANGER: self-referential
- $long_cb = sub {
- # wbuf is unset or empty, here; $cb may add to it
- my $more = eval { $cb->() };
- if ($@ || !$self->{sock}) { # something bad happened...
- $long_cb = undef;
- my $diff = now() - $t0;
- if ($@) {
- err($self,
- "%s during long response[$fd] - %0.6f",
- $@, $diff);
- }
- out($self, " deferred[$fd] aborted - %0.6f", $diff);
- $self->close;
- } elsif ($more) { # $self->{wbuf}:
- $self->update_idle_time;
-
- # COMPRESS users all share the same DEFLATE context.
- # Flush it here to ensure clients don't see
- # each other's data
- $self->zflush;
-
- # no recursion, schedule another call ASAP
- # but only after all pending writes are done
- my $wbuf = $self->{wbuf} ||= [];
- push @$wbuf, $long_cb;
-
- # wbuf may be populated by $cb, no need to rearm if so:
- $self->requeue if scalar(@$wbuf) == 1;
- } else { # all done!
- $long_cb = undef;
- res($self, '.');
- out($self, " deferred[$fd] done - %0.6f", now() - $t0);
- my $wbuf = $self->{wbuf};
- $self->requeue unless $wbuf && @$wbuf;
- }
- };
- $self->write($long_cb); # kick off!
+ $self->{long_cb} = [ $cb, now(), @args ];
+ long_step($self); # kick off!
undef;
}
@@ -686,6 +690,18 @@ sub mid_lookup ($$) {
(undef, undef);
}
+sub xref_range_i {
+ my ($self, $beg, $end) = @_;
+ my $ng = $self->{ng};
+ my $r = $ng->mm->msg_range($beg, $end);
+ @$r or return;
+ more($self, join("\r\n", map {
+ my $num = $_->[0];
+ "$num ".xref($self, $ng, $num, $_->[1]);
+ } @$r));
+ 1;
+}
+
sub hdr_xref ($$$) { # optimize XHDR Xref [range] for rtin
my ($self, $xhdr, $range) = @_;
@@ -699,19 +715,8 @@ sub hdr_xref ($$$) { # optimize XHDR Xref [range] for rtin
$range = $self->{article} unless defined $range;
my $r = get_range($self, $range);
return $r unless ref $r;
- my $ng = $self->{ng};
- my $mm = $ng->mm;
- my ($beg, $end) = @$r;
more($self, $xhdr ? r221 : r225);
- long_response($self, sub {
- my $r = $mm->msg_range($beg, $end);
- @$r or return;
- more($self, join("\r\n", map {
- my $num = $_->[0];
- "$num ".xref($self, $ng, $num, $_->[1]);
- } @$r));
- 1;
- });
+ long_response($self, \&xref_range_i, @$r);
}
}
^ permalink raw reply related [relevance 5%]
* [PATCH 0/8] nntp: eliminate response-duration cyclic refs
@ 2019-12-21 7:59 7% Eric Wong
2019-12-21 8:00 5% ` [PATCH 2/8] nntp: remove cyclic refs from long_response Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2019-12-21 7:59 UTC (permalink / raw)
To: meta
While there are no known memory leaks in our code, the short-lived
reference cycle introduced by the long_response API is a potential
source of bugs.
Change the long_response API and update callers to pass
parameter explicitly to named subroutines instead of anonymous
ones.
Since creating new anonymous subs (aka "closures") take a fair
amount of memory in Perl, this ought to reduce memory pressure,
as well, since we're now only creating lightweight references to
long-lived subs. The tradeoff is we make a few more hash
lookups since we no longer cache some hash lookups into local
variables onto the stack.
Eric Wong (8):
nntp: get_range: return scalarref for $beg
nntp: remove cyclic refs from long_response
nntp: hdr_searchmsg: use named sub for numeric range response
nntp: cmd_xrover: use named sub for long_response
nntp: cmd_listgroup: use named subs for long_response
nntp: cmd_newnews: use named sub for long_response
nntp: hdr_msg_id: use named sub for long_response
nntp: cmd_xover: use named sub for long_response
lib/PublicInbox/NNTP.pm | 293 +++++++++++++++++++++-------------------
1 file changed, 152 insertions(+), 141 deletions(-)
^ permalink raw reply [relevance 7%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-12-21 7:59 7% [PATCH 0/8] nntp: eliminate response-duration cyclic refs Eric Wong
2019-12-21 8:00 5% ` [PATCH 2/8] nntp: remove cyclic refs from long_response Eric Wong
2020-01-08 10:44 [PATCH 0/2] nntpd and daemon bugfixes Eric Wong
2020-01-08 10:44 5% ` [PATCH 2/2] nntp: correctly log long response errors Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).