* [PATCH 0/3] more networking/daemon fixes
@ 2016-05-24 4:22 7% Eric Wong
2016-05-24 4:22 6% ` [PATCH 2/3] http: fix various race conditions Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2016-05-24 4:22 UTC (permalink / raw)
To: meta
This series finally reinstates process limiting by queueing
processes instead of rejecting and falling back to dumb.
Falling back to dumb fails badly if we have to switch in
the middle of a pack negotiation.
I've tested it a bunch with aborted git clones of large
repositories with our own -httpd (using async) and fixed a few
longstanding bugs. The generic PSGI code still needs work
to resume properly and to avoid leaking references.
Eric Wong (3):
standardize timer-related event-loop code
http: fix various race conditions
git-http-backend: use qspawn to limit running processes
lib/PublicInbox/EvCleanup.pm | 41 ++++++++++++++++++++
lib/PublicInbox/GitHTTPBackend.pm | 38 ++++++++-----------
lib/PublicInbox/HTTP.pm | 80 +++++++++++++++++++--------------------
lib/PublicInbox/HTTPD/Async.pm | 3 +-
lib/PublicInbox/NNTP.pm | 30 +++++++--------
lib/PublicInbox/Qspawn.pm | 52 +++++++++++++++++++++++++
t/qspawn.t | 60 +++++++++++++++++++++++++++++
7 files changed, 223 insertions(+), 81 deletions(-)
^ permalink raw reply [relevance 7%]
* [PATCH 2/3] http: fix various race conditions
2016-05-24 4:22 7% [PATCH 0/3] more networking/daemon fixes Eric Wong
@ 2016-05-24 4:22 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2016-05-24 4:22 UTC (permalink / raw)
To: meta
We no longer override Danga::Socket::event_write and instead
re-enable reads by queuing up another callback in the $close
response callback. This is necessary because event_write may not be
completely done writing a response, only the existing buffered data.
Furthermore, the {closed} field can almost be set at any time when
writing, so we must check it before acting on pipelined requests as
well as during write callbacks in more().
---
lib/PublicInbox/HTTP.pm | 60 ++++++++++++++++++++++++-------------------------
1 file changed, 30 insertions(+), 30 deletions(-)
diff --git a/lib/PublicInbox/HTTP.pm b/lib/PublicInbox/HTTP.pm
index 00c9a04..6aae5c8 100644
--- a/lib/PublicInbox/HTTP.pm
+++ b/lib/PublicInbox/HTTP.pm
@@ -39,7 +39,10 @@ sub process_pipelineq () {
my $q = $pipelineq;
$pipet = undef;
$pipelineq = [];
- rbuf_process($_) foreach @$q;
+ foreach (@$q) {
+ next if $_->{closed};
+ rbuf_process($_);
+ }
}
# Use the same configuration parameter as git since this is primarily
@@ -228,26 +231,36 @@ sub identity_wcb ($) {
sub { $self->write(\($_[0])) if $_[0] ne '' }
}
+sub next_request ($) {
+ my ($self) = @_;
+ $self->watch_write(0);
+ if ($self->{rbuf} eq '') { # wait for next request
+ $self->watch_read(1);
+ } else { # avoid recursion for pipelined requests
+ push @$pipelineq, $self;
+ $pipet ||= PublicInbox::EvCleanup::asap(*process_pipelineq);
+ }
+}
+
+sub response_done ($$) {
+ my ($self, $alive) = @_;
+ my $env = $self->{env};
+ $self->{env} = undef;
+ $self->write("0\r\n\r\n") if $alive == 2;
+ $self->write(sub { $alive ? next_request($self) : $self->close });
+ if (my $obj = $env->{'pi-httpd.inbox'}) {
+ # grace period for reaping resources
+ $WEAKEN->{"$obj"} = $obj;
+ PublicInbox::EvCleanup::later(*weaken_task);
+ }
+}
+
sub response_write {
my ($self, $env, $res) = @_;
my $alive = response_header_write($self, $env, $res);
my $write = $alive == 2 ? chunked_wcb($self) : identity_wcb($self);
- my $close = sub {
- $self->write("0\r\n\r\n") if $alive == 2;
- if ($alive) {
- $self->event_write; # watch for readability if done
- } else {
- Danga::Socket::write($self, sub { $self->close });
- }
- if (my $obj = $env->{'pi-httpd.inbox'}) {
- # grace period for reaping resources
- $WEAKEN->{"$obj"} = $obj;
- $weakt ||= PublicInbox::EvCleanup::later(*weaken_task);
- }
- $self->{env} = undef;
- };
-
+ my $close = sub { response_done($self, $alive) };
if (defined(my $body = $res->[2])) {
if (ref $body eq 'ARRAY') {
$write->($_) foreach @$body;
@@ -278,6 +291,7 @@ sub response_write {
use constant MSG_MORE => ($^O eq 'linux') ? 0x8000 : 0;
sub more ($$) {
my $self = $_[0];
+ return if $self->{closed};
if (MSG_MORE && !$self->{write_buf_size}) {
my $n = send($self->{sock}, $_[1], MSG_MORE);
if (defined $n) {
@@ -290,20 +304,6 @@ sub more ($$) {
$self->write($_[1]);
}
-# overrides existing Danga::Socket method
-sub event_write {
- my ($self) = @_;
- # only continue watching for readability when we are done writing:
- return if $self->write(undef) != 1;
-
- if ($self->{rbuf} eq '') { # wait for next request
- $self->watch_read(1);
- } else { # avoid recursion for pipelined requests
- push @$pipelineq, $self;
- $pipet ||= PublicInbox::EvCleanup::asap(*process_pipelineq);
- }
-}
-
sub input_prepare {
my ($self, $env) = @_;
my $input = $null_io;
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2016-05-24 4:22 7% [PATCH 0/3] more networking/daemon fixes Eric Wong
2016-05-24 4:22 6% ` [PATCH 2/3] http: fix various race conditions Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).