[PATCH 0/4] www_coderepo vs cgit fallback fixes

[PATCH 0/4] www_coderepo vs cgit fallback fixes

[Eric Wong]

2/4 is critical in that it fixes occasional 500 errors noticed
in my cgit -> www_coderepo migration.  There's a few other
problems noticed an fixed along the way as well; having to
contort our code to PSGI expectations leads to some awkwardness,
unfortunately :<

Eric Wong (4):
  qspawn: use fallback response code from CGI program
  httpd/async: retry reads properly when parsing headers
  qspawn: consistently return 500 on premature EOF
  qspawn: fix EINTR with generic PSGI servers

 lib/PublicInbox/GitHTTPBackend.pm |  1 +
 lib/PublicInbox/HTTPD/Async.pm    |  8 +++++---
 lib/PublicInbox/Qspawn.pm         | 18 +++++++++++++-----
 3 files changed, 19 insertions(+), 8 deletions(-)

[PATCH 1/4] qspawn: use fallback response code from CGI program

[Eric Wong]

Prefer to use the original (cgit||git-http-backend) HTTP
response code if our fallback to WwwCoderepo fails.  404
codes is typically more appropriate than 500 for these things.
---
 lib/PublicInbox/GitHTTPBackend.pm | 1 +
 lib/PublicInbox/Qspawn.pm         | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/GitHTTPBackend.pm b/lib/PublicInbox/GitHTTPBackend.pm
index 5b879871..74432429 100644
--- a/lib/PublicInbox/GitHTTPBackend.pm
+++ b/lib/PublicInbox/GitHTTPBackend.pm
@@ -151,6 +151,7 @@ sub parse_cgi_headers { # {parse_hdr} for Qspawn
 	if ($code == 404 && $ctx->{www} && !$ctx->{_coderepo_tried}++) {
 		my $wcb = delete $ctx->{env}->{'qspawn.wcb'};
 		$ctx->{env}->{'plack.skip-deflater'} = 1; # prevent 2x gzip
+		$ctx->{env}->{'qspawn.fallback'} = $code;
 		my $res = $ctx->{www}->coderepo->srv($ctx);
 		# for ->psgi_return_init_cb
 		$ctx->{env}->{'qspawn.wcb'} = $wcb;
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index ab0ef25c..f8cbffcb 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -109,8 +109,9 @@ sub finalize ($$) {
 	return if $self->{passed}; # another command chained it
 	if (my $wcb = delete $env->{'qspawn.wcb'}) {
 		# have we started writing, yet?
+		my $code = delete $env->{'qspawn.fallback'} // 500;
 		require PublicInbox::WwwStatic;
-		$wcb->(PublicInbox::WwwStatic::r(500));
+		$wcb->(PublicInbox::WwwStatic::r($code));
 	}
 }
 

[PATCH 2/4] httpd/async: retry reads properly when parsing headers

[Eric Wong]

While git-http-backend sends headers with one write syscall,
upstream cgit still trickles them out line-by-line and we need to
account for that and retry Qspawn {parse_hdr} callbacks.
---
 lib/PublicInbox/HTTPD/Async.pm | 8 +++++---
 lib/PublicInbox/Qspawn.pm      | 4 ++--
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/HTTPD/Async.pm b/lib/PublicInbox/HTTPD/Async.pm
index e03daafa..7bbab1e1 100644
--- a/lib/PublicInbox/HTTPD/Async.pm
+++ b/lib/PublicInbox/HTTPD/Async.pm
@@ -14,7 +14,7 @@
 # arg: arg for {cb}
 # end_obj: CODE or object which responds to ->event_step when ->close is called
 package PublicInbox::HTTPD::Async;
-use strict;
+use v5.12;
 use parent qw(PublicInbox::DS);
 use Errno qw(EAGAIN);
 use PublicInbox::Syscall qw(EPOLLIN);
@@ -44,9 +44,9 @@ sub new {
 
 sub event_step {
 	my ($self) = @_;
-	if (my $cb = delete $self->{cb}) {
+	if (defined $self->{cb}) {
 		# this may call async_pass when headers are done
-		$cb->(my $refcnt_guard = delete $self->{arg});
+		$self->{cb}->($self->{arg});
 	} elsif (my $sock = $self->{sock}) {
 		# $http may be undef if discarding body output from cgit on 404
 		my $http = $self->{http} or return $self->close;
@@ -76,6 +76,7 @@ sub event_step {
 # may be PublicInbox::GzipFilter or $PublicInbox::Qspawn::qx_fh
 sub async_pass {
 	my ($self, $http, $ofh, $bref) = @_;
+	delete @$self{qw(cb arg)};
 	# In case the client HTTP connection ($http) dies, it
 	# will automatically close this ($self) object.
 	$http->{forward} = $self;
@@ -94,6 +95,7 @@ sub async_pass {
 sub close {
 	my $self = $_[0];
 	$self->SUPER::close; # DS::close
+	delete @$self{qw(cb arg)};
 
 	# we defer this to the next timer loop since close is deferred
 	if (my $end_obj = delete $self->{end_obj}) {
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index f8cbffcb..7984e35a 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -134,7 +134,7 @@ sub start ($$$) {
 	}
 }
 
-sub psgi_qx_init_cb {
+sub psgi_qx_init_cb { # this may be PublicInbox::HTTPD::Async {cb}
 	my ($self) = @_;
 	my $async = delete $self->{async}; # PublicInbox::HTTPD::Async
 	my ($r, $buf);
@@ -223,7 +223,7 @@ sub rd_hdr ($) {
 	$ret;
 }
 
-sub psgi_return_init_cb {
+sub psgi_return_init_cb { # this may be PublicInbox::HTTPD::Async {cb}
 	my ($self) = @_;
 	my $r = rd_hdr($self) or return;
 	my $env = $self->{psgi_env};

[PATCH 3/4] qspawn: consistently return 500 on premature EOF

[Eric Wong]

If {parse_hdr} callback doesn't handle it, we need to break the
loop if the CGI process dies prematurely.  This doesn't fix a
currently known problem, but theoretically a SIGKILL could hit
(cgit || git-http-backend) while -netd or -httpd survives.
---
 lib/PublicInbox/Qspawn.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 7984e35a..5d3ec309 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -210,6 +210,13 @@ sub rd_hdr ($) {
 			if ($@) {
 				warn "parse_hdr: $@";
 				$ret = [ 500, [], [ "Internal error\n" ] ];
+			} elsif (!defined($ret) && !$r) {
+				my $cmd = $self->{cmd} // [ '(?)' ];
+				my $env = $self->{psgi_env};
+				warn <<EOM;
+EOF parsing headers from @$cmd ($self->{psgi_env}->{REQUEST_URI})
+EOM
+				$ret = [ 500, [], [ "Internal error\n" ] ];
 			}
 		} else {
 			# caller should notify us when it's ready:

[PATCH 4/4] qspawn: fix EINTR with generic PSGI servers

[Eric Wong]

Using the `next' operator doesn't work with `do {} (until|while)'
loops, so change it to use `until {}'.  I've never encountered
this problem in-the-wild, but I only use -(netd|httpd).
---
 lib/PublicInbox/Qspawn.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 5d3ec309..f2beb4b2 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -201,7 +201,7 @@ sub rd_hdr ($) {
 	my $total_rd = 0;
 	my $hdr_buf = $self->{hdr_buf};
 	my ($ph_cb, $ph_arg) = @{$self->{parse_hdr}};
-	do {
+	until (defined($ret)) {
 		my $r = sysread($self->{rpipe}, $$hdr_buf, 4096,
 				length($$hdr_buf));
 		if (defined($r)) {
@@ -225,7 +225,7 @@ EOM
 			warn "error reading header: $!";
 			$ret = [ 500, [], [ "Internal error\n" ] ];
 		}
-	} until (defined $ret);
+	}
 	delete $self->{parse_hdr}; # done parsing headers
 	$ret;
 }