* [PATCH 00/14] convert solver to use pi-httpd.async
@ 2019-01-27 4:03 7% Eric Wong
2019-01-27 4:03 6% ` [PATCH 03/14] qspawn|getlinebody: support streaming filters Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-01-27 4:03 UTC (permalink / raw)
To: meta
Much of the groundwork for this was laid in the now-abandoned
"repobrowse" branch. The goal was to improves fairness as we no
longer wait synchronously on git (apply|update-index|ls-files)
processes and can requests for other clients.
The end result was slightly (2-3%?) slower with all the
callbacks, but reducing "git apply" invocations by relying on
pathnames (instead of stdin) made the end result ~20% faster for
a large (64) patch series.
Email::Simple (via Email::MIME/PublicInbox::MIME) remains a
performance bottleneck, as it does a lot of unnecessary header
parsing and hash-table populating we don't care about; but I'm
not sure if I'll have time to address that.
Eric Wong (14):
httpd/async: remove needless sysread wrapper
qspawn: implement psgi_return and use it for githttpbackend
qspawn|getlinebody: support streaming filters
qspawn|httpd/async: improve and fix out-of-date comments
httpd/async: stop running command if client disconnects
qspawn: implement psgi_qx
t/qspawn.t: psgi_qx stderr test
view: swap CRLF for LF in HTML output
solver: rewrite to use Qspawn->psgi_qx and pi-httpd.async
solver: hold patches in temporary directory
solver: reduce "git apply" invocations
qspawn: decode $? for user-friendliness
viewvcs: do not show final error message twice
solver: crank up max patches to 9999
lib/PublicInbox/GetlineBody.pm | 16 +-
lib/PublicInbox/Git.pm | 2 +-
lib/PublicInbox/GitHTTPBackend.pm | 64 +---
lib/PublicInbox/HTTPD/Async.pm | 27 +-
lib/PublicInbox/Qspawn.pm | 143 +++++++-
lib/PublicInbox/SolverGit.pm | 532 +++++++++++++++++-------------
lib/PublicInbox/View.pm | 4 +
lib/PublicInbox/ViewVCS.pm | 50 ++-
t/qspawn.t | 12 +-
t/solver_git.t | 22 +-
10 files changed, 543 insertions(+), 329 deletions(-)
--
EW
^ permalink raw reply [relevance 7%]
* [PATCH 03/14] qspawn|getlinebody: support streaming filters
2019-01-27 4:03 7% [PATCH 00/14] convert solver to use pi-httpd.async Eric Wong
@ 2019-01-27 4:03 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-01-27 4:03 UTC (permalink / raw)
To: meta
This is intended for wrapping "git show" and "git diff"
processes in the future and to prevent it from monopolizing
callers.
This will us to better handle backpressure from gigantic
commits.
---
lib/PublicInbox/GetlineBody.pm | 16 +++++++++++++---
lib/PublicInbox/Qspawn.pm | 21 +++++++++++++++++++--
2 files changed, 32 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/GetlineBody.pm b/lib/PublicInbox/GetlineBody.pm
index ea07f3d..0a922fd 100644
--- a/lib/PublicInbox/GetlineBody.pm
+++ b/lib/PublicInbox/GetlineBody.pm
@@ -13,8 +13,13 @@ use strict;
use warnings;
sub new {
- my ($class, $rpipe, $end, $buf) = @_;
- bless { rpipe => $rpipe, end => $end, buf => $buf }, $class;
+ my ($class, $rpipe, $end, $buf, $filter) = @_;
+ bless {
+ rpipe => $rpipe,
+ end => $end,
+ buf => $buf,
+ filter => $filter || 0,
+ }, $class;
}
# close should always be called after getline returns undef,
@@ -24,8 +29,13 @@ sub DESTROY { $_[0]->close }
sub getline {
my ($self) = @_;
+ my $filter = $self->{filter};
+ return if $filter == -1; # last call was EOF
+
my $buf = delete $self->{buf}; # initial buffer
- defined $buf ? $buf : $self->{rpipe}->getline;
+ $buf = $self->{rpipe}->getline unless defined $buf;
+ $self->{filter} = -1 unless defined $buf; # set EOF for next call
+ $filter ? $filter->($buf) : $buf;
}
sub close {
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index b80dac1..3247cd0 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -9,6 +9,7 @@ package PublicInbox::Qspawn;
use strict;
use warnings;
use PublicInbox::Spawn qw(popen_rd);
+require Plack::Util;
my $def_limiter;
sub new ($$$;) {
@@ -60,11 +61,25 @@ sub start {
}
}
+# create a filter for "push"-based streaming PSGI writes used by HTTPD::Async
+sub filter_fh ($$) {
+ my ($fh, $filter) = @_;
+ Plack::Util::inline_object(
+ close => sub {
+ $fh->write($filter->(undef));
+ $fh->close;
+ },
+ write => sub {
+ $fh->write($filter->($_[0]));
+ });
+}
+
sub psgi_return {
my ($self, $env, $limiter, $parse_hdr) = @_;
my ($fh, $rpipe);
my $end = sub {
- if (my $err = $self->finish) {
+ my $err = $self->finish;
+ if ($err && !$env->{'qspawn.quiet'}) {
$err = join(' ', @{$self->{args}->[0]}).": $err\n";
$env->{'psgi.errors'}->print($err);
}
@@ -84,6 +99,7 @@ sub psgi_return {
my $cb = sub {
my $r = $rd_hdr->() or return;
$rd_hdr = undef;
+ my $filter = delete $env->{'qspawn.filter'};
if (scalar(@$r) == 3) { # error
if ($async) {
$async->close; # calls rpipe->close
@@ -94,11 +110,12 @@ sub psgi_return {
$res->($r);
} elsif ($async) {
$fh = $res->($r); # scalar @$r == 2
+ $fh = filter_fh($fh, $filter) if $filter;
$async->async_pass($env->{'psgix.io'}, $fh, \$buf);
} else { # for synchronous PSGI servers
require PublicInbox::GetlineBody;
$r->[2] = PublicInbox::GetlineBody->new($rpipe, $end,
- $buf);
+ $buf, $filter);
$res->($r);
}
};
--
EW
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-01-27 4:03 7% [PATCH 00/14] convert solver to use pi-httpd.async Eric Wong
2019-01-27 4:03 6% ` [PATCH 03/14] qspawn|getlinebody: support streaming filters Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).