* [PATCH 2/2] githttpbackend: clamp to one smart HTTP request at-a-time
2016-04-28 1:56 5% [PATCH 0/2] githttpbackend: dumb HTTP fallbacks Eric Wong
@ 2016-04-28 1:56 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2016-04-28 1:56 UTC (permalink / raw)
To: meta
Server admins may not be able to afford to have too many
git-pack-objects processes running at once. Since PSGI
HTTP servers should already be configured to use multiple
processes for other requests; limit concurrency of smart
backends to one; and fall back to dumb responses if we're
already generating a pack.
---
lib/PublicInbox/GitHTTPBackend.pm | 12 ++++++++++++
t/httpd.t | 2 +-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/GitHTTPBackend.pm b/lib/PublicInbox/GitHTTPBackend.pm
index c44c67d..a7cac10 100644
--- a/lib/PublicInbox/GitHTTPBackend.pm
+++ b/lib/PublicInbox/GitHTTPBackend.pm
@@ -10,6 +10,14 @@ use Fcntl qw(:seek);
use IO::File;
use PublicInbox::Spawn qw(spawn);
+# TODO: make configurable, but keep in mind it's better to have
+# multiple -httpd worker processes which are already scaled to
+# the proper number of CPUs and memory. git-pack-objects(1) may
+# also use threads and bust memory limits, too, so I recommend
+# limiting threads to 1 (via `pack.threads` knob in git) for serving.
+my $LIMIT = 1;
+my $nr_running = 0;
+
# n.b. serving "description" and "cloneurl" should be innocuous enough to
# not cause problems. serving "config" might...
my @text = qw[HEAD info/refs
@@ -31,6 +39,8 @@ sub r {
sub serve {
my ($cgi, $git, $path) = @_;
+ return serve_dumb($cgi, $git, $path) if $nr_running >= $LIMIT;
+
my $service = $cgi->param('service') || '';
if ($service =~ /\Agit-\w+-pack\z/ || $path =~ /\Agit-\w+-pack\z/) {
my $ok = serve_smart($cgi, $git, $path);
@@ -174,6 +184,7 @@ sub serve_smart {
$wpipe = $in = undef;
$buf = '';
my ($vin, $fh, $res);
+ $nr_running++;
my $end = sub {
if ($fh) {
$fh->close;
@@ -182,6 +193,7 @@ sub serve_smart {
if ($rpipe) {
$rpipe->close; # _may_ be Danga::Socket::close
$rpipe = undef;
+ $nr_running--;
}
if (defined $pid && $pid != waitpid($pid, 0)) {
$err->print("git http-backend ($git_dir): $?\n");
diff --git a/t/httpd.t b/t/httpd.t
index 0379031..781fe03 100644
--- a/t/httpd.t
+++ b/t/httpd.t
@@ -104,7 +104,7 @@ EOF
is(system(qw(git clone -q --mirror),
"http://$host:$port/$group", "$tmpdir/clone.git"),
- 0, 'clone successful');
+ 0, 'smart clone successful');
# ensure dumb cloning works, too:
is(system('git', "--git-dir=$maindir",
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/2] githttpbackend: dumb HTTP fallbacks
@ 2016-04-28 1:56 5% Eric Wong
2016-04-28 1:56 7% ` [PATCH 2/2] githttpbackend: clamp to one smart HTTP request at-a-time Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2016-04-28 1:56 UTC (permalink / raw)
To: meta
Smart HTTP cloning is nice for bandwidth utilization, but it
can eat up memory and CPU on a busy server; so fall back to
dumb HTTP if there is already a running backend or if an admin
decides to set `http.uploadpack' to `false' entirely.
Not that our large static file serving is great when it comes
to slow clients, at the moment; but of course I shall
improve on that...
Eric Wong (2):
githttpbackend: fall back to dumb if smart HTTP is off
githttpbackend: clamp to one smart HTTP request at-a-time
lib/PublicInbox/GitHTTPBackend.pm | 52 ++++++++++++++++++++++++++++-----------
t/httpd.t | 9 +++++++
2 files changed, 46 insertions(+), 15 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2016-04-28 1:56 5% [PATCH 0/2] githttpbackend: dumb HTTP fallbacks Eric Wong
2016-04-28 1:56 7% ` [PATCH 2/2] githttpbackend: clamp to one smart HTTP request at-a-time Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).