From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id CB3C21F463; Fri, 27 Sep 2019 21:01:01 +0000 (UTC) Date: Fri, 27 Sep 2019 21:01:01 +0000 From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater Message-ID: <20190927210101.q22ilbp64x6uba7i@dcvr> References: <20190926015038.29770-1-e@80x24.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20190926015038.29770-1-e@80x24.org> List-Id: The httpd-supplied write callback is the leak culprit under Perl 5.16.3. undef-ing it immediately after use keeps a repeated "git fetch" loop from monotonically increasing memory and FD use on the Perl shipped with RHEL/CentOS 7.x. Other endpoints tested showed no increase in memory use under constant load with "ab -HAccept-Encoding:gzip -k", including the async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via SolverGit module. --- Note: I initially tried this change, but thought it only slowed down the leaking because I had not yet discovered the workaround in commit cd71a869c7e9c811 ("ds: workaround a memory leak in Perl 5.16.x"). Now that both leaks are worked around, memory usage is completely flat when repeating a single request of any type with gzip-enabled. lib/PublicInbox/Qspawn.pm | 4 ++++ script/public-inbox-httpd | 8 +------- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm index 5a30064..cb3dc51 100644 --- a/lib/PublicInbox/Qspawn.pm +++ b/lib/PublicInbox/Qspawn.pm @@ -281,6 +281,10 @@ sub psgi_return { $buf, $filter); $wcb->($r); } + + # Workaround a leak under Perl 5.16.3 when combined with + # Plack::Middleware::Deflater: + $wcb = undef; }; $limiter ||= $def_limiter ||= PublicInbox::Qspawn::Limiter->new(32); my $start_cb = sub { # may run later, much later... diff --git a/script/public-inbox-httpd b/script/public-inbox-httpd index 9b869f9..b2464f4 100755 --- a/script/public-inbox-httpd +++ b/script/public-inbox-httpd @@ -24,13 +24,7 @@ my $refresh = sub { my $www = PublicInbox::WWW->new; $www->preload; $app = builder { - # Perl 5.16.3 leaks in our "push" response code path - # (e.g. Qspawn) due to something in - # Plack::Util::response_cb, regardless of whether the - # client is sending Accept-Encoding:gzip requests. - # perl5180delta documents many leak fixes, so assume - # 5.18+ is safe for now and bump the check as-need: - $] >= 5.018000 and eval { + eval { enable 'Deflater', content_type => [ qw( text/html -- EW