user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater
  2019-09-26  1:50  6% [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Eric Wong
  2019-09-26  1:50  7% ` [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x Eric Wong
@ 2019-09-27 21:01  5% ` Eric Wong
  1 sibling, 0 replies; 3+ results
From: Eric Wong @ 2019-09-27 21:01 UTC (permalink / raw)
  To: meta

The httpd-supplied write callback is the leak culprit under Perl
5.16.3.  undef-ing it immediately after use keeps a repeated
"git fetch" loop from monotonically increasing memory and FD use
on the Perl shipped with RHEL/CentOS 7.x.

Other endpoints tested showed no increase in memory use under
constant load with "ab -HAccept-Encoding:gzip -k", including the
async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via
SolverGit module.
---
 Note: I initially tried this change, but thought it only slowed
 down the leaking because I had not yet discovered the
 workaround in commit cd71a869c7e9c811
 ("ds: workaround a memory leak in Perl 5.16.x").

 Now that both leaks are worked around, memory usage is completely
 flat when repeating a single request of any type with gzip-enabled.

 lib/PublicInbox/Qspawn.pm | 4 ++++
 script/public-inbox-httpd | 8 +-------
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 5a30064..cb3dc51 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -281,6 +281,10 @@ sub psgi_return {
 								$buf, $filter);
 			$wcb->($r);
 		}
+
+		# Workaround a leak under Perl 5.16.3 when combined with
+		# Plack::Middleware::Deflater:
+		$wcb = undef;
 	};
 	$limiter ||= $def_limiter ||= PublicInbox::Qspawn::Limiter->new(32);
 	my $start_cb = sub { # may run later, much later...
diff --git a/script/public-inbox-httpd b/script/public-inbox-httpd
index 9b869f9..b2464f4 100755
--- a/script/public-inbox-httpd
+++ b/script/public-inbox-httpd
@@ -24,13 +24,7 @@ my $refresh = sub {
 		my $www = PublicInbox::WWW->new;
 		$www->preload;
 		$app = builder {
-			# Perl 5.16.3 leaks in our "push" response code path
-			# (e.g. Qspawn) due to something in
-			# Plack::Util::response_cb, regardless of whether the
-			# client is sending Accept-Encoding:gzip requests.
-			# perl5180delta documents many leak fixes, so assume
-			# 5.18+ is safe for now and bump the check as-need:
-			$] >= 5.018000 and eval {
+			eval {
 				enable 'Deflater',
 					content_type => [ qw(
 						text/html
-- 
EW


^ permalink raw reply	[relevance 5%]

* [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x
  2019-09-26  1:50  6% [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Eric Wong
@ 2019-09-26  1:50  7% ` Eric Wong
  2019-09-27 21:01  5% ` [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater Eric Wong
  1 sibling, 0 replies; 3+ results
From: Eric Wong @ 2019-09-26  1:50 UTC (permalink / raw)
  To: meta

The perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7 is
affected by a memory leak in Perl when calling `ref' on
blessed references.  This resulted in a very slow leak that
manifests more quickly with a nonstop "git fetch" loop.

Use Scalar::Util::blessed to work around the issue.
Tested overnight on a CentOS 7 VM.

cf. https://rt.perl.org/Public/Bug/Display.html?id=114340
---
 lib/PublicInbox/DS.pm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index 30a9641..7f7cb85 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -24,6 +24,7 @@ use parent qw(Exporter);
 our @EXPORT_OK = qw(now msg_more);
 use warnings;
 use 5.010_001;
+use Scalar::Util qw(blessed);
 
 use PublicInbox::Syscall qw(:epoll);
 use PublicInbox::Tmpfile;
@@ -178,10 +179,12 @@ sub next_tick () {
     my $q = $nextq;
     $nextq = [];
     for (@$q) {
-        if (ref($_) eq 'CODE') {
-            $_->();
-        } else {
+        # we avoid "ref" on blessed refs to workaround a Perl 5.16.3 leak:
+        # https://rt.perl.org/Public/Bug/Display.html?id=114340
+        if (blessed($_)) {
             $_->event_step;
+        } else {
+            $_->();
         }
     }
 }
-- 
EW


^ permalink raw reply	[relevance 7%]

* [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7
@ 2019-09-26  1:50  6% Eric Wong
  2019-09-26  1:50  7% ` [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x Eric Wong
  2019-09-27 21:01  5% ` [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater Eric Wong
  0 siblings, 2 replies; 3+ results
From: Eric Wong @ 2019-09-26  1:50 UTC (permalink / raw)
  To: meta

After many hours of reviewing our code in PublicInbox::Qspawn,
PublicInbox::GitHTTPBackend, and PublicInbox::HTTP and finding
nothing but cleanups and documentation improvements; it seems
the leaks affecting lore is down to bugs in Perl 5.16.3.

After removing the warning for Deflater being missing in
d883d4a93b23be134038e28f421eafca70c3d838
("httpd: get rid of Deflater warning"), I missed that my
own CentOS 7 VM was missing that module so was unable to
reproduce the FD leaks :x

The first patch is a straightforward workaround that I was
able to test without Plack::Middleware::Deflater being installed.

The second patch stops loading Deflater in httpd on Perls
earlier than 5.18.  (But I haven't built or tested 5.18 myself).
Enabling gzip in varnish will be needed for 5.16 users.

Independently of this fix, I've long been considering replacing
Deflater with a buffer-to-gzip layer which would reduce memory
pressure from intermediate uncompressed strings.  This might
serve as impetus to move that idea along (and of course I'd
test it heavily under CentOS 7 :>).

Anybody using custom .psgi files on Perl 5.16.x will need to
make adjustments to disable the Deflater in that.

Eric Wong (2):
  ds: workaround a memory leak in Perl 5.16.x
  httpd: disable Deflater middleware by default on Perl <5.18

 lib/PublicInbox/DS.pm     | 9 ++++++---
 script/public-inbox-httpd | 8 +++++++-
 2 files changed, 13 insertions(+), 4 deletions(-)

-- 
EW


^ permalink raw reply	[relevance 6%]

Results 1-3 of 3 | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-09-26  1:50  6% [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Eric Wong
2019-09-26  1:50  7% ` [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x Eric Wong
2019-09-27 21:01  5% ` [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater Eric Wong

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).