user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7
Date: Thu, 26 Sep 2019 21:16:20 +0000
Message-ID: <20190926211620.GB11146@dcvr> (raw)
In-Reply-To: <20190926123606.GA10467@chatter.i7.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 26, 2019 at 01:50:36AM +0000, Eric Wong wrote:
> > After many hours of reviewing our code in PublicInbox::Qspawn,
> > PublicInbox::GitHTTPBackend, and PublicInbox::HTTP and finding
> > nothing but cleanups and documentation improvements; it seems
> > the leaks affecting lore is down to bugs in Perl 5.16.3.
> > 
> > After removing the warning for Deflater being missing in
> > d883d4a93b23be134038e28f421eafca70c3d838
> > ("httpd: get rid of Deflater warning"), I missed that my
> > own CentOS 7 VM was missing that module so was unable to
> > reproduce the FD leaks :x
> > 
> > The first patch is a straightforward workaround that I was
> > able to test without Plack::Middleware::Deflater being installed.
> > 
> > The second patch stops loading Deflater in httpd on Perls
> > earlier than 5.18.  (But I haven't built or tested 5.18 myself).
> > Enabling gzip in varnish will be needed for 5.16 users.
> 
> Yay, I can confirm that the latest master fixes all the FD leaks that have
> been plaguing lore.kernel.org for the past few weeks. The number of pipes is
> stable and there are no (deleted) tempfiles showing up. Thanks so much for
> this!

Good to know!  Also, how's memory use?

It ought to be stable if Perl is doing the right thing; but I
haven't tested all the endpoints thoroughly on 5.16.3
(especially the mbox.gz ones)

> Can you elaborate on gzip+varnish changes? I'm assuming it's something as
> simple as:
> 
> @@ -63,6 +63,10 @@
>        } else {
>                /* short TTL for up-to-dateness, our PSGI is not that slow */
>                set beresp.ttl = 10s;
> +               /* Compress text responses on CentOS7 */
> +               if (beresp.http.content-type ~ "text") {
> +                       set beresp.do_gzip = true;
> +               }

*shrug*  Whatever the varnish docs says :)

I still prefer to do gzip in the Perl process to minimize IPC
overhead (and to eventually make things easier-to-deploy w/o
needing Varnish).

I think I can add gzip support to WwwStream and WwwAtomStream
pretty easily, and we're already using a custom gzipper for
the mboxrd endpoints.

  reply index

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-26  1:50 Eric Wong
2019-09-26  1:50 ` [PATCH 1/2] ds: workaround a memory leak in Perl 5.16.x Eric Wong
2019-09-26  1:50 ` [PATCH 2/2] httpd: disable Deflater middleware by default on Perl <5.18 Eric Wong
2019-09-26 12:36 ` [PATCH 0/2] leak workarounds for Perl 5.16 on CentOS/RHEL 7 Konstantin Ryabitsev
2019-09-26 21:16   ` Eric Wong [this message]
2019-09-27 21:01 ` [PATCH 3/2] qspawn: workaround Perl 5.16.3 leak, re-enable Deflater Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190926211620.GB11146@dcvr \
    --to=e@80x24.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git