user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: how's memory usage on public-inbox-httpd?
Date: Thu, 6 Jun 2019 22:29:38 +0000
Message-ID: <20190606222938.y7nt3uankntkktly@dcvr> (raw)
In-Reply-To: <20190606221904.GB4087@chatter.i7.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Jun 06, 2019 at 10:10:09PM +0000, Eric Wong wrote:
> > > > All those endpoints should detect backpressure from a slow
> > > > client (varnish/nginx in your case) using the ->getline method.
> > > 
> > > Wouldn't that spike up and down? The size I'm seeing stays pretty constant
> > > without any significant changes across requests.
> > 
> > Nope.  That's the thing with glibc malloc not wanting to trim
> > the heap for good benchmarks.
> > 
> > You could also try starting with MALLOC_MMAP_THRESHOLD_=131072
> > in env (or some smaller/larger number in bytes) to force it to
> > use mmap in more cases instead of sbrk.
> 
> I've restarted the process and I'm running mmap -x $PID | tail -1 on it once
> a minute. I'll try to collect this data for a while and see if I can notice
> significant increases and correlate that with access logs.  From the first
> few minutes of running I see:
> 
> Thu Jun  6 22:06:03 UTC 2019
> total kB          298160  102744   96836
> Thu Jun  6 22:07:03 UTC 2019
> total kB          355884  154968  147664
> Thu Jun  6 22:08:03 UTC 2019
> total kB          355884  154980  147664
> Thu Jun  6 22:09:03 UTC 2019
> total kB          359976  156788  148336
> Thu Jun  6 22:10:03 UTC 2019
> total kB          359976  156788  148336
> Thu Jun  6 22:11:03 UTC 2019
> total kB          359976  156788  148336
> Thu Jun  6 22:12:03 UTC 2019
> total kB          365464  166612  158160
> Thu Jun  6 22:13:03 UTC 2019
> total kB          366884  167908  159456
> Thu Jun  6 22:14:03 UTC 2019
> total kB          366884  167908  159456
> Thu Jun  6 22:15:03 UTC 2019
> total kB          366884  167908  159456

Would also be good to correlate that to open sockets, too.
(168M is probably normal for 64-bit, I'm still on 32-bit and its <100M).
I'm not happy with that memory use, even; but it's better than
gigabytes.

> > Without concurrent connections; I can't see that happening
> > unless there's a single message which is gigabytes in size.  I'm
> > already irked that Email::MIME requires slurping entire emails
> > into memory; but it should not be using more than one
> > Email::MIME object in memory-at-a-time for a single client.
> > 
> > Anything from varnish/nginx logs can't keep up for some reason?
> 
> Speaking of logs, I did notice that even though we're passing -1
> /var/log/public-inbox/httpd.out.log, that file stays empty. There's
> nttpd.out.log there, which is non-empty, so that's curious:
> 
> # ls -ahl
> total 2.6M
> drwx------.  2 publicinbox publicinbox  177 Jun  6 22:05 .
> drwxr-xr-x. 21 root        root        4.0K Jun  2 03:12 ..
> -rw-r--r--.  1 publicinbox publicinbox    0 Jun  6 22:05 httpd.out.log
> -rw-r--r--.  1 publicinbox publicinbox 422K Jun  6 22:04 nntpd.out.log
> -rw-r--r--.  1 publicinbox publicinbox 771K May 12 01:02 nntpd.out.log-20190512.gz
> -rw-r--r--.  1 publicinbox publicinbox 271K May 19 03:45 nntpd.out.log-20190519.gz
> -rw-r--r--.  1 publicinbox publicinbox  86K May 25 22:23 nntpd.out.log-20190526.gz
> -rw-r--r--.  1 publicinbox publicinbox 1.1M Jun  2 00:52 nntpd.out.log-20190602
> 
> Could it be that stdout is not being written out and is just perpetually
> buffered? That could explain the ever-growing size.

There's no HTTP access logging by default.
AccessLog::Timed is commented out in examples/public-inbox.psgi;
and the example uses syswrite, even.

Also, PublicInbox::Daemon definitely enables autoflush on STDOUT.

  reply index

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-12-01 19:44 Eric Wong
2019-06-06 19:04 ` Konstantin Ryabitsev
2019-06-06 20:37   ` Eric Wong
2019-06-06 21:45     ` Konstantin Ryabitsev
2019-06-06 22:10       ` Eric Wong
2019-06-06 22:19         ` Konstantin Ryabitsev
2019-06-06 22:29           ` Eric Wong [this message]
2019-06-10 10:09             ` [RFC] optionally support glibc malloc_info via SIGCONT Eric Wong
2019-06-09  8:39         ` how's memory usage on public-inbox-httpd? Eric Wong
2019-06-12 17:08           ` Eric Wong
2019-06-06 20:54   ` Eric Wong
2019-10-16 22:10   ` Eric Wong
2019-10-18 19:23     ` Konstantin Ryabitsev
2019-10-19  0:11       ` Eric Wong
2019-10-22 17:28         ` Konstantin Ryabitsev
2019-10-22 19:11           ` Eric Wong
2019-10-28 23:24         ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190606222938.y7nt3uankntkktly@dcvr \
    --to=e@80x24.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git