user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: how's memory usage on public-inbox-httpd?
Date: Thu, 6 Jun 2019 18:19:04 -0400
Message-ID: <20190606221904.GB4087@chatter.i7.local> (raw)
In-Reply-To: <20190606221009.y4fe2e2rervvq3z4@dcvr>

On Thu, Jun 06, 2019 at 10:10:09PM +0000, Eric Wong wrote:
>> > All those endpoints should detect backpressure from a slow
>> > client (varnish/nginx in your case) using the ->getline method.
>>
>> Wouldn't that spike up and down? The size I'm seeing stays pretty constant
>> without any significant changes across requests.
>
>Nope.  That's the thing with glibc malloc not wanting to trim
>the heap for good benchmarks.
>
>You could also try starting with MALLOC_MMAP_THRESHOLD_=131072
>in env (or some smaller/larger number in bytes) to force it to
>use mmap in more cases instead of sbrk.

I've restarted the process and I'm running mmap -x $PID | tail -1 on it 
once a minute. I'll try to collect this data for a while and see if I 
can notice significant increases and correlate that with access logs.  
From the first few minutes of running I see:

Thu Jun  6 22:06:03 UTC 2019
total kB          298160  102744   96836
Thu Jun  6 22:07:03 UTC 2019
total kB          355884  154968  147664
Thu Jun  6 22:08:03 UTC 2019
total kB          355884  154980  147664
Thu Jun  6 22:09:03 UTC 2019
total kB          359976  156788  148336
Thu Jun  6 22:10:03 UTC 2019
total kB          359976  156788  148336
Thu Jun  6 22:11:03 UTC 2019
total kB          359976  156788  148336
Thu Jun  6 22:12:03 UTC 2019
total kB          365464  166612  158160
Thu Jun  6 22:13:03 UTC 2019
total kB          366884  167908  159456
Thu Jun  6 22:14:03 UTC 2019
total kB          366884  167908  159456
Thu Jun  6 22:15:03 UTC 2019
total kB          366884  167908  159456

>Without concurrent connections; I can't see that happening
>unless there's a single message which is gigabytes in size.  I'm
>already irked that Email::MIME requires slurping entire emails
>into memory; but it should not be using more than one
>Email::MIME object in memory-at-a-time for a single client.
>
>Anything from varnish/nginx logs can't keep up for some reason?

Speaking of logs, I did notice that even though we're passing -1 
/var/log/public-inbox/httpd.out.log, that file stays empty. There's 
nttpd.out.log there, which is non-empty, so that's curious:

# ls -ahl
total 2.6M
drwx------.  2 publicinbox publicinbox  177 Jun  6 22:05 .
drwxr-xr-x. 21 root        root        4.0K Jun  2 03:12 ..
-rw-r--r--.  1 publicinbox publicinbox    0 Jun  6 22:05 httpd.out.log
-rw-r--r--.  1 publicinbox publicinbox 422K Jun  6 22:04 nntpd.out.log
-rw-r--r--.  1 publicinbox publicinbox 771K May 12 01:02 nntpd.out.log-20190512.gz
-rw-r--r--.  1 publicinbox publicinbox 271K May 19 03:45 nntpd.out.log-20190519.gz
-rw-r--r--.  1 publicinbox publicinbox  86K May 25 22:23 nntpd.out.log-20190526.gz
-rw-r--r--.  1 publicinbox publicinbox 1.1M Jun  2 00:52 nntpd.out.log-20190602

Could it be that stdout is not being written out and is just perpetually 
buffered? That could explain the ever-growing size.

-K

  reply index

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-12-01 19:44 Eric Wong
2019-06-06 19:04 ` Konstantin Ryabitsev
2019-06-06 20:37   ` Eric Wong
2019-06-06 21:45     ` Konstantin Ryabitsev
2019-06-06 22:10       ` Eric Wong
2019-06-06 22:19         ` Konstantin Ryabitsev [this message]
2019-06-06 22:29           ` Eric Wong
2019-06-10 10:09             ` [RFC] optionally support glibc malloc_info via SIGCONT Eric Wong
2019-06-09  8:39         ` how's memory usage on public-inbox-httpd? Eric Wong
2019-06-12 17:08           ` Eric Wong
2019-06-06 20:54   ` Eric Wong
2019-10-16 22:10   ` Eric Wong
2019-10-18 19:23     ` Konstantin Ryabitsev
2019-10-19  0:11       ` Eric Wong
2019-10-22 17:28         ` Konstantin Ryabitsev
2019-10-22 19:11           ` Eric Wong
2019-10-28 23:24         ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190606221904.GB4087@chatter.i7.local \
    --to=konstantin@linuxfoundation.org \
    --cc=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git