user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Konstantin Ryabitsev <>
To: Eric Wong <>
Subject: Re: how's memory usage on public-inbox-httpd?
Date: Thu, 6 Jun 2019 15:04:55 -0400	[thread overview]
Message-ID: <20190606190455.GA17362@chatter.i7.local> (raw)
In-Reply-To: <20181201194429.d5aldesjkb56il5c@dcvr>


This is an old-ish discussion, but we finally had a chance to run the 
httpd daemon for a long time without restarting it to add more lists, 
and the memory usage on it is actually surprising:

$ ps -eF | grep public-inbox
publici+ 17741     1  0 52667 24836   8 May24 ?        00:00:00 /usr/bin/perl -w /usr/local/bin/public-inbox-nntpd -1 /var/log/public-inbox/nntpd.out.log
publici+ 17744 17741  0 69739 90288   9 May24 ?        00:38:43 /usr/bin/perl -w /usr/local/bin/public-inbox-nntpd -1 /var/log/public-inbox/nntpd.out.log
publici+ 18273     1  0 52599 23832   9 May24 ?        00:00:00 /usr/bin/perl -w /usr/local/bin/public-inbox-httpd -1 /var/log/public-inbox/httpd.out.log
publici+ 18275 18273  4 5016115 19713872 10 May24 ?    13:59:13 /usr/bin/perl -w /usr/local/bin/public-inbox-httpd -1 /var/log/public-inbox/httpd.out.log

You'll notice that process 18275 has been running since May 24 and takes 
up 19GB in RSS. This is a 16-core 64-GB system, so it's not necessarily 
super alarming, but seems large. :)

Is that normal, and if not, what can I do to help troubleshoot where 
it's all going?


On Sat, Dec 01, 2018 at 07:44:29PM +0000, Eric Wong wrote:
>I haven't been around much, so not working on public-inbox means
>fewer restarts :x
>On my 2-core VM, I've been noticing public-inbox-httpd memory
>spikes into the 500MB range, which is gross...  It seems caused
>by slow clients and large threads/mbox downloads.  The PSGI code
>only loads one email per-client in memory at-a-time when using
>-httpd; but that adds up with many clients and larger messages.
>I run two -httpd workers, one-per-core, but also varnish and an
>experimental Ruby/C reverse-buffering proxy (yahns) for HTTPS.
>The problem seems to be varnish isn't reading from -httpd fast
>enough (and I lack CPU cores), but decreasing the niceness of
>varnish seems to help with the problem...

  reply	other threads:[~2019-06-06 19:05 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-12-01 19:44 Eric Wong
2019-06-06 19:04 ` Konstantin Ryabitsev [this message]
2019-06-06 20:37   ` Eric Wong
2019-06-06 21:45     ` Konstantin Ryabitsev
2019-06-06 22:10       ` Eric Wong
2019-06-06 22:19         ` Konstantin Ryabitsev
2019-06-06 22:29           ` Eric Wong
2019-06-10 10:09             ` [RFC] optionally support glibc malloc_info via SIGCONT Eric Wong
2019-06-09  8:39         ` how's memory usage on public-inbox-httpd? Eric Wong
2019-06-12 17:08           ` Eric Wong
2019-06-06 20:54   ` Eric Wong
2019-10-16 22:10   ` Eric Wong
2019-10-18 19:23     ` Konstantin Ryabitsev
2019-10-19  0:11       ` Eric Wong
2019-10-22 17:28         ` Konstantin Ryabitsev
2019-10-22 19:11           ` Eric Wong
2019-10-28 23:24         ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190606190455.GA17362@chatter.i7.local \ \ \ \
    --subject='Re: how'\''s memory usage on public-inbox-httpd?' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ \
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
 note: .onion URLs require Tor:

code repositories for project(s) associated with this inbox:

AGPL code for this site: git clone