From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-3.9 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id 44548211B3;
	Sat,  1 Dec 2018 19:44:29 +0000 (UTC)
Date: Sat, 1 Dec 2018 19:44:29 +0000
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Subject: how's memory usage on public-inbox-httpd?
Message-ID: <20181201194429.d5aldesjkb56il5c@dcvr>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
List-Id: <meta.public-inbox.org>

I haven't been around much, so not working on public-inbox means
fewer restarts :x

On my 2-core VM, I've been noticing public-inbox-httpd memory
spikes into the 500MB range, which is gross...  It seems caused
by slow clients and large threads/mbox downloads.  The PSGI code
only loads one email per-client in memory at-a-time when using
-httpd; but that adds up with many clients and larger messages.

I run two -httpd workers, one-per-core, but also varnish and an
experimental Ruby/C reverse-buffering proxy (yahns) for HTTPS.

The problem seems to be varnish isn't reading from -httpd fast
enough (and I lack CPU cores), but decreasing the niceness of
varnish seems to help with the problem...