From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.4 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id CCEE31F462 for ; Thu, 6 Jun 2019 19:05:00 +0000 (UTC) Received: by mail-it1-x141.google.com with SMTP id r135so1707706ith.1 for ; Thu, 06 Jun 2019 12:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=E1pg/lDVQ4az9FEF8Mie639lTpd4HqSuMRQ7rTFqZII=; b=S+Gg2zg1ZmYMgjp+ECdYbExiEpPa0/j8qMKVsG09YMcfu7x6MzsWQ4f3zatyL4h2u9 P8HFFg7p9yaqL0Xs3FlJ8lXNHg48Ux5HrYtGsaQxHLQehW6Ak/YGoGQqfvvkZ8dEnZ0Y v3/JKM9JBPC8f1uSaOdgFi5k+xmdlmkKKrYu0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=E1pg/lDVQ4az9FEF8Mie639lTpd4HqSuMRQ7rTFqZII=; b=WQx1S96LOzEw/V3nAhY7dQAa1s6qELNcCKeVuofvIOXhgYcZ5KZgg33LNAA7XLSmq6 e/7coNyMkhyIRcjld0RQZfHZc6Ji3ZX+kAx7NT4OHl/Jkz9N6f0sUKtZZd3NCjvMtLTo 6x2dx/2iBjzoBUelFvUy9wEM/WwCOM8+m6n9gQ9D+3iaLtQMELz6Wk1Q2XUOFGVz328G hjAAKhPvyDMDaTlouzxjQsw7m1lgdlvTFfZ5sGOfNfy9KPR0AtoeD7U7/aVh518ODZlD UzyXUX6FGwMz9A2S1mTsIEnJAX9Q5RJlwW3x6quQ/3HHQipVXGVMqj2cFH1Tl7P0F0ya bW8w== X-Gm-Message-State: APjAAAVr+897IOGxaT9nWKkXkwbFO8rB5gAY2E2pDxpIYbhVwpJ1ccwc /08p5QWDEECL1NPgKSA6a0NfGA== X-Google-Smtp-Source: APXvYqyPlS0j3B3g9HN89Bh9lKpBoPDIe7f8H2nMLie6GBqE3oNBZeXc2mBSBZzSdOQSRcOm+OqAVw== X-Received: by 2002:a02:84e6:: with SMTP id f93mr33179818jai.73.1559847898238; Thu, 06 Jun 2019 12:04:58 -0700 (PDT) Received: from chatter.i7.local (192-0-228-88.cpe.teksavvy.com. [192.0.228.88]) by smtp.gmail.com with ESMTPSA id f4sm862470iok.56.2019.06.06.12.04.56 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 06 Jun 2019 12:04:57 -0700 (PDT) Date: Thu, 6 Jun 2019 15:04:55 -0400 From: Konstantin Ryabitsev To: Eric Wong Cc: meta@public-inbox.org Subject: Re: how's memory usage on public-inbox-httpd? Message-ID: <20190606190455.GA17362@chatter.i7.local> References: <20181201194429.d5aldesjkb56il5c@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline In-Reply-To: <20181201194429.d5aldesjkb56il5c@dcvr> User-Agent: Mutt/1.11.4 (2019-03-13) List-Id: Hello: This is an old-ish discussion, but we finally had a chance to run the httpd daemon for a long time without restarting it to add more lists, and the memory usage on it is actually surprising: $ ps -eF | grep public-inbox publici+ 17741 1 0 52667 24836 8 May24 ? 00:00:00 /usr/bin/perl -w /usr/local/bin/public-inbox-nntpd -1 /var/log/public-inbox/nntpd.out.log publici+ 17744 17741 0 69739 90288 9 May24 ? 00:38:43 /usr/bin/perl -w /usr/local/bin/public-inbox-nntpd -1 /var/log/public-inbox/nntpd.out.log publici+ 18273 1 0 52599 23832 9 May24 ? 00:00:00 /usr/bin/perl -w /usr/local/bin/public-inbox-httpd -1 /var/log/public-inbox/httpd.out.log publici+ 18275 18273 4 5016115 19713872 10 May24 ? 13:59:13 /usr/bin/perl -w /usr/local/bin/public-inbox-httpd -1 /var/log/public-inbox/httpd.out.log You'll notice that process 18275 has been running since May 24 and takes up 19GB in RSS. This is a 16-core 64-GB system, so it's not necessarily super alarming, but seems large. :) Is that normal, and if not, what can I do to help troubleshoot where it's all going? -K On Sat, Dec 01, 2018 at 07:44:29PM +0000, Eric Wong wrote: >I haven't been around much, so not working on public-inbox means >fewer restarts :x > >On my 2-core VM, I've been noticing public-inbox-httpd memory >spikes into the 500MB range, which is gross... It seems caused >by slow clients and large threads/mbox downloads. The PSGI code >only loads one email per-client in memory at-a-time when using >-httpd; but that adds up with many clients and larger messages. > >I run two -httpd workers, one-per-core, but also varnish and an >experimental Ruby/C reverse-buffering proxy (yahns) for HTTPS. > >The problem seems to be varnish isn't reading from -httpd fast >enough (and I lack CPU cores), but decreasing the niceness of >varnish seems to help with the problem...