From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 423BB1F462 for ; Thu, 6 Jun 2019 22:19:08 +0000 (UTC) Received: by mail-io1-xd2c.google.com with SMTP id n5so1472535ioc.7 for ; Thu, 06 Jun 2019 15:19:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=MLOvfXu+v8Jz+qM9TfE4PiAXQT9zXog72L0yodyIi+o=; b=R3mqL5XOe0oAFVdoxHLoJGXm767cu8bB9F4J7k+zDRLnw0MVxkXzwNXoCgdSiS8a0s Yod5jW8xK5qqYebPwh86QtghH1zS3HY3Kx3Pyh2dsEoPVjK51TX3cxmykcB/7Os4NpE7 5orS6MluQBEkyRI7SoTuK7zVHTJp5FyCgG0z0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=MLOvfXu+v8Jz+qM9TfE4PiAXQT9zXog72L0yodyIi+o=; b=o6Ssz5KwMYW7umVpFu332KhOY6bmDxWwBsAGFIW4Iq01SIYve9Wa8bGuhTRzuQ0YcV Hh7BU+BY7P6uKgZV3xgAtSiCUo0+7CiRmN+0xPkQQQb39K5Y8+rznqzTHt5Duws+XqjK wtFO2xHfEqqKe+9DVWs+mwHQnGMpOcUMCLXRwAu225aYO8n7D0W6//zHyercnk4Cmx8f BehcxQWoAv49hFPS6p5rjFGoiGODfVbI1x1ROVUnfm+htCJsE89dZJ1yRsKQcNXUqAZj FHHLO2WA9BaXnSLwGjkEJGEnWOl4/69wirABtk7MM6dDfBlm9BlBjarrPSeGx3vFm+8s 94gw== X-Gm-Message-State: APjAAAV8fm3gOHcWVWfuuaHdeHcqD7AVZCDX5S8V741nsr6dbBBcPfvQ /OFENTXKnw8SKbkGR+DLC6QBkb6Efl6vTg== X-Google-Smtp-Source: APXvYqxi3MgHOK3ePKyjtHGnUtdQMp2FJ3O7McbmGVDOCfS5EzQuq/pArRoI1wIA3NGdtlYBXaOK/w== X-Received: by 2002:a5d:9a11:: with SMTP id s17mr33753087iol.267.1559859547170; Thu, 06 Jun 2019 15:19:07 -0700 (PDT) Received: from chatter.i7.local (192-0-228-88.cpe.teksavvy.com. [192.0.228.88]) by smtp.gmail.com with ESMTPSA id r139sm58404iod.61.2019.06.06.15.19.05 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 06 Jun 2019 15:19:06 -0700 (PDT) Date: Thu, 6 Jun 2019 18:19:04 -0400 From: Konstantin Ryabitsev To: Eric Wong Cc: meta@public-inbox.org Subject: Re: how's memory usage on public-inbox-httpd? Message-ID: <20190606221904.GB4087@chatter.i7.local> References: <20181201194429.d5aldesjkb56il5c@dcvr> <20190606190455.GA17362@chatter.i7.local> <20190606203752.7wpdla5ynemjlshs@dcvr> <20190606214509.GA4087@chatter.i7.local> <20190606221009.y4fe2e2rervvq3z4@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline In-Reply-To: <20190606221009.y4fe2e2rervvq3z4@dcvr> User-Agent: Mutt/1.11.4 (2019-03-13) List-Id: On Thu, Jun 06, 2019 at 10:10:09PM +0000, Eric Wong wrote: >> > All those endpoints should detect backpressure from a slow >> > client (varnish/nginx in your case) using the ->getline method. >> >> Wouldn't that spike up and down? The size I'm seeing stays pretty constant >> without any significant changes across requests. > >Nope. That's the thing with glibc malloc not wanting to trim >the heap for good benchmarks. > >You could also try starting with MALLOC_MMAP_THRESHOLD_=131072 >in env (or some smaller/larger number in bytes) to force it to >use mmap in more cases instead of sbrk. I've restarted the process and I'm running mmap -x $PID | tail -1 on it once a minute. I'll try to collect this data for a while and see if I can notice significant increases and correlate that with access logs. >From the first few minutes of running I see: Thu Jun 6 22:06:03 UTC 2019 total kB 298160 102744 96836 Thu Jun 6 22:07:03 UTC 2019 total kB 355884 154968 147664 Thu Jun 6 22:08:03 UTC 2019 total kB 355884 154980 147664 Thu Jun 6 22:09:03 UTC 2019 total kB 359976 156788 148336 Thu Jun 6 22:10:03 UTC 2019 total kB 359976 156788 148336 Thu Jun 6 22:11:03 UTC 2019 total kB 359976 156788 148336 Thu Jun 6 22:12:03 UTC 2019 total kB 365464 166612 158160 Thu Jun 6 22:13:03 UTC 2019 total kB 366884 167908 159456 Thu Jun 6 22:14:03 UTC 2019 total kB 366884 167908 159456 Thu Jun 6 22:15:03 UTC 2019 total kB 366884 167908 159456 >Without concurrent connections; I can't see that happening >unless there's a single message which is gigabytes in size. I'm >already irked that Email::MIME requires slurping entire emails >into memory; but it should not be using more than one >Email::MIME object in memory-at-a-time for a single client. > >Anything from varnish/nginx logs can't keep up for some reason? Speaking of logs, I did notice that even though we're passing -1 /var/log/public-inbox/httpd.out.log, that file stays empty. There's nttpd.out.log there, which is non-empty, so that's curious: # ls -ahl total 2.6M drwx------. 2 publicinbox publicinbox 177 Jun 6 22:05 . drwxr-xr-x. 21 root root 4.0K Jun 2 03:12 .. -rw-r--r--. 1 publicinbox publicinbox 0 Jun 6 22:05 httpd.out.log -rw-r--r--. 1 publicinbox publicinbox 422K Jun 6 22:04 nntpd.out.log -rw-r--r--. 1 publicinbox publicinbox 771K May 12 01:02 nntpd.out.log-20190512.gz -rw-r--r--. 1 publicinbox publicinbox 271K May 19 03:45 nntpd.out.log-20190519.gz -rw-r--r--. 1 publicinbox publicinbox 86K May 25 22:23 nntpd.out.log-20190526.gz -rw-r--r--. 1 publicinbox publicinbox 1.1M Jun 2 00:52 nntpd.out.log-20190602 Could it be that stdout is not being written out and is just perpetually buffered? That could explain the ever-growing size. -K