Re: how's memory usage on public-inbox-httpd?

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

On Thu, Jun 06, 2019 at 08:37:52PM +0000, Eric Wong wrote:
>Do you have commit 7d02b9e64455831d3bda20cd2e64e0c15dc07df5?
>("view: stop storing all MIME objects on large threads")
>That was most significant.

Yes. We're running 743ac758 with a few cherry-picked patches on top of 
that (like epoch roll-over fix).

>Otherwise it's probably a combination of several things...
>httpd and nntpd both supports streaming, arbitrarily  large
>endpoints (all.mbox.gz, and /T/, /t/, /t.mbox.gz threads with
>thousands of messages, giant NNTP BODY/ARTICLE ranges).
>
>All those endpoints should detect backpressure from a slow
>client (varnish/nginx in your case) using the ->getline method.

Wouldn't that spike up and down? The size I'm seeing stays pretty 
constant without any significant changes across requests.

>Also, are you only using the default of -W/--worker-process=1
>on a 16-core machine?  Just checked public-inbox-httpd(8), the
>-W switch is documented :)  You can use SIGTTIN/TTOU to
>increase, decrease workers w/o restarting, too.

D'oh, yes... though it's not been a problem yet. :) I'm not sure I want 
to bump that up, though, if that means we're going to have multiple 
19GB-sized processes instead of one. :)

>Do you have any stats on the number of simultaneous connections
>public-inbox-httpd/nginx/varnish handles (and logging of that
>info at peek)?  (perhaps running "ss -tan" periodically)(*)

We don't collect that info, but I'm not sure it's the number of 
concurrent connections that's the culprit, as there is no fluctuation in 
RSS size based on the number of responses.

To answer the questions in your follow-up:

It would appear to be all in anon memory. Mem_usage [1] reports:

# ./Mem_usage 18275
Backed by file:
  Executable                r-x  16668
  Write/Exec (jump tables)  rwx  0
  RO data                   r--  106908
  Data                      rw-  232
  Unreadable                ---  94072
  Unknown                        0
Anonymous:
  Writable code (stack)     rwx  0
  Data (malloc, mmap)       rw-  19988892
  RO data                   r--  0
  Unreadable                ---  0
  Unknown                        12

I've been looking at lsof -p of that process and I see sqlite and xapian 
showing up and disappearing. The lkml ones are being accessed almost all 
the time, but even there I see them showing up with different FD 
entries, so they are being closed and reopened properly.

Hope this helps.

-K

.. [1] https://elinux.org/images/d/d3/Mem_usage