how's memory use? May 2020 edition

how's memory use? May 2020 edition

[Eric Wong]

Hey all, if possible; I'd like to know the memory use of your
daemons (particularly -httpd), relevant pmap(1) (or equivalent)
output, and version of public-inbox in use.

I'm primarily a GNU/Linux user, so much of the following is
glibc-specific.  If you use another malloc (or libc) I'd also be
interested to know.  I expect my changes to be sympathetic to
the way all reasonable malloc implementations work.

Over the past few months, my processes have been using less RAM.
With 1.5.0 on 64-bit systems, I don't see -httpd go stay at or
above ~80MB RSS, 32-bit systems ~50 MB.  It might spike for
giant messages, but the change to preload encodings[1] seems
to let malloc to trim the top of the heap more consistently.

The biggest message in an inbox is still a factor, and I use
this to find the largest blob in a git repo:

    git cat-file --batch-check --batch-all-objects --unordered | \
      awk '$2 == "blob" && $3 > max { max = $3; oid = $1 } END {print oid, max}'

It's usually spam, which won't get served if
"public-inbox-learn spam"-ed away.

Linux-based systems with `procps' installed can use pmap to show
anonymous mappings (not sure about other OSes):

    pmap $PID | grep -w anon

On a "beefy" 64-bit workstation running -httpd, there's only
one giant anonymous region (and several smaller ones probably
not used by malloc):

    00055df38140000  63540K rw---   [ anon ]

Above is for the process which hosts http://czquwvybam4bgbro.onion/

On a lesser VM (still 64-bit) which hosts http://hjrcffqmbrq6wope.onion/,
the heap is split since the lack of space caused sbrk(2) to fail
and forced malloc to use mmap(2) to create a new (sub) heap:

    00005575d3d4a000  30616K rw---   [ anon ]
    00005575d5b30000  13852K rw---   [ anon ]

glibc malloc defaults to a sliding window for mmap, so messages
which are beyond that window won't risk fragmenting the main
heap for their in-memory representation.

For the curious, the Linux mallopt(3) manpage also documents
environment variables which can be used to set a fixed mmap
window, trim threshold, and several other malloc knobs.
However, one of my goals is to get things working as well as
possible out-of-the-box so users won't need to fiddle with
knobs :>

-nntpd uses significantly less memory than -httpd since it:

1) doesn't split MIME parts
2) doesn't decode quoted-printable or base64
3) doesn't do character set conversions

STARTTLS or NNTPS for OpenSSL requires a significant amount of
per-socket memory, though I'm not sure how many NNTP readers
there are and if they use TLS.


[1] - https://public-inbox.org/meta/20200508015901.GA27432@dcvr/
      ("www: preload: load all encodings at startup")

Re: how's memory use? May 2020 edition

[Konstantin Ryabitsev]

On Tue, May 12, 2020 at 08:37:34AM +0000, Eric Wong wrote:
> Hey all, if possible; I'd like to know the memory use of your
> daemons (particularly -httpd), relevant pmap(1) (or equivalent)
> output, and version of public-inbox in use.

This is on lore.kernel.org. We upgraded to 1.5.0 yesterday, so this is 
only after a day of running, but this usually covers a lot of traffic.
We run with -W4, hence 4 different outputs:

# pgrep -f public-inbox-httpd | xargs pmap | grep anon
0000000002093000  23568K rw---   [ anon ]
00007f981c6dc000     84K rw---   [ anon ]
00007f981d2c5000      4K rw---   [ anon ]
00007f982802f000     20K rw---   [ anon ]
00007f982824c000     16K rw---   [ anon ]
00007f982865c000    184K rw---   [ anon ]
00007f9828da8000      8K rw---   [ anon ]
00007f9828fc2000      8K rw---   [ anon ]
00007f9829351000      4K rw---   [ anon ]
00007f982953f000    160K rw---   [ anon ]
00007f9829572000      4K rw---   [ anon ]
00007f9829575000      4K rw---   [ anon ]
00007fffddbe2000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
0000000002093000  23568K rw---   [ anon ]
0000000003797000 235060K rw---   [ anon ]
00007f981a8fd000   2736K rw---   [ anon ]
00007f981c6dc000     84K rw---   [ anon ]
00007f981d2c5000      4K rw---   [ anon ]
00007f982802f000     20K rw---   [ anon ]
00007f982824c000     16K rw---   [ anon ]
00007f982865c000    184K rw---   [ anon ]
00007f9828da8000      8K rw---   [ anon ]
00007f9828fc2000      8K rw---   [ anon ]
00007f9829351000      4K rw---   [ anon ]
00007f982953f000    160K rw---   [ anon ]
00007f9829572000      4K rw---   [ anon ]
00007f9829575000      4K rw---   [ anon ]
00007fffddbe2000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
0000000002093000  23568K rw---   [ anon ]
0000000003797000 216568K rw---   [ anon ]
00007f98196cc000   4724K rw---   [ anon ]
00007f9819b69000   4876K rw---   [ anon ]
00007f981a02c000   2736K rw---   [ anon ]
00007f981aeb5000   3496K rw---   [ anon ]
00007f981b350000   1628K rw---   [ anon ]
00007f981c6dc000     84K rw---   [ anon ]
00007f981d2c5000      4K rw---   [ anon ]
00007f982802f000     20K rw---   [ anon ]
00007f982824c000     16K rw---   [ anon ]
00007f982865c000    184K rw---   [ anon ]
00007f9828da8000      8K rw---   [ anon ]
00007f9828fc2000      8K rw---   [ anon ]
00007f9829351000      4K rw---   [ anon ]
00007f982953f000    160K rw---   [ anon ]
00007f9829572000      4K rw---   [ anon ]
00007f9829575000      4K rw---   [ anon ]
00007fffddbe2000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
0000000002093000  23568K rw---   [ anon ]
0000000003797000 241724K rw---   [ anon ]
00007f981a8e2000   2736K rw---   [ anon ]
00007f981b16f000   1964K rw---   [ anon ]
00007f981b35a000   1300K rw---   [ anon ]
00007f981c6dc000     84K rw---   [ anon ]
00007f981d2c5000      4K rw---   [ anon ]
00007f982802f000     20K rw---   [ anon ]
00007f982824c000     16K rw---   [ anon ]
00007f982865c000    184K rw---   [ anon ]
00007f9828da8000      8K rw---   [ anon ]
00007f9828fc2000      8K rw---   [ anon ]
00007f9829351000      4K rw---   [ anon ]
00007f982953f000    160K rw---   [ anon ]
00007f9829572000      4K rw---   [ anon ]
00007f9829575000      4K rw---   [ anon ]
00007fffddbe2000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
0000000002093000  23568K rw---   [ anon ]
0000000003797000 202632K rw---   [ anon ]
00007f981c6dc000     84K rw---   [ anon ]
00007f981d2c5000      4K rw---   [ anon ]
00007f982802f000     20K rw---   [ anon ]
00007f982824c000     16K rw---   [ anon ]
00007f982865c000    184K rw---   [ anon ]
00007f9828da8000      8K rw---   [ anon ]
00007f9828fc2000      8K rw---   [ anon ]
00007f9829351000      4K rw---   [ anon ]
00007f982953f000    160K rw---   [ anon ]
00007f9829572000      4K rw---   [ anon ]
00007f9829575000      4K rw---   [ anon ]
00007fffddbe2000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]

Best,
-K

Re: how's memory use? May 2020 edition

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, May 12, 2020 at 08:37:34AM +0000, Eric Wong wrote:
> > Hey all, if possible; I'd like to know the memory use of your
> > daemons (particularly -httpd), relevant pmap(1) (or equivalent)
> > output, and version of public-inbox in use.
> 
> This is on lore.kernel.org. We upgraded to 1.5.0 yesterday, so this is 
> only after a day of running, but this usually covers a lot of traffic.
> We run with -W4, hence 4 different outputs:

Thanks for the info!  I forgot to ask in the original post, but
having an idea of active connections per worker might be useful
too.  I rarely see more than a few dozen, myself.

> # pgrep -f public-inbox-httpd | xargs pmap | grep anon

Would've been easier for humans to read the output if each
process were individually broken out, but I can figure it out
from addresses below :>

> 0000000002093000  23568K rw---   [ anon ]

OK, that looks like the heap of the master.

> 00007f981c6dc000     84K rw---   [ anon ]
> 00007f981d2c5000      4K rw---   [ anon ]
> 00007f982802f000     20K rw---   [ anon ]
> 00007f982824c000     16K rw---   [ anon ]
> 00007f982865c000    184K rw---   [ anon ]
> 00007f9828da8000      8K rw---   [ anon ]
> 00007f9828fc2000      8K rw---   [ anon ]
> 00007f9829351000      4K rw---   [ anon ]
> 00007f982953f000    160K rw---   [ anon ]
> 00007f9829572000      4K rw---   [ anon ]
> 00007f9829575000      4K rw---   [ anon ]
> 00007fffddbe2000      8K r-x--   [ anon ]
> ffffffffff600000      4K r-x--   [ anon ]

Probably somethings used by glibc internally, or maybe
SQLite, Xapian.  Good thing is the above mappings now
get shared with children and are copy-on-write

OK, onto another process:

> 0000000002093000  23568K rw---   [ anon ]

That looks inherited with the parent.

> 0000000003797000 235060K rw---   [ anon ]

Ah, so that's probably the main heap after forking (I forget
my 64-bit process uses -W0, so no workers in that setup).

Anyways, 250-300MB seems a lot better than things were for lore
few months ago (closer to ~1G per worker, IIRC?).

I've still got a some pure Perl ideas (and plenty with
Inline::C), though I'll probably prioritize other things, first,
such as IMAP.

> 00007f981a8fd000   2736K rw---   [ anon ]

Not sure where the above comes from, but it's an odd allocation
that seems to get pulled in by most other workers, independently.

> 00007f981c6dc000     84K rw---   [ anon ]
> 00007f981d2c5000      4K rw---   [ anon ]
> 00007f982802f000     20K rw---   [ anon ]
> 00007f982824c000     16K rw---   [ anon ]
> 00007f982865c000    184K rw---   [ anon ]
> 00007f9828da8000      8K rw---   [ anon ]
> 00007f9828fc2000      8K rw---   [ anon ]
> 00007f9829351000      4K rw---   [ anon ]
> 00007f982953f000    160K rw---   [ anon ]
> 00007f9829572000      4K rw---   [ anon ]
> 00007f9829575000      4K rw---   [ anon ]
> 00007fffddbe2000      8K r-x--   [ anon ]
> ffffffffff600000      4K r-x--   [ anon ]

That all looks shared from the parent, good.

> 0000000002093000  23568K rw---   [ anon ]
> 0000000003797000 216568K rw---   [ anon ]

OK, similar to the other worker.

> 00007f98196cc000   4724K rw---   [ anon ]
> 00007f9819b69000   4876K rw---   [ anon ]
> 00007f981a02c000   2736K rw---   [ anon ]
> 00007f981aeb5000   3496K rw---   [ anon ]
> 00007f981b350000   1628K rw---   [ anon ]

Weird, going to need to source dive into other dependencies to
figure this out, but also not a lot compared to the main 200MB+
heap, either.

I've got some odd ones like those, too, and they seem to
persist...

<snip 00007f981c6dc000 - ffffffffff600000>

> 0000000002093000  23568K rw---   [ anon ]
> 0000000003797000 241724K rw---   [ anon ]

> 00007f981a8e2000   2736K rw---   [ anon ]
> 00007f981b16f000   1964K rw---   [ anon ]
> 00007f981b35a000   1300K rw---   [ anon ]

Ditto for these mysterious allocations

<snip 00007f981c6dc000 - ffffffffff600000>

> 0000000002093000  23568K rw---   [ anon ]
> 0000000003797000 202632K rw---   [ anon ]

Probably the least busy process, and no odd >1MB mappings.

Anyways, things seem looking much better than they were in the
past.  Regexp matching and split() for MIME is still a problem,
and some lists like linux-mtd having some giant multi-MB spam
that gets crawled...

Thanks again for the info!