user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
Subject: how's memory use? May 2020 edition
Date: Tue, 12 May 2020 08:37:34 +0000	[thread overview]
Message-ID: <20200512083734.GA13688@dcvr> (raw)

Hey all, if possible; I'd like to know the memory use of your
daemons (particularly -httpd), relevant pmap(1) (or equivalent)
output, and version of public-inbox in use.

I'm primarily a GNU/Linux user, so much of the following is
glibc-specific.  If you use another malloc (or libc) I'd also be
interested to know.  I expect my changes to be sympathetic to
the way all reasonable malloc implementations work.

Over the past few months, my processes have been using less RAM.
With 1.5.0 on 64-bit systems, I don't see -httpd go stay at or
above ~80MB RSS, 32-bit systems ~50 MB.  It might spike for
giant messages, but the change to preload encodings[1] seems
to let malloc to trim the top of the heap more consistently.

The biggest message in an inbox is still a factor, and I use
this to find the largest blob in a git repo:

    git cat-file --batch-check --batch-all-objects --unordered | \
      awk '$2 == "blob" && $3 > max { max = $3; oid = $1 } END {print oid, max}'

It's usually spam, which won't get served if
"public-inbox-learn spam"-ed away.

Linux-based systems with `procps' installed can use pmap to show
anonymous mappings (not sure about other OSes):

    pmap $PID | grep -w anon

On a "beefy" 64-bit workstation running -httpd, there's only
one giant anonymous region (and several smaller ones probably
not used by malloc):

    00055df38140000  63540K rw---   [ anon ]

Above is for the process which hosts http://czquwvybam4bgbro.onion/

On a lesser VM (still 64-bit) which hosts http://hjrcffqmbrq6wope.onion/,
the heap is split since the lack of space caused sbrk(2) to fail
and forced malloc to use mmap(2) to create a new (sub) heap:

    00005575d3d4a000  30616K rw---   [ anon ]
    00005575d5b30000  13852K rw---   [ anon ]

glibc malloc defaults to a sliding window for mmap, so messages
which are beyond that window won't risk fragmenting the main
heap for their in-memory representation.

For the curious, the Linux mallopt(3) manpage also documents
environment variables which can be used to set a fixed mmap
window, trim threshold, and several other malloc knobs.
However, one of my goals is to get things working as well as
possible out-of-the-box so users won't need to fiddle with
knobs :>

-nntpd uses significantly less memory than -httpd since it:

1) doesn't split MIME parts
2) doesn't decode quoted-printable or base64
3) doesn't do character set conversions

STARTTLS or NNTPS for OpenSSL requires a significant amount of
per-socket memory, though I'm not sure how many NNTP readers
there are and if they use TLS.

[1] -
      ("www: preload: load all encodings at startup")

             reply	other threads:[~2020-05-12  8:37 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-12  8:37 Eric Wong [this message]
2020-05-14 20:57 ` how's memory use? May 2020 edition Konstantin Ryabitsev
2020-05-15  5:23   ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200512083734.GA13688@dcvr \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).