user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <>
Subject: [PATCH] view: stop storing all MIME objects on large threads
Date: Tue,  8 Jan 2019 00:46:06 +0000
Message-ID: <> (raw)

While we try to discard the $smsg (SearchMsg) objects quickly,
they remain referenced via $node (SearchThread::Msg) objects,
which are stored forever in $ctx->{mapping} to cull redundant
words out of subjects in the thread skeleton.

This significantly cuts memory bloat with large search results
with '&x=t'.  Now, the search results overhead of
SearchThread::Msg and linked objects are stable at around 350K
instead of ~7M per response in a rough test (there's more
savings to be had in the same areas).

Several hundred kilobytes is still huge and a large per-client
cost; but it's far better than MEGABYTES per-client.
 lib/PublicInbox/ | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/ b/lib/PublicInbox/
index bb49c03..416cb4a 100644
--- a/lib/PublicInbox/
+++ b/lib/PublicInbox/
@@ -218,7 +218,7 @@ sub index_entry {
 	$rv .= $subj . "\n";
 	$rv .= _th_index_lite($mid_raw, \$irt, $id, $ctx);
 	my @tocc;
-	my $mime = $smsg->{mime};
+	my $mime = delete $smsg->{mime}; # critical to memory use
 	my $hdr = $mime->header_obj;
 	my $from = _hdr_names_html($hdr, 'From');
 	obfuscate_addrs($obfs_ibx, $from) if $obfs_ibx;

                 reply index

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:

 note: .onion URLs require Tor:

AGPL code for this site: git clone