From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,BAYES_00,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no
	autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id B80BE1F601
	for <meta@public-inbox.org>; Wed,  7 Dec 2022 21:20:13 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org;
	s=selector1; t=1670448013;
	bh=hAAkxabRmaBm+iHJP2u0B6jOHqnduTGf3mVH4MZ+BoY=;
	h=Date:From:To:Subject:From;
	b=211avwDYf7qyaHimFNN+Z8SfZHmJiug6eAE6fFOHHbShnSrNpNJx6HUcMcN7zS++d
	 WgpQ9xO4dcpcvt4+3+kp9j7PmBQvCx00sPFnGQV/Dsuddfq/kyB5Me1k34Rmi4Ocij
	 IglRY8R+oPNAKj3TuwGBaxwyJHzMpTE9kRSKfhMk=
Date: Wed, 7 Dec 2022 21:20:13 +0000
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: netd/httpd memory usage
Message-ID: <20221207212013.GA25466@dcvr>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
List-Id: <meta.public-inbox.org>

While memory use has mostly gotten under control with scratchpad
avoidance done around Sep (after v1.9.0), it's still higher than
I like :<

Suspiciously, PublicInbox::Smsg::load_from_data still seems to
be causing a big chunk of long-lived allocations on my system:

sub load_from_data ($$) {
	my ($self) = $_[0]; # data = $_[1]
	utf8::decode($_[1]);
	(
		$self->{subject},
		$self->{from},
		$self->{references},

		# To: and Cc: are stored to optimize HDR/XHDR in NNTP since
		# some NNTP clients will use that for message displays.
		# NNTP only, and only stored in Over(view), not Xapian
		$self->{to},
		$self->{cc},

		$self->{blob},
		$self->{mid},

		# NNTP only
		$self->{bytes},
		$self->{lines}
	) = split(/\n/, $_[1]);
}

I'm only on Perl 5.28.1, and looking at Perl 5.3x deltas I see
see some regexp-related leak fixes, perhaps an upgrade will fix
it...

(or work on https://80x24.org/mwrap-perl.git to expand into C
backtraces is required...)

Of course, even single-threaded git packing is using massive memory
use with the linux.git forkgroup :<