From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id C8B8B1F54E for ; Sat, 23 Jul 2022 06:12:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1658556736; bh=VOI18MTLuIz3xIQTyis5MwPLbZpacSRC8YE0cjEkkx0=; h=From:To:Subject:Date:From; b=gF8V0ODnCg8GHpIbv5FVML6pJFCDoXSSK3WrwYwJaAwFIEIO6R+yLX7EDzTXlk4Mr E3D9mc32/3kABUUhURAG/hKrrKM0UUdZiw+dIRdkuzNJ7fTTlLryzq9AbOHKZcmse8 4rkzUAu/f0Beltbth3HTJ85T8OyH9F9sABF6Uo60= From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] pop3: reduce memory use while generating the mailbox cache Date: Sat, 23 Jul 2022 06:12:16 +0000 Message-Id: <20220723061216.3776894-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: While the cache itself is relatively compact for 50K messages, generating it was inefficient due to our schema and Over.pm APIs being designed for NNTP. While we won't change our schema for now, we can choose better DBI APIs to use and limit our ephemeral memory use. This amounts to a 60% reduction in memory usage and a 5-10% speedup against org.kernel.vger.git.0: { echo 'USER '$(uuidgen)'@org.kernel.vger.git.0' echo PASS anonymous echo STAT echo QUIT } | nc $HOST $PORT --- lib/PublicInbox/POP3.pm | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/PublicInbox/POP3.pm b/lib/PublicInbox/POP3.pm index 60eedea7..203c91a6 100644 --- a/lib/PublicInbox/POP3.pm +++ b/lib/PublicInbox/POP3.pm @@ -148,12 +148,21 @@ sub _stat_cache ($) { my ($self) = @_; my ($beg, $end) = (($self->{uid_dele} // -1) + 1, $self->{uid_max}); PublicInbox::IMAP::uid_clamp($self, \$beg, \$end); - my $opt = { limit => PublicInbox::IMAP::UID_SLICE }; - my $m = $self->{ibx}->over(1)->do_get(<<'', $opt, $beg, $end); + my (@cache, $m); + my $sth = $self->{ibx}->over(1)->dbh->prepare_cached(<<'', undef, 1); SELECT num,ddd FROM over WHERE num >= ? AND num <= ? ORDER BY num ASC - [ map { ($_->{num}, $_->{bytes} + 0, $_->{blob}) } @$m ]; + $sth->execute($beg, $end); + do { + $m = $sth->fetchall_arrayref({}, 1000); + for my $x (@$m) { + PublicInbox::Over::load_from_row($x); + push(@cache, $x->{num}, $x->{bytes} + 0, $x->{blob}); + undef $x; # saves ~1.5M memory w/ 50k messages + } + } while (scalar(@$m) && ($beg = $cache[-3] + 1)); + \@cache; } sub cmd_stat {