From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 88D051F55B for ; Mon, 1 Jun 2020 10:06:57 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/13] smsg: remove tricky {mime} field Date: Mon, 1 Jun 2020 10:06:44 +0000 Message-Id: <20200601100657.14700-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Storing a large PublicInbox::Eml (or in the past, Email::MIME) object inside a small PublicInbox::Smsg object has historically been bloat-prone[1] since there may be many small smsgs in memory at once Hundreds or thousands of $smsg objects can linger in memory due to search results and message threading operations. So keep $eml and $smsg objects independent of each other, for now. Instead, we'll introduce a $smsg->populate($eml) API to handle filling in the keys for the importer, indexer, and non-SQLite-using WWW users. Furthermore, $smsg->$field dispatch has always been measurably faster than $smsg->{$field} access in NNTP. Since $smsg->$field became read-only with the removal of $smsg->{mime}, we can abandon the $smsg->$field invocations and favor of direct hash access. [1] the prime example being what commit 7d02b9e64455831d fixed ("view: stop storing all MIME objects on large threads") Eric Wong (13): inbox: introduce smsg_eml method wwwatomstream: convert callers to use smsg_eml v2writable: fix non-sensical interpolation in BUG message import: modernize to use Perl 5.10 features smsg: introduce ->populate method smsg: get rid of ->wrap initializer, too inbox: msg_by_*: remove $(size)ref args www: remove smsg_mime API and adjust callers nntp: smsg_range_i: favor ->{$field} lookups when possible smsg: get rid of remaining {mime} users smsg: remove ->bytes and ->lines methods smsg: remove remaining accessor methods wwwatomstream: drop smsg->{mid} fallback for non-SQLite Documentation/mknews.perl | 7 +- lib/PublicInbox/ExtMsg.pm | 2 +- lib/PublicInbox/Feed.pm | 8 +- lib/PublicInbox/Import.pm | 69 ++++++++--------- lib/PublicInbox/Inbox.pm | 32 ++++---- lib/PublicInbox/Mbox.pm | 2 +- lib/PublicInbox/NNTP.pm | 14 +++- lib/PublicInbox/OverIdx.pm | 3 +- lib/PublicInbox/SearchIdx.pm | 33 ++++----- lib/PublicInbox/SearchView.pm | 6 +- lib/PublicInbox/Smsg.pm | 123 +++++++++++-------------------- lib/PublicInbox/SolverGit.pm | 4 +- lib/PublicInbox/V2Writable.pm | 11 +-- lib/PublicInbox/View.pm | 63 ++++++++-------- lib/PublicInbox/WwwAtomStream.pm | 8 +- t/altid.t | 3 +- t/altid_v2.t | 3 +- t/import.t | 3 +- t/search.t | 46 +++++++----- t/v2mda.t | 4 +- t/v2writable.t | 5 +- 21 files changed, 207 insertions(+), 242 deletions(-)