From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 00/13] smsg: remove tricky {mime} field
Date: Mon, 1 Jun 2020 10:06:44 +0000 [thread overview]
Message-ID: <20200601100657.14700-1-e@yhbt.net> (raw)
Storing a large PublicInbox::Eml (or in the past, Email::MIME)
object inside a small PublicInbox::Smsg object has historically
been bloat-prone[1] since there may be many small smsgs in
memory at once
Hundreds or thousands of $smsg objects can linger in memory due
to search results and message threading operations. So keep
$eml and $smsg objects independent of each other, for now.
Instead, we'll introduce a $smsg->populate($eml) API to handle
filling in the keys for the importer, indexer, and
non-SQLite-using WWW users.
Furthermore, $smsg->$field dispatch has always been measurably
faster than $smsg->{$field} access in NNTP. Since $smsg->$field
became read-only with the removal of $smsg->{mime}, we can
abandon the $smsg->$field invocations and favor of direct hash
access.
[1] the prime example being what commit 7d02b9e64455831d fixed
("view: stop storing all MIME objects on large threads")
Eric Wong (13):
inbox: introduce smsg_eml method
wwwatomstream: convert callers to use smsg_eml
v2writable: fix non-sensical interpolation in BUG message
import: modernize to use Perl 5.10 features
smsg: introduce ->populate method
smsg: get rid of ->wrap initializer, too
inbox: msg_by_*: remove $(size)ref args
www: remove smsg_mime API and adjust callers
nntp: smsg_range_i: favor ->{$field} lookups when possible
smsg: get rid of remaining {mime} users
smsg: remove ->bytes and ->lines methods
smsg: remove remaining accessor methods
wwwatomstream: drop smsg->{mid} fallback for non-SQLite
Documentation/mknews.perl | 7 +-
lib/PublicInbox/ExtMsg.pm | 2 +-
lib/PublicInbox/Feed.pm | 8 +-
lib/PublicInbox/Import.pm | 69 ++++++++---------
lib/PublicInbox/Inbox.pm | 32 ++++----
lib/PublicInbox/Mbox.pm | 2 +-
lib/PublicInbox/NNTP.pm | 14 +++-
lib/PublicInbox/OverIdx.pm | 3 +-
lib/PublicInbox/SearchIdx.pm | 33 ++++-----
lib/PublicInbox/SearchView.pm | 6 +-
lib/PublicInbox/Smsg.pm | 123 +++++++++++--------------------
lib/PublicInbox/SolverGit.pm | 4 +-
lib/PublicInbox/V2Writable.pm | 11 +--
lib/PublicInbox/View.pm | 63 ++++++++--------
lib/PublicInbox/WwwAtomStream.pm | 8 +-
t/altid.t | 3 +-
t/altid_v2.t | 3 +-
t/import.t | 3 +-
t/search.t | 46 +++++++-----
t/v2mda.t | 4 +-
t/v2writable.t | 5 +-
21 files changed, 207 insertions(+), 242 deletions(-)
next reply other threads:[~2020-06-01 10:06 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-01 10:06 Eric Wong [this message]
2020-06-01 10:06 ` [PATCH 01/13] inbox: introduce smsg_eml method Eric Wong
2020-06-01 10:06 ` [PATCH 02/13] wwwatomstream: convert callers to use smsg_eml Eric Wong
2020-06-01 10:06 ` [PATCH 03/13] v2writable: fix non-sensical interpolation in BUG message Eric Wong
2020-06-01 10:06 ` [PATCH 04/13] import: modernize to use Perl 5.10 features Eric Wong
2020-06-01 10:06 ` [PATCH 05/13] smsg: introduce ->populate method Eric Wong
2020-06-01 10:06 ` [PATCH 06/13] smsg: get rid of ->wrap initializer, too Eric Wong
2020-06-01 10:06 ` [PATCH 07/13] inbox: msg_by_*: remove $(size)ref args Eric Wong
2020-06-01 10:06 ` [PATCH 08/13] www: remove smsg_mime API and adjust callers Eric Wong
2020-06-01 10:06 ` [PATCH 09/13] nntp: smsg_range_i: favor ->{$field} lookups when possible Eric Wong
2020-06-01 10:06 ` [PATCH 10/13] smsg: get rid of remaining {mime} users Eric Wong
2020-06-01 10:06 ` [PATCH 11/13] smsg: remove ->bytes and ->lines methods Eric Wong
2020-06-01 10:06 ` [PATCH 12/13] smsg: remove remaining accessor methods Eric Wong
2020-06-01 10:06 ` [PATCH 13/13] wwwatomstream: drop smsg->{mid} fallback for non-SQLite Eric Wong
2020-06-01 16:45 ` [PATCH 00/13] smsg: remove tricky {mime} field Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200601100657.14700-1-e@yhbt.net \
--to=e@yhbt.net \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).