* [PATCH 01/13] inbox: introduce smsg_eml method
2020-06-01 10:06 7% [PATCH 00/13] smsg: remove tricky {mime} field Eric Wong
@ 2020-06-01 10:06 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-06-01 10:06 UTC (permalink / raw)
To: meta
The goal of this is to eventually remove the $smsg->{mime} field
which is easy-to-misuse and cause memory explosions which
necessitated fixes like commit 7d02b9e64455831d
("view: stop storing all MIME objects on large threads").
---
lib/PublicInbox/Inbox.pm | 6 ++++++
lib/PublicInbox/SolverGit.pm | 4 ++--
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index c295b2677e4..bd1489e3699 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -334,6 +334,12 @@ sub smsg_mime {
}
}
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ my $bref = msg_by_smsg($self, $smsg) or return;
+ PublicInbox::Eml->new($bref);
+}
+
sub mid2num($$) {
my ($self, $mid) = @_;
my $mm = mm($self) or return;
diff --git a/lib/PublicInbox/SolverGit.pm b/lib/PublicInbox/SolverGit.pm
index f718e28cbd5..b1cb1ae97c4 100644
--- a/lib/PublicInbox/SolverGit.pm
+++ b/lib/PublicInbox/SolverGit.pm
@@ -233,8 +233,8 @@ sub find_extract_diffs ($$$) {
my $diffs = [];
foreach my $smsg (@$msgs) {
- $ibx->smsg_mime($smsg) or next;
- delete($smsg->{mime})->each_part(\&extract_diff,
+ my $eml = $ibx->smsg_eml($smsg) or next;
+ $eml->each_part(\&extract_diff,
[$self, $diffs, $pre, $post, $ibx, $smsg], 1);
}
@$diffs ? $diffs : undef;
^ permalink raw reply related [relevance 7%]
* [PATCH 00/13] smsg: remove tricky {mime} field
@ 2020-06-01 10:06 7% Eric Wong
2020-06-01 10:06 7% ` [PATCH 01/13] inbox: introduce smsg_eml method Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-06-01 10:06 UTC (permalink / raw)
To: meta
Storing a large PublicInbox::Eml (or in the past, Email::MIME)
object inside a small PublicInbox::Smsg object has historically
been bloat-prone[1] since there may be many small smsgs in
memory at once
Hundreds or thousands of $smsg objects can linger in memory due
to search results and message threading operations. So keep
$eml and $smsg objects independent of each other, for now.
Instead, we'll introduce a $smsg->populate($eml) API to handle
filling in the keys for the importer, indexer, and
non-SQLite-using WWW users.
Furthermore, $smsg->$field dispatch has always been measurably
faster than $smsg->{$field} access in NNTP. Since $smsg->$field
became read-only with the removal of $smsg->{mime}, we can
abandon the $smsg->$field invocations and favor of direct hash
access.
[1] the prime example being what commit 7d02b9e64455831d fixed
("view: stop storing all MIME objects on large threads")
Eric Wong (13):
inbox: introduce smsg_eml method
wwwatomstream: convert callers to use smsg_eml
v2writable: fix non-sensical interpolation in BUG message
import: modernize to use Perl 5.10 features
smsg: introduce ->populate method
smsg: get rid of ->wrap initializer, too
inbox: msg_by_*: remove $(size)ref args
www: remove smsg_mime API and adjust callers
nntp: smsg_range_i: favor ->{$field} lookups when possible
smsg: get rid of remaining {mime} users
smsg: remove ->bytes and ->lines methods
smsg: remove remaining accessor methods
wwwatomstream: drop smsg->{mid} fallback for non-SQLite
Documentation/mknews.perl | 7 +-
lib/PublicInbox/ExtMsg.pm | 2 +-
lib/PublicInbox/Feed.pm | 8 +-
lib/PublicInbox/Import.pm | 69 ++++++++---------
lib/PublicInbox/Inbox.pm | 32 ++++----
lib/PublicInbox/Mbox.pm | 2 +-
lib/PublicInbox/NNTP.pm | 14 +++-
lib/PublicInbox/OverIdx.pm | 3 +-
lib/PublicInbox/SearchIdx.pm | 33 ++++-----
lib/PublicInbox/SearchView.pm | 6 +-
lib/PublicInbox/Smsg.pm | 123 +++++++++++--------------------
lib/PublicInbox/SolverGit.pm | 4 +-
lib/PublicInbox/V2Writable.pm | 11 +--
lib/PublicInbox/View.pm | 63 ++++++++--------
lib/PublicInbox/WwwAtomStream.pm | 8 +-
t/altid.t | 3 +-
t/altid_v2.t | 3 +-
t/import.t | 3 +-
t/search.t | 46 +++++++-----
t/v2mda.t | 4 +-
t/v2writable.t | 5 +-
21 files changed, 207 insertions(+), 242 deletions(-)
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-06-01 10:06 7% [PATCH 00/13] smsg: remove tricky {mime} field Eric Wong
2020-06-01 10:06 7% ` [PATCH 01/13] inbox: introduce smsg_eml method Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).