* [PATCH 02/13] wwwatomstream: convert callers to use smsg_eml
2020-06-01 10:06 6% [PATCH 00/13] smsg: remove tricky {mime} field Eric Wong
@ 2020-06-01 10:06 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-06-01 10:06 UTC (permalink / raw)
To: meta
We can simplify WwwAtomStream callbacks by performing ->smsg_eml
calls in the `feed_entry' sub itself. This simplifies callers,
by reducing the number of places which can load an Eml object
into memory.
---
lib/PublicInbox/Feed.pm | 2 +-
lib/PublicInbox/SearchView.pm | 2 +-
lib/PublicInbox/WwwAtomStream.pm | 9 +++++----
3 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/Feed.pm b/lib/PublicInbox/Feed.pm
index e64628be830..b770a35077c 100644
--- a/lib/PublicInbox/Feed.pm
+++ b/lib/PublicInbox/Feed.pm
@@ -12,7 +12,7 @@ use PublicInbox::Smsg; # this loads w/o Search::Xapian
sub generate_i {
my ($ctx) = @_;
while (my $smsg = shift @{$ctx->{msgs}}) {
- $ctx->{-inbox}->smsg_mime($smsg) and return $smsg;
+ return $smsg;
}
}
diff --git a/lib/PublicInbox/SearchView.pm b/lib/PublicInbox/SearchView.pm
index 4336e4d9b2d..249cf53926d 100644
--- a/lib/PublicInbox/SearchView.pm
+++ b/lib/PublicInbox/SearchView.pm
@@ -340,7 +340,7 @@ sub adump_i {
my $smsg = eval {
PublicInbox::Smsg::from_mitem($mi, $ctx->{srch});
} or next;
- $ctx->{-inbox}->smsg_mime($smsg) and return $smsg;
+ return $smsg;
}
}
diff --git a/lib/PublicInbox/WwwAtomStream.pm b/lib/PublicInbox/WwwAtomStream.pm
index c3fbb1a7cef..6ed0cb212d6 100644
--- a/lib/PublicInbox/WwwAtomStream.pm
+++ b/lib/PublicInbox/WwwAtomStream.pm
@@ -12,6 +12,7 @@ use warnings;
use POSIX qw(strftime);
use Digest::SHA qw(sha1_hex);
use PublicInbox::Address;
+use PublicInbox::MID qw(mids);
use PublicInbox::Hval qw(ascii_html mid_href);
use PublicInbox::MsgTime qw(msg_timestamp);
@@ -99,9 +100,9 @@ sub atom_header {
sub feed_entry {
my ($self, $smsg) = @_;
my $ctx = $self->{ctx};
- my $mid = $smsg->mid; # may extract Message-ID from {mime}
- my $mime = delete $smsg->{mime};
- my $hdr = $mime->header_obj;
+ my $eml = $ctx->{-inbox}->smsg_eml($smsg) or return '';
+ my $hdr = $eml->header_obj;
+ my $mid = $smsg->{mid} // mids($hdr)->[0];
my $irt = PublicInbox::View::in_reply_to($hdr);
my $uuid = to_uuid($mid);
my $base = $ctx->{feed_base_url};
@@ -141,7 +142,7 @@ sub feed_entry {
qq(<pre\nstyle="white-space:pre-wrap">);
$ctx->{obuf} = \$s;
$ctx->{mhref} = $href;
- PublicInbox::View::multipart_text_as_html($mime, $ctx);
+ PublicInbox::View::multipart_text_as_html($eml, $ctx);
delete $ctx->{obuf};
$s .= '</pre></div></content></entry>';
}
^ permalink raw reply related [relevance 7%]
* [PATCH 00/13] smsg: remove tricky {mime} field
@ 2020-06-01 10:06 6% Eric Wong
2020-06-01 10:06 7% ` [PATCH 02/13] wwwatomstream: convert callers to use smsg_eml Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-06-01 10:06 UTC (permalink / raw)
To: meta
Storing a large PublicInbox::Eml (or in the past, Email::MIME)
object inside a small PublicInbox::Smsg object has historically
been bloat-prone[1] since there may be many small smsgs in
memory at once
Hundreds or thousands of $smsg objects can linger in memory due
to search results and message threading operations. So keep
$eml and $smsg objects independent of each other, for now.
Instead, we'll introduce a $smsg->populate($eml) API to handle
filling in the keys for the importer, indexer, and
non-SQLite-using WWW users.
Furthermore, $smsg->$field dispatch has always been measurably
faster than $smsg->{$field} access in NNTP. Since $smsg->$field
became read-only with the removal of $smsg->{mime}, we can
abandon the $smsg->$field invocations and favor of direct hash
access.
[1] the prime example being what commit 7d02b9e64455831d fixed
("view: stop storing all MIME objects on large threads")
Eric Wong (13):
inbox: introduce smsg_eml method
wwwatomstream: convert callers to use smsg_eml
v2writable: fix non-sensical interpolation in BUG message
import: modernize to use Perl 5.10 features
smsg: introduce ->populate method
smsg: get rid of ->wrap initializer, too
inbox: msg_by_*: remove $(size)ref args
www: remove smsg_mime API and adjust callers
nntp: smsg_range_i: favor ->{$field} lookups when possible
smsg: get rid of remaining {mime} users
smsg: remove ->bytes and ->lines methods
smsg: remove remaining accessor methods
wwwatomstream: drop smsg->{mid} fallback for non-SQLite
Documentation/mknews.perl | 7 +-
lib/PublicInbox/ExtMsg.pm | 2 +-
lib/PublicInbox/Feed.pm | 8 +-
lib/PublicInbox/Import.pm | 69 ++++++++---------
lib/PublicInbox/Inbox.pm | 32 ++++----
lib/PublicInbox/Mbox.pm | 2 +-
lib/PublicInbox/NNTP.pm | 14 +++-
lib/PublicInbox/OverIdx.pm | 3 +-
lib/PublicInbox/SearchIdx.pm | 33 ++++-----
lib/PublicInbox/SearchView.pm | 6 +-
lib/PublicInbox/Smsg.pm | 123 +++++++++++--------------------
lib/PublicInbox/SolverGit.pm | 4 +-
lib/PublicInbox/V2Writable.pm | 11 +--
lib/PublicInbox/View.pm | 63 ++++++++--------
lib/PublicInbox/WwwAtomStream.pm | 8 +-
t/altid.t | 3 +-
t/altid_v2.t | 3 +-
t/import.t | 3 +-
t/search.t | 46 +++++++-----
t/v2mda.t | 4 +-
t/v2writable.t | 5 +-
21 files changed, 207 insertions(+), 242 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-06-01 10:06 6% [PATCH 00/13] smsg: remove tricky {mime} field Eric Wong
2020-06-01 10:06 7% ` [PATCH 02/13] wwwatomstream: convert callers to use smsg_eml Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).