* [PATCH 5/7] searchmsg: remove unused fields for PSGI in Xapian results
2019-01-10 21:35 7% [PATCH 0/7] psgi: more memory reductions Eric Wong
@ 2019-01-10 21:35 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-01-10 21:35 UTC (permalink / raw)
To: meta
These fields are only necessary in NNTP and not even stored in
Xapian; so keeping them around for the PSGI web UI search
results wastes nearly 80K when loading large result sets.
---
lib/PublicInbox/SearchMsg.pm | 13 ++++++++++++-
t/search.t | 10 ++++++----
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/SearchMsg.pm b/lib/PublicInbox/SearchMsg.pm
index 65e085f..292efce 100644
--- a/lib/PublicInbox/SearchMsg.pm
+++ b/lib/PublicInbox/SearchMsg.pm
@@ -57,11 +57,14 @@ sub load_from_data ($$) {
# To: and Cc: are stored to optimize HDR/XHDR in NNTP since
# some NNTP clients will use that for message displays.
+ # NNTP only, and only stored in Over(view), not Xapian
$self->{to},
$self->{cc},
$self->{blob},
$self->{mid},
+
+ # NNTP only
$self->{bytes},
$self->{lines}
) = split(/\n/, $_[1]);
@@ -79,10 +82,18 @@ sub load_expand {
$self;
}
+# Only called by PSGI interface, not NNTP
sub load_doc {
my ($class, $doc) = @_;
my $self = bless {}, $class;
- load_expand($self, $doc);
+ my $smsg = load_expand($self, $doc);
+
+ from_name($smsg); # fill in {from_name} so we can delete {from}
+
+ # drop NNTP-only fields which aren't relevant to PSGI results:
+ # saves ~80K on a 200 item search result:
+ delete @$smsg{qw(from ts to cc bytes lines)};
+ $smsg;
}
# :bytes and :lines metadata in RFC 3977
diff --git a/t/search.t b/t/search.t
index 3c758e6..6415a64 100644
--- a/t/search.t
+++ b/t/search.t
@@ -341,7 +341,7 @@ $ibx->with_umask(sub {
is(scalar(@$res), 1,
"searched $pfx successfully for From:");
foreach my $smsg (@$res) {
- like($smsg->from, qr/Laggy Sender/,
+ like($smsg->from_name, qr/Laggy Sender/,
"From appears with $pfx");
}
}
@@ -358,16 +358,18 @@ $ibx->with_umask(sub {
$res = $ro->query('q:theatre');
is(scalar(@$res), 1, 'only one quoted body');
- like($res->[0]->from, qr/\AQuoter/, 'got quoted body') if scalar(@$res);
+ like($res->[0]->from_name, qr/\AQuoter/,
+ 'got quoted body') if (scalar(@$res));
$res = $ro->query('nq:theatre');
is(scalar @$res, 1, 'only one non-quoted body');
- like($res->[0]->from, qr/\ANon-Quoter/, 'got non-quoted body') if scalar(@$res);
+ like($res->[0]->from_name, qr/\ANon-Quoter/,
+ 'got non-quoted body') if (scalar(@$res));
foreach my $pfx (qw(b: bs:)) {
$res = $ro->query($pfx . 'theatre');
is(scalar @$res, 2, "searched both bodies for $pfx");
- like($res->[0]->from, qr/\ANon-Quoter/,
+ like($res->[0]->from_name, qr/\ANon-Quoter/,
"non-quoter first for $pfx") if scalar(@$res);
}
}
--
EW
^ permalink raw reply related [relevance 6%]
* [PATCH 0/7] psgi: more memory reductions
@ 2019-01-10 21:35 7% Eric Wong
2019-01-10 21:35 6% ` [PATCH 5/7] searchmsg: remove unused fields for PSGI in Xapian results Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-01-10 21:35 UTC (permalink / raw)
To: meta
While of these are as significant as the patch avoid inadvertant
MIME objects storage in threads(*), they add up to some meaningful
reductions and can make it easier for memory-starved VPS to serve
serve public-inboxes.
I've diffed output of /T/, /t/ and &x=t endpoints of various HTML
pages before and after without finding differences.
There's definitely more that can be done in this area, though...
Sprinkling Devel::Size::total_size calls in various places (mostly
->getline iterators/callbacks ) was instrumental in the development
of these patches.
(*) https://public-inbox.org/meta/20190108004606.23760-1-e@80x24.org/
("view: stop storing all MIME objects on large threads")
Eric Wong (7):
httpd: remove psgix.harakiri reference
searchmsg: get rid of termlist scanning for mid
searchmsg: remove Xapian::Document field
searchview: drop unused {seen} hashref
searchmsg: remove unused fields for PSGI in Xapian results
over: cull unneeded fields for get_thread
view: more culling for search threads
lib/PublicInbox/HTTPD.pm | 1 -
lib/PublicInbox/Inbox.pm | 5 ++--
lib/PublicInbox/Over.pm | 19 ++++++++-----
lib/PublicInbox/SearchIdx.pm | 6 ++--
lib/PublicInbox/SearchMsg.pm | 49 ++++++++++++++++-----------------
lib/PublicInbox/SearchThread.pm | 5 ++++
lib/PublicInbox/SearchView.pm | 1 -
lib/PublicInbox/View.pm | 10 +++++--
t/search.t | 10 ++++---
9 files changed, 60 insertions(+), 46 deletions(-)
--
EW
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-01-10 21:35 7% [PATCH 0/7] psgi: more memory reductions Eric Wong
2019-01-10 21:35 6% ` [PATCH 5/7] searchmsg: remove unused fields for PSGI in Xapian results Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).