user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 0/7] psgi: more memory reductions
@ 2019-01-10 21:35  7% Eric Wong
  2019-01-10 21:35  6% ` [PATCH 3/7] searchmsg: remove Xapian::Document field Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-01-10 21:35 UTC (permalink / raw)
  To: meta

While of these are as significant as the patch avoid inadvertant
MIME objects storage in threads(*), they add up to some meaningful
reductions and can make it easier for memory-starved VPS to serve
serve public-inboxes.

I've diffed output of /T/, /t/ and &x=t endpoints of various HTML
pages before and after without finding differences.

There's definitely more that can be done in this area, though...

Sprinkling Devel::Size::total_size calls in various places (mostly
->getline iterators/callbacks ) was instrumental in the development
of these patches.

(*) https://public-inbox.org/meta/20190108004606.23760-1-e@80x24.org/
    ("view: stop storing all MIME objects on large threads")

Eric Wong (7):
  httpd: remove psgix.harakiri reference
  searchmsg: get rid of termlist scanning for mid
  searchmsg: remove Xapian::Document field
  searchview: drop unused {seen} hashref
  searchmsg: remove unused fields for PSGI in Xapian results
  over: cull unneeded fields for get_thread
  view: more culling for search threads

 lib/PublicInbox/HTTPD.pm        |  1 -
 lib/PublicInbox/Inbox.pm        |  5 ++--
 lib/PublicInbox/Over.pm         | 19 ++++++++-----
 lib/PublicInbox/SearchIdx.pm    |  6 ++--
 lib/PublicInbox/SearchMsg.pm    | 49 ++++++++++++++++-----------------
 lib/PublicInbox/SearchThread.pm |  5 ++++
 lib/PublicInbox/SearchView.pm   |  1 -
 lib/PublicInbox/View.pm         | 10 +++++--
 t/search.t                      | 10 ++++---
 9 files changed, 60 insertions(+), 46 deletions(-)

-- 
EW


^ permalink raw reply	[relevance 7%]

* [PATCH 3/7] searchmsg: remove Xapian::Document field
  2019-01-10 21:35  7% [PATCH 0/7] psgi: more memory reductions Eric Wong
@ 2019-01-10 21:35  6% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-01-10 21:35 UTC (permalink / raw)
  To: meta

We don't need to be carrying this around with the many SearchMsg
objects we have.  This saves about 20K from a large SearchView
"&x=t" response.
---
 lib/PublicInbox/SearchIdx.pm |  6 +++---
 lib/PublicInbox/SearchMsg.pm | 19 +++++++------------
 2 files changed, 10 insertions(+), 15 deletions(-)

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index fd48169..7a8ebf3 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -282,7 +282,7 @@ sub index_body ($$$) {
 sub add_xapian ($$$$$) {
 	my ($self, $mime, $num, $oid, $mids, $mid0) = @_;
 	my $smsg = PublicInbox::SearchMsg->new($mime);
-	my $doc = $smsg->{doc};
+	my $doc = Search::Xapian::Document->new;
 	my $subj = $smsg->subject;
 	add_val($doc, PublicInbox::Search::TS(), $smsg->ts);
 	my @ds = gmtime($smsg->ds);
@@ -439,8 +439,8 @@ sub remove_by_oid {
 	for (; $head != $tail; $head->inc) {
 		my $docid = $head->get_docid;
 		my $doc = $db->get_document($docid);
-		my $smsg = PublicInbox::SearchMsg->wrap($doc, $mid);
-		$smsg->load_expand;
+		my $smsg = PublicInbox::SearchMsg->wrap($mid);
+		$smsg->load_expand($doc);
 		if ($smsg->{blob} eq $oid) {
 			push(@delete, $docid);
 		}
diff --git a/lib/PublicInbox/SearchMsg.pm b/lib/PublicInbox/SearchMsg.pm
index 5a2ca83..65e085f 100644
--- a/lib/PublicInbox/SearchMsg.pm
+++ b/lib/PublicInbox/SearchMsg.pm
@@ -13,20 +13,18 @@ use Time::Local qw(timegm);
 
 sub new {
 	my ($class, $mime) = @_;
-	my $doc = Search::Xapian::Document->new;
-	bless { doc => $doc, mime => $mime }, $class;
+	bless { mime => $mime }, $class;
 }
 
 sub wrap {
-	my ($class, $doc, $mid) = @_;
-	bless { doc => $doc, mime => undef, mid => $mid }, $class;
+	my ($class, $mid) = @_;
+	bless { mid => $mid }, $class;
 }
 
 sub get {
 	my ($class, $head, $db, $mid) = @_;
 	my $doc_id = $head->get_docid;
-	my $doc = $db->get_document($doc_id);
-	load_expand(wrap($class, $doc, $mid))
+	load_expand(wrap($class, $mid), $db->get_document($doc_id));
 }
 
 sub get_val ($$) {
@@ -70,8 +68,7 @@ sub load_from_data ($$) {
 }
 
 sub load_expand {
-	my ($self) = @_;
-	my $doc = $self->{doc};
+	my ($self, $doc) = @_;
 	my $data = $doc->get_data or return;
 	$self->{ts} = get_val($doc, PublicInbox::Search::TS());
 	my $dt = get_val($doc, PublicInbox::Search::DT());
@@ -84,8 +81,8 @@ sub load_expand {
 
 sub load_doc {
 	my ($class, $doc) = @_;
-	my $self = bless { doc => $doc }, $class;
-	$self->load_expand;
+	my $self = bless {}, $class;
+	load_expand($self, $doc);
 }
 
 # :bytes and :lines metadata in RFC 3977
@@ -164,8 +161,6 @@ sub mid ($;$) {
 		$self->{mid} = $mid;
 	} elsif (defined(my $rv = $self->{mid})) {
 		$rv;
-	} elsif ($self->{doc}) {
-		die "SHOULD NOT HAPPEN\n";
 	} else {
 		die "NO {mime} for mid\n" unless $self->{mime};
 		$self->_extract_mid; # v1 w/o Xapian
-- 
EW


^ permalink raw reply related	[relevance 6%]

Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-01-10 21:35  7% [PATCH 0/7] psgi: more memory reductions Eric Wong
2019-01-10 21:35  6% ` [PATCH 3/7] searchmsg: remove Xapian::Document field Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).