user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 04/10] search: fix space regressions from recent changes
Date: Fri,  9 Sep 2016 00:01:25 +0000	[thread overview]
Message-ID: <20160909000131.18584-5-e@80x24.org> (raw)
In-Reply-To: <20160909000131.18584-1-e@80x24.org>

As of Xapian 1.0.4 (from 2007) is possible to use
Search::Xapian::QueryParser::add_prefix multiple times with the
same user field name but different term prefixes.

This brings my current git@vger mirror from 6.5GB to 2.1GB
(both sizes are after xapian-compact).
---
 lib/PublicInbox/Search.pm    | 15 +++++++++------
 lib/PublicInbox/SearchIdx.pm | 25 ++++---------------------
 2 files changed, 13 insertions(+), 27 deletions(-)

diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index f74129d..c8e297f 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -60,20 +60,23 @@ my %bool_pfx_external = (
 my %prob_prefix = (
 	# for mairix compatibility
 	s => 'S',
-	m => 'Q', # 'mid' is exact, 'm' can do partial
+	m => 'XMID', # 'mid:' (bool) is exact, 'm:' (prob) can do partial
 	f => 'A',
 	t => 'XTO',
-	tc => 'XTC',
+	tc => 'XTO XCC',
 	c => 'XCC',
-	tcf => 'XTCF',
-	b => 'XBODY',
-	bs => 'XBS',
+	tcf => 'XTO XCC A',
+	b => 'XNQ XQUOT',
+	bs => 'XNQ XQUOT S',
 
 	# n.b.: leaving out "a:" alias for "tcf:" even though
 	# mairix supports it.  It is only mentioned in passing in mairix(1)
 	# and the extra two letters are not significantly longer.
 	q => 'XQUOT',
 	nq => 'XNQ',
+
+	# default:
+	'' => 'XMID S A XNQ XQUOT',
 );
 
 # not documenting m: and mid: for now, the using the URLs works w/o Xapian
@@ -241,7 +244,7 @@ EOF
 	}
 
 	while (my ($name, $prefix) = each %prob_prefix) {
-		$qp->add_prefix($name, $prefix);
+		$qp->add_prefix($name, $_) foreach split(/ /, $prefix);
 	}
 
 	$self->{query_parser} = $qp;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index cd27a29..ae89060 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -129,15 +129,9 @@ sub index_users ($$) {
 
 	$tg->index_text($from, 1, 'A'); # A - author
 	$tg->increase_termpos;
-
 	$tg->index_text($to, 1, 'XTO') if $to ne '';
+	$tg->increase_termpos;
 	$tg->index_text($cc, 1, 'XCC') if $cc ne '';
-	my $tc = join("\t", $to, $cc);
-	$tg->index_text($tc, 1, 'XTC') if $tc ne '';
-	my $tcf = join("\t", $tc, $from);
-	$tg->index_text($tcf, 1, 'XTCF') if $tcf ne '';
-
-	$tg->index_text($from);
 	$tg->increase_termpos;
 }
 
@@ -173,12 +167,7 @@ sub add_message {
 		my $tg = $self->term_generator;
 
 		$tg->set_document($doc);
-		if ($subj) {
-			$tg->index_text($subj, 1, 'S');
-			$tg->index_text($subj, 1, 'XBS');
-		}
-		$tg->increase_termpos;
-		$tg->index_text($subj) if $subj;
+		$tg->index_text($subj, 1, 'S') if $subj;
 		$tg->increase_termpos;
 
 		index_users($tg, $smsg);
@@ -204,25 +193,19 @@ sub add_message {
 			if (@quot) {
 				my $s = join("\n", @quot);
 				@quot = ();
-				$tg->index_text($s, 1, 'XQUOT');
-				$tg->index_text($s, 0, 'XBS');
-				$tg->index_text($s, 0, 'XBODY');
-				$tg->index_text($s, 0);
+				$tg->index_text($s, 0, 'XQUOT');
 				$tg->increase_termpos;
 			}
 			if (@orig) {
 				my $s = join("\n", @orig);
 				@orig = ();
 				$tg->index_text($s, 1, 'XNQ');
-				$tg->index_text($s, 1, 'XBS');
-				$tg->index_text($s, 1, 'XBODY');
-				$tg->index_text($s);
 				$tg->increase_termpos;
 			}
 		});
 
 		link_message($self, $smsg, $old_tid);
-		$tg->index_text($mid, 1);
+		$tg->index_text($mid, 1, 'XMID');
 		$doc->set_data($smsg->to_doc_data($blob));
 
 		if (my $altid = $self->{-altid}) {
-- 
EW


  parent reply	other threads:[~2016-09-09  0:01 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-09-09  0:01 [PATCH 0/10] search: more mairix prefix compatibility Eric Wong
2016-09-09  0:01 ` [PATCH 01/10] search: allow searching user fields (To/Cc/From) Eric Wong
2016-09-09  0:01 ` [PATCH 02/10] search: drop longer subject: prefix for search Eric Wong
2016-09-09  0:01 ` [PATCH 03/10] search: more granular message body searching Eric Wong
2016-09-09  0:01 ` Eric Wong [this message]
2016-09-09  0:01 ` [PATCH 05/10] search: match quote detection behavior of view Eric Wong
2016-09-09  0:01 ` [PATCH 06/10] search: increase term positions for each quoted hunk Eric Wong
2016-09-09  0:01 ` [PATCH 07/10] search: fix compatibility with Debian wheezy Eric Wong
2016-09-09  0:01 ` [PATCH 08/10] search: avoid mindlessly calling body_set Eric Wong
2016-09-09  0:01 ` [PATCH 09/10] search: match the behavior of WWW for indexing text Eric Wong
2016-09-09  0:01 ` [PATCH 10/10] search: index attachment filenames Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160909000131.18584-5-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).