user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: meta@public-inbox.org
Subject: [PATCH] v1: allow upgrading indexlevel=basic to 'medium' or 'full'
Date: Fri, 20 Jul 2018 06:58:45 +0000
Message-ID: <20180720065845.fho2shm2tjgacq3l@whir> (raw)
In-Reply-To: <877elrf4gq.fsf@xmission.com>

"Eric W. Biederman" <ebiederm@xmission.com> wrote:
> Eric Wong <e@80x24.org> writes:
> > I think being able to transition from an existing "basic" to
> > (full|medium) can be straightforward, too, with the untested
> > patch below.
> >
> > But most other transitions between levels will require an expensive
> > --reindex.   (full|medium) => basic would be easy, too, but
> > it'll be up to the user to remove the Xapian files.
> 
> I would still use "$self->{indexlevel} =~ m/(full|medium)/" instead of
> ne 'basic' just in case we start recognizing another level.
 
Good catch.

> Otherwise your change seems to make sense.  I have not really dug into
> the reindex case where the point is to pick up things that were
> previously missed.  I can definitely see that code should be guarded
> with an indexlevel check.

Unfortunately, this only works with v1 right now; and supporting
it for v2 would not be as easy... (I wouldn't consider it a priority,
either).

--------8<------
Subject: [PATCH] v1: allow upgrading indexlevel=basic to 'medium' or 'full'

For v1 repos, we don't need to write any metadata to Xapian
and changing from 'basic' to 'medium' or 'full' will work.

For v2, the metadata for indexing is stored in msgmap (because
the Xapian databases are partitioned for parallelism), so a
reindex is required.
---
 lib/PublicInbox/SearchIdx.pm |  5 +++--
 t/v1reindex.t                | 21 ++++++++++++++++++++-
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index bb60506..1d259a8 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -27,6 +27,8 @@ use constant {
 	DEBUG => !!$ENV{DEBUG},
 };
 
+my $xapianlevels = qr/\A(?:full|medium)\z/;
+
 my %GIT_ESC = (
 	a => "\a",
 	b => "\b",
@@ -365,7 +367,6 @@ sub add_xapian ($$$$$) {
 sub add_message {
 	# mime = Email::MIME object
 	my ($self, $mime, $bytes, $num, $oid, $mid0) = @_;
-	my $xapianlevels = qr/\A(?:full|medium)\z/;
 	my $mids = mids($mime->header_obj);
 	$mid0 = $mids->[0] unless defined $mid0; # v1 compatibility
 	unless (defined $num) { # v1
@@ -714,7 +715,7 @@ sub _index_sync {
 			}
 			$dbh->commit;
 		}
-		if ($mkey && $newest) {
+		if ($mkey && $newest && $self->{indexlevel} =~ $xapianlevels) {
 			my $cur = $xdb->get_metadata($mkey);
 			if (need_update($self, $cur, $newest)) {
 				$xdb->set_metadata($mkey, $newest);
diff --git a/t/v1reindex.t b/t/v1reindex.t
index d97938d..75380f0 100644
--- a/t/v1reindex.t
+++ b/t/v1reindex.t
@@ -124,9 +124,10 @@ $rw = PublicInbox::SearchIdx->new($ibx, 1);
 	ok(-d $xap, 'Xapian directories recreated');
 	delete $ibx->{mm};
 	is_deeply([ $ibx->mm->minmax ], $minmax, 'minmax unchanged');
+	my $mset = $ibx->search->query('hello world', {mset=>1});
+	isnt(0, $mset->size, 'got Xapian search results');
 }
 
-
 ok(unlink "$mainrepo/public-inbox/msgmap.sqlite3", 'remove msgmap');
 remove_tree($xap);
 ok(!-d $xap, 'Xapian directories removed again');
@@ -144,7 +145,25 @@ $rw = PublicInbox::SearchIdx->new($ibx, 1);
 	ok(-d $xap, 'Xapian directories recreated');
 	delete $ibx->{mm};
 	is_deeply([ $ibx->mm->minmax ], $minmax, 'minmax unchanged');
+	my $mset = $ibx->search->reopen->query('hello world', {mset=>1});
+	is(0, $mset->size, "no Xapian search results");
 }
 
+# upgrade existing basic to medium
+# note: changing indexlevels is not yet supported in v2,
+# and may not be without more effort
+$ibx_config->{indexlevel} = 'medium';
+$ibx = PublicInbox::Inbox->new($ibx_config);
+$rw = PublicInbox::SearchIdx->new($ibx, 1);
+# no removals
+{
+	my @warn;
+	local $SIG{__WARN__} = sub { push @warn, @_ };
+	eval { $rw->index_sync };
+	is($@, '', 'no error from indexing');
+	is_deeply(\@warn, [], 'no warnings');
+	my $mset = $ibx->search->reopen->query('hello world', {mset=>1});
+	isnt(0, $mset->size, 'search OK after basic -> medium');
+}
 
 done_testing();

  reply index

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-17 23:27 [PATCH 0/3] Making the search indexes optional ebiederm
2018-07-17 23:30 ` [PATCH 1/3] SearchIdx.pm: Make indexing search positions optional Eric W. Biederman
2018-07-17 23:30 ` [PATCH 2/3] SearchIdx: Add the mechanism for making all Xapian indexing optional Eric W. Biederman
2018-07-17 23:30 ` [PATCH 3/3] SearchIdx: Allow the amount of indexing be configured Eric W. Biederman
2018-07-18 10:22   ` Eric Wong
2018-07-18 16:00     ` ebiederm
2018-07-18 16:31       ` Eric Wong
2018-07-18 16:52         ` [PATCH v2 1/3] Making the search indexes optional ebiederm
2018-07-18 16:53           ` [PATCH v2 1/3] SearchIdx.pm: Make indexing search positions optional Eric W. Biederman
2018-07-18 16:53           ` [PATCH v2 2/3] SearchIdx: Add the mechanism for making all Xapian indexing optional Eric W. Biederman
2018-07-18 16:53           ` [PATCH v2 3/3] SearchIdx: Allow the amount of indexing be configured Eric W. Biederman
2018-07-19 21:51             ` [PATCH] tests: fixup indexlevel setting in tests Eric Wong
2018-07-18 17:32           ` [PATCH v2 3/4] public-inbox-init: Initialize indexlevel ebiederm
2018-07-19  3:52           ` [PATCH v2 1/3] Making the search indexes optional Eric Wong
2018-07-19 18:47             ` ebiederm
2018-07-20  6:58               ` Eric Wong [this message]
2018-07-18 10:17 ` [PATCH 0/3] " Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180720065845.fho2shm2tjgacq3l@whir \
    --to=e@80x24.org \
    --cc=ebiederm@xmission.com \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox