user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 7/9] convert: fix --no-index switch
    2020-02-02  6:52  5% ` [PATCH 1/9] t/multi-mid.t: extra test for -convert highwater mark Eric Wong
@ 2020-02-02  6:52  6% ` Eric Wong
  1 sibling, 0 replies; 3+ results
From: Eric Wong @ 2020-02-02  6:52 UTC (permalink / raw)
  To: meta

The (currently undocumented) "--no-index" flag did not trigger
the V2Writable->done call necessary to make the import
successful.

Fixes: eea47b676127bcdb ("convert: preserve highwater mark from v1 msgmap")
---
 script/public-inbox-convert | 6 ++----
 t/convert-compact.t         | 3 +++
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/script/public-inbox-convert b/script/public-inbox-convert
index 281313e0..68b0b31a 100755
--- a/script/public-inbox-convert
+++ b/script/public-inbox-convert
@@ -151,7 +151,5 @@ if (my $mm = $old->mm) {
 	chop(my $cmt = $epoch0->qx(qw(rev-parse --verify), $head));
 	$v2w->last_epoch_commit(0, $cmt);
 }
-if ($index) {
-	$v2w->index_sync({reindex => 1});
-	$v2w->done;
-}
+$v2w->index_sync({reindex => 1}) if $index;
+$v2w->done;
diff --git a/t/convert-compact.t b/t/convert-compact.t
index fc46083b..f56d7817 100644
--- a/t/convert-compact.t
+++ b/t/convert-compact.t
@@ -76,6 +76,9 @@ my $hwm = do {
 };
 ok(defined($hwm) && $hwm > 0, "highwater mark set #$hwm");
 
+$cmd = [ '-convert', '--no-index', $ibx->{inboxdir}, "$tmpdir/no-index" ];
+ok(run_script($cmd, undef, $rdr), 'convert --no-index works');
+
 $cmd = [ '-convert', $ibx->{inboxdir}, "$tmpdir/v2" ];
 ok(run_script($cmd, undef, $rdr), 'convert works');
 @xdir = glob("$tmpdir/v2/xap*/*");

^ permalink raw reply related	[relevance 6%]

* [PATCH 1/9] t/multi-mid.t: extra test for -convert highwater mark
  @ 2020-02-02  6:52  5% ` Eric Wong
  2020-02-02  6:52  6% ` [PATCH 7/9] convert: fix --no-index switch Eric Wong
  1 sibling, 0 replies; 3+ results
From: Eric Wong @ 2020-02-02  6:52 UTC (permalink / raw)
  To: meta

This is derived from a real-world test case where I encounterd
multiple Message-IDs in a v1 inbox causing regen problems.

Fixes: eea47b676127bcdb ("convert: preserve highwater mark from v1 msgmap")
---
 MANIFEST      |  1 +
 t/multi-mid.t | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
 create mode 100644 t/multi-mid.t

diff --git a/MANIFEST b/MANIFEST
index 3736c777..5eb5d53a 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -250,6 +250,7 @@ t/mime.t
 t/msg_iter.t
 t/msgmap.t
 t/msgtime.t
+t/multi-mid.t
 t/nntp.t
 t/nntpd-tls.t
 t/nntpd.t
diff --git a/t/multi-mid.t b/t/multi-mid.t
new file mode 100644
index 00000000..0d283a0d
--- /dev/null
+++ b/t/multi-mid.t
@@ -0,0 +1,61 @@
+# Copyright (C) 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use Test::More;
+use PublicInbox::MIME;
+use PublicInbox::TestCommon;
+use PublicInbox::InboxWritable;
+use PublicInbox::SearchIdx;
+require_git(2.6);
+require_mods(qw(DBD::SQLite));
+
+my $addr = 'test@example.com';
+my $bad = PublicInbox::MIME->new(<<EOF);
+Message-ID: <a\@example.com>
+Message-ID: <b\@example.com>
+From: a\@example.com
+To: $addr
+Date: Fri, 02 Oct 1993 00:00:00 +0000
+Subject: bad
+
+EOF
+
+my $good = PublicInbox::MIME->new(<<EOF);
+Message-ID: <b\@example.com>
+Date: Fri, 02 Oct 1993 00:00:00 +0000
+From: b\@example.com
+To: $addr
+Subject: good
+
+EOF
+
+for my $order ([$bad, $good], [$good, $bad]) {
+	my $before;
+	my ($tmpdir, $for_destroy) = tmpdir();
+	my $ibx = PublicInbox::InboxWritable->new({
+		inboxdir => "$tmpdir/v1",
+		name => 'test-v1',
+		indexlevel => 'basic',
+		-primary_address => $addr,
+	}, my $creat_opt = {});
+	if ('setup v1 inbox') {
+		my $im = $ibx->importer(0);
+		ok($im->add($_), 'added '.$_->header('Subject')) for @$order;
+		$im->done;
+		my $s = PublicInbox::SearchIdx->new($ibx, 1);
+		$s->index_sync;
+		$before = [ $ibx->mm->minmax ];
+		$ibx->cleanup;
+	}
+	my $rdr = { 1 => \(my $out = ''), 2 => \(my $err = '') };
+	my $cmd = [ '-convert', $ibx->{inboxdir}, "$tmpdir/v2" ];
+	ok(run_script($cmd, undef, $rdr),
+		'convert to v2');
+	$err =~ s!\AW: $tmpdir/v1 not configured[^\n]+\n!!s;
+	is($err, '', 'no errors or warnings from -convert');
+	$ibx->{version} = 2;
+	$ibx->{inboxdir} = "$tmpdir/v2";
+	is_deeply([$ibx->mm->minmax], $before,
+		'min, max article numbers unchanged');
+}
+
+done_testing();

^ permalink raw reply related	[relevance 5%]

* [PATCH] convert: preserve highwater mark from v1 msgmap
@ 2020-01-30  7:51  7% Eric Wong
  0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-01-30  7:51 UTC (permalink / raw)
  To: meta

If we're reusing the msgmap from a v1 inbox, we also need to
ensure the highwater mark doesn't get doubled in the v1->v2
conversion by internally triggering the equivalent of
"--reindex" on a fresh v2 inbox.

This was needed to convert an indexed v1 inbox which featured
messages with multiple Message-IDs in it.  Fresh, unindexed
clones of v1 inboxes would not have been affected by this.
---
 script/public-inbox-convert | 16 +++++++++++++---
 t/convert-compact.t         |  8 ++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/script/public-inbox-convert b/script/public-inbox-convert
index 56a810eb..8ac111a2 100755
--- a/script/public-inbox-convert
+++ b/script/public-inbox-convert
@@ -7,6 +7,7 @@ use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
 use PublicInbox::InboxWritable;
 use PublicInbox::Config;
 use PublicInbox::V2Writable;
+use PublicInbox::Git;
 use PublicInbox::Spawn qw(spawn);
 use Cwd 'abs_path';
 use File::Copy 'cp'; # preserves permissions:
@@ -144,10 +145,19 @@ $w = $r = undef;
 close $rd or die "close fast-export: $!\n";
 waitpid($pid, 0) or die "waitpid failed: $!\n";
 $? == 0 or die "fast-export failed: $?\n";
-my $mm = $old->mm;
-$mm->{dbh}->sqlite_backup_to_file("$new_dir/msgmap.sqlite3") if $mm;
 $v2w->done;
+if (my $mm = $old->mm) {
+	$old->cleanup;
+	$mm->{dbh}->sqlite_backup_to_file("$new_dir/msgmap.sqlite3");
+
+	# we want to trigger a reindex, not a from scratch index if
+	# we're reusing the msgmap from an existing v1 installation.
+	$v2w->idx_init;
+	my $epoch0 = PublicInbox::Git->new($v2w->git_init(0));
+	chop(my $cmt = $epoch0->qx(qw(rev-parse --verify), $head));
+	$v2w->last_epoch_commit(0, $cmt);
+}
 if ($index) {
-	$v2w->index_sync;
+	$v2w->index_sync({reindex => 1});
 	$v2w->done;
 }
diff --git a/t/convert-compact.t b/t/convert-compact.t
index 6e4a8268..fc46083b 100644
--- a/t/convert-compact.t
+++ b/t/convert-compact.t
@@ -69,6 +69,13 @@ is(scalar(@xdir), 1, 'got one xapian directory after compact');
 is(((stat($xdir[0]))[2]) & 07777, 0755,
 	'sharedRepository respected on v1 compact');
 
+my $hwm = do {
+	my $mm = $ibx->mm;
+	$ibx->cleanup;
+	$mm->num_highwater;
+};
+ok(defined($hwm) && $hwm > 0, "highwater mark set #$hwm");
+
 $cmd = [ '-convert', $ibx->{inboxdir}, "$tmpdir/v2" ];
 ok(run_script($cmd, undef, $rdr), 'convert works');
 @xdir = glob("$tmpdir/v2/xap*/*");
@@ -83,6 +90,7 @@ my $env = { NPROC => 2 };
 ok(run_script($cmd, $env, $rdr), 'v2 compact works');
 $ibx->{inboxdir} = "$tmpdir/v2";
 $ibx->{version} = 2;
+is($ibx->mm->num_highwater, $hwm, 'highwater mark unchanged in v2 inbox');
 
 @xdir = glob("$tmpdir/v2/xap*/*");
 foreach (@xdir) {

^ permalink raw reply related	[relevance 7%]

Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-01-30  7:51  7% [PATCH] convert: preserve highwater mark from v1 msgmap Eric Wong
2020-02-02  6:52     [PATCH 0/9] convert: more fixes and tests Eric Wong
2020-02-02  6:52  5% ` [PATCH 1/9] t/multi-mid.t: extra test for -convert highwater mark Eric Wong
2020-02-02  6:52  6% ` [PATCH 7/9] convert: fix --no-index switch Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).