Re: [WIP] extindex: support --dedupe[=MSGID]

[Eric Wong <e@80x24.org>]

Eric Wong <e@80x24.org> wrote:
> Anyways, before this patch, "-extindex --dedupe" was taking ~5
> min to no-op every message (after the initial full --dedupe run
> which took over a day to run).  Now that it appears to be doing
> many more dedupes, so it could take 40-ish hours to finish the
> dedupe run on the machine running <https://yhbt.net/lore/all/>.

Nearly 44 hours (on a busy system), but that was only because of
a bug which caused half the work to get skipped (fix below).

Xapian glass (1.4.x) really falls down with large shards and
making -xcpdb work with extindex to reshard into more shards
will be required to maintain performance as an extindex grows...

diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index 442ded46..2311161e 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -895,12 +895,13 @@ sub eidx_dedupe ($$$) {
 	return unless eidxq_lock_acquire($self);
 	my ($iter, $cur_mid);
 	my $min_id = 0;
+	my $idx = 0;
 	local $sync->{-regen_fmt} = "dedupe %u/".$self->{oidx}->max."\n";
 
 	# note: we could write this query more intelligently,
 	# but that causes lock contention with read-only processes
 dedupe_restart:
-	$cur_mid = shift @$msgids;
+	$cur_mid = $msgids->[$idx];
 	if ($cur_mid eq '') { # all Message-IDs
 		$iter = $self->{oidx}->dbh->prepare(<<EOS);
 SELECT mid,id FROM msgid WHERE id > ? ORDER BY id ASC
@@ -945,7 +946,7 @@ EOS
 			goto dedupe_restart;
 		}
 	}
-	goto dedupe_restart if scalar(@$msgids);
+	goto dedupe_restart if defined($msgids->[++$idx]);
 
 	my $n = delete $sync->{dedupe_cull};
 	if (my $pr = $sync->{-opt}->{-progress}) {