diff options
author | Eric Wong <e@80x24.org> | 2021-07-25 00:11:01 +0000 |
---|---|---|
committer | Eric Wong <e@80x24.org> | 2021-07-25 00:19:19 +0000 |
commit | 613a1e038f9bd7b4c9287bda621b6e19eac24684 (patch) | |
tree | aa89c153ed541bc962a590b15c8f0a186e44c68e /script | |
parent | 3416172dbe6d05cc3272829d5448323cea3c8961 (diff) | |
download | public-inbox-613a1e038f9bd7b4c9287bda621b6e19eac24684.tar.gz |
Sometimes I just want to dedupe a single Message-ID to test something, and this lets me do it. This patch appears to do what its supposed to. But it also appears to be finding duplicates that were previously missed. That's a good thing, but I wish I understood what seems to be fixed :x I'm not sure why the previous ExtSearchIdx.pm (blob 357312b8) was causing messages to be missed, even, and why this patch seems to fix it... And it's not infinite looping, either. Anyways, before this patch, "-extindex --dedupe" was taking ~5 min to no-op every message (after the initial full --dedupe run which took over a day to run). No-op --dedupes now take just under 2 hours to scan every single cross-posted message for a no-op dedupe. The initial dedupe took nearly 44 hours on my system for <https://yhbt.net/lore/all/> due to SATA-2 TLC SSD latency on 3 gigantic Xapian shards. Running --dedupe with this change seems to prevent /BUG\?.*?not deduplicated properly/ stderr messages from being triggered by View.pm. Current versions of -extindex do not seem susceptible to introducing duplicates.
Diffstat (limited to 'script')
-rwxr-xr-x | script/public-inbox-extindex | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/script/public-inbox-extindex b/script/public-inbox-extindex index dcb12e5a..addd5ac6 100755 --- a/script/public-inbox-extindex +++ b/script/public-inbox-extindex @@ -17,7 +17,7 @@ usage: public-inbox-extindex [options] [EXTINDEX_DIR] [INBOX_DIR...] --batch-size=BYTES flush changes to OS after a given number of bytes --max-size=BYTES do not index messages larger than the given size --gc perform garbage collection instead of indexing - --dedupe fix prior deduplication errors + --dedupe[=MSGID] fix prior deduplication errors (may be repeated) --verbose | -v increase verbosity (may be repeated) --dry-run | -n dry-run on --dedupe @@ -29,7 +29,7 @@ GetOptions($opt, qw(verbose|v+ reindex rethread compact|c+ jobs|j=i fsync|sync! indexlevel|index-level|L=s max_size|max-size=s batch_size|batch-size=s - dedupe gc commit-interval=i watch scan! dry-run|n + dedupe:s@ gc commit-interval=i watch scan! dry-run|n all help|h)) or die $help; if ($opt->{help}) { print $help; exit 0 }; |