* [PATCH 2/9] extindex: delete stale messages from over.sqlite3
2020-12-15 2:02 6% ` [PATCH 0/9] extindex: " Eric Wong
@ 2020-12-15 2:02 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-12-15 2:02 UTC (permalink / raw)
To: meta
In addition to removing stale messages from Xapian, we must
also remove them from over.sqlite3.
---
lib/PublicInbox/ExtSearchIdx.pm | 1 +
t/extsearch.t | 12 ++++++++++++
2 files changed, 13 insertions(+)
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index 394a89d4..ec86a7c0 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -533,6 +533,7 @@ DELETE FROM xref3 WHERE ibx_id = ? AND xnum = ? AND oidbin = ?
my $xr3 = $self->{oidx}->get_xref3($docid);
my $idx = $self->idx_shard($docid);
if (scalar(@$xr3) == 0) { # all gone
+ $self->{oidx}->delete_by_num($docid);
$self->{oidx}->eidxq_del($docid);
$idx->shard_remove($docid);
} else { # enqueue for reindex of remaining messages
diff --git a/t/extsearch.t b/t/extsearch.t
index f16ec0d1..4a8a9f49 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -268,18 +268,30 @@ if ('reindex catches missed messages') {
my $new = $oidx->get_art($max + 1);
is($new->{subject}, $eml->header('Subject'), 'new message added');
+ $es->{xdb}->reopen;
+ my $mset = $es->mset("mid:$new->{mid}");
+ is($mset->size, 1, 'previously unseen, now indexed in Xapian');
+
ok($im->remove($eml), 'remove new message from v2 inbox');
$im->done;
my $cmt_c = $v2ibx->mm->last_commit_xap($schema_version, 0);
is($oidx->eidx_meta($lc_key, $cmt_c), $cmt_b,
'bump lc-v2 meta again to skip v2 remove');
$err = '';
+ $oidx->dbh_close;
ok(run_script([qw(-extindex --reindex), "$home/extindex",
$v2ibx->{inboxdir}], undef, $opt),
'--reindex for stale');
@err = split(/^/, $err);
is(scalar(@err), 1, 'only one warning');
like($err[0], qr/\(#$new->{num}\): stale/, 'got stale message warning');
+ is($oidx->get_art($new->{num}), undef,
+ 'stale message gone from over');
+ is_deeply($oidx->get_xref3($new->{num}), [],
+ 'stale message has no xref3');
+ $es->{xdb}->reopen;
+ $mset = $es->mset("mid:$new->{mid}");
+ is($mset->size, 0, 'stale mid gone Xapian');
}
if ('remove v1test and test gc') {
^ permalink raw reply related [relevance 7%]
* [PATCH 0/9] extindex: --reindex support
@ 2020-12-15 2:02 6% ` Eric Wong
2020-12-15 2:02 7% ` [PATCH 2/9] extindex: delete stale messages from over.sqlite3 Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-12-15 2:02 UTC (permalink / raw)
To: meta
Patches 1 and 2 are resends, the rest have gone through a lot of
changes and I'm probably ready to run this live on the
extindex which holds my lore mirror onion
<http://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/all/>
Eric Wong (9):
extindex: preliminary --reindex support
extindex: delete stale messages from over.sqlite3
over: sort xref3 by xnum if ibx_id repeats
extindex: support --rethread and content bifurcation
extsearchidx: reindex works on Xapian, too
extsearchidx: checkpoint releases locks
extsearchidx: simplify reindex code paths
extsearchidx: reindex releases over.sqlite3 handles properly
searchidxshard: simplify newline elimination
lib/PublicInbox/ExtSearchIdx.pm | 369 +++++++++++++++++++++++++++++-
lib/PublicInbox/Over.pm | 5 +-
lib/PublicInbox/OverIdx.pm | 23 ++
lib/PublicInbox/SearchIdx.pm | 13 +-
lib/PublicInbox/SearchIdxShard.pm | 20 +-
lib/PublicInbox/V2Writable.pm | 11 +-
t/extsearch.t | 133 ++++++++++-
7 files changed, 542 insertions(+), 32 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-11 3:37 [PATCH] extindex: preliminary --reindex support Eric Wong
2020-12-15 2:02 6% ` [PATCH 0/9] extindex: " Eric Wong
2020-12-15 2:02 7% ` [PATCH 2/9] extindex: delete stale messages from over.sqlite3 Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).