* [PATCH 3/3] extindex: --gc: avoid SQLite lock conflict on shard cleanup
2021-07-06 12:42 5% [PATCH 0/3] extindex: dedupe support, + gc fix Eric Wong
@ 2021-07-06 12:42 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-07-06 12:42 UTC (permalink / raw)
To: meta
Xapian shard cleanup only requires read-only access to
over.sqlite3, so avoid opening it with read-write access since
create_tables will hit lock conflicts on "INSERT OR IGNORE"
statements.
---
lib/PublicInbox/SearchIdxShard.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/SearchIdxShard.pm b/lib/PublicInbox/SearchIdxShard.pm
index 1598faeb..8635f585 100644
--- a/lib/PublicInbox/SearchIdxShard.pm
+++ b/lib/PublicInbox/SearchIdxShard.pm
@@ -78,8 +78,9 @@ sub shard_close {
sub shard_over_check {
my ($self, $over) = @_;
if ($self->{-ipc_req} && $over->{dbh}) {
- # can't send DB handles over IPC
- $over = ref($over)->new($over->{dbh}->sqlite_db_filename);
+ # can't send DB handles over IPC, and use read-only to avoid
+ # create_tables lock conflict:
+ $over = PublicInbox::Over->new($over->{dbh}->sqlite_db_filename)
}
$self->ipc_do('over_check', $over);
}
^ permalink raw reply related [relevance 7%]
* [PATCH 0/3] extindex: dedupe support, + gc fix
@ 2021-07-06 12:42 5% Eric Wong
2021-07-06 12:42 7% ` [PATCH 3/3] extindex: --gc: avoid SQLite lock conflict on shard cleanup Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-07-06 12:42 UTC (permalink / raw)
To: meta
I'm still not sure how the duplicates got into my extindices;
but the problem doesn't seem reproducible at the moment so
maybe the original bug was fixed.
Since there's already dedupe failures from past indexing, the
--dedupe switch here should help us get rid of them. It's only
lightly tested, but it seems to be working.
There's also a minor fix for --gc, too.
Eric Wong (3):
eml: relax warn_ignore regexps for current Email::Address::XS
extindex: implement --dedupe to fix old extindices
extindex: --gc: avoid SQLite lock conflict on shard cleanup
lib/PublicInbox/Eml.pm | 4 +-
lib/PublicInbox/ExtSearchIdx.pm | 96 +++++++++++++++++++++++++++++++
lib/PublicInbox/OverIdx.pm | 20 +++++++
lib/PublicInbox/SearchIdxShard.pm | 5 +-
script/public-inbox-extindex | 13 ++++-
t/extsearch.t | 11 ++++
6 files changed, 142 insertions(+), 7 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-07-06 12:42 5% [PATCH 0/3] extindex: dedupe support, + gc fix Eric Wong
2021-07-06 12:42 7% ` [PATCH 3/3] extindex: --gc: avoid SQLite lock conflict on shard cleanup Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).