From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 5/5] extindex: fix w/ Xapian 1.2.21..1.2.24
Date: Sun, 17 Jan 2021 07:09:59 +0000 [thread overview]
Message-ID: <20210117070959.16826-6-e@80x24.org> (raw)
In-Reply-To: <20210117070959.16826-1-e@80x24.org>
From: Eric Wong <e@yhbt.net>
Xapian v1.2.21..v1.2.24 failed to set the close-on-exec flag
on the flintlock FD, causing "git cat-file" processes to
hold onto the lock and prevent subsequent Xapian::WritableDatabase
from locking the DB. So cleanup git processes after committing
the miscidx transaction.
---
lib/PublicInbox/ExtSearchIdx.pm | 1 +
lib/PublicInbox/SearchIdx.pm | 5 ++++-
lib/PublicInbox/V2Writable.pm | 12 ++++++++++--
3 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index 85959a95..c782a62a 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -1003,6 +1003,7 @@ sub idx_init { # similar to V2Writable
$self->with_umask(\&_idx_init, $self, $opt);
$self->{oidx}->begin_lazy;
$self->{oidx}->eidx_prep;
+ $self->git->batch_prepare;
$self->{midx}->begin_txn;
}
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index adced076..7f7b980d 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -105,8 +105,11 @@ sub load_xapian_writable () {
$DB_CREATE_OR_OPEN = eval($xap.'::DB_CREATE_OR_OPEN()');
$DB_OPEN = eval($xap.'::DB_OPEN()');
my $ver = (eval($xap.'::major_version()') << 16) |
- (eval($xap.'::minor_version()') << 8);
+ (eval($xap.'::minor_version()') << 8) |
+ eval($xap.'::revision()');
$DB_NO_SYNC = 0x4 if $ver >= 0x10400;
+ # Xapian v1.2.21..v1.2.24 were missing close-on-exec on OFD locks
+ $X->{CLOEXEC_UNSET} = 1 if $ver >= 0x010215 && $ver <= 0x010218;
1;
}
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 54004fd7..0104f87a 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -17,6 +17,7 @@ use PublicInbox::InboxWritable;
use PublicInbox::OverIdx;
use PublicInbox::Msgmap;
use PublicInbox::Spawn qw(spawn popen_rd run_die);
+use PublicInbox::Search;
use PublicInbox::SearchIdx qw(log2stack is_ancestor check_size is_bad_blob);
use IO::Handle; # ->autoflush
use File::Temp ();
@@ -608,7 +609,11 @@ shard[$i] bad echo:$echo != $i waiting for txn commit
}
my $midx = $self->{midx}; # misc index
- $midx->commit_txn if $midx;
+ if ($midx) {
+ $midx->commit_txn;
+ $PublicInbox::Search::X{CLOEXEC_UNSET} and
+ $self->git->cleanup;
+ }
# last_commit is special, don't commit these until
# Xapian shards are done:
@@ -618,7 +623,10 @@ shard[$i] bad echo:$echo != $i waiting for txn commit
$dbh->commit;
$dbh->begin_work;
}
- $midx->begin_txn if $midx;
+ if ($midx) {
+ $self->git->batch_prepare;
+ $midx->begin_txn;
+ }
}
$self->{total_bytes} += $self->{transact_bytes};
$self->{transact_bytes} = 0;
prev parent reply other threads:[~2021-01-17 7:10 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-17 7:09 [PATCH 0/5] fixes for older Perls and Xapian Eric Wong
2021-01-17 7:09 ` [PATCH 1/5] address: pairs: enable pure Perl version Eric Wong
2021-01-17 7:09 ` [PATCH 2/5] initialize scalar for `vec' perlop modification Eric Wong
2021-01-17 7:09 ` [PATCH 3/5] t/thread-index-gap: support older DBI Eric Wong
2021-01-17 7:09 ` [PATCH 4/5] t/shared_kv: workaround old File::Spec Eric Wong
2021-01-17 7:09 ` Eric Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210117070959.16826-6-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).