* [PATCH] workaround Xapian OFD locks w/o close-on-exec
@ 2018-05-24 8:32 7% Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2018-05-24 8:32 UTC (permalink / raw)
To: meta
Xapian v1.2.21..v1.2.24 (inclusive) use OFD locks but failed to
set the close-on-exec flag on those locks. So we must continue
to work around those old versions by ensuring Xapian file
descriptors aren't held any longer than necessary when in
long-running git processes.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
---
lib/PublicInbox/SearchIdx.pm | 2 ++
t/convert-compact.t | 5 ++++-
t/v2mirror.t | 2 +-
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 74f9267..107cd34 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -666,6 +666,7 @@ sub _index_sync {
$self->{over}->rollback_lazy;
$self->{over}->disconnect;
+ $git->cleanup;
delete $self->{txn};
$xdb->cancel_transaction;
$xdb = _xdb_release($self);
@@ -696,6 +697,7 @@ sub _index_sync {
}
}
$self->commit_txn_lazy;
+ $git->cleanup;
$xdb = _xdb_release($self);
# let another process do some work... <
if (!$newest) {
diff --git a/t/convert-compact.t b/t/convert-compact.t
index e923200..def0343 100644
--- a/t/convert-compact.t
+++ b/t/convert-compact.t
@@ -40,7 +40,10 @@ ok($im->add($mime), 'added one message');
ok($im->remove($mime), 'remove message');
ok($im->add($mime), 'added message again');
$im->done;
-PublicInbox::SearchIdx->new($ibx, 1)->index_sync;
+for (1..2) {
+ eval { PublicInbox::SearchIdx->new($ibx, 1)->index_sync; };
+ is($@, '', 'no errors syncing');
+}
is(((stat("$ibx->{mainrepo}/public-inbox"))[2]) & 07777, 0755,
'sharedRepository respected for v1');
diff --git a/t/v2mirror.t b/t/v2mirror.t
index 9e0c9e1..4cd369b 100644
--- a/t/v2mirror.t
+++ b/t/v2mirror.t
@@ -50,7 +50,7 @@ for my $i (1..9) {
$mime->header_set('Subject', "subject = $i");
ok($v2w->add($mime), "add msg $i OK");
}
-$v2w->barrier;
+$v2w->done;
my %opts = (
LocalAddr => '127.0.0.1',
--
EW
^ permalink raw reply related [relevance 7%]
* Re: [PATCH 0/4] test fixes for latest CPAN modules
@ 2018-05-24 8:41 5% ` Eric Wong
2018-05-24 15:15 0% ` Konstantin Ryabitsev
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2018-05-24 8:41 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Sorry I missed your reply! Yes, you're correct -- applying the above patch
> makes all tests pass. I'm guessing it's also the reason why I'm seeing the
> "already locked" error in my other tests, too:
>
> $ public-inbox-index /var/lib/public-inbox/meta.git
> Exception: Unable to get write lock on
> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked
OK, that Xapian bug should be worked around in
https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/
And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
("workaround Xapian OFD locks w/o close-on-exec")
Thanks for the testing!
^ permalink raw reply [relevance 5%]
* Re: [PATCH 0/4] test fixes for latest CPAN modules
2018-05-24 8:41 5% ` Eric Wong
@ 2018-05-24 15:15 0% ` Konstantin Ryabitsev
2018-05-30 11:57 0% ` Eric Wong
0 siblings, 1 reply; 4+ results
From: Konstantin Ryabitsev @ 2018-05-24 15:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
[-- Attachment #1.1: Type: text/plain, Size: 1758 bytes --]
On 05/24/18 04:41, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
>> Sorry I missed your reply! Yes, you're correct -- applying the above patch
>> makes all tests pass. I'm guessing it's also the reason why I'm seeing the
>> "already locked" error in my other tests, too:
>>
>> $ public-inbox-index /var/lib/public-inbox/meta.git
>> Exception: Unable to get write lock on
>> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked
>
> OK, that Xapian bug should be worked around in
> https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/
>
> And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
> ("workaround Xapian OFD locks w/o close-on-exec")
Hmm... I'm still getting the same error, though, if I try to re-run
public-inbox-index:
$ git clone --mirror https://public-inbox.org/meta
Cloning into bare repository 'meta.git'...
remote: Counting objects: 7524, done.
remote: Compressing objects: 100% (164/164), done.
remote: Total 7524 (delta 52), reused 0 (delta 0)
Receiving objects: 100% (7524/7524), 3.80 MiB | 13.91 MiB/s, done.
Resolving deltas: 100% (3770/3770), done.
$ export PI_CONFIG=/etc/public-inbox/config
$ public-inbox-index /var/lib/public-inbox/meta.git
The above succeeds, but if I try to run it again right away:
$ public-inbox-index /var/lib/public-inbox/meta.git
Exception: Unable to get write lock on
/var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked
Is this easier solved by upgrading to the version of xapian > 1.2.22? I
can probably nag the Epel packager to do that if it's a legitimate bug
in Xapian.
Best,
--
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [relevance 0%]
* Re: [PATCH 0/4] test fixes for latest CPAN modules
2018-05-24 15:15 0% ` Konstantin Ryabitsev
@ 2018-05-30 11:57 0% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2018-05-30 11:57 UTC (permalink / raw)
To: Konstantin Ryabitsev; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On 05/24/18 04:41, Eric Wong wrote:
> > OK, that Xapian bug should be worked around in
> > https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/
> >
> > And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
> > ("workaround Xapian OFD locks w/o close-on-exec")
>
> Hmm... I'm still getting the same error, though, if I try to re-run
> public-inbox-index:
>
> $ git clone --mirror https://public-inbox.org/meta
<snip>
> $ public-inbox-index /var/lib/public-inbox/meta.git
>
> The above succeeds, but if I try to run it again right away:
>
> $ public-inbox-index /var/lib/public-inbox/meta.git
> Exception: Unable to get write lock on
> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked
Odd, I tried that exact sequence with the v1.2.22 tag in
xapian.git on my Debian machine to reproduce the problem
and make that patch.
> Is this easier solved by upgrading to the version of xapian > 1.2.22? I
> can probably nag the Epel packager to do that if it's a legitimate bug
> in Xapian.
Maybe... I wonder if there's other patches in the Epel package
or other system-level differences causing this.
^ permalink raw reply [relevance 0%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2018-05-11 19:20 [PATCH 0/4] test fixes for latest CPAN modules Eric Wong
2018-05-11 19:27 ` Konstantin Ryabitsev
2018-05-11 19:38 ` Eric Wong
2018-05-11 19:50 ` Konstantin Ryabitsev
2018-05-16 5:12 ` Eric Wong
2018-05-23 21:51 ` Konstantin Ryabitsev
2018-05-24 8:41 5% ` Eric Wong
2018-05-24 15:15 0% ` Konstantin Ryabitsev
2018-05-30 11:57 0% ` Eric Wong
2018-05-24 8:32 7% [PATCH] workaround Xapian OFD locks w/o close-on-exec Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).