user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH] workaround Xapian OFD locks w/o close-on-exec
@ 2018-05-24  8:32  7% Eric Wong
  0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2018-05-24  8:32 UTC (permalink / raw)
  To: meta

Xapian v1.2.21..v1.2.24 (inclusive) use OFD locks but failed to
set the close-on-exec flag on those locks.  So we must continue
to work around those old versions by ensuring Xapian file
descriptors aren't held any longer than necessary when in
long-running git processes.

Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
---
 lib/PublicInbox/SearchIdx.pm | 2 ++
 t/convert-compact.t          | 5 ++++-
 t/v2mirror.t                 | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 74f9267..107cd34 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -666,6 +666,7 @@ sub _index_sync {
 
 		$self->{over}->rollback_lazy;
 		$self->{over}->disconnect;
+		$git->cleanup;
 		delete $self->{txn};
 		$xdb->cancel_transaction;
 		$xdb = _xdb_release($self);
@@ -696,6 +697,7 @@ sub _index_sync {
 			}
 		}
 		$self->commit_txn_lazy;
+		$git->cleanup;
 		$xdb = _xdb_release($self);
 		# let another process do some work... <
 		if (!$newest) {
diff --git a/t/convert-compact.t b/t/convert-compact.t
index e923200..def0343 100644
--- a/t/convert-compact.t
+++ b/t/convert-compact.t
@@ -40,7 +40,10 @@ ok($im->add($mime), 'added one message');
 ok($im->remove($mime), 'remove message');
 ok($im->add($mime), 'added message again');
 $im->done;
-PublicInbox::SearchIdx->new($ibx, 1)->index_sync;
+for (1..2) {
+	eval { PublicInbox::SearchIdx->new($ibx, 1)->index_sync; };
+	is($@, '', 'no errors syncing');
+}
 
 is(((stat("$ibx->{mainrepo}/public-inbox"))[2]) & 07777, 0755,
 	'sharedRepository respected for v1');
diff --git a/t/v2mirror.t b/t/v2mirror.t
index 9e0c9e1..4cd369b 100644
--- a/t/v2mirror.t
+++ b/t/v2mirror.t
@@ -50,7 +50,7 @@ for my $i (1..9) {
 	$mime->header_set('Subject', "subject = $i");
 	ok($v2w->add($mime), "add msg $i OK");
 }
-$v2w->barrier;
+$v2w->done;
 
 my %opts = (
 	LocalAddr => '127.0.0.1',
-- 
EW


^ permalink raw reply related	[relevance 7%]

* Re: [PATCH 0/4] test fixes for latest CPAN modules
  @ 2018-05-24  8:41  5%           ` Eric Wong
  2018-05-24 15:15  0%             ` Konstantin Ryabitsev
  0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2018-05-24  8:41 UTC (permalink / raw)
  To: Konstantin Ryabitsev; +Cc: meta

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Sorry I missed your reply! Yes, you're correct -- applying the above patch
> makes all tests pass. I'm guessing it's also the reason why I'm seeing the
> "already locked" error in my other tests, too:
> 
> $ public-inbox-index /var/lib/public-inbox/meta.git
> Exception: Unable to get write lock on
> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked

OK, that Xapian bug should be worked around in
https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/

And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
("workaround Xapian OFD locks w/o close-on-exec")

Thanks for the testing!

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/4] test fixes for latest CPAN modules
  2018-05-24  8:41  5%           ` Eric Wong
@ 2018-05-24 15:15  0%             ` Konstantin Ryabitsev
  2018-05-30 11:57  0%               ` Eric Wong
  0 siblings, 1 reply; 4+ results
From: Konstantin Ryabitsev @ 2018-05-24 15:15 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta


[-- Attachment #1.1: Type: text/plain, Size: 1758 bytes --]

On 05/24/18 04:41, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
>> Sorry I missed your reply! Yes, you're correct -- applying the above patch
>> makes all tests pass. I'm guessing it's also the reason why I'm seeing the
>> "already locked" error in my other tests, too:
>>
>> $ public-inbox-index /var/lib/public-inbox/meta.git
>> Exception: Unable to get write lock on
>> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked
> 
> OK, that Xapian bug should be worked around in
> https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/
> 
> And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
> ("workaround Xapian OFD locks w/o close-on-exec")

Hmm... I'm still getting the same error, though, if I try to re-run
public-inbox-index:

$ git clone --mirror https://public-inbox.org/meta
Cloning into bare repository 'meta.git'...
remote: Counting objects: 7524, done.
remote: Compressing objects: 100% (164/164), done.
remote: Total 7524 (delta 52), reused 0 (delta 0)
Receiving objects: 100% (7524/7524), 3.80 MiB | 13.91 MiB/s, done.
Resolving deltas: 100% (3770/3770), done.
$ export PI_CONFIG=/etc/public-inbox/config
$ public-inbox-index /var/lib/public-inbox/meta.git

The above succeeds, but if I try to run it again right away:

$ public-inbox-index /var/lib/public-inbox/meta.git
Exception: Unable to get write lock on
/var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked

Is this easier solved by upgrading to the version of xapian > 1.2.22? I
can probably nag the Epel packager to do that if it's a legitimate bug
in Xapian.

Best,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[relevance 0%]

* Re: [PATCH 0/4] test fixes for latest CPAN modules
  2018-05-24 15:15  0%             ` Konstantin Ryabitsev
@ 2018-05-30 11:57  0%               ` Eric Wong
  0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2018-05-30 11:57 UTC (permalink / raw)
  To: Konstantin Ryabitsev; +Cc: meta

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On 05/24/18 04:41, Eric Wong wrote:
> > OK, that Xapian bug should be worked around in
> > https://public-inbox.org/meta/20180524083216.30440-1-e@80x24.org/
> > 
> > And pushed as commit c339a14b11c57e7b330f4efef6903bcbea2b3463
> > ("workaround Xapian OFD locks w/o close-on-exec")
> 
> Hmm... I'm still getting the same error, though, if I try to re-run
> public-inbox-index:
> 
> $ git clone --mirror https://public-inbox.org/meta

<snip>

> $ public-inbox-index /var/lib/public-inbox/meta.git
> 
> The above succeeds, but if I try to run it again right away:
> 
> $ public-inbox-index /var/lib/public-inbox/meta.git
> Exception: Unable to get write lock on
> /var/lib/public-inbox/meta.git/public-inbox/xapian15: already locked

Odd, I tried that exact sequence with the v1.2.22 tag in
xapian.git on my Debian machine to reproduce the problem
and make that patch.

> Is this easier solved by upgrading to the version of xapian > 1.2.22? I
> can probably nag the Epel packager to do that if it's a legitimate bug
> in Xapian.

Maybe... I wonder if there's other patches in the Epel package
or other system-level differences causing this.

^ permalink raw reply	[relevance 0%]

Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2018-05-11 19:20     [PATCH 0/4] test fixes for latest CPAN modules Eric Wong
2018-05-11 19:27     ` Konstantin Ryabitsev
2018-05-11 19:38       ` Eric Wong
2018-05-11 19:50         ` Konstantin Ryabitsev
2018-05-16  5:12           ` Eric Wong
2018-05-23 21:51             ` Konstantin Ryabitsev
2018-05-24  8:41  5%           ` Eric Wong
2018-05-24 15:15  0%             ` Konstantin Ryabitsev
2018-05-30 11:57  0%               ` Eric Wong
2018-05-24  8:32  7% [PATCH] workaround Xapian OFD locks w/o close-on-exec Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).