* [PATCH] searchidx: v1: fix retries when Xapian and Msgmap are out-of-sync
@ 2020-06-05 2:01 6% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-06-05 2:01 UTC (permalink / raw)
To: meta
We forcibly stop git-log here, so erroring out on git-log close
failures is wrong since it sees SIGPIPE. Noticed while
reindexing a large v1 inbox for IMAP changes.
Fixes: b32b47fb12a3043d ("index: "git log" failures are fatal")
---
lib/PublicInbox/SearchIdx.pm | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 5c161b9a..f7462aa7 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -754,10 +754,7 @@ sub _index_sync {
my $xdb = $self->begin_txn_lazy;
my $mm = _msgmap_init($self);
do {
- if ($xlog) {
- close($xlog) or die "git log failed: \$?=$?";
- $xlog = undef;
- }
+ $xlog = undef; # stop previous git-log via SIGPIPE
$last_commit = _last_x_commit($self, $mm);
$lx = reindex_from($opts->{reindex}, $last_commit);
^ permalink raw reply related [relevance 6%]
* [PATCH] index: "git log" failures are fatal
@ 2019-11-04 0:43 7% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-11-04 0:43 UTC (permalink / raw)
To: meta
While I've never seen "git log" fail on its own, it could happen
one day and we should be prepared to abort indexing when it
happens.
Beef up tests for t/spawn.t to ensure close() behaves
on popen_rd the way we expect it to.
---
lib/PublicInbox/SearchIdx.pm | 8 ++++++--
lib/PublicInbox/V2Writable.pm | 5 +++--
t/spawn.t | 4 ++--
3 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index b2d71a1f..f265fa7f 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -590,6 +590,7 @@ sub read_log {
$newest ||= $latest;
}
}
+ close($log) or die "git log failed: \$?=$?";
# get the leftovers
foreach my $blob (keys %D) {
my $mime = do_cat_mail($git, $blob, \$bytes) or next;
@@ -632,7 +633,7 @@ sub _git_log {
--no-notes --no-color --no-renames
--diff-filter=AM), $range);
++$fcount while <$fh>;
- close $fh;
+ close $fh or die "git log failed: \$?=$?";
my $high = $self->{mm}->num_highwater;
$pr->("$fcount\n") if $pr; # continue previous line
$self->{ntodo} = $fcount;
@@ -713,7 +714,10 @@ sub _index_sync {
my $xdb = $self->begin_txn_lazy;
my $mm = _msgmap_init($self);
do {
- $xlog = undef;
+ if ($xlog) {
+ close($xlog) or die "git log failed: \$?=$?";
+ $xlog = undef;
+ }
$last_commit = _last_x_commit($self, $mm);
$lx = reindex_from($opts->{reindex}, $last_commit);
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 1825da2c..02f313b0 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -1128,6 +1128,7 @@ sub sync_prepare ($$$) {
--no-notes --no-color --no-renames
--diff-filter=AM), $range, '--', 'm');
++$n while <$fh>;
+ close $fh or die "git log failed: \$?=$?";
$pr->("$n\n") if $pr;
$regen_max += $n;
}
@@ -1195,7 +1196,7 @@ sub unindex ($$$$) {
unindex_oid($self, $git, $1, $unindexed);
}
delete $self->{reindex_pipe};
- $fh = undef;
+ close $fh or die "git log failed: \$?=$?";
return unless $sync->{-opt}->{prune};
my $after = scalar keys %$unindexed;
@@ -1251,7 +1252,7 @@ sub index_epoch ($$$) {
mark_deleted($self, $sync, $git, $1);
}
}
- $fh = undef;
+ close $fh or die "git log failed: \$?=$?";
delete $self->{reindex_pipe};
update_last_commit($self, $git, $i, $cmt) if defined $cmt;
}
diff --git a/t/spawn.t b/t/spawn.t
index aba1a26c..ebebfb57 100644
--- a/t/spawn.t
+++ b/t/spawn.t
@@ -71,13 +71,13 @@ use PublicInbox::Spawn qw(which spawn popen_rd);
is($buf, "hello\n", 'tied gets works');
is(sysread($fh, $buf, 6), 0, 'sysread got EOF');
$? = 1;
- close $fh;
+ ok(close($fh), 'close succeeds');
is($?, 0, '$? set properly');
}
{
my $fh = popen_rd([qw(false)]);
- close $fh;
+ ok(!close($fh), 'close fails on false');
isnt($?, 0, '$? set properly: '.$?);
}
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-11-04 0:43 7% [PATCH] index: "git log" failures are fatal Eric Wong
2020-06-05 2:01 6% [PATCH] searchidx: v1: fix retries when Xapian and Msgmap are out-of-sync Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).