* [PATCH 4/4] lei q|up|convert: common finish_output to detect errors
2023-11-15 9:21 6% [PATCH 0/4] lei convert: support idempotent v2 outputs Eric Wong
@ 2023-11-15 9:21 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-15 9:21 UTC (permalink / raw)
To: meta
We need to consistently check the exit code of pigz|gzip|xz|bzip2
when writing to compressed mboxes (or bad storage).
---
lib/PublicInbox/LeiConvert.pm | 4 ++--
lib/PublicInbox/LeiToMail.pm | 11 +++++++++++
lib/PublicInbox/LeiXSearch.pm | 9 +--------
3 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 9d2479b0..8f628562 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -33,9 +33,9 @@ sub process_inputs { # via wq_do
local $PublicInbox::DS::in_loop = 0; # force synchronous awaitpid
$self->SUPER::process_inputs;
my $lei = $self->{lei};
- delete $lei->{1};
my $l2m = delete $lei->{l2m};
- delete $self->{wcb}; # commit
+ delete $self->{wcb}; # may close connections
+ $l2m->finish_output($lei) if $l2m;
if (my $v2w = delete $lei->{v2w}) { $v2w->done } # may die
my $nr_w = delete($l2m->{-nr_write}) // 0;
my $d = (delete($l2m->{-nr_seen}) // 0) - $nr_w;
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 0d62888d..007191bb 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -609,6 +609,17 @@ sub _pre_augment_mbox {
undef;
}
+sub finish_output {
+ my ($self, $lei) = @_;
+ my $out = delete $lei->{1} // die 'BUG: no lei->{1}';
+ my $old = delete $lei->{old_1};
+ $lei->{1} = $old if $old;
+ return if $out->close; # reaps gzip|pigz|xz|bzip2
+ my $msg = "E: Error closing $lei->{ovv}->{dst}";
+ $? ? $lei->child_error($?) : ($msg .= " ($!)");
+ die $msg;
+}
+
sub _do_augment_mbox {
my ($self, $lei) = @_;
return unless $self->{seekable};
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 5e36c11a..cee3ad07 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -393,14 +393,7 @@ sub query_done { # EOF callback for main daemon
$lei->sto_done_request;
$lei->{ovv}->ovv_end($lei);
if ($l2m) { # close() calls LeiToMail reap_compress
- if (my $out = delete $lei->{old_1}) {
- if (my $mbout = $lei->{1}) { # compressor pipe process
- $mbout->close or die <<"";
-Error closing $lei->{ovv}->{dst}: \$!=$! \$?=$?
-
- }
- $lei->{1} = $out;
- }
+ $l2m->finish_output($lei);
if ($l2m->lock_free) {
$l2m->poke_dst;
$lei->poke_mua;
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] lei convert: support idempotent v2 outputs
@ 2023-11-15 9:21 6% Eric Wong
2023-11-15 9:21 7% ` [PATCH 4/4] lei q|up|convert: common finish_output to detect errors Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-15 9:21 UTC (permalink / raw)
To: meta
This may make it easier for public-inbox admins to forcibly
inject missing messages from existing mbox*/maildir/IMAP/NNTP
archives.
1/4 was only needed to get 2/4 working, but 3/4 makes
it unnecessary with our current codebase (though we may
still need 1/4 in the future).
4/4 was noticed while working on 3/4.
Eric Wong (4):
lei: fix idempotent STDERR redirect in workers
lei convert: fix repeat and idempotent v2 output
lei: avoid extra fork for v2 outputs
lei q|up|convert: common finish_output to detect errors
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 9 ++++++---
lib/PublicInbox/LeiOverview.pm | 4 ++--
lib/PublicInbox/LeiToMail.pm | 33 +++++++++++++++++++++------------
lib/PublicInbox/LeiXSearch.pm | 13 +------------
lib/PublicInbox/V2Writable.pm | 1 -
t/lei-convert.t | 31 ++++++++++++++++++++++++++++++-
7 files changed, 61 insertions(+), 32 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-15 9:21 6% [PATCH 0/4] lei convert: support idempotent v2 outputs Eric Wong
2023-11-15 9:21 7% ` [PATCH 4/4] lei q|up|convert: common finish_output to detect errors Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).