* [PATCH 4/6] import: switch to using ProcessPipe
2021-01-02 9:13 7% [PATCH 0/6] process pipe improvements Eric Wong
@ 2021-01-02 9:13 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-01-02 9:13 UTC (permalink / raw)
To: meta
This saves us a few lines of code, but also prevents misreaping
by sibling processes.
---
lib/PublicInbox/Import.pm | 23 +++++++++--------------
1 file changed, 9 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index 47a529ff..b5780d2b 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -54,9 +54,9 @@ sub new {
sub gfi_start {
my ($self) = @_;
- return ($self->{in}, $self->{out}) if $self->{pid};
+ return ($self->{in}, $self->{out}) if $self->{in};
- my ($in_r, $pid, $out_r, $out_w);
+ my ($in_r, $out_r, $out_w);
pipe($out_r, $out_w) or die "pipe failed: $!";
$self->lock_acquire;
@@ -72,13 +72,11 @@ sub gfi_start {
chomp @t;
$self->{-tree} = { map { $_ => 1 } @t };
}
- my @cmd = ('git', "--git-dir=$git->{git_dir}",
- qw(fast-import --quiet --done --date-format=raw));
- ($in_r, $pid) = popen_rd(\@cmd, undef, { 0 => $out_r });
+ $in_r = $self->{in} = $git->popen(qw(fast-import
+ --quiet --done --date-format=raw),
+ undef, { 0 => $out_r });
$out_w->autoflush(1);
- $self->{in} = $in_r;
$self->{out} = $out_w;
- $self->{pid} = $pid;
$self->{nchg} = 0;
};
if ($@) {
@@ -162,14 +160,14 @@ sub check_remove_v1 {
sub checkpoint {
my ($self) = @_;
- return unless $self->{pid};
+ return unless $self->{in};
print { $self->{out} } "checkpoint\n" or wfail;
undef;
}
sub progress {
my ($self, $msg) = @_;
- return unless $self->{pid};
+ return unless $self->{in};
print { $self->{out} } "progress $msg\n" or wfail;
readline($self->{in}) eq "progress $msg\n" or die
"progress $msg not received\n";
@@ -218,7 +216,7 @@ sub barrier {
# used for v2
sub get_mark {
my ($self, $mark) = @_;
- die "not active\n" unless $self->{pid};
+ die "not active\n" unless $self->{in};
my ($r, $w) = $self->gfi_start;
print $w "get-mark $mark\n" or wfail;
defined(my $oid = <$r>) or die "get-mark failed, need git 2.6.0+\n";
@@ -481,10 +479,7 @@ sub done {
eval {
my $r = delete $self->{in} or die 'BUG: missing {in} when done';
print $w "done\n" or wfail;
- my $pid = delete $self->{pid} or
- die 'BUG: missing {pid} when done';
- waitpid($pid, 0) == $pid or die 'fast-import did not finish';
- $? == 0 or die "fast-import failed: $?";
+ close $r or die "fast-import failed: $?"; # ProcessPipe::CLOSE
};
my $wait_err = $@;
my $nchg = delete $self->{nchg};
^ permalink raw reply related [relevance 6%]
* [PATCH 0/6] process pipe improvements
@ 2021-01-02 9:13 7% Eric Wong
2021-01-02 9:13 6% ` [PATCH 4/6] import: switch to using ProcessPipe Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-01-02 9:13 UTC (permalink / raw)
To: meta
The DS::in_loop clobbering in MboxReader annoyed me and drove
this series.
I wanted to do it for the bidirectional pipes with
"git cat-file --batch", too, but there seems to be some
lifetime management issues which get hard-to-control
with things like the the waitpid(..,WNOHANG) call
in GitAsyncCat. Maybe switching --batch + DS to use UNIX
sockets can be done to save FDs (or I'm too brain-damaged
to figure this out).
But right now, all of our codebase is robust against children
attempting to reap siblings (or PIDs of former siblings)
Eric Wong (6):
processpipe: allow synchronous close to set $?
processpipe: lazy-require PublicInbox::DS for dwaitpid
git: qx: waitpid synchronously via ProcessPipe->CLOSE
import: switch to using ProcessPipe
git: manifest_entry: use ProcessPipe via popen_rd
qspawn: switch to ProcessPipe via popen_rd
lib/PublicInbox/Git.pm | 20 +++++++++-----
lib/PublicInbox/Import.pm | 23 +++++++---------
lib/PublicInbox/MboxReader.pm | 3 ---
lib/PublicInbox/ProcessPipe.pm | 49 +++++++++++++++++++++-------------
lib/PublicInbox/Qspawn.pm | 15 +++++------
lib/PublicInbox/Spawn.pm | 10 +++----
t/mbox_reader.t | 17 ++++++++++++
t/spawn.t | 38 ++++++++++++++++++++++++++
8 files changed, 119 insertions(+), 56 deletions(-)
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-01-02 9:13 7% [PATCH 0/6] process pipe improvements Eric Wong
2021-01-02 9:13 6% ` [PATCH 4/6] import: switch to using ProcessPipe Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).