* [PATCH 03/18] cindex: use `local' for pipes between processes
2023-11-13 13:15 6% [PATCH 00/18] cindex: some --associate work Eric Wong
@ 2023-11-13 13:15 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-13 13:15 UTC (permalink / raw)
To: meta
We can let these pipes get auto-closed upon leaving the process
subroutine scope.
---
lib/PublicInbox/CodeSearchIdx.pm | 19 +++++++++----------
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm
index 5b80db44..ffd443e1 100644
--- a/lib/PublicInbox/CodeSearchIdx.pm
+++ b/lib/PublicInbox/CodeSearchIdx.pm
@@ -522,8 +522,8 @@ sub dump_roots_start {
close $fh;
# dump_roots | sort -k1,1 | OFS=' ' uniq_fold >to_root_id
my ($sort_opt, $fold_opt);
- pipe($sort_opt->{0}, my $sort_w);
- pipe($fold_opt->{0}, $sort_opt->{1});
+ pipe(local $sort_opt->{0}, my $sort_w);
+ pipe(local $fold_opt->{0}, local $sort_opt->{1});
my @sort = (@SORT, '-k1,1');
my $dst = "$TMPDIR/to_root_id";
open $fold_opt->{1}, '>', $dst;
@@ -560,8 +560,8 @@ EOM
sub dump_ibx_start {
my ($self, $associate) = @_;
my ($sort_opt, $fold_opt);
- pipe($sort_opt->{0}, $DUMP_IBX_WPIPE);
- pipe($fold_opt->{0}, $sort_opt->{1});
+ pipe(local $sort_opt->{0}, $DUMP_IBX_WPIPE);
+ pipe(local $fold_opt->{0}, local $sort_opt->{1});
my @sort = (@SORT, '-k1,1'); # sort only on ASSOC_PFX
# pipeline: dump_ibx | sort -k1,1 | uniq_fold >to_ibx_id
open $fold_opt->{1}, '>', "$TMPDIR/to_ibx_id";
@@ -952,8 +952,8 @@ sub init_prune ($) {
for (0..$#IDX_SHARDS) { push @delve, "$self->{xpfx}/$_" }
my $run_prune = PublicInbox::OnDestroy->new($$, \&run_prune, $self);
my ($sort_opt, $sed_opt, $delve_opt);
- pipe($sed_opt->{0}, $delve_opt->{1});
- pipe($sort_opt->{0}, $sed_opt->{1});
+ pipe(local $sed_opt->{0}, local $delve_opt->{1});
+ pipe(local $sort_opt->{0}, local $sed_opt->{1});
open($sort_opt->{1}, '+>', "$TMPDIR/indexed_commits");
run_await([@SORT, '-u'], $CMD_ENV, $sort_opt, \&cmd_done, $run_prune);
run_await(\@sed, $CMD_ENV, $sed_opt, \&cmd_done, $run_prune);
@@ -986,13 +986,12 @@ sub run_prune { # OnDestroy when `git config extensions.objectFormat' are done
# ) | awk | sort | comm | cidx_read_comm()
my ($awk_opt, $sort_opt, $batch_opt);
my $comm_opt = { -C => "$TMPDIR" };
- pipe($awk_opt->{0}, $batch_opt->{1});
- pipe($sort_opt->{0}, $awk_opt->{1});
- pipe($comm_opt->{0}, $sort_opt->{1});
+ pipe(local $awk_opt->{0}, local $batch_opt->{1});
+ pipe(local $sort_opt->{0}, local $awk_opt->{1});
+ pipe(local $comm_opt->{0}, local $sort_opt->{1});
run_await(\@AWK, $CMD_ENV, $awk_opt, \&cmd_done);
run_await([@SORT, '-u'], $CMD_ENV, $sort_opt, \&cmd_done);
my $comm_rd = popen_rd(\@COMM, $CMD_ENV, $comm_opt, \&cmd_done, \@COMM);
- %$_ = () for ($awk_opt, $sort_opt, $comm_opt); # comm_rd is blocking :<
PublicInbox::CidxComm->new($comm_rd, $self); # calls cidx_read_comm
my $git_ver = PublicInbox::Git::git_version();
push @PRUNE_BATCH, '--buffer' if $git_ver ge v2.6;
^ permalink raw reply related [relevance 7%]
* [PATCH 00/18] cindex: some --associate work
@ 2023-11-13 13:15 6% Eric Wong
2023-11-13 13:15 7% ` [PATCH 03/18] cindex: use `local' for pipes between processes Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-13 13:15 UTC (permalink / raw)
To: meta
Still very much in flux, but some treewide cleanups in there...
And I've been wondering if "join" is a better word than
"associate" to denote the relationship between inboxes
and coderepos.
But "join" (even if we use join(1) internally) probably
implies strict relationships, whereas our current "associate"
is always going to be fuzzy due to patchids being fuzzy
and blobs OIDs being abbreviated in patches.
I'm also thinking about moving --associate-* CLI switches
into suboptions (e.g. what getsubopt(3) supports), so:
--associate=aggressive,prefixes=patchid+dfblob
But Perl doesn't ship with getsubopt(3) emulation
out-of-the-box
Eric Wong (18):
cindex: check `say' errors w/ close or ->flush
tmpfile: check `stat' errors, use autodie for unlink
cindex: use `local' for pipes between processes
xap_helper_cxx: use write_file helper
xap_helper_cxx: make the build process ccache-friendly
xap_helper_cxx: use -pipe by default in CXXFLAGS
xap_client: spawn C++ xap_helper directly
treewide: update read_all to avoid eof|close checks
spawn: don't append to scalarrefs on stdout/stderr
cindex: imply --all with --associate w/o -I/--only
cindex: delay associate until prune+indexing finish
xap_helper: Perl dump_ibx respects `-m MAX'
cidx_xap_helper_aux: complain about truncated inputs
xap_helper: stricter and harsher error handling
xap_helper: better variable naming for key buffer
cindex: do not guess integer maximum for Xapian
cindex: rename associate-max => window
cindex: support --associate-aggressive shortcut
lib/PublicInbox/CidxComm.pm | 6 +-
lib/PublicInbox/CidxXapHelperAux.pm | 6 +-
lib/PublicInbox/CodeSearchIdx.pm | 122 ++++++++++-----
lib/PublicInbox/Gcf2.pm | 3 +-
lib/PublicInbox/IO.pm | 18 ++-
lib/PublicInbox/LeiInput.pm | 10 +-
lib/PublicInbox/LeiMirror.pm | 10 +-
lib/PublicInbox/LeiToMail.pm | 3 +-
lib/PublicInbox/Spawn.pm | 4 +-
lib/PublicInbox/TestCommon.pm | 6 +-
lib/PublicInbox/Tmpfile.pm | 10 +-
lib/PublicInbox/XapClient.pm | 28 ++--
lib/PublicInbox/XapHelper.pm | 30 ++--
lib/PublicInbox/XapHelperCxx.pm | 55 +++----
lib/PublicInbox/xap_helper.h | 233 ++++++++++++----------------
script/public-inbox-cindex | 3 +-
script/public-inbox-learn | 2 +-
script/public-inbox-mda | 2 +-
script/public-inbox-purge | 2 +-
t/spawn.t | 2 +-
t/xap_helper.t | 27 ++--
21 files changed, 287 insertions(+), 295 deletions(-)
Yay, less code!
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-13 13:15 6% [PATCH 00/18] cindex: some --associate work Eric Wong
2023-11-13 13:15 7% ` [PATCH 03/18] cindex: use `local' for pipes between processes Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).