* [PATCH 05/18] xap_helper_cxx: make the build process ccache-friendly
2023-11-13 13:15 6% [PATCH 00/18] cindex: some --associate work Eric Wong
@ 2023-11-13 13:15 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-13 13:15 UTC (permalink / raw)
To: meta
We need to have stable filenames and separate compilation
from the linkage stage for ccache to hit. So avoid the use
of a temporary directory and instead rely on a lock file to
guard against parallel builds.
---
lib/PublicInbox/XapHelperCxx.pm | 32 +++++++++++++++++++-------------
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/XapHelperCxx.pm b/lib/PublicInbox/XapHelperCxx.pm
index 0891074d..7aa52416 100644
--- a/lib/PublicInbox/XapHelperCxx.pm
+++ b/lib/PublicInbox/XapHelperCxx.pm
@@ -7,7 +7,7 @@
# The resulting executable is not linked to Perl in any way.
package PublicInbox::XapHelperCxx;
use v5.12;
-use PublicInbox::Spawn qw(run_qx which);
+use PublicInbox::Spawn qw(run_die run_qx which);
use PublicInbox::IO qw(read_all write_file);
use PublicInbox::Search;
use Fcntl qw(SEEK_SET);
@@ -24,8 +24,8 @@ my @pm_dep = map { $srcpfx.$_ } qw(Search.pm CodeSearch.pm);
my $ldflags = '-Wl,-O1';
$ldflags .= ' -Wl,--compress-debug-sections=zlib' if $^O ne 'openbsd';
my $xflags = ($ENV{CXXFLAGS} // '-Wall -ggdb3 -O0') . ' ' .
- ($ENV{LDFLAGS} // $ldflags) .
- qq{ -DTHREADID=}.PublicInbox::Search::THREADID;
+ ' -DTHREADID=' . PublicInbox::Search::THREADID .
+ ' ' . ($ENV{LDFLAGS} // $ldflags);
my $xap_modversion;
sub xap_cfg (@) {
@@ -58,12 +58,12 @@ sub build () {
die "mkdir($dir): $err" if !-d $dir;
}
use autodie;
- require File::Temp;
require PublicInbox::CodeSearch;
+ require PublicInbox::Lock;
+ require PublicInbox::OnDestroy;
my ($prog) = ($bin =~ m!/([^/]+)\z!);
- my $tmp = File::Temp->newdir(DIR => $dir) // die "newdir: $!";
- my $src = "$tmp/$prog.cpp";
- open my $fh, '>', $src;
+ my $lk = PublicInbox::Lock->new("$dir/$prog.lock")->lock_for_scope;
+ open my $fh, '>', "$dir/$prog.cpp";
for (@srcs) {
say $fh qq(# line 1 "$_");
open my $rfh, '<', $_;
@@ -73,6 +73,10 @@ sub build () {
print $fh PublicInbox::CodeSearch::generate_cxx();
close $fh;
+ opendir my $dh, '.';
+ my $restore = PublicInbox::OnDestroy->new(\&chdir, $dh);
+ chdir $dir;
+
# xap_modversion may be set by needs_rebuild
$xap_modversion //= xap_cfg('--modversion');
my $fl = xap_cfg(qw(--libs --cflags));
@@ -84,14 +88,16 @@ sub build () {
# distributed packages.
$^O eq 'netbsd' and $fl =~ s/(\A|[ \t])\-L([^ \t]+)([ \t]|\z)/
"$1-L$2 -Wl,-rpath=$2$3"/egsx;
-
- my $cmd = "$cxx $src $fl $xflags -o $tmp/$prog";
- system($cmd) and die "$cmd failed: \$?=$?";
- write_file '>', "$tmp/XFLAGS", $xflags, "\n";
- write_file '>', "$tmp/xap_modversion", $xap_modversion, "\n";
+ my @xflags = split(/\s+/, "$fl $xflags");
+ my @cflags = grep(!/\A-(?:Wl|l|L)/, @xflags);
+ run_die([$cxx, '-c', "$prog.cpp", @cflags]);
+ run_die([$cxx, '-o', "$prog.tmp", "$prog.o", @xflags]);
+ unlink "$prog.cpp", "$prog.o";
+ write_file '>', 'XFLAGS.tmp', $xflags, "\n";
+ write_file '>', 'xap_modversion.tmp', $xap_modversion, "\n";
undef $xap_modversion; # do we ever build() twice?
# not quite atomic, but close enough :P
- rename("$tmp/$_", "$dir/$_") for ($prog, qw(XFLAGS xap_modversion));
+ rename("$_.tmp", $_) for ($prog, qw(XFLAGS xap_modversion));
}
sub check_build () {
^ permalink raw reply related [relevance 7%]
* [PATCH 00/18] cindex: some --associate work
@ 2023-11-13 13:15 6% Eric Wong
2023-11-13 13:15 7% ` [PATCH 05/18] xap_helper_cxx: make the build process ccache-friendly Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-13 13:15 UTC (permalink / raw)
To: meta
Still very much in flux, but some treewide cleanups in there...
And I've been wondering if "join" is a better word than
"associate" to denote the relationship between inboxes
and coderepos.
But "join" (even if we use join(1) internally) probably
implies strict relationships, whereas our current "associate"
is always going to be fuzzy due to patchids being fuzzy
and blobs OIDs being abbreviated in patches.
I'm also thinking about moving --associate-* CLI switches
into suboptions (e.g. what getsubopt(3) supports), so:
--associate=aggressive,prefixes=patchid+dfblob
But Perl doesn't ship with getsubopt(3) emulation
out-of-the-box
Eric Wong (18):
cindex: check `say' errors w/ close or ->flush
tmpfile: check `stat' errors, use autodie for unlink
cindex: use `local' for pipes between processes
xap_helper_cxx: use write_file helper
xap_helper_cxx: make the build process ccache-friendly
xap_helper_cxx: use -pipe by default in CXXFLAGS
xap_client: spawn C++ xap_helper directly
treewide: update read_all to avoid eof|close checks
spawn: don't append to scalarrefs on stdout/stderr
cindex: imply --all with --associate w/o -I/--only
cindex: delay associate until prune+indexing finish
xap_helper: Perl dump_ibx respects `-m MAX'
cidx_xap_helper_aux: complain about truncated inputs
xap_helper: stricter and harsher error handling
xap_helper: better variable naming for key buffer
cindex: do not guess integer maximum for Xapian
cindex: rename associate-max => window
cindex: support --associate-aggressive shortcut
lib/PublicInbox/CidxComm.pm | 6 +-
lib/PublicInbox/CidxXapHelperAux.pm | 6 +-
lib/PublicInbox/CodeSearchIdx.pm | 122 ++++++++++-----
lib/PublicInbox/Gcf2.pm | 3 +-
lib/PublicInbox/IO.pm | 18 ++-
lib/PublicInbox/LeiInput.pm | 10 +-
lib/PublicInbox/LeiMirror.pm | 10 +-
lib/PublicInbox/LeiToMail.pm | 3 +-
lib/PublicInbox/Spawn.pm | 4 +-
lib/PublicInbox/TestCommon.pm | 6 +-
lib/PublicInbox/Tmpfile.pm | 10 +-
lib/PublicInbox/XapClient.pm | 28 ++--
lib/PublicInbox/XapHelper.pm | 30 ++--
lib/PublicInbox/XapHelperCxx.pm | 55 +++----
lib/PublicInbox/xap_helper.h | 233 ++++++++++++----------------
script/public-inbox-cindex | 3 +-
script/public-inbox-learn | 2 +-
script/public-inbox-mda | 2 +-
script/public-inbox-purge | 2 +-
t/spawn.t | 2 +-
t/xap_helper.t | 27 ++--
21 files changed, 287 insertions(+), 295 deletions(-)
Yay, less code!
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-13 13:15 6% [PATCH 00/18] cindex: some --associate work Eric Wong
2023-11-13 13:15 7% ` [PATCH 05/18] xap_helper_cxx: make the build process ccache-friendly Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).