* [PATCH 12/14] admin: resolve_git_dir respects symlinks
2023-11-28 14:56 6% [PATCH 00/14] IT'S ALIVE! www loads cindex join data Eric Wong
@ 2023-11-28 14:56 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-28 14:56 UTC (permalink / raw)
To: meta
Absolute pathnames of git coderepos are stored in the cindex,
but we should favor paths relative to $ENV{PWD} since it
respects symlinks in the heirarchy.
Respecting symlinks makes it easier to migrate cindex to
new storage as old storage wears out and to relocate the
storage device onto another machine.
---
lib/PublicInbox/Admin.pm | 25 +++++++++++++++++++++----
t/admin.t | 12 ++++++++++++
2 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/Admin.pm b/lib/PublicInbox/Admin.pm
index 893f4a1b..cc9d2171 100644
--- a/lib/PublicInbox/Admin.pm
+++ b/lib/PublicInbox/Admin.pm
@@ -63,15 +63,32 @@ sub resolve_inboxdir {
$dir;
}
+sub valid_pwd {
+ my $pwd = $ENV{PWD} // return;
+ my @st_pwd = stat $pwd or return;
+ my @st_cwd = stat '.' or die "stat(.): $!";
+ "@st_pwd[1,0]" eq "@st_cwd[1,0]" ? $pwd : undef;
+}
+
sub resolve_git_dir {
- my ($cd) = @_;
+ my ($cd) = @_; # cd may be `undef' for cwd
# try v1 bare git dirs
+ my $pwd = valid_pwd();
+ my $env;
+ defined($pwd) && substr($cd // '/', 0, 1) ne '/' and
+ $env->{PWD} = "$pwd/$cd";
my $cmd = [ qw(git rev-parse --git-dir) ];
- my $dir = run_qx($cmd, undef, {-C => $cd});
+ my $dir = run_qx($cmd, $env, { -C => $cd });
die "error in @$cmd (cwd:${\($cd // '.')}): $?\n" if $?;
chomp $dir;
- # --absolute-git-dir requires git v2.13.0+
- $dir = rel2abs_collapsed($dir, $cd) if $dir !~ m!\A/!;
+ # --absolute-git-dir requires git v2.13.0+, and we want to
+ # respect symlinks when $ENV{PWD} if $ENV{PWD} ne abs_path('.')
+ # since we store absolute GIT_DIR paths in cindex.
+ if (substr($dir, 0, 1) ne '/') {
+ substr($cd // '/', 0, 1) eq '/' or
+ $cd = File::Spec->rel2abs($cd, $pwd);
+ $dir = rel2abs_collapsed($dir, $cd);
+ }
$dir;
}
diff --git a/t/admin.t b/t/admin.t
index 20e3deb7..586938d0 100644
--- a/t/admin.t
+++ b/t/admin.t
@@ -6,6 +6,7 @@ use v5.10.1;
use PublicInbox::TestCommon;
use PublicInbox::Import;
use_ok 'PublicInbox::Admin';
+use autodie;
my $v1 = create_inbox 'v1', -no_gc => 1, sub {};
my ($tmpdir, $for_destroy) = tmpdir();
my $git_dir = $v1->{inboxdir};
@@ -23,6 +24,17 @@ SKIP: {
};
*resolve_inboxdir = \&PublicInbox::Admin::resolve_inboxdir;
+*resolve_git_dir = \&PublicInbox::Admin::resolve_git_dir;
+
+{
+ symlink $git_dir, my $sym = "$tmpdir/v1-symlink.git";
+ for my $d ('') { # TODO: should work inside $sym/objects
+ local $ENV{PWD} = $sym.$d;
+ chdir $sym.$d;
+ is resolve_git_dir('.'), $sym,
+ "symlink preserved from {SYMLINKDIR}.git$d";
+ }
+}
# v1
is(resolve_inboxdir($git_dir), $git_dir, 'top-level GIT_DIR resolved');
^ permalink raw reply related [relevance 7%]
* [PATCH 00/14] IT'S ALIVE! www loads cindex join data
@ 2023-11-28 14:56 6% Eric Wong
2023-11-28 14:56 7% ` [PATCH 12/14] admin: resolve_git_dir respects symlinks Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-28 14:56 UTC (permalink / raw)
To: meta
8/14 is the killer one which actually makes the cindex data
useful for WWW and powering solver. Keep in mind, I've had
to cap solver at 3 coderepos as a temporary measure since
there's a lot of "weak" joins we should be weeding out.
More documentation coming, but cindex joins are very much
a fuzzy thing which will have to deal with false positives
and such. So figuring out the scoring for sanity would
make sense...
Fortunately, --join=aggressive,reset only takes ~1 hour for me,
so probably 1/3 that on modern hardware. Incremental
`-cindex --join' (no suboptions) usually takes <5 minutes if
done frequently.
New performance problem: solver could definitely be smarter
about dealing with common roots/groups. For the longest time,
I've only had 1 coderepo per-inbox, having hundreds is wacky.
Actual searching against the cindex isn't done, yet, but
that's kinda straightforward.
Eric Wong (14):
test_common: create_*: detect changes all parameters
t/cindex*: require SCM_RIGHTS for these tests
codesearch: eliminate redundant substitutions
solver: schedule cleanup after synchronous git->check
xap_helper.h: move cindex endpoints to separate file
xap_helper: implement mset endpoint for WWW, IMAP, etc...
hval: use File::Spec to make relative paths for href
www: load and use cindex join data
git: speed up ->git_path for non-worktrees
cindex: require `-g GIT_DIR' or `-r PROJECT_ROOT'
git: speed up Git->new by 5% or so
admin: resolve_git_dir respects symlinks
cindex: extra quit checks
www: start working on a repo listing
Documentation/public-inbox-cindex.pod | 2 +-
MANIFEST | 3 +
Makefile.PL | 8 +-
lib/PublicInbox/Admin.pm | 25 +-
lib/PublicInbox/CodeSearch.pm | 162 ++++++++++-
lib/PublicInbox/CodeSearchIdx.pm | 52 ++--
lib/PublicInbox/Config.pm | 39 ++-
lib/PublicInbox/Git.pm | 27 +-
lib/PublicInbox/Hval.pm | 12 +-
lib/PublicInbox/RepoList.pm | 39 +++
lib/PublicInbox/Search.pm | 42 +++
lib/PublicInbox/SearchIdx.pm | 10 +-
lib/PublicInbox/SolverGit.pm | 9 +-
lib/PublicInbox/TestCommon.pm | 35 ++-
lib/PublicInbox/View.pm | 7 +-
lib/PublicInbox/WWW.pm | 1 +
lib/PublicInbox/WwwCoderepo.pm | 44 ++-
lib/PublicInbox/WwwStream.pm | 11 +-
lib/PublicInbox/WwwText.pm | 19 +-
lib/PublicInbox/XapHelper.pm | 51 ++--
lib/PublicInbox/XapHelperCxx.pm | 14 +-
lib/PublicInbox/xap_helper.h | 379 +++++++-------------------
lib/PublicInbox/xh_cidx.h | 244 +++++++++++++++++
lib/PublicInbox/xh_mset.h | 96 +++++++
script/public-inbox-cindex | 38 ++-
t/admin.t | 12 +
t/cindex-join.t | 9 +-
t/cindex.t | 91 ++++++-
t/xap_helper.t | 53 +++-
xt/solver.t | 3 +-
30 files changed, 1111 insertions(+), 426 deletions(-)
create mode 100644 lib/PublicInbox/RepoList.pm
create mode 100644 lib/PublicInbox/xh_cidx.h
create mode 100644 lib/PublicInbox/xh_mset.h
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-28 14:56 6% [PATCH 00/14] IT'S ALIVE! www loads cindex join data Eric Wong
2023-11-28 14:56 7% ` [PATCH 12/14] admin: resolve_git_dir respects symlinks Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).