* [PATCH 0/7] gcf2: libgit2-based cat-file alternative
@ 2020-09-19 9:37 7% Eric Wong
2020-09-19 9:37 7% ` [PATCH 2/7] t/gcf2: test changes to alternates Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-09-19 9:37 UTC (permalink / raw)
To: meta
This allows a single cat-file-like process to handle multiple
inboxes; instead of having a "git cat-file --batch" process for
every inbox; saving pipes and process table space.
The preliminary code was done months ago, but I struggled to put
all the pieces together in a coherent way. My brain has been
scattered :x
I finally decided to make the gcf2 process a global singleton
(per-worker) to avoid complexity elsewhere in the config...
It doesn't detect or release unlinked packs + indices, yet,
so "git gc" may not free disk space until restarted.
Otherwise it does detect new epochs and seems mostly working
otherwise...
Eric Wong (7):
gcf2: libgit2-based git cat-file alternative
t/gcf2: test changes to alternates
add gcf2 client and executable script
gcf2: transparently retry on missing OID
gcf2*: more descriptive package descriptions
gcf2: require git dir with OID
gcf2: wire up read-only daemons and rm -gcf2 script
MANIFEST | 5 +
lib/PublicInbox/Daemon.pm | 11 +++
lib/PublicInbox/Gcf2.pm | 89 ++++++++++++++++++
lib/PublicInbox/Gcf2Client.pm | 62 +++++++++++++
lib/PublicInbox/Git.pm | 41 ++++++---
lib/PublicInbox/GitAsyncCat.pm | 73 +++++++++++++--
lib/PublicInbox/IMAP.pm | 2 +-
lib/PublicInbox/gcf2_libgit2.h | 142 +++++++++++++++++++++++++++++
script/public-inbox-httpd | 1 +
t/gcf2.t | 162 +++++++++++++++++++++++++++++++++
t/gcf2_client.t | 90 ++++++++++++++++++
11 files changed, 652 insertions(+), 26 deletions(-)
create mode 100644 lib/PublicInbox/Gcf2.pm
create mode 100644 lib/PublicInbox/Gcf2Client.pm
create mode 100644 lib/PublicInbox/gcf2_libgit2.h
create mode 100644 t/gcf2.t
create mode 100644 t/gcf2_client.t
^ permalink raw reply [relevance 7%]
* [PATCH 2/7] t/gcf2: test changes to alternates
2020-09-19 9:37 7% [PATCH 0/7] gcf2: libgit2-based cat-file alternative Eric Wong
@ 2020-09-19 9:37 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-09-19 9:37 UTC (permalink / raw)
To: meta; +Cc: Eric Wong
From: Eric Wong <e@yhbt.net>
Calling ->add_alternate won't pick up new additions to
$OBJDIR/info/alternates, unfornately. Thus v2 inboxes will
need to do something to invalidate Gcf2 objects.
---
t/gcf2.t | 68 +++++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 60 insertions(+), 8 deletions(-)
diff --git a/t/gcf2.t b/t/gcf2.t
index c67efb6c..9056b340 100644
--- a/t/gcf2.t
+++ b/t/gcf2.t
@@ -7,24 +7,74 @@ use Test::More;
use Fcntl qw(:seek);
use IO::Handle ();
use POSIX qw(_exit);
+use Cwd qw(abs_path);
require_mods('PublicInbox::Gcf2');
use_ok 'PublicInbox::Gcf2';
+use PublicInbox::Import;
+my ($tmpdir, $for_destroy) = tmpdir();
+
my $gcf2 = PublicInbox::Gcf2::new();
is(ref($gcf2), 'PublicInbox::Gcf2', '::new works');
+my $COPYING = 'dba13ed2ddf783ee8118c6a581dbf75305f816a3';
+open my $agpl, '<', 'COPYING' or BAIL_OUT "AGPL-3 missing: $!";
+$agpl = do { local $/; <$agpl> };
+
+PublicInbox::Import::init_bare($tmpdir);
+my $fi_data = './t/git.fast-import-data';
+my $rdr = {};
+open $rdr->{0}, '<', $fi_data or BAIL_OUT $!;
+xsys([qw(git fast-import --quiet)], { GIT_DIR => $tmpdir }, $rdr);
+is($?, 0, 'fast-import succeeded');
+$gcf2->add_alternate("$tmpdir/objects");
+
+{
+ my ($r, $w);
+ pipe($r, $w) or BAIL_OUT $!;
+ my $tree = 'fdbc43725f21f485051c17463b50185f4c3cf88c';
+ $gcf2->cat_oid(fileno($w), $tree);
+ close $w;
+ is("$tree tree 30\n", <$r>, 'tree header ok');
+ $r = do { local $/; <$r> };
+ is(chop($r), "\n", 'got trailing newline');
+ is(length($r), 30, 'tree length matches');
+}
+
chomp(my $objdir = xqx([qw(git rev-parse --git-path objects)]));
if ($objdir =~ /\A--git-path\n/) { # git <2.5
chomp($objdir = xqx([qw(git rev-parse --git-dir)]));
$objdir .= '/objects';
- $objdir = undef unless -d $objdir;
}
+if ($objdir && -d $objdir) {
+ $objdir = abs_path($objdir);
+ open my $alt, '>>', "$tmpdir/objects/info/alternates" or
+ BAIL_OUT $!;
+ print $alt $objdir, "\n" or BAIL_OUT $!;
+ close $alt or BAIL_OUT $!;
-my $COPYING = 'dba13ed2ddf783ee8118c6a581dbf75305f816a3';
-open my $agpl, '<', 'COPYING' or BAIL_OUT "AGPL-3 missing: $!";
-$agpl = do { local $/; <$agpl> };
+ # calling gcf2->add_alternate on an already-added path won't
+ # cause alternates to be reloaded, so we do
+ # $gcf2->add_alternate($objdir) later on instead of
+ # $gcf2->add_alternate("$tmpdir/objects");
+ # $objdir = "$tmpdir/objects";
+} else {
+ $objdir = undef
+}
+
+my $nr = $ENV{TEST_LEAK_NR};
+my $cat = $ENV{TEST_LEAK_CAT} // 10;
+diag "checking for leaks... (TEST_LEAK_NR=$nr TEST_LEAK_CAT=$cat)" if $nr;
SKIP: {
- skip 'not in git worktree', 15 unless defined($objdir);
+ skip 'not in git worktree', 21 unless defined($objdir);
$gcf2->add_alternate($objdir);
+ eval { $gcf2->add_alternate($objdir) };
+ ok(!$@, 'no error adding alternate redundantly');
+ if ($nr) {
+ diag "adding alternate $nr times redundantly";
+ $gcf2->add_alternate($objdir) for (1..$nr);
+ diag 'done adding redundant alternates';
+ }
+
open my $fh, '+>', undef or BAIL_OUT "open: $!";
my $fd = fileno($fh);
$fh->autoflush(1);
@@ -52,6 +102,10 @@ SKIP: {
};
$ck_copying->('regular file');
+ $gcf2 = PublicInbox::Gcf2::new();
+ $gcf2->add_alternate("$tmpdir/objects");
+ $ck_copying->('alternates respected');
+
$^O eq 'linux' or skip('pipe tests are Linux-only', 12);
my $size = -s $fh;
for my $blk (1, 0) {
@@ -86,11 +140,9 @@ SKIP: {
}
}
-if (my $nr = $ENV{TEST_LEAK_NR}) {
+if ($nr) {
open my $null, '>', '/dev/null' or BAIL_OUT "open /dev/null: $!";
my $fd = fileno($null);
- my $cat = $ENV{TEST_LEAK_CAT} // 10;
- diag "checking for leaks... (TEST_LEAK_NR=$nr TEST_LEAK_CAT=$cat)";
local $SIG{PIPE} = 'IGNORE';
my ($r, $w);
pipe($r, $w);
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-09-19 9:37 7% [PATCH 0/7] gcf2: libgit2-based cat-file alternative Eric Wong
2020-09-19 9:37 7% ` [PATCH 2/7] t/gcf2: test changes to alternates Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).