user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 0/7] gcf2: libgit2-based cat-file alternative
@ 2020-09-19  9:37  7% Eric Wong
  2020-09-19  9:37  5% ` [PATCH 3/7] add gcf2 client and executable script Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-09-19  9:37 UTC (permalink / raw)
  To: meta

This allows a single cat-file-like process to handle multiple
inboxes; instead of having a "git cat-file --batch" process for
every inbox; saving pipes and process table space.

The preliminary code was done months ago, but I struggled to put
all the pieces together in a coherent way.  My brain has been
scattered :x

I finally decided to make the gcf2 process a global singleton
(per-worker) to avoid complexity elsewhere in the config...

It doesn't detect or release unlinked packs + indices, yet,
so "git gc" may not free disk space until restarted.

Otherwise it does detect new epochs and seems mostly working
otherwise...

Eric Wong (7):
  gcf2: libgit2-based git cat-file alternative
  t/gcf2: test changes to alternates
  add gcf2 client and executable script
  gcf2: transparently retry on missing OID
  gcf2*: more descriptive package descriptions
  gcf2: require git dir with OID
  gcf2: wire up read-only daemons and rm -gcf2 script

 MANIFEST                       |   5 +
 lib/PublicInbox/Daemon.pm      |  11 +++
 lib/PublicInbox/Gcf2.pm        |  89 ++++++++++++++++++
 lib/PublicInbox/Gcf2Client.pm  |  62 +++++++++++++
 lib/PublicInbox/Git.pm         |  41 ++++++---
 lib/PublicInbox/GitAsyncCat.pm |  73 +++++++++++++--
 lib/PublicInbox/IMAP.pm        |   2 +-
 lib/PublicInbox/gcf2_libgit2.h | 142 +++++++++++++++++++++++++++++
 script/public-inbox-httpd      |   1 +
 t/gcf2.t                       | 162 +++++++++++++++++++++++++++++++++
 t/gcf2_client.t                |  90 ++++++++++++++++++
 11 files changed, 652 insertions(+), 26 deletions(-)
 create mode 100644 lib/PublicInbox/Gcf2.pm
 create mode 100644 lib/PublicInbox/Gcf2Client.pm
 create mode 100644 lib/PublicInbox/gcf2_libgit2.h
 create mode 100644 t/gcf2.t
 create mode 100644 t/gcf2_client.t

^ permalink raw reply	[relevance 7%]

* [PATCH 3/7] add gcf2 client and executable script
  2020-09-19  9:37  7% [PATCH 0/7] gcf2: libgit2-based cat-file alternative Eric Wong
@ 2020-09-19  9:37  5% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-09-19  9:37 UTC (permalink / raw)
  To: meta; +Cc: Eric Wong

From: Eric Wong <e@yhbt.net>

This should be able to replace multiple `git cat-file' for blob
retrieval, but adjustments may be needed.
---
 Documentation/public-inbox-gcf2.pod | 63 +++++++++++++++++++++++++++++
 MANIFEST                            |  4 ++
 Makefile.PL                         |  5 +++
 lib/PublicInbox/Gcf2Client.pm       | 35 ++++++++++++++++
 script/public-inbox-gcf2            | 14 +++++++
 t/gcf2_client.t                     | 47 +++++++++++++++++++++
 6 files changed, 168 insertions(+)
 create mode 100644 Documentation/public-inbox-gcf2.pod
 create mode 100644 lib/PublicInbox/Gcf2Client.pm
 create mode 100755 script/public-inbox-gcf2
 create mode 100644 t/gcf2_client.t

diff --git a/Documentation/public-inbox-gcf2.pod b/Documentation/public-inbox-gcf2.pod
new file mode 100644
index 00000000..813fbe7f
--- /dev/null
+++ b/Documentation/public-inbox-gcf2.pod
@@ -0,0 +1,63 @@
+=head1 NAME
+
+public-inbox-gcf2 - internal libgit2-based blob retriever
+
+=head1 SYNOPSIS
+
+	This is an internal command used by public-inbox.
+	It may change unrecognizably or cease to exist at some point
+
+=head1 DESCRIPTION
+
+public-inbox-gcf2 is an optional internal process used by
+public-inbox daemons for read-only access to underlying git
+repositories.
+
+Users are NOT expected to run public-inbox-gcf2 on their own.
+It replaces multiple C<git cat-file --batch> processes by treating
+any git repos it knows about as alternates.
+
+None of its behaviors are stable and it is ALL subject to change
+at any time.
+
+Any lines written to its standard input prefixed with a C</>
+are interpreted as a git directory.  That git directory
+will be suffixed with "/objects" and treated as an alternate.
+It writes nothing to stdout in this case.
+
+Otherwise it behaves like C<git cat-file --batch>, but only accepts
+unabbreviated hexadecimal object IDs in its standard input.
+Its output format is identical to C<git cat-file --batch>.  It
+only works for L<public-inbox-v2-format(5)> inboxes and v1
+inboxes indexed by L<public-inbox-index(1)>.
+
+=head1 OPTIONS
+
+=head1 ENVIRONMENT
+
+=over 8
+
+=item PERL_INLINE_DIRECTORY
+
+This must be set unless C<~/.cache/public-inbox/inline-c>
+exists.  C<public-inbox-gcf2> uses L<Inline::C> and libgit2
+and compiles a small shim on its first run.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2020 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<git-cat-file(1)>
diff --git a/MANIFEST b/MANIFEST
index 0d3a7073..91457dab 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -26,6 +26,7 @@ Documentation/public-inbox-config.pod
 Documentation/public-inbox-convert.pod
 Documentation/public-inbox-daemon.pod
 Documentation/public-inbox-edit.pod
+Documentation/public-inbox-gcf2.pod
 Documentation/public-inbox-httpd.pod
 Documentation/public-inbox-imapd.pod
 Documentation/public-inbox-index.pod
@@ -129,6 +130,7 @@ lib/PublicInbox/Filter/RubyLang.pm
 lib/PublicInbox/Filter/SubjectTag.pm
 lib/PublicInbox/Filter/Vger.pm
 lib/PublicInbox/Gcf2.pm
+lib/PublicInbox/Gcf2Client.pm
 lib/PublicInbox/GetlineBody.pm
 lib/PublicInbox/Git.pm
 lib/PublicInbox/GitAsyncCat.pm
@@ -221,6 +223,7 @@ sa_config/user/.spamassassin/user_prefs
 script/public-inbox-compact
 script/public-inbox-convert
 script/public-inbox-edit
+script/public-inbox-gcf2
 script/public-inbox-httpd
 script/public-inbox-imapd
 script/public-inbox-index
@@ -278,6 +281,7 @@ t/filter_rubylang.t
 t/filter_subjecttag.t
 t/filter_vger.t
 t/gcf2.t
+t/gcf2_client.t
 t/git-http-backend.psgi
 t/git.fast-import-data
 t/git.t
diff --git a/Makefile.PL b/Makefile.PL
index 3fe9acf8..5a268362 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -71,6 +71,11 @@ $v->{gz_docs} = [ map { "$_.gz" } (@{$v->{docs}},@{$v->{docs_html}}) ];
 $v->{rsync_docs} = [ @{$v->{gz_docs}}, @{$v->{docs}},
 	@{$v->{docs_html}}, qw(NEWS.atom NEWS.atom.gz)];
 
+# filter out public-inbox-gcf2 from the website, it's an internal command
+for my $var (qw(gz_docs rsync_docs)) {
+	@{$v->{$var}} = grep(!/-gcf2/, @{$v->{$var}});
+}
+
 # external manpages which we host ourselves, since some packages
 # (currently just Xapian) doesn't host manpages themselves.
 my @xman = qw(copydatabase.1 xapian-compact.1);
diff --git a/lib/PublicInbox/Gcf2Client.pm b/lib/PublicInbox/Gcf2Client.pm
new file mode 100644
index 00000000..71fbb1d1
--- /dev/null
+++ b/lib/PublicInbox/Gcf2Client.pm
@@ -0,0 +1,35 @@
+# Copyright (C) 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+package PublicInbox::Gcf2Client;
+use strict;
+use parent 'PublicInbox::Git';
+use PublicInbox::Spawn qw(popen_rd);
+use IO::Handle ();
+
+sub new {
+	my $self = shift->SUPER::new('/nonexistent');
+	my ($out_r, $out_w);
+	pipe($out_r, $out_w) or $self->fail("pipe failed: $!");
+	my $cmd = [ 'public-inbox-gcf2' ];
+	@$self{qw(in pid)} = popen_rd($cmd, undef, { 0 => $out_r });
+	$self->{inflight} = [];
+	$self->{out} = $out_w;
+	fcntl($out_w, 1031, 4096) if $^O eq 'linux'; # 1031: F_SETPIPE_SZ
+	$out_w->autoflush(1);
+	$self;
+}
+
+sub add_git_dir {
+	my ($self, $git_dir) = @_;
+
+	# ensure buffers are drained, length($git_dir) may exceed
+	# PIPE_BUF on platforms where PIPE_BUF is only 512 bytes
+	my $inflight = $self->{inflight};
+	while (scalar(@$inflight)) {
+		$self->cat_async_step($inflight);
+	}
+	print { $self->{out} } $git_dir, "\n" or
+				$self->fail("write error: $!");
+}
+
+1;
diff --git a/script/public-inbox-gcf2 b/script/public-inbox-gcf2
new file mode 100755
index 00000000..51811698
--- /dev/null
+++ b/script/public-inbox-gcf2
@@ -0,0 +1,14 @@
+#!perl -w
+# Copyright (C) 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+eval { require PublicInbox::Gcf2 };
+die "libgit2 development package or Inline::C missing for $0: $@\n" if $@;
+my $gcf2 = PublicInbox::Gcf2::new();
+while (<STDIN>) {
+	chomp;
+	if (m!\A/!) { # +/path/to/git-dir
+		$gcf2->add_alternate("$_/objects");
+	} else {
+		$gcf2->cat_oid(1, $_);
+	}
+}
diff --git a/t/gcf2_client.t b/t/gcf2_client.t
new file mode 100644
index 00000000..39f9f296
--- /dev/null
+++ b/t/gcf2_client.t
@@ -0,0 +1,47 @@
+#!perl -w
+# Copyright (C) 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use PublicInbox::TestCommon;
+use Test::More;
+use Cwd qw(getcwd);
+use PublicInbox::Import;
+
+require_mods('PublicInbox::Gcf2');
+use_ok 'PublicInbox::Gcf2Client';
+my ($tmpdir, $for_destroy) = tmpdir();
+PublicInbox::Import::init_bare($tmpdir);
+my $fi_data = './t/git.fast-import-data';
+my $rdr = {};
+open $rdr->{0}, '<', $fi_data or BAIL_OUT $!;
+xsys([qw(git fast-import --quiet)], { GIT_DIR => $tmpdir }, $rdr);
+is($?, 0, 'fast-import succeeded');
+
+my $tree = 'fdbc43725f21f485051c17463b50185f4c3cf88c';
+my $called = 0;
+{
+	local $ENV{PATH} = getcwd()."/blib/script:$ENV{PATH}";
+	my $gcf2c = PublicInbox::Gcf2Client->new;
+	$gcf2c->add_git_dir($tmpdir);
+	$gcf2c->cat_async($tree, sub {
+		my ($bref, $oid, $type, $size, $arg) = @_;
+		is($oid, $tree, 'got expected OID');
+		is($size, 30, 'got expected length');
+		is($type, 'tree', 'got tree type');
+		is(length($$bref), 30, 'got a tree');
+		is($arg, 'hi', 'arg passed');
+		$called++;
+	}, 'hi');
+	my $trunc = substr($tree, 0, 39);
+	$gcf2c->cat_async($trunc, sub {
+		my ($bref, $oid, $type, $size, $arg) = @_;
+		is(undef, $bref, 'missing bref is undef');
+		is($oid, $trunc, 'truncated OID printed');
+		is($type, 'missing', 'type is "missing"');
+		is($size, undef, 'size is undef');
+		is($arg, 'bye', 'arg passed when missing');
+		$called++;
+	}, 'bye');
+}
+is($called, 2, 'cat_async callbacks hit');
+done_testing;

^ permalink raw reply related	[relevance 5%]

Results 1-2 of 2 | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2020-09-19  9:37  7% [PATCH 0/7] gcf2: libgit2-based cat-file alternative Eric Wong
2020-09-19  9:37  5% ` [PATCH 3/7] add gcf2 client and executable script Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).