From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Subject: [PATCH 6/6] upload-pack: provide a hook for running pack-objects
Date: Wed, 18 May 2016 18:45:37 -0400 [thread overview]
Message-ID: <20160518224537.GF22443@sigill.intra.peff.net> (raw)
In-Reply-To: <20160518223712.GA18317@sigill.intra.peff.net>
When upload-pack serves a client request, it turns to
pack-objects to do the heavy lifting of creating a
packfile. There's no easy way to intercept the call to
pack-objects, but there are a few good reasons to want to do
so:
1. If you're debugging a client or server issue with
fetching, you may want to store a copy of the generated
packfile.
2. If you're gathering data from real-world fetches for
performance analysis or debugging, storing a copy of
the arguments and stdin lets you replay the pack
generation at your leisure.
3. You may want to insert a caching layer around
pack-objects; it is the most CPU- and memory-intensive
part of serving a fetch, and its output is a pure
function[1] of its input, making it an ideal place to
consolidate identical requests.
This patch adds a simple "hook" interface to intercept calls
to pack-objects. The new test demonstrates how it can be
used for debugging (using it for caching is a
straightforward extension; the tricky part is writing the
actual caching layer).
This hook is unlike the normal hook scripts found in the
"hooks/" directory of a repository. Because we promise that
upload-pack is safe to run in an untrusted repository, we
cannot execute arbitrary code or commands found in the
repository (neither in hooks/, nor in the config). So
instead, this hook is triggered from a config variable that
is explicitly ignored in the per-repo config.
The config variable holds the actual shell command to run as
the hook. Another approach would be to simply treat it as a
boolean: "should I respect the upload-pack hooks in this
repo?", and then run the script from "hooks/" as we usually
do. However, that isn't as flexible; there's no way to run a
hook approved by the site administrator (e.g., in
"/etc/gitconfig") on a repository whose contents are not
trusted. The approach taken by this patch is more
fine-grained, if a little less conventional for git hooks
(it does behave similar to other configured commands like
diff.external, etc).
[1] Pack-objects isn't _actually_ a pure function. Its
output depends on the exact packing of the object
database, and if multi-threading is used for delta
compression, can even differ racily. But for the
purposes of caching, that's OK; of the many possible
outputs for a given input, it is sufficient only that we
output one of them.
Signed-off-by: Jeff King <peff@peff.net>
---
Documentation/config.txt | 15 +++++++++++
t/t5544-pack-objects-hook.sh | 62 ++++++++++++++++++++++++++++++++++++++++++++
upload-pack.c | 13 +++++++++-
3 files changed, 89 insertions(+), 1 deletion(-)
create mode 100755 t/t5544-pack-objects-hook.sh
diff --git a/Documentation/config.txt b/Documentation/config.txt
index e4cd291..b9b0541 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2872,6 +2872,21 @@ uploadpack.keepAlive::
`uploadpack.keepAlive` seconds. Setting this option to 0
disables keepalive packets entirely. The default is 5 seconds.
+uploadpack.packObjectsHook::
+ If this option is set, when `upload-pack` would run
+ `git pack-objects` to create a packfile for a client, it will
+ run this shell command instead. The `pack-objects` command and
+ arguments it _would_ have run (including the `git pack-objects`
+ at the beginning) are appended to the shell command. The stdin
+ and stdout of the hook are treated as if `pack-objects` itself
+ was run. I.e., `upload-pack` will feed input intended for
+ `pack-objects` to the hook, and expects a completed packfile on
+ stdout.
++
+Note that this configuration variable is ignored if it is seen in the
+repository-level config (this is a safety measure against fetching from
+untrusted repositories).
+
url.<base>.insteadOf::
Any URL that starts with this value will be rewritten to
start, instead, with <base>. In cases where some site serves a
diff --git a/t/t5544-pack-objects-hook.sh b/t/t5544-pack-objects-hook.sh
new file mode 100755
index 0000000..4357af1
--- /dev/null
+++ b/t/t5544-pack-objects-hook.sh
@@ -0,0 +1,62 @@
+#!/bin/sh
+
+test_description='test custom script in place of pack-objects'
+. ./test-lib.sh
+
+test_expect_success 'create some history to fetch' '
+ test_commit one &&
+ test_commit two
+'
+
+test_expect_success 'create debugging hook script' '
+ write_script .git/hook <<-\EOF
+ echo >&2 "hook running"
+ echo "$*" >hook.args
+ cat >hook.stdin
+ "$@" <hook.stdin >hook.stdout
+ cat hook.stdout
+ EOF
+'
+
+clear_hook_results () {
+ rm -rf .git/hook.* dst.git
+}
+
+test_expect_success 'hook runs via global config' '
+ clear_hook_results &&
+ test_config_global uploadpack.packObjectsHook ./hook &&
+ git clone --no-local . dst.git 2>stderr &&
+ grep "hook running" stderr
+'
+
+test_expect_success 'hook outputs are sane' '
+ # check that we recorded a usable pack
+ git index-pack --stdin <.git/hook.stdout &&
+
+ # check that we recorded args and stdin. We do not check
+ # the full argument list or the exact pack contents, as it would make
+ # the test brittle. So just sanity check that we could replay
+ # the packing procedure.
+ grep "^git" .git/hook.args &&
+ $(cat .git/hook.args) <.git/hook.stdin >replay
+'
+
+test_expect_success 'hook runs from -c config' '
+ clear_hook_results &&
+ git clone --no-local \
+ -u "git -c uploadpack.packObjectsHook=./hook upload-pack" \
+ . dst.git 2>stderr &&
+ grep "hook running" stderr
+'
+
+test_expect_success 'hook does not run from repo config' '
+ clear_hook_results &&
+ test_config uploadpack.packObjectsHook "./hook" &&
+ git clone --no-local . dst.git 2>stderr &&
+ ! grep "hook running" stderr &&
+ test_path_is_missing .git/hook.args &&
+ test_path_is_missing .git/hook.stdin &&
+ test_path_is_missing .git/hook.stdout
+'
+
+test_done
diff --git a/upload-pack.c b/upload-pack.c
index f19444d..8979be6 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -52,6 +52,7 @@ static int keepalive = 5;
static int use_sideband;
static int advertise_refs;
static int stateless_rpc;
+static const char *pack_objects_hook;
static void reset_timeout(void)
{
@@ -93,6 +94,14 @@ static void create_pack_file(void)
int i;
FILE *pipe_fd;
+ if (!pack_objects_hook)
+ pack_objects.git_cmd = 1;
+ else {
+ argv_array_push(&pack_objects.args, pack_objects_hook);
+ argv_array_push(&pack_objects.args, "git");
+ pack_objects.use_shell = 1;
+ }
+
if (shallow_nr) {
argv_array_push(&pack_objects.args, "--shallow-file");
argv_array_push(&pack_objects.args, "");
@@ -115,7 +124,6 @@ static void create_pack_file(void)
pack_objects.in = -1;
pack_objects.out = -1;
pack_objects.err = -1;
- pack_objects.git_cmd = 1;
if (start_command(&pack_objects))
die("git upload-pack: unable to fork git-pack-objects");
@@ -812,6 +820,9 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
keepalive = git_config_int(var, value);
if (!keepalive)
keepalive = -1;
+ } else if (current_config_scope() != CONFIG_SCOPE_REPO) {
+ if (!strcmp("uploadpack.packobjectshook", var))
+ return git_config_string(&pack_objects_hook, var, value);
}
return parse_hide_refs_config(var, value, "uploadpack");
}
--
2.8.2.888.gecb1fe3
next prev parent reply other threads:[~2016-05-18 22:45 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-18 22:37 [PATCH/RFC 0/6] pack-objects hook for upload-pack Jeff King
2016-05-18 22:39 ` [PATCH 1/6] git_config_with_options: drop "found" counting Jeff King
2016-05-18 22:39 ` [PATCH 2/6] git_config_parse_parameter: refactor cleanup code Jeff King
2016-05-18 22:41 ` [PATCH 3/6] config: set up config_source for command-line config Jeff King
2016-05-18 22:43 ` [PATCH 4/6] config: return configset value for current_config_ functions Jeff King
2016-05-19 0:08 ` Jeff King
2016-05-26 7:47 ` Duy Nguyen
2016-05-26 16:42 ` Junio C Hamano
2016-05-26 16:50 ` Jeff King
2016-05-26 17:36 ` Junio C Hamano
2016-05-27 0:41 ` Jeff King
2016-05-27 2:11 ` Junio C Hamano
2016-05-27 0:32 ` Jeff King
2016-05-18 22:44 ` [PATCH 5/6] config: add a notion of "scope" Jeff King
2016-05-18 22:45 ` Jeff King [this message]
2016-05-19 0:14 ` [PATCH 6/6] upload-pack: provide a hook for running pack-objects Jeff King
2016-05-19 10:12 ` Ævar Arnfjörð Bjarmason
2016-05-19 12:08 ` Jeff King
2016-05-19 14:54 ` Ævar Arnfjörð Bjarmason
2016-05-26 5:37 ` Jeff King
2016-05-25 0:59 ` [PATCH/RFC 0/6] pack-objects hook for upload-pack Junio C Hamano
2016-05-26 5:44 ` Jeff King
2016-05-26 16:44 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160518224537.GF22443@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).