[PATCH] archive: add archive.restrictRemote option

[PATCH] archive: add archive.restrictRemote option

[Jeff King]

From: Scott J. Goldman <scottjg@github.com>

In commit ee27ca4, we started restricting remote git-archive
invocations to only accessing reachable commits. This
matches what upload-pack allows, but does restrict some
useful cases (e.g., HEAD:foo). We loosened this in 0f544ee,
which allows `foo:bar` as long as `foo` is a ref tip.
However, that still doesn't allow many useful things, like:

  1. Commits accessible from a ref, like `foo^:bar`, which
     is reachable

  2. Arbitrary sha1s, even if they are reachable.

We can do a full object-reachability check for these cases,
but it can be quite expensive if the client has sent us the
sha1 of a tree; we have to visit every sub-tree of every
commit in the worst case.

Let's instead give site admins an escape hatch, in case they
prefer the more liberal behavior.  For many sites, the full
object database is public anyway (e.g., if you allow dumb
walker access), or the site admin may simply decide the
security/convenience tradeoff is not worth it.

This patch adds a new config option to turn off the
restrictions added in ee27ca4. It defaults to on, meaning
there is no change in behavior by default.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-archive.txt |  7 +++++++
 archive.c                     | 13 +++++++++++--
 t/t5000-tar-tree.sh           |  9 +++++++++
 3 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index b97aaab..486cb08 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -121,6 +121,13 @@ tar.<format>.remote::
 	user-defined formats, but true for the "tar.gz" and "tgz"
 	formats.
 
+archive.restrictRemote::
+	If true, archives can only be requested by refnames. If false,
+	allows remote archive requests from arbitrary revisions. This
+	can be a security hazard, as archives could be requested for
+	unreachable commits that have been pruned from the repository.
+	Defaults to true.
+
 [[ATTRIBUTES]]
 ATTRIBUTES
 ----------
diff --git a/archive.c b/archive.c
index 346f3b2..13eb23a 100644
--- a/archive.c
+++ b/archive.c
@@ -17,6 +17,7 @@ static char const * const archive_usage[] = {
 static const struct archiver **archivers;
 static int nr_archivers;
 static int alloc_archivers;
+static int git_archive_restrict_remote = 1;
 
 void register_archiver(struct archiver *ar)
 {
@@ -257,7 +258,7 @@ static void parse_treeish_arg(const char **argv,
 	unsigned char sha1[20];
 
 	/* Remotes are only allowed to fetch actual refs */
-	if (remote) {
+	if (remote && git_archive_restrict_remote) {
 		char *ref = NULL;
 		const char *colon = strchr(name, ':');
 		int refnamelen = colon ? colon - name : strlen(name);
@@ -401,6 +402,14 @@ static int parse_archive_args(int argc, const char **argv,
 	return argc;
 }
 
+static int git_default_archive_config(const char *var, const char *value,
+				      void *cb)
+{
+	if (!strcmp(var, "archive.restrictremote"))
+		git_archive_restrict_remote = git_config_bool(var, value);
+	return git_default_config(var, value, cb);
+}
+
 int write_archive(int argc, const char **argv, const char *prefix,
 		  int setup_prefix, const char *name_hint, int remote)
 {
@@ -411,7 +420,7 @@ int write_archive(int argc, const char **argv, const char *prefix,
 	if (setup_prefix && prefix == NULL)
 		prefix = setup_git_directory_gently(&nongit);
 
-	git_config(git_default_config, NULL);
+	git_config(git_default_archive_config, NULL);
 	init_tar_archiver();
 	init_zip_archiver();
 
diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
index 05f011d..eba285f 100755
--- a/t/t5000-tar-tree.sh
+++ b/t/t5000-tar-tree.sh
@@ -213,6 +213,15 @@ test_expect_success 'clients cannot access unreachable commits' '
 	test_must_fail git archive --remote=. $sha1 >remote.tar
 '
 
+test_expect_success 'clients can access unreachable commits' '
+	test_commit unreachable1 &&
+	sha1=`git rev-parse HEAD` &&
+	git reset --hard HEAD^ &&
+	git archive $sha1 >remote.tar &&
+	test_config archive.restrictRemote false &&
+	git archive --remote=. $sha1 >remote.tar
+'
+
 test_expect_success 'setup tar filters' '
 	git config tar.tar.foo.command "tr ab ba" &&
 	git config tar.bar.command "tr ab ba" &&
-- 
1.8.5.2.500.g8060133

Re: [PATCH] archive: add archive.restrictRemote option

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> From: Scott J. Goldman <scottjg@github.com>
>
> In commit ee27ca4, we started restricting remote git-archive
> invocations to only accessing reachable commits. This
> matches what upload-pack allows, but does restrict some
> useful cases (e.g., HEAD:foo). We loosened this in 0f544ee,
> which allows `foo:bar` as long as `foo` is a ref tip.
> However, that still doesn't allow many useful things, like:
>
>   1. Commits accessible from a ref, like `foo^:bar`, which
>      is reachable
>
>   2. Arbitrary sha1s, even if they are reachable.
>
> We can do a full object-reachability check for these cases,
> but it can be quite expensive if the client has sent us the
> sha1 of a tree; we have to visit every sub-tree of every
> commit in the worst case.
>
> Let's instead give site admins an escape hatch, in case they
> prefer the more liberal behavior.  For many sites, the full
> object database is public anyway (e.g., if you allow dumb
> walker access), or the site admin may simply decide the
> security/convenience tradeoff is not worth it.
>
> This patch adds a new config option to turn off the
> restrictions added in ee27ca4. It defaults to on, meaning
> there is no change in behavior by default.
>
> Signed-off-by: Jeff King <peff@peff.net>

Thanks.

Do GitHub people have general aversion against signing off (or
sending out, for that matter) their own patches, unless they were
already active here before they joined GitHub, by the way?

I like the general idea and this escape hatch would be a good thing
to have.

A few comments:

 - Seeing the word combination "restrict"+"remote" before reading
   the explanation made me think "hmph, only allow remote archive
   from certain hosts but not from others?"  We are restricting the
   objects and only on the remote usage, not restricting remotes, so
   somebody else may be able to come up with a less misleading name.

 - It might be better to call the escape hatch "allow something"
   that defaults to "false".  It is merely the issue of perception,
   but having a knob to be limiting that defaults to true gives a
   wrong impression that in an ideal world remote archive ought to
   be loose and we are artificially limiting it by default.

But these are just my "reactions"; neither is an objection to the
posted patch as-is.

> ---
>  Documentation/git-archive.txt |  7 +++++++
>  archive.c                     | 13 +++++++++++--
>  t/t5000-tar-tree.sh           |  9 +++++++++
>  3 files changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
> index b97aaab..486cb08 100644
> --- a/Documentation/git-archive.txt
> +++ b/Documentation/git-archive.txt
> @@ -121,6 +121,13 @@ tar.<format>.remote::
>  	user-defined formats, but true for the "tar.gz" and "tgz"
>  	formats.
>  
> +archive.restrictRemote::
> +	If true, archives can only be requested by refnames. If false,

As this does not affect local use of "git archive", "requested by
refnames" may need to be clarified further.  Perhaps "remote
archives can be requested only for published refnames" or something.

Just to help starting further discussion to pick brains of others,
this paragraph could have been like this, I would think.

    archive.serveArbitraryObjectToRemote::

        By default, remote archives can be requested only for
        published refnames (e.g. "git archive --remote=origin
        master" is OK, but "git archive --remote=origin ae9677f" is
        not), to prevent peeking into unreachable commits that have
        been pruned from the repository.  This configuration
        variable can be set to bypass this security measure.

The phrase "serve arbitrary object to remote" would reflect the
purpose of the escape hatch better, I would think, but it is not a
great short-and-sweet name.

Thanks.

Re: [PATCH] archive: add archive.restrictRemote option

[Jeff King]

On Thu, Feb 27, 2014 at 10:37:30AM -0800, Junio C Hamano wrote:

> > Signed-off-by: Jeff King <peff@peff.net>
> 
> Thanks.
> 
> Do GitHub people have general aversion against signing off (or
> sending out, for that matter) their own patches, unless they were
> already active here before they joined GitHub, by the way?

Mostly it is that I clean up the patches and commit messages before
sending them out. Michael sends out his own patches because they are
already perfect by the time I see them. :)

I can certainly get S-O-B from GitHubbers, but my impression of the DCO
is that it does not matter; as the first link in the signoff chain, I am
certifying that the patch meets the licensing requirements. Of course, I
know that because of my relationship to the author and our employer,
which is something that isn't encoded in the headers. A S-O-B from the
author would perhaps make it more obvious what happened.

> I like the general idea and this escape hatch would be a good thing
> to have.
> 
> A few comments:
> 
>  - Seeing the word combination "restrict"+"remote" before reading
>    the explanation made me think "hmph, only allow remote archive
>    from certain hosts but not from others?"  We are restricting the
>    objects and only on the remote usage, not restricting remotes, so
>    somebody else may be able to come up with a less misleading name.
>
>  - It might be better to call the escape hatch "allow something"
>    that defaults to "false".  It is merely the issue of perception,
>    but having a knob to be limiting that defaults to true gives a
>    wrong impression that in an ideal world remote archive ought to
>    be loose and we are artificially limiting it by default.

After reading your first point, I came up with
"archive.allowRemoteUnreachable", which also satisfies the second. I do
not have a strong opinion.

> > +archive.restrictRemote::
> > +	If true, archives can only be requested by refnames. If false,
> 
> As this does not affect local use of "git archive", "requested by
> refnames" may need to be clarified further.  Perhaps "remote
> archives can be requested only for published refnames" or something.

I was hoping to be vague. If we really want to get into specifics, we
should probably document the current rules (refnames, and sub-trees of
refnames). It might be a good thing to document that anyway, though. And
by doing so, it would become obvious why one would want to set this
option. I'll see what I can come up with.

-Peff

[PATCH v2 0/2] lifting upload-archive restrictions

[Jeff King]

On Fri, Feb 28, 2014 at 04:07:09AM -0500, Jeff King wrote:

> > As this does not affect local use of "git archive", "requested by
> > refnames" may need to be clarified further.  Perhaps "remote
> > archives can be requested only for published refnames" or something.
> 
> I was hoping to be vague. If we really want to get into specifics, we
> should probably document the current rules (refnames, and sub-trees of
> refnames). It might be a good thing to document that anyway, though. And
> by doing so, it would become obvious why one would want to set this
> option. I'll see what I can come up with.

Here's a series to handle this; I think the end result is much nicer. I
ended up calling the option "uploadarchive.allowUnreachable". By moving
it there instead of under "archive", it is more clear that this is about
the _serving_ end of the remote connection, and the word "remote"
becomes redundant.

  [1/2]: docs: clarify remote restrictions for git-upload-archive
  [2/2]: add uploadarchive.allowUnreachable option

-Peff

[PATCH v2 1/2] docs: clarify remote restrictions for git-upload-archive

[Jeff King]

Commits ee27ca4 and 0f544ee introduced rules by which
git-upload-archive would restrict clients from accessing
unreachable objects. However, we never documented those
rules anywhere, nor their reason for being. Let's do so now.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-archive.txt        |  5 ++++-
 Documentation/git-upload-archive.txt | 26 ++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index b97aaab..cfa1e4e 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -65,7 +65,10 @@ OPTIONS
 
 --remote=<repo>::
 	Instead of making a tar archive from the local repository,
-	retrieve a tar archive from a remote repository.
+	retrieve a tar archive from a remote repository. Note that the
+	remote repository may place restrictions on which sha1
+	expressions may be allowed in `<tree-ish>`. See
+	linkgit:git-upload-archive[1] for details.
 
 --exec=<git-upload-archive>::
 	Used with --remote to specify the path to the
diff --git a/Documentation/git-upload-archive.txt b/Documentation/git-upload-archive.txt
index d09bbb5..8ae65d8 100644
--- a/Documentation/git-upload-archive.txt
+++ b/Documentation/git-upload-archive.txt
@@ -20,6 +20,32 @@ This command is usually not invoked directly by the end user.  The UI
 for the protocol is on the 'git archive' side, and the program pair
 is meant to be used to get an archive from a remote repository.
 
+SECURITY
+--------
+
+In order to protect the privacy of objects that have been removed from
+history but may not yet have been pruned, `git-upload-archive` avoids
+serving archives for commits and trees that are not reachable from the
+repository's refs.  However, because calculating object reachability is
+computationally expensive, `git-upload-archive` implements a stricter
+but easier-to-check set of rules:
+
+  1. Clients may request a commit or tree that is pointed to directly by
+     a ref. E.g., `git archive --remote=origin v1.0`.
+
+  2. Clients may request a sub-tree within a commit or tree using the
+     `ref:path` syntax. E.g., `git archive --remote=origin v1.0:Documentation`.
+
+  3. Clients may _not_ use other sha1 expressions, even if the end
+     result is reachable. E.g., neither a relative commit like `master^`
+     nor a literal sha1 like `abcd1234` is allowed, even if the result
+     is reachable from the refs.
+
+Note that rule 3 disallows many cases that do not have any privacy
+implications. These rules are subject to change in future versions of
+git, and the server accessed by `git archive --remote` may or may not
+follow these exact rules.
+
 OPTIONS
 -------
 <directory>::
-- 
1.8.5.2.500.g8060133

[PATCH v2 2/2] add uploadarchive.allowUnreachable option

[Jeff King]

From: Scott J. Goldman <scottjg@github.com>

In commit ee27ca4, we started restricting remote git-archive
invocations to only accessing reachable commits. This
matches what upload-pack allows, but does restrict some
useful cases (e.g., HEAD:foo). We loosened this in 0f544ee,
which allows `foo:bar` as long as `foo` is a ref tip.
However, that still doesn't allow many useful things, like:

  1. Commits accessible from a ref, like `foo^:bar`, which
     are reachable

  2. Arbitrary sha1s, even if they are reachable.

We can do a full object-reachability check for these cases,
but it can be quite expensive if the client has sent us the
sha1 of a tree; we have to visit every sub-tree of every
commit in the worst case.

Let's instead give site admins an escape hatch, in case they
prefer the more liberal behavior.  For many sites, the full
object database is public anyway (e.g., if you allow dumb
walker access), or the site admin may simply decide the
security/convenience tradeoff is not worth it.

This patch adds a new config option to disable the
restrictions added in ee27ca4. It defaults to off, meaning
there is no change in behavior by default.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/config.txt             |  7 +++++++
 Documentation/git-upload-archive.txt |  6 ++++++
 archive.c                            | 13 +++++++++++--
 t/t5000-tar-tree.sh                  |  9 +++++++++
 4 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 040197b..62f0a4e 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2334,6 +2334,13 @@ transfer.unpackLimit::
 	not set, the value of this variable is used instead.
 	The default value is 100.
 
+uploadarchive.allowUnreachable::
+	If true, allow clients to use `git archive --remote` to request
+	any tree, whether reachable from the ref tips or not. See the
+	discussion in the `SECURITY` section of
+	linkgit:git-upload-archive[1] for more details. Defaults to
+	`false`.
+
 uploadpack.hiderefs::
 	String(s) `upload-pack` uses to decide which refs to omit
 	from its initial advertisement.  Use more than one
diff --git a/Documentation/git-upload-archive.txt b/Documentation/git-upload-archive.txt
index 8ae65d8..cbef61b 100644
--- a/Documentation/git-upload-archive.txt
+++ b/Documentation/git-upload-archive.txt
@@ -46,6 +46,12 @@ implications. These rules are subject to change in future versions of
 git, and the server accessed by `git archive --remote` may or may not
 follow these exact rules.
 
+If the config option `uploadArchive.allowUnreachable` is true, these
+rules are ignored, and clients may use arbitrary sha1 expressions.
+This is useful if you do not care about the privacy of unreachable
+objects, or if your object database is already publicly available for
+access via non-smart-http.
+
 OPTIONS
 -------
 <directory>::
diff --git a/archive.c b/archive.c
index 346f3b2..7d0976f 100644
--- a/archive.c
+++ b/archive.c
@@ -17,6 +17,7 @@ static char const * const archive_usage[] = {
 static const struct archiver **archivers;
 static int nr_archivers;
 static int alloc_archivers;
+static int remote_allow_unreachable;
 
 void register_archiver(struct archiver *ar)
 {
@@ -257,7 +258,7 @@ static void parse_treeish_arg(const char **argv,
 	unsigned char sha1[20];
 
 	/* Remotes are only allowed to fetch actual refs */
-	if (remote) {
+	if (remote && !remote_allow_unreachable) {
 		char *ref = NULL;
 		const char *colon = strchr(name, ':');
 		int refnamelen = colon ? colon - name : strlen(name);
@@ -401,6 +402,14 @@ static int parse_archive_args(int argc, const char **argv,
 	return argc;
 }
 
+static int git_default_archive_config(const char *var, const char *value,
+				      void *cb)
+{
+	if (!strcmp(var, "uploadarchive.allowunreachable"))
+		remote_allow_unreachable = git_config_bool(var, value);
+	return git_default_config(var, value, cb);
+}
+
 int write_archive(int argc, const char **argv, const char *prefix,
 		  int setup_prefix, const char *name_hint, int remote)
 {
@@ -411,7 +420,7 @@ int write_archive(int argc, const char **argv, const char *prefix,
 	if (setup_prefix && prefix == NULL)
 		prefix = setup_git_directory_gently(&nongit);
 
-	git_config(git_default_config, NULL);
+	git_config(git_default_archive_config, NULL);
 	init_tar_archiver();
 	init_zip_archiver();
 
diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
index 05f011d..1cf0a4e 100755
--- a/t/t5000-tar-tree.sh
+++ b/t/t5000-tar-tree.sh
@@ -213,6 +213,15 @@ test_expect_success 'clients cannot access unreachable commits' '
 	test_must_fail git archive --remote=. $sha1 >remote.tar
 '
 
+test_expect_success 'upload-archive can allow unreachable commits' '
+	test_commit unreachable1 &&
+	sha1=`git rev-parse HEAD` &&
+	git reset --hard HEAD^ &&
+	git archive $sha1 >remote.tar &&
+	test_config uploadarchive.allowUnreachable true &&
+	git archive --remote=. $sha1 >remote.tar
+'
+
 test_expect_success 'setup tar filters' '
 	git config tar.tar.foo.command "tr ab ba" &&
 	git config tar.bar.command "tr ab ba" &&
-- 
1.8.5.2.500.g8060133

Re: [PATCH] archive: add archive.restrictRemote option

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Feb 27, 2014 at 10:37:30AM -0800, Junio C Hamano wrote:
>
>> > Signed-off-by: Jeff King <peff@peff.net>
>> 
>> Thanks.
>> 
>> Do GitHub people have general aversion against signing off (or
>> sending out, for that matter) their own patches, unless they were
>> already active here before they joined GitHub, by the way?
>
> Mostly it is that I clean up the patches and commit messages before
> sending them out. Michael sends out his own patches because they are
> already perfect by the time I see them. :)
>
> I can certainly get S-O-B from GitHubbers, but my impression of the DCO
> is that it does not matter...
> ... A S-O-B from the
> author would perhaps make it more obvious what happened.

Oh, I was not saying the practice was not legit.  It was just that I
expected a bit more from GitHub, a leading company that evangelises
use of Git ;-)

> I was hoping to be vague. If we really want to get into specifics, we
> should probably document the current rules (refnames, and sub-trees of
> refnames). It might be a good thing to document that anyway, though. And
> by doing so, it would become obvious why one would want to set this
> option. I'll see what I can come up with.

Thanks.

Re: [PATCH v2 0/2] lifting upload-archive restrictions

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Here's a series to handle this; I think the end result is much nicer. I
> ended up calling the option "uploadarchive.allowUnreachable". By moving
> it there instead of under "archive", it is more clear that this is about
> the _serving_ end of the remote connection, and the word "remote"
> becomes redundant.

Yeah, the original was already good but I like this version much
better.  I'm glad I bikeshedded ;-)

Thanks.