[PATCH] http.c: shell command evaluation for extraheader

[PATCH] http.c: shell command evaluation for extraheader

[Colin Arnott]

The http.extraHeader config parameter currently only supports storing
constant values. There are two main use cases where this fails:

  0. Sensitive payloads: frequently this config parameter is used to pass
     authentication credentials in place of or in addition to the
     Authorization header, however since this value is required to be in
     the clear this can create security issues.

  1. Mutating headers: some headers, especially new authentication
     schemes, leverage short lived tokens that change over time.

There do exist solutions with current tools for these use cases, however
none are optimal:

  0. Shell alias: by aliasing over git with a call to git that includes the
     config directive and evaluates the header value inline, you can
     fake the desired mutability:
       `alias git='git -c http.extraHeader="$(gpg -d crypt.gpg)"'`
     This presents two problems:
     a. aliasing over commands can be confusing to new users, since git
        config information is stored in shell configs
     b. this solution scales only to your shell, not all shells

  1. Global hook: you could implement a hook that writes the config
     entry before fetch / pull actions, so that it is up to date, but
     this does nothing to secure it.

  2. git-credential-helper: the credential helper interface already
     supports shelling out to arbitrary binaries or scripts, however
     this interface can only be used to populate the Authorization
     header.

The optimal solution involves extending the current implementation of
http.extraHeader parsing to allow for arbitrary shell command execution.
There seem to be two paradigms for such features:

  0. Overloading with '!' prefixes: seen in alias.* and credential.helper

  1. New "Cmd" suffix parameters: seen in sendemail.toCmd sendemail.ccCmd

While the latter may be more clear without documentation, the addition
of a new config parameter seemed more complex for the codebase. As such,
new documentation is included.

Several edge cases came up during implementation and the following
design decisions were made:

  0. Stdin and stderr for the child_process are exposed to the user:
     this allows commands that print status information via stderr, and
     accept input to function. The use case considered is text input for
     decryption and error handling that is out of scope for git.

  1. Failure to exec: if either the file does not exist, or any other
     exec related failure occurs, no error is presented to the user,
     and the header is not included

  2. Non-zero return code: if the child_process returns a non-zero
     value, no error is presented to the user, the return value is
     consumed, and the header is not included in the request.

  3. Headers starting with the '!' character require a shell command to
     create: because no escaping syntax was implemented, the following
     is required for such headers: "!printf '!magic: abra'"

Signed-off-by: Colin Arnott <colin@urandom.co.uk>
---
 Documentation/config.txt    |  7 +++++++
 http.c                      | 20 ++++++++++++++++++++
 t/t5551-http-fetch-smart.sh |  6 ++++--
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index f57e9cf10..4b2171d60 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1918,6 +1918,13 @@ http.extraHeader::
 	more than one such entry exists, all of them are added as extra
 	headers.  To allow overriding the settings inherited from the system
 	config, an empty value will reset the extra headers to the empty list.
+	If the value is prefixed with an exclamation point, it will
+	be treated as a shell command.  For example, defining
+	"http.extraHeader = !gpg -d < secure_header.gpg", will pass the
+	decrypted header, if the command does not exec cleanly or has a
+	non-zero return value, no header will be added.  Note that shell
+	commands will be executed from the top-level directory of a
+	repository, which may not necessarily be the current directory.
 
 http.cookieFile::
 	The pathname of a file containing previously stored cookie lines,
diff --git a/http.c b/http.c
index 31755023a..11103df41 100644
--- a/http.c
+++ b/http.c
@@ -380,6 +380,26 @@ static int http_options(const char *var, const char *value, void *cb)
 		} else if (!*value) {
 			curl_slist_free_all(extra_http_headers);
 			extra_http_headers = NULL;
+		} else if (value[0] == '!') {
+			struct child_process cp = CHILD_PROCESS_INIT;
+			cp.git_cmd = 0;
+			cp.in = 0;
+			cp.out = -1;
+			cp.err = 0;
+			cp.use_shell = 1;
+			argv_array_push(&cp.args, value + 1);
+			if (!start_command(&cp)) {
+				struct strbuf output;
+				strbuf_init(&output, 0);
+				strbuf_read(&output, cp.out, 0);
+				close(cp.out);
+				cp.out = -1;
+				if (!finish_command(&cp)) {
+					extra_http_headers =
+						curl_slist_append(extra_http_headers, output.buf);
+				}
+				strbuf_release(&output);
+			}
 		} else {
 			extra_http_headers =
 				curl_slist_append(extra_http_headers, value);
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index f5721b4a5..039afc76a 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -351,7 +351,9 @@ test_expect_success 'custom http headers' '
 	test_must_fail git -c http.extraheader="x-magic-two: cadabra" \
 		fetch "$HTTPD_URL/smart_headers/repo.git" &&
 	git -c http.extraheader="x-magic-one: abra" \
-	    -c http.extraheader="x-magic-two: cadabra" \
+	    -c http.extraheader="!printf \"x-magic-two: cadabra\"" \
+	    -c http.extraheader="!printf \"x-magic-three: alakazam; exit 2\"" \
+	    -c http.extraheader="!shellcommanddoesnotexist" \
 	    fetch "$HTTPD_URL/smart_headers/repo.git" &&
 	git update-index --add --cacheinfo 160000,$(git rev-parse HEAD),sub &&
 	git config -f .gitmodules submodule.sub.path sub &&
@@ -360,7 +362,7 @@ test_expect_success 'custom http headers' '
 	git submodule init sub &&
 	test_must_fail git submodule update sub &&
 	git -c http.extraheader="x-magic-one: abra" \
-	    -c http.extraheader="x-magic-two: cadabra" \
+	    -c http.extraheader="!printf \"x-magic-two: cadabra\"" \
 		submodule update sub
 '
 
-- 
2.16.2

Re: [PATCH] http.c: shell command evaluation for extraheader

[Johannes Schindelin]

Hi Colin,

On Sun, 4 Mar 2018, Colin Arnott wrote:

> The http.extraHeader config parameter currently only supports storing
> constant values. There are two main use cases where this fails:
> 
>   0. Sensitive payloads: frequently this config parameter is used to pass
>      authentication credentials in place of or in addition to the
>      Authorization header, however since this value is required to be in
>      the clear this can create security issues.
> 
>   1. Mutating headers: some headers, especially new authentication
>      schemes, leverage short lived tokens that change over time.
> 
> There do exist solutions with current tools for these use cases, however
> none are optimal:
> 
>   0. Shell alias: by aliasing over git with a call to git that includes the
>      config directive and evaluates the header value inline, you can
>      fake the desired mutability:
>        `alias git='git -c http.extraHeader="$(gpg -d crypt.gpg)"'`
>      This presents two problems:
>      a. aliasing over commands can be confusing to new users, since git
>         config information is stored in shell configs
>      b. this solution scales only to your shell, not all shells
> 
>   1. Global hook: you could implement a hook that writes the config
>      entry before fetch / pull actions, so that it is up to date, but
>      this does nothing to secure it.
> 
>   2. git-credential-helper: the credential helper interface already
>      supports shelling out to arbitrary binaries or scripts, however
>      this interface can only be used to populate the Authorization
>      header.

As the credential-helper is already intended for sensitive data, and as it
already allows to interact with a helper, I would strongly assume that it
would make more sense to try to extend that feature (instead of the simple
extraHeader one).

This would also help alleviate all the quoting/dequoting issues involved
with shell scripting.

Besides, the http.extraHeader feature was designed to accommodate all
kinds of extra headers, not only authentication ones (and indeed, the
authentication was only intended for use in build agents, where both
environment and logging can be controlled rather tightly).

I also see that in your implementation, only the extraHeader value is
evaluated, without any access to the rest of the metadata (such as URL,
and optionally specified user).

It would probably get a little more complicated than a shell script to
write a credential-helper that will always be asked to generate an
authentication, but I think even a moderate-level Perl script could be
used for that, and it *would* know the URL and user for which the
credentials are intended...

Ciao,
Johannes

Re: [PATCH] http.c: shell command evaluation for extraheader

[Colin Arnott]

Johannes,

On March 5, 2018 1:47 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> As the credential-helper is already intended for sensitive data, and as it
> already allows to interact with a helper, I would strongly assume that it
> would make more sense to try to extend that feature (instead of the simple
> extraHeader one).

To confirm you are suggesting that the credential struct, defined in credential.h, be extended to include a headers array, like so:
--- a/credential.h
+++ b/credential.h
@@ -15,6 +15,7 @@ struct credential {
        char *protocol;
        char *host;
        char *path;
+       char **headers
 };
 
 #define CREDENTIAL_INIT { STRING_LIST_INIT_DUP }

> This would also help alleviate all the quoting/dequoting issues involved
> with shell scripting.
> 
> Besides, the http.extraHeader feature was designed to accommodate all
> kinds of extra headers, not only authentication ones (and indeed, the
> authentication was only intended for use in build agents, where both
> environment and logging can be controlled rather tightly).

I realise that my examples are scoped for auth, but I can conceive of other mutating headers that are not explicitly authentication related, and could benefit from shell execution before fetch, pull, push actions.

> I also see that in your implementation, only the extraHeader value is
> evaluated, without any access to the rest of the metadata (such as URL,
> and optionally specified user).
>
> It would probably get a little more complicated than a shell script to
> write a credential-helper that will always be asked to generate an
> authentication, but I think even a moderate-level Perl script could be
> used for that, and it would know the URL and user for which the
> credentials are intended...

You are correct; the scope provided by http.<url>.* is enough to meet my use cases, however I agree the lack of access to metadata limits what can be done within in the context of the shell, and makes the case for a credential-helper implementation stronger. I think there is something to be said about the simplicity and user-friendliness of allowing shell scripts for semi-complex config options, but authentication is a task that should be handled well and centrally, thus extending the credential-api makes sense.

​Without Wax,
Colin Arnott​

Re: [PATCH] http.c: shell command evaluation for extraheader

[Johannes Schindelin]

Hi Colin,

On Tue, 6 Mar 2018, Colin Arnott wrote:

> On March 5, 2018 1:47 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> 
> > As the credential-helper is already intended for sensitive data, and
> > as it already allows to interact with a helper, I would strongly
> > assume that it would make more sense to try to extend that feature
> > (instead of the simple extraHeader one).
> 
> To confirm you are suggesting that the credential struct, defined in credential.h, be extended to include a headers array, like so:
> --- a/credential.h
> +++ b/credential.h
> @@ -15,6 +15,7 @@ struct credential {
>         char *protocol;
>         char *host;
>         char *path;
> +       char **headers
>  };
>  
>  #define CREDENTIAL_INIT { STRING_LIST_INIT_DUP }

I think that was what I had in mind, yes.

> > This would also help alleviate all the quoting/dequoting issues involved
> > with shell scripting.
> > 
> > Besides, the http.extraHeader feature was designed to accommodate all
> > kinds of extra headers, not only authentication ones (and indeed, the
> > authentication was only intended for use in build agents, where both
> > environment and logging can be controlled rather tightly).
> 
> I realise that my examples are scoped for auth, but I can conceive of
> other mutating headers that are not explicitly authentication related,
> and could benefit from shell execution before fetch, pull, push actions.

I can conceive of yet another use case that would benefit from shell
execution in these scenarios: attacks. That is why I am extremely hesitant
to go that route.

But then, I am not the maintainer of Git. If you can convince him, you're
good to go.

> > I also see that in your implementation, only the extraHeader value is
> > evaluated, without any access to the rest of the metadata (such as URL,
> > and optionally specified user).
> >
> > It would probably get a little more complicated than a shell script to
> > write a credential-helper that will always be asked to generate an
> > authentication, but I think even a moderate-level Perl script could be
> > used for that, and it would know the URL and user for which the
> > credentials are intended...
> 
> You are correct; the scope provided by http.<url>.* is enough to meet my
> use cases, however I agree the lack of access to metadata limits what
> can be done within in the context of the shell, and makes the case for a
> credential-helper implementation stronger. I think there is something to
> be said about the simplicity and user-friendliness of allowing shell
> scripts for semi-complex config options, but authentication is a task
> that should be handled well and centrally, thus extending the
> credential-api makes sense.

Yes, I agree.

Ciao,
Johannes