[PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Hello! I have an RFC to update the existing credential helper design in
order to allow for some new scenarios, and future evolution of auth methods
that Git hosts may wish to provide. I outline the background, summary of
changes and some challenges below. I also attach a series of patches to
illustrate the design proposal.

One missing element from the patches are extensive tests of the new
behaviour. It appears existing tests focus either on the credential helper
protocol/format, or rely on testing basic authentication only via an Apache
webserver. In order to have a full end to end test coverage of these new
features it make be that we need a more comprehensive test bed to mock these
more nuanced authentication methods. I lean on the experts on the list for
advice here.


Background
==========

Git uses a variety of protocols [1]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [2], and Negotiate (RFC 2478)
[3]. Git uses a extensible model where credential helpers can provide
credentials for protocols [4]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [5], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [6]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Leverage newest identity standards, enhancements, and threat
     mitigations - all without updating Git.
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[n]=value, where n
    is a zero based index of the values.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[n].

 3. Teach Git to specify authentication schemes other than Basic in
    subsequent HTTP requests based on credential helper responses.


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [7] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [8] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [8] authority, a credential helper can use OpenID Connect's
Discovery [9] and Dynamic Client Registration [9] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[0]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[1]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [11]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future flexibility
==================

By allowing the credential helpers decide the best authentication scheme, we
can allow the remote Git server to both offer new schemes (or remove old
ones) that enlightened credential helpers could take immediate advantage of,
and to use credentials that are much more tightly scoped and bound to the
specific request.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[0]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper would return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>



Should Git not control the set of authentication schemes?
=========================================================

One concern that the reader may have regarding these changes is in allowing
helpers to select the authentication mechanism to use, it may be possible
that a weaker form of authentication is used.

Take for example a Git remote server that responds with the following
authentication schemes:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Negotiate ...
WWW-Authenticate: Basic ...


Today Git (and libcurl) prefer to Negotiate over Basic authentication [12].
If a helper responded with authtype=basic Git would now be using a "less
secure" mechanism.

The reason we still propose the credential helper decide on the
authentication scheme is that Git is not the best placed entity to decide
what type of authentication should be used for a particular request (see
Design Principle 3).

OAuth Bearer tokens are often bundled in Basic Authorization headers [13],
but given that the tokens are/can be short-lived and have a highly scoped
set of permissions, this solution could be argued as being more secure than
something like NTLM [14]. Similarly, the user may wish to be consulted on
selecting a particular user account, or directly selecting an authentication
mechanism for a request that otherwise they would not be able to use.

Also, as new authentication protocols appear Git does not need to be
modified or updated for the user to take advantage of them; the credential
helpers take on the responsibility of learning and selecting the "best"
option.


Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [1] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [2] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [3] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [4] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [5] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [6] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [7] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [8] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [9] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [10] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [11] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616

 * [12] libcurl http.c pickoneauth Function
   https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416

 * [13] Git Credential Manager GitHub Host Provider (using PAT as password)
   https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157

 * [14] NT LAN Manager (NTLM) Authentication Protocol
   https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4

Matthew John Cheetham (8):
  wincred: ignore unknown lines (do not die)
  netrc: ignore unknown lines (do not die)
  osxkeychain: clarify that we ignore unknown lines
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests
  http: store all request headers on active_request_slot
  http: move proactive auth to first slot creation
  http: set specific auth scheme depending on credential

 Documentation/git-credential.txt              |  18 ++
 .../netrc/git-credential-netrc.perl           |   5 +-
 .../osxkeychain/git-credential-osxkeychain.c  |   5 +
 .../wincred/git-credential-wincred.c          |   7 +-
 credential.c                                  |  18 ++
 credential.h                                  |  11 +
 git-curl-compat.h                             |   7 +
 http-push.c                                   | 103 ++++-----
 http-walker.c                                 |   2 +-
 http.c                                        | 199 +++++++++++++-----
 http.h                                        |   4 +-
 remote-curl.c                                 |  36 ++--
 t/lib-httpd/apache.conf                       |  13 ++
 t/t5551-http-fetch-smart.sh                   |  46 ++++
 14 files changed, 335 insertions(+), 139 deletions(-)


base-commit: dd3f6c4cae7e3b15ce984dce8593ff7569650e24
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
-- 
gitgitgadget

[PATCH 1/8] wincred: ignore unknown lines (do not die)

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

It is the expectation that credential helpers be liberal in what they
accept and conservative in what they return, to allow for future growth
and evolution of the protocol/interaction.

All of the other helpers (store, cache, osxkeychain, libsecret,
gnome-keyring) except `netrc` currently ignore any credential lines
that are not recognised, whereas the Windows helper (wincred) instead
dies.

Fix the discrepancy and ignore unknown lines in the wincred helper.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 contrib/credential/wincred/git-credential-wincred.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/contrib/credential/wincred/git-credential-wincred.c b/contrib/credential/wincred/git-credential-wincred.c
index 5091048f9c6..ead6e267c78 100644
--- a/contrib/credential/wincred/git-credential-wincred.c
+++ b/contrib/credential/wincred/git-credential-wincred.c
@@ -278,8 +278,11 @@ static void read_credential(void)
 			wusername = utf8_to_utf16_dup(v);
 		} else if (!strcmp(buf, "password"))
 			password = utf8_to_utf16_dup(v);
-		else
-			die("unrecognized input");
+		/*
+		 * Ignore other lines; we don't know what they mean, but
+		 * this future-proofs us when later versions of git do
+		 * learn new lines, and the helpers are updated to match.
+		 */
 	}
 }
 
-- 
gitgitgadget

[PATCH 2/8] netrc: ignore unknown lines (do not die)

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Contrary to the documentation on credential helpers, as well as the help
text for git-credential-netrc itself, this helper will `die` when
presented with an unknown property/attribute/token.

Correct the behaviour here by skipping and ignoring any tokens that are
unknown. This means all helpers in the tree are consistent and ignore
any unknown credential properties/attributes.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 contrib/credential/netrc/git-credential-netrc.perl | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/contrib/credential/netrc/git-credential-netrc.perl b/contrib/credential/netrc/git-credential-netrc.perl
index bc57cc65884..9fb998ae090 100755
--- a/contrib/credential/netrc/git-credential-netrc.perl
+++ b/contrib/credential/netrc/git-credential-netrc.perl
@@ -356,7 +356,10 @@ sub read_credential_data_from_stdin {
 		next unless m/^([^=]+)=(.+)/;
 
 		my ($token, $value) = ($1, $2);
-		die "Unknown search token $token" unless exists $q{$token};
+
+		# skip any unknown tokens
+		next unless exists $q{$token};
+
 		$q{$token} = $value;
 		log_debug("We were given search token $token and value $value");
 	}
-- 
gitgitgadget

[PATCH 3/8] osxkeychain: clarify that we ignore unknown lines

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Like in all the other credential helpers, the osxkeychain helper
ignores unknown credential lines.

Add a comment (a la the other helpers) to make it clear and explicit
that this is the desired behaviour.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 contrib/credential/osxkeychain/git-credential-osxkeychain.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/contrib/credential/osxkeychain/git-credential-osxkeychain.c b/contrib/credential/osxkeychain/git-credential-osxkeychain.c
index bf77748d602..e29cc28779d 100644
--- a/contrib/credential/osxkeychain/git-credential-osxkeychain.c
+++ b/contrib/credential/osxkeychain/git-credential-osxkeychain.c
@@ -159,6 +159,11 @@ static void read_credential(void)
 			username = xstrdup(v);
 		else if (!strcmp(buf, "password"))
 			password = xstrdup(v);
+		/*
+		 * Ignore other lines; we don't know what they mean, but
+		 * this future-proofs us when later versions of git do
+		 * learn new lines, and the helpers are updated to match.
+		 */
 	}
 }
 
-- 
gitgitgadget

Re: [PATCH 3/8] osxkeychain: clarify that we ignore unknown lines

[Derrick Stolee]

On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Like in all the other credential helpers, the osxkeychain helper
> ignores unknown credential lines.
> 
> Add a comment (a la the other helpers) to make it clear and explicit
> that this is the desired behaviour.

I recommend that these first three patches be submitted for full
review and merging, since they seem important independent of this
RFC.

Thanks,
-Stolee

Re: [PATCH 3/8] osxkeychain: clarify that we ignore unknown lines

[Matthew John Cheetham]

On 2022-09-19 09:12, Derrick Stolee wrote:
> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Like in all the other credential helpers, the osxkeychain helper
>> ignores unknown credential lines.
>>
>> Add a comment (a la the other helpers) to make it clear and explicit
>> that this is the desired behaviour.
> 
> I recommend that these first three patches be submitted for full
> review and merging, since they seem important independent of this
> RFC.
> 
> Thanks,
> -Stolee

That's a fair point. I will submit these independently.

Thanks,
Matthew

[PATCH 4/8] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 10 +++++++
 http.c       | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6a9d4e3de07 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,14 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +139,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index 5d0502f51fd..091321af98e 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,81 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	const char *z = NULL;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val)) val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		const char **v = values->v + values->nr - 1;
+		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
+
+		free((void*)*v);
+		*v = append;
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (skip_iprefix(buf.buf, "http/", &z))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1829,6 +1904,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers

[Derrick Stolee]

On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:

> +	/**
> +	 * A `strvec` of WWW-Authenticate header values. Each string
> +	 * is the value of a WWW-Authenticate header in an HTTP response,
> +	 * in the order they were received in the response.
> +	 */
> +	struct strvec wwwauth_headers;

I like this careful documentation.

> +	unsigned header_is_last_match:1;

But then this member is unclear how it is attached. It could use its
own "for internal use" comment if we don't want to describe it in full
detail here.

> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = eltsize * nmemb;
> +	struct strvec *values = &http_auth.wwwauth_headers;
> +	struct strbuf buf = STRBUF_INIT;
> +	const char *val;
> +	const char *z = NULL;
> +
> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	strbuf_add(&buf, ptr, size);
> +
> +	/* Strip the CRLF that should be present at the end of each field */

Is it really a CRLF? Or just an LF?

> +	strbuf_trim_trailing_newline(&buf);

Thankfully, this will trim an LF _or_ CR/LF pair, so either way would be fine.

> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
> +		while (isspace(*val)) val++;

Break the "val++;" to its own line:

		while (isspace(*val))
			val++;

While we are here, do we need to be careful about the end of the string at
this point? Is it possible that the server will send all spaces up until the
maximum header size (as mentioned in the message)?

> +
> +		strvec_push(values, val);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> +	}
> +
> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 */
> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
> +		const char **v = values->v + values->nr - 1;

I suppose we expect leading spaces as critical to this header, right?

> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);

We might have better luck using a strbuf, initializing it with the expected
size and using strbuf_add() to append the strings. Maybe I'm just prematurely
optimizing, though.

> +
> +		free((void*)*v);
> +		*v = append;
> +
> +		goto exit;
> +	}
> +
> +	/* This is the start of a new header we don't care about */
> +	http_auth.header_is_last_match = 0;
> +
> +	/*
> +	 * If this is a HTTP status line and not a header field, this signals
> +	 * a different HTTP response. libcurl writes all the output of all
> +	 * response headers of all responses, including redirects.
> +	 * We only care about the last HTTP request response's headers so clear
> +	 * the existing array.
> +	 */
> +	if (skip_iprefix(buf.buf, "http/", &z))
> +		strvec_clear(values);
> +
> +exit:
> +	strbuf_release(&buf);
> +	return size;
> +}
> +
>  size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
>  {
>  	return nmemb;
> @@ -1829,6 +1904,8 @@ static int http_request(const char *url,
>  					 fwrite_buffer);
>  	}
>  
> +	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);

Nice integration point!

Thanks,
-Stolee

Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2022-09-19 09:21, Derrick Stolee wrote:
> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
> 
>> +	/**
>> +	 * A `strvec` of WWW-Authenticate header values. Each string
>> +	 * is the value of a WWW-Authenticate header in an HTTP response,
>> +	 * in the order they were received in the response.
>> +	 */
>> +	struct strvec wwwauth_headers;
> 
> I like this careful documentation.
> 
>> +	unsigned header_is_last_match:1;
> 
> But then this member is unclear how it is attached. It could use its
> own "for internal use" comment if we don't want to describe it in full
> detail here.

A fair point. I will update in a future iteration.

>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> +{
>> +	size_t size = eltsize * nmemb;
>> +	struct strvec *values = &http_auth.wwwauth_headers;
>> +	struct strbuf buf = STRBUF_INIT;
>> +	const char *val;
>> +	const char *z = NULL;
>> +
>> +	/*
>> +	 * Header lines may not come NULL-terminated from libcurl so we must
>> +	 * limit all scans to the maximum length of the header line, or leverage
>> +	 * strbufs for all operations.
>> +	 *
>> +	 * In addition, it is possible that header values can be split over
>> +	 * multiple lines as per RFC 2616 (even though this has since been
>> +	 * deprecated in RFC 7230). A continuation header field value is
>> +	 * identified as starting with a space or horizontal tab.
>> +	 *
>> +	 * The formal definition of a header field as given in RFC 2616 is:
>> +	 *
>> +	 *   message-header = field-name ":" [ field-value ]
>> +	 *   field-name     = token
>> +	 *   field-value    = *( field-content | LWS )
>> +	 *   field-content  = <the OCTETs making up the field-value
>> +	 *                    and consisting of either *TEXT or combinations
>> +	 *                    of token, separators, and quoted-string>
>> +	 */
>> +
>> +	strbuf_add(&buf, ptr, size);
>> +
>> +	/* Strip the CRLF that should be present at the end of each field */
> 
> Is it really a CRLF? Or just an LF?

It is indeed an CRLF, agnostic of platform. HTTP defines CRLF as the
end-of-line marker for all entities other than the body.

See RFC 2616 section 2.2: https://www.rfc-editor.org/rfc/rfc2616#section-2.2

>> +	strbuf_trim_trailing_newline(&buf);
> 
> Thankfully, this will trim an LF _or_ CR/LF pair, so either way would be fine.
> 
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> +		while (isspace(*val)) val++;
> 
> Break the "val++;" to its own line:
> 
> 		while (isspace(*val))
> 			val++;

Sure! Sorry I missed this one.

> While we are here, do we need to be careful about the end of the string at
> this point? Is it possible that the server will send all spaces up until the
> maximum header size (as mentioned in the message)?
> 
>> +
>> +		strvec_push(values, val);
>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
>> +	}
>> +
>> +	/*
>> +	 * This line could be a continuation of the previously matched header
>> +	 * field. If this is the case then we should append this value to the
>> +	 * end of the previously consumed value.
>> +	 */
>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> +		const char **v = values->v + values->nr - 1;
> 
> I suppose we expect leading spaces as critical to this header, right?

Leading (and trailing) spaces are not part of the header value.

From RFC 2616 section 2.2 regarding header field values:

"All linear white space, including folding, has the same semantics as SP.
A recipient MAY replace any linear white space with a single SP before
interpreting the field value or forwarding the message downstream."

>> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
> 
> We might have better luck using a strbuf, initializing it with the expected
> size and using strbuf_add() to append the strings. Maybe I'm just prematurely
> optimizing, though.

This code path is used to re-join/fold a header value continuation, which is
pretty rare in the wild (if at all with modern web servers).

>> +
>> +		free((void*)*v);
>> +		*v = append;
>> +
>> +		goto exit;
>> +	}
>> +
>> +	/* This is the start of a new header we don't care about */
>> +	http_auth.header_is_last_match = 0;
>> +
>> +	/*
>> +	 * If this is a HTTP status line and not a header field, this signals
>> +	 * a different HTTP response. libcurl writes all the output of all
>> +	 * response headers of all responses, including redirects.
>> +	 * We only care about the last HTTP request response's headers so clear
>> +	 * the existing array.
>> +	 */
>> +	if (skip_iprefix(buf.buf, "http/", &z))
>> +		strvec_clear(values);
>> +
>> +exit:
>> +	strbuf_release(&buf);
>> +	return size;
>> +}
>> +
>>  size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
>>  {
>>  	return nmemb;
>> @@ -1829,6 +1904,8 @@ static int http_request(const char *url,
>>  					 fwrite_buffer);
>>  	}
>>  
>> +	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
> 
> Nice integration point!
> 
> Thanks,
> -Stolee

Thanks,
Matthew

Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers

[Derrick Stolee]

On 9/21/2022 6:24 PM, Matthew John Cheetham wrote:
> On 2022-09-19 09:21, Derrick Stolee wrote:
>> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:

>>> +
>>> +		strvec_push(values, val);
>>> +		http_auth.header_is_last_match = 1;
>>> +		goto exit;
>>> +	}
>>> +
>>> +	/*
>>> +	 * This line could be a continuation of the previously matched header
>>> +	 * field. If this is the case then we should append this value to the
>>> +	 * end of the previously consumed value.
>>> +	 */
>>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>>> +		const char **v = values->v + values->nr - 1;
>>
>> I suppose we expect leading spaces as critical to this header, right?
> 
> Leading (and trailing) spaces are not part of the header value.
> 
> From RFC 2616 section 2.2 regarding header field values:
> 
> "All linear white space, including folding, has the same semantics as SP.
> A recipient MAY replace any linear white space with a single SP before
> interpreting the field value or forwarding the message downstream."
> 
>>> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
>>
>> We might have better luck using a strbuf, initializing it with the expected
>> size and using strbuf_add() to append the strings. Maybe I'm just prematurely
>> optimizing, though.
> 
> This code path is used to re-join/fold a header value continuation, which is
> pretty rare in the wild (if at all with modern web servers).

I think the point is that I noticed that you removed the leading whitespace
in a header's first line, but additional whitespace after this first space
will be included in the concatenated content of the header value.

As long as that is the intention, then I'm happy here.

Thanks,
-Stolee

[PATCH 5/8] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[n]` properties where `n` is a
zero-indexed number, reflecting the order the WWW-Authenticate headers
appeared in the HTTP response.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  9 +++++++
 credential.c                     | 12 +++++++++
 t/lib-httpd/apache.conf          | 13 +++++++++
 t/t5551-http-fetch-smart.sh      | 46 ++++++++++++++++++++++++++++++++
 4 files changed, 80 insertions(+)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f18673017f5..7d4a788c63d 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -151,6 +151,15 @@ Git understands the following attributes:
 	were read (e.g., `url=https://example.com` would behave as if
 	`protocol=https` and `host=example.com` had been provided). This
 	can help callers avoid parsing URLs themselves.
+
+`wwwauth[n]`::
+
+	When an HTTP response is received that includes one or more
+	'WWW-Authenticate' authentication headers, these can be passed to Git
+	(and subsequent credential helpers) with these attributes.
+	Each 'WWW-Authenticate' header value should be passed as a separate
+	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
+	appear in the HTTP response.
 +
 Note that specifying a protocol is mandatory and if the URL
 doesn't specify a hostname (e.g., "cert:///path/to/file") the
diff --git a/credential.c b/credential.c
index 897b4679333..4ad40323fc7 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	int i = 0;
+	for (; i < vec->nr; i++) {
+		const char *full_key = xstrfmt("%s[%d]", key, i);
+		credential_write_item(fp, full_key, vec->v[i], 0);
+		free((void*)full_key);
+	}
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 497b9b9d927..fe118d76f98 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -235,6 +235,19 @@ SSLEngine On
 	Require valid-user
 </LocationMatch>
 
+# Advertise two additional auth methods above "Basic".
+# Neither of them actually work but serve test cases showing these
+# additional auth headers are consumed correctly.
+<Location /auth-wwwauth/>
+	AuthType Basic
+	AuthName "git-auth"
+	AuthUserFile passwd
+	Require valid-user
+	SetEnvIf Authorization "^\S+" authz
+	Header always add WWW-Authenticate "Bearer authority=https://login.example.com" env=!authz
+	Header always add WWW-Authenticate "FooAuth foo=bar baz=1" env=!authz
+</Location>
+
 RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]
 RewriteCond %{REQUEST_URI} /git-receive-pack$
 RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 6a38294a476..c99d8e253df 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -564,6 +564,52 @@ test_expect_success 'http auth forgets bogus credentials' '
 	expect_askpass both user@host
 '
 
+test_expect_success 'http auth sends www-auth headers to credential helper' '
+	write_script git-credential-tee <<-\EOF &&
+		cmd=$1
+		teefile=credential-$cmd
+		if [ -f "$teefile" ]; then
+			rm $teefile
+		fi
+		(
+			while read line;
+			do
+				if [ -z "$line" ]; then
+					exit 0
+				fi
+				echo "$line" >> $teefile
+				echo $line
+			done
+		) | git credential-store $cmd
+	EOF
+
+	cat >expected-get <<-EOF &&
+	protocol=http
+	host=127.0.0.1:5551
+	wwwauth[0]=Bearer authority=https://login.example.com
+	wwwauth[1]=FooAuth foo=bar baz=1
+	wwwauth[2]=Basic realm="git-auth"
+	EOF
+
+	cat >expected-store <<-EOF &&
+	protocol=http
+	host=127.0.0.1:5551
+	username=user@host
+	password=pass@host
+	EOF
+
+	rm -f .git-credentials &&
+	test_config credential.helper tee &&
+	set_askpass user@host pass@host &&
+	(
+		PATH="$PWD:$PATH" &&
+		git ls-remote "$HTTPD_URL/auth-wwwauth/smart/repo.git"
+	) &&
+	expect_askpass both user@host &&
+	test_cmp expected-get credential-get &&
+	test_cmp expected-store credential-store
+'
+
 test_expect_success 'client falls back from v2 to v0 to match server' '
 	GIT_TRACE_PACKET=$PWD/trace \
 	GIT_TEST_PROTOCOL_VERSION=2 \
-- 
gitgitgadget

Re: [PATCH 5/8] credential: add WWW-Authenticate header to cred requests

[Derrick Stolee]

On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>

> In this case we send multiple `wwwauth[n]` properties where `n` is a
> zero-indexed number, reflecting the order the WWW-Authenticate headers
> appeared in the HTTP response.
> @@ -151,6 +151,15 @@ Git understands the following attributes:
>  	were read (e.g., `url=https://example.com` would behave as if
>  	`protocol=https` and `host=example.com` had been provided). This
>  	can help callers avoid parsing URLs themselves.
> +
> +`wwwauth[n]`::
> +
> +	When an HTTP response is received that includes one or more
> +	'WWW-Authenticate' authentication headers, these can be passed to Git
> +	(and subsequent credential helpers) with these attributes.
> +	Each 'WWW-Authenticate' header value should be passed as a separate
> +	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
> +	appear in the HTTP response.
>  +
>  Note that specifying a protocol is mandatory and if the URL
>  doesn't specify a hostname (e.g., "cert:///path/to/file") the

This "+" means that this paragraph should be connected to the previous
one, so it seems that you've inserted your new value in the middle of
the `url` key. You'll want to move yours to be after those two connected
paragraphs. Your diff hunk should look like this:

--- >8 ---

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f18673017f..127ae29be3 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -160,6 +160,15 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[n]`::
+
+	When an HTTP response is received that includes one or more
+	'WWW-Authenticate' authentication headers, these can be passed to Git
+	(and subsequent credential helpers) with these attributes.
+	Each 'WWW-Authenticate' header value should be passed as a separate
+	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
+	appear in the HTTP response.
+
 GIT
 ---
 Part of the linkgit:git[1] suite


--- >8 ---

> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
> index 497b9b9d927..fe118d76f98 100644
> --- a/t/lib-httpd/apache.conf
> +++ b/t/lib-httpd/apache.conf
> @@ -235,6 +235,19 @@ SSLEngine On
>  	Require valid-user
>  </LocationMatch>
>  
> +# Advertise two additional auth methods above "Basic".
> +# Neither of them actually work but serve test cases showing these
> +# additional auth headers are consumed correctly.
> +<Location /auth-wwwauth/>
> +	AuthType Basic
> +	AuthName "git-auth"
> +	AuthUserFile passwd
> +	Require valid-user
> +	SetEnvIf Authorization "^\S+" authz
> +	Header always add WWW-Authenticate "Bearer authority=https://login.example.com" env=!authz
> +	Header always add WWW-Authenticate "FooAuth foo=bar baz=1" env=!authz
> +</Location>
> +

This is cool that you've figured out how to make our Apache tests
add these headers! Maybe we won't need that extra test helper like
I thought (unless we want to confirm the second request sends the
right information).

> +test_expect_success 'http auth sends www-auth headers to credential helper' '
> +	write_script git-credential-tee <<-\EOF &&
> +		cmd=$1
> +		teefile=credential-$cmd
> +		if [ -f "$teefile" ]; then

I think we prefer using "test" over the braces (and linebreak
before then) like this:

		if test -n "$teefile"
		then

> +			rm $teefile
> +		fi

Alternatively, you could always run "rm -f $teefile" for
simplicity.

> +		(
> +			while read line;
> +			do
> +				if [ -z "$line" ]; then
> +					exit 0
> +				fi
> +				echo "$line" >> $teefile
> +				echo $line
> +			done
> +		) | git credential-store $cmd

Since I'm not sure, I'll ask the question: do we need the sub-shell
here, or could we pipe directly off of the "done"? Like this:

		while read line;
		do
			if [ -z "$line" ]; then
				exit 0
			fi
			echo "$line" >> $teefile
			echo $line
		done | git credential-store $cmd

> +	EOF


> +	cat >expected-get <<-EOF &&
> +	protocol=http
> +	host=127.0.0.1:5551
> +	wwwauth[0]=Bearer authority=https://login.example.com
> +	wwwauth[1]=FooAuth foo=bar baz=1
> +	wwwauth[2]=Basic realm="git-auth"
> +	EOF
> +
> +	cat >expected-store <<-EOF &&
> +	protocol=http
> +	host=127.0.0.1:5551
> +	username=user@host
> +	password=pass@host
> +	EOF
> +
> +	rm -f .git-credentials &&
> +	test_config credential.helper tee &&
> +	set_askpass user@host pass@host &&
> +	(
> +		PATH="$PWD:$PATH" &&
> +		git ls-remote "$HTTPD_URL/auth-wwwauth/smart/repo.git"
> +	) &&
> +	expect_askpass both user@host &&
> +	test_cmp expected-get credential-get &&
> +	test_cmp expected-store credential-store

Elegant check for both calls.

Thanks,
-Stolee

Re: [PATCH 5/8] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2022-09-19 09:33, Derrick Stolee wrote:
> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
>> In this case we send multiple `wwwauth[n]` properties where `n` is a
>> zero-indexed number, reflecting the order the WWW-Authenticate headers
>> appeared in the HTTP response.
>> @@ -151,6 +151,15 @@ Git understands the following attributes:
>>  	were read (e.g., `url=https://example.com` would behave as if
>>  	`protocol=https` and `host=example.com` had been provided). This
>>  	can help callers avoid parsing URLs themselves.
>> +
>> +`wwwauth[n]`::
>> +
>> +	When an HTTP response is received that includes one or more
>> +	'WWW-Authenticate' authentication headers, these can be passed to Git
>> +	(and subsequent credential helpers) with these attributes.
>> +	Each 'WWW-Authenticate' header value should be passed as a separate
>> +	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
>> +	appear in the HTTP response.
>>  +
>>  Note that specifying a protocol is mandatory and if the URL
>>  doesn't specify a hostname (e.g., "cert:///path/to/file") the
> 
> This "+" means that this paragraph should be connected to the previous
> one, so it seems that you've inserted your new value in the middle of
> the `url` key. You'll want to move yours to be after those two connected
> paragraphs. Your diff hunk should look like this:
> 
> --- >8 ---
> 
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index f18673017f..127ae29be3 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -160,6 +160,15 @@ empty string.
>  Components which are missing from the URL (e.g., there is no
>  username in the example above) will be left unset.
>  
> +`wwwauth[n]`::
> +
> +	When an HTTP response is received that includes one or more
> +	'WWW-Authenticate' authentication headers, these can be passed to Git
> +	(and subsequent credential helpers) with these attributes.
> +	Each 'WWW-Authenticate' header value should be passed as a separate
> +	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
> +	appear in the HTTP response.
> +
>  GIT
>  ---
>  Part of the linkgit:git[1] suite
> 
> 
> --- >8 ---

Thanks for catching!

>> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
>> index 497b9b9d927..fe118d76f98 100644
>> --- a/t/lib-httpd/apache.conf
>> +++ b/t/lib-httpd/apache.conf
>> @@ -235,6 +235,19 @@ SSLEngine On
>>  	Require valid-user
>>  </LocationMatch>
>>  
>> +# Advertise two additional auth methods above "Basic".
>> +# Neither of them actually work but serve test cases showing these
>> +# additional auth headers are consumed correctly.
>> +<Location /auth-wwwauth/>
>> +	AuthType Basic
>> +	AuthName "git-auth"
>> +	AuthUserFile passwd
>> +	Require valid-user
>> +	SetEnvIf Authorization "^\S+" authz
>> +	Header always add WWW-Authenticate "Bearer authority=https://login.example.com" env=!authz
>> +	Header always add WWW-Authenticate "FooAuth foo=bar baz=1" env=!authz
>> +</Location>
>> +
> 
> This is cool that you've figured out how to make our Apache tests
> add these headers! Maybe we won't need that extra test helper like
> I thought (unless we want to confirm the second request sends the
> right information).

This will exercise the new header parsing and passing the info to the helper
but will indeed not test the response. I feel like a test helper would be
beneficial still.. what I've done here doesn't feel 100% clean or complete.

>> +test_expect_success 'http auth sends www-auth headers to credential helper' '
>> +	write_script git-credential-tee <<-\EOF &&
>> +		cmd=$1
>> +		teefile=credential-$cmd
>> +		if [ -f "$teefile" ]; then
> 
> I think we prefer using "test" over the braces (and linebreak
> before then) like this:
> 
> 		if test -n "$teefile"
> 		then
> 
>> +			rm $teefile
>> +		fi
> 
> Alternatively, you could always run "rm -f $teefile" for
> simplicity.
I like simple :-)

>> +		(
>> +			while read line;
>> +			do
>> +				if [ -z "$line" ]; then
>> +					exit 0
>> +				fi
>> +				echo "$line" >> $teefile
>> +				echo $line
>> +			done
>> +		) | git credential-store $cmd
> 
> Since I'm not sure, I'll ask the question: do we need the sub-shell
> here, or could we pipe directly off of the "done"? Like this:
> 
> 		while read line;
> 		do
> 			if [ -z "$line" ]; then
> 				exit 0
> 			fi
> 			echo "$line" >> $teefile
> 			echo $line
> 		done | git credential-store $cmd

That we can.. I will update in next iteration.

>> +	EOF
> 
> 
>> +	cat >expected-get <<-EOF &&
>> +	protocol=http
>> +	host=127.0.0.1:5551
>> +	wwwauth[0]=Bearer authority=https://login.example.com
>> +	wwwauth[1]=FooAuth foo=bar baz=1
>> +	wwwauth[2]=Basic realm="git-auth"
>> +	EOF
>> +
>> +	cat >expected-store <<-EOF &&
>> +	protocol=http
>> +	host=127.0.0.1:5551
>> +	username=user@host
>> +	password=pass@host
>> +	EOF
>> +
>> +	rm -f .git-credentials &&
>> +	test_config credential.helper tee &&
>> +	set_askpass user@host pass@host &&
>> +	(
>> +		PATH="$PWD:$PATH" &&
>> +		git ls-remote "$HTTPD_URL/auth-wwwauth/smart/repo.git"
>> +	) &&
>> +	expect_askpass both user@host &&
>> +	test_cmp expected-get credential-get &&
>> +	test_cmp expected-store credential-store
> 
> Elegant check for both calls.
> 
> Thanks,
> -Stolee

Thanks,
Matthew

[PATCH 6/8] http: store all request headers on active_request_slot

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Once a list of headers has been set on the curl handle, it is not
possible to recover that `struct curl_slist` instance to add or modify
headers.

In future commits we will want to modify the set of request headers in
response to an authentication challenge/401 response from the server,
with information provided by a credential helper.

There are a number of different places where curl is used for an HTTP
request, and they do not have a common handling of request headers.
However, given that they all do call the `start_active_slot()` function,
either directly or indirectly via `run_slot()` or `run_one_slot()`, we
use this as the point to set the `CURLOPT_HTTPHEADER` option just
before the request is made.

We collect all request headers in a `struct curl_slist` on the
`struct active_request_slot` that is obtained from a call to
`get_active_slot(int)`. This function now takes a single argument to
define if the initial set of headers on the slot should include the
"Pragma: no-cache" header, along with all extra headers specified via
`http.extraHeader` config values.

The active request slot obtained from `get_active_slot(int)` will always
contain a fresh set of default headers and any headers set in previous
usages of this slot will be freed.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http-push.c   | 103 ++++++++++++++++++++++----------------------------
 http-walker.c |   2 +-
 http.c        |  82 ++++++++++++++++++----------------------
 http.h        |   4 +-
 remote-curl.c |  36 +++++++++---------
 5 files changed, 101 insertions(+), 126 deletions(-)

diff --git a/http-push.c b/http-push.c
index 5f4340a36e6..2b40959b376 100644
--- a/http-push.c
+++ b/http-push.c
@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,
 	curl_easy_setopt(curl, CURLOPT_UPLOAD, 1);
 }
 
-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)
+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,
+	struct remote_lock *lock, enum dav_header_flag options)
 {
 	struct strbuf buf = STRBUF_INIT;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
 	if (options & DAV_HEADER_IF) {
 		strbuf_addf(&buf, "If: (<%s>)", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_LOCK) {
 		strbuf_addf(&buf, "Lock-Token: <%s>", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_TIMEOUT) {
 		strbuf_addf(&buf, "Timeout: Second-%ld", lock->timeout);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	strbuf_release(&buf);
 
-	return dav_headers;
+	return headers;
 }
 
 static void finish_request(struct transfer_request *request);
@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)
 
 	request->url = get_remote_object_url(repo->url, hex, 1);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MKCOL);
@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)
 	strbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);
 	request->url = strbuf_detach(&buf, NULL);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http(slot->curl, request->url, DAV_PUT,
@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)
 static void start_move(struct transfer_request *request)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MOVE);
-	dav_headers = curl_slist_append(dav_headers, request->dest);
-	dav_headers = curl_slist_append(dav_headers, "Overwrite: T");
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
+	slot->headers = curl_slist_append(slot->headers, request->dest);
+	slot->headers = curl_slist_append(slot->headers, "Overwrite: T");
 
 	if (start_active_slot(slot)) {
 		request->slot = slot;
@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
 	lock->refreshing = 1;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_LOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)
 	}
 
 	lock->refreshing = 0;
-	curl_slist_free_all(dav_headers);
 
 	return rc;
 }
@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	char *ep;
 	char timeout_header[25];
 	struct remote_lock *lock = NULL;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	char *escaped;
 
@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	while (ep) {
 		char saved_character = ep[1];
 		ep[1] = '\0';
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
 		curl_setup_http_get(slot->curl, url, DAV_MKCOL);
 		if (start_active_slot(slot)) {
@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	strbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);
 	free(escaped);
 
+	slot = get_active_slot(0);
+	slot->results = &results;
+
 	xsnprintf(timeout_header, sizeof(timeout_header), "Timeout: Second-%ld", timeout);
-	dav_headers = curl_slist_append(dav_headers, timeout_header);
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
+	slot->headers = curl_slist_append(slot->headers, timeout_header);
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
 
-	slot = get_active_slot();
-	slot->results = &results;
 	curl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	CALLOC_ARRAY(lock, 1);
@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 		fprintf(stderr, "Unable to start LOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
 
@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct remote_lock *prev = repo->locks;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_LOCK);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)
 		fprintf(stderr, "Unable to start UNLOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
-
 	if (repo->locks == lock) {
 		repo->locks = lock->next;
 	} else {
@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	struct remote_ls_ctx ls;
 
@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,
 
 	strbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 1");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 1");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,
 	free(url);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 }
 
 static void get_remote_object_list(unsigned char parent)
@@ -1199,7 +1189,6 @@ static int locking_available(void)
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	int lock_flags = 0;
 	char *escaped;
@@ -1208,14 +1197,14 @@ static int locking_available(void)
 	strbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);
 	free(escaped);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 0");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 0");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, repo->url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1257,7 +1246,6 @@ static int locking_available(void)
 
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 
 	return lock_flags;
 }
@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers;
-
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
 	strbuf_addf(&out_buffer.buf, "%s\n", oid_to_hex(oid));
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF);
+
 	curl_setup_http(slot->curl, lock->url, DAV_PUT,
 			&out_buffer, fwrite_null);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)
 	struct buffer buffer = { STRBUF_INIT, 0 };
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 
 	remote_ls("refs/", (PROCESS_FILES | RECURSIVE),
 		  add_remote_info_ref, &buffer.buf);
 	if (!aborted) {
-		dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
+		slot->headers = append_dav_token_headers(slot->headers, lock,
+			DAV_HEADER_IF);
+
 		curl_setup_http(slot->curl, lock->url, DAV_PUT,
 				&buffer, fwrite_null);
-		curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 		if (start_active_slot(slot)) {
 			run_active_slot(slot);
@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)
 	if (dry_run)
 		return 0;
 	url = xstrfmt("%s%s", repo->url, remote_ref->name);
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
 	curl_setup_http_get(slot->curl, url, DAV_DELETE);
 	if (start_active_slot(slot)) {
diff --git a/http-walker.c b/http-walker.c
index b8f0f98ae14..8747de2fcdb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)
 	 * Use a callback to process the result, since another request
 	 * may fail and need to have alternates loaded before continuing
 	 */
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_alternates_response;
 	alt_req.walker = walker;
 	slot->callback_data = &alt_req;
diff --git a/http.c b/http.c
index 091321af98e..42616f746b1 100644
--- a/http.c
+++ b/http.c
@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =
 	| CURLAUTH_DIGEST_IE
 	| CURLAUTH_DIGEST;
 
-static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
 static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
 
 static struct curl_slist *host_resolutions;
@@ -1132,11 +1130,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 	if (remote)
 		var_override(&http_proxy_authmethod, remote->http_proxy_authmethod);
 
-	pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma: no-cache");
-	no_pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma:");
-
 	{
 		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
 		if (http_max_requests)
@@ -1198,6 +1191,8 @@ void http_cleanup(void)
 
 	while (slot != NULL) {
 		struct active_request_slot *next = slot->next;
+		if (slot->headers)
+			curl_slist_free_all(slot->headers);
 		if (slot->curl) {
 			xmulti_remove_handle(slot);
 			curl_easy_cleanup(slot->curl);
@@ -1214,12 +1209,6 @@ void http_cleanup(void)
 
 	string_list_clear(&extra_http_headers, 0);
 
-	curl_slist_free_all(pragma_header);
-	pragma_header = NULL;
-
-	curl_slist_free_all(no_pragma_header);
-	no_pragma_header = NULL;
-
 	curl_slist_free_all(host_resolutions);
 	host_resolutions = NULL;
 
@@ -1254,7 +1243,18 @@ void http_cleanup(void)
 	FREE_AND_NULL(cached_accept_language);
 }
 
-struct active_request_slot *get_active_slot(void)
+static struct curl_slist *http_copy_default_headers(void)
+{
+	struct curl_slist *headers = NULL;
+	const struct string_list_item *item;
+
+	for_each_string_list_item(item, &extra_http_headers)
+		headers = curl_slist_append(headers, item->string);
+
+	return headers;
+}
+
+struct active_request_slot *get_active_slot(int no_pragma_header)
 {
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
@@ -1276,6 +1276,7 @@ struct active_request_slot *get_active_slot(void)
 		newslot->curl = NULL;
 		newslot->in_use = 0;
 		newslot->next = NULL;
+		newslot->headers = NULL;
 
 		slot = active_queue_head;
 		if (!slot) {
@@ -1293,6 +1294,15 @@ struct active_request_slot *get_active_slot(void)
 		curl_session_count++;
 	}
 
+	if (slot->headers)
+		curl_slist_free_all(slot->headers);
+
+	slot->headers = http_copy_default_headers();
+
+	if (!no_pragma_header)
+		slot->headers = curl_slist_append(slot->headers,
+			"Pragma: no-cache");
+
 	active_requests++;
 	slot->in_use = 1;
 	slot->results = NULL;
@@ -1302,7 +1312,6 @@ struct active_request_slot *get_active_slot(void)
 	curl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);
 	if (curl_save_cookies)
 		curl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
 	curl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);
 	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
 	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);
@@ -1334,9 +1343,12 @@ struct active_request_slot *get_active_slot(void)
 
 int start_active_slot(struct active_request_slot *slot)
 {
-	CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);
+	CURLMcode curlm_result;
 	int num_transfers;
 
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);
+	curlm_result = curl_multi_add_handle(curlm, slot->curl);
+
 	if (curlm_result != CURLM_OK &&
 	    curlm_result != CURLM_CALL_MULTI_PERFORM) {
 		warning("curl_multi_add_handle failed: %s",
@@ -1651,17 +1663,6 @@ int run_one_slot(struct active_request_slot *slot,
 	return handle_curl_result(results);
 }
 
-struct curl_slist *http_copy_default_headers(void)
-{
-	struct curl_slist *headers = NULL;
-	const struct string_list_item *item;
-
-	for_each_string_list_item(item, &extra_http_headers)
-		headers = curl_slist_append(headers, item->string);
-
-	return headers;
-}
-
 static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)
 {
 	char *ptr;
@@ -1879,12 +1880,11 @@ static int http_request(const char *url,
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *headers = http_copy_default_headers();
-	struct strbuf buf = STRBUF_INIT;
+	int no_cache = options && options->no_cache;
 	const char *accept_language;
 	int ret;
 
-	slot = get_active_slot();
+	slot = get_active_slot(!no_cache);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);
 
 	if (!result) {
@@ -1909,27 +1909,23 @@ static int http_request(const char *url,
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-		headers = curl_slist_append(headers, accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			accept_language);
 
-	strbuf_addstr(&buf, "Pragma:");
-	if (options && options->no_cache)
-		strbuf_addstr(&buf, " no-cache");
 	if (options && options->initial_request &&
 	    http_follow_config == HTTP_FOLLOW_INITIAL)
 		curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);
 
-	headers = curl_slist_append(headers, buf.buf);
-
 	/* Add additional headers here */
 	if (options && options->extra_headers) {
 		const struct string_list_item *item;
 		for_each_string_list_item(item, options->extra_headers) {
-			headers = curl_slist_append(headers, item->string);
+			slot->headers = curl_slist_append(slot->headers,
+				item->string);
 		}
 	}
 
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
 	curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);
 
@@ -1947,9 +1943,6 @@ static int http_request(const char *url,
 		curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
 				options->effective_url);
 
-	curl_slist_free_all(headers);
-	strbuf_release(&buf);
-
 	return ret;
 }
 
@@ -2310,12 +2303,10 @@ struct http_pack_request *new_direct_http_pack_request(
 		goto abort;
 	}
 
-	preq->slot = get_active_slot();
+	preq->slot = get_active_slot(1);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
-	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
-		no_pragma_header);
 
 	/*
 	 * If there is data present from a previous transfer attempt,
@@ -2480,14 +2471,13 @@ struct http_object_request *new_http_object_request(const char *base_url,
 		}
 	}
 
-	freq->slot = get_active_slot();
+	freq->slot = get_active_slot(1);
 
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
 
 	/*
 	 * If we have successfully processed data from a previous fetch
diff --git a/http.h b/http.h
index 3c94c479100..a304cc408b2 100644
--- a/http.h
+++ b/http.h
@@ -22,6 +22,7 @@ struct slot_results {
 struct active_request_slot {
 	CURL *curl;
 	int in_use;
+	struct curl_slist *headers;
 	CURLcode curl_result;
 	long http_code;
 	int *finished;
@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
 curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);
 
 /* Slot lifecycle functions */
-struct active_request_slot *get_active_slot(void);
+struct active_request_slot *get_active_slot(int no_pragma_header);
 int start_active_slot(struct active_request_slot *slot);
 void run_active_slot(struct active_request_slot *slot);
 void finish_all_active_slots(void);
@@ -64,7 +65,6 @@ void step_active_slots(void);
 void http_init(struct remote *remote, const char *url,
 	       int proactive_auth);
 void http_cleanup(void);
-struct curl_slist *http_copy_default_headers(void);
 
 extern long int git_curl_ipresolve;
 extern int active_requests;
diff --git a/remote-curl.c b/remote-curl.c
index 72dfb8fb86a..edbd4504beb 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,
 static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	struct strbuf buf = STRBUF_INIT;
 	int err;
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000");
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);
 
 	err = run_slot(slot, results);
 
-	curl_slist_free_all(headers);
 	strbuf_release(&buf);
 	return err;
 }
@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)
 static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
 	size_t gzip_size = 0;
@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 			needs_100_continue = 1;
 	}
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
-	headers = curl_slist_append(headers, needs_100_continue ?
+retry:
+	slot = get_active_slot(0);
+
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, needs_100_continue ?
 		"Expect: 100-continue" : "Expect:");
 
 	/* Add Accept-Language header */
 	if (rpc->hdr_accept_language)
-		headers = curl_slist_append(headers, rpc->hdr_accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->hdr_accept_language);
 
 	/* Add the extra Git-Protocol header */
 	if (rpc->protocol_header)
-		headers = curl_slist_append(headers, rpc->protocol_header);
-
-retry:
-	slot = get_active_slot();
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->protocol_header);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -955,7 +953,8 @@ retry:
 		/* The request body is large and the size cannot be predicted.
 		 * We must use chunked encoding to send it.
 		 */
-		headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+		slot->headers = curl_slist_append(slot->headers,
+			"Transfer-Encoding: chunked");
 		rpc->initial_buffer = 1;
 		curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);
 		curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);
@@ -1002,7 +1001,8 @@ retry:
 
 		gzip_size = stream.total_out;
 
-		headers = curl_slist_append(headers, "Content-Encoding: gzip");
+		slot->headers = curl_slist_append(slot->headers,
+			"Content-Encoding: gzip");
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));
 
@@ -1025,7 +1025,6 @@ retry:
 		}
 	}
 
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);
 	rpc_in_data.rpc = rpc;
 	rpc_in_data.slot = slot;
@@ -1055,7 +1054,6 @@ retry:
 	if (stateless_connect)
 		packet_response_end(rpc->in);
 
-	curl_slist_free_all(headers);
 	free(gzip_body);
 	return err;
 }
-- 
gitgitgadget

[PATCH 7/8] http: move proactive auth to first slot creation

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rather than proactively seek credentials to authenticate a request at
`http_init()` time, do it when the first `active_request_slot` is
created.

Because credential helpers may modify the headers used for a request we
can only auth when a slot is created (when we can first start to gather
request headers).

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/http.c b/http.c
index 42616f746b1..8e107ff19b8 100644
--- a/http.c
+++ b/http.c
@@ -514,18 +514,18 @@ static int curl_empty_auth_enabled(void)
 	return 0;
 }
 
-static void init_curl_http_auth(CURL *result)
+static void init_curl_http_auth(struct active_request_slot *slot)
 {
 	if (!http_auth.username || !*http_auth.username) {
 		if (curl_empty_auth_enabled())
-			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
+			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
 	}
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
+	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
 }
 
 /* *var must be free-able */
@@ -900,9 +900,6 @@ static CURL *get_curl_handle(void)
 #endif
 	}
 
-	if (http_proactive_auth)
-		init_curl_http_auth(result);
-
 	if (getenv("GIT_SSL_VERSION"))
 		ssl_version = getenv("GIT_SSL_VERSION");
 	if (ssl_version && *ssl_version) {
@@ -1259,6 +1256,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
 
+	int proactive_auth = 0;
 	int num_transfers;
 
 	/* Wait for a slot to open up if the queue is full */
@@ -1281,6 +1279,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 		slot = active_queue_head;
 		if (!slot) {
 			active_queue_head = newslot;
+
+			/* Auth first slot if asked for proactive auth */
+			proactive_auth = http_proactive_auth;
 		} else {
 			while (slot->next != NULL)
 				slot = slot->next;
@@ -1335,8 +1336,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 
 	curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
-	if (http_auth.password || curl_empty_auth_enabled())
-		init_curl_http_auth(slot->curl);
+
+	if (http_auth.password || curl_empty_auth_enabled() || proactive_auth)
+		init_curl_http_auth(slot);
 
 	return slot;
 }
-- 
gitgitgadget

[PATCH 8/8] http: set specific auth scheme depending on credential

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a new credential field `authtype` that can be used by
credential helpers to indicate the type of the credential or
authentication mechanism to use for a request.

Modify http.c to now specify the correct authentication scheme or
credential type when authenticating the curl handle. If the new
`authtype` field in the credential structure is `NULL` or "Basic" then
use the existing username/password options. If the field is "Bearer"
then use the OAuth bearer token curl option. Otherwise, the `authtype`
field is the authentication scheme and the `password` field is the
raw, unencoded value.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  9 +++++++++
 credential.c                     |  5 +++++
 credential.h                     |  1 +
 git-curl-compat.h                |  7 +++++++
 http.c                           | 24 +++++++++++++++++++++---
 5 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 7d4a788c63d..3b6ef6f4906 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -152,6 +152,15 @@ Git understands the following attributes:
 	`protocol=https` and `host=example.com` had been provided). This
 	can help callers avoid parsing URLs themselves.
 
+`authtype`::
+
+	Indicates the type of authentication scheme used. If this is not
+	present the default is "Basic".
+	Known values include "Basic", "Digest", and "Bearer".
+	If an unknown value is provided, this is taken as the authentication
+	scheme for the `Authorization` header, and the `password` field is
+	used as the raw unencoded authorization parameters of the same header.
+
 `wwwauth[n]`::
 
 	When an HTTP response is received that includes one or more
diff --git a/credential.c b/credential.c
index 4ad40323fc7..9d4a0f3fd51 100644
--- a/credential.c
+++ b/credential.c
@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "path")) {
 			free(c->path);
 			c->path = xstrdup(value);
+		} else if (!strcmp(key, "authtype")) {
+			free(c->authtype);
+			c->authtype = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "authtype", c->authtype, 0);
 	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
diff --git a/credential.h b/credential.h
index 6a9d4e3de07..a6572aacf1d 100644
--- a/credential.h
+++ b/credential.h
@@ -135,6 +135,7 @@ struct credential {
 	char *protocol;
 	char *host;
 	char *path;
+	char *authtype;
 };
 
 #define CREDENTIAL_INIT { \
diff --git a/git-curl-compat.h b/git-curl-compat.h
index 56a83b6bbd8..74732500a9f 100644
--- a/git-curl-compat.h
+++ b/git-curl-compat.h
@@ -126,4 +126,11 @@
 #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS
 #endif
 
+/**
+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.
+ */
+#if LIBCURL_VERSION_NUM >= 0x073D00
+#define GIT_CURL_HAVE_CURLAUTH_BEARER
+#endif
+
 #endif
diff --git a/http.c b/http.c
index 8e107ff19b8..d8913b2c641 100644
--- a/http.c
+++ b/http.c
@@ -516,7 +516,8 @@ static int curl_empty_auth_enabled(void)
 
 static void init_curl_http_auth(struct active_request_slot *slot)
 {
-	if (!http_auth.username || !*http_auth.username) {
+	if (!http_auth.authtype &&
+		(!http_auth.username || !*http_auth.username)) {
 		if (curl_empty_auth_enabled())
 			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
@@ -524,8 +525,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
+	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
+				|| !strcasecmp(http_auth.authtype, "digest")) {
+		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
+			http_auth.username);
+		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
+			http_auth.password);
+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
+	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
+		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
+		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
+			http_auth.password);
+#endif
+	} else {
+		struct strbuf auth = STRBUF_INIT;
+		strbuf_addf(&auth, "Authorization: %s %s",
+			http_auth.authtype, http_auth.password);
+		slot->headers = curl_slist_append(slot->headers, auth.buf);
+		strbuf_release(&auth);
+	}
 }
 
 /* *var must be free-able */
-- 
gitgitgadget

Re: [PATCH 8/8] http: set specific auth scheme depending on credential

[Derrick Stolee]

On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Introduce a new credential field `authtype` that can be used by
> credential helpers to indicate the type of the credential or
> authentication mechanism to use for a request.
> 
> Modify http.c to now specify the correct authentication scheme or
> credential type when authenticating the curl handle. If the new
> `authtype` field in the credential structure is `NULL` or "Basic" then
> use the existing username/password options. If the field is "Bearer"
> then use the OAuth bearer token curl option. Otherwise, the `authtype`
> field is the authentication scheme and the `password` field is the
> raw, unencoded value.


> @@ -524,8 +525,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
>  
>  	credential_fill(&http_auth);
>  
> -	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
> -	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
> +	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
> +				|| !strcasecmp(http_auth.authtype, "digest")) {
> +		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
> +			http_auth.username);
> +		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
> +			http_auth.password);
> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
> +	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
> +		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
> +		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
> +			http_auth.password);
> +#endif
> +	} else {
> +		struct strbuf auth = STRBUF_INIT;
> +		strbuf_addf(&auth, "Authorization: %s %s",
> +			http_auth.authtype, http_auth.password);
> +		slot->headers = curl_slist_append(slot->headers, auth.buf);
> +		strbuf_release(&auth);
> +	}
>  }

It would be good to have a test here, and the only way I can think
to add it would be to modify one of the test credential helpers to
indicate that OAuth is being used.

The test would somehow need to be careful about the curl version,
though, and I'm not sure if we have prior work for writing prereqs
based on the linked curl version.

Thanks,
-Stolee

Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[Derrick Stolee]

On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
> Hello! I have an RFC to update the existing credential helper design in
> order to allow for some new scenarios, and future evolution of auth methods
> that Git hosts may wish to provide. I outline the background, summary of
> changes and some challenges below. I also attach a series of patches to
> illustrate the design proposal.

It's unfortunate that we didn't get to talk about this during the
contributor summit, but it is super-technical and worth looking closely
at all the details. 

> One missing element from the patches are extensive tests of the new
> behaviour. It appears existing tests focus either on the credential helper
> protocol/format, or rely on testing basic authentication only via an Apache
> webserver. In order to have a full end to end test coverage of these new
> features it make be that we need a more comprehensive test bed to mock these
> more nuanced authentication methods. I lean on the experts on the list for
> advice here.

The microsoft/git fork has a feature (the GVFS Protocol) that requires a
custom HTTP server as a test helper. We might need a similar test helper
to return these WWW-Authenticate headers and check the full request list
from Git matches the spec. Doing that while also executing the proper Git
commands to serve the HTTP bodies is hopefully not too large. It might be
nice to adapt such a helper to replace the need for a full Apache install
in our test suite, but that's an independent concern from this RFC.

> Limitations
> ===========
> 
> Because this credential model was built mostly for password based
> authentication systems, it's somewhat limited. In particular:
> 
>  1. To generate valid credentials, additional information about the request
>     (or indeed the requestee and their device) may be required. For example,
>     OAuth is based around scopes. A scope, like "git.read", might be
>     required to read data from the remote. However, the remote cannot tell
>     the credential helper what scope is required for this request.
> 
>  2. This system is not fully extensible. Each time a new type of
>     authentication (like OAuth Bearer) is invented, Git needs updates before
>     credential helpers can take advantage of it (or leverage a new
>     capability in libcurl).
> 
> 
> Goals
> =====
> 
>  * As a user with multiple federated cloud identities:

I'm not sure if you mentioned it anywhere else, but this is specifically
for cases where a user might have multiple identities _on the same host
by DNS name_. The credential.useHttpPath config option might seem like it
could help here, but the credential helper might pick the wrong identity
that is the most-recent login. Either this workflow will require the user
to re-login with every new URL or the fetches/clones will fail when the
guess is wrong and the user would need to learn how to log into that other
identity.

Please correct me if I'm wrong about any of this, but the details of your
goals make it clear that the workflow will be greatly improved:

>    * Reach out to a remote and have my credential helper automatically
>      prompt me for the correct identity.
>    * Leverage existing authentication systems built-in to many operating
>      systems and devices to boost security and reduce reliance on passwords.
> 
>  * As a Git host and/or cloud identity provider:
>    
>    * Leverage newest identity standards, enhancements, and threat
>      mitigations - all without updating Git.
>    * Enforce security policies (like requiring two-factor authentication)
>      dynamically.
>    * Allow integration with third party standard based identity providers in
>      enterprises allowing customers to have a single plane of control for
>      critical identities with access to source code.

I had a question with this part of your proposal:

>     Because the extra information forms an ordered list, and the existing
>     credential helper I/O format only provides for simple key=value pairs,
>     we introduce a new convention for transmitting an ordered list of
>     values. Key names that are suffixed with a C-style array syntax should
>     have values considered to form an order list, i.e. key[n]=value, where n
>     is a zero based index of the values.
>     
>     For the WWW-Authenticate header values we opt to use the key wwwauth[n].
...
> Git sends over standard input:
> 
> protocol=https
> host=example.com
> wwwauth[0]=Bearer realm="login.example", scope="git.readwrite"
> wwwauth[1]=Basic realm="login.example"

The important part here is that we provide a way to specify a multi-valued
key as opposed to a "last one wins" key, right?

Using empty braces (wwwauth[]) would suffice to indicate this, right? That
allows us to not care about the values inside the braces. The biggest
issues I see with a value in the braces are:

1. What if it isn't an integer?
2. What if we are missing a value?
3. What if they come out of order?

Without a value inside, then the order in which they appear provides
implicit indices in their multi-valued list.

Other than that, I support this idea and will start looking at the code
now.

Thanks,
-Stolee

Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[Derrick Stolee]

On 9/19/2022 12:08 PM, Derrick Stolee wrote:
> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:

>> protocol=https
>> host=example.com
>> wwwauth[0]=Bearer realm="login.example", scope="git.readwrite"
>> wwwauth[1]=Basic realm="login.example"
> 
> The important part here is that we provide a way to specify a multi-valued
> key as opposed to a "last one wins" key, right?
> 
> Using empty braces (wwwauth[]) would suffice to indicate this, right? That
> allows us to not care about the values inside the braces. The biggest
> issues I see with a value in the braces are:
> 
> 1. What if it isn't an integer?
> 2. What if we are missing a value?
> 3. What if they come out of order?
> 
> Without a value inside, then the order in which they appear provides
> implicit indices in their multi-valued list.

After looking at the code, it would not be difficult at all to make this
change in-place for these patches. But I won't push too hard if there is
some reason to keep the index values.
 
> Other than that, I support this idea and will start looking at the code
> now.

I took a look and provided feedback as I could. Patches 6 and 7 eluded
me only because I'm so unfamiliar with the http.c code and don't have
time to learn it today.

I mentioned that patches 1-3 could easily be picked up as a topic while
the rest of the series is considered carefully.

I tried to add some mentions of testing, but you've already tested more
than I expected, by adding the headers to the Apache output.

Thanks,
-Stolee

Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham]

On 2022-09-19 09:08, Derrick Stolee wrote:
> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:
>> Hello! I have an RFC to update the existing credential helper design in
>> order to allow for some new scenarios, and future evolution of auth methods
>> that Git hosts may wish to provide. I outline the background, summary of
>> changes and some challenges below. I also attach a series of patches to
>> illustrate the design proposal.
> 
> It's unfortunate that we didn't get to talk about this during the
> contributor summit, but it is super-technical and worth looking closely
> at all the details. 
> 
>> One missing element from the patches are extensive tests of the new
>> behaviour. It appears existing tests focus either on the credential helper
>> protocol/format, or rely on testing basic authentication only via an Apache
>> webserver. In order to have a full end to end test coverage of these new
>> features it make be that we need a more comprehensive test bed to mock these
>> more nuanced authentication methods. I lean on the experts on the list for
>> advice here.
> 
> The microsoft/git fork has a feature (the GVFS Protocol) that requires a
> custom HTTP server as a test helper. We might need a similar test helper
> to return these WWW-Authenticate headers and check the full request list
> from Git matches the spec. Doing that while also executing the proper Git
> commands to serve the HTTP bodies is hopefully not too large. It might be
> nice to adapt such a helper to replace the need for a full Apache install
> in our test suite, but that's an independent concern from this RFC.

That's a good reference and possible solution to the testing question, and
definitely something I can look at adding. I just wanted another pair of
eyes and thoughts on any other options that I may have been missing in the
existing testing repertoire, before embarking on writing such a test helper.

>> Limitations
>> ===========
>>
>> Because this credential model was built mostly for password based
>> authentication systems, it's somewhat limited. In particular:
>>
>>  1. To generate valid credentials, additional information about the request
>>     (or indeed the requestee and their device) may be required. For example,
>>     OAuth is based around scopes. A scope, like "git.read", might be
>>     required to read data from the remote. However, the remote cannot tell
>>     the credential helper what scope is required for this request.
>>
>>  2. This system is not fully extensible. Each time a new type of
>>     authentication (like OAuth Bearer) is invented, Git needs updates before
>>     credential helpers can take advantage of it (or leverage a new
>>     capability in libcurl).
>>
>>
>> Goals
>> =====
>>
>>  * As a user with multiple federated cloud identities:
> 
> I'm not sure if you mentioned it anywhere else, but this is specifically
> for cases where a user might have multiple identities _on the same host
> by DNS name_. The credential.useHttpPath config option might seem like it
> could help here, but the credential helper might pick the wrong identity
> that is the most-recent login. Either this workflow will require the user
> to re-login with every new URL or the fetches/clones will fail when the
> guess is wrong and the user would need to learn how to log into that other
> identity.
> 
> Please correct me if I'm wrong about any of this, but the details of your
> goals make it clear that the workflow will be greatly improved:

Such a scenario where multiple identities may be available for the same DNS
hostname would indeed be improved (with an appropriately enlightened
credential helper of course). As you mentioned, credential.useHttpPath can
also be used to workaround such a situation, but that just creates another
problem in that users need to provide the same set of credentials for each
repository with a full remote URL path that use the same identity.

By providing information about the auth challenge (including parameters
like authority or realm if present) would allow credential helpers select
or filter known identities and credentials automatically, avoiding user
input.

>>    * Reach out to a remote and have my credential helper automatically
>>      prompt me for the correct identity.
>>    * Leverage existing authentication systems built-in to many operating
>>      systems and devices to boost security and reduce reliance on passwords.
>>
>>  * As a Git host and/or cloud identity provider:
>>    
>>    * Leverage newest identity standards, enhancements, and threat
>>      mitigations - all without updating Git.
>>    * Enforce security policies (like requiring two-factor authentication)
>>      dynamically.
>>    * Allow integration with third party standard based identity providers in
>>      enterprises allowing customers to have a single plane of control for
>>      critical identities with access to source code.
> 
> I had a question with this part of your proposal:
> 
>>     Because the extra information forms an ordered list, and the existing
>>     credential helper I/O format only provides for simple key=value pairs,
>>     we introduce a new convention for transmitting an ordered list of
>>     values. Key names that are suffixed with a C-style array syntax should
>>     have values considered to form an order list, i.e. key[n]=value, where n
>>     is a zero based index of the values.
>>     
>>     For the WWW-Authenticate header values we opt to use the key wwwauth[n].
> ...
>> Git sends over standard input:
>>
>> protocol=https
>> host=example.com
>> wwwauth[0]=Bearer realm="login.example", scope="git.readwrite"
>> wwwauth[1]=Basic realm="login.example"
> 
> The important part here is that we provide a way to specify a multi-valued
> key as opposed to a "last one wins" key, right?
> 
> Using empty braces (wwwauth[]) would suffice to indicate this, right? That
> allows us to not care about the values inside the braces. The biggest
> issues I see with a value in the braces are:
> 
> 1. What if it isn't an integer?
> 2. What if we are missing a value?
> 3. What if they come out of order?
> 
> Without a value inside, then the order in which they appear provides
> implicit indices in their multi-valued list.
> 
> Other than that, I support this idea and will start looking at the code
> now.

There are two important things this extension to the I/O format provides:
1) multi-valued keys, and 2) ordering to the multiple values.

You are correct that dropping the integer index still means we still meet
requirement 1, and implicitly meet requirement 2. In this proposal I was
just being explicit in the ordering - it's not something I'm overly
attached to however, and may indeed make parsing or identifiying these
multi-valued keys easier on the credential helper side of things.

> Thanks,
> -Stolee

Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers

[Lessley Dennington]

This is a really exciting idea! Based on your patches, it seems to be a
great opportunity to add extensibility and flexibility to the credential
helper model without huge disruptions to the codebase. Well done!

On 9/13/22 12:25 PM, Matthew John Cheetham via GitGitGadget wrote:
>   3. Teach Git to specify authentication schemes other than Basic in
>      subsequent HTTP requests based on credential helper responses.
> 
This!! Yes!!
> 
> ...
> wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
> 
I think sending the fields individually (as you describe in this doc and
implement in your patches) is the right call. In my opinion, it's more
legible, consistent with the remote response, and aligns with your goal of
minimizing authentication-related actions in Git.

Best,

Lessley

[PATCH v2 0/6] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server based
heavily on git-daemon, with simple authentication configurable by command
line args.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Leverage newest identity standards, enhancements, and threat
     mitigations - all without updating Git.
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].

 3. Teach Git to specify authentication schemes other than Basic in
    subsequent HTTP requests based on credential helper responses.


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future flexibility
==================

By allowing the credential helpers decide the best authentication scheme, we
can allow the remote Git server to both offer new schemes (or remove old
ones) that enlightened credential helpers could take immediate advantage of,
and to use credentials that are much more tightly scoped and bound to the
specific request.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper would return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>



Should Git not control the set of authentication schemes?
=========================================================

One concern that the reader may have regarding these changes is in allowing
helpers to select the authentication mechanism to use, it may be possible
that a weaker form of authentication is used.

Take for example a Git remote server that responds with the following
authentication schemes:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Negotiate ...
WWW-Authenticate: Basic ...


Today Git (and libcurl) prefer to Negotiate over Basic authentication [13].
If a helper responded with authtype=basic Git would now be using a "less
secure" mechanism.

The reason we still propose the credential helper decide on the
authentication scheme is that Git is not the best placed entity to decide
what type of authentication should be used for a particular request (see
Design Principle 3).

OAuth Bearer tokens are often bundled in Basic Authorization headers [14],
but given that the tokens are/can be short-lived and have a highly scoped
set of permissions, this solution could be argued as being more secure than
something like NTLM [15]. Similarly, the user may wish to be consulted on
selecting a particular user account, or directly selecting an authentication
mechanism for a request that otherwise they would not be able to use.

Also, as new authentication protocols appear Git does not need to be
modified or updated for the user to take advantage of them; the credential
helpers take on the responsibility of learning and selecting the "best"
option.


Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616

 * [13] libcurl http.c pickoneauth Function
   https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416

 * [14] Git Credential Manager GitHub Host Provider (using PAT as password)
   https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157

 * [15] NT LAN Manager (NTLM) Authentication Protocol
   https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.

Matthew John Cheetham (6):
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests
  http: store all request headers on active_request_slot
  http: move proactive auth to first slot creation
  http: set specific auth scheme depending on credential
  t5556-http-auth: add test for HTTP auth hdr logic

 Documentation/git-credential.txt          |   18 +
 Makefile                                  |    2 +
 contrib/buildsystems/CMakeLists.txt       |   13 +
 credential.c                              |   18 +
 credential.h                              |   16 +
 git-curl-compat.h                         |   10 +
 http-push.c                               |  103 +-
 http-walker.c                             |    2 +-
 http.c                                    |  200 +++-
 http.h                                    |    4 +-
 remote-curl.c                             |   36 +-
 t/helper/.gitignore                       |    1 +
 t/helper/test-credential-helper-replay.sh |   14 +
 t/helper/test-http-server.c               | 1134 +++++++++++++++++++++
 t/t5556-http-auth.sh                      |  260 +++++
 15 files changed, 1695 insertions(+), 136 deletions(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh
 create mode 100644 t/helper/test-http-server.c
 create mode 100755 t/t5556-http-auth.sh


base-commit: 9c32cfb49c60fa8173b9666db02efe3b45a8522f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v1:

 1:  6426f9c3954 < -:  ----------- wincred: ignore unknown lines (do not die)
 2:  ae5c1bfc092 < -:  ----------- netrc: ignore unknown lines (do not die)
 3:  2ece562a595 < -:  ----------- osxkeychain: clarify that we ignore unknown lines
 4:  78e66d56605 ! 1:  f297c78f60a http: read HTTP WWW-Authenticate response headers
     @@ credential.h: struct credential {
      +	 * in the order they were received in the response.
      +	 */
      +	struct strvec wwwauth_headers;
     ++
     ++	/**
     ++	 * Internal use only. Used to keep track of split header fields
     ++	 * in order to fold multiple lines into one value.
     ++	 */
      +	unsigned header_is_last_match:1;
      +
       	unsigned approved:1,
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +
      +	/* Start of a new WWW-Authenticate header */
      +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
     -+		while (isspace(*val)) val++;
     ++		while (isspace(*val))
     ++			val++;
      +
      +		strvec_push(values, val);
      +		http_auth.header_is_last_match = 1;
 5:  936545004b8 ! 2:  0838d992744 credential: add WWW-Authenticate header to cred requests
     @@ Commit message
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Documentation/git-credential.txt ##
     -@@ Documentation/git-credential.txt: Git understands the following attributes:
     - 	were read (e.g., `url=https://example.com` would behave as if
     - 	`protocol=https` and `host=example.com` had been provided). This
     - 	can help callers avoid parsing URLs themselves.
     -+
     -+`wwwauth[n]`::
     +@@ Documentation/git-credential.txt: empty string.
     + Components which are missing from the URL (e.g., there is no
     + username in the example above) will be left unset.
     + 
     ++`wwwauth[]`::
      +
      +	When an HTTP response is received that includes one or more
      +	'WWW-Authenticate' authentication headers, these can be passed to Git
      +	(and subsequent credential helpers) with these attributes.
      +	Each 'WWW-Authenticate' header value should be passed as a separate
     -+	attribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers
     -+	appear in the HTTP response.
     - +
     - Note that specifying a protocol is mandatory and if the URL
     - doesn't specify a hostname (e.g., "cert:///path/to/file") the
     ++	attribute 'wwwauth[]' where the order of the attributes is the same
     ++	as they appear in the HTTP response.
     ++
     + GIT
     + ---
     + Part of the linkgit:git[1] suite
      
       ## credential.c ##
      @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
     @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const
      +				    const struct strvec *vec)
      +{
      +	int i = 0;
     ++	const char *full_key = xstrfmt("%s[]", key);
      +	for (; i < vec->nr; i++) {
     -+		const char *full_key = xstrfmt("%s[%d]", key, i);
      +		credential_write_item(fp, full_key, vec->v[i], 0);
     -+		free((void*)full_key);
      +	}
     ++	free((void*)full_key);
      +}
      +
       void credential_write(const struct credential *c, FILE *fp)
     @@ credential.c: void credential_write(const struct credential *c, FILE *fp)
       }
       
       static int run_credential_helper(struct credential *c,
     -
     - ## t/lib-httpd/apache.conf ##
     -@@ t/lib-httpd/apache.conf: SSLEngine On
     - 	Require valid-user
     - </LocationMatch>
     - 
     -+# Advertise two additional auth methods above "Basic".
     -+# Neither of them actually work but serve test cases showing these
     -+# additional auth headers are consumed correctly.
     -+<Location /auth-wwwauth/>
     -+	AuthType Basic
     -+	AuthName "git-auth"
     -+	AuthUserFile passwd
     -+	Require valid-user
     -+	SetEnvIf Authorization "^\S+" authz
     -+	Header always add WWW-Authenticate "Bearer authority=https://login.example.com" env=!authz
     -+	Header always add WWW-Authenticate "FooAuth foo=bar baz=1" env=!authz
     -+</Location>
     -+
     - RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]
     - RewriteCond %{REQUEST_URI} /git-receive-pack$
     - RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]
     -
     - ## t/t5551-http-fetch-smart.sh ##
     -@@ t/t5551-http-fetch-smart.sh: test_expect_success 'http auth forgets bogus credentials' '
     - 	expect_askpass both user@host
     - '
     - 
     -+test_expect_success 'http auth sends www-auth headers to credential helper' '
     -+	write_script git-credential-tee <<-\EOF &&
     -+		cmd=$1
     -+		teefile=credential-$cmd
     -+		if [ -f "$teefile" ]; then
     -+			rm $teefile
     -+		fi
     -+		(
     -+			while read line;
     -+			do
     -+				if [ -z "$line" ]; then
     -+					exit 0
     -+				fi
     -+				echo "$line" >> $teefile
     -+				echo $line
     -+			done
     -+		) | git credential-store $cmd
     -+	EOF
     -+
     -+	cat >expected-get <<-EOF &&
     -+	protocol=http
     -+	host=127.0.0.1:5551
     -+	wwwauth[0]=Bearer authority=https://login.example.com
     -+	wwwauth[1]=FooAuth foo=bar baz=1
     -+	wwwauth[2]=Basic realm="git-auth"
     -+	EOF
     -+
     -+	cat >expected-store <<-EOF &&
     -+	protocol=http
     -+	host=127.0.0.1:5551
     -+	username=user@host
     -+	password=pass@host
     -+	EOF
     -+
     -+	rm -f .git-credentials &&
     -+	test_config credential.helper tee &&
     -+	set_askpass user@host pass@host &&
     -+	(
     -+		PATH="$PWD:$PATH" &&
     -+		git ls-remote "$HTTPD_URL/auth-wwwauth/smart/repo.git"
     -+	) &&
     -+	expect_askpass both user@host &&
     -+	test_cmp expected-get credential-get &&
     -+	test_cmp expected-store credential-store
     -+'
     -+
     - test_expect_success 'client falls back from v2 to v0 to match server' '
     - 	GIT_TRACE_PACKET=$PWD/trace \
     - 	GIT_TEST_PROTOCOL_VERSION=2 \
 6:  20843e2051e = 3:  c62fef65f46 http: store all request headers on active_request_slot
 7:  cae7180bc37 = 4:  a790c01f9f2 http: move proactive auth to first slot creation
 8:  7f827067f55 ! 5:  b0b7cd7ee5e http: set specific auth scheme depending on credential
     @@ Commit message
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Documentation/git-credential.txt ##
     -@@ Documentation/git-credential.txt: Git understands the following attributes:
     - 	`protocol=https` and `host=example.com` had been provided). This
     - 	can help callers avoid parsing URLs themselves.
     +@@ Documentation/git-credential.txt: username in the example above) will be left unset.
     + 	attribute 'wwwauth[]' where the order of the attributes is the same
     + 	as they appear in the HTTP response.
       
      +`authtype`::
      +
     @@ Documentation/git-credential.txt: Git understands the following attributes:
      +	scheme for the `Authorization` header, and the `password` field is
      +	used as the raw unencoded authorization parameters of the same header.
      +
     - `wwwauth[n]`::
     - 
     - 	When an HTTP response is received that includes one or more
     + GIT
     + ---
     + Part of the linkgit:git[1] suite
      
       ## credential.c ##
      @@ credential.c: void credential_clear(struct credential *c)
     @@ git-curl-compat.h
       
      +/**
      + * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.
     ++ * However, only 7.69.0 fixes a bug where Bearer headers were not
     ++ * actually sent with reused connections on subsequent transfers
     ++ * (curl/curl@dea17b519dc1).
      + */
     -+#if LIBCURL_VERSION_NUM >= 0x073D00
     ++#if LIBCURL_VERSION_NUM >= 0x074500
      +#define GIT_CURL_HAVE_CURLAUTH_BEARER
      +#endif
      +
 -:  ----------- > 6:  f3f13ed8c82 t5556-http-auth: add test for HTTP auth hdr logic

-- 
gitgitgadget

[PATCH v2 1/6] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 ++++++++++
 http.c       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index 5d0502f51fd..03d43d352e7 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	const char *z = NULL;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		const char **v = values->v + values->nr - 1;
+		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
+
+		free((void*)*v);
+		*v = append;
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (skip_iprefix(buf.buf, "http/", &z))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1829,6 +1905,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

[PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[n]` properties where `n` is a
zero-indexed number, reflecting the order the WWW-Authenticate headers
appeared in the HTTP response.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  9 +++++++++
 credential.c                     | 12 ++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f18673017f5..0ff3cbc25b9 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -160,6 +160,15 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received that includes one or more
+	'WWW-Authenticate' authentication headers, these can be passed to Git
+	(and subsequent credential helpers) with these attributes.
+	Each 'WWW-Authenticate' header value should be passed as a separate
+	attribute 'wwwauth[]' where the order of the attributes is the same
+	as they appear in the HTTP response.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/credential.c b/credential.c
index 897b4679333..8a3ad6c0ae2 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	int i = 0;
+	const char *full_key = xstrfmt("%s[]", key);
+	for (; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free((void*)full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
-- 
gitgitgadget

Re: [PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests

[Jeff Hostetler]

On 10/21/22 1:07 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Add the value of the WWW-Authenticate response header to credential
> requests. Credential helpers that understand and support HTTP
> authentication and authorization can use this standard header (RFC 2616
> Section 14.47 [1]) to generate valid credentials.
> 
> WWW-Authenticate headers can contain information pertaining to the
> authority, authentication mechanism, or extra parameters/scopes that are
> required.
> 
> The current I/O format for credential helpers only allows for unique
> names for properties/attributes, so in order to transmit multiple header
> values (with a specific order) we introduce a new convention whereby a
> C-style array syntax is used in the property name to denote multiple
> ordered values for the same property.
> 
> In this case we send multiple `wwwauth[n]` properties where `n` is a
 > zero-indexed number, reflecting the order the WWW-Authenticate headers
 > appeared in the HTTP response.

Here (and maybe in the cover letter) you mention `wwwauth[n]` and `n`...
> +`wwwauth[]`::
> +
> +	When an HTTP response is received that includes one or more
> +	'WWW-Authenticate' authentication headers, these can be passed to Git
> +	(and subsequent credential helpers) with these attributes.
> +	Each 'WWW-Authenticate' header value should be passed as a separate
> +	attribute 'wwwauth[]' where the order of the attributes is the same
> +	as they appear in the HTTP response.

...but here you don't include the `n`.

[...]
> +static void credential_write_strvec(FILE *fp, const char *key,
> +				    const struct strvec *vec)
> +{
> +	int i = 0;
> +	const char *full_key = xstrfmt("%s[]", key);

...nor here.

Jeff

Re: [PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2022-10-28 11:22, Jeff Hostetler wrote:
> On 10/21/22 1:07 PM, Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add the value of the WWW-Authenticate response header to credential
>> requests. Credential helpers that understand and support HTTP
>> authentication and authorization can use this standard header (RFC 2616
>> Section 14.47 [1]) to generate valid credentials.
>>
>> WWW-Authenticate headers can contain information pertaining to the
>> authority, authentication mechanism, or extra parameters/scopes that are
>> required.
>>
>> The current I/O format for credential helpers only allows for unique
>> names for properties/attributes, so in order to transmit multiple header
>> values (with a specific order) we introduce a new convention whereby a
>> C-style array syntax is used in the property name to denote multiple
>> ordered values for the same property.
>>
>> In this case we send multiple `wwwauth[n]` properties where `n` is a
>> zero-indexed number, reflecting the order the WWW-Authenticate headers
>> appeared in the HTTP response.
> 
> Here (and maybe in the cover letter) you mention `wwwauth[n]` and `n`...
>> +`wwwauth[]`::
>> +
>> +    When an HTTP response is received that includes one or more
>> +    'WWW-Authenticate' authentication headers, these can be passed to Git
>> +    (and subsequent credential helpers) with these attributes.
>> +    Each 'WWW-Authenticate' header value should be passed as a separate
>> +    attribute 'wwwauth[]' where the order of the attributes is the same
>> +    as they appear in the HTTP response.
> 
> ...but here you don't include the `n`.
> 
> [...]
>> +static void credential_write_strvec(FILE *fp, const char *key,
>> +                    const struct strvec *vec)
>> +{
>> +    int i = 0;
>> +    const char *full_key = xstrfmt("%s[]", key);
> 
> ...nor here.
> 
Ah. This is an oversight in my v2 rebasing! Will fix in v3.

Thanks,
Matthew

[PATCH v2 3/6] http: store all request headers on active_request_slot

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Once a list of headers has been set on the curl handle, it is not
possible to recover that `struct curl_slist` instance to add or modify
headers.

In future commits we will want to modify the set of request headers in
response to an authentication challenge/401 response from the server,
with information provided by a credential helper.

There are a number of different places where curl is used for an HTTP
request, and they do not have a common handling of request headers.
However, given that they all do call the `start_active_slot()` function,
either directly or indirectly via `run_slot()` or `run_one_slot()`, we
use this as the point to set the `CURLOPT_HTTPHEADER` option just
before the request is made.

We collect all request headers in a `struct curl_slist` on the
`struct active_request_slot` that is obtained from a call to
`get_active_slot(int)`. This function now takes a single argument to
define if the initial set of headers on the slot should include the
"Pragma: no-cache" header, along with all extra headers specified via
`http.extraHeader` config values.

The active request slot obtained from `get_active_slot(int)` will always
contain a fresh set of default headers and any headers set in previous
usages of this slot will be freed.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http-push.c   | 103 ++++++++++++++++++++++----------------------------
 http-walker.c |   2 +-
 http.c        |  82 ++++++++++++++++++----------------------
 http.h        |   4 +-
 remote-curl.c |  36 +++++++++---------
 5 files changed, 101 insertions(+), 126 deletions(-)

diff --git a/http-push.c b/http-push.c
index 5f4340a36e6..2b40959b376 100644
--- a/http-push.c
+++ b/http-push.c
@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,
 	curl_easy_setopt(curl, CURLOPT_UPLOAD, 1);
 }
 
-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)
+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,
+	struct remote_lock *lock, enum dav_header_flag options)
 {
 	struct strbuf buf = STRBUF_INIT;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
 	if (options & DAV_HEADER_IF) {
 		strbuf_addf(&buf, "If: (<%s>)", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_LOCK) {
 		strbuf_addf(&buf, "Lock-Token: <%s>", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_TIMEOUT) {
 		strbuf_addf(&buf, "Timeout: Second-%ld", lock->timeout);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	strbuf_release(&buf);
 
-	return dav_headers;
+	return headers;
 }
 
 static void finish_request(struct transfer_request *request);
@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)
 
 	request->url = get_remote_object_url(repo->url, hex, 1);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MKCOL);
@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)
 	strbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);
 	request->url = strbuf_detach(&buf, NULL);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http(slot->curl, request->url, DAV_PUT,
@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)
 static void start_move(struct transfer_request *request)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MOVE);
-	dav_headers = curl_slist_append(dav_headers, request->dest);
-	dav_headers = curl_slist_append(dav_headers, "Overwrite: T");
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
+	slot->headers = curl_slist_append(slot->headers, request->dest);
+	slot->headers = curl_slist_append(slot->headers, "Overwrite: T");
 
 	if (start_active_slot(slot)) {
 		request->slot = slot;
@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
 	lock->refreshing = 1;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_LOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)
 	}
 
 	lock->refreshing = 0;
-	curl_slist_free_all(dav_headers);
 
 	return rc;
 }
@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	char *ep;
 	char timeout_header[25];
 	struct remote_lock *lock = NULL;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	char *escaped;
 
@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	while (ep) {
 		char saved_character = ep[1];
 		ep[1] = '\0';
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
 		curl_setup_http_get(slot->curl, url, DAV_MKCOL);
 		if (start_active_slot(slot)) {
@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	strbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);
 	free(escaped);
 
+	slot = get_active_slot(0);
+	slot->results = &results;
+
 	xsnprintf(timeout_header, sizeof(timeout_header), "Timeout: Second-%ld", timeout);
-	dav_headers = curl_slist_append(dav_headers, timeout_header);
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
+	slot->headers = curl_slist_append(slot->headers, timeout_header);
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
 
-	slot = get_active_slot();
-	slot->results = &results;
 	curl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	CALLOC_ARRAY(lock, 1);
@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 		fprintf(stderr, "Unable to start LOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
 
@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct remote_lock *prev = repo->locks;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_LOCK);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)
 		fprintf(stderr, "Unable to start UNLOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
-
 	if (repo->locks == lock) {
 		repo->locks = lock->next;
 	} else {
@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	struct remote_ls_ctx ls;
 
@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,
 
 	strbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 1");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 1");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,
 	free(url);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 }
 
 static void get_remote_object_list(unsigned char parent)
@@ -1199,7 +1189,6 @@ static int locking_available(void)
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	int lock_flags = 0;
 	char *escaped;
@@ -1208,14 +1197,14 @@ static int locking_available(void)
 	strbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);
 	free(escaped);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 0");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 0");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, repo->url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1257,7 +1246,6 @@ static int locking_available(void)
 
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 
 	return lock_flags;
 }
@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers;
-
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
 	strbuf_addf(&out_buffer.buf, "%s\n", oid_to_hex(oid));
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF);
+
 	curl_setup_http(slot->curl, lock->url, DAV_PUT,
 			&out_buffer, fwrite_null);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)
 	struct buffer buffer = { STRBUF_INIT, 0 };
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 
 	remote_ls("refs/", (PROCESS_FILES | RECURSIVE),
 		  add_remote_info_ref, &buffer.buf);
 	if (!aborted) {
-		dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
+		slot->headers = append_dav_token_headers(slot->headers, lock,
+			DAV_HEADER_IF);
+
 		curl_setup_http(slot->curl, lock->url, DAV_PUT,
 				&buffer, fwrite_null);
-		curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 		if (start_active_slot(slot)) {
 			run_active_slot(slot);
@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)
 	if (dry_run)
 		return 0;
 	url = xstrfmt("%s%s", repo->url, remote_ref->name);
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
 	curl_setup_http_get(slot->curl, url, DAV_DELETE);
 	if (start_active_slot(slot)) {
diff --git a/http-walker.c b/http-walker.c
index b8f0f98ae14..8747de2fcdb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)
 	 * Use a callback to process the result, since another request
 	 * may fail and need to have alternates loaded before continuing
 	 */
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_alternates_response;
 	alt_req.walker = walker;
 	slot->callback_data = &alt_req;
diff --git a/http.c b/http.c
index 03d43d352e7..f2ebb17c8c4 100644
--- a/http.c
+++ b/http.c
@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =
 	| CURLAUTH_DIGEST_IE
 	| CURLAUTH_DIGEST;
 
-static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
 static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
 
 static struct curl_slist *host_resolutions;
@@ -1133,11 +1131,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 	if (remote)
 		var_override(&http_proxy_authmethod, remote->http_proxy_authmethod);
 
-	pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma: no-cache");
-	no_pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma:");
-
 	{
 		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
 		if (http_max_requests)
@@ -1199,6 +1192,8 @@ void http_cleanup(void)
 
 	while (slot != NULL) {
 		struct active_request_slot *next = slot->next;
+		if (slot->headers)
+			curl_slist_free_all(slot->headers);
 		if (slot->curl) {
 			xmulti_remove_handle(slot);
 			curl_easy_cleanup(slot->curl);
@@ -1215,12 +1210,6 @@ void http_cleanup(void)
 
 	string_list_clear(&extra_http_headers, 0);
 
-	curl_slist_free_all(pragma_header);
-	pragma_header = NULL;
-
-	curl_slist_free_all(no_pragma_header);
-	no_pragma_header = NULL;
-
 	curl_slist_free_all(host_resolutions);
 	host_resolutions = NULL;
 
@@ -1255,7 +1244,18 @@ void http_cleanup(void)
 	FREE_AND_NULL(cached_accept_language);
 }
 
-struct active_request_slot *get_active_slot(void)
+static struct curl_slist *http_copy_default_headers(void)
+{
+	struct curl_slist *headers = NULL;
+	const struct string_list_item *item;
+
+	for_each_string_list_item(item, &extra_http_headers)
+		headers = curl_slist_append(headers, item->string);
+
+	return headers;
+}
+
+struct active_request_slot *get_active_slot(int no_pragma_header)
 {
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
@@ -1277,6 +1277,7 @@ struct active_request_slot *get_active_slot(void)
 		newslot->curl = NULL;
 		newslot->in_use = 0;
 		newslot->next = NULL;
+		newslot->headers = NULL;
 
 		slot = active_queue_head;
 		if (!slot) {
@@ -1294,6 +1295,15 @@ struct active_request_slot *get_active_slot(void)
 		curl_session_count++;
 	}
 
+	if (slot->headers)
+		curl_slist_free_all(slot->headers);
+
+	slot->headers = http_copy_default_headers();
+
+	if (!no_pragma_header)
+		slot->headers = curl_slist_append(slot->headers,
+			"Pragma: no-cache");
+
 	active_requests++;
 	slot->in_use = 1;
 	slot->results = NULL;
@@ -1303,7 +1313,6 @@ struct active_request_slot *get_active_slot(void)
 	curl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);
 	if (curl_save_cookies)
 		curl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
 	curl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);
 	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
 	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);
@@ -1335,9 +1344,12 @@ struct active_request_slot *get_active_slot(void)
 
 int start_active_slot(struct active_request_slot *slot)
 {
-	CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);
+	CURLMcode curlm_result;
 	int num_transfers;
 
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);
+	curlm_result = curl_multi_add_handle(curlm, slot->curl);
+
 	if (curlm_result != CURLM_OK &&
 	    curlm_result != CURLM_CALL_MULTI_PERFORM) {
 		warning("curl_multi_add_handle failed: %s",
@@ -1652,17 +1664,6 @@ int run_one_slot(struct active_request_slot *slot,
 	return handle_curl_result(results);
 }
 
-struct curl_slist *http_copy_default_headers(void)
-{
-	struct curl_slist *headers = NULL;
-	const struct string_list_item *item;
-
-	for_each_string_list_item(item, &extra_http_headers)
-		headers = curl_slist_append(headers, item->string);
-
-	return headers;
-}
-
 static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)
 {
 	char *ptr;
@@ -1880,12 +1881,11 @@ static int http_request(const char *url,
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *headers = http_copy_default_headers();
-	struct strbuf buf = STRBUF_INIT;
+	int no_cache = options && options->no_cache;
 	const char *accept_language;
 	int ret;
 
-	slot = get_active_slot();
+	slot = get_active_slot(!no_cache);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);
 
 	if (!result) {
@@ -1910,27 +1910,23 @@ static int http_request(const char *url,
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-		headers = curl_slist_append(headers, accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			accept_language);
 
-	strbuf_addstr(&buf, "Pragma:");
-	if (options && options->no_cache)
-		strbuf_addstr(&buf, " no-cache");
 	if (options && options->initial_request &&
 	    http_follow_config == HTTP_FOLLOW_INITIAL)
 		curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);
 
-	headers = curl_slist_append(headers, buf.buf);
-
 	/* Add additional headers here */
 	if (options && options->extra_headers) {
 		const struct string_list_item *item;
 		for_each_string_list_item(item, options->extra_headers) {
-			headers = curl_slist_append(headers, item->string);
+			slot->headers = curl_slist_append(slot->headers,
+				item->string);
 		}
 	}
 
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
 	curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);
 
@@ -1948,9 +1944,6 @@ static int http_request(const char *url,
 		curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
 				options->effective_url);
 
-	curl_slist_free_all(headers);
-	strbuf_release(&buf);
-
 	return ret;
 }
 
@@ -2311,12 +2304,10 @@ struct http_pack_request *new_direct_http_pack_request(
 		goto abort;
 	}
 
-	preq->slot = get_active_slot();
+	preq->slot = get_active_slot(1);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
-	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
-		no_pragma_header);
 
 	/*
 	 * If there is data present from a previous transfer attempt,
@@ -2481,14 +2472,13 @@ struct http_object_request *new_http_object_request(const char *base_url,
 		}
 	}
 
-	freq->slot = get_active_slot();
+	freq->slot = get_active_slot(1);
 
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
 
 	/*
 	 * If we have successfully processed data from a previous fetch
diff --git a/http.h b/http.h
index 3c94c479100..a304cc408b2 100644
--- a/http.h
+++ b/http.h
@@ -22,6 +22,7 @@ struct slot_results {
 struct active_request_slot {
 	CURL *curl;
 	int in_use;
+	struct curl_slist *headers;
 	CURLcode curl_result;
 	long http_code;
 	int *finished;
@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
 curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);
 
 /* Slot lifecycle functions */
-struct active_request_slot *get_active_slot(void);
+struct active_request_slot *get_active_slot(int no_pragma_header);
 int start_active_slot(struct active_request_slot *slot);
 void run_active_slot(struct active_request_slot *slot);
 void finish_all_active_slots(void);
@@ -64,7 +65,6 @@ void step_active_slots(void);
 void http_init(struct remote *remote, const char *url,
 	       int proactive_auth);
 void http_cleanup(void);
-struct curl_slist *http_copy_default_headers(void);
 
 extern long int git_curl_ipresolve;
 extern int active_requests;
diff --git a/remote-curl.c b/remote-curl.c
index 72dfb8fb86a..edbd4504beb 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,
 static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	struct strbuf buf = STRBUF_INIT;
 	int err;
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000");
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);
 
 	err = run_slot(slot, results);
 
-	curl_slist_free_all(headers);
 	strbuf_release(&buf);
 	return err;
 }
@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)
 static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
 	size_t gzip_size = 0;
@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 			needs_100_continue = 1;
 	}
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
-	headers = curl_slist_append(headers, needs_100_continue ?
+retry:
+	slot = get_active_slot(0);
+
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, needs_100_continue ?
 		"Expect: 100-continue" : "Expect:");
 
 	/* Add Accept-Language header */
 	if (rpc->hdr_accept_language)
-		headers = curl_slist_append(headers, rpc->hdr_accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->hdr_accept_language);
 
 	/* Add the extra Git-Protocol header */
 	if (rpc->protocol_header)
-		headers = curl_slist_append(headers, rpc->protocol_header);
-
-retry:
-	slot = get_active_slot();
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->protocol_header);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -955,7 +953,8 @@ retry:
 		/* The request body is large and the size cannot be predicted.
 		 * We must use chunked encoding to send it.
 		 */
-		headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+		slot->headers = curl_slist_append(slot->headers,
+			"Transfer-Encoding: chunked");
 		rpc->initial_buffer = 1;
 		curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);
 		curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);
@@ -1002,7 +1001,8 @@ retry:
 
 		gzip_size = stream.total_out;
 
-		headers = curl_slist_append(headers, "Content-Encoding: gzip");
+		slot->headers = curl_slist_append(slot->headers,
+			"Content-Encoding: gzip");
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));
 
@@ -1025,7 +1025,6 @@ retry:
 		}
 	}
 
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);
 	rpc_in_data.rpc = rpc;
 	rpc_in_data.slot = slot;
@@ -1055,7 +1054,6 @@ retry:
 	if (stateless_connect)
 		packet_response_end(rpc->in);
 
-	curl_slist_free_all(headers);
 	free(gzip_body);
 	return err;
 }
-- 
gitgitgadget

[PATCH v2 4/6] http: move proactive auth to first slot creation

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rather than proactively seek credentials to authenticate a request at
`http_init()` time, do it when the first `active_request_slot` is
created.

Because credential helpers may modify the headers used for a request we
can only auth when a slot is created (when we can first start to gather
request headers).

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/http.c b/http.c
index f2ebb17c8c4..17b47195d22 100644
--- a/http.c
+++ b/http.c
@@ -515,18 +515,18 @@ static int curl_empty_auth_enabled(void)
 	return 0;
 }
 
-static void init_curl_http_auth(CURL *result)
+static void init_curl_http_auth(struct active_request_slot *slot)
 {
 	if (!http_auth.username || !*http_auth.username) {
 		if (curl_empty_auth_enabled())
-			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
+			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
 	}
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
+	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
 }
 
 /* *var must be free-able */
@@ -901,9 +901,6 @@ static CURL *get_curl_handle(void)
 #endif
 	}
 
-	if (http_proactive_auth)
-		init_curl_http_auth(result);
-
 	if (getenv("GIT_SSL_VERSION"))
 		ssl_version = getenv("GIT_SSL_VERSION");
 	if (ssl_version && *ssl_version) {
@@ -1260,6 +1257,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
 
+	int proactive_auth = 0;
 	int num_transfers;
 
 	/* Wait for a slot to open up if the queue is full */
@@ -1282,6 +1280,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 		slot = active_queue_head;
 		if (!slot) {
 			active_queue_head = newslot;
+
+			/* Auth first slot if asked for proactive auth */
+			proactive_auth = http_proactive_auth;
 		} else {
 			while (slot->next != NULL)
 				slot = slot->next;
@@ -1336,8 +1337,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 
 	curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
-	if (http_auth.password || curl_empty_auth_enabled())
-		init_curl_http_auth(slot->curl);
+
+	if (http_auth.password || curl_empty_auth_enabled() || proactive_auth)
+		init_curl_http_auth(slot);
 
 	return slot;
 }
-- 
gitgitgadget

[PATCH v2 5/6] http: set specific auth scheme depending on credential

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a new credential field `authtype` that can be used by
credential helpers to indicate the type of the credential or
authentication mechanism to use for a request.

Modify http.c to now specify the correct authentication scheme or
credential type when authenticating the curl handle. If the new
`authtype` field in the credential structure is `NULL` or "Basic" then
use the existing username/password options. If the field is "Bearer"
then use the OAuth bearer token curl option. Otherwise, the `authtype`
field is the authentication scheme and the `password` field is the
raw, unencoded value.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  9 +++++++++
 credential.c                     |  5 +++++
 credential.h                     |  1 +
 git-curl-compat.h                | 10 ++++++++++
 http.c                           | 24 +++++++++++++++++++++---
 5 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 0ff3cbc25b9..82ade09b5e9 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -169,6 +169,15 @@ username in the example above) will be left unset.
 	attribute 'wwwauth[]' where the order of the attributes is the same
 	as they appear in the HTTP response.
 
+`authtype`::
+
+	Indicates the type of authentication scheme used. If this is not
+	present the default is "Basic".
+	Known values include "Basic", "Digest", and "Bearer".
+	If an unknown value is provided, this is taken as the authentication
+	scheme for the `Authorization` header, and the `password` field is
+	used as the raw unencoded authorization parameters of the same header.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/credential.c b/credential.c
index 8a3ad6c0ae2..a556f9f375a 100644
--- a/credential.c
+++ b/credential.c
@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "path")) {
 			free(c->path);
 			c->path = xstrdup(value);
+		} else if (!strcmp(key, "authtype")) {
+			free(c->authtype);
+			c->authtype = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "authtype", c->authtype, 0);
 	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
diff --git a/credential.h b/credential.h
index 6f2e5bc610b..8d580b054d0 100644
--- a/credential.h
+++ b/credential.h
@@ -140,6 +140,7 @@ struct credential {
 	char *protocol;
 	char *host;
 	char *path;
+	char *authtype;
 };
 
 #define CREDENTIAL_INIT { \
diff --git a/git-curl-compat.h b/git-curl-compat.h
index 56a83b6bbd8..839049f6dfe 100644
--- a/git-curl-compat.h
+++ b/git-curl-compat.h
@@ -126,4 +126,14 @@
 #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS
 #endif
 
+/**
+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.
+ * However, only 7.69.0 fixes a bug where Bearer headers were not
+ * actually sent with reused connections on subsequent transfers
+ * (curl/curl@dea17b519dc1).
+ */
+#if LIBCURL_VERSION_NUM >= 0x074500
+#define GIT_CURL_HAVE_CURLAUTH_BEARER
+#endif
+
 #endif
diff --git a/http.c b/http.c
index 17b47195d22..ac620bcbf0c 100644
--- a/http.c
+++ b/http.c
@@ -517,7 +517,8 @@ static int curl_empty_auth_enabled(void)
 
 static void init_curl_http_auth(struct active_request_slot *slot)
 {
-	if (!http_auth.username || !*http_auth.username) {
+	if (!http_auth.authtype &&
+		(!http_auth.username || !*http_auth.username)) {
 		if (curl_empty_auth_enabled())
 			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
@@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
+	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
+				|| !strcasecmp(http_auth.authtype, "digest")) {
+		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
+			http_auth.username);
+		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
+			http_auth.password);
+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
+	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
+		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
+		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
+			http_auth.password);
+#endif
+	} else {
+		struct strbuf auth = STRBUF_INIT;
+		strbuf_addf(&auth, "Authorization: %s %s",
+			http_auth.authtype, http_auth.password);
+		slot->headers = curl_slist_append(slot->headers, auth.buf);
+		strbuf_release(&auth);
+	}
 }
 
 /* *var must be free-able */
-- 
gitgitgadget

[PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a series of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers can respond
to requests that contain WWW-Authenticate information with the ability
to select the response Authenticate header scheme.

Introduce a mini HTTP server helper that provides a frontend for the
git-http-backend, with support for arbitrary authentication schemes.
The test-http-server is based heavily on the git-daemon, and forwards
all successfully authenticated requests to the http-backend.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                                  |    2 +
 contrib/buildsystems/CMakeLists.txt       |   13 +
 t/helper/.gitignore                       |    1 +
 t/helper/test-credential-helper-replay.sh |   14 +
 t/helper/test-http-server.c               | 1134 +++++++++++++++++++++
 t/t5556-http-auth.sh                      |  260 +++++
 6 files changed, 1424 insertions(+)
 create mode 100755 t/helper/test-credential-helper-replay.sh
 create mode 100644 t/helper/test-http-server.c
 create mode 100755 t/t5556-http-auth.sh

diff --git a/Makefile b/Makefile
index d93ad956e58..39b130f711d 100644
--- a/Makefile
+++ b/Makefile
@@ -1500,6 +1500,8 @@ else
 	endif
 	BASIC_CFLAGS += $(CURL_CFLAGS)
 
+	TEST_PROGRAMS_NEED_X += test-http-server
+
 	REMOTE_CURL_PRIMARY = git-remote-http$X
 	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
 	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 787738e6fa3..45251695ce0 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -989,6 +989,19 @@ set(wrapper_scripts
 set(wrapper_test_scripts
 	test-fake-ssh test-tool)
 
+if(CURL_FOUND)
+       list(APPEND wrapper_test_scripts test-http-server)
+
+       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+       target_link_libraries(test-http-server common-main)
+
+       if(MSVC)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+       endif()
+endif()
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..1a94ab6eed5 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+test-http-server
diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
new file mode 100755
index 00000000000..03e5e63dad6
--- /dev/null
+++ b/t/helper/test-credential-helper-replay.sh
@@ -0,0 +1,14 @@
+cmd=$1
+teefile=$cmd-actual.cred
+catfile=$cmd-response.cred
+rm -f $teefile
+while read line;
+do
+	if test -z "$line"; then
+		break;
+	fi
+	echo "$line" >> $teefile
+done
+if test "$cmd" = "get"; then
+	cat $catfile
+fi
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..92139c04c90
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,1134 @@
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--anonymous-allowed]\n"
+"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
+;
+
+/* Timeout, and initial timeout */
+static unsigned int timeout;
+static unsigned int init_timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+static void set_keep_alive(int sockfd)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+//////////////////////////////////////////////////////////////////
+// The code in this section is used by "worker" instances to service
+// a single connection from a client.  The worker talks to the client
+// on 0 and 1.
+//////////////////////////////////////////////////////////////////
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK       = 0,
+	/*
+	 * Various errors while processing the request and/or the response.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_IO_ERROR = 1<<0,
+	/*
+	 * Close the socket and clean up.  Does not imply an error.
+	 */
+	WR_HANGUP   = 1<<1,
+
+	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+};
+
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	ssize_t content_length;
+};
+
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = -1 \
+	}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
+static enum worker_result send_http_error(
+	int fd,
+	int http_code, const char *http_code_name,
+	int retry_after_seconds, struct string_list *response_headers,
+	enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n",
+		    http_code, http_code_name);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
+	if (retry_after_seconds > 0)
+		strbuf_addf  (&response_header, "Retry-After: %d\r\n", retry_after_seconds);
+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_OK | WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* store common request headers separately */
+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
+			req->content_length = strtol(hv, &hp, 10);
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->content_length >= 0)
+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req, const char *user)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	if (write(1, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+			req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
+				req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
+				req->content_type);
+	if (req->content_length >= 0)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
+				(intmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(1);
+	close(0);
+	return !!res;
+}
+
+enum auth_result {
+	AUTH_UNKNOWN = 0,
+	AUTH_DENY = 1,
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	const char *scheme;
+	const char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(struct strbuf *scheme)
+{
+	int i;
+	struct auth_module *mod;
+	for (i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme->buf))
+			return mod;
+	}
+
+	return NULL;
+}
+
+static void add_auth_module(struct auth_module *mod)
+{
+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+	auth_modules[auth_modules_nr++] = mod;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/* ask all auth modules to validate the request */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (!split[0] || !split[1]) continue;
+
+			// trim trailing space ' '
+			strbuf_setlen(split[0], split[0]->len - 1);
+
+			mod = get_auth_module(split[0]);
+			if (mod) {
+
+				for_each_string_list_item(token, mod->tokens) {
+					if (!strcmp(split[1]->buf, token->string)) {
+						result = AUTH_ALLOW;
+						goto done;
+					}
+				}
+
+				if (result != AUTH_UNKNOWN)
+					goto done;
+			}
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (allow_anonymous)
+			break;
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
+	}
+
+	strbuf_list_free(split);
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
+	if (is_git_request(req))
+		return do__git(req, user);
+
+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
+			       WR_OK | WR_HANGUP);
+}
+
+static enum worker_result worker(void)
+{
+	struct req req = REQ__INIT;
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0);
+
+	while (1) {
+		req__release(&req);
+
+		alarm(init_timeout ? init_timeout : timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+
+		wr = dispatch(&req);
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+	}
+
+	close(0);
+	close(1);
+
+	return !!(wr & WR_IO_ERROR);
+}
+
+//////////////////////////////////////////////////////////////////
+// This section contains the listener and child-process management
+// code used by the primary instance to accept incoming connections
+// and dispatch them to async child process "worker" instances.
+//////////////////////////////////////////////////////////////////
+
+static int addrcmp(const struct sockaddr_storage *s1,
+		   const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+} *firstborn;
+
+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child *newborn, **cradle;
+
+	newborn = xcalloc(1, sizeof(*newborn));
+	live_children++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+/*
+ * This gets called if the number of connections grows
+ * past "max_connections".
+ *
+ * We kill the newest connection from a duplicate IP.
+ */
+static void kill_some_child(void)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+static void check_dead_children(void)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			const char *dead = "";
+			if (status)
+				dead = " (with error)";
+			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
+
+			/* remove the child */
+			*cradle = blanket->next;
+			live_children--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child();
+		sleep(1);  /* give it some time to die */
+		check_dead_children();
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	if (!reuseaddr)
+		return 0;
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd);
+
+	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	if (!listen_addr->nr)
+		setup_named_sock("127.0.0.1", listen_port, socklist);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children();
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+//////////////////////////////////////////////////////////////////
+// This section is executed by both the primary instance and all
+// worker instances.  So, yes, each child-process re-parses the
+// command line argument and re-discovers how it should behave.
+//////////////////////////////////////////////////////////////////
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+	struct auth_module *mod = NULL;
+
+	trace2_cmd_name("test-http-server");
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--init-timeout=", &v)) {
+			init_timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+		if (skip_prefix(arg, "--allow-anonymous", &v)) {
+			allow_anonymous = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--auth=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			// trim trailing ':'
+			if (p[1])
+				strbuf_setlen(p[0], p[0]->len - 1);
+
+			if (get_auth_module(p[0])) {
+				error("duplicate auth scheme '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			mod = xmalloc(sizeof(struct auth_module));
+			mod->scheme = xstrdup(p[0]->buf);
+			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
+			mod->tokens = xmalloc(sizeof(struct string_list));
+			string_list_init_dup(mod->tokens);
+
+			add_auth_module(mod);
+
+			strbuf_list_free(p);
+			continue;
+		}
+		if (skip_prefix(arg, "--auth-token=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			if (!p[1]) {
+				error("missing token value '%s'\n", v);
+				usage(test_http_auth_usage);
+			}
+
+			// trim trailing ':'
+			strbuf_setlen(p[0], p[0]->len - 1);
+
+			mod = get_auth_module(p[0]);
+			if (!mod) {
+				error("auth scheme not defined '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			string_list_append(mod->tokens, p[1]->buf);
+			strbuf_list_free(p);
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..43f1791a0fe
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,260 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
+REPO_DIR="$(pwd)"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$(pwd)"/pid-file.pid
+SERVER_LOG="$(pwd)"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
+	&& export CREDENTIAL_HELPER
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
+per_test_cleanup () {
+	stop_http_server &&
+	rm -f OUT.* &&
+	rm -f *.cred
+}
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server --allow-anonymous &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=bearer:secret-token &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-token
+	authtype=bearer
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-token
+	authtype=bearer
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=basic:$USERPASS64 &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	authtype=basic
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	authtype=basic
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=foobar:alg=test\ widget=1 \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=foobar:SECRET-FOOBAR-VALUE &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=SECRET-FOOBAR-VALUE
+	authtype=foobar
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=SECRET-FOOBAR-VALUE
+	authtype=foobar
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=bearer:secret-token &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >erase-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-token
+	authtype=bearer
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-token
+	authtype=bearer
+	EOF
+
+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp erase-expected.cred erase-actual.cred
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Derrick Stolee]

On 10/21/22 1:08 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>

> @@ -1500,6 +1500,8 @@ else
>  	endif
>  	BASIC_CFLAGS += $(CURL_CFLAGS)
>  
> +	TEST_PROGRAMS_NEED_X += test-http-server
> +
>  	REMOTE_CURL_PRIMARY = git-remote-http$X
>  	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
>  	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)

This hunk is in the "else" block of "ifdef NO_CURL",
so this makes sense for why TEST_PROGRAMS_NEED_X is
augmented here, away from other instances.

> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
> index 787738e6fa3..45251695ce0 100644
> --- a/contrib/buildsystems/CMakeLists.txt
> +++ b/contrib/buildsystems/CMakeLists.txt
> @@ -989,6 +989,19 @@ set(wrapper_scripts
>  set(wrapper_test_scripts
>  	test-fake-ssh test-tool)
>  
> +if(CURL_FOUND)
> +       list(APPEND wrapper_test_scripts test-http-server)
> +
> +       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
> +       target_link_libraries(test-http-server common-main)
> +
> +       if(MSVC)
> +               set_target_properties(test-http-server
> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
> +               set_target_properties(test-http-server
> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
> +       endif()
> +endif()

And this file has the pattern of many "if(CURL_FOUND)"
blocks with isolated purposes, so it makes sense to
have this be an isolated change instead of grouped with
a different case.

> diff --git a/t/helper/.gitignore b/t/helper/.gitignore
> index 8c2ddcce95f..1a94ab6eed5 100644
> --- a/t/helper/.gitignore
> +++ b/t/helper/.gitignore
> @@ -1,2 +1,3 @@
>  /test-tool
>  /test-fake-ssh
> +test-http-server

Should this start with a "/" like the other entries?

> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
> new file mode 100755
> index 00000000000..03e5e63dad6
> --- /dev/null
> +++ b/t/helper/test-credential-helper-replay.sh
> @@ -0,0 +1,14 @@
> +cmd=$1
> +teefile=$cmd-actual.cred
> +catfile=$cmd-response.cred
> +rm -f $teefile
> +while read line;
> +do
> +	if test -z "$line"; then
> +		break;
> +	fi
> +	echo "$line" >> $teefile
> +done
> +if test "$cmd" = "get"; then
> +	cat $catfile
> +fi

Should this be a helper method within another script, such
as t/lib-credential.sh or t/lib-httpd.sh? The read over
stdin will still work, as in this example:

read_chunk() {
	while read line; do
		case "$line" in
		--) break ;;
		*) echo "$line" ;;
		esac
	done
}

> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c

> @@ -0,0 +1,1134 @@
> +#include "config.h"
> +#include "run-command.h"
> +#include "strbuf.h"
> +#include "string-list.h"
> +#include "trace2.h"
> +#include "version.h"
> +#include "dir.h"
> +#include "date.h"
> +
> +#define TR2_CAT "test-http-server"
> +
> +static const char *pid_file;
> +static int verbose;
> +static int reuseaddr;
> +
> +static const char test_http_auth_usage[] =
> +"http-server [--verbose]\n"
> +"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
> +"           [--reuseaddr] [--pid-file=<file>]\n"
> +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
> +"           [--anonymous-allowed]\n"
> +"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
> +;

These are a lot of options to implement all at once. They are probably
simple enough, but depending on the implementation and tests, it might
be helpful to split this patch into smaller ones that introduce these
options along with the tests that exercise each. That will help
verify that they are being tested properly instead of needing to track
back and forth across the patch for each one.

> +
> +/* Timeout, and initial timeout */
> +static unsigned int timeout;
> +static unsigned int init_timeout;
> +
> +static void logreport(const char *label, const char *err, va_list params)
> +{
> +	struct strbuf msg = STRBUF_INIT;
> +
> +	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
> +	strbuf_vaddf(&msg, err, params);
> +	strbuf_addch(&msg, '\n');
> +
> +	fwrite(msg.buf, sizeof(char), msg.len, stderr);
> +	fflush(stderr);
> +
> +	strbuf_release(&msg);
> +}
> +
> +__attribute__((format (printf, 1, 2)))
> +static void logerror(const char *err, ...)
> +{
> +	va_list params;
> +	va_start(params, err);
> +	logreport("error", err, params);
> +	va_end(params);
> +}
> +
> +__attribute__((format (printf, 1, 2)))
> +static void loginfo(const char *err, ...)
> +{
> +	va_list params;
> +	if (!verbose)
> +		return;
> +	va_start(params, err);
> +	logreport("info", err, params);
> +	va_end(params);
> +}

I wonder how much of this we need or is just a nice thing. I would
err on the side of making things as simple as possible, but being
able to debug this test server may be important based on your
experience.

> +static void set_keep_alive(int sockfd)
> +{
> +	int ka = 1;
> +
> +	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
> +		if (errno != ENOTSOCK)
> +			logerror("unable to set SO_KEEPALIVE on socket: %s",
> +				strerror(errno));
> +	}
> +}
> +
> +//////////////////////////////////////////////////////////////////
> +// The code in this section is used by "worker" instances to service
> +// a single connection from a client.  The worker talks to the client
> +// on 0 and 1.
> +//////////////////////////////////////////////////////////////////

Use /* */ style comments. You can repeat the asterisks to get a
similar visual block.

> +
> +enum worker_result {
> +	/*
> +	 * Operation successful.
> +	 * Caller *might* keep the socket open and allow keep-alive.
> +	 */
> +	WR_OK       = 0,
> +	/*
> +	 * Various errors while processing the request and/or the response.
> +	 * Close the socket and clean up.
> +	 * Exit child-process with non-zero status.
> +	 */
> +	WR_IO_ERROR = 1<<0,
> +	/*
> +	 * Close the socket and clean up.  Does not imply an error.
> +	 */
> +	WR_HANGUP   = 1<<1,

nit: add a whitespace line between an item and the next
item's comment.

> +
> +	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
> +};

(I read, but have no comments on the http-server boilerplate.)

> +
> +enum auth_result {
> +	AUTH_UNKNOWN = 0,
> +	AUTH_DENY = 1,
> +	AUTH_ALLOW = 2,
> +};
> +
> +struct auth_module {
> +	const char *scheme;
> +	const char *challenge_params;

Later, I notice that you set challenge_params using an
xstrdup() so this shouldn't be const and you should
free it in any freeing code.

> +	struct string_list *tokens;
> +};
> +
> +static int allow_anonymous;
> +static struct auth_module **auth_modules = NULL;
> +static size_t auth_modules_nr = 0;
> +static size_t auth_modules_alloc = 0;

So, we are setting up a number of potential auth modules,
each of which has a scheme to match a request to the module,
and a list of tokens that would be considered worthy of the
AUTH_ALLOW result. Otherwise, if the scheme matches but no
token matches, we get AUTH_DENY. Finally, if no scheme matches
we get AUTH_UNKNOWN.

This concept might be worth a comment here around the data
structures before we get into how that is implemented.

> +static struct auth_module *get_auth_module(struct strbuf *scheme)
> +{
> +	int i;
> +	struct auth_module *mod;
> +	for (i = 0; i < auth_modules_nr; i++) {
> +		mod = auth_modules[i];
> +		if (!strcasecmp(mod->scheme, scheme->buf))
> +			return mod;
> +	}
> +
> +	return NULL;
> +}

Matching the input scheme against the list of modules.

Only complaint: there is no reason that 'scheme' needs t
be a strbuf, but could be a 'const char *' here.

> +static void add_auth_module(struct auth_module *mod)
> +{
> +	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
> +	auth_modules[auth_modules_nr++] = mod;
> +}

nit: this could be located earlier, next to the list
definition, or delayed until it is needed. That would
allow get_auth_module() to be closer to its first use.

> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
> +{
> +	enum auth_result result = AUTH_UNKNOWN;
> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
> +	struct auth_module *mod;
> +
> +	struct string_list_item *hdr;
> +	struct string_list_item *token;
> +	const char *v;
> +	struct strbuf **split = NULL;
> +	int i;
> +	char *challenge;
> +
> +	/* ask all auth modules to validate the request */
> +	for_each_string_list_item(hdr, &req->header_list) {
> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
> +			split = strbuf_split_str(v, ' ', 2);
> +			if (!split[0] || !split[1]) continue;

For each valid request header...

> +			// trim trailing space ' '
> +			strbuf_setlen(split[0], split[0]->len - 1);
> +
> +			mod = get_auth_module(split[0]);
> +			if (mod) {

...get an appropriate module, if it exists...

> +
> +				for_each_string_list_item(token, mod->tokens) {
> +					if (!strcmp(split[1]->buf, token->string)) {
> +						result = AUTH_ALLOW;
> +						goto done;
> +					}
> +				}
> +
> +				if (result != AUTH_UNKNOWN)
> +					goto done;

...and report if we find a valid token.

Here, it seems I was wrong in my expectation of AUTH_DENY:
if a matching module exists but no token exists in that
module, then we keep searching other modules. 

> +			}
> +		}
> +	}
> +
> +done:
> +	switch (result) {
> +	case AUTH_ALLOW:
> +		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
> +		*user = "VALID_TEST_USER";
> +		*wr = WR_OK;
> +		break;
> +
> +	case AUTH_DENY:
> +		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
> +		/* fall-through */

I'm not sure that I see a case where this is possible. Maybe
we should have a 'result = AUTH_DENY' at the start of the
"if (mod)" block, followed by a 'goto done' in all cases
instead of "if (result != AUTH_UNKNOWN)"?

> +	case AUTH_UNKNOWN:
> +		if (allow_anonymous)
> +			break;

If we do not require auth, then we want to continue if there
is no matching authentication.

> +		for (i = 0; i < auth_modules_nr; i++) {
> +			mod = auth_modules[i];
> +			if (mod->challenge_params)
> +				challenge = xstrfmt("WWW-Authenticate: %s %s",
> +						    mod->scheme,
> +						    mod->challenge_params);
> +			else
> +				challenge = xstrfmt("WWW-Authenticate: %s",
> +						    mod->scheme);
> +			string_list_append(&hdrs, challenge);
> +		}
> +		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);

However, here is the critical piece about how servers will
start to act with the new WWW-Authenticate header usage in
the Git credential helper interface. This will be critical
in the testing for Git to retry the credential helper while
passing these authentications schemes from the installed
modules.

> +	}
> +
> +	strbuf_list_free(split);
> +	string_list_clear(&hdrs, 0);
> +
> +	return result == AUTH_ALLOW ||
> +	      (result == AUTH_UNKNOWN && allow_anonymous);

Did it work? Or did it not need to work? I'm interested to
investigate the case that the client sent an authentication
header that matches a module but doesn't match any tokens,
but we allow anonymous access, anyway. Is that a 400? Or
is that a 401?

> +static enum worker_result dispatch(struct req *req)
> +{
> +	enum worker_result wr = WR_OK;
> +	const char *user = NULL;
> +
> +	if (!is_authed(req, &user, &wr))
> +		return wr;

If we are not authed, send the 401 response.

> +	if (is_git_request(req))
> +		return do__git(req, user);

If we are authed, then pass through to the Git response.

> +	return send_http_error(1, 501, "Not Implemented", -1, NULL,
> +			       WR_OK | WR_HANGUP);

If the Git request fails, we don't care. This is a test.
Just pass a 500-level error and the client will barf,
letting us know that something went wrong.

> +static void kill_some_child(void)

> +static void check_dead_children(void)

These technically sound methods have unfortunate names.
Using something like "connection" over "child" might
alleviate some of the horror. (I initially wanted to
suggest "subprocess" but you compare live_children to
max_connections in the next method, so connection seemed
appropriate.)

> +static struct strvec cld_argv = STRVEC_INIT;
> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
> +{
> +	struct child_process cld = CHILD_PROCESS_INIT;
> +
> +	if (max_connections && live_children >= max_connections) {
> +		kill_some_child();
> +		sleep(1);  /* give it some time to die */
> +		check_dead_children();
> +		if (live_children >= max_connections) {
> +			close(incoming);
> +			logerror("Too many children, dropping connection");
> +			return;
> +		}
> +	}

Do we anticipate exercising concurrent requests in our
tests? Perhaps it's not worth putting a cap on the
connection count so we can keep the test helpers simple.

> +	if (addr->sa_family == AF_INET) {
> +		char buf[128] = "";
> +		struct sockaddr_in *sin_addr = (void *) addr;
> +		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
> +		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
> +		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
> +				 ntohs(sin_addr->sin_port));
> +#ifndef NO_IPV6
> +	} else if (addr->sa_family == AF_INET6) {
> +		char buf[128] = "";
> +		struct sockaddr_in6 *sin6_addr = (void *) addr;
> +		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
> +		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
> +		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
> +				 ntohs(sin6_addr->sin6_port));
> +#endif
> +	}
> +
> +	strvec_pushv(&cld.args, cld_argv.v);
> +	cld.in = incoming;
> +	cld.out = dup(incoming);
> +
> +	if (cld.out < 0)
> +		logerror("could not dup() `incoming`");
> +	else if (start_command(&cld))
> +		logerror("unable to fork");
> +	else
> +		add_child(&cld, addr, addrlen);
> +}
> +

I scanned the socket creation code, but my eyes were
glazing over. I'm definitely in the camp of "if it works,
that's enough for our tests." If we start to rely on this
test harness in more places, we can improve any shortcomings
as they arise.

> +//////////////////////////////////////////////////////////////////
> +// This section is executed by both the primary instance and all
> +// worker instances.  So, yes, each child-process re-parses the
> +// command line argument and re-discovers how it should behave.
> +//////////////////////////////////////////////////////////////////
> +
> +int cmd_main(int argc, const char **argv)
> +{
> +	int listen_port = 0;
> +	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
> +	int worker_mode = 0;
> +	int i;
> +	struct auth_module *mod = NULL;
> +
> +	trace2_cmd_name("test-http-server");
> +	setup_git_directory_gently(NULL);
> +
> +	for (i = 1; i < argc; i++) {
> +		const char *arg = argv[i];
> +		const char *v;
> +
> +		if (skip_prefix(arg, "--listen=", &v)) {
> +			string_list_append(&listen_addr, xstrdup_tolower(v));
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--port=", &v)) {
> +			char *end;
> +			unsigned long n;
> +			n = strtoul(v, &end, 0);
> +			if (*v && !*end) {
> +				listen_port = n;
> +				continue;
> +			}
> +		}
> +		if (!strcmp(arg, "--worker")) {
> +			worker_mode = 1;
> +			trace2_cmd_mode("worker");
> +			continue;
> +		}
> +		if (!strcmp(arg, "--verbose")) {
> +			verbose = 1;
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--timeout=", &v)) {
> +			timeout = atoi(v);
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--init-timeout=", &v)) {
> +			init_timeout = atoi(v);
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--max-connections=", &v)) {
> +			max_connections = atoi(v);
> +			if (max_connections < 0)
> +				max_connections = 0; /* unlimited */
> +			continue;
> +		}
> +		if (!strcmp(arg, "--reuseaddr")) {
> +			reuseaddr = 1;
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--pid-file=", &v)) {
> +			pid_file = v;
> +			continue;
> +		}

ok, most of these arguments are actually about the per-connection
subprocesses.

> +		if (skip_prefix(arg, "--allow-anonymous", &v)) {
> +			allow_anonymous = 1;
> +			continue;
> +		}

Here is how we choose to allo anonymous access.

> +		if (skip_prefix(arg, "--auth=", &v)) {
> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
> +
> +			if (!p[0]) {
> +				error("invalid argument '%s'", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			// trim trailing ':'
> +			if (p[1])
> +				strbuf_setlen(p[0], p[0]->len - 1);
> +
> +			if (get_auth_module(p[0])) {
> +				error("duplicate auth scheme '%s'\n", p[0]->buf);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			mod = xmalloc(sizeof(struct auth_module));
> +			mod->scheme = xstrdup(p[0]->buf);
> +			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;

Here, you xstrdup() into a 'const char *', but you are really
passing ownership so it shouldn't be conts.

> +			mod->tokens = xmalloc(sizeof(struct string_list));

nit: this could also be "CALLOC_ARRAY(mod->tokens, 1);"

> +			string_list_init_dup(mod->tokens);
> +
> +			add_auth_module(mod);
> +
> +			strbuf_list_free(p);
> +			continue;

Ok, we gain the auth schemes from the command line.

> +		}
> +		if (skip_prefix(arg, "--auth-token=", &v)) {
> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
> +			if (!p[0]) {
> +				error("invalid argument '%s'", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			if (!p[1]) {
> +				error("missing token value '%s'\n", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			// trim trailing ':'

Use /* */ (Aside: I'm surprised we don't have a build option in
DEVELOPER=1 that catches the use of these comments.)

> +			strbuf_setlen(p[0], p[0]->len - 1);
> +
> +			mod = get_auth_module(p[0]);
> +			if (!mod) {
> +				error("auth scheme not defined '%s'\n", p[0]->buf);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			string_list_append(mod->tokens, p[1]->buf);
> +			strbuf_list_free(p);
> +			continue;
> +		}

And the token lists. It is important that the scheme is added
before any token is added.

> +		fprintf(stderr, "error: unknown argument '%s'\n", arg);
> +		usage(test_http_auth_usage);
> +	}
> +
> +	/* avoid splitting a message in the middle */
> +	setvbuf(stderr, NULL, _IOFBF, 4096);
> +
> +	if (listen_port == 0)
> +		listen_port = DEFAULT_GIT_PORT;
> +
> +	/*
> +	 * If no --listen=<addr> args are given, the setup_named_sock()
> +	 * code will use receive a NULL address and set INADDR_ANY.
> +	 * This exposes both internal and external interfaces on the
> +	 * port.
> +	 *
> +	 * Disallow that and default to the internal-use-only loopback
> +	 * address.
> +	 */
> +	if (!listen_addr.nr)
> +		string_list_append(&listen_addr, "127.0.0.1");
> +
> +	/*
> +	 * worker_mode is set in our own child process instances
> +	 * (that are bound to a connected socket from a client).
> +	 */
> +	if (worker_mode)
> +		return worker();
> +
> +	/*
> +	 * `cld_argv` is a bit of a clever hack. The top-level instance
> +	 * of test-http-server does the normal bind/listen/accept stuff.
> +	 * For each incoming socket, the top-level process spawns
> +	 * a child instance of test-http-server *WITH* the additional
> +	 * `--worker` argument. This causes the child to set `worker_mode`
> +	 * and immediately call `worker()` using the connected socket (and
> +	 * without the usual need for fork() or threads).
> +	 *
> +	 * The magic here is made possible because `cld_argv` is static
> +	 * and handle() (called by service_loop()) knows about it.
> +	 */
> +	strvec_push(&cld_argv, argv[0]);
> +	strvec_push(&cld_argv, "--worker");
> +	for (i = 1; i < argc; ++i)
> +		strvec_push(&cld_argv, argv[i]);
> +
> +	/*
> +	 * Setup primary instance to listen for connections.
> +	 */
> +	return serve(&listen_addr, listen_port);
> +}

And complete the thing with some boilerplate.

This was a lot to read, and the interesting bits are all mixed in
with the http server code, which is less interesting to what we
are trying to accomplish. It would be beneficial to split this
into one or two patches before we actually introduce the tests.

The most important thing that I think would be helpful is to
isolate all the authentication behavior into its own patch so
we can see how those connections from the command-line arguments
affect the behavior of the server responses.

I think ideally we would have the following split:

 1. All server boilerblate. All requests 500 not-implemented.

 2. Add Git fall-through with no authentication. Add the tests
    that are intended to allow anonymous auth.

 3. Add authentication data structures read from command-line,
    but not processed at all in the logic.

 4. Act on the authentication data structures to alter the
    requests. Add the tests that use these authentication
    schemes.

I could easily see a case for combining 1&2 as well as 3&4,
for slightly larger but more completely-testable changes at
every step.

From what I read, I don't think there is much to change in
the end result of the code, but it definitely was hard to read
the important things when surrounded by many lines of
boilerplate.

> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh

I'm going to pause here and come back to the test script in
a separate reply.

Thanks,
-Stolee

Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Jeff Hostetler]

On 10/28/22 11:08 AM, Derrick Stolee wrote:
> }
> 
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> 
>> @@ -0,0 +1,1134 @@
>> +#include "config.h"
>> +#include "run-command.h"
>> +#include "strbuf.h"
>> +#include "string-list.h"
>> +#include "trace2.h"
>> +#include "version.h"
>> +#include "dir.h"
>> +#include "date.h"
>> +
>> +#define TR2_CAT "test-http-server"
>> +
>> +static const char *pid_file;
>> +static int verbose;
>> +static int reuseaddr;
>> +
>> +static const char test_http_auth_usage[] =
>> +"http-server [--verbose]\n"
>> +"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
>> +"           [--reuseaddr] [--pid-file=<file>]\n"
>> +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
>> +"           [--anonymous-allowed]\n"
>> +"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
>> +;
> 
> These are a lot of options to implement all at once. They are probably
> simple enough, but depending on the implementation and tests, it might
> be helpful to split this patch into smaller ones that introduce these
> options along with the tests that exercise each. That will help
> verify that they are being tested properly instead of needing to track
> back and forth across the patch for each one.

how many of these options were inherited from test-gvfs-protocol or
from upstream git-daemon?  If most came from git-daemon, it's probably
easier to see that this was a cut-n-paste from it if it comes over in
one commit, since all of the OPT_ processing, usage(), and static global
state vars will come over together I would think -- rather than to build
up the arg parsing bit by bit.  More on this in a minute...


>> +
>> +/* Timeout, and initial timeout */
>> +static unsigned int timeout;
>> +static unsigned int init_timeout;
>> +
>> +static void logreport(const char *label, const char *err, va_list params)
>> +{
>> +	struct strbuf msg = STRBUF_INIT;
>> +
>> +	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
>> +	strbuf_vaddf(&msg, err, params);
>> +	strbuf_addch(&msg, '\n');
>> +
>> +	fwrite(msg.buf, sizeof(char), msg.len, stderr);
>> +	fflush(stderr);
>> +
>> +	strbuf_release(&msg);
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void logerror(const char *err, ...)
>> +{
>> +	va_list params;
>> +	va_start(params, err);
>> +	logreport("error", err, params);
>> +	va_end(params);
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void loginfo(const char *err, ...)
>> +{
>> +	va_list params;
>> +	if (!verbose)
>> +		return;
>> +	va_start(params, err);
>> +	logreport("info", err, params);
>> +	va_end(params);
>> +}

...Maybe it would be easier to see/diff this large new test server
if we copied `daemon.c` into this source file in 1 commit and then
converted it to what you have now in 1 commit -- so that only new
code shows up here.  For example, all of the above logreport, logerror,
and loginfo routines would show up as new in the copy commit, but not
in the edit commit.  However, that may lead to too much noise when
you actually get into the meat of the auth changes, maybe.


> I wonder how much of this we need or is just a nice thing. I would
> err on the side of making things as simple as possible, but being
> able to debug this test server may be important based on your
> experience.

i'd vote to keep it.

[...]
>> +static void kill_some_child(void)
> 
>> +static void check_dead_children(void)
> 
> These technically sound methods have unfortunate names.
> Using something like "connection" over "child" might
> alleviate some of the horror. (I initially wanted to
> suggest "subprocess" but you compare live_children to
> max_connections in the next method, so connection seemed
> appropriate.)

These names were inherited from `daemon.c` IIRC. I wouldn't change
them since it'll just introduce noise when diffing.  Especially,
if we do the copy commit first.


[...]
>> +static struct strvec cld_argv = STRVEC_INIT;
>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>> +{
>> +	struct child_process cld = CHILD_PROCESS_INIT;
>> +
>> +	if (max_connections && live_children >= max_connections) {
>> +		kill_some_child();
>> +		sleep(1);  /* give it some time to die */
>> +		check_dead_children();
>> +		if (live_children >= max_connections) {
>> +			close(incoming);
>> +			logerror("Too many children, dropping connection");
>> +			return;
>> +		}
>> +	}
> 
> Do we anticipate exercising concurrent requests in our
> tests? Perhaps it's not worth putting a cap on the
> connection count so we can keep the test helpers simple.

again, this code was inherited from `daemon.c`, so we could leave it.

[...]
>> +			mod = xmalloc(sizeof(struct auth_module));
>> +			mod->scheme = xstrdup(p[0]->buf);
>> +			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
> 
> Here, you xstrdup() into a 'const char *', but you are really
> passing ownership so it shouldn't be conts.

There is a strbuf_detach() that will let you steal the buffer from the
strbuf if that would help.


[...]
> This was a lot to read, and the interesting bits are all mixed in
> with the http server code, which is less interesting to what we
> are trying to accomplish. It would be beneficial to split this
> into one or two patches before we actually introduce the tests.

agreed. it is big, but it does make sense.  perhaps doing the
copy daemon.c commit and then see how this commit diffs from it
would make it more manageable. (not sure, but worth a try.)

[...]
>  From what I read, I don't think there is much to change in
> the end result of the code, but it definitely was hard to read
> the important things when surrounded by many lines of
> boilerplate.

agreed. i think the end result is good.

Thanks
Jeff

Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Matthew John Cheetham]

On 2022-10-28 12:14, Jeff Hostetler wrote:
>
>
> On 10/28/22 11:08 AM, Derrick Stolee wrote:
>> }
>>
>>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>>
>>> @@ -0,0 +1,1134 @@
>>> +#include "config.h"
>>> +#include "run-command.h"
>>> +#include "strbuf.h"
>>> +#include "string-list.h"
>>> +#include "trace2.h"
>>> +#include "version.h"
>>> +#include "dir.h"
>>> +#include "date.h"
>>> +
>>> +#define TR2_CAT "test-http-server"
>>> +
>>> +static const char *pid_file;
>>> +static int verbose;
>>> +static int reuseaddr;
>>> +
>>> +static const char test_http_auth_usage[] =
>>> +"http-server [--verbose]\n"
>>> +"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
>>> +"           [--reuseaddr] [--pid-file=<file>]\n"
>>> +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
>>> +"           [--anonymous-allowed]\n"
>>> +"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
>>> +;
>>
>> These are a lot of options to implement all at once. They are probably
>> simple enough, but depending on the implementation and tests, it might
>> be helpful to split this patch into smaller ones that introduce these
>> options along with the tests that exercise each. That will help
>> verify that they are being tested properly instead of needing to track
>> back and forth across the patch for each one.
>
> how many of these options were inherited from test-gvfs-protocol or
> from upstream git-daemon?  If most came from git-daemon, it's probably
> easier to see that this was a cut-n-paste from it if it comes over in
> one commit, since all of the OPT_ processing, usage(), and static global
> state vars will come over together I would think -- rather than to build
> up the arg parsing bit by bit.  More on this in a minute...
>

Only --anonymous-allowed, --auth and --auth-token are added over git-daemon.

>
>>> +
>>> +/* Timeout, and initial timeout */
>>> +static unsigned int timeout;
>>> +static unsigned int init_timeout;
>>> +
>>> +static void logreport(const char *label, const char *err, va_list params)
>>> +{
>>> +    struct strbuf msg = STRBUF_INIT;
>>> +
>>> +    strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
>>> +    strbuf_vaddf(&msg, err, params);
>>> +    strbuf_addch(&msg, '\n');
>>> +
>>> +    fwrite(msg.buf, sizeof(char), msg.len, stderr);
>>> +    fflush(stderr);
>>> +
>>> +    strbuf_release(&msg);
>>> +}
>>> +
>>> +__attribute__((format (printf, 1, 2)))
>>> +static void logerror(const char *err, ...)
>>> +{
>>> +    va_list params;
>>> +    va_start(params, err);
>>> +    logreport("error", err, params);
>>> +    va_end(params);
>>> +}
>>> +
>>> +__attribute__((format (printf, 1, 2)))
>>> +static void loginfo(const char *err, ...)
>>> +{
>>> +    va_list params;
>>> +    if (!verbose)
>>> +        return;
>>> +    va_start(params, err);
>>> +    logreport("info", err, params);
>>> +    va_end(params);
>>> +}
>
> ...Maybe it would be easier to see/diff this large new test server
> if we copied `daemon.c` into this source file in 1 commit and then
> converted it to what you have now in 1 commit -- so that only new
> code shows up here.  For example, all of the above logreport, logerror,
> and loginfo routines would show up as new in the copy commit, but not
> in the edit commit.  However, that may lead to too much noise when
> you actually get into the meat of the auth changes, maybe.

I take from git-daemon and the test-gvfs-protocol helper from microsoft/git
fork, but then also delete lots of not required pieces too just as much as
I have added. Copying git-daemon.c, to then delete, and then add feels like
lots of noise.

>> I wonder how much of this we need or is just a nice thing. I would
>> err on the side of making things as simple as possible, but being
>> able to debug this test server may be important based on your
>> experience.
>
> i'd vote to keep it.
>
> [...]
>>> +static void kill_some_child(void)
>>
>>> +static void check_dead_children(void)
>>
>> These technically sound methods have unfortunate names.
>> Using something like "connection" over "child" might
>> alleviate some of the horror. (I initially wanted to
>> suggest "subprocess" but you compare live_children to
>> max_connections in the next method, so connection seemed
>> appropriate.)
>
> These names were inherited from `daemon.c` IIRC. I wouldn't change
> them since it'll just introduce noise when diffing.  Especially,
> if we do the copy commit first.

Indeed. These functions are untouched from daemon.c. I do plan to split
this mega-patch up however in to a single 'add the boilerplate' based on
git-daemon patch, then add the extra pieces like HTTP request parsing and
the auth pieces in a v3.

> [...]
>>> +static struct strvec cld_argv = STRVEC_INIT;
>>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>>> +{
>>> +    struct child_process cld = CHILD_PROCESS_INIT;
>>> +
>>> +    if (max_connections && live_children >= max_connections) {
>>> +        kill_some_child();
>>> +        sleep(1);  /* give it some time to die */
>>> +        check_dead_children();
>>> +        if (live_children >= max_connections) {
>>> +            close(incoming);
>>> +            logerror("Too many children, dropping connection");
>>> +            return;
>>> +        }
>>> +    }
>>
>> Do we anticipate exercising concurrent requests in our
>> tests? Perhaps it's not worth putting a cap on the
>> connection count so we can keep the test helpers simple.
>
> again, this code was inherited from `daemon.c`, so we could leave it.
>
> [...]
>>> +            mod = xmalloc(sizeof(struct auth_module));
>>> +            mod->scheme = xstrdup(p[0]->buf);
>>> +            mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
>>
>> Here, you xstrdup() into a 'const char *', but you are really
>> passing ownership so it shouldn't be conts.
>
> There is a strbuf_detach() that will let you steal the buffer from the
> strbuf if that would help.

Will update in v3 to drop the const.

> [...]
>> This was a lot to read, and the interesting bits are all mixed in
>> with the http server code, which is less interesting to what we
>> are trying to accomplish. It would be beneficial to split this
>> into one or two patches before we actually introduce the tests.
>
> agreed. it is big, but it does make sense.  perhaps doing the
> copy daemon.c commit and then see how this commit diffs from it
> would make it more manageable. (not sure, but worth a try.)
>
> [...]
>>  From what I read, I don't think there is much to change in
>> the end result of the code, but it definitely was hard to read
>> the important things when surrounded by many lines of
>> boilerplate.
>
> agreed. i think the end result is good.
>
> Thanks
> Jeff
>
>

Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Derrick Stolee]

On 11/1/22 7:14 PM, Matthew John Cheetham wrote:
> On 2022-10-28 12:14, Jeff Hostetler wrote:
>> On 10/28/22 11:08 AM, Derrick Stolee wrote:

>>>> +static void kill_some_child(void)
>>>
>>>> +static void check_dead_children(void)
>>>
>>> These technically sound methods have unfortunate names.
>>> Using something like "connection" over "child" might
>>> alleviate some of the horror. (I initially wanted to
>>> suggest "subprocess" but you compare live_children to
>>> max_connections in the next method, so connection seemed
>>> appropriate.)
>>
>> These names were inherited from `daemon.c` IIRC. I wouldn't change
>> them since it'll just introduce noise when diffing.  Especially,
>> if we do the copy commit first.
> 
> Indeed. These functions are untouched from daemon.c. I do plan to split
> this mega-patch up however in to a single 'add the boilerplate' based on
> git-daemon patch, then add the extra pieces like HTTP request parsing and
> the auth pieces in a v3.

If these are copied from daemon.c, it may be worth trying
to lib-ify these data structures and code so they can be
shared across the two places. That can also come up as a
cleanup later, too.

For now, don't bother changing the names since they exist
somewhere else.
 
>> [...]
>>>> +static struct strvec cld_argv = STRVEC_INIT;
>>>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>>>> +{
>>>> +    struct child_process cld = CHILD_PROCESS_INIT;
>>>> +
>>>> +    if (max_connections && live_children >= max_connections) {
>>>> +        kill_some_child();
>>>> +        sleep(1);  /* give it some time to die */
>>>> +        check_dead_children();
>>>> +        if (live_children >= max_connections) {
>>>> +            close(incoming);
>>>> +            logerror("Too many children, dropping connection");
>>>> +            return;
>>>> +        }
>>>> +    }
>>>
>>> Do we anticipate exercising concurrent requests in our
>>> tests? Perhaps it's not worth putting a cap on the
>>> connection count so we can keep the test helpers simple.
>>
>> again, this code was inherited from `daemon.c`, so we could leave it.

I wonder how much could be extracted from daemon.c using a
copy into a 'daemon-lib.c' with methods defined in 'daemon-lib.h'
then consumed from this file instead. Not sure it's worth the
churn to daemon.c, though.

Thanks,
-Stolee

Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic

[Matthew John Cheetham]

On 2022-10-28 08:08, Derrick Stolee wrote:
> On 10/21/22 1:08 PM, Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
>> @@ -1500,6 +1500,8 @@ else
>>  	endif
>>  	BASIC_CFLAGS += $(CURL_CFLAGS)
>>  
>> +	TEST_PROGRAMS_NEED_X += test-http-server
>> +
>>  	REMOTE_CURL_PRIMARY = git-remote-http$X
>>  	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
>>  	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
> 
> This hunk is in the "else" block of "ifdef NO_CURL",
> so this makes sense for why TEST_PROGRAMS_NEED_X is
> augmented here, away from other instances.
> 
>> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
>> index 787738e6fa3..45251695ce0 100644
>> --- a/contrib/buildsystems/CMakeLists.txt
>> +++ b/contrib/buildsystems/CMakeLists.txt
>> @@ -989,6 +989,19 @@ set(wrapper_scripts
>>  set(wrapper_test_scripts
>>  	test-fake-ssh test-tool)
>>  
>> +if(CURL_FOUND)
>> +       list(APPEND wrapper_test_scripts test-http-server)
>> +
>> +       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
>> +       target_link_libraries(test-http-server common-main)
>> +
>> +       if(MSVC)
>> +               set_target_properties(test-http-server
>> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
>> +               set_target_properties(test-http-server
>> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
>> +       endif()
>> +endif()
> 
> And this file has the pattern of many "if(CURL_FOUND)"
> blocks with isolated purposes, so it makes sense to
> have this be an isolated change instead of grouped with
> a different case.
> 
>> diff --git a/t/helper/.gitignore b/t/helper/.gitignore
>> index 8c2ddcce95f..1a94ab6eed5 100644
>> --- a/t/helper/.gitignore
>> +++ b/t/helper/.gitignore
>> @@ -1,2 +1,3 @@
>>  /test-tool
>>  /test-fake-ssh
>> +test-http-server
> 
> Should this start with a "/" like the other entries?

That it probably should! Will update.

>> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
>> new file mode 100755
>> index 00000000000..03e5e63dad6
>> --- /dev/null
>> +++ b/t/helper/test-credential-helper-replay.sh
>> @@ -0,0 +1,14 @@
>> +cmd=$1
>> +teefile=$cmd-actual.cred
>> +catfile=$cmd-response.cred
>> +rm -f $teefile
>> +while read line;
>> +do
>> +	if test -z "$line"; then
>> +		break;
>> +	fi
>> +	echo "$line" >> $teefile
>> +done
>> +if test "$cmd" = "get"; then
>> +	cat $catfile
>> +fi
> 
> Should this be a helper method within another script, such
> as t/lib-credential.sh or t/lib-httpd.sh? The read over
> stdin will still work, as in this example:
> 
> read_chunk() {
> 	while read line; do
> 		case "$line" in
> 		--) break ;;
> 		*) echo "$line" ;;
> 		esac
> 	done
> }

This script file is used as a credential helper that is invoked by Git.
We specify that Git should use this credential helper in the tests using
the -c option:

  CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
	  && export CREDENTIAL_HELPER
..
   git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&


Would extracting a read_chunk() function to one of the lib-* test scripts
be worth it given we already need another entry script anyway?

What other scripts would be calling read_chunk()?


>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> 
>> @@ -0,0 +1,1134 @@
>> +#include "config.h"
>> +#include "run-command.h"
>> +#include "strbuf.h"
>> +#include "string-list.h"
>> +#include "trace2.h"
>> +#include "version.h"
>> +#include "dir.h"
>> +#include "date.h"
>> +
>> +#define TR2_CAT "test-http-server"
>> +
>> +static const char *pid_file;
>> +static int verbose;
>> +static int reuseaddr;
>> +
>> +static const char test_http_auth_usage[] =
>> +"http-server [--verbose]\n"
>> +"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
>> +"           [--reuseaddr] [--pid-file=<file>]\n"
>> +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
>> +"           [--anonymous-allowed]\n"
>> +"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
>> +;
> 
> These are a lot of options to implement all at once. They are probably
> simple enough, but depending on the implementation and tests, it might
> be helpful to split this patch into smaller ones that introduce these
> options along with the tests that exercise each. That will help
> verify that they are being tested properly instead of needing to track
> back and forth across the patch for each one.

I plan to split this patch in to several in a v3.

>> +
>> +/* Timeout, and initial timeout */
>> +static unsigned int timeout;
>> +static unsigned int init_timeout;
>> +
>> +static void logreport(const char *label, const char *err, va_list params)
>> +{
>> +	struct strbuf msg = STRBUF_INIT;
>> +
>> +	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
>> +	strbuf_vaddf(&msg, err, params);
>> +	strbuf_addch(&msg, '\n');
>> +
>> +	fwrite(msg.buf, sizeof(char), msg.len, stderr);
>> +	fflush(stderr);
>> +
>> +	strbuf_release(&msg);
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void logerror(const char *err, ...)
>> +{
>> +	va_list params;
>> +	va_start(params, err);
>> +	logreport("error", err, params);
>> +	va_end(params);
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void loginfo(const char *err, ...)
>> +{
>> +	va_list params;
>> +	if (!verbose)
>> +		return;
>> +	va_start(params, err);
>> +	logreport("info", err, params);
>> +	va_end(params);
>> +}
> 
> I wonder how much of this we need or is just a nice thing. I would
> err on the side of making things as simple as possible, but being
> able to debug this test server may be important based on your
> experience.

These are useful to debug failures. Plus they also come from my copy
from daemon.c, so didn't want to touch/delete too much from that
starting point.

>> +static void set_keep_alive(int sockfd)
>> +{
>> +	int ka = 1;
>> +
>> +	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
>> +		if (errno != ENOTSOCK)
>> +			logerror("unable to set SO_KEEPALIVE on socket: %s",
>> +				strerror(errno));
>> +	}
>> +}
>> +
>> +//////////////////////////////////////////////////////////////////
>> +// The code in this section is used by "worker" instances to service
>> +// a single connection from a client.  The worker talks to the client
>> +// on 0 and 1.
>> +//////////////////////////////////////////////////////////////////
> 
> Use /* */ style comments. You can repeat the asterisks to get a
> similar visual block.

Yep!

>> +
>> +enum worker_result {
>> +	/*
>> +	 * Operation successful.
>> +	 * Caller *might* keep the socket open and allow keep-alive.
>> +	 */
>> +	WR_OK       = 0,
>> +	/*
>> +	 * Various errors while processing the request and/or the response.
>> +	 * Close the socket and clean up.
>> +	 * Exit child-process with non-zero status.
>> +	 */
>> +	WR_IO_ERROR = 1<<0,
>> +	/*
>> +	 * Close the socket and clean up.  Does not imply an error.
>> +	 */
>> +	WR_HANGUP   = 1<<1,
> 
> nit: add a whitespace line between an item and the next
> item's comment.

Sure

>> +
>> +	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
>> +};
> 
> (I read, but have no comments on the http-server boilerplate.)
> 
>> +
>> +enum auth_result {
>> +	AUTH_UNKNOWN = 0,
>> +	AUTH_DENY = 1,
>> +	AUTH_ALLOW = 2,
>> +};
>> +
>> +struct auth_module {
>> +	const char *scheme;
>> +	const char *challenge_params;
> 
> Later, I notice that you set challenge_params using an
> xstrdup() so this shouldn't be const and you should
> free it in any freeing code.

One question on this suggestion.. where would be appropriate to
free said char*? We need them for the lifetime of the process,
and they never grown in number beyond initial allocation from
parsing command line args.

I could move to stack alloc these in `cmd_main` and instead pass
a pointer to the `auth_modules` and count down through every
serve/handle etc function, rather than rely on them being global?

Thoughts or preferences?

>> +	struct string_list *tokens;
>> +};
>> +
>> +static int allow_anonymous;
>> +static struct auth_module **auth_modules = NULL;
>> +static size_t auth_modules_nr = 0;
>> +static size_t auth_modules_alloc = 0;
> 
> So, we are setting up a number of potential auth modules,
> each of which has a scheme to match a request to the module,
> and a list of tokens that would be considered worthy of the
> AUTH_ALLOW result. Otherwise, if the scheme matches but no
> token matches, we get AUTH_DENY. Finally, if no scheme matches
> we get AUTH_UNKNOWN.
> 
> This concept might be worth a comment here around the data
> structures before we get into how that is implemented.
> 
>> +static struct auth_module *get_auth_module(struct strbuf *scheme)
>> +{
>> +	int i;
>> +	struct auth_module *mod;
>> +	for (i = 0; i < auth_modules_nr; i++) {
>> +		mod = auth_modules[i];
>> +		if (!strcasecmp(mod->scheme, scheme->buf))
>> +			return mod;
>> +	}
>> +
>> +	return NULL;
>> +}
> 
> Matching the input scheme against the list of modules.
> 
> Only complaint: there is no reason that 'scheme' needs t
> be a strbuf, but could be a 'const char *' here.

True.

>> +static void add_auth_module(struct auth_module *mod)
>> +{
>> +	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
>> +	auth_modules[auth_modules_nr++] = mod;
>> +}
> 
> nit: this could be located earlier, next to the list
> definition, or delayed until it is needed. That would
> allow get_auth_module() to be closer to its first use.

Not sure I follow.. are you saying I should move `add_auth_module`
to earlier in the file?

>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
>> +{
>> +	enum auth_result result = AUTH_UNKNOWN;
>> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
>> +	struct auth_module *mod;
>> +
>> +	struct string_list_item *hdr;
>> +	struct string_list_item *token;
>> +	const char *v;
>> +	struct strbuf **split = NULL;
>> +	int i;
>> +	char *challenge;
>> +
>> +	/* ask all auth modules to validate the request */
>> +	for_each_string_list_item(hdr, &req->header_list) {
>> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
>> +			split = strbuf_split_str(v, ' ', 2);
>> +			if (!split[0] || !split[1]) continue;
> 
> For each valid request header...
> 
>> +			// trim trailing space ' '
>> +			strbuf_setlen(split[0], split[0]->len - 1);
>> +
>> +			mod = get_auth_module(split[0]);
>> +			if (mod) {
> 
> ...get an appropriate module, if it exists...
> 
>> +
>> +				for_each_string_list_item(token, mod->tokens) {
>> +					if (!strcmp(split[1]->buf, token->string)) {
>> +						result = AUTH_ALLOW;
>> +						goto done;
>> +					}
>> +				}
>> +
>> +				if (result != AUTH_UNKNOWN)
>> +					goto done;
> 
> ...and report if we find a valid token.
> 
> Here, it seems I was wrong in my expectation of AUTH_DENY:
> if a matching module exists but no token exists in that
> module, then we keep searching other modules. 

AUTH_DENY denies a request immediately and stops searching other modules.
AUTH_ALLOW approves the request and stops looking at other modules.
AUTH_UNKNOWN means this module didn't match or 'decide' to reject, so keep
looking/asking other modules.

After reading you review, I think it may be better to change this to
more closely match your expectations (and how typical servers behave):

Return AUTH_ALLOW if we find a matching valid token for the module.
If we match a module and do NOT find a token, then return AUTH_DENY.
Otherwise return AUTH_UNKNOWN - this means the user provided some auth
mechanism we don't understand, or no auth at all.

>> +			}
>> +		}
>> +	}
>> +
>> +done:
>> +	switch (result) {
>> +	case AUTH_ALLOW:
>> +		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
>> +		*user = "VALID_TEST_USER";
>> +		*wr = WR_OK;
>> +		break;
>> +
>> +	case AUTH_DENY:
>> +		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
>> +		/* fall-through */
> 
> I'm not sure that I see a case where this is possible. Maybe
> we should have a 'result = AUTH_DENY' at the start of the
> "if (mod)" block, followed by a 'goto done' in all cases
> instead of "if (result != AUTH_UNKNOWN)"?

In this version, you're correct.. AUTH_DENY is never returned.
This tri-state response from an auth module is an oversight from an earlier
local version - sorry for the confusion here, and thanks for catching!
I will update in a v3 to match sane expectations.

>> +	case AUTH_UNKNOWN:
>> +		if (allow_anonymous)
>> +			break;
> 
> If we do not require auth, then we want to continue if there
> is no matching authentication.
> 
>> +		for (i = 0; i < auth_modules_nr; i++) {
>> +			mod = auth_modules[i];
>> +			if (mod->challenge_params)
>> +				challenge = xstrfmt("WWW-Authenticate: %s %s",
>> +						    mod->scheme,
>> +						    mod->challenge_params);
>> +			else
>> +				challenge = xstrfmt("WWW-Authenticate: %s",
>> +						    mod->scheme);
>> +			string_list_append(&hdrs, challenge);
>> +		}
>> +		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
> 
> However, here is the critical piece about how servers will
> start to act with the new WWW-Authenticate header usage in
> the Git credential helper interface. This will be critical
> in the testing for Git to retry the credential helper while
> passing these authentications schemes from the installed
> modules.
> 
>> +	}
>> +
>> +	strbuf_list_free(split);
>> +	string_list_clear(&hdrs, 0);
>> +
>> +	return result == AUTH_ALLOW ||
>> +	      (result == AUTH_UNKNOWN && allow_anonymous);
> 
> Did it work? Or did it not need to work? I'm interested to
> investigate the case that the client sent an authentication
> header that matches a module but doesn't match any tokens,
> but we allow anonymous access, anyway. Is that a 400? Or
> is that a 401?

It should probably be a 401 as the credentials are understood, but
are just 'bad'.

>> +static enum worker_result dispatch(struct req *req)
>> +{
>> +	enum worker_result wr = WR_OK;
>> +	const char *user = NULL;
>> +
>> +	if (!is_authed(req, &user, &wr))
>> +		return wr;
> 
> If we are not authed, send the 401 response.
> 
>> +	if (is_git_request(req))
>> +		return do__git(req, user);
> 
> If we are authed, then pass through to the Git response.
> 
>> +	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>> +			       WR_OK | WR_HANGUP);
> 
> If the Git request fails, we don't care. This is a test.
> Just pass a 500-level error and the client will barf,
> letting us know that something went wrong.

Correct assessment!

>> +static void kill_some_child(void)
> 
>> +static void check_dead_children(void)
> 
> These technically sound methods have unfortunate names.
> Using something like "connection" over "child" might
> alleviate some of the horror. (I initially wanted to
> suggest "subprocess" but you compare live_children to
> max_connections in the next method, so connection seemed
> appropriate.)

These are copied exactly from git-daemon, so I'd rather
avoid the churn in renaming things.

>> +static struct strvec cld_argv = STRVEC_INIT;
>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>> +{
>> +	struct child_process cld = CHILD_PROCESS_INIT;
>> +
>> +	if (max_connections && live_children >= max_connections) {
>> +		kill_some_child();
>> +		sleep(1);  /* give it some time to die */
>> +		check_dead_children();
>> +		if (live_children >= max_connections) {
>> +			close(incoming);
>> +			logerror("Too many children, dropping connection");
>> +			return;
>> +		}
>> +	}
> 
> Do we anticipate exercising concurrent requests in our
> tests? Perhaps it's not worth putting a cap on the
> connection count so we can keep the test helpers simple.

Probably not, but again.. 100% of the boilerplate here came from
the prior art in daemon.c, so didn't want to touch any of it!
I'm happy to start deleting things however if needed?

>> +	if (addr->sa_family == AF_INET) {
>> +		char buf[128] = "";
>> +		struct sockaddr_in *sin_addr = (void *) addr;
>> +		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
>> +		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
>> +		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
>> +				 ntohs(sin_addr->sin_port));
>> +#ifndef NO_IPV6
>> +	} else if (addr->sa_family == AF_INET6) {
>> +		char buf[128] = "";
>> +		struct sockaddr_in6 *sin6_addr = (void *) addr;
>> +		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
>> +		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
>> +		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
>> +				 ntohs(sin6_addr->sin6_port));
>> +#endif
>> +	}
>> +
>> +	strvec_pushv(&cld.args, cld_argv.v);
>> +	cld.in = incoming;
>> +	cld.out = dup(incoming);
>> +
>> +	if (cld.out < 0)
>> +		logerror("could not dup() `incoming`");
>> +	else if (start_command(&cld))
>> +		logerror("unable to fork");
>> +	else
>> +		add_child(&cld, addr, addrlen);
>> +}
>> +
> 
> I scanned the socket creation code, but my eyes were
> glazing over. I'm definitely in the camp of "if it works,
> that's enough for our tests." If we start to rely on this
> test harness in more places, we can improve any shortcomings
> as they arise.
> 
>> +//////////////////////////////////////////////////////////////////
>> +// This section is executed by both the primary instance and all
>> +// worker instances.  So, yes, each child-process re-parses the
>> +// command line argument and re-discovers how it should behave.
>> +//////////////////////////////////////////////////////////////////
>> +
>> +int cmd_main(int argc, const char **argv)
>> +{
>> +	int listen_port = 0;
>> +	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
>> +	int worker_mode = 0;
>> +	int i;
>> +	struct auth_module *mod = NULL;
>> +
>> +	trace2_cmd_name("test-http-server");
>> +	setup_git_directory_gently(NULL);
>> +
>> +	for (i = 1; i < argc; i++) {
>> +		const char *arg = argv[i];
>> +		const char *v;
>> +
>> +		if (skip_prefix(arg, "--listen=", &v)) {
>> +			string_list_append(&listen_addr, xstrdup_tolower(v));
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--port=", &v)) {
>> +			char *end;
>> +			unsigned long n;
>> +			n = strtoul(v, &end, 0);
>> +			if (*v && !*end) {
>> +				listen_port = n;
>> +				continue;
>> +			}
>> +		}
>> +		if (!strcmp(arg, "--worker")) {
>> +			worker_mode = 1;
>> +			trace2_cmd_mode("worker");
>> +			continue;
>> +		}
>> +		if (!strcmp(arg, "--verbose")) {
>> +			verbose = 1;
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--timeout=", &v)) {
>> +			timeout = atoi(v);
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--init-timeout=", &v)) {
>> +			init_timeout = atoi(v);
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--max-connections=", &v)) {
>> +			max_connections = atoi(v);
>> +			if (max_connections < 0)
>> +				max_connections = 0; /* unlimited */
>> +			continue;
>> +		}
>> +		if (!strcmp(arg, "--reuseaddr")) {
>> +			reuseaddr = 1;
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--pid-file=", &v)) {
>> +			pid_file = v;
>> +			continue;
>> +		}
> 
> ok, most of these arguments are actually about the per-connection
> subprocesses.
> 
>> +		if (skip_prefix(arg, "--allow-anonymous", &v)) {
>> +			allow_anonymous = 1;
>> +			continue;
>> +		}
> 
> Here is how we choose to allo anonymous access.
> 
>> +		if (skip_prefix(arg, "--auth=", &v)) {
>> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
>> +
>> +			if (!p[0]) {
>> +				error("invalid argument '%s'", v);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			// trim trailing ':'
>> +			if (p[1])
>> +				strbuf_setlen(p[0], p[0]->len - 1);
>> +
>> +			if (get_auth_module(p[0])) {
>> +				error("duplicate auth scheme '%s'\n", p[0]->buf);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			mod = xmalloc(sizeof(struct auth_module));
>> +			mod->scheme = xstrdup(p[0]->buf);
>> +			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
> 
> Here, you xstrdup() into a 'const char *', but you are really
> passing ownership so it shouldn't be conts.
Ok

> 
>> +			mod->tokens = xmalloc(sizeof(struct string_list));
> 
> nit: this could also be "CALLOC_ARRAY(mod->tokens, 1);"
Sure!
>> +			string_list_init_dup(mod->tokens);
>> +
>> +			add_auth_module(mod);
>> +
>> +			strbuf_list_free(p);
>> +			continue;
> 
> Ok, we gain the auth schemes from the command line.
> 
>> +		}
>> +		if (skip_prefix(arg, "--auth-token=", &v)) {
>> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
>> +			if (!p[0]) {
>> +				error("invalid argument '%s'", v);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			if (!p[1]) {
>> +				error("missing token value '%s'\n", v);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			// trim trailing ':'
> 
> Use /* */ (Aside: I'm surprised we don't have a build option in
> DEVELOPER=1 that catches the use of these comments.)
Me too! Appologies here.
>> +			strbuf_setlen(p[0], p[0]->len - 1);
>> +
>> +			mod = get_auth_module(p[0]);
>> +			if (!mod) {
>> +				error("auth scheme not defined '%s'\n", p[0]->buf);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			string_list_append(mod->tokens, p[1]->buf);
>> +			strbuf_list_free(p);
>> +			continue;
>> +		}
> 
> And the token lists. It is important that the scheme is added
> before any token is added.
> 
>> +		fprintf(stderr, "error: unknown argument '%s'\n", arg);
>> +		usage(test_http_auth_usage);
>> +	}
>> +
>> +	/* avoid splitting a message in the middle */
>> +	setvbuf(stderr, NULL, _IOFBF, 4096);
>> +
>> +	if (listen_port == 0)
>> +		listen_port = DEFAULT_GIT_PORT;
>> +
>> +	/*
>> +	 * If no --listen=<addr> args are given, the setup_named_sock()
>> +	 * code will use receive a NULL address and set INADDR_ANY.
>> +	 * This exposes both internal and external interfaces on the
>> +	 * port.
>> +	 *
>> +	 * Disallow that and default to the internal-use-only loopback
>> +	 * address.
>> +	 */
>> +	if (!listen_addr.nr)
>> +		string_list_append(&listen_addr, "127.0.0.1");
>> +
>> +	/*
>> +	 * worker_mode is set in our own child process instances
>> +	 * (that are bound to a connected socket from a client).
>> +	 */
>> +	if (worker_mode)
>> +		return worker();
>> +
>> +	/*
>> +	 * `cld_argv` is a bit of a clever hack. The top-level instance
>> +	 * of test-http-server does the normal bind/listen/accept stuff.
>> +	 * For each incoming socket, the top-level process spawns
>> +	 * a child instance of test-http-server *WITH* the additional
>> +	 * `--worker` argument. This causes the child to set `worker_mode`
>> +	 * and immediately call `worker()` using the connected socket (and
>> +	 * without the usual need for fork() or threads).
>> +	 *
>> +	 * The magic here is made possible because `cld_argv` is static
>> +	 * and handle() (called by service_loop()) knows about it.
>> +	 */
>> +	strvec_push(&cld_argv, argv[0]);
>> +	strvec_push(&cld_argv, "--worker");
>> +	for (i = 1; i < argc; ++i)
>> +		strvec_push(&cld_argv, argv[i]);
>> +
>> +	/*
>> +	 * Setup primary instance to listen for connections.
>> +	 */
>> +	return serve(&listen_addr, listen_port);
>> +}
> 
> And complete the thing with some boilerplate.
> 
> This was a lot to read, and the interesting bits are all mixed in
> with the http server code, which is less interesting to what we
> are trying to accomplish. It would be beneficial to split this
> into one or two patches before we actually introduce the tests.
> 
> The most important thing that I think would be helpful is to
> isolate all the authentication behavior into its own patch so
> we can see how those connections from the command-line arguments
> affect the behavior of the server responses.
> 
> I think ideally we would have the following split:
> 
>  1. All server boilerblate. All requests 500 not-implemented.
> 
>  2. Add Git fall-through with no authentication. Add the tests
>     that are intended to allow anonymous auth.
> 
>  3. Add authentication data structures read from command-line,
>     but not processed at all in the logic.
> 
>  4. Act on the authentication data structures to alter the
>     requests. Add the tests that use these authentication
>     schemes.
> 
> I could easily see a case for combining 1&2 as well as 3&4,
> for slightly larger but more completely-testable changes at
> every step.
I agree, and my appologies for not splitting these out.
I'll follow up with a split that should make more sense.
> From what I read, I don't think there is much to change in
> the end result of the code, but it definitely was hard to read
> the important things when surrounded by many lines of
> boilerplate.
> 
>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
> 
> I'm going to pause here and come back to the test script in
> a separate reply.
> 
> Thanks,
> -Stolee
Thanks,
Matthew

git-credential.txt

[M Hickford]

Reading git-credential.txt, I'm not quite clear:

1. Are the new wwwauth[] and authtype attributes populated by Git and passed to helpers? Or vice versa?
2. Should a storage helper store these attributes? If so, must the values be treated as confidential?

Re: git-credential.txt

[Matthew John Cheetham]

On 2022-10-24 19:26, M Hickford wrote:
> Reading git-credential.txt, I'm not quite clear:
> 
> 1. Are the new wwwauth[] and authtype attributes populated by Git and passed to helpers? Or vice versa?

The wwwauth[] attribute is from Git -> helper, and the authtype attribute is
from helper -> Git. I can update the doc to make this more explicit.

> 2. Should a storage helper store these attributes? If so, must the values be treated as confidential?

Good question. A simple credential helper may wish to inspect these headers only
to differentiate the different authentication schemes available (basic, bearer,
etc) and return a credential of the correct/available type (and include an
`authtype` attribute in the response).

However it's unlikely such a helper would need to store the wwwauth[] values
as verbatim unless it can directly understand the parameters of the challenges.
The addition of this attribute is for credential helpers to gain more context
about the auth challenge from the remote.

For example, a helper may receive a bearer challenge including minimum required
OAuth scopes and an authentication authority:

wwwauth[]=Bearer authority=login.example.com/oauth scopes="code_rw userinfo_read"

Using these extra parameters the helper can try and locate an existing stored
credential that satisfies the request.

Such an enlightened helper would need to query stored credentials looking for
matching metadata including the authority, and a bearer token that has at least
the minimum required scopes (but could have a superset).

Thanks,
Matthew

[PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server based
heavily on git-daemon, with simple authentication configurable by command
line args.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Leverage newest identity standards, enhancements, and threat
     mitigations - all without updating Git.
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].

 3. Teach Git to specify authentication schemes other than Basic in
    subsequent HTTP requests based on credential helper responses.


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future flexibility
==================

By allowing the credential helpers decide the best authentication scheme, we
can allow the remote Git server to both offer new schemes (or remove old
ones) that enlightened credential helpers could take immediate advantage of,
and to use credentials that are much more tightly scoped and bound to the
specific request.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper would return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>



Should Git not control the set of authentication schemes?
=========================================================

One concern that the reader may have regarding these changes is in allowing
helpers to select the authentication mechanism to use, it may be possible
that a weaker form of authentication is used.

Take for example a Git remote server that responds with the following
authentication schemes:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Negotiate ...
WWW-Authenticate: Basic ...


Today Git (and libcurl) prefer to Negotiate over Basic authentication [13].
If a helper responded with authtype=basic Git would now be using a "less
secure" mechanism.

The reason we still propose the credential helper decide on the
authentication scheme is that Git is not the best placed entity to decide
what type of authentication should be used for a particular request (see
Design Principle 3).

OAuth Bearer tokens are often bundled in Basic Authorization headers [14],
but given that the tokens are/can be short-lived and have a highly scoped
set of permissions, this solution could be argued as being more secure than
something like NTLM [15]. Similarly, the user may wish to be consulted on
selecting a particular user account, or directly selecting an authentication
mechanism for a request that otherwise they would not be able to use.

Also, as new authentication protocols appear Git does not need to be
modified or updated for the user to take advantage of them; the credential
helpers take on the responsibility of learning and selecting the "best"
option.


Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616

 * [13] libcurl http.c pickoneauth Function
   https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416

 * [14] Git Credential Manager GitHub Host Provider (using PAT as password)
   https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157

 * [15] NT LAN Manager (NTLM) Authentication Protocol
   https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.

Matthew John Cheetham (11):
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests
  http: store all request headers on active_request_slot
  http: move proactive auth to first slot creation
  http: set specific auth scheme depending on credential
  test-http-server: add stub HTTP server test helper
  test-http-server: add HTTP error response function
  test-http-server: add HTTP request parsing
  test-http-server: pass Git requests to http-backend
  test-http-server: add simple authentication
  t5556: add HTTP authentication tests

 Documentation/git-credential.txt          |   29 +-
 Makefile                                  |    2 +
 contrib/buildsystems/CMakeLists.txt       |   13 +
 credential.c                              |   18 +
 credential.h                              |   16 +
 git-curl-compat.h                         |   10 +
 http-push.c                               |  103 +-
 http-walker.c                             |    2 +-
 http.c                                    |  200 +++-
 http.h                                    |    4 +-
 remote-curl.c                             |   36 +-
 t/helper/.gitignore                       |    1 +
 t/helper/test-credential-helper-replay.sh |   14 +
 t/helper/test-http-server.c               | 1146 +++++++++++++++++++++
 t/t5556-http-auth.sh                      |  260 +++++
 15 files changed, 1717 insertions(+), 137 deletions(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh
 create mode 100644 t/helper/test-http-server.c
 create mode 100755 t/t5556-http-auth.sh


base-commit: 9c32cfb49c60fa8173b9666db02efe3b45a8522f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v2:

  1:  f297c78f60a =  1:  f297c78f60a http: read HTTP WWW-Authenticate response headers
  2:  0838d992744 !  2:  e45e23406a5 credential: add WWW-Authenticate header to cred requests
     @@ Commit message
          C-style array syntax is used in the property name to denote multiple
          ordered values for the same property.
      
     -    In this case we send multiple `wwwauth[n]` properties where `n` is a
     -    zero-indexed number, reflecting the order the WWW-Authenticate headers
     -    appeared in the HTTP response.
     +    In this case we send multiple `wwwauth[]` properties where the order
     +    that the repeated attributes appear in the conversation reflects the
     +    order that the WWW-Authenticate headers appeared in the HTTP response.
      
          [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Documentation/git-credential.txt ##
     +@@ Documentation/git-credential.txt: separated by an `=` (equals) sign, followed by a newline.
     + The key may contain any bytes except `=`, newline, or NUL. The value may
     + contain any bytes except newline or NUL.
     + 
     +-In both cases, all bytes are treated as-is (i.e., there is no quoting,
     ++Attributes with keys that end with C-style array brackets `[]` can have
     ++multiple values. Each instance of a multi-valued attribute forms an
     ++ordered list of values - the order of the repeated attributes defines
     ++the order of the values. An empty multi-valued attribute (`key[]=\n`)
     ++acts to clear any previous entries and reset the list.
     ++
     ++In all cases, all bytes are treated as-is (i.e., there is no quoting,
     + and one cannot transmit a value with newline or NUL in it). The list of
     + attributes is terminated by a blank line or end-of-file.
     + 
      @@ Documentation/git-credential.txt: empty string.
       Components which are missing from the URL (e.g., there is no
       username in the example above) will be left unset.
       
      +`wwwauth[]`::
      +
     -+	When an HTTP response is received that includes one or more
     -+	'WWW-Authenticate' authentication headers, these can be passed to Git
     -+	(and subsequent credential helpers) with these attributes.
     -+	Each 'WWW-Authenticate' header value should be passed as a separate
     -+	attribute 'wwwauth[]' where the order of the attributes is the same
     -+	as they appear in the HTTP response.
     ++	When an HTTP response is received by Git that includes one or more
     ++	'WWW-Authenticate' authentication headers, these will be passed by Git
     ++	to credential helpers.
     ++	Each 'WWW-Authenticate' header value is passed as a multi-valued
     ++	attribute 'wwwauth[]', where the order of the attributes is the same as
     ++	they appear in the HTTP response.
      +
       GIT
       ---
  3:  c62fef65f46 =  3:  65ac638b8a0 http: store all request headers on active_request_slot
  4:  a790c01f9f2 =  4:  4d75ca29cc5 http: move proactive auth to first slot creation
  5:  b0b7cd7ee5e !  5:  2f38427aa8d http: set specific auth scheme depending on credential
     @@ Commit message
      
       ## Documentation/git-credential.txt ##
      @@ Documentation/git-credential.txt: username in the example above) will be left unset.
     - 	attribute 'wwwauth[]' where the order of the attributes is the same
     - 	as they appear in the HTTP response.
     + 	attribute 'wwwauth[]', where the order of the attributes is the same as
     + 	they appear in the HTTP response.
       
      +`authtype`::
      +
     -+	Indicates the type of authentication scheme used. If this is not
     -+	present the default is "Basic".
     ++	Indicates the type of authentication scheme that should be used by Git.
     ++	Credential helpers may reply to a request from Git with this attribute,
     ++	such that subsequent authenticated requests include the correct
     ++	`Authorization` header.
     ++	If this attribute is not present, the default value is "Basic".
      +	Known values include "Basic", "Digest", and "Bearer".
      +	If an unknown value is provided, this is taken as the authentication
      +	scheme for the `Authorization` header, and the `password` field is
  6:  f3f13ed8c82 !  6:  4947e81546a t5556-http-auth: add test for HTTP auth hdr logic
     @@ Metadata
      Author: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Commit message ##
     -    t5556-http-auth: add test for HTTP auth hdr logic
     +    test-http-server: add stub HTTP server test helper
      
     -    Add a series of tests to exercise the HTTP authentication header parsing
     -    and the interop with credential helpers. Credential helpers can respond
     -    to requests that contain WWW-Authenticate information with the ability
     -    to select the response Authenticate header scheme.
     +    Introduce a mini HTTP server helper that in the future will be enhanced
     +    to provide a frontend for the git-http-backend, with support for
     +    arbitrary authentication schemes.
      
     -    Introduce a mini HTTP server helper that provides a frontend for the
     -    git-http-backend, with support for arbitrary authentication schemes.
     -    The test-http-server is based heavily on the git-daemon, and forwards
     -    all successfully authenticated requests to the http-backend.
     +    Right now, test-http-server is a pared-down copy of the git-daemon that
     +    always returns a 501 Not Implemented response to all callers.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
     @@ t/helper/.gitignore
      @@
       /test-tool
       /test-fake-ssh
     -+test-http-server
     -
     - ## t/helper/test-credential-helper-replay.sh (new) ##
     -@@
     -+cmd=$1
     -+teefile=$cmd-actual.cred
     -+catfile=$cmd-response.cred
     -+rm -f $teefile
     -+while read line;
     -+do
     -+	if test -z "$line"; then
     -+		break;
     -+	fi
     -+	echo "$line" >> $teefile
     -+done
     -+if test "$cmd" = "get"; then
     -+	cat $catfile
     -+fi
     ++/test-http-server
      
       ## t/helper/test-http-server.c (new) ##
      @@
     @@ t/helper/test-http-server.c (new)
      +"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
      +"           [--reuseaddr] [--pid-file=<file>]\n"
      +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
     -+"           [--anonymous-allowed]\n"
     -+"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
      +;
      +
      +/* Timeout, and initial timeout */
     @@ t/helper/test-http-server.c (new)
      +	}
      +}
      +
     -+//////////////////////////////////////////////////////////////////
     -+// The code in this section is used by "worker" instances to service
     -+// a single connection from a client.  The worker talks to the client
     -+// on 0 and 1.
     -+//////////////////////////////////////////////////////////////////
     ++/*
     ++ * The code in this section is used by "worker" instances to service
     ++ * a single connection from a client.  The worker talks to the client
     ++ * on 0 and 1.
     ++ */
      +
      +enum worker_result {
      +	/*
     @@ t/helper/test-http-server.c (new)
      +	 * Caller *might* keep the socket open and allow keep-alive.
      +	 */
      +	WR_OK       = 0,
     ++
      +	/*
      +	 * Various errors while processing the request and/or the response.
      +	 * Close the socket and clean up.
      +	 * Exit child-process with non-zero status.
      +	 */
      +	WR_IO_ERROR = 1<<0,
     ++
      +	/*
      +	 * Close the socket and clean up.  Does not imply an error.
      +	 */
     @@ t/helper/test-http-server.c (new)
      +	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
      +};
      +
     -+/*
     -+ * Fields from a parsed HTTP request.
     -+ */
     -+struct req {
     -+	struct strbuf start_line;
     -+
     -+	const char *method;
     -+	const char *http_version;
     -+
     -+	struct strbuf uri_path;
     -+	struct strbuf query_args;
     -+
     -+	struct string_list header_list;
     -+	const char *content_type;
     -+	ssize_t content_length;
     -+};
     -+
     -+#define REQ__INIT { \
     -+	.start_line = STRBUF_INIT, \
     -+	.uri_path = STRBUF_INIT, \
     -+	.query_args = STRBUF_INIT, \
     -+	.header_list = STRING_LIST_INIT_NODUP, \
     -+	.content_type = NULL, \
     -+	.content_length = -1 \
     -+	}
     -+
     -+static void req__release(struct req *req)
     -+{
     -+	strbuf_release(&req->start_line);
     -+
     -+	strbuf_release(&req->uri_path);
     -+	strbuf_release(&req->query_args);
     -+
     -+	string_list_clear(&req->header_list, 0);
     -+}
     -+
     -+static enum worker_result send_http_error(
     -+	int fd,
     -+	int http_code, const char *http_code_name,
     -+	int retry_after_seconds, struct string_list *response_headers,
     -+	enum worker_result wr_in)
     -+{
     -+	struct strbuf response_header = STRBUF_INIT;
     -+	struct strbuf response_content = STRBUF_INIT;
     -+	struct string_list_item *h;
     -+	enum worker_result wr;
     -+
     -+	strbuf_addf(&response_content, "Error: %d %s\r\n",
     -+		    http_code, http_code_name);
     -+	if (retry_after_seconds > 0)
     -+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
     -+			    retry_after_seconds);
     -+
     -+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
     -+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
     -+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
     -+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
     -+	if (retry_after_seconds > 0)
     -+		strbuf_addf  (&response_header, "Retry-After: %d\r\n", retry_after_seconds);
     -+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
     -+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
     -+	if (response_headers)
     -+		for_each_string_list_item(h, response_headers)
     -+			strbuf_addf(&response_header, "%s\r\n", h->string);
     -+	strbuf_addstr(&response_header, "\r\n");
     -+
     -+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
     -+		logerror("unable to write response header");
     -+		wr = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
     -+		logerror("unable to write response content body");
     -+		wr = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	wr = wr_in;
     -+
     -+done:
     -+	strbuf_release(&response_header);
     -+	strbuf_release(&response_content);
     -+
     -+	return wr;
     -+}
     -+
     -+/*
     -+ * Read the HTTP request up to the start of the optional message-body.
     -+ * We do this byte-by-byte because we have keep-alive turned on and
     -+ * cannot rely on an EOF.
     -+ *
     -+ * https://tools.ietf.org/html/rfc7230
     -+ *
     -+ * We cannot call die() here because our caller needs to properly
     -+ * respond to the client and/or close the socket before this
     -+ * child exits so that the client doesn't get a connection reset
     -+ * by peer error.
     -+ */
     -+static enum worker_result req__read(struct req *req, int fd)
     -+{
     -+	struct strbuf h = STRBUF_INIT;
     -+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
     -+	int nr_start_line_fields;
     -+	const char *uri_target;
     -+	const char *query;
     -+	char *hp;
     -+	const char *hv;
     -+
     -+	enum worker_result result = WR_OK;
     -+
     -+	/*
     -+	 * Read line 0 of the request and split it into component parts:
     -+	 *
     -+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
     -+	 *
     -+	 */
     -+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
     -+		result = WR_OK | WR_HANGUP;
     -+		goto done;
     -+	}
     -+
     -+	strbuf_trim_trailing_newline(&req->start_line);
     -+
     -+	nr_start_line_fields = string_list_split(&start_line_fields,
     -+						 req->start_line.buf,
     -+						 ' ', -1);
     -+	if (nr_start_line_fields != 3) {
     -+		logerror("could not parse request start-line '%s'",
     -+			 req->start_line.buf);
     -+		result = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	req->method = xstrdup(start_line_fields.items[0].string);
     -+	req->http_version = xstrdup(start_line_fields.items[2].string);
     -+
     -+	uri_target = start_line_fields.items[1].string;
     -+
     -+	if (strcmp(req->http_version, "HTTP/1.1")) {
     -+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
     -+			 req->http_version);
     -+		result = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	query = strchr(uri_target, '?');
     -+
     -+	if (query) {
     -+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
     -+		strbuf_trim_trailing_dir_sep(&req->uri_path);
     -+		strbuf_addstr(&req->query_args, query + 1);
     -+	} else {
     -+		strbuf_addstr(&req->uri_path, uri_target);
     -+		strbuf_trim_trailing_dir_sep(&req->uri_path);
     -+	}
     -+
     -+	/*
     -+	 * Read the set of HTTP headers into a string-list.
     -+	 */
     -+	while (1) {
     -+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
     -+			goto done;
     -+		strbuf_trim_trailing_newline(&h);
     -+
     -+		if (!h.len)
     -+			goto done; /* a blank line ends the header */
     -+
     -+		hp = strbuf_detach(&h, NULL);
     -+		string_list_append(&req->header_list, hp);
     -+
     -+		/* store common request headers separately */
     -+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
     -+			req->content_type = hv;
     -+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
     -+			req->content_length = strtol(hv, &hp, 10);
     -+		}
     -+	}
     -+
     -+	/*
     -+	 * We do not attempt to read the <message-body>, if it exists.
     -+	 * We let our caller read/chunk it in as appropriate.
     -+	 */
     -+
     -+done:
     -+	string_list_clear(&start_line_fields, 0);
     -+
     -+	/*
     -+	 * This is useful for debugging the request, but very noisy.
     -+	 */
     -+	if (trace2_is_enabled()) {
     -+		struct string_list_item *item;
     -+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
     -+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
     -+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
     -+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
     -+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
     -+		if (req->content_length >= 0)
     -+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
     -+		if (req->content_type)
     -+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
     -+		for_each_string_list_item(item, &req->header_list)
     -+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
     -+	}
     -+
     -+	return result;
     -+}
     -+
     -+static int is_git_request(struct req *req)
     -+{
     -+	static regex_t *smart_http_regex;
     -+	static int initialized;
     -+
     -+	if (!initialized) {
     -+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
     -+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
     -+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
     -+			    REG_EXTENDED)) {
     -+			warning("could not compile smart HTTP regex");
     -+			smart_http_regex = NULL;
     -+		}
     -+		initialized = 1;
     -+	}
     -+
     -+	return smart_http_regex &&
     -+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
     -+}
     -+
     -+static enum worker_result do__git(struct req *req, const char *user)
     -+{
     -+	const char *ok = "HTTP/1.1 200 OK\r\n";
     -+	struct child_process cp = CHILD_PROCESS_INIT;
     -+	int res;
     -+
     -+	if (write(1, ok, strlen(ok)) < 0)
     -+		return error(_("could not send '%s'"), ok);
     -+
     -+	if (user)
     -+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
     -+
     -+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     -+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     -+			req->uri_path.buf);
     -+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
     -+	if (req->query_args.len)
     -+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
     -+				req->query_args.buf);
     -+	if (req->content_type)
     -+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
     -+				req->content_type);
     -+	if (req->content_length >= 0)
     -+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
     -+				(intmax_t)req->content_length);
     -+	cp.git_cmd = 1;
     -+	strvec_push(&cp.args, "http-backend");
     -+	res = run_command(&cp);
     -+	close(1);
     -+	close(0);
     -+	return !!res;
     -+}
     -+
     -+enum auth_result {
     -+	AUTH_UNKNOWN = 0,
     -+	AUTH_DENY = 1,
     -+	AUTH_ALLOW = 2,
     -+};
     -+
     -+struct auth_module {
     -+	const char *scheme;
     -+	const char *challenge_params;
     -+	struct string_list *tokens;
     -+};
     -+
     -+static int allow_anonymous;
     -+static struct auth_module **auth_modules = NULL;
     -+static size_t auth_modules_nr = 0;
     -+static size_t auth_modules_alloc = 0;
     -+
     -+static struct auth_module *get_auth_module(struct strbuf *scheme)
     -+{
     -+	int i;
     -+	struct auth_module *mod;
     -+	for (i = 0; i < auth_modules_nr; i++) {
     -+		mod = auth_modules[i];
     -+		if (!strcasecmp(mod->scheme, scheme->buf))
     -+			return mod;
     -+	}
     -+
     -+	return NULL;
     -+}
     -+
     -+static void add_auth_module(struct auth_module *mod)
     -+{
     -+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
     -+	auth_modules[auth_modules_nr++] = mod;
     -+}
     -+
     -+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
     -+{
     -+	enum auth_result result = AUTH_UNKNOWN;
     -+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
     -+	struct auth_module *mod;
     -+
     -+	struct string_list_item *hdr;
     -+	struct string_list_item *token;
     -+	const char *v;
     -+	struct strbuf **split = NULL;
     -+	int i;
     -+	char *challenge;
     -+
     -+	/* ask all auth modules to validate the request */
     -+	for_each_string_list_item(hdr, &req->header_list) {
     -+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
     -+			split = strbuf_split_str(v, ' ', 2);
     -+			if (!split[0] || !split[1]) continue;
     -+
     -+			// trim trailing space ' '
     -+			strbuf_setlen(split[0], split[0]->len - 1);
     -+
     -+			mod = get_auth_module(split[0]);
     -+			if (mod) {
     -+
     -+				for_each_string_list_item(token, mod->tokens) {
     -+					if (!strcmp(split[1]->buf, token->string)) {
     -+						result = AUTH_ALLOW;
     -+						goto done;
     -+					}
     -+				}
     -+
     -+				if (result != AUTH_UNKNOWN)
     -+					goto done;
     -+			}
     -+		}
     -+	}
     -+
     -+done:
     -+	switch (result) {
     -+	case AUTH_ALLOW:
     -+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
     -+		*user = "VALID_TEST_USER";
     -+		*wr = WR_OK;
     -+		break;
     -+
     -+	case AUTH_DENY:
     -+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
     -+		/* fall-through */
     -+
     -+	case AUTH_UNKNOWN:
     -+		if (allow_anonymous)
     -+			break;
     -+		for (i = 0; i < auth_modules_nr; i++) {
     -+			mod = auth_modules[i];
     -+			if (mod->challenge_params)
     -+				challenge = xstrfmt("WWW-Authenticate: %s %s",
     -+						    mod->scheme,
     -+						    mod->challenge_params);
     -+			else
     -+				challenge = xstrfmt("WWW-Authenticate: %s",
     -+						    mod->scheme);
     -+			string_list_append(&hdrs, challenge);
     -+		}
     -+		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
     -+	}
     -+
     -+	strbuf_list_free(split);
     -+	string_list_clear(&hdrs, 0);
     -+
     -+	return result == AUTH_ALLOW ||
     -+	      (result == AUTH_UNKNOWN && allow_anonymous);
     -+}
     -+
     -+static enum worker_result dispatch(struct req *req)
     -+{
     -+	enum worker_result wr = WR_OK;
     -+	const char *user = NULL;
     -+
     -+	if (!is_authed(req, &user, &wr))
     -+		return wr;
     -+
     -+	if (is_git_request(req))
     -+		return do__git(req, user);
     -+
     -+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
     -+			       WR_OK | WR_HANGUP);
     -+}
     -+
      +static enum worker_result worker(void)
      +{
     -+	struct req req = REQ__INIT;
     ++	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
      +	char *client_addr = getenv("REMOTE_ADDR");
      +	char *client_port = getenv("REMOTE_PORT");
      +	enum worker_result wr = WR_OK;
     @@ t/helper/test-http-server.c (new)
      +	set_keep_alive(0);
      +
      +	while (1) {
     -+		req__release(&req);
     -+
     -+		alarm(init_timeout ? init_timeout : timeout);
     -+		wr = req__read(&req, 0);
     -+		alarm(0);
     -+
     -+		if (wr & WR_STOP_THE_MUSIC)
     -+			break;
     ++		if (write_in_full(1, response, strlen(response)) < 0) {
     ++			logerror("unable to write response");
     ++			wr = WR_IO_ERROR;
     ++		}
      +
     -+		wr = dispatch(&req);
      +		if (wr & WR_STOP_THE_MUSIC)
      +			break;
      +	}
     @@ t/helper/test-http-server.c (new)
      +	return !!(wr & WR_IO_ERROR);
      +}
      +
     -+//////////////////////////////////////////////////////////////////
     -+// This section contains the listener and child-process management
     -+// code used by the primary instance to accept incoming connections
     -+// and dispatch them to async child process "worker" instances.
     -+//////////////////////////////////////////////////////////////////
     ++/*
     ++ * This section contains the listener and child-process management
     ++ * code used by the primary instance to accept incoming connections
     ++ * and dispatch them to async child process "worker" instances.
     ++ */
      +
      +static int addrcmp(const struct sockaddr_storage *s1,
      +		   const struct sockaddr_storage *s2)
     @@ t/helper/test-http-server.c (new)
      +
      +	set_keep_alive(sockfd);
      +
     -+	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
     ++	if (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {
      +		logerror("Could not bind to %s: %s",
      +			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
      +			 strerror(errno));
     @@ t/helper/test-http-server.c (new)
      +	return service_loop(&socklist);
      +}
      +
     -+//////////////////////////////////////////////////////////////////
     -+// This section is executed by both the primary instance and all
     -+// worker instances.  So, yes, each child-process re-parses the
     -+// command line argument and re-discovers how it should behave.
     -+//////////////////////////////////////////////////////////////////
     ++/*
     ++ * This section is executed by both the primary instance and all
     ++ * worker instances.  So, yes, each child-process re-parses the
     ++ * command line argument and re-discovers how it should behave.
     ++ */
      +
      +int cmd_main(int argc, const char **argv)
      +{
     @@ t/helper/test-http-server.c (new)
      +	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
      +	int worker_mode = 0;
      +	int i;
     -+	struct auth_module *mod = NULL;
      +
      +	trace2_cmd_name("test-http-server");
      +	setup_git_directory_gently(NULL);
     @@ t/helper/test-http-server.c (new)
      +			pid_file = v;
      +			continue;
      +		}
     -+		if (skip_prefix(arg, "--allow-anonymous", &v)) {
     -+			allow_anonymous = 1;
     -+			continue;
     -+		}
     -+		if (skip_prefix(arg, "--auth=", &v)) {
     -+			struct strbuf **p = strbuf_split_str(v, ':', 2);
     -+
     -+			if (!p[0]) {
     -+				error("invalid argument '%s'", v);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			// trim trailing ':'
     -+			if (p[1])
     -+				strbuf_setlen(p[0], p[0]->len - 1);
     -+
     -+			if (get_auth_module(p[0])) {
     -+				error("duplicate auth scheme '%s'\n", p[0]->buf);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			mod = xmalloc(sizeof(struct auth_module));
     -+			mod->scheme = xstrdup(p[0]->buf);
     -+			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
     -+			mod->tokens = xmalloc(sizeof(struct string_list));
     -+			string_list_init_dup(mod->tokens);
     -+
     -+			add_auth_module(mod);
     -+
     -+			strbuf_list_free(p);
     -+			continue;
     -+		}
     -+		if (skip_prefix(arg, "--auth-token=", &v)) {
     -+			struct strbuf **p = strbuf_split_str(v, ':', 2);
     -+			if (!p[0]) {
     -+				error("invalid argument '%s'", v);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			if (!p[1]) {
     -+				error("missing token value '%s'\n", v);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			// trim trailing ':'
     -+			strbuf_setlen(p[0], p[0]->len - 1);
     -+
     -+			mod = get_auth_module(p[0]);
     -+			if (!mod) {
     -+				error("auth scheme not defined '%s'\n", p[0]->buf);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			string_list_append(mod->tokens, p[1]->buf);
     -+			strbuf_list_free(p);
     -+			continue;
     -+		}
      +
      +		fprintf(stderr, "error: unknown argument '%s'\n", arg);
      +		usage(test_http_auth_usage);
     @@ t/helper/test-http-server.c (new)
      +	 */
      +	return serve(&listen_addr, listen_port);
      +}
     -
     - ## t/t5556-http-auth.sh (new) ##
     -@@
     -+#!/bin/sh
     -+
     -+test_description='test http auth header and credential helper interop'
     -+
     -+. ./test-lib.sh
     -+
     -+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
     -+
     -+# Setup a repository
     -+#
     -+REPO_DIR="$(pwd)"/repo
     -+
     -+# Setup some lookback URLs where test-http-server will be listening.
     -+# We will spawn it directly inside the repo directory, so we avoid
     -+# any need to configure directory mappings etc - we only serve this
     -+# repository from the root '/' of the server.
     -+#
     -+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
     -+ORIGIN_URL=http://$HOST_PORT/
     -+
     -+# The pid-file is created by test-http-server when it starts.
     -+# The server will shutdown if/when we delete it (this is easier than
     -+# killing it by PID).
     -+#
     -+PID_FILE="$(pwd)"/pid-file.pid
     -+SERVER_LOG="$(pwd)"/OUT.server.log
     -+
     -+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     -+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
     -+	&& export CREDENTIAL_HELPER
     -+
     -+test_expect_success 'setup repos' '
     -+	test_create_repo "$REPO_DIR" &&
     -+	git -C "$REPO_DIR" branch -M main
     -+'
     -+
     -+stop_http_server () {
     -+	if ! test -f "$PID_FILE"
     -+	then
     -+		return 0
     -+	fi
     -+	#
     -+	# The server will shutdown automatically when we delete the pid-file.
     -+	#
     -+	rm -f "$PID_FILE"
     -+	#
     -+	# Give it a few seconds to shutdown (mainly to completely release the
     -+	# port before the next test start another instance and it attempts to
     -+	# bind to it).
     -+	#
     -+	for k in 0 1 2 3 4
     -+	do
     -+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
     -+		then
     -+			return 0
     -+		fi
     -+		sleep 1
     -+	done
     -+
     -+	echo "stop_http_server: timeout waiting for server shutdown"
     -+	return 1
     -+}
     -+
     -+start_http_server () {
     -+	#
     -+	# Launch our server into the background in repo_dir.
     -+	#
     -+	(
     -+		cd "$REPO_DIR"
     -+		test-http-server --verbose \
     -+			--listen=127.0.0.1 \
     -+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
     -+			--reuseaddr \
     -+			--pid-file="$PID_FILE" \
     -+			"$@" \
     -+			2>"$SERVER_LOG" &
     -+	)
     -+	#
     -+	# Give it a few seconds to get started.
     -+	#
     -+	for k in 0 1 2 3 4
     -+	do
     -+		if test -f "$PID_FILE"
     -+		then
     -+			return 0
     -+		fi
     -+		sleep 1
     -+	done
     -+
     -+	echo "start_http_server: timeout waiting for server startup"
     -+	return 1
     -+}
     -+
     -+per_test_cleanup () {
     -+	stop_http_server &&
     -+	rm -f OUT.* &&
     -+	rm -f *.cred
     -+}
     -+
     -+test_expect_success 'http auth anonymous no challenge' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server --allow-anonymous &&
     -+
     -+	# Attempt to read from a protected repository
     -+	git ls-remote $ORIGIN_URL
     -+'
     -+
     -+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=bearer:secret-token &&
     -+
     -+	cat >get-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >store-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-token
     -+	authtype=bearer
     -+	EOF
     -+
     -+	cat >get-response.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-token
     -+	authtype=bearer
     -+	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
     -+'
     -+
     -+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=basic:$USERPASS64 &&
     -+
     -+	cat >get-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >store-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     -+	authtype=basic
     -+	EOF
     -+
     -+	cat >get-response.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     -+	authtype=basic
     -+	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
     -+'
     -+
     -+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server \
     -+		--auth=foobar:alg=test\ widget=1 \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=foobar:SECRET-FOOBAR-VALUE &&
     -+
     -+	cat >get-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=foobar alg=test widget=1
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >store-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=SECRET-FOOBAR-VALUE
     -+	authtype=foobar
     -+	EOF
     -+
     -+	cat >get-response.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=SECRET-FOOBAR-VALUE
     -+	authtype=foobar
     -+	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
     -+'
     -+
     -+test_expect_success 'http auth www-auth headers to credential helper invalid' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=bearer:secret-token &&
     -+
     -+	cat >get-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >erase-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=invalid-token
     -+	authtype=bearer
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >get-response.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=invalid-token
     -+	authtype=bearer
     -+	EOF
     -+
     -+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp erase-expected.cred erase-actual.cred
     -+'
     -+
     -+test_done
  -:  ----------- >  7:  93bdf1d7060 test-http-server: add HTTP error response function
  -:  ----------- >  8:  b3e9156755f test-http-server: add HTTP request parsing
  -:  ----------- >  9:  5fb248c074a test-http-server: pass Git requests to http-backend
  -:  ----------- > 10:  192f09b9de4 test-http-server: add simple authentication
  -:  ----------- > 11:  b64d2f2c473 t5556: add HTTP authentication tests

-- 
gitgitgadget

[PATCH v3 01/11] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 ++++++++++
 http.c       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index 5d0502f51fd..03d43d352e7 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	const char *z = NULL;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		const char **v = values->v + values->nr - 1;
+		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
+
+		free((void*)*v);
+		*v = append;
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (skip_iprefix(buf.buf, "http/", &z))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1829,6 +1905,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

[PATCH v3 02/11] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt | 17 ++++++++++++++++-
 credential.c                     | 12 ++++++++++++
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f18673017f5..791a57dddfb 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,15 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
+	Each 'WWW-Authenticate' header value is passed as a multi-valued
+	attribute 'wwwauth[]', where the order of the attributes is the same as
+	they appear in the HTTP response.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/credential.c b/credential.c
index 897b4679333..8a3ad6c0ae2 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	int i = 0;
+	const char *full_key = xstrfmt("%s[]", key);
+	for (; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free((void*)full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
-- 
gitgitgadget

[PATCH v3 03/11] http: store all request headers on active_request_slot

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Once a list of headers has been set on the curl handle, it is not
possible to recover that `struct curl_slist` instance to add or modify
headers.

In future commits we will want to modify the set of request headers in
response to an authentication challenge/401 response from the server,
with information provided by a credential helper.

There are a number of different places where curl is used for an HTTP
request, and they do not have a common handling of request headers.
However, given that they all do call the `start_active_slot()` function,
either directly or indirectly via `run_slot()` or `run_one_slot()`, we
use this as the point to set the `CURLOPT_HTTPHEADER` option just
before the request is made.

We collect all request headers in a `struct curl_slist` on the
`struct active_request_slot` that is obtained from a call to
`get_active_slot(int)`. This function now takes a single argument to
define if the initial set of headers on the slot should include the
"Pragma: no-cache" header, along with all extra headers specified via
`http.extraHeader` config values.

The active request slot obtained from `get_active_slot(int)` will always
contain a fresh set of default headers and any headers set in previous
usages of this slot will be freed.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http-push.c   | 103 ++++++++++++++++++++++----------------------------
 http-walker.c |   2 +-
 http.c        |  82 ++++++++++++++++++----------------------
 http.h        |   4 +-
 remote-curl.c |  36 +++++++++---------
 5 files changed, 101 insertions(+), 126 deletions(-)

diff --git a/http-push.c b/http-push.c
index 5f4340a36e6..2b40959b376 100644
--- a/http-push.c
+++ b/http-push.c
@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,
 	curl_easy_setopt(curl, CURLOPT_UPLOAD, 1);
 }
 
-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)
+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,
+	struct remote_lock *lock, enum dav_header_flag options)
 {
 	struct strbuf buf = STRBUF_INIT;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
 	if (options & DAV_HEADER_IF) {
 		strbuf_addf(&buf, "If: (<%s>)", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_LOCK) {
 		strbuf_addf(&buf, "Lock-Token: <%s>", lock->token);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	if (options & DAV_HEADER_TIMEOUT) {
 		strbuf_addf(&buf, "Timeout: Second-%ld", lock->timeout);
-		dav_headers = curl_slist_append(dav_headers, buf.buf);
+		headers = curl_slist_append(headers, buf.buf);
 		strbuf_reset(&buf);
 	}
 	strbuf_release(&buf);
 
-	return dav_headers;
+	return headers;
 }
 
 static void finish_request(struct transfer_request *request);
@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)
 
 	request->url = get_remote_object_url(repo->url, hex, 1);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MKCOL);
@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)
 	strbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);
 	request->url = strbuf_detach(&buf, NULL);
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http(slot->curl, request->url, DAV_PUT,
@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)
 static void start_move(struct transfer_request *request)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	curl_setup_http_get(slot->curl, request->url, DAV_MOVE);
-	dav_headers = curl_slist_append(dav_headers, request->dest);
-	dav_headers = curl_slist_append(dav_headers, "Overwrite: T");
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
+	slot->headers = curl_slist_append(slot->headers, request->dest);
+	slot->headers = curl_slist_append(slot->headers, "Overwrite: T");
 
 	if (start_active_slot(slot)) {
 		request->slot = slot;
@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
 	lock->refreshing = 1;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF | DAV_HEADER_TIMEOUT);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_LOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)
 	}
 
 	lock->refreshing = 0;
-	curl_slist_free_all(dav_headers);
 
 	return rc;
 }
@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	char *ep;
 	char timeout_header[25];
 	struct remote_lock *lock = NULL;
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	char *escaped;
 
@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	while (ep) {
 		char saved_character = ep[1];
 		ep[1] = '\0';
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
 		curl_setup_http_get(slot->curl, url, DAV_MKCOL);
 		if (start_active_slot(slot)) {
@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 	strbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);
 	free(escaped);
 
+	slot = get_active_slot(0);
+	slot->results = &results;
+
 	xsnprintf(timeout_header, sizeof(timeout_header), "Timeout: Second-%ld", timeout);
-	dav_headers = curl_slist_append(dav_headers, timeout_header);
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
+	slot->headers = curl_slist_append(slot->headers, timeout_header);
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
 
-	slot = get_active_slot();
-	slot->results = &results;
 	curl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	CALLOC_ARRAY(lock, 1);
@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
 		fprintf(stderr, "Unable to start LOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
 
@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct remote_lock *prev = repo->locks;
-	struct curl_slist *dav_headers;
 	int rc = 0;
 
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_LOCK);
+
 	curl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)
 		fprintf(stderr, "Unable to start UNLOCK request\n");
 	}
 
-	curl_slist_free_all(dav_headers);
-
 	if (repo->locks == lock) {
 		repo->locks = lock->next;
 	} else {
@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	struct remote_ls_ctx ls;
 
@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,
 
 	strbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 1");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 1");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,
 	free(url);
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 }
 
 static void get_remote_object_list(unsigned char parent)
@@ -1199,7 +1189,6 @@ static int locking_available(void)
 	struct slot_results results;
 	struct strbuf in_buffer = STRBUF_INIT;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers = http_copy_default_headers();
 	struct xml_ctx ctx;
 	int lock_flags = 0;
 	char *escaped;
@@ -1208,14 +1197,14 @@ static int locking_available(void)
 	strbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);
 	free(escaped);
 
-	dav_headers = curl_slist_append(dav_headers, "Depth: 0");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = curl_slist_append(slot->headers, "Depth: 0");
+	slot->headers = curl_slist_append(slot->headers,
+		"Content-Type: text/xml");
+
 	curl_setup_http(slot->curl, repo->url, DAV_PROPFIND,
 			&out_buffer, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);
 
 	if (start_active_slot(slot)) {
@@ -1257,7 +1246,6 @@ static int locking_available(void)
 
 	strbuf_release(&out_buffer.buf);
 	strbuf_release(&in_buffer);
-	curl_slist_free_all(dav_headers);
 
 	return lock_flags;
 }
@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)
 	struct active_request_slot *slot;
 	struct slot_results results;
 	struct buffer out_buffer = { STRBUF_INIT, 0 };
-	struct curl_slist *dav_headers;
-
-	dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
 	strbuf_addf(&out_buffer.buf, "%s\n", oid_to_hex(oid));
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
+	slot->headers = append_dav_token_headers(slot->headers, lock,
+		DAV_HEADER_IF);
+
 	curl_setup_http(slot->curl, lock->url, DAV_PUT,
 			&out_buffer, fwrite_null);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)
 	struct buffer buffer = { STRBUF_INIT, 0 };
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *dav_headers;
 
 	remote_ls("refs/", (PROCESS_FILES | RECURSIVE),
 		  add_remote_info_ref, &buffer.buf);
 	if (!aborted) {
-		dav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);
 
-		slot = get_active_slot();
+		slot = get_active_slot(0);
 		slot->results = &results;
+		slot->headers = append_dav_token_headers(slot->headers, lock,
+			DAV_HEADER_IF);
+
 		curl_setup_http(slot->curl, lock->url, DAV_PUT,
 				&buffer, fwrite_null);
-		curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
 
 		if (start_active_slot(slot)) {
 			run_active_slot(slot);
@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)
 	if (dry_run)
 		return 0;
 	url = xstrfmt("%s%s", repo->url, remote_ref->name);
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->results = &results;
 	curl_setup_http_get(slot->curl, url, DAV_DELETE);
 	if (start_active_slot(slot)) {
diff --git a/http-walker.c b/http-walker.c
index b8f0f98ae14..8747de2fcdb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)
 	 * Use a callback to process the result, since another request
 	 * may fail and need to have alternates loaded before continuing
 	 */
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 	slot->callback_func = process_alternates_response;
 	alt_req.walker = walker;
 	slot->callback_data = &alt_req;
diff --git a/http.c b/http.c
index 03d43d352e7..f2ebb17c8c4 100644
--- a/http.c
+++ b/http.c
@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =
 	| CURLAUTH_DIGEST_IE
 	| CURLAUTH_DIGEST;
 
-static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
 static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
 
 static struct curl_slist *host_resolutions;
@@ -1133,11 +1131,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 	if (remote)
 		var_override(&http_proxy_authmethod, remote->http_proxy_authmethod);
 
-	pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma: no-cache");
-	no_pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma:");
-
 	{
 		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
 		if (http_max_requests)
@@ -1199,6 +1192,8 @@ void http_cleanup(void)
 
 	while (slot != NULL) {
 		struct active_request_slot *next = slot->next;
+		if (slot->headers)
+			curl_slist_free_all(slot->headers);
 		if (slot->curl) {
 			xmulti_remove_handle(slot);
 			curl_easy_cleanup(slot->curl);
@@ -1215,12 +1210,6 @@ void http_cleanup(void)
 
 	string_list_clear(&extra_http_headers, 0);
 
-	curl_slist_free_all(pragma_header);
-	pragma_header = NULL;
-
-	curl_slist_free_all(no_pragma_header);
-	no_pragma_header = NULL;
-
 	curl_slist_free_all(host_resolutions);
 	host_resolutions = NULL;
 
@@ -1255,7 +1244,18 @@ void http_cleanup(void)
 	FREE_AND_NULL(cached_accept_language);
 }
 
-struct active_request_slot *get_active_slot(void)
+static struct curl_slist *http_copy_default_headers(void)
+{
+	struct curl_slist *headers = NULL;
+	const struct string_list_item *item;
+
+	for_each_string_list_item(item, &extra_http_headers)
+		headers = curl_slist_append(headers, item->string);
+
+	return headers;
+}
+
+struct active_request_slot *get_active_slot(int no_pragma_header)
 {
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
@@ -1277,6 +1277,7 @@ struct active_request_slot *get_active_slot(void)
 		newslot->curl = NULL;
 		newslot->in_use = 0;
 		newslot->next = NULL;
+		newslot->headers = NULL;
 
 		slot = active_queue_head;
 		if (!slot) {
@@ -1294,6 +1295,15 @@ struct active_request_slot *get_active_slot(void)
 		curl_session_count++;
 	}
 
+	if (slot->headers)
+		curl_slist_free_all(slot->headers);
+
+	slot->headers = http_copy_default_headers();
+
+	if (!no_pragma_header)
+		slot->headers = curl_slist_append(slot->headers,
+			"Pragma: no-cache");
+
 	active_requests++;
 	slot->in_use = 1;
 	slot->results = NULL;
@@ -1303,7 +1313,6 @@ struct active_request_slot *get_active_slot(void)
 	curl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);
 	if (curl_save_cookies)
 		curl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
 	curl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);
 	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
 	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);
@@ -1335,9 +1344,12 @@ struct active_request_slot *get_active_slot(void)
 
 int start_active_slot(struct active_request_slot *slot)
 {
-	CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);
+	CURLMcode curlm_result;
 	int num_transfers;
 
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);
+	curlm_result = curl_multi_add_handle(curlm, slot->curl);
+
 	if (curlm_result != CURLM_OK &&
 	    curlm_result != CURLM_CALL_MULTI_PERFORM) {
 		warning("curl_multi_add_handle failed: %s",
@@ -1652,17 +1664,6 @@ int run_one_slot(struct active_request_slot *slot,
 	return handle_curl_result(results);
 }
 
-struct curl_slist *http_copy_default_headers(void)
-{
-	struct curl_slist *headers = NULL;
-	const struct string_list_item *item;
-
-	for_each_string_list_item(item, &extra_http_headers)
-		headers = curl_slist_append(headers, item->string);
-
-	return headers;
-}
-
 static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)
 {
 	char *ptr;
@@ -1880,12 +1881,11 @@ static int http_request(const char *url,
 {
 	struct active_request_slot *slot;
 	struct slot_results results;
-	struct curl_slist *headers = http_copy_default_headers();
-	struct strbuf buf = STRBUF_INIT;
+	int no_cache = options && options->no_cache;
 	const char *accept_language;
 	int ret;
 
-	slot = get_active_slot();
+	slot = get_active_slot(!no_cache);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);
 
 	if (!result) {
@@ -1910,27 +1910,23 @@ static int http_request(const char *url,
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-		headers = curl_slist_append(headers, accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			accept_language);
 
-	strbuf_addstr(&buf, "Pragma:");
-	if (options && options->no_cache)
-		strbuf_addstr(&buf, " no-cache");
 	if (options && options->initial_request &&
 	    http_follow_config == HTTP_FOLLOW_INITIAL)
 		curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);
 
-	headers = curl_slist_append(headers, buf.buf);
-
 	/* Add additional headers here */
 	if (options && options->extra_headers) {
 		const struct string_list_item *item;
 		for_each_string_list_item(item, options->extra_headers) {
-			headers = curl_slist_append(headers, item->string);
+			slot->headers = curl_slist_append(slot->headers,
+				item->string);
 		}
 	}
 
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
 	curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);
 
@@ -1948,9 +1944,6 @@ static int http_request(const char *url,
 		curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
 				options->effective_url);
 
-	curl_slist_free_all(headers);
-	strbuf_release(&buf);
-
 	return ret;
 }
 
@@ -2311,12 +2304,10 @@ struct http_pack_request *new_direct_http_pack_request(
 		goto abort;
 	}
 
-	preq->slot = get_active_slot();
+	preq->slot = get_active_slot(1);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
-	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
-		no_pragma_header);
 
 	/*
 	 * If there is data present from a previous transfer attempt,
@@ -2481,14 +2472,13 @@ struct http_object_request *new_http_object_request(const char *base_url,
 		}
 	}
 
-	freq->slot = get_active_slot();
+	freq->slot = get_active_slot(1);
 
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
 
 	/*
 	 * If we have successfully processed data from a previous fetch
diff --git a/http.h b/http.h
index 3c94c479100..a304cc408b2 100644
--- a/http.h
+++ b/http.h
@@ -22,6 +22,7 @@ struct slot_results {
 struct active_request_slot {
 	CURL *curl;
 	int in_use;
+	struct curl_slist *headers;
 	CURLcode curl_result;
 	long http_code;
 	int *finished;
@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
 curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);
 
 /* Slot lifecycle functions */
-struct active_request_slot *get_active_slot(void);
+struct active_request_slot *get_active_slot(int no_pragma_header);
 int start_active_slot(struct active_request_slot *slot);
 void run_active_slot(struct active_request_slot *slot);
 void finish_all_active_slots(void);
@@ -64,7 +65,6 @@ void step_active_slots(void);
 void http_init(struct remote *remote, const char *url,
 	       int proactive_auth);
 void http_cleanup(void);
-struct curl_slist *http_copy_default_headers(void);
 
 extern long int git_curl_ipresolve;
 extern int active_requests;
diff --git a/remote-curl.c b/remote-curl.c
index 72dfb8fb86a..edbd4504beb 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,
 static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	struct strbuf buf = STRBUF_INIT;
 	int err;
 
-	slot = get_active_slot();
+	slot = get_active_slot(0);
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000");
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);
 
 	err = run_slot(slot, results);
 
-	curl_slist_free_all(headers);
 	strbuf_release(&buf);
 	return err;
 }
@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)
 static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
 	size_t gzip_size = 0;
@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 			needs_100_continue = 1;
 	}
 
-	headers = curl_slist_append(headers, rpc->hdr_content_type);
-	headers = curl_slist_append(headers, rpc->hdr_accept);
-	headers = curl_slist_append(headers, needs_100_continue ?
+retry:
+	slot = get_active_slot(0);
+
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);
+	slot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);
+	slot->headers = curl_slist_append(slot->headers, needs_100_continue ?
 		"Expect: 100-continue" : "Expect:");
 
 	/* Add Accept-Language header */
 	if (rpc->hdr_accept_language)
-		headers = curl_slist_append(headers, rpc->hdr_accept_language);
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->hdr_accept_language);
 
 	/* Add the extra Git-Protocol header */
 	if (rpc->protocol_header)
-		headers = curl_slist_append(headers, rpc->protocol_header);
-
-retry:
-	slot = get_active_slot();
+		slot->headers = curl_slist_append(slot->headers,
+			rpc->protocol_header);
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
@@ -955,7 +953,8 @@ retry:
 		/* The request body is large and the size cannot be predicted.
 		 * We must use chunked encoding to send it.
 		 */
-		headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+		slot->headers = curl_slist_append(slot->headers,
+			"Transfer-Encoding: chunked");
 		rpc->initial_buffer = 1;
 		curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);
 		curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);
@@ -1002,7 +1001,8 @@ retry:
 
 		gzip_size = stream.total_out;
 
-		headers = curl_slist_append(headers, "Content-Encoding: gzip");
+		slot->headers = curl_slist_append(slot->headers,
+			"Content-Encoding: gzip");
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));
 
@@ -1025,7 +1025,6 @@ retry:
 		}
 	}
 
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);
 	rpc_in_data.rpc = rpc;
 	rpc_in_data.slot = slot;
@@ -1055,7 +1054,6 @@ retry:
 	if (stateless_connect)
 		packet_response_end(rpc->in);
 
-	curl_slist_free_all(headers);
 	free(gzip_body);
 	return err;
 }
-- 
gitgitgadget

Re: [PATCH v3 03/11] http: store all request headers on active_request_slot

[Glen Choo]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> Once a list of headers has been set on the curl handle, it is not
> possible to recover that `struct curl_slist` instance to add or modify
> headers.
>
> In future commits we will want to modify the set of request headers in
> response to an authentication challenge/401 response from the server,
> with information provided by a credential helper.
>
> There are a number of different places where curl is used for an HTTP
> request, and they do not have a common handling of request headers.
> However, given that they all do call the `start_active_slot()` function,
> either directly or indirectly via `run_slot()` or `run_one_slot()`, we
> use this as the point to set the `CURLOPT_HTTPHEADER` option just
> before the request is made.
>
> We collect all request headers in a `struct curl_slist` on the
> `struct active_request_slot` that is obtained from a call to
> `get_active_slot(int)`. This function now takes a single argument to
> define if the initial set of headers on the slot should include the
> "Pragma: no-cache" header, along with all extra headers specified via
> `http.extraHeader` config values.

I admit that I'm not that familiar with the http subsystem, so I'll
focus on the style.

If I'm reading this patch correctly, there are two related, but distinct
changes:

- store and modify the headers on the slot
- change how headers are initialized and remove now-unncessary libcurl
  calls that set headers

Both are simple, but given the number of LoCs changed, I found it quite
difficult to track which LoCs were part of which work. Could this be
broken up into two patches instead, i.e.:

- store headers on the slot without changing how they are initialized
- add extra header initialization logic to get_active_slot() and remove
  the unnecessary libcurl calls

[PATCH v3 04/11] http: move proactive auth to first slot creation

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rather than proactively seek credentials to authenticate a request at
`http_init()` time, do it when the first `active_request_slot` is
created.

Because credential helpers may modify the headers used for a request we
can only auth when a slot is created (when we can first start to gather
request headers).

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/http.c b/http.c
index f2ebb17c8c4..17b47195d22 100644
--- a/http.c
+++ b/http.c
@@ -515,18 +515,18 @@ static int curl_empty_auth_enabled(void)
 	return 0;
 }
 
-static void init_curl_http_auth(CURL *result)
+static void init_curl_http_auth(struct active_request_slot *slot)
 {
 	if (!http_auth.username || !*http_auth.username) {
 		if (curl_empty_auth_enabled())
-			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
+			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
 	}
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
+	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
 }
 
 /* *var must be free-able */
@@ -901,9 +901,6 @@ static CURL *get_curl_handle(void)
 #endif
 	}
 
-	if (http_proactive_auth)
-		init_curl_http_auth(result);
-
 	if (getenv("GIT_SSL_VERSION"))
 		ssl_version = getenv("GIT_SSL_VERSION");
 	if (ssl_version && *ssl_version) {
@@ -1260,6 +1257,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 	struct active_request_slot *slot = active_queue_head;
 	struct active_request_slot *newslot;
 
+	int proactive_auth = 0;
 	int num_transfers;
 
 	/* Wait for a slot to open up if the queue is full */
@@ -1282,6 +1280,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 		slot = active_queue_head;
 		if (!slot) {
 			active_queue_head = newslot;
+
+			/* Auth first slot if asked for proactive auth */
+			proactive_auth = http_proactive_auth;
 		} else {
 			while (slot->next != NULL)
 				slot = slot->next;
@@ -1336,8 +1337,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)
 
 	curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
-	if (http_auth.password || curl_empty_auth_enabled())
-		init_curl_http_auth(slot->curl);
+
+	if (http_auth.password || curl_empty_auth_enabled() || proactive_auth)
+		init_curl_http_auth(slot);
 
 	return slot;
 }
-- 
gitgitgadget

[PATCH v3 05/11] http: set specific auth scheme depending on credential

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a new credential field `authtype` that can be used by
credential helpers to indicate the type of the credential or
authentication mechanism to use for a request.

Modify http.c to now specify the correct authentication scheme or
credential type when authenticating the curl handle. If the new
`authtype` field in the credential structure is `NULL` or "Basic" then
use the existing username/password options. If the field is "Bearer"
then use the OAuth bearer token curl option. Otherwise, the `authtype`
field is the authentication scheme and the `password` field is the
raw, unencoded value.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt | 12 ++++++++++++
 credential.c                     |  5 +++++
 credential.h                     |  1 +
 git-curl-compat.h                | 10 ++++++++++
 http.c                           | 24 +++++++++++++++++++++---
 5 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 791a57dddfb..9069bfb2d50 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -175,6 +175,18 @@ username in the example above) will be left unset.
 	attribute 'wwwauth[]', where the order of the attributes is the same as
 	they appear in the HTTP response.
 
+`authtype`::
+
+	Indicates the type of authentication scheme that should be used by Git.
+	Credential helpers may reply to a request from Git with this attribute,
+	such that subsequent authenticated requests include the correct
+	`Authorization` header.
+	If this attribute is not present, the default value is "Basic".
+	Known values include "Basic", "Digest", and "Bearer".
+	If an unknown value is provided, this is taken as the authentication
+	scheme for the `Authorization` header, and the `password` field is
+	used as the raw unencoded authorization parameters of the same header.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/credential.c b/credential.c
index 8a3ad6c0ae2..a556f9f375a 100644
--- a/credential.c
+++ b/credential.c
@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "path")) {
 			free(c->path);
 			c->path = xstrdup(value);
+		} else if (!strcmp(key, "authtype")) {
+			free(c->authtype);
+			c->authtype = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "authtype", c->authtype, 0);
 	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
diff --git a/credential.h b/credential.h
index 6f2e5bc610b..8d580b054d0 100644
--- a/credential.h
+++ b/credential.h
@@ -140,6 +140,7 @@ struct credential {
 	char *protocol;
 	char *host;
 	char *path;
+	char *authtype;
 };
 
 #define CREDENTIAL_INIT { \
diff --git a/git-curl-compat.h b/git-curl-compat.h
index 56a83b6bbd8..839049f6dfe 100644
--- a/git-curl-compat.h
+++ b/git-curl-compat.h
@@ -126,4 +126,14 @@
 #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS
 #endif
 
+/**
+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.
+ * However, only 7.69.0 fixes a bug where Bearer headers were not
+ * actually sent with reused connections on subsequent transfers
+ * (curl/curl@dea17b519dc1).
+ */
+#if LIBCURL_VERSION_NUM >= 0x074500
+#define GIT_CURL_HAVE_CURLAUTH_BEARER
+#endif
+
 #endif
diff --git a/http.c b/http.c
index 17b47195d22..ac620bcbf0c 100644
--- a/http.c
+++ b/http.c
@@ -517,7 +517,8 @@ static int curl_empty_auth_enabled(void)
 
 static void init_curl_http_auth(struct active_request_slot *slot)
 {
-	if (!http_auth.username || !*http_auth.username) {
+	if (!http_auth.authtype &&
+		(!http_auth.username || !*http_auth.username)) {
 		if (curl_empty_auth_enabled())
 			curl_easy_setopt(slot->curl, CURLOPT_USERPWD, ":");
 		return;
@@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
 
 	credential_fill(&http_auth);
 
-	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
+	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
+				|| !strcasecmp(http_auth.authtype, "digest")) {
+		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
+			http_auth.username);
+		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
+			http_auth.password);
+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
+	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
+		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
+		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
+			http_auth.password);
+#endif
+	} else {
+		struct strbuf auth = STRBUF_INIT;
+		strbuf_addf(&auth, "Authorization: %s %s",
+			http_auth.authtype, http_auth.password);
+		slot->headers = curl_slist_append(slot->headers, auth.buf);
+		strbuf_release(&auth);
+	}
 }
 
 /* *var must be free-able */
-- 
gitgitgadget

Re: [PATCH v3 05/11] http: set specific auth scheme depending on credential

[Glen Choo]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Introduce a new credential field `authtype` that can be used by
> credential helpers to indicate the type of the credential or
> authentication mechanism to use for a request.
>
> Modify http.c to now specify the correct authentication scheme or
> credential type when authenticating the curl handle. If the new
> `authtype` field in the credential structure is `NULL` or "Basic" then
> use the existing username/password options. If the field is "Bearer"
> then use the OAuth bearer token curl option. Otherwise, the `authtype`
> field is the authentication scheme and the `password` field is the
> raw, unencoded value.
>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  Documentation/git-credential.txt | 12 ++++++++++++
>  credential.c                     |  5 +++++
>  credential.h                     |  1 +
>  git-curl-compat.h                | 10 ++++++++++
>  http.c                           | 24 +++++++++++++++++++++---
>  5 files changed, 49 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index 791a57dddfb..9069bfb2d50 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -175,6 +175,18 @@ username in the example above) will be left unset.
>  	attribute 'wwwauth[]', where the order of the attributes is the same as
>  	they appear in the HTTP response.
>  
> +`authtype`::
> +
> +	Indicates the type of authentication scheme that should be used by Git.
> +	Credential helpers may reply to a request from Git with this attribute,
> +	such that subsequent authenticated requests include the correct
> +	`Authorization` header.
> +	If this attribute is not present, the default value is "Basic".
> +	Known values include "Basic", "Digest", and "Bearer".
> +	If an unknown value is provided, this is taken as the authentication
> +	scheme for the `Authorization` header, and the `password` field is
> +	used as the raw unencoded authorization parameters of the same header.
> +

[...]

> @@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
>  
>  	credential_fill(&http_auth);
>  
> -	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
> -	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
> +	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
> +				|| !strcasecmp(http_auth.authtype, "digest")) {
> +		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
> +			http_auth.username);
> +		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
> +			http_auth.password);
> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
> +	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
> +		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
> +		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
> +			http_auth.password);
> +#endif
> +	} else {
> +		struct strbuf auth = STRBUF_INIT;
> +		strbuf_addf(&auth, "Authorization: %s %s",
> +			http_auth.authtype, http_auth.password);
> +		slot->headers = curl_slist_append(slot->headers, auth.buf);
> +		strbuf_release(&auth);
> +	}

As expected, a "Bearer" authtype doesn't require passing a username to
curl, but as you noted in the cover letter, credential helpers were
designed with username-password authentication in mind, which raises the
question of what a credential helper should do with "Bearer"
credentials.

e.g. it is not clear to me where the "username" comes from in the tests, e.g.

  +test_expect_success 'http auth www-auth headers to credential helper basic valid' '
  +	test_when_finished "per_test_cleanup" &&
  +	# base64("alice:secret-passwd")
  +	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
  +	export USERPASS64 &&
  +
  +	start_http_server \
  +		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
  +		--auth=basic:realm=\"example.com\" \
  +		--auth-token=basic:$USERPASS64 &&
  +
  +	cat >get-expected.cred <<-EOF &&
  +	protocol=http
  +	host=$HOST_PORT
  +	wwwauth[]=bearer authority="id.example.com" q=1 p=0
  +	wwwauth[]=basic realm="example.com"
  +	EOF
  +
  +	cat >store-expected.cred <<-EOF &&
  +	protocol=http
  +	host=$HOST_PORT
  +	username=alice
  +	password=secret-passwd
  +	authtype=basic
  +	EOF
  +
  +	cat >get-response.cred <<-EOF &&
  +	protocol=http
  +	host=$HOST_PORT
  +	username=alice
  +	password=secret-passwd
  +	authtype=basic
  +	EOF
  +
  +	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
  +
  +	test_cmp get-expected.cred get-actual.cred &&
  +	test_cmp store-expected.cred store-actual.cred
  +'

I'm not sure how we plan to handle this. Some approaches I can see are:

- We require that credential helpers set a reasonable value for
  "username". Presumably most credential helpers generating bearer
  tokens have some idea of user identity, so this might be reasonable,
  though it is wasteful, since we never use it in a meaningul way, e.g.
  I don't think Git asks the credential helper for "username=alice" and
  the credential helper decides to return the 'alice' credential instead
  of the 'bob' credential (but I could be mistaken).

- We require that credential helpers set _some_ value for "username",
  even if it is bogus. If so, we should communicate this explicitly.

- It is okay for "username" to be missing. This seems like the most
  elegant approach for credential helpers. I'm not sure if we're there
  yet with this series, e.g. http.c::handle_curl_result() reads:

    else if (results->http_code == 401) {
      if (http_auth.username && http_auth.password) {
        credential_reject(&http_auth);
        return HTTP_NOAUTH;

  which seems to assume both a username _and_ password. If the username
  is missing, we presumably don't send "erase", which might be a problem
  for revoked access tokens (though presumably not an issue for OIDC id
  tokens).

Re: [PATCH v3 05/11] http: set specific auth scheme depending on credential

[Matthew John Cheetham]

On 2022-11-09 15:40, Glen Choo wrote:
> "Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Introduce a new credential field `authtype` that can be used by
>> credential helpers to indicate the type of the credential or
>> authentication mechanism to use for a request.
>>
>> Modify http.c to now specify the correct authentication scheme or
>> credential type when authenticating the curl handle. If the new
>> `authtype` field in the credential structure is `NULL` or "Basic" then
>> use the existing username/password options. If the field is "Bearer"
>> then use the OAuth bearer token curl option. Otherwise, the `authtype`
>> field is the authentication scheme and the `password` field is the
>> raw, unencoded value.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  Documentation/git-credential.txt | 12 ++++++++++++
>>  credential.c                     |  5 +++++
>>  credential.h                     |  1 +
>>  git-curl-compat.h                | 10 ++++++++++
>>  http.c                           | 24 +++++++++++++++++++++---
>>  5 files changed, 49 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
>> index 791a57dddfb..9069bfb2d50 100644
>> --- a/Documentation/git-credential.txt
>> +++ b/Documentation/git-credential.txt
>> @@ -175,6 +175,18 @@ username in the example above) will be left unset.
>>  	attribute 'wwwauth[]', where the order of the attributes is the same as
>>  	they appear in the HTTP response.
>>  
>> +`authtype`::
>> +
>> +	Indicates the type of authentication scheme that should be used by Git.
>> +	Credential helpers may reply to a request from Git with this attribute,
>> +	such that subsequent authenticated requests include the correct
>> +	`Authorization` header.
>> +	If this attribute is not present, the default value is "Basic".
>> +	Known values include "Basic", "Digest", and "Bearer".
>> +	If an unknown value is provided, this is taken as the authentication
>> +	scheme for the `Authorization` header, and the `password` field is
>> +	used as the raw unencoded authorization parameters of the same header.
>> +
> 
> [...]
> 
>> @@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)
>>  
>>  	credential_fill(&http_auth);
>>  
>> -	curl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);
>> -	curl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);
>> +	if (!http_auth.authtype || !strcasecmp(http_auth.authtype, "basic")
>> +				|| !strcasecmp(http_auth.authtype, "digest")) {
>> +		curl_easy_setopt(slot->curl, CURLOPT_USERNAME,
>> +			http_auth.username);
>> +		curl_easy_setopt(slot->curl, CURLOPT_PASSWORD,
>> +			http_auth.password);
>> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER
>> +	} else if (!strcasecmp(http_auth.authtype, "bearer")) {
>> +		curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);
>> +		curl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,
>> +			http_auth.password);
>> +#endif
>> +	} else {
>> +		struct strbuf auth = STRBUF_INIT;
>> +		strbuf_addf(&auth, "Authorization: %s %s",
>> +			http_auth.authtype, http_auth.password);
>> +		slot->headers = curl_slist_append(slot->headers, auth.buf);
>> +		strbuf_release(&auth);
>> +	}
> 
> As expected, a "Bearer" authtype doesn't require passing a username to
> curl, but as you noted in the cover letter, credential helpers were
> designed with username-password authentication in mind, which raises the
> question of what a credential helper should do with "Bearer"
> credentials.
> 
> e.g. it is not clear to me where the "username" comes from in the tests, e.g.
> 
>   +test_expect_success 'http auth www-auth headers to credential helper basic valid' '
>   +	test_when_finished "per_test_cleanup" &&
>   +	# base64("alice:secret-passwd")
>   +	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
>   +	export USERPASS64 &&
>   +
>   +	start_http_server \
>   +		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
>   +		--auth=basic:realm=\"example.com\" \
>   +		--auth-token=basic:$USERPASS64 &&
>   +
>   +	cat >get-expected.cred <<-EOF &&
>   +	protocol=http
>   +	host=$HOST_PORT
>   +	wwwauth[]=bearer authority="id.example.com" q=1 p=0
>   +	wwwauth[]=basic realm="example.com"
>   +	EOF
>   +
>   +	cat >store-expected.cred <<-EOF &&
>   +	protocol=http
>   +	host=$HOST_PORT
>   +	username=alice
>   +	password=secret-passwd
>   +	authtype=basic
>   +	EOF
>   +
>   +	cat >get-response.cred <<-EOF &&
>   +	protocol=http
>   +	host=$HOST_PORT
>   +	username=alice
>   +	password=secret-passwd
>   +	authtype=basic
>   +	EOF
>   +
>   +	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
>   +
>   +	test_cmp get-expected.cred get-actual.cred &&
>   +	test_cmp store-expected.cred store-actual.cred
>   +'
> 
> I'm not sure how we plan to handle this. Some approaches I can see are:
> 
> - We require that credential helpers set a reasonable value for
>   "username". Presumably most credential helpers generating bearer
>   tokens have some idea of user identity, so this might be reasonable,
>   though it is wasteful, since we never use it in a meaningul way, e.g.
>   I don't think Git asks the credential helper for "username=alice" and
>   the credential helper decides to return the 'alice' credential instead
>   of the 'bob' credential (but I could be mistaken).
> 
> - We require that credential helpers set _some_ value for "username",
>   even if it is bogus. If so, we should communicate this explicitly.
> 
> - It is okay for "username" to be missing. This seems like the most
>   elegant approach for credential helpers. I'm not sure if we're there
>   yet with this series, e.g. http.c::handle_curl_result() reads:
> 
>     else if (results->http_code == 401) {
>       if (http_auth.username && http_auth.password) {
>         credential_reject(&http_auth);
>         return HTTP_NOAUTH;
> 
>   which seems to assume both a username _and_ password. If the username
>   is missing, we presumably don't send "erase", which might be a problem
>   for revoked access tokens (though presumably not an issue for OIDC id
>   tokens).
You are correct here that a missing username here may cause some unexpected
issues, and there should be more test coverage here.

My recent v4 iteration has actually dropped the `authtype` patches here,
and I'll pick these back up along with these concerns in a future series.
Splitting this in to a future series is probably a good idea as I feel
there's going to need to be several cleanup patches adjacent to the core
new-feature patch, so I wouldn't want to polute this series :)

Thanks!
Matthew

[PATCH v3 06/11] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.

Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                            |   2 +
 contrib/buildsystems/CMakeLists.txt |  13 +
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 685 ++++++++++++++++++++++++++++
 4 files changed, 701 insertions(+)
 create mode 100644 t/helper/test-http-server.c

diff --git a/Makefile b/Makefile
index d93ad956e58..39b130f711d 100644
--- a/Makefile
+++ b/Makefile
@@ -1500,6 +1500,8 @@ else
 	endif
 	BASIC_CFLAGS += $(CURL_CFLAGS)
 
+	TEST_PROGRAMS_NEED_X += test-http-server
+
 	REMOTE_CURL_PRIMARY = git-remote-http$X
 	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
 	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 787738e6fa3..45251695ce0 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -989,6 +989,19 @@ set(wrapper_scripts
 set(wrapper_test_scripts
 	test-fake-ssh test-tool)
 
+if(CURL_FOUND)
+       list(APPEND wrapper_test_scripts test-http-server)
+
+       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+       target_link_libraries(test-http-server common-main)
+
+       if(MSVC)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+       endif()
+endif()
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..18f1f741305
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,685 @@
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+/* Timeout, and initial timeout */
+static unsigned int timeout;
+static unsigned int init_timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+static void set_keep_alive(int sockfd)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client.  The worker talks to the client
+ * on 0 and 1.
+ */
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK       = 0,
+
+	/*
+	 * Various errors while processing the request and/or the response.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_IO_ERROR = 1<<0,
+
+	/*
+	 * Close the socket and clean up.  Does not imply an error.
+	 */
+	WR_HANGUP   = 1<<1,
+
+	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+};
+
+static enum worker_result worker(void)
+{
+	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0);
+
+	while (1) {
+		if (write_in_full(1, response, strlen(response)) < 0) {
+			logerror("unable to write response");
+			wr = WR_IO_ERROR;
+		}
+
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+	}
+
+	close(0);
+	close(1);
+
+	return !!(wr & WR_IO_ERROR);
+}
+
+/*
+ * This section contains the listener and child-process management
+ * code used by the primary instance to accept incoming connections
+ * and dispatch them to async child process "worker" instances.
+ */
+
+static int addrcmp(const struct sockaddr_storage *s1,
+		   const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+} *firstborn;
+
+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child *newborn, **cradle;
+
+	newborn = xcalloc(1, sizeof(*newborn));
+	live_children++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+/*
+ * This gets called if the number of connections grows
+ * past "max_connections".
+ *
+ * We kill the newest connection from a duplicate IP.
+ */
+static void kill_some_child(void)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+static void check_dead_children(void)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			const char *dead = "";
+			if (status)
+				dead = " (with error)";
+			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
+
+			/* remove the child */
+			*cradle = blanket->next;
+			live_children--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child();
+		sleep(1);  /* give it some time to die */
+		check_dead_children();
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	if (!reuseaddr)
+		return 0;
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd);
+
+	if (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	if (!listen_addr->nr)
+		setup_named_sock("127.0.0.1", listen_port, socklist);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children();
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances.  So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+
+	trace2_cmd_name("test-http-server");
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--init-timeout=", &v)) {
+			init_timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
-- 
gitgitgadget

Re: [PATCH v3 06/11] test-http-server: add stub HTTP server test helper

[Derrick Stolee]

On 11/2/22 6:09 PM, Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Introduce a mini HTTP server helper that in the future will be enhanced
> to provide a frontend for the git-http-backend, with support for
> arbitrary authentication schemes.
> 
> Right now, test-http-server is a pared-down copy of the git-daemon that
> always returns a 501 Not Implemented response to all callers.

Thanks for splitting this out. I ran a diff between daemon.c and
this version of t/helper/test-http-server.c. Most of the diff was
functionality removed from daemon.c, and the small bits that were
new to this file are either comments detailing how the helper
works or custom bits related to the test environment (like the
pid file). It was much easier to validate that these changes made
sense.

Looking good.

Thanks,
-Stolee

[PATCH v3 07/11] test-http-server: add HTTP error response function

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 59 +++++++++++++++++++++++++++++++++----
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 18f1f741305..53508639714 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -97,9 +97,59 @@ enum worker_result {
 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
 };
 
+static enum worker_result send_http_error(
+	int fd,
+	int http_code, const char *http_code_name,
+	int retry_after_seconds, struct string_list *response_headers,
+	enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n",
+		    http_code, http_code_name);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
 static enum worker_result worker(void)
 {
-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -110,11 +160,8 @@ static enum worker_result worker(void)
 	set_keep_alive(0);
 
 	while (1) {
-		if (write_in_full(1, response, strlen(response)) < 0) {
-			logerror("unable to write response");
-			wr = WR_IO_ERROR;
-		}
-
+		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
+			WR_OK | WR_HANGUP);
 		if (wr & WR_STOP_THE_MUSIC)
 			break;
 	}
-- 
gitgitgadget

[PATCH v3 08/11] test-http-server: add HTTP request parsing

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add ability to parse HTTP requests to the test-http-server test helper.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 176 +++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 2 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 53508639714..7bde678e264 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -97,6 +97,42 @@ enum worker_result {
 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
 };
 
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	ssize_t content_length;
+};
+
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = -1 \
+	}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
 static enum worker_result send_http_error(
 	int fd,
 	int http_code, const char *http_code_name,
@@ -148,8 +184,136 @@ done:
 	return wr;
 }
 
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_OK | WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* store common request headers separately */
+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
+			req->content_length = strtol(hv, &hp, 10);
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->content_length >= 0)
+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
+			       WR_OK | WR_HANGUP);
+}
+
 static enum worker_result worker(void)
 {
+	struct req req = REQ__INIT;
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -160,8 +324,16 @@ static enum worker_result worker(void)
 	set_keep_alive(0);
 
 	while (1) {
-		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
-			WR_OK | WR_HANGUP);
+		req__release(&req);
+
+		alarm(init_timeout ? init_timeout : timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+
+		wr = dispatch(&req);
 		if (wr & WR_STOP_THE_MUSIC)
 			break;
 	}
-- 
gitgitgadget

[PATCH v3 09/11] test-http-server: pass Git requests to http-backend

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Teach the test-http-sever test helper to forward Git requests to the
`git-http-backend`.

Introduce a new test script t5556-http-auth.sh that spins up the test
HTTP server and attempts an `ls-remote` on the served repository,
without any authentication.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c |  56 +++++++++++++++++++
 t/t5556-http-auth.sh        | 105 ++++++++++++++++++++++++++++++++++++
 2 files changed, 161 insertions(+)
 create mode 100755 t/t5556-http-auth.sh

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 7bde678e264..9f1d6b58067 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -305,8 +305,64 @@ done:
 	return result;
 }
 
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req, const char *user)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	if (write(1, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+			req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
+				req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
+				req->content_type);
+	if (req->content_length >= 0)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
+				(intmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(1);
+	close(0);
+	return !!res;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	if (is_git_request(req))
+		return do__git(req, NULL);
+
 	return send_http_error(1, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
 }
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..78da151f122
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,105 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
+REPO_DIR="$(pwd)"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$(pwd)"/pid-file.pid
+SERVER_LOG="$(pwd)"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
+per_test_cleanup () {
+	stop_http_server &&
+	rm -f OUT.*
+}
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server --allow-anonymous &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
+test_done
-- 
gitgitgadget

[PATCH v3 10/11] test-http-server: add simple authentication

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
command-line arguments. Incoming requests are compared against the set
of valid schemes and tokens and only approved if a matching token is
found, or if no auth was provided and anonymous auth is enabled.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 188 +++++++++++++++++++++++++++++++++++-
 1 file changed, 187 insertions(+), 1 deletion(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 9f1d6b58067..9a458743d13 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -18,6 +18,8 @@ static const char test_http_auth_usage[] =
 "           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
 "           [--reuseaddr] [--pid-file=<file>]\n"
 "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--anonymous-allowed]\n"
+"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
 ;
 
 /* Timeout, and initial timeout */
@@ -358,10 +360,136 @@ static enum worker_result do__git(struct req *req, const char *user)
 	return !!res;
 }
 
+enum auth_result {
+	/* No auth module matches the request. */
+	AUTH_UNKNOWN = 0,
+
+	/* Auth module denied the request. */
+	AUTH_DENY = 1,
+
+	/* Auth module successfully validated the request. */
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	char *scheme;
+	char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(const char *scheme)
+{
+	int i;
+	struct auth_module *mod;
+	for (i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme))
+			return mod;
+	}
+
+	return NULL;
+}
+
+static void add_auth_module(struct auth_module *mod)
+{
+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+	auth_modules[auth_modules_nr++] = mod;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/*
+	 * Check all auth modules and try to validate the request.
+	 * The first module that matches a valid token approves the request.
+	 * If no module is found, or if there is no valid token, then 401 error.
+	 * Otherwise, only permit the request if anonymous auth is enabled.
+	 */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (!split[0] || !split[1]) continue;
+
+			/* trim trailing space ' ' */
+			strbuf_setlen(split[0], split[0]->len - 1);
+
+			mod = get_auth_module(split[0]->buf);
+			if (mod) {
+				result = AUTH_DENY;
+
+				for_each_string_list_item(token, mod->tokens) {
+					if (!strcmp(split[1]->buf, token->string)) {
+						result = AUTH_ALLOW;
+						break;
+					}
+				}
+
+				goto done;
+			}
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (result != AUTH_DENY && allow_anonymous)
+			break;
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
+	}
+
+	strbuf_list_free(split);
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
 	if (is_git_request(req))
-		return do__git(req, NULL);
+		return do__git(req, user);
 
 	return send_http_error(1, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
@@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)
 	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
 	int worker_mode = 0;
 	int i;
+	struct auth_module *mod = NULL;
 
 	trace2_cmd_name("test-http-server");
 	setup_git_directory_gently(NULL);
@@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)
 			pid_file = v;
 			continue;
 		}
+		if (skip_prefix(arg, "--allow-anonymous", &v)) {
+			allow_anonymous = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--auth=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			/* trim trailing ':' */
+			if (p[1])
+				strbuf_setlen(p[0], p[0]->len - 1);
+
+			if (get_auth_module(p[0]->buf)) {
+				error("duplicate auth scheme '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			mod = xmalloc(sizeof(struct auth_module));
+			mod->scheme = xstrdup(p[0]->buf);
+			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
+			CALLOC_ARRAY(mod->tokens, 1);
+			string_list_init_dup(mod->tokens);
+
+			add_auth_module(mod);
+
+			strbuf_list_free(p);
+			continue;
+		}
+		if (skip_prefix(arg, "--auth-token=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			if (!p[1]) {
+				error("missing token value '%s'\n", v);
+				usage(test_http_auth_usage);
+			}
+
+			/* trim trailing ':' */
+			strbuf_setlen(p[0], p[0]->len - 1);
+
+			mod = get_auth_module(p[0]->buf);
+			if (!mod) {
+				error("auth scheme not defined '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			string_list_append(mod->tokens, p[1]->buf);
+			strbuf_list_free(p);
+			continue;
+		}
 
 		fprintf(stderr, "error: unknown argument '%s'\n", arg);
 		usage(test_http_auth_usage);
-- 
gitgitgadget

[PATCH v3 11/11] t5556: add HTTP authentication tests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a series of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers can respond
to requests that contain WWW-Authenticate information with the ability
to select the response Authenticate header scheme.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-credential-helper-replay.sh |  14 ++
 t/t5556-http-auth.sh                      | 157 +++++++++++++++++++++-
 2 files changed, 170 insertions(+), 1 deletion(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh

diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
new file mode 100755
index 00000000000..03e5e63dad6
--- /dev/null
+++ b/t/helper/test-credential-helper-replay.sh
@@ -0,0 +1,14 @@
+cmd=$1
+teefile=$cmd-actual.cred
+catfile=$cmd-response.cred
+rm -f $teefile
+while read line;
+do
+	if test -z "$line"; then
+		break;
+	fi
+	echo "$line" >> $teefile
+done
+if test "$cmd" = "get"; then
+	cat $catfile
+fi
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 78da151f122..43f1791a0fe 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -26,6 +26,8 @@ PID_FILE="$(pwd)"/pid-file.pid
 SERVER_LOG="$(pwd)"/OUT.server.log
 
 PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
+	&& export CREDENTIAL_HELPER
 
 test_expect_success 'setup repos' '
 	test_create_repo "$REPO_DIR" &&
@@ -91,7 +93,8 @@ start_http_server () {
 
 per_test_cleanup () {
 	stop_http_server &&
-	rm -f OUT.*
+	rm -f OUT.* &&
+	rm -f *.cred
 }
 
 test_expect_success 'http auth anonymous no challenge' '
@@ -102,4 +105,156 @@ test_expect_success 'http auth anonymous no challenge' '
 	git ls-remote $ORIGIN_URL
 '
 
+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=bearer:secret-token &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-token
+	authtype=bearer
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-token
+	authtype=bearer
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=basic:$USERPASS64 &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	authtype=basic
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	authtype=basic
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=foobar:alg=test\ widget=1 \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=foobar:SECRET-FOOBAR-VALUE &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=SECRET-FOOBAR-VALUE
+	authtype=foobar
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=SECRET-FOOBAR-VALUE
+	authtype=foobar
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=bearer:secret-token &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >erase-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-token
+	authtype=bearer
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-token
+	authtype=bearer
+	EOF
+
+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp erase-expected.cred erase-actual.cred
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[M Hickford]

On Wed, 2 Nov 2022 at 22:09, Matthew John Cheetham via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> `authtype`::
>
> Indicates the type of authentication scheme that should be used by Git.
> Credential helpers may reply to a request from Git with this attribute,
> such that subsequent authenticated requests include the correct
> `Authorization` header.
> If this attribute is not present, the default value is "Basic".
> Known values include "Basic", "Digest", and "Bearer".
> If an unknown value is provided, this is taken as the authentication
> scheme for the `Authorization` header, and the `password` field is
> used as the raw unencoded authorization parameters of the same header.

Do you have an example using authtype=Digest? Would the helper
populate the password field with the user's verbatim password or the
Digest challenge response? Put another way, is the Digest
challenge-response logic in Git (libcurl) or the helper?

https://www.rfc-editor.org/rfc/rfc7616#section-3.4

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham]

On 2022-11-03 12:00, M Hickford wrote:
> On Wed, 2 Nov 2022 at 22:09, Matthew John Cheetham via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> `authtype`::
>>
>> Indicates the type of authentication scheme that should be used by Git.
>> Credential helpers may reply to a request from Git with this attribute,
>> such that subsequent authenticated requests include the correct
>> `Authorization` header.
>> If this attribute is not present, the default value is "Basic".
>> Known values include "Basic", "Digest", and "Bearer".
>> If an unknown value is provided, this is taken as the authentication
>> scheme for the `Authorization` header, and the `password` field is
>> used as the raw unencoded authorization parameters of the same header.
> 
> Do you have an example using authtype=Digest? Would the helper
> populate the password field with the user's verbatim password or the
> Digest challenge response? Put another way, is the Digest
> challenge-response logic in Git (libcurl) or the helper?
> 
> https://www.rfc-editor.org/rfc/rfc7616#section-3.4
Digest should be handled by libcurl, but you've spotted that I missed
configuring libcurl here to select digest over basic for a returned
username and password.

You may have noticed I've dropped these `authtype`/response config
patches from the latest iteration (v4) as I intend to expand this part
in a separate future series. I'll be sure to specifically test and handle
digest here! Thanks for spotting :)

Thanks,
Matthew

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Derrick Stolee]

On 11/2/22 6:09 PM, Matthew John Cheetham via GitGitGadget wrote:
> Following from my original RFC submission [0], this submission is considered
> ready for full review. This patch series is now based on top of current
> master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
> separately submitted patches [1] to fix up the other credential helpers'
> behaviour.

> Updates in v3
> =============
> 
>  * Split final patch that added the test-http-server in to several, easier
>    to review patches.
> 
>  * Updated wording in git-credential.txt to clarify which side of the
>    credential helper protocol is sending/receiving the new wwwauth and
>    authtype attributes.

You also updated some commit messages based on v2 feedback. Thanks!

The commit splitting you did in this version is greatly appreciated.
I found this version to be in good shape. It's a solid foundation to
build upon (if any future work is necessary).

Thanks,
-Stolee

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Glen Choo]

Hi Matthew!

We covered this series in Review Club. As usual, participants will send
their own feedback on this thread, but you may also find the meeting
notes handy:

  https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit?pli=1#

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> Background
> ==========
>
> [...]
>
> Limitations
> ===========
>
> [...]
>
> Goals
> =====
>
> [...]
>
> Design Principles
> =================
>
> [...]

Thanks for the well-written cover letter! I suspect that not many folks
are familiar with the history and workings of credential helpers, the
current state of auth and how credential helper limitations create
challenges for auth.

I've learned a lot reading this, and it makes the motivations of this
series clear :)

> Proposed Changes
> ================
>
>  1. Teach Git to read HTTP response headers, specifically the standard
>     WWW-Authenticate (RFC 7235 Section 4.1) headers.
>
>  2. Teach Git to include extra information about HTTP responses that require
>     authentication when calling credential helpers. Specifically the
>     WWW-Authenticate header information.
>     
>     Because the extra information forms an ordered list, and the existing
>     credential helper I/O format only provides for simple key=value pairs,
>     we introduce a new convention for transmitting an ordered list of
>     values. Key names that are suffixed with a C-style array syntax should
>     have values considered to form an order list, i.e. key[]=value, where
>     the order of the key=value pairs in the stream specifies the order.
>     
>     For the WWW-Authenticate header values we opt to use the key wwwauth[].
>
>  3. Teach Git to specify authentication schemes other than Basic in
>     subsequent HTTP requests based on credential helper responses.
>

From a reading of this section + the subject line, it's not immediately
obvious that 3. also requires extending the credential helper protocol
to include the "authtype" field. IMO it's significant enough to warrant
an explicit call-out.

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham]

On 2022-11-09 15:06, Glen Choo wrote:
>> Proposed Changes
>> ================
>>
>>  1. Teach Git to read HTTP response headers, specifically the standard
>>     WWW-Authenticate (RFC 7235 Section 4.1) headers.
>>
>>  2. Teach Git to include extra information about HTTP responses that require
>>     authentication when calling credential helpers. Specifically the
>>     WWW-Authenticate header information.
>>     
>>     Because the extra information forms an ordered list, and the existing
>>     credential helper I/O format only provides for simple key=value pairs,
>>     we introduce a new convention for transmitting an ordered list of
>>     values. Key names that are suffixed with a C-style array syntax should
>>     have values considered to form an order list, i.e. key[]=value, where
>>     the order of the key=value pairs in the stream specifies the order.
>>     
>>     For the WWW-Authenticate header values we opt to use the key wwwauth[].
>>
>>  3. Teach Git to specify authentication schemes other than Basic in
>>     subsequent HTTP requests based on credential helper responses.
>>
> 
> From a reading of this section + the subject line, it's not immediately
> obvious that 3. also requires extending the credential helper protocol
> to include the "authtype" field. IMO it's significant enough to warrant
> an explicit call-out.
After some consideration I've decided to split out #3 here to a future patch
series. Parts 1 and 2 surround Git to credential helper contextual information
which is still useful in it's own right. Part 3 should really be expanded here
to better cover and explain the reverse helper-to-Git direction, whereby
helpers can modify Git's response headers to the remote.

With 1+2 most of the benefits of having an enlightened helper understand the
auth challenge, and intelligently select identities is still possible. Remotes
just need to continue to extract tokens from the basic Authorization header as
they do today until then.


Thanks,
Matthew

Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers

[Junio C Hamano]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> Testing these new additions, I introduce a new test helper test-http-server
> that acts as a frontend to git-http-backend; a mini HTTP server based
> heavily on git-daemon, with simple authentication configurable by command
> line args.

I did not try to figure out the reason but the topic with its tests
seem to break in 'seen' the linux-cmake-ctest CI job.

  https://github.com/git/git/actions/runs/3562942886/jobs/5985179202

but the same test does not break under usual "make test".

Can people who are interested in the cmake-ctest stuff take a look?

It is tempting to eject the ab/cmake-nix-and-ci topic that is
already in 'next', under the theory that what that topic does to the
tests "works" for some tests but not for others, and this topic is
an unfortunate collateral damage whose tests weren't something the
other topic did not support well.  If the cmake-ctest stuff is in
such a shape, then it may have been a bit premature to merge it
down.

Thanks.

[PATCH v4 0/8] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server based
heavily on git-daemon, with simple authentication configurable by command
line args.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.

Matthew John Cheetham (8):
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests
  test-http-server: add stub HTTP server test helper
  test-http-server: add HTTP error response function
  test-http-server: add HTTP request parsing
  test-http-server: pass Git requests to http-backend
  test-http-server: add simple authentication
  t5556: add HTTP authentication tests

 Documentation/git-credential.txt          |   18 +-
 Makefile                                  |    2 +
 contrib/buildsystems/CMakeLists.txt       |   13 +
 credential.c                              |   13 +
 credential.h                              |   15 +
 http.c                                    |   78 ++
 t/helper/.gitignore                       |    1 +
 t/helper/test-credential-helper-replay.sh |   14 +
 t/helper/test-http-server.c               | 1146 +++++++++++++++++++++
 t/t5556-http-auth.sh                      |  223 ++++
 10 files changed, 1522 insertions(+), 1 deletion(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh
 create mode 100644 t/helper/test-http-server.c
 create mode 100755 t/t5556-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v3:

  1:  f297c78f60a =  1:  b5b56ccd941 http: read HTTP WWW-Authenticate response headers
  2:  e45e23406a5 !  2:  d02875dda7c credential: add WWW-Authenticate header to cred requests
     @@ Documentation/git-credential.txt: empty string.
      +	to credential helpers.
      +	Each 'WWW-Authenticate' header value is passed as a multi-valued
      +	attribute 'wwwauth[]', where the order of the attributes is the same as
     -+	they appear in the HTTP response.
     ++	they appear in the HTTP response. This attribute is 'one-way' from Git
     ++	to pass additional information to credential helpers.
      +
     + Unrecognised attributes are silently discarded.
     + 
       GIT
     - ---
     - Part of the linkgit:git[1] suite
      
       ## credential.c ##
      @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
  3:  65ac638b8a0 <  -:  ----------- http: store all request headers on active_request_slot
  4:  4d75ca29cc5 <  -:  ----------- http: move proactive auth to first slot creation
  5:  2f38427aa8d <  -:  ----------- http: set specific auth scheme depending on credential
  6:  4947e81546a =  3:  07a1845ea56 test-http-server: add stub HTTP server test helper
  7:  93bdf1d7060 =  4:  98dd286db7c test-http-server: add HTTP error response function
  8:  b3e9156755f =  5:  5c4e36e23ee test-http-server: add HTTP request parsing
  9:  5fb248c074a =  6:  0a0f4fd10c8 test-http-server: pass Git requests to http-backend
 10:  192f09b9de4 =  7:  794256754c1 test-http-server: add simple authentication
 11:  b64d2f2c473 !  8:  8ecf6383522 t5556: add HTTP authentication tests
     @@ Commit message
          t5556: add HTTP authentication tests
      
          Add a series of tests to exercise the HTTP authentication header parsing
     -    and the interop with credential helpers. Credential helpers can respond
     -    to requests that contain WWW-Authenticate information with the ability
     -    to select the response Authenticate header scheme.
     +    and the interop with credential helpers. Credential helpers will receive
     +    WWW-Authenticate information in credential requests.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
       	git ls-remote $ORIGIN_URL
       '
       
     -+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=bearer:secret-token &&
     -+
     -+	cat >get-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+
     -+	cat >store-expected.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-token
     -+	authtype=bearer
     -+	EOF
     -+
     -+	cat >get-response.cred <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-token
     -+	authtype=bearer
     -+	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
     -+'
     -+
      +test_expect_success 'http auth www-auth headers to credential helper basic valid' '
      +	test_when_finished "per_test_cleanup" &&
      +	# base64("alice:secret-passwd")
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	export USERPASS64 &&
      +
      +	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
      +		--auth=basic:realm=\"example.com\" \
      +		--auth-token=basic:$USERPASS64 &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
      +	wwwauth[]=basic realm="example.com"
      +	EOF
      +
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
     -+	authtype=basic
      +	EOF
      +
      +	cat >get-response.cred <<-EOF &&
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
     -+	authtype=basic
      +	EOF
      +
      +	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	test_cmp store-expected.cred store-actual.cred
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '
     ++test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
      +	test_when_finished "per_test_cleanup" &&
     ++	# base64("alice:secret-passwd")
     ++	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     ++	export USERPASS64 &&
     ++
      +	start_http_server \
      +		--auth=foobar:alg=test\ widget=1 \
      +		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
      +		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=foobar:SECRET-FOOBAR-VALUE &&
     ++		--auth-token=basic:$USERPASS64 &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
     -+	password=SECRET-FOOBAR-VALUE
     -+	authtype=foobar
     ++	password=secret-passwd
      +	EOF
      +
      +	cat >get-response.cred <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
     -+	password=SECRET-FOOBAR-VALUE
     -+	authtype=foobar
     ++	password=secret-passwd
      +	EOF
      +
      +	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +
      +test_expect_success 'http auth www-auth headers to credential helper invalid' '
      +	test_when_finished "per_test_cleanup" &&
     ++	# base64("alice:secret-passwd")
     ++	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     ++	export USERPASS64 &&
      +	start_http_server \
      +		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
      +		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=bearer:secret-token &&
     ++		--auth-token=basic:$USERPASS64 &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
     -+	password=invalid-token
     -+	authtype=bearer
     ++	password=invalid-passwd
      +	wwwauth[]=bearer authority="id.example.com" q=1 p=0
      +	wwwauth[]=basic realm="example.com"
      +	EOF
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
     -+	password=invalid-token
     -+	authtype=bearer
     ++	password=invalid-passwd
      +	EOF
      +
      +	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&

-- 
gitgitgadget

[PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 ++++++++++
 http.c       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index 8a5ba3f4776..c4e9cd73e14 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	const char *z = NULL;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		const char **v = values->v + values->nr - 1;
+		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
+
+		free((void*)*v);
+		*v = append;
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (skip_iprefix(buf.buf, "http/", &z))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1940,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = eltsize * nmemb;
> +	struct strvec *values = &http_auth.wwwauth_headers;
> +	struct strbuf buf = STRBUF_INIT;
> +	const char *val;
> +	const char *z = NULL;
> +
> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	strbuf_add(&buf, ptr, size);
> +
> +	/* Strip the CRLF that should be present at the end of each field */
> +	strbuf_trim_trailing_newline(&buf);
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
> +		while (isspace(*val))
> +			val++;

Per the RFC [1]: 

> The field value MAY be preceded by any amount of LWS, though a single SP
> is preferred.

And LWS (linear whitespace) is defined as:

> CRLF           = CR LF 
> LWS            = [CRLF] 1*( SP | HT )

and 'isspace()' includes CR, LF, SP, and HT [2]. 

Looks good!

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4-2
[2] https://linux.die.net/man/3/isspace

> +
> +		strvec_push(values, val);

I had the same question about "what happens with an empty 'val' here?" as
Stolee did earlier [3], but I *think* the "zero length" (i.e., single null
terminator) will be copied successfully. It's probably worth testing that
explicitly, though (I see you set up tests in later patches - ideally a 
"www-authenticate:<mix of whitespace>" line could be tested there).

[3] https://lore.kernel.org/git/9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com/

> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> +	}
> +
> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 */
> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
> +		const char **v = values->v + values->nr - 1;
> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);

In this case (where the line is a continuation of a 'www-authenticate'
header), it looks like the code here expects *exactly* one LWS at the start
of the line ('isspace(*buf.buf)' requiring at least one space to append the
header, 'ptr + 1' skipping no more than one). But, according to the RFC, it
could be more than one:

> Header fields can be extended over multiple lines by preceding each extra
> line with at least one SP or HT.

So I think 'buf.buf' might need to have all preceding spaces removed, like
you did in the "Start of a new WWW-Authenticate header" block.

Also, if you're copying 'ptr' into 'buf' to avoid issues from a missing null
terminator, wouldn't you want to use 'buf.buf' (instead of 'ptr') in
'xstrfmt()'?

> +
> +		free((void*)*v);
> +		*v = append;

I was about to suggest (optionally) rewriting this to use 'strvec_pop()' and
'strvec_push_nodup()':

	strvec_pop(values); 
	strvec_push_nodup(values, append);

to maybe make this a bit easier to follow, but unfortunately
'strvec_push_nodup()' isn't available outside of 'strvec.c'. If you did want
to use 'strvec' functions, you could remove the 'static' from
'strvec_push_nodup()' and add it to 'strvec.h' it in a later reroll, but I
don't consider that change "blocking" or even important enough to warrant
its own reroll. 

> +
> +		goto exit;
> +	}
> +
> +	/* This is the start of a new header we don't care about */
> +	http_auth.header_is_last_match = 0;
> +
> +	/*
> +	 * If this is a HTTP status line and not a header field, this signals
> +	 * a different HTTP response. libcurl writes all the output of all
> +	 * response headers of all responses, including redirects.
> +	 * We only care about the last HTTP request response's headers so clear
> +	 * the existing array.
> +	 */
> +	if (skip_iprefix(buf.buf, "http/", &z))
> +		strvec_clear(values);

The comments describing the intended behavior (as well as the commit
message) are clear and explain the somewhat esoteric (at least to my
untrained eye ;) ) code. Thanks!

> +
> +exit:
> +	strbuf_release(&buf);
> +	return size;
> +}
> +
>  size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
>  {
>  	return nmemb;
> @@ -1864,6 +1940,8 @@ static int http_request(const char *url,
>  					 fwrite_buffer);
>  	}
>  
> +	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
> +
>  	accept_language = http_get_accept_language_header();
>  
>  	if (accept_language)

Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2022-12-14 15:15, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> +{
>> +	size_t size = eltsize * nmemb;
>> +	struct strvec *values = &http_auth.wwwauth_headers;
>> +	struct strbuf buf = STRBUF_INIT;
>> +	const char *val;
>> +	const char *z = NULL;
>> +
>> +	/*
>> +	 * Header lines may not come NULL-terminated from libcurl so we must
>> +	 * limit all scans to the maximum length of the header line, or leverage
>> +	 * strbufs for all operations.
>> +	 *
>> +	 * In addition, it is possible that header values can be split over
>> +	 * multiple lines as per RFC 2616 (even though this has since been
>> +	 * deprecated in RFC 7230). A continuation header field value is
>> +	 * identified as starting with a space or horizontal tab.
>> +	 *
>> +	 * The formal definition of a header field as given in RFC 2616 is:
>> +	 *
>> +	 *   message-header = field-name ":" [ field-value ]
>> +	 *   field-name     = token
>> +	 *   field-value    = *( field-content | LWS )
>> +	 *   field-content  = <the OCTETs making up the field-value
>> +	 *                    and consisting of either *TEXT or combinations
>> +	 *                    of token, separators, and quoted-string>
>> +	 */
>> +
>> +	strbuf_add(&buf, ptr, size);
>> +
>> +	/* Strip the CRLF that should be present at the end of each field */
>> +	strbuf_trim_trailing_newline(&buf);
>> +
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> +		while (isspace(*val))
>> +			val++;
> 
> Per the RFC [1]: 
> 
>> The field value MAY be preceded by any amount of LWS, though a single SP
>> is preferred.
> 
> And LWS (linear whitespace) is defined as:
> 
>> CRLF           = CR LF 
>> LWS            = [CRLF] 1*( SP | HT )
> 
> and 'isspace()' includes CR, LF, SP, and HT [2]. 
> 
> Looks good!
> 
> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4-2
> [2] https://linux.die.net/man/3/isspace
> 
>> +
>> +		strvec_push(values, val);
> 
> I had the same question about "what happens with an empty 'val' here?" as
> Stolee did earlier [3], but I *think* the "zero length" (i.e., single null
> terminator) will be copied successfully. It's probably worth testing that
> explicitly, though (I see you set up tests in later patches - ideally a 
> "www-authenticate:<mix of whitespace>" line could be tested there).
> 
> [3] https://lore.kernel.org/git/9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com/

There is a bug here. Empty header values would indeed be appended
successfully, but this eventually results in empty values for `wwwauth[]`
being sent over to credential helpers (which should treat the empty value as
a reset of the existing list!!)

Really, empty values should be ignored.
My next iteration should hopefully be a bit more careful around these cases.

>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
>> +	}
>> +
>> +	/*
>> +	 * This line could be a continuation of the previously matched header
>> +	 * field. If this is the case then we should append this value to the
>> +	 * end of the previously consumed value.
>> +	 */
>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> +		const char **v = values->v + values->nr - 1;
>> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
> 
> In this case (where the line is a continuation of a 'www-authenticate'
> header), it looks like the code here expects *exactly* one LWS at the start
> of the line ('isspace(*buf.buf)' requiring at least one space to append the
> header, 'ptr + 1' skipping no more than one). But, according to the RFC, it
> could be more than one:
> 
>> Header fields can be extended over multiple lines by preceding each extra
>> line with at least one SP or HT.
> 
> So I think 'buf.buf' might need to have all preceding spaces removed, like
> you did in the "Start of a new WWW-Authenticate header" block.
> 
> Also, if you're copying 'ptr' into 'buf' to avoid issues from a missing null
> terminator, wouldn't you want to use 'buf.buf' (instead of 'ptr') in
> 'xstrfmt()'?

Sure! Good points.

>> +
>> +		free((void*)*v);
>> +		*v = append;
> 
> I was about to suggest (optionally) rewriting this to use 'strvec_pop()' and
> 'strvec_push_nodup()':
> 
> 	strvec_pop(values); 
> 	strvec_push_nodup(values, append);
> 
> to maybe make this a bit easier to follow, but unfortunately
> 'strvec_push_nodup()' isn't available outside of 'strvec.c'. If you did want
> to use 'strvec' functions, you could remove the 'static' from
> 'strvec_push_nodup()' and add it to 'strvec.h' it in a later reroll, but I
> don't consider that change "blocking" or even important enough to warrant
> its own reroll. 

That wouldn't be too much effort, and would help simplify overall the move
to using `strbuf_` functions. Check my next iteration for this.

>> +
>> +		goto exit;
>> +	}
>> +
>> +	/* This is the start of a new header we don't care about */
>> +	http_auth.header_is_last_match = 0;
>> +
>> +	/*
>> +	 * If this is a HTTP status line and not a header field, this signals
>> +	 * a different HTTP response. libcurl writes all the output of all
>> +	 * response headers of all responses, including redirects.
>> +	 * We only care about the last HTTP request response's headers so clear
>> +	 * the existing array.
>> +	 */
>> +	if (skip_iprefix(buf.buf, "http/", &z))
>> +		strvec_clear(values);
> 
> The comments describing the intended behavior (as well as the commit
> message) are clear and explain the somewhat esoteric (at least to my
> untrained eye ;) ) code. Thanks!
> 
>> +
>> +exit:
>> +	strbuf_release(&buf);
>> +	return size;
>> +}
>> +
>>  size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
>>  {
>>  	return nmemb;
>> @@ -1864,6 +1940,8 @@ static int http_request(const char *url,
>>  					 fwrite_buffer);
>>  	}
>>  
>> +	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
>> +
>>  	accept_language = http_get_accept_language_header();
>>  
>>  	if (accept_language)
> 

Thanks,
Matthew

Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 12 2022, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> [...]
>  /* Initialize a credential structure, setting all fields to empty. */
> diff --git a/http.c b/http.c
> index 8a5ba3f4776..c4e9cd73e14 100644
> --- a/http.c
> +++ b/http.c
> @@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  	return nmemb;
>  }
>  
> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = eltsize * nmemb;

Just out of general paranoia: use st_mult() here, not "*" (checks for
overflows)?

> +	struct strvec *values = &http_auth.wwwauth_headers;
> +	struct strbuf buf = STRBUF_INIT;
> +	const char *val;
> +	const char *z = NULL;

Why NULL-init the "z" here, but not the "val"? Both look like they
should be un-init'd. We also tend to call a throw-away char pointer "p",
not "z", but anyway (more below).... 

> +
> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	strbuf_add(&buf, ptr, size);
> +
> +	/* Strip the CRLF that should be present at the end of each field */
> +	strbuf_trim_trailing_newline(&buf);
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
> +		while (isspace(*val))
> +			val++;

As we already have a "struct strbuf" here, maybe we can instead
consistently use the strbuf functions, e.g. strbuf_ltrim() in this case.

I haven't reviewed this in detail, maybe it's not easy or worth it
here...

> +
> +		strvec_push(values, val);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> +	}
> +
> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 */
> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
> +		const char **v = values->v + values->nr - 1;

It makes no difference to the compiler, but perhaps using []-indexing
here is more idiomatic, for getting the nth member of this strvec?

> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
> +
> +		free((void*)*v);

Is this reaching into the strvec & manually memory-managing it
unavoidable, or can we use strvec_pop() etc?

> +		*v = append;
> +
> +		goto exit;
> +	}
> +
> +	/* This is the start of a new header we don't care about */
> +	http_auth.header_is_last_match = 0;
> +
> +	/*
> +	 * If this is a HTTP status line and not a header field, this signals
> +	 * a different HTTP response. libcurl writes all the output of all
> +	 * response headers of all responses, including redirects.
> +	 * We only care about the last HTTP request response's headers so clear
> +	 * the existing array.
> +	 */
> +	if (skip_iprefix(buf.buf, "http/", &z))

...Don't you want to just skip this "z" variable altogether and use
istarts_with() instead? All you seem to care about is whether it starts
with it, not what the offset is.

Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2022-12-15 01:27, Ævar Arnfjörð Bjarmason wrote:

> 
> On Mon, Dec 12 2022, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>> [...]
>>  /* Initialize a credential structure, setting all fields to empty. */
>> diff --git a/http.c b/http.c
>> index 8a5ba3f4776..c4e9cd73e14 100644
>> --- a/http.c
>> +++ b/http.c
>> @@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>  	return nmemb;
>>  }
>>  
>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> +{
>> +	size_t size = eltsize * nmemb;
> 
> Just out of general paranoia: use st_mult() here, not "*" (checks for
> overflows)?

Sure! Good point.

>> +	struct strvec *values = &http_auth.wwwauth_headers;
>> +	struct strbuf buf = STRBUF_INIT;
>> +	const char *val;
>> +	const char *z = NULL;
> 
> Why NULL-init the "z" here, but not the "val"? Both look like they
> should be un-init'd. We also tend to call a throw-away char pointer "p",
> not "z", but anyway (more below).... 
> 
>> +
>> +	/*
>> +	 * Header lines may not come NULL-terminated from libcurl so we must
>> +	 * limit all scans to the maximum length of the header line, or leverage
>> +	 * strbufs for all operations.
>> +	 *
>> +	 * In addition, it is possible that header values can be split over
>> +	 * multiple lines as per RFC 2616 (even though this has since been
>> +	 * deprecated in RFC 7230). A continuation header field value is
>> +	 * identified as starting with a space or horizontal tab.
>> +	 *
>> +	 * The formal definition of a header field as given in RFC 2616 is:
>> +	 *
>> +	 *   message-header = field-name ":" [ field-value ]
>> +	 *   field-name     = token
>> +	 *   field-value    = *( field-content | LWS )
>> +	 *   field-content  = <the OCTETs making up the field-value
>> +	 *                    and consisting of either *TEXT or combinations
>> +	 *                    of token, separators, and quoted-string>
>> +	 */
>> +
>> +	strbuf_add(&buf, ptr, size);
>> +
>> +	/* Strip the CRLF that should be present at the end of each field */
>> +	strbuf_trim_trailing_newline(&buf);
>> +
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> +		while (isspace(*val))
>> +			val++;
> 
> As we already have a "struct strbuf" here, maybe we can instead
> consistently use the strbuf functions, e.g. strbuf_ltrim() in this case.

That's a good point. I can move to using strbuf functions entirely.

> I haven't reviewed this in detail, maybe it's not easy or worth it
> here...
> 
>> +
>> +		strvec_push(values, val);
>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
>> +	}
>> +
>> +	/*
>> +	 * This line could be a continuation of the previously matched header
>> +	 * field. If this is the case then we should append this value to the
>> +	 * end of the previously consumed value.
>> +	 */
>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> +		const char **v = values->v + values->nr - 1;
> 
> It makes no difference to the compiler, but perhaps using []-indexing
> here is more idiomatic, for getting the nth member of this strvec?

Sure!

>> +		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
>> +
>> +		free((void*)*v);
> 
> Is this reaching into the strvec & manually memory-managing it
> unavoidable, or can we use strvec_pop() etc?

Again, good point. I can rework this to pop and push a new, joined value.

>> +		*v = append;
>> +
>> +		goto exit;
>> +	}
>> +
>> +	/* This is the start of a new header we don't care about */
>> +	http_auth.header_is_last_match = 0;
>> +
>> +	/*
>> +	 * If this is a HTTP status line and not a header field, this signals
>> +	 * a different HTTP response. libcurl writes all the output of all
>> +	 * response headers of all responses, including redirects.
>> +	 * We only care about the last HTTP request response's headers so clear
>> +	 * the existing array.
>> +	 */
>> +	if (skip_iprefix(buf.buf, "http/", &z))
> 
> ...Don't you want to just skip this "z" variable altogether and use
> istarts_with() instead? All you seem to care about is whether it starts
> with it, not what the offset is.
> 

Again, a good point. Thanks for the suggestions. My next iteration will include
this.

Thanks,
Matthew

[PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt | 18 +++++++++++++++++-
 credential.c                     | 12 ++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..bf0de0e9408 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,16 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
+	Each 'WWW-Authenticate' header value is passed as a multi-valued
+	attribute 'wwwauth[]', where the order of the attributes is the same as
+	they appear in the HTTP response. This attribute is 'one-way' from Git
+	to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..8a3ad6c0ae2 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	int i = 0;
+	const char *full_key = xstrfmt("%s[]", key);
+	for (; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free((void*)full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
-- 
gitgitgadget

Re: [PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Add the value of the WWW-Authenticate response header to credential
> requests. Credential helpers that understand and support HTTP
> authentication and authorization can use this standard header (RFC 2616
> Section 14.47 [1]) to generate valid credentials.
> 
> WWW-Authenticate headers can contain information pertaining to the
> authority, authentication mechanism, or extra parameters/scopes that are
> required.
> 
> The current I/O format for credential helpers only allows for unique
> names for properties/attributes, so in order to transmit multiple header
> values (with a specific order) we introduce a new convention whereby a
> C-style array syntax is used in the property name to denote multiple
> ordered values for the same property.
> 
> In this case we send multiple `wwwauth[]` properties where the order
> that the repeated attributes appear in the conversation reflects the
> order that the WWW-Authenticate headers appeared in the HTTP response.
> 
> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

...

> +Attributes with keys that end with C-style array brackets `[]` can have
> +multiple values. Each instance of a multi-valued attribute forms an
> +ordered list of values - the order of the repeated attributes defines
> +the order of the values. An empty multi-valued attribute (`key[]=\n`)
> +acts to clear any previous entries and reset the list.
> +

The commit message & documentation changes (here and the 'www-auth[]'
definition below) are concise, easy-to-understand explanations of what
you're doing here with the 'www-authenticate' header values.

>  
> @@ -160,6 +166,16 @@ empty string.
>  Components which are missing from the URL (e.g., there is no
>  username in the example above) will be left unset.
>  
> +`wwwauth[]`::
> +
> +	When an HTTP response is received by Git that includes one or more
> +	'WWW-Authenticate' authentication headers, these will be passed by Git
> +	to credential helpers.
> +	Each 'WWW-Authenticate' header value is passed as a multi-valued
> +	attribute 'wwwauth[]', where the order of the attributes is the same as
> +	they appear in the HTTP response. This attribute is 'one-way' from Git
> +	to pass additional information to credential helpers.

nit: if you're trying to get a paragraph break between "...to credential
helpers." and "Each 'WWW-Authenticate' header value", you need to add an
explicit break:

-------- 8< --------

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index bf0de0e940..50759153ef 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -171,10 +171,11 @@ username in the example above) will be left unset.
 	When an HTTP response is received by Git that includes one or more
 	'WWW-Authenticate' authentication headers, these will be passed by Git
 	to credential helpers.
-	Each 'WWW-Authenticate' header value is passed as a multi-valued
-	attribute 'wwwauth[]', where the order of the attributes is the same as
-	they appear in the HTTP response. This attribute is 'one-way' from Git
-	to pass additional information to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
 
 Unrecognised attributes are silently discarded.
 
-------- >8 --------

You can test to see how the docs look by running 'make doc' from the
repository root and looking at the generated 'git-credential.html' (note
that, if you've installed Git dependencies with Homebrew, you might need to
specify 'XML_CATALOG_FILES=$(brew --prefix)/etc/xml/catalog' to get it to
work).

> +
>  Unrecognised attributes are silently discarded.
>  
>  GIT
> diff --git a/credential.c b/credential.c
> index 897b4679333..8a3ad6c0ae2 100644
> --- a/credential.c
> +++ b/credential.c
> @@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>  	fprintf(fp, "%s=%s\n", key, value);
>  }
>  
> +static void credential_write_strvec(FILE *fp, const char *key,
> +				    const struct strvec *vec)
> +{
> +	int i = 0;
> +	const char *full_key = xstrfmt("%s[]", key);
> +	for (; i < vec->nr; i++) {
> +		credential_write_item(fp, full_key, vec->v[i], 0);
> +	}
> +	free((void*)full_key);
> +}
> +
>  void credential_write(const struct credential *c, FILE *fp)
>  {
>  	credential_write_item(fp, "protocol", c->protocol, 1);
> @@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
>  	credential_write_item(fp, "path", c->path, 0);
>  	credential_write_item(fp, "username", c->username, 0);
>  	credential_write_item(fp, "password", c->password, 0);
> +	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);

This implementation looks good to me.

>  }
>  
>  static int run_credential_helper(struct credential *c,

Re: [PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2022-12-14 15:15, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add the value of the WWW-Authenticate response header to credential
>> requests. Credential helpers that understand and support HTTP
>> authentication and authorization can use this standard header (RFC 2616
>> Section 14.47 [1]) to generate valid credentials.
>>
>> WWW-Authenticate headers can contain information pertaining to the
>> authority, authentication mechanism, or extra parameters/scopes that are
>> required.
>>
>> The current I/O format for credential helpers only allows for unique
>> names for properties/attributes, so in order to transmit multiple header
>> values (with a specific order) we introduce a new convention whereby a
>> C-style array syntax is used in the property name to denote multiple
>> ordered values for the same property.
>>
>> In this case we send multiple `wwwauth[]` properties where the order
>> that the repeated attributes appear in the conversation reflects the
>> order that the WWW-Authenticate headers appeared in the HTTP response.
>>
>> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
> 
> ...
> 
>> +Attributes with keys that end with C-style array brackets `[]` can have
>> +multiple values. Each instance of a multi-valued attribute forms an
>> +ordered list of values - the order of the repeated attributes defines
>> +the order of the values. An empty multi-valued attribute (`key[]=\n`)
>> +acts to clear any previous entries and reset the list.
>> +
> 
> The commit message & documentation changes (here and the 'www-auth[]'
> definition below) are concise, easy-to-understand explanations of what
> you're doing here with the 'www-authenticate' header values.
> 
>>  
>> @@ -160,6 +166,16 @@ empty string.
>>  Components which are missing from the URL (e.g., there is no
>>  username in the example above) will be left unset.
>>  
>> +`wwwauth[]`::
>> +
>> +	When an HTTP response is received by Git that includes one or more
>> +	'WWW-Authenticate' authentication headers, these will be passed by Git
>> +	to credential helpers.
>> +	Each 'WWW-Authenticate' header value is passed as a multi-valued
>> +	attribute 'wwwauth[]', where the order of the attributes is the same as
>> +	they appear in the HTTP response. This attribute is 'one-way' from Git
>> +	to pass additional information to credential helpers.
> 
> nit: if you're trying to get a paragraph break between "...to credential
> helpers." and "Each 'WWW-Authenticate' header value", you need to add an
> explicit break:
> 
> -------- 8< --------
> 
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index bf0de0e940..50759153ef 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -171,10 +171,11 @@ username in the example above) will be left unset.
>  	When an HTTP response is received by Git that includes one or more
>  	'WWW-Authenticate' authentication headers, these will be passed by Git
>  	to credential helpers.
> -	Each 'WWW-Authenticate' header value is passed as a multi-valued
> -	attribute 'wwwauth[]', where the order of the attributes is the same as
> -	they appear in the HTTP response. This attribute is 'one-way' from Git
> -	to pass additional information to credential helpers.
> ++
> +Each 'WWW-Authenticate' header value is passed as a multi-valued
> +attribute 'wwwauth[]', where the order of the attributes is the same as
> +they appear in the HTTP response. This attribute is 'one-way' from Git
> +to pass additional information to credential helpers.
>  
>  Unrecognised attributes are silently discarded.
>  
> -------- >8 --------
> 
> You can test to see how the docs look by running 'make doc' from the
> repository root and looking at the generated 'git-credential.html' (note
> that, if you've installed Git dependencies with Homebrew, you might need to
> specify 'XML_CATALOG_FILES=$(brew --prefix)/etc/xml/catalog' to get it to
> work).

Thanks! Yes, I was intending there to be a line break. Thanks for the tip;
will be addressed in the next iteration.

>> +
>>  Unrecognised attributes are silently discarded.
>>  
>>  GIT
>> diff --git a/credential.c b/credential.c
>> index 897b4679333..8a3ad6c0ae2 100644
>> --- a/credential.c
>> +++ b/credential.c
>> @@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>>  	fprintf(fp, "%s=%s\n", key, value);
>>  }
>>  
>> +static void credential_write_strvec(FILE *fp, const char *key,
>> +				    const struct strvec *vec)
>> +{
>> +	int i = 0;
>> +	const char *full_key = xstrfmt("%s[]", key);
>> +	for (; i < vec->nr; i++) {
>> +		credential_write_item(fp, full_key, vec->v[i], 0);
>> +	}
>> +	free((void*)full_key);
>> +}
>> +
>>  void credential_write(const struct credential *c, FILE *fp)
>>  {
>>  	credential_write_item(fp, "protocol", c->protocol, 1);
>> @@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
>>  	credential_write_item(fp, "path", c->path, 0);
>>  	credential_write_item(fp, "username", c->username, 0);
>>  	credential_write_item(fp, "password", c->password, 0);
>> +	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
> 
> This implementation looks good to me.
> 
>>  }
>>  
>>  static int run_credential_helper(struct credential *c,
> 

Thanks,
Matthew

[PATCH v4 3/8] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.

Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                            |   2 +
 contrib/buildsystems/CMakeLists.txt |  13 +
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 685 ++++++++++++++++++++++++++++
 4 files changed, 701 insertions(+)
 create mode 100644 t/helper/test-http-server.c

diff --git a/Makefile b/Makefile
index b258fdbed86..1eb795bbfd4 100644
--- a/Makefile
+++ b/Makefile
@@ -1611,6 +1611,8 @@ else
 	endif
 	BASIC_CFLAGS += $(CURL_CFLAGS)
 
+	TEST_PROGRAMS_NEED_X += test-http-server
+
 	REMOTE_CURL_PRIMARY = git-remote-http$X
 	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
 	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..e9b9bfbb437 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -989,6 +989,19 @@ set(wrapper_scripts
 set(wrapper_test_scripts
 	test-fake-ssh test-tool)
 
+if(CURL_FOUND)
+       list(APPEND wrapper_test_scripts test-http-server)
+
+       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+       target_link_libraries(test-http-server common-main)
+
+       if(MSVC)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+               set_target_properties(test-http-server
+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+       endif()
+endif()
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..18f1f741305
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,685 @@
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+/* Timeout, and initial timeout */
+static unsigned int timeout;
+static unsigned int init_timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+static void set_keep_alive(int sockfd)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client.  The worker talks to the client
+ * on 0 and 1.
+ */
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK       = 0,
+
+	/*
+	 * Various errors while processing the request and/or the response.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_IO_ERROR = 1<<0,
+
+	/*
+	 * Close the socket and clean up.  Does not imply an error.
+	 */
+	WR_HANGUP   = 1<<1,
+
+	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+};
+
+static enum worker_result worker(void)
+{
+	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0);
+
+	while (1) {
+		if (write_in_full(1, response, strlen(response)) < 0) {
+			logerror("unable to write response");
+			wr = WR_IO_ERROR;
+		}
+
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+	}
+
+	close(0);
+	close(1);
+
+	return !!(wr & WR_IO_ERROR);
+}
+
+/*
+ * This section contains the listener and child-process management
+ * code used by the primary instance to accept incoming connections
+ * and dispatch them to async child process "worker" instances.
+ */
+
+static int addrcmp(const struct sockaddr_storage *s1,
+		   const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+} *firstborn;
+
+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child *newborn, **cradle;
+
+	newborn = xcalloc(1, sizeof(*newborn));
+	live_children++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+/*
+ * This gets called if the number of connections grows
+ * past "max_connections".
+ *
+ * We kill the newest connection from a duplicate IP.
+ */
+static void kill_some_child(void)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+static void check_dead_children(void)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			const char *dead = "";
+			if (status)
+				dead = " (with error)";
+			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
+
+			/* remove the child */
+			*cradle = blanket->next;
+			live_children--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child();
+		sleep(1);  /* give it some time to die */
+		check_dead_children();
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	if (!reuseaddr)
+		return 0;
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd);
+
+	if (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
+{
+	if (!listen_addr->nr)
+		setup_named_sock("127.0.0.1", listen_port, socklist);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children();
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances.  So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+
+	trace2_cmd_name("test-http-server");
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--init-timeout=", &v)) {
+			init_timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
-- 
gitgitgadget

Re: [PATCH v4 3/8] test-http-server: add stub HTTP server test helper

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Introduce a mini HTTP server helper that in the future will be enhanced
> to provide a frontend for the git-http-backend, with support for
> arbitrary authentication schemes.

I really like this approach, particularly because it opens up the
possibility of writing more fine-grained tests in other contexts (e.g.,
testing how a bundle-uri client handles different kinds of erroneous server
responses by intercepting and customizing those responses).

> 
> Right now, test-http-server is a pared-down copy of the git-daemon that
> always returns a 501 Not Implemented response to all callers.
> 
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  Makefile                            |   2 +
>  contrib/buildsystems/CMakeLists.txt |  13 +
>  t/helper/.gitignore                 |   1 +
>  t/helper/test-http-server.c         | 685 ++++++++++++++++++++++++++++
>  4 files changed, 701 insertions(+)
>  create mode 100644 t/helper/test-http-server.c
> 
> diff --git a/Makefile b/Makefile
> index b258fdbed86..1eb795bbfd4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1611,6 +1611,8 @@ else
>  	endif
>  	BASIC_CFLAGS += $(CURL_CFLAGS)
>  
> +	TEST_PROGRAMS_NEED_X += test-http-server

This works because all usage of 'TEST_PROGRAMS_NEED_X' are either lazily
evaluated (in the case of 'TEST_PROGRAMS') or are assigned later in the
'Makefile' than the addition here (in the case of 'test_bindir_programs'). 

On a related note, I think it would be helpful to mention 'test-http-server'
in the "=== Optional library: libcurl ===" section of the documentation at
the top of the Makefile, to clarify that it (like 'git-http-fetch' and
'git-http-push') are not built.

> +
>  	REMOTE_CURL_PRIMARY = git-remote-http$X
>  	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
>  	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
> index 2f6e0197ffa..e9b9bfbb437 100644
> --- a/contrib/buildsystems/CMakeLists.txt
> +++ b/contrib/buildsystems/CMakeLists.txt
> @@ -989,6 +989,19 @@ set(wrapper_scripts
>  set(wrapper_test_scripts
>  	test-fake-ssh test-tool)
>  
> +if(CURL_FOUND)
> +       list(APPEND wrapper_test_scripts test-http-server)
> +
> +       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
> +       target_link_libraries(test-http-server common-main)
> +
> +       if(MSVC)
> +               set_target_properties(test-http-server
> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
> +               set_target_properties(test-http-server
> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
> +       endif()
> +endif()
>  
>  foreach(script ${wrapper_scripts})
>  	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
> diff --git a/t/helper/.gitignore b/t/helper/.gitignore
> index 8c2ddcce95f..9aa9c752997 100644
> --- a/t/helper/.gitignore
> +++ b/t/helper/.gitignore
> @@ -1,2 +1,3 @@
>  /test-tool
>  /test-fake-ssh
> +/test-http-server
> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> new file mode 100644
> index 00000000000..18f1f741305
> --- /dev/null
> +++ b/t/helper/test-http-server.c

A lot of the functions in this file are modified versions of ones in
'daemon.c'. It would help reviewers/future readers to mention that in the
commit message. 

My comments are mostly going to be around the similarities/differences from
'daemon.c', hopefully to understand how 'test-http-server' is meant to be
used.

> +static void logreport(const char *label, const char *err, va_list params)
> +{
> +	struct strbuf msg = STRBUF_INIT;
> +
> +	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
> +	strbuf_vaddf(&msg, err, params);
> +	strbuf_addch(&msg, '\n');
> +
> +	fwrite(msg.buf, sizeof(char), msg.len, stderr);
> +	fflush(stderr);
> +
> +	strbuf_release(&msg);

This looks like the 'LOG_DESTINATION_STDERR' case of 'logreport()' in
'daemon.c', but adds a "label" to represent the priority. Makes sense; these
logs will be helpful to have in stderr when running tests, and the priority
will be captured as well.

> +}
> +
> +__attribute__((format (printf, 1, 2)))
> +static void logerror(const char *err, ...)
> +{
> +	va_list params;
> +	va_start(params, err);
> +	logreport("error", err, params);
> +	va_end(params);
> +}
> +
> +__attribute__((format (printf, 1, 2)))
> +static void loginfo(const char *err, ...)
> +{
> +	va_list params;
> +	if (!verbose)
> +		return;
> +	va_start(params, err);
> +	logreport("info", err, params);
> +	va_end(params);
> +}

These two functions replace the "priority" int with the "label" string, but
otherwise capture the same information.

> +
> +static void set_keep_alive(int sockfd)

This function is identical to its 'daemon.c' counterpart; its usage in
'test-http-server.c' doesn't indicate any need to differ.

> +
> +/*
> + * The code in this section is used by "worker" instances to service
> + * a single connection from a client.  The worker talks to the client
> + * on 0 and 1.
> + */
> +
> +enum worker_result {
> +	/*
> +	 * Operation successful.
> +	 * Caller *might* keep the socket open and allow keep-alive.
> +	 */
> +	WR_OK       = 0,
> +
> +	/*
> +	 * Various errors while processing the request and/or the response.
> +	 * Close the socket and clean up.
> +	 * Exit child-process with non-zero status.
> +	 */
> +	WR_IO_ERROR = 1<<0,
> +
> +	/*
> +	 * Close the socket and clean up.  Does not imply an error.
> +	 */
> +	WR_HANGUP   = 1<<1,
> +
> +	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),

As much as I love the name, I'm not sure having this value defined makes
much sense as its own "state". AFAICT, 'WR_IO_ERROR' means "error AND exit",
but 'WR_HANGUP' just means "exit", so the latter is a superset of the
former. Even if you interpret 'WR_HANGUP' as "*no* error and exit", that
makes it and 'WR_IO_ERROR' mutually exclusive, so the "combined" state
doesn't represent anything "real".

> +};
> +
> +static enum worker_result worker(void)
> +{
> +	const char *response = "HTTP/1.1 501 Not Implemented\r\n";

Here's the hardcoded 501 error, as mentioned in the commit message.

> +	char *client_addr = getenv("REMOTE_ADDR");
> +	char *client_port = getenv("REMOTE_PORT");
> +	enum worker_result wr = WR_OK;
> +
> +	if (client_addr)
> +		loginfo("Connection from %s:%s", client_addr, client_port);
> +
> +	set_keep_alive(0);
> +
> +	while (1) {
> +		if (write_in_full(1, response, strlen(response)) < 0) {
> +			logerror("unable to write response");
> +			wr = WR_IO_ERROR;
> +		}

This tries to write the response out to stdout (optional nit: you could use
'STDOUT_FILENO' instead of '1' to make this clearer), and sets 'WR_IO_ERROR'
if it fails. 

> +
> +		if (wr & WR_STOP_THE_MUSIC)
> +			break;

This will trigger if 'wr' is 'WR_HANGUP' *or* 'WR_IO_ERROR'. Is that
intentional? If it is, I think 'wr != 'WR_OK' might make that more obvious?

> +	}
> +
> +	close(0);
> +	close(1);
> +
> +	return !!(wr & WR_IO_ERROR);

Then finish by closing out 'stdin' and 'stdout', and returning '0' for "no
error", '1' for "error".

> +}
> +
> +/*
> + * This section contains the listener and child-process management
> + * code used by the primary instance to accept incoming connections
> + * and dispatch them to async child process "worker" instances.
> + */
> +
> +static int addrcmp(const struct sockaddr_storage *s1,


Identical to 'daemon.c'.

> +static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
> +{
> +	struct child *newborn, **cradle;
> +
> +	newborn = xcalloc(1, sizeof(*newborn));
> +	live_children++;
> +	memcpy(&newborn->cld, cld, sizeof(*cld));
> +	memcpy(&newborn->address, addr, addrlen);
> +	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
> +		if (!addrcmp(&(*cradle)->address, &newborn->address))
> +			break;
> +	newborn->next = *cradle;
> +	*cradle = newborn;
> +}

This is mostly the same as 'daemon.c', but uses 'xcalloc()' instead of
'CALLOC_ARRAY()'. The latter is an alias for the former, so this is fine.

> +static void kill_some_child(void)

...

> +static void check_dead_children(void)
Both of these are identical to 'daemon.c'.

> +
> +static struct strvec cld_argv = STRVEC_INIT;
> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)

This matches 'daemon.c' except for the addition of:

> +	if (cld.out < 0)
> +		logerror("could not dup() `incoming`");

The extra context provided by this message could be helpful in debugging. If
nothing else, it doesn't hurt.

> +	else if (start_command(&cld))
> +		logerror("unable to fork");
> +	else
> +		add_child(&cld, addr, addrlen);
> +}
> +
> +static void child_handler(int signo)

...

> +static int set_reuse_addr(int sockfd)

...

> +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)

...

> +#ifndef NO_IPV6
> +
> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
...

> +#else /* NO_IPV6 */
> +
> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)

All of these functions match 'daemon.c' (save for some whitespace fixups).

> +
> +static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
> +{
> +	if (!listen_addr->nr)
> +		setup_named_sock("127.0.0.1", listen_port, socklist);

This is the only difference in this function from 'daemon.c' (there, the
first arg is 'NULL', which ends up mapping to 'INADDR_ANY'). Why the change
in default?

> +	else {
> +		int i, socknum;
> +		for (i = 0; i < listen_addr->nr; i++) {
> +			socknum = setup_named_sock(listen_addr->items[i].string,
> +						   listen_port, socklist);
> +
> +			if (socknum == 0)
> +				logerror("unable to allocate any listen sockets for host %s on port %u",
> +					 listen_addr->items[i].string, listen_port);
> +		}
> +	}
> +}
> +
> +static int service_loop(struct socketlist *socklist)

This function differs from 'daemon.c' by using removal of the 'pid_file' to
force a graceful shutdown of the server.

> +{
> +	struct pollfd *pfd;
> +	int i;
> +
> +	CALLOC_ARRAY(pfd, socklist->nr);
> +
> +	for (i = 0; i < socklist->nr; i++) {
> +		pfd[i].fd = socklist->list[i];
> +		pfd[i].events = POLLIN;
> +	}
> +
> +	signal(SIGCHLD, child_handler);
> +
> +	for (;;) {
> +		int i;
> +		int nr_ready;
> +		int timeout = (pid_file ? 100 : -1);
> +
> +		check_dead_children();
> +
> +		nr_ready = poll(pfd, socklist->nr, timeout);

Setting a timeout here (if 'pid_file' is present) allows us to operate in a
mode where the removal of a 'pid_file' indicates that the server should shut
down.

> +		if (nr_ready < 0) {

'nr_ready < 0' indicates an error [1]; handle the same way as 'daemon.c'.

[1] https://man7.org/linux/man-pages/man2/poll.2.html

> +			if (errno != EINTR) {
> +				logerror("Poll failed, resuming: %s",
> +				      strerror(errno));
> +				sleep(1);
> +			}
> +			continue;
> +		}
> +		else if (nr_ready == 0) {

'nr_ready == 0' indicates a polling timeout (see [1] above)...

> +			/*
> +			 * If we have a pid_file, then we watch it.
> +			 * If someone deletes it, we shutdown the service.
> +			 * The shell scripts in the test suite will use this.
> +			 */
> +			if (!pid_file || file_exists(pid_file))
> +				continue;
> +			goto shutdown;

...and that timeout exists so that we can check whether the 'pid_file' still
exists and, if so, shut down gracefully.

> +		}
> +

Otherwise, 'nr_ready > 1', so handle the polled events.

> +		for (i = 0; i < socklist->nr; i++) {
> +			if (pfd[i].revents & POLLIN) {
> +				union {
> +					struct sockaddr sa;
> +					struct sockaddr_in sai;
> +#ifndef NO_IPV6
> +					struct sockaddr_in6 sai6;
> +#endif
> +				} ss;
> +				socklen_t sslen = sizeof(ss);
> +				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
> +				if (incoming < 0) {
> +					switch (errno) {
> +					case EAGAIN:
> +					case EINTR:
> +					case ECONNABORTED:
> +						continue;
> +					default:
> +						die_errno("accept returned");
> +					}
> +				}
> +				handle(incoming, &ss.sa, sslen);
> +			}
> +		}
> +	}
> +
> +shutdown:
> +	loginfo("Starting graceful shutdown (pid-file gone)");
> +	for (i = 0; i < socklist->nr; i++)
> +		close(socklist->list[i]);
> +
> +	return 0;

This addition logs the shutdown and closes out sockets. Looks good!

> +}
> +
> +static int serve(struct string_list *listen_addr, int listen_port)
> +{
> +	struct socketlist socklist = { NULL, 0, 0 };
> +
> +	socksetup(listen_addr, listen_port, &socklist);
> +	if (socklist.nr == 0)
> +		die("unable to allocate any listen sockets on port %u",
> +		    listen_port);
> +
> +	loginfo("Ready to rumble");

I thought this was a leftover debug printout, but it turns out that
'serve()' in 'daemon.c' has the same message. :) 

> +
> +	/*
> +	 * Wait to create the pid-file until we've setup the sockets
> +	 * and are open for business.
> +	 */
> +	if (pid_file)
> +		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
> +
> +	return service_loop(&socklist);
> +}
> +
> +/*
> + * This section is executed by both the primary instance and all
> + * worker instances.  So, yes, each child-process re-parses the
> + * command line argument and re-discovers how it should behave.
> + */
> +
> +int cmd_main(int argc, const char **argv)
> +{
> +	int listen_port = 0;
> +	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
> +	int worker_mode = 0;
> +	int i;
> +
> +	trace2_cmd_name("test-http-server");
> +	setup_git_directory_gently(NULL);

Since this isn't part of 'test-tool', it needs to do its own trace2 setup,
but it seems to be missing some of the relevant function calls. Could you
include 'trace2_cmd_list_config()' and 'trace2_cmd_list_env_vars()' as well? 

> +
> +	for (i = 1; i < argc; i++) {

Can this loop be replaced with 'parse_options()' and the appropriate 'struct
option[]'? Newer test helpers ('test-bundle-uri', 'test-cache-tree',
'test-getcwd') have been using it, and it generally seems much easier to
work with/more flexible than a custom 'if()' block (handling option
negation, interpreting both '--option=<value>' and '--option value' syntax
etc.).

That said, it looks this was mostly pulled from 'daemon.c' (which might
predate 'parse_options()'), so I'd also understand if you want to keep it as
similar to that as possible. Up to you!

> +	/* avoid splitting a message in the middle */
> +	setvbuf(stderr, NULL, _IOFBF, 4096);
> +
> +	if (listen_port == 0)
> +		listen_port = DEFAULT_GIT_PORT;
> +
> +	/*
> +	 * If no --listen=<addr> args are given, the setup_named_sock()
> +	 * code will use receive a NULL address and set INADDR_ANY.
> +	 * This exposes both internal and external interfaces on the
> +	 * port.
> +	 *
> +	 * Disallow that and default to the internal-use-only loopback
> +	 * address.
> +	 */
> +	if (!listen_addr.nr)
> +		string_list_append(&listen_addr, "127.0.0.1");
> +
> +	/*
> +	 * worker_mode is set in our own child process instances
> +	 * (that are bound to a connected socket from a client).
> +	 */
> +	if (worker_mode)
> +		return worker();
> +
> +	/*
> +	 * `cld_argv` is a bit of a clever hack. The top-level instance
> +	 * of test-http-server does the normal bind/listen/accept stuff.
> +	 * For each incoming socket, the top-level process spawns
> +	 * a child instance of test-http-server *WITH* the additional
> +	 * `--worker` argument. This causes the child to set `worker_mode`
> +	 * and immediately call `worker()` using the connected socket (and
> +	 * without the usual need for fork() or threads).
> +	 *
> +	 * The magic here is made possible because `cld_argv` is static
> +	 * and handle() (called by service_loop()) knows about it.
> +	 */
> +	strvec_push(&cld_argv, argv[0]);
> +	strvec_push(&cld_argv, "--worker");
> +	for (i = 1; i < argc; ++i)
> +		strvec_push(&cld_argv, argv[i]);
> +
> +	/*
> +	 * Setup primary instance to listen for connections.
> +	 */
> +	return serve(&listen_addr, listen_port);

The rest of the function is "new", but is well-documented and appears to
work as intended.

> +}

One last note/suggestion - while a lot of the functions in
'test-http-server.c' are modified from those in 'daemon.c', there are a fair
number of identical functions as well. Would it be possible to libify some
of 'daemon.c's functions (mainly by creating a 'daemon.h' and making the
functions non-static) so that they don't need to be copied?

Re: [PATCH v4 3/8] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham]

On 2022-12-14 15:16, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Introduce a mini HTTP server helper that in the future will be enhanced
>> to provide a frontend for the git-http-backend, with support for
>> arbitrary authentication schemes.
> 
> I really like this approach, particularly because it opens up the
> possibility of writing more fine-grained tests in other contexts (e.g.,
> testing how a bundle-uri client handles different kinds of erroneous server
> responses by intercepting and customizing those responses).

Having a mini server we can play around with makes it easier to simulate a
'bad' server, rather than use a real one like Apache and try and coerce it
in to doing 'bad' things.

>>
>> Right now, test-http-server is a pared-down copy of the git-daemon that
>> always returns a 501 Not Implemented response to all callers.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  Makefile                            |   2 +
>>  contrib/buildsystems/CMakeLists.txt |  13 +
>>  t/helper/.gitignore                 |   1 +
>>  t/helper/test-http-server.c         | 685 ++++++++++++++++++++++++++++
>>  4 files changed, 701 insertions(+)
>>  create mode 100644 t/helper/test-http-server.c
>>
>> diff --git a/Makefile b/Makefile
>> index b258fdbed86..1eb795bbfd4 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1611,6 +1611,8 @@ else
>>  	endif
>>  	BASIC_CFLAGS += $(CURL_CFLAGS)
>>  
>> +	TEST_PROGRAMS_NEED_X += test-http-server
> 
> This works because all usage of 'TEST_PROGRAMS_NEED_X' are either lazily
> evaluated (in the case of 'TEST_PROGRAMS') or are assigned later in the
> 'Makefile' than the addition here (in the case of 'test_bindir_programs'). 
> 
> On a related note, I think it would be helpful to mention 'test-http-server'
> in the "=== Optional library: libcurl ===" section of the documentation at
> the top of the Makefile, to clarify that it (like 'git-http-fetch' and
> 'git-http-push') are not built.

Upon closer inspection I noticed we don't actuall depend on libcurl here.
In my next iteration I've reworked the test helper to share some code with
daemon.c and changed where we add `test-http-server` in the Makefiles to
be the same as `test-fake-ssh`.

>> +
>>  	REMOTE_CURL_PRIMARY = git-remote-http$X
>>  	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
>>  	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
>> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
>> index 2f6e0197ffa..e9b9bfbb437 100644
>> --- a/contrib/buildsystems/CMakeLists.txt
>> +++ b/contrib/buildsystems/CMakeLists.txt
>> @@ -989,6 +989,19 @@ set(wrapper_scripts
>>  set(wrapper_test_scripts
>>  	test-fake-ssh test-tool)
>>  
>> +if(CURL_FOUND)
>> +       list(APPEND wrapper_test_scripts test-http-server)
>> +
>> +       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
>> +       target_link_libraries(test-http-server common-main)
>> +
>> +       if(MSVC)
>> +               set_target_properties(test-http-server
>> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
>> +               set_target_properties(test-http-server
>> +                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
>> +       endif()
>> +endif()
>>  
>>  foreach(script ${wrapper_scripts})
>>  	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
>> diff --git a/t/helper/.gitignore b/t/helper/.gitignore
>> index 8c2ddcce95f..9aa9c752997 100644
>> --- a/t/helper/.gitignore
>> +++ b/t/helper/.gitignore
>> @@ -1,2 +1,3 @@
>>  /test-tool
>>  /test-fake-ssh
>> +/test-http-server
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> new file mode 100644
>> index 00000000000..18f1f741305
>> --- /dev/null
>> +++ b/t/helper/test-http-server.c
> 
> A lot of the functions in this file are modified versions of ones in
> 'daemon.c'. It would help reviewers/future readers to mention that in the
> commit message. 

I appreciate the thorough effort here in understanding what those daemon.c
functions do. Hopefully the next iteration will help other reviewers as I'm
going to be extracting the identical functions to share them between daemon.c
and test-http-server.c.

> My comments are mostly going to be around the similarities/differences from
> 'daemon.c', hopefully to understand how 'test-http-server' is meant to be
> used.
> 
>> +static void logreport(const char *label, const char *err, va_list params)
>> +{
>> +	struct strbuf msg = STRBUF_INIT;
>> +
>> +	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
>> +	strbuf_vaddf(&msg, err, params);
>> +	strbuf_addch(&msg, '\n');
>> +
>> +	fwrite(msg.buf, sizeof(char), msg.len, stderr);
>> +	fflush(stderr);
>> +
>> +	strbuf_release(&msg);
> 
> This looks like the 'LOG_DESTINATION_STDERR' case of 'logreport()' in
> 'daemon.c', but adds a "label" to represent the priority. Makes sense; these
> logs will be helpful to have in stderr when running tests, and the priority
> will be captured as well.
> 
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void logerror(const char *err, ...)
>> +{
>> +	va_list params;
>> +	va_start(params, err);
>> +	logreport("error", err, params);
>> +	va_end(params);
>> +}
>> +
>> +__attribute__((format (printf, 1, 2)))
>> +static void loginfo(const char *err, ...)
>> +{
>> +	va_list params;
>> +	if (!verbose)
>> +		return;
>> +	va_start(params, err);
>> +	logreport("info", err, params);
>> +	va_end(params);
>> +}
> 
> These two functions replace the "priority" int with the "label" string, but
> otherwise capture the same information.
> 
>> +
>> +static void set_keep_alive(int sockfd)
> 
> This function is identical to its 'daemon.c' counterpart; its usage in
> 'test-http-server.c' doesn't indicate any need to differ.
> 
>> +
>> +/*
>> + * The code in this section is used by "worker" instances to service
>> + * a single connection from a client.  The worker talks to the client
>> + * on 0 and 1.
>> + */
>> +
>> +enum worker_result {
>> +	/*
>> +	 * Operation successful.
>> +	 * Caller *might* keep the socket open and allow keep-alive.
>> +	 */
>> +	WR_OK       = 0,
>> +
>> +	/*
>> +	 * Various errors while processing the request and/or the response.
>> +	 * Close the socket and clean up.
>> +	 * Exit child-process with non-zero status.
>> +	 */
>> +	WR_IO_ERROR = 1<<0,
>> +
>> +	/*
>> +	 * Close the socket and clean up.  Does not imply an error.
>> +	 */
>> +	WR_HANGUP   = 1<<1,
>> +
>> +	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
> 
> As much as I love the name, I'm not sure having this value defined makes
> much sense as its own "state". AFAICT, 'WR_IO_ERROR' means "error AND exit",
> but 'WR_HANGUP' just means "exit", so the latter is a superset of the
> former. Even if you interpret 'WR_HANGUP' as "*no* error and exit", that
> makes it and 'WR_IO_ERROR' mutually exclusive, so the "combined" state
> doesn't represent anything "real".

Fair point. Will remove this extra value in next iteration.

>> +};
>> +
>> +static enum worker_result worker(void)
>> +{
>> +	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
> 
> Here's the hardcoded 501 error, as mentioned in the commit message.
> 
>> +	char *client_addr = getenv("REMOTE_ADDR");
>> +	char *client_port = getenv("REMOTE_PORT");
>> +	enum worker_result wr = WR_OK;
>> +
>> +	if (client_addr)
>> +		loginfo("Connection from %s:%s", client_addr, client_port);
>> +
>> +	set_keep_alive(0);
>> +
>> +	while (1) {
>> +		if (write_in_full(1, response, strlen(response)) < 0) {
>> +			logerror("unable to write response");
>> +			wr = WR_IO_ERROR;
>> +		}
> 
> This tries to write the response out to stdout (optional nit: you could use
> 'STDOUT_FILENO' instead of '1' to make this clearer), and sets 'WR_IO_ERROR'
> if it fails. 

Good point; will use `STDOUT_FILENO` in all applicable places in next iteration.

>> +
>> +		if (wr & WR_STOP_THE_MUSIC)
>> +			break;
> 
> This will trigger if 'wr' is 'WR_HANGUP' *or* 'WR_IO_ERROR'. Is that
> intentional? If it is, I think 'wr != 'WR_OK' might make that more obvious?
> 
>> +	}
>> +
>> +	close(0);
>> +	close(1);
>> +
>> +	return !!(wr & WR_IO_ERROR);
> 
> Then finish by closing out 'stdin' and 'stdout', and returning '0' for "no
> error", '1' for "error".
> 
>> +}
>> +
>> +/*
>> + * This section contains the listener and child-process management
>> + * code used by the primary instance to accept incoming connections
>> + * and dispatch them to async child process "worker" instances.
>> + */
>> +
>> +static int addrcmp(const struct sockaddr_storage *s1,
> 
> 
> Identical to 'daemon.c'.
> 
>> +static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
>> +{
>> +	struct child *newborn, **cradle;
>> +
>> +	newborn = xcalloc(1, sizeof(*newborn));
>> +	live_children++;
>> +	memcpy(&newborn->cld, cld, sizeof(*cld));
>> +	memcpy(&newborn->address, addr, addrlen);
>> +	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
>> +		if (!addrcmp(&(*cradle)->address, &newborn->address))
>> +			break;
>> +	newborn->next = *cradle;
>> +	*cradle = newborn;
>> +}
> 
> This is mostly the same as 'daemon.c', but uses 'xcalloc()' instead of
> 'CALLOC_ARRAY()'. The latter is an alias for the former, so this is fine.
> 
>> +static void kill_some_child(void)
> 
> ...
> 
>> +static void check_dead_children(void)
> Both of these are identical to 'daemon.c'.
> 
>> +
>> +static struct strvec cld_argv = STRVEC_INIT;
>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
> 
> This matches 'daemon.c' except for the addition of:
> 
>> +	if (cld.out < 0)
>> +		logerror("could not dup() `incoming`");
> 
> The extra context provided by this message could be helpful in debugging. If
> nothing else, it doesn't hurt.
> 
>> +	else if (start_command(&cld))
>> +		logerror("unable to fork");
>> +	else
>> +		add_child(&cld, addr, addrlen);
>> +}
>> +
>> +static void child_handler(int signo)
> 
> ...
> 
>> +static int set_reuse_addr(int sockfd)
> 
> ...
> 
>> +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
> 
> ...
> 
>> +#ifndef NO_IPV6
>> +
>> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
> ...
> 
>> +#else /* NO_IPV6 */
>> +
>> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
> 
> All of these functions match 'daemon.c' (save for some whitespace fixups).
> 
>> +
>> +static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
>> +{
>> +	if (!listen_addr->nr)
>> +		setup_named_sock("127.0.0.1", listen_port, socklist);
> 
> This is the only difference in this function from 'daemon.c' (there, the
> first arg is 'NULL', which ends up mapping to 'INADDR_ANY'). Why the change
> in default?

Next iteration will share implementation with daemon.c.

>> +	else {
>> +		int i, socknum;
>> +		for (i = 0; i < listen_addr->nr; i++) {
>> +			socknum = setup_named_sock(listen_addr->items[i].string,
>> +						   listen_port, socklist);
>> +
>> +			if (socknum == 0)
>> +				logerror("unable to allocate any listen sockets for host %s on port %u",
>> +					 listen_addr->items[i].string, listen_port);
>> +		}
>> +	}
>> +}
>> +
>> +static int service_loop(struct socketlist *socklist)
> 
> This function differs from 'daemon.c' by using removal of the 'pid_file' to
> force a graceful shutdown of the server.
> 
>> +{
>> +	struct pollfd *pfd;
>> +	int i;
>> +
>> +	CALLOC_ARRAY(pfd, socklist->nr);
>> +
>> +	for (i = 0; i < socklist->nr; i++) {
>> +		pfd[i].fd = socklist->list[i];
>> +		pfd[i].events = POLLIN;
>> +	}
>> +
>> +	signal(SIGCHLD, child_handler);
>> +
>> +	for (;;) {
>> +		int i;
>> +		int nr_ready;
>> +		int timeout = (pid_file ? 100 : -1);
>> +
>> +		check_dead_children();
>> +
>> +		nr_ready = poll(pfd, socklist->nr, timeout);
> 
> Setting a timeout here (if 'pid_file' is present) allows us to operate in a
> mode where the removal of a 'pid_file' indicates that the server should shut
> down.
> 
>> +		if (nr_ready < 0) {
> 
> 'nr_ready < 0' indicates an error [1]; handle the same way as 'daemon.c'.
> 
> [1] https://man7.org/linux/man-pages/man2/poll.2.html
> 
>> +			if (errno != EINTR) {
>> +				logerror("Poll failed, resuming: %s",
>> +				      strerror(errno));
>> +				sleep(1);
>> +			}
>> +			continue;
>> +		}
>> +		else if (nr_ready == 0) {
> 
> 'nr_ready == 0' indicates a polling timeout (see [1] above)...
> 
>> +			/*
>> +			 * If we have a pid_file, then we watch it.
>> +			 * If someone deletes it, we shutdown the service.
>> +			 * The shell scripts in the test suite will use this.
>> +			 */
>> +			if (!pid_file || file_exists(pid_file))
>> +				continue;
>> +			goto shutdown;
> 
> ...and that timeout exists so that we can check whether the 'pid_file' still
> exists and, if so, shut down gracefully.
> 
>> +		}
>> +
> 
> Otherwise, 'nr_ready > 1', so handle the polled events.
> 
>> +		for (i = 0; i < socklist->nr; i++) {
>> +			if (pfd[i].revents & POLLIN) {
>> +				union {
>> +					struct sockaddr sa;
>> +					struct sockaddr_in sai;
>> +#ifndef NO_IPV6
>> +					struct sockaddr_in6 sai6;
>> +#endif
>> +				} ss;
>> +				socklen_t sslen = sizeof(ss);
>> +				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
>> +				if (incoming < 0) {
>> +					switch (errno) {
>> +					case EAGAIN:
>> +					case EINTR:
>> +					case ECONNABORTED:
>> +						continue;
>> +					default:
>> +						die_errno("accept returned");
>> +					}
>> +				}
>> +				handle(incoming, &ss.sa, sslen);
>> +			}
>> +		}
>> +	}
>> +
>> +shutdown:
>> +	loginfo("Starting graceful shutdown (pid-file gone)");
>> +	for (i = 0; i < socklist->nr; i++)
>> +		close(socklist->list[i]);
>> +
>> +	return 0;
> 
> This addition logs the shutdown and closes out sockets. Looks good!
> 
>> +}
>> +
>> +static int serve(struct string_list *listen_addr, int listen_port)
>> +{
>> +	struct socketlist socklist = { NULL, 0, 0 };
>> +
>> +	socksetup(listen_addr, listen_port, &socklist);
>> +	if (socklist.nr == 0)
>> +		die("unable to allocate any listen sockets on port %u",
>> +		    listen_port);
>> +
>> +	loginfo("Ready to rumble");
> 
> I thought this was a leftover debug printout, but it turns out that
> 'serve()' in 'daemon.c' has the same message. :) 

Indeed! This made me chuckle when I first saw it..

>> +
>> +	/*
>> +	 * Wait to create the pid-file until we've setup the sockets
>> +	 * and are open for business.
>> +	 */
>> +	if (pid_file)
>> +		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
>> +
>> +	return service_loop(&socklist);
>> +}
>> +
>> +/*
>> + * This section is executed by both the primary instance and all
>> + * worker instances.  So, yes, each child-process re-parses the
>> + * command line argument and re-discovers how it should behave.
>> + */
>> +
>> +int cmd_main(int argc, const char **argv)
>> +{
>> +	int listen_port = 0;
>> +	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
>> +	int worker_mode = 0;
>> +	int i;
>> +
>> +	trace2_cmd_name("test-http-server");
>> +	setup_git_directory_gently(NULL);
> 
> Since this isn't part of 'test-tool', it needs to do its own trace2 setup,
> but it seems to be missing some of the relevant function calls. Could you
> include 'trace2_cmd_list_config()' and 'trace2_cmd_list_env_vars()' as well? 

Sure!

>> +
>> +	for (i = 1; i < argc; i++) {
> 
> Can this loop be replaced with 'parse_options()' and the appropriate 'struct
> option[]'? Newer test helpers ('test-bundle-uri', 'test-cache-tree',
> 'test-getcwd') have been using it, and it generally seems much easier to
> work with/more flexible than a custom 'if()' block (handling option
> negation, interpreting both '--option=<value>' and '--option value' syntax
> etc.).
> 
> That said, it looks this was mostly pulled from 'daemon.c' (which might
> predate 'parse_options()'), so I'd also understand if you want to keep it as
> similar to that as possible. Up to you!

For now I think I'll keep it the same style as daemon.c.

>> +	/* avoid splitting a message in the middle */
>> +	setvbuf(stderr, NULL, _IOFBF, 4096);
>> +
>> +	if (listen_port == 0)
>> +		listen_port = DEFAULT_GIT_PORT;
>> +
>> +	/*
>> +	 * If no --listen=<addr> args are given, the setup_named_sock()
>> +	 * code will use receive a NULL address and set INADDR_ANY.
>> +	 * This exposes both internal and external interfaces on the
>> +	 * port.
>> +	 *
>> +	 * Disallow that and default to the internal-use-only loopback
>> +	 * address.
>> +	 */
>> +	if (!listen_addr.nr)
>> +		string_list_append(&listen_addr, "127.0.0.1");
>> +
>> +	/*
>> +	 * worker_mode is set in our own child process instances
>> +	 * (that are bound to a connected socket from a client).
>> +	 */
>> +	if (worker_mode)
>> +		return worker();
>> +
>> +	/*
>> +	 * `cld_argv` is a bit of a clever hack. The top-level instance
>> +	 * of test-http-server does the normal bind/listen/accept stuff.
>> +	 * For each incoming socket, the top-level process spawns
>> +	 * a child instance of test-http-server *WITH* the additional
>> +	 * `--worker` argument. This causes the child to set `worker_mode`
>> +	 * and immediately call `worker()` using the connected socket (and
>> +	 * without the usual need for fork() or threads).
>> +	 *
>> +	 * The magic here is made possible because `cld_argv` is static
>> +	 * and handle() (called by service_loop()) knows about it.
>> +	 */
>> +	strvec_push(&cld_argv, argv[0]);
>> +	strvec_push(&cld_argv, "--worker");
>> +	for (i = 1; i < argc; ++i)
>> +		strvec_push(&cld_argv, argv[i]);
>> +
>> +	/*
>> +	 * Setup primary instance to listen for connections.
>> +	 */
>> +	return serve(&listen_addr, listen_port);
> 
> The rest of the function is "new", but is well-documented and appears to
> work as intended.
> 
>> +}
> 
> One last note/suggestion - while a lot of the functions in
> 'test-http-server.c' are modified from those in 'daemon.c', there are a fair
> number of identical functions as well. Would it be possible to libify some
> of 'daemon.c's functions (mainly by creating a 'daemon.h' and making the
> functions non-static) so that they don't need to be copied?
> 

Watch for my next iteration for this!

Thanks,
Matthew

[PATCH v4 4/8] test-http-server: add HTTP error response function

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 59 +++++++++++++++++++++++++++++++++----
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 18f1f741305..53508639714 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -97,9 +97,59 @@ enum worker_result {
 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
 };
 
+static enum worker_result send_http_error(
+	int fd,
+	int http_code, const char *http_code_name,
+	int retry_after_seconds, struct string_list *response_headers,
+	enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n",
+		    http_code, http_code_name);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
 static enum worker_result worker(void)
 {
-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -110,11 +160,8 @@ static enum worker_result worker(void)
 	set_keep_alive(0);
 
 	while (1) {
-		if (write_in_full(1, response, strlen(response)) < 0) {
-			logerror("unable to write response");
-			wr = WR_IO_ERROR;
-		}
-
+		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
+			WR_OK | WR_HANGUP);
 		if (wr & WR_STOP_THE_MUSIC)
 			break;
 	}
-- 
gitgitgadget

Re: [PATCH v4 4/8] test-http-server: add HTTP error response function

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> +static enum worker_result send_http_error(
> +	int fd,
> +	int http_code, const char *http_code_name,
> +	int retry_after_seconds, struct string_list *response_headers,
> +	enum worker_result wr_in)
> +{
> +	struct strbuf response_header = STRBUF_INIT;
> +	struct strbuf response_content = STRBUF_INIT;
> +	struct string_list_item *h;
> +	enum worker_result wr;
> +
> +	strbuf_addf(&response_content, "Error: %d %s\r\n",
> +		    http_code, http_code_name);
> +	if (retry_after_seconds > 0)
> +		strbuf_addf(&response_content, "Retry-After: %d\r\n",
> +			    retry_after_seconds);
> +
> +	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
> +	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
> +	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
> +	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
> +	if (retry_after_seconds > 0)
> +		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
> +	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
> +	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
> +	if (response_headers)
> +		for_each_string_list_item(h, response_headers)
> +			strbuf_addf(&response_header, "%s\r\n", h->string);
> +	strbuf_addstr(&response_header, "\r\n");
> +
> +	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
> +		logerror("unable to write response header");
> +		wr = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
> +		logerror("unable to write response content body");
> +		wr = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	wr = wr_in;

By setting this here, if there's a 'goto done' added sometime in the future
that doesn't explicitly set 'wr' first, it'll trigger a compiler error.
That's good for a case like this, where we don't want to assume a "default"
for 'wr' before handling it.

> +
> +done:
> +	strbuf_release(&response_header);
> +	strbuf_release(&response_content);
> +
> +	return wr;
> +}
> +
>  static enum worker_result worker(void)
>  {
> -	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
>  	char *client_addr = getenv("REMOTE_ADDR");
>  	char *client_port = getenv("REMOTE_PORT");
>  	enum worker_result wr = WR_OK;
> @@ -110,11 +160,8 @@ static enum worker_result worker(void)
>  	set_keep_alive(0);
>  
>  	while (1) {
> -		if (write_in_full(1, response, strlen(response)) < 0) {
> -			logerror("unable to write response");
> -			wr = WR_IO_ERROR;
> -		}
> -
> +		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
> +			WR_OK | WR_HANGUP);

This is a nice incremental improvement on the original hardcoded response.

>  		if (wr & WR_STOP_THE_MUSIC)
>  			break;
>  	}

[PATCH v4 5/8] test-http-server: add HTTP request parsing

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add ability to parse HTTP requests to the test-http-server test helper.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 176 +++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 2 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 53508639714..7bde678e264 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -97,6 +97,42 @@ enum worker_result {
 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
 };
 
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	ssize_t content_length;
+};
+
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = -1 \
+	}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
 static enum worker_result send_http_error(
 	int fd,
 	int http_code, const char *http_code_name,
@@ -148,8 +184,136 @@ done:
 	return wr;
 }
 
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_OK | WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* store common request headers separately */
+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
+			req->content_length = strtol(hv, &hp, 10);
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->content_length >= 0)
+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
+			       WR_OK | WR_HANGUP);
+}
+
 static enum worker_result worker(void)
 {
+	struct req req = REQ__INIT;
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -160,8 +324,16 @@ static enum worker_result worker(void)
 	set_keep_alive(0);
 
 	while (1) {
-		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
-			WR_OK | WR_HANGUP);
+		req__release(&req);
+
+		alarm(init_timeout ? init_timeout : timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr & WR_STOP_THE_MUSIC)
+			break;
+
+		wr = dispatch(&req);
 		if (wr & WR_STOP_THE_MUSIC)
 			break;
 	}
-- 
gitgitgadget

Re: [PATCH v4 5/8] test-http-server: add HTTP request parsing

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> +/*
> + * Read the HTTP request up to the start of the optional message-body.
> + * We do this byte-by-byte because we have keep-alive turned on and
> + * cannot rely on an EOF.
> + *
> + * https://tools.ietf.org/html/rfc7230
> + *
> + * We cannot call die() here because our caller needs to properly
> + * respond to the client and/or close the socket before this
> + * child exits so that the client doesn't get a connection reset
> + * by peer error.
> + */
> +static enum worker_result req__read(struct req *req, int fd)
> +{
> +	struct strbuf h = STRBUF_INIT;
> +	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
> +	int nr_start_line_fields;
> +	const char *uri_target;
> +	const char *query;
> +	char *hp;
> +	const char *hv;
> +
> +	enum worker_result result = WR_OK;
> +
> +	/*
> +	 * Read line 0 of the request and split it into component parts:
> +	 *
> +	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
> +	 *
> +	 */
> +	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
> +		result = WR_OK | WR_HANGUP;
> +		goto done;
> +	}
> +
> +	strbuf_trim_trailing_newline(&req->start_line);
> +
> +	nr_start_line_fields = string_list_split(&start_line_fields,
> +						 req->start_line.buf,
> +						 ' ', -1);
> +	if (nr_start_line_fields != 3) {
> +		logerror("could not parse request start-line '%s'",
> +			 req->start_line.buf);
> +		result = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	req->method = xstrdup(start_line_fields.items[0].string);
> +	req->http_version = xstrdup(start_line_fields.items[2].string);
> +
> +	uri_target = start_line_fields.items[1].string;
> +
> +	if (strcmp(req->http_version, "HTTP/1.1")) {
> +		logerror("unsupported version '%s' (expecting HTTP/1.1)",
> +			 req->http_version);
> +		result = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	query = strchr(uri_target, '?');
> +
> +	if (query) {
> +		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
> +		strbuf_addstr(&req->query_args, query + 1);
> +	} else {
> +		strbuf_addstr(&req->uri_path, uri_target);
> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
> +	}

This "line 0" parsing looks good, and aligns with the RFC you linked
(specifically section 3.1.1 [1]).

[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1

> +
> +	/*
> +	 * Read the set of HTTP headers into a string-list.
> +	 */
> +	while (1) {
> +		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
> +			goto done;
> +		strbuf_trim_trailing_newline(&h);
> +
> +		if (!h.len)
> +			goto done; /* a blank line ends the header */
> +
> +		hp = strbuf_detach(&h, NULL);
> +		string_list_append(&req->header_list, hp);
> +
> +		/* store common request headers separately */
> +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
> +			req->content_type = hv;
> +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
> +			req->content_length = strtol(hv, &hp, 10);
> +		}

The "separately" is somewhat confusing - you unconditionally add 'hp' to
'req->header_list', so the "Content-Type" and "Content-Length" headers are
included there as well. If that's the desired behavior, a comment like "Also
store common headers as 'req' fields" might be clearer.

> +	}
> +
> +	/*
> +	 * We do not attempt to read the <message-body>, if it exists.
> +	 * We let our caller read/chunk it in as appropriate.
> +	 */
> +
> +done:
> +	string_list_clear(&start_line_fields, 0);
> +
> +	/*
> +	 * This is useful for debugging the request, but very noisy.
> +	 */
> +	if (trace2_is_enabled()) {

'trace2_printf()' is gated internally by 'trace2_enabled' anyway, so I don't
think this 'if()' is necessary. You could add a 'DEBUG_HTTP_SERVER'
preprocessor directive (like 'DEBUG_CACHE_TREE' in 'cache-tree.c') if you
wanted to prevent these printouts unless a developer sets it to '1'.

> +		struct string_list_item *item;
> +		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
> +		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
> +		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
> +		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
> +		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
> +		if (req->content_length >= 0)
> +			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
> +		if (req->content_type)
> +			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
> +		for_each_string_list_item(item, &req->header_list)
> +			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
> +	}
> +
> +	return result;
> +}
> +
> +static enum worker_result dispatch(struct req *req)
> +{
> +	return send_http_error(1, 501, "Not Implemented", -1, NULL,
> +			       WR_OK | WR_HANGUP);

Although the request is now being read & parsed, the response creation code
is still a hardcoded "Not Implemented". This means that the now-parsed 'req'
is be temporarily unused, but I think that's reasonable (since it allows for
breaking up the implementation of 'test-http-server' into multiple, less
overwhelming patches).

> +}
> +
>  static enum worker_result worker(void)
>  {
> +	struct req req = REQ__INIT;
>  	char *client_addr = getenv("REMOTE_ADDR");
>  	char *client_port = getenv("REMOTE_PORT");
>  	enum worker_result wr = WR_OK;
> @@ -160,8 +324,16 @@ static enum worker_result worker(void)
>  	set_keep_alive(0);
>  
>  	while (1) {
> -		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
> -			WR_OK | WR_HANGUP);
> +		req__release(&req);
> +
> +		alarm(init_timeout ? init_timeout : timeout);
> +		wr = req__read(&req, 0);
> +		alarm(0);

I know 'init_timeout' and 'timeout' were pulled from 'daemon.c', but what's
the difference between them/why do they both exist? It looks like
'init_timeout' just acts as a permanent override to the value of 'timeout'.

> +
> +		if (wr & WR_STOP_THE_MUSIC)
> +			break;
> +
> +		wr = dispatch(&req);
>  		if (wr & WR_STOP_THE_MUSIC)
>  			break;
>  	}

Re: [PATCH v4 5/8] test-http-server: add HTTP request parsing

[Matthew John Cheetham]

On 2022-12-14 15:18, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> +/*
>> + * Read the HTTP request up to the start of the optional message-body.
>> + * We do this byte-by-byte because we have keep-alive turned on and
>> + * cannot rely on an EOF.
>> + *
>> + * https://tools.ietf.org/html/rfc7230
>> + *
>> + * We cannot call die() here because our caller needs to properly
>> + * respond to the client and/or close the socket before this
>> + * child exits so that the client doesn't get a connection reset
>> + * by peer error.
>> + */
>> +static enum worker_result req__read(struct req *req, int fd)
>> +{
>> +	struct strbuf h = STRBUF_INIT;
>> +	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
>> +	int nr_start_line_fields;
>> +	const char *uri_target;
>> +	const char *query;
>> +	char *hp;
>> +	const char *hv;
>> +
>> +	enum worker_result result = WR_OK;
>> +
>> +	/*
>> +	 * Read line 0 of the request and split it into component parts:
>> +	 *
>> +	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
>> +	 *
>> +	 */
>> +	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
>> +		result = WR_OK | WR_HANGUP;
>> +		goto done;
>> +	}
>> +
>> +	strbuf_trim_trailing_newline(&req->start_line);
>> +
>> +	nr_start_line_fields = string_list_split(&start_line_fields,
>> +						 req->start_line.buf,
>> +						 ' ', -1);
>> +	if (nr_start_line_fields != 3) {
>> +		logerror("could not parse request start-line '%s'",
>> +			 req->start_line.buf);
>> +		result = WR_IO_ERROR;
>> +		goto done;
>> +	}
>> +
>> +	req->method = xstrdup(start_line_fields.items[0].string);
>> +	req->http_version = xstrdup(start_line_fields.items[2].string);
>> +
>> +	uri_target = start_line_fields.items[1].string;
>> +
>> +	if (strcmp(req->http_version, "HTTP/1.1")) {
>> +		logerror("unsupported version '%s' (expecting HTTP/1.1)",
>> +			 req->http_version);
>> +		result = WR_IO_ERROR;
>> +		goto done;
>> +	}
>> +
>> +	query = strchr(uri_target, '?');
>> +
>> +	if (query) {
>> +		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
>> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
>> +		strbuf_addstr(&req->query_args, query + 1);
>> +	} else {
>> +		strbuf_addstr(&req->uri_path, uri_target);
>> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
>> +	}
> 
> This "line 0" parsing looks good, and aligns with the RFC you linked
> (specifically section 3.1.1 [1]).
> 
> [1] https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1
> 
>> +
>> +	/*
>> +	 * Read the set of HTTP headers into a string-list.
>> +	 */
>> +	while (1) {
>> +		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
>> +			goto done;
>> +		strbuf_trim_trailing_newline(&h);
>> +
>> +		if (!h.len)
>> +			goto done; /* a blank line ends the header */
>> +
>> +		hp = strbuf_detach(&h, NULL);
>> +		string_list_append(&req->header_list, hp);
>> +
>> +		/* store common request headers separately */
>> +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
>> +			req->content_type = hv;
>> +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
>> +			req->content_length = strtol(hv, &hp, 10);
>> +		}
> 
> The "separately" is somewhat confusing - you unconditionally add 'hp' to
> 'req->header_list', so the "Content-Type" and "Content-Length" headers are
> included there as well. If that's the desired behavior, a comment like "Also
> store common headers as 'req' fields" might be clearer.

Will clarify this comment in next roll. You are correct, we *also* store these
common headers on `struct req`.

>> +	}
>> +
>> +	/*
>> +	 * We do not attempt to read the <message-body>, if it exists.
>> +	 * We let our caller read/chunk it in as appropriate.
>> +	 */
>> +
>> +done:
>> +	string_list_clear(&start_line_fields, 0);
>> +
>> +	/*
>> +	 * This is useful for debugging the request, but very noisy.
>> +	 */
>> +	if (trace2_is_enabled()) {
> 
> 'trace2_printf()' is gated internally by 'trace2_enabled' anyway, so I don't
> think this 'if()' is necessary. You could add a 'DEBUG_HTTP_SERVER'
> preprocessor directive (like 'DEBUG_CACHE_TREE' in 'cache-tree.c') if you
> wanted to prevent these printouts unless a developer sets it to '1'.

The overarching `trace2_is_enabled()` call is to avoid any possible repeated
evaluation within `trace2_printf` for potentially multiple request headers.

>> +		struct string_list_item *item;
>> +		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
>> +		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
>> +		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
>> +		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
>> +		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
>> +		if (req->content_length >= 0)
>> +			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
>> +		if (req->content_type)
>> +			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
>> +		for_each_string_list_item(item, &req->header_list)
>> +			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
>> +	}
>> +
>> +	return result;
>> +}
>> +
>> +static enum worker_result dispatch(struct req *req)
>> +{
>> +	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>> +			       WR_OK | WR_HANGUP);
> 
> Although the request is now being read & parsed, the response creation code
> is still a hardcoded "Not Implemented". This means that the now-parsed 'req'
> is be temporarily unused, but I think that's reasonable (since it allows for
> breaking up the implementation of 'test-http-server' into multiple, less
> overwhelming patches).
> 
>> +}
>> +
>>  static enum worker_result worker(void)
>>  {
>> +	struct req req = REQ__INIT;
>>  	char *client_addr = getenv("REMOTE_ADDR");
>>  	char *client_port = getenv("REMOTE_PORT");
>>  	enum worker_result wr = WR_OK;
>> @@ -160,8 +324,16 @@ static enum worker_result worker(void)
>>  	set_keep_alive(0);
>>  
>>  	while (1) {
>> -		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
>> -			WR_OK | WR_HANGUP);
>> +		req__release(&req);
>> +
>> +		alarm(init_timeout ? init_timeout : timeout);
>> +		wr = req__read(&req, 0);
>> +		alarm(0);
> 
> I know 'init_timeout' and 'timeout' were pulled from 'daemon.c', but what's
> the difference between them/why do they both exist? It looks like
> 'init_timeout' just acts as a permanent override to the value of 'timeout'.

Good catch. This split made sense in daemon.c whereby the `--timeout` are would
be passed to the `git-upload-pack` command, and `--init-timeout` is used as the
timeout value for the daemon server itself.

In the test HTTP server we don't need the differentiation so I'll just use the
simpler `--timeout` arg.

>> +
>> +		if (wr & WR_STOP_THE_MUSIC)
>> +			break;
>> +
>> +		wr = dispatch(&req);
>>  		if (wr & WR_STOP_THE_MUSIC)
>>  			break;
>>  	}
> 

Thanks,
Matthew

[PATCH v4 6/8] test-http-server: pass Git requests to http-backend

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Teach the test-http-sever test helper to forward Git requests to the
`git-http-backend`.

Introduce a new test script t5556-http-auth.sh that spins up the test
HTTP server and attempts an `ls-remote` on the served repository,
without any authentication.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c |  56 +++++++++++++++++++
 t/t5556-http-auth.sh        | 105 ++++++++++++++++++++++++++++++++++++
 2 files changed, 161 insertions(+)
 create mode 100755 t/t5556-http-auth.sh

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 7bde678e264..9f1d6b58067 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -305,8 +305,64 @@ done:
 	return result;
 }
 
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req, const char *user)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	if (write(1, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+			req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
+				req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
+				req->content_type);
+	if (req->content_length >= 0)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
+				(intmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(1);
+	close(0);
+	return !!res;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	if (is_git_request(req))
+		return do__git(req, NULL);
+
 	return send_http_error(1, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
 }
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..78da151f122
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,105 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
+REPO_DIR="$(pwd)"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$(pwd)"/pid-file.pid
+SERVER_LOG="$(pwd)"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
+per_test_cleanup () {
+	stop_http_server &&
+	rm -f OUT.*
+}
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server --allow-anonymous &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Teach the test-http-sever test helper to forward Git requests to the
> `git-http-backend`.
> 
> Introduce a new test script t5556-http-auth.sh that spins up the test
> HTTP server and attempts an `ls-remote` on the served repository,
> without any authentication.
> 
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  t/helper/test-http-server.c |  56 +++++++++++++++++++
>  t/t5556-http-auth.sh        | 105 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 161 insertions(+)
>  create mode 100755 t/t5556-http-auth.sh
> 
> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> index 7bde678e264..9f1d6b58067 100644
> --- a/t/helper/test-http-server.c
> +++ b/t/helper/test-http-server.c
> @@ -305,8 +305,64 @@ done:
>  	return result;
>  }
>  
> +static int is_git_request(struct req *req)
> +{
> +	static regex_t *smart_http_regex;
> +	static int initialized;
> +
> +	if (!initialized) {
> +		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
> +		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
> +			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
> +			    REG_EXTENDED)) {

Could you explain the reasoning behind this regex (e.g., in a comment)? What
sorts of valid/invalid requests does it represent? Is that the full set of
requests that are "valid" to Git, or is it a test-specific subset?

> +			warning("could not compile smart HTTP regex");
> +			smart_http_regex = NULL;
> +		}
> +		initialized = 1;
> +	}
> +
> +	return smart_http_regex &&
> +		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
> +}
> +
> +static enum worker_result do__git(struct req *req, const char *user)
> +{
> +	const char *ok = "HTTP/1.1 200 OK\r\n";
> +	struct child_process cp = CHILD_PROCESS_INIT;
> +	int res;
> +
> +	if (write(1, ok, strlen(ok)) < 0)
> +		return error(_("could not send '%s'"), ok);

Is it correct to hardcode the response status to '200 OK'? Even when
'http-backend' exits with an error?

> +
> +	if (user)
> +		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);

I'm guessing that 'user' isn't used until a later patch? I think it might be
better to not introduce that arg at all until it's needed (it'll put the
usage of 'user' in context with how its value is determined), rather than
hardcode it to 'NULL' for now.

> +
> +	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
> +	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
> +			req->uri_path.buf);
> +	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
> +	if (req->query_args.len)
> +		strvec_pushf(&cp.env, "QUERY_STRING=%s",
> +				req->query_args.buf);
> +	if (req->content_type)
> +		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
> +				req->content_type);
> +	if (req->content_length >= 0)
> +		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
> +				(intmax_t)req->content_length);
> +	cp.git_cmd = 1;
> +	strvec_push(&cp.args, "http-backend");
> +	res = run_command(&cp);

I'm not super familiar with 'http-backend' but as long as it 1) uses the
content passed into the environment to parse the request, and 2) writes the
response to stdout, I think this is right.

> +	close(1);
> +	close(0);
> +	return !!res;
> +}
> +
>  static enum worker_result dispatch(struct req *req)
>  {
> +	if (is_git_request(req))
> +		return do__git(req, NULL);
> +
>  	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>  			       WR_OK | WR_HANGUP);
>  }
> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
> new file mode 100755
> index 00000000000..78da151f122
> --- /dev/null
> +++ b/t/t5556-http-auth.sh
> @@ -0,0 +1,105 @@
> +#!/bin/sh
> +
> +test_description='test http auth header and credential helper interop'
> +
> +. ./test-lib.sh
> +
> +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
> +
> +# Setup a repository
> +#
> +REPO_DIR="$(pwd)"/repo

nit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with
what I see in other tests. 

Also, if you're creating a repo in its own subdirectory ('repo'), you can
set 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating
a repo at the root level of the test output dir - it can help avoid
potential weird/unexpected behavior as a result of being in a repo inside of
another repo.

> +
> +# Setup some lookback URLs where test-http-server will be listening.
> +# We will spawn it directly inside the repo directory, so we avoid
> +# any need to configure directory mappings etc - we only serve this
> +# repository from the root '/' of the server.
> +#
> +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
> +ORIGIN_URL=http://$HOST_PORT/
> +
> +# The pid-file is created by test-http-server when it starts.
> +# The server will shutdown if/when we delete it (this is easier than
> +# killing it by PID).
> +#
> +PID_FILE="$(pwd)"/pid-file.pid
> +SERVER_LOG="$(pwd)"/OUT.server.log
> +
> +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
> +
> +test_expect_success 'setup repos' '
> +	test_create_repo "$REPO_DIR" &&
> +	git -C "$REPO_DIR" branch -M main
> +'
> +
> +stop_http_server () {
> +	if ! test -f "$PID_FILE"
> +	then
> +		return 0
> +	fi
> +	#
> +	# The server will shutdown automatically when we delete the pid-file.
> +	#
> +	rm -f "$PID_FILE"
> +	#
> +	# Give it a few seconds to shutdown (mainly to completely release the
> +	# port before the next test start another instance and it attempts to
> +	# bind to it).
> +	#
> +	for k in 0 1 2 3 4
> +	do
> +		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
> +		then
> +			return 0
> +		fi
> +		sleep 1
> +	done
> +
> +	echo "stop_http_server: timeout waiting for server shutdown"
> +	return 1
> +}
> +
> +start_http_server () {
> +	#
> +	# Launch our server into the background in repo_dir.
> +	#
> +	(
> +		cd "$REPO_DIR"
> +		test-http-server --verbose \
> +			--listen=127.0.0.1 \
> +			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
> +			--reuseaddr \
> +			--pid-file="$PID_FILE" \
> +			"$@" \
> +			2>"$SERVER_LOG" &
> +	)
> +	#
> +	# Give it a few seconds to get started.
> +	#
> +	for k in 0 1 2 3 4
> +	do
> +		if test -f "$PID_FILE"
> +		then
> +			return 0
> +		fi
> +		sleep 1
> +	done
> +
> +	echo "start_http_server: timeout waiting for server startup"
> +	return 1
> +}

These start/stop functions look good to me!

> +
> +per_test_cleanup () {
> +	stop_http_server &&
> +	rm -f OUT.*
> +}
> +
> +test_expect_success 'http auth anonymous no challenge' '
> +	test_when_finished "per_test_cleanup" &&
> +	start_http_server --allow-anonymous &&

The '--allow-anonymous' option isn't added until patch 7 [1], so the test
will fail in this patch. I think the easiest way to solve that is to remove
it here (although I think it's fine to leave the title "anonymous no
challenge", though), then add it in patch 7. 

[1] https://lore.kernel.org/git/794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com/

> +
> +	# Attempt to read from a protected repository
> +	git ls-remote $ORIGIN_URL
> +'
> +
> +test_done

Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend

[Matthew John Cheetham]

On 2022-12-14 15:20, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Teach the test-http-sever test helper to forward Git requests to the
>> `git-http-backend`.
>>
>> Introduce a new test script t5556-http-auth.sh that spins up the test
>> HTTP server and attempts an `ls-remote` on the served repository,
>> without any authentication.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  t/helper/test-http-server.c |  56 +++++++++++++++++++
>>  t/t5556-http-auth.sh        | 105 ++++++++++++++++++++++++++++++++++++
>>  2 files changed, 161 insertions(+)
>>  create mode 100755 t/t5556-http-auth.sh
>>
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> index 7bde678e264..9f1d6b58067 100644
>> --- a/t/helper/test-http-server.c
>> +++ b/t/helper/test-http-server.c
>> @@ -305,8 +305,64 @@ done:
>>  	return result;
>>  }
>>  
>> +static int is_git_request(struct req *req)
>> +{
>> +	static regex_t *smart_http_regex;
>> +	static int initialized;
>> +
>> +	if (!initialized) {
>> +		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
>> +		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
>> +			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
>> +			    REG_EXTENDED)) {
> 
> Could you explain the reasoning behind this regex (e.g., in a comment)? What
> sorts of valid/invalid requests does it represent? Is that the full set of
> requests that are "valid" to Git, or is it a test-specific subset?

Explanatory comment will be added in next iteration. These are the valid Git
endpoints for the dumb and smart HTTP protocols as specified in the tech docs.

>> +			warning("could not compile smart HTTP regex");
>> +			smart_http_regex = NULL;
>> +		}
>> +		initialized = 1;
>> +	}
>> +
>> +	return smart_http_regex &&
>> +		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
>> +}
>> +
>> +static enum worker_result do__git(struct req *req, const char *user)
>> +{
>> +	const char *ok = "HTTP/1.1 200 OK\r\n";
>> +	struct child_process cp = CHILD_PROCESS_INIT;
>> +	int res;
>> +
>> +	if (write(1, ok, strlen(ok)) < 0)
>> +		return error(_("could not send '%s'"), ok);
> 
> Is it correct to hardcode the response status to '200 OK'? Even when
> 'http-backend' exits with an error?

We always respond with a 200 OK response even if the http-backend process exits
with an error. This helper is intended only to be used to exercise the HTTP
auth handling in the Git client, and specifically around authentication (not
handled by http-backend).

If we wanted to respond with a more 'valid' HTTP response status then we'd need
to buffer the output of http-backend, wait for and grok the exit status of the
process, then write the HTTP status line followed by the http-backend output.
This is outside of the scope of this test helper's use at time of writing.

Important auth responses (401) we are handling prior to getting to this point.

The above will also be summarised in a comment on the next roll.

>> +
>> +	if (user)
>> +		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
> 
> I'm guessing that 'user' isn't used until a later patch? I think it might be
> better to not introduce that arg at all until it's needed (it'll put the
> usage of 'user' in context with how its value is determined), rather than
> hardcode it to 'NULL' for now.

Good point!

>> +
>> +	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
>> +	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
>> +			req->uri_path.buf);
>> +	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
>> +	if (req->query_args.len)
>> +		strvec_pushf(&cp.env, "QUERY_STRING=%s",
>> +				req->query_args.buf);
>> +	if (req->content_type)
>> +		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
>> +				req->content_type);
>> +	if (req->content_length >= 0)
>> +		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
>> +				(intmax_t)req->content_length);
>> +	cp.git_cmd = 1;
>> +	strvec_push(&cp.args, "http-backend");
>> +	res = run_command(&cp);
> 
> I'm not super familiar with 'http-backend' but as long as it 1) uses the
> content passed into the environment to parse the request, and 2) writes the
> response to stdout, I think this is right.
> 
>> +	close(1);
>> +	close(0);
>> +	return !!res;
>> +}
>> +
>>  static enum worker_result dispatch(struct req *req)
>>  {
>> +	if (is_git_request(req))
>> +		return do__git(req, NULL);
>> +
>>  	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>>  			       WR_OK | WR_HANGUP);
>>  }
>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
>> new file mode 100755
>> index 00000000000..78da151f122
>> --- /dev/null
>> +++ b/t/t5556-http-auth.sh
>> @@ -0,0 +1,105 @@
>> +#!/bin/sh
>> +
>> +test_description='test http auth header and credential helper interop'
>> +
>> +. ./test-lib.sh
>> +
>> +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
>> +
>> +# Setup a repository
>> +#
>> +REPO_DIR="$(pwd)"/repo
> 
> nit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with
> what I see in other tests. 

I don't see this? In fact I see more usages of `$(pwd)` than your suggestion.

> Also, if you're creating a repo in its own subdirectory ('repo'), you can
> set 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating
> a repo at the root level of the test output dir - it can help avoid
> potential weird/unexpected behavior as a result of being in a repo inside of
> another repo.

However.. after setting `TEST_NO_CREATE_REPO=1` I was getting CI failures
around a missing PWD, so my next iteration uses the `$TRASH_DIRECTORY` variable
explicitly in paths instead :-)

>> +
>> +# Setup some lookback URLs where test-http-server will be listening.
>> +# We will spawn it directly inside the repo directory, so we avoid
>> +# any need to configure directory mappings etc - we only serve this
>> +# repository from the root '/' of the server.
>> +#
>> +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
>> +ORIGIN_URL=http://$HOST_PORT/
>> +
>> +# The pid-file is created by test-http-server when it starts.
>> +# The server will shutdown if/when we delete it (this is easier than
>> +# killing it by PID).
>> +#
>> +PID_FILE="$(pwd)"/pid-file.pid
>> +SERVER_LOG="$(pwd)"/OUT.server.log
>> +
>> +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
>> +
>> +test_expect_success 'setup repos' '
>> +	test_create_repo "$REPO_DIR" &&
>> +	git -C "$REPO_DIR" branch -M main
>> +'
>> +
>> +stop_http_server () {
>> +	if ! test -f "$PID_FILE"
>> +	then
>> +		return 0
>> +	fi
>> +	#
>> +	# The server will shutdown automatically when we delete the pid-file.
>> +	#
>> +	rm -f "$PID_FILE"
>> +	#
>> +	# Give it a few seconds to shutdown (mainly to completely release the
>> +	# port before the next test start another instance and it attempts to
>> +	# bind to it).
>> +	#
>> +	for k in 0 1 2 3 4
>> +	do
>> +		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
>> +		then
>> +			return 0
>> +		fi
>> +		sleep 1
>> +	done
>> +
>> +	echo "stop_http_server: timeout waiting for server shutdown"
>> +	return 1
>> +}
>> +
>> +start_http_server () {
>> +	#
>> +	# Launch our server into the background in repo_dir.
>> +	#
>> +	(
>> +		cd "$REPO_DIR"
>> +		test-http-server --verbose \
>> +			--listen=127.0.0.1 \
>> +			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
>> +			--reuseaddr \
>> +			--pid-file="$PID_FILE" \
>> +			"$@" \
>> +			2>"$SERVER_LOG" &
>> +	)
>> +	#
>> +	# Give it a few seconds to get started.
>> +	#
>> +	for k in 0 1 2 3 4
>> +	do
>> +		if test -f "$PID_FILE"
>> +		then
>> +			return 0
>> +		fi
>> +		sleep 1
>> +	done
>> +
>> +	echo "start_http_server: timeout waiting for server startup"
>> +	return 1
>> +}
> 
> These start/stop functions look good to me!
> 
>> +
>> +per_test_cleanup () {
>> +	stop_http_server &&
>> +	rm -f OUT.*
>> +}
>> +
>> +test_expect_success 'http auth anonymous no challenge' '
>> +	test_when_finished "per_test_cleanup" &&
>> +	start_http_server --allow-anonymous &&
> 
> The '--allow-anonymous' option isn't added until patch 7 [1], so the test
> will fail in this patch. I think the easiest way to solve that is to remove
> it here (although I think it's fine to leave the title "anonymous no
> challenge", though), then add it in patch 7. 
> 
> [1] https://lore.kernel.org/git/794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com/

Good catch! Will fix.

>> +
>> +	# Attempt to read from a protected repository
>> +	git ls-remote $ORIGIN_URL
>> +'
>> +
>> +test_done
> 

Thanks,
Matthew

Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend

[Victoria Dye]

Matthew John Cheetham wrote:
> 
> On 2022-12-14 15:20, Victoria Dye wrote:
>> nit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with
>> what I see in other tests. 
> 
> I don't see this? In fact I see more usages of `$(pwd)` than your suggestion.

To be honest, I'm not sure how I missed this. '$(pwd)' *is* quite common in
the tests, although it does seem to be used mostly in individual tests
rather than file-level variables (although that's not universally true, e.g.
using it to set 'CURR_DIR' in 't9400-diff-highlight.sh'). 

So, contrary to my earlier comment, this seems best left up to (your)
personal preference than any concrete rule.

> 
>> Also, if you're creating a repo in its own subdirectory ('repo'), you can
>> set 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating
>> a repo at the root level of the test output dir - it can help avoid
>> potential weird/unexpected behavior as a result of being in a repo inside of
>> another repo.
> 
> However.. after setting `TEST_NO_CREATE_REPO=1` I was getting CI failures
> around a missing PWD, so my next iteration uses the `$TRASH_DIRECTORY` variable
> explicitly in paths instead :-)

You're right, I was completely misreading the purpose of
'TEST_OUTPUT_DIRECTORY' in 'test-lib.sh':

> if test -z "$TEST_OUTPUT_DIRECTORY"
> then
> 	# Similarly, override this to store the test-results subdir
> 	# elsewhere
> 	TEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY
> fi

"the test-results subdir" != "the test working directory". As you pointed
out, '$TRASH_DIRECTORY' would be the variable to use here.

[PATCH v4 7/8] test-http-server: add simple authentication

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
command-line arguments. Incoming requests are compared against the set
of valid schemes and tokens and only approved if a matching token is
found, or if no auth was provided and anonymous auth is enabled.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 188 +++++++++++++++++++++++++++++++++++-
 1 file changed, 187 insertions(+), 1 deletion(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 9f1d6b58067..9a458743d13 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -18,6 +18,8 @@ static const char test_http_auth_usage[] =
 "           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
 "           [--reuseaddr] [--pid-file=<file>]\n"
 "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--anonymous-allowed]\n"
+"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
 ;
 
 /* Timeout, and initial timeout */
@@ -358,10 +360,136 @@ static enum worker_result do__git(struct req *req, const char *user)
 	return !!res;
 }
 
+enum auth_result {
+	/* No auth module matches the request. */
+	AUTH_UNKNOWN = 0,
+
+	/* Auth module denied the request. */
+	AUTH_DENY = 1,
+
+	/* Auth module successfully validated the request. */
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	char *scheme;
+	char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(const char *scheme)
+{
+	int i;
+	struct auth_module *mod;
+	for (i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme))
+			return mod;
+	}
+
+	return NULL;
+}
+
+static void add_auth_module(struct auth_module *mod)
+{
+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+	auth_modules[auth_modules_nr++] = mod;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/*
+	 * Check all auth modules and try to validate the request.
+	 * The first module that matches a valid token approves the request.
+	 * If no module is found, or if there is no valid token, then 401 error.
+	 * Otherwise, only permit the request if anonymous auth is enabled.
+	 */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (!split[0] || !split[1]) continue;
+
+			/* trim trailing space ' ' */
+			strbuf_setlen(split[0], split[0]->len - 1);
+
+			mod = get_auth_module(split[0]->buf);
+			if (mod) {
+				result = AUTH_DENY;
+
+				for_each_string_list_item(token, mod->tokens) {
+					if (!strcmp(split[1]->buf, token->string)) {
+						result = AUTH_ALLOW;
+						break;
+					}
+				}
+
+				goto done;
+			}
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (result != AUTH_DENY && allow_anonymous)
+			break;
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
+	}
+
+	strbuf_list_free(split);
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
 	if (is_git_request(req))
-		return do__git(req, NULL);
+		return do__git(req, user);
 
 	return send_http_error(1, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
@@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)
 	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
 	int worker_mode = 0;
 	int i;
+	struct auth_module *mod = NULL;
 
 	trace2_cmd_name("test-http-server");
 	setup_git_directory_gently(NULL);
@@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)
 			pid_file = v;
 			continue;
 		}
+		if (skip_prefix(arg, "--allow-anonymous", &v)) {
+			allow_anonymous = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--auth=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			/* trim trailing ':' */
+			if (p[1])
+				strbuf_setlen(p[0], p[0]->len - 1);
+
+			if (get_auth_module(p[0]->buf)) {
+				error("duplicate auth scheme '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			mod = xmalloc(sizeof(struct auth_module));
+			mod->scheme = xstrdup(p[0]->buf);
+			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
+			CALLOC_ARRAY(mod->tokens, 1);
+			string_list_init_dup(mod->tokens);
+
+			add_auth_module(mod);
+
+			strbuf_list_free(p);
+			continue;
+		}
+		if (skip_prefix(arg, "--auth-token=", &v)) {
+			struct strbuf **p = strbuf_split_str(v, ':', 2);
+			if (!p[0]) {
+				error("invalid argument '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			if (!p[1]) {
+				error("missing token value '%s'\n", v);
+				usage(test_http_auth_usage);
+			}
+
+			/* trim trailing ':' */
+			strbuf_setlen(p[0], p[0]->len - 1);
+
+			mod = get_auth_module(p[0]->buf);
+			if (!mod) {
+				error("auth scheme not defined '%s'\n", p[0]->buf);
+				usage(test_http_auth_usage);
+			}
+
+			string_list_append(mod->tokens, p[1]->buf);
+			strbuf_list_free(p);
+			continue;
+		}
 
 		fprintf(stderr, "error: unknown argument '%s'\n", arg);
 		usage(test_http_auth_usage);
-- 
gitgitgadget

Re: [PATCH v4 7/8] test-http-server: add simple authentication

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
> +{
> +	enum auth_result result = AUTH_UNKNOWN;
> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
> +	struct auth_module *mod;
> +
> +	struct string_list_item *hdr;
> +	struct string_list_item *token;
> +	const char *v;
> +	struct strbuf **split = NULL;
> +	int i;
> +	char *challenge;
> +
> +	/*
> +	 * Check all auth modules and try to validate the request.
> +	 * The first module that matches a valid token approves the request.
> +	 * If no module is found, or if there is no valid token, then 401 error.
> +	 * Otherwise, only permit the request if anonymous auth is enabled.
> +	 */
> +	for_each_string_list_item(hdr, &req->header_list) {
> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {

Is only one "Authorization:" header allowed? If so, adding a 'break;' at the
end of this if-statement would make that clearer. If not, what's the
expected allow/deny behavior if e.g. one header is ALLOW'd by one auth
module, and another header is DENY'd by a different auth module?

> +			split = strbuf_split_str(v, ' ', 2);
> +			if (!split[0] || !split[1]) continue;
> +
> +			/* trim trailing space ' ' */
> +			strbuf_setlen(split[0], split[0]->len - 1);
> +
> +			mod = get_auth_module(split[0]->buf);
> +			if (mod) {
> +				result = AUTH_DENY;
> +
> +				for_each_string_list_item(token, mod->tokens) {
> +					if (!strcmp(split[1]->buf, token->string)) {
> +						result = AUTH_ALLOW;
> +						break;
> +					}
> +				}
> +
> +				goto done;
> +			}
> +		}
> +	}
> +
> +done:
> +	switch (result) {
> +	case AUTH_ALLOW:
> +		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
> +		*user = "VALID_TEST_USER";
> +		*wr = WR_OK;
> +		break;
> +
> +	case AUTH_DENY:
> +		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
> +		/* fall-through */
> +
> +	case AUTH_UNKNOWN:
> +		if (result != AUTH_DENY && allow_anonymous)
> +			break;

I think this just needs to be 'if (allow_anonymous)' - we already know
'result' is 'AUTH_UNKNOWN' once we reach this block.

> +		for (i = 0; i < auth_modules_nr; i++) {
> +			mod = auth_modules[i];
> +			if (mod->challenge_params)
> +				challenge = xstrfmt("WWW-Authenticate: %s %s",
> +						    mod->scheme,
> +						    mod->challenge_params);
> +			else
> +				challenge = xstrfmt("WWW-Authenticate: %s",
> +						    mod->scheme);
> +			string_list_append(&hdrs, challenge);
> +		}
> +		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
> +	}
> +
> +	strbuf_list_free(split);
> +	string_list_clear(&hdrs, 0);
> +
> +	return result == AUTH_ALLOW ||
> +	      (result == AUTH_UNKNOWN && allow_anonymous);

So if a user is explicitly denied, even with 'allow_anonymous', this fails?
Is there a test case that uses that behavior and/or is that standard auth
behavior? Otherwise, it'd be simpler to skip the 'is_authed()' check (in
'dispatch()') altogether if 'allow_anonymous' is enabled.

> +}
> +
>  static enum worker_result dispatch(struct req *req)
>  {
> +	enum worker_result wr = WR_OK;
> +	const char *user = NULL;
> +
> +	if (!is_authed(req, &user, &wr))
> +		return wr;
> +
>  	if (is_git_request(req))
> -		return do__git(req, NULL);
> +		return do__git(req, user);
>  
>  	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>  			       WR_OK | WR_HANGUP);
> @@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)
>  	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
>  	int worker_mode = 0;
>  	int i;
> +	struct auth_module *mod = NULL;
>  
>  	trace2_cmd_name("test-http-server");
>  	setup_git_directory_gently(NULL);
> @@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)
>  			pid_file = v;
>  			continue;
>  		}
> +		if (skip_prefix(arg, "--allow-anonymous", &v)) {
> +			allow_anonymous = 1;
> +			continue;
> +		}
> +		if (skip_prefix(arg, "--auth=", &v)) {
...

> +		}
> +		if (skip_prefix(arg, "--auth-token=", &v)) {
> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
> +			if (!p[0]) {
> +				error("invalid argument '%s'", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			if (!p[1]) {
> +				error("missing token value '%s'\n", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			/* trim trailing ':' */
> +			strbuf_setlen(p[0], p[0]->len - 1);
> +
> +			mod = get_auth_module(p[0]->buf);
> +			if (!mod) {
> +				error("auth scheme not defined '%s'\n", p[0]->buf);
> +				usage(test_http_auth_usage);
> +			}

Does this mean that '--auth' needs to be specified before '--auth-token' to
avoid the "auth scheme not defined" error? If so, this could be made less
fragile by just setting the string value of the arg in this 'if()' block,
then processing the value after the option-parsing loop.

> +
> +			string_list_append(mod->tokens, p[1]->buf);
> +			strbuf_list_free(p);
> +			continue;
> +		}
>  
>  		fprintf(stderr, "error: unknown argument '%s'\n", arg);
>  		usage(test_http_auth_usage);

I think a test (in this patch) showing how the auth headers are handled by
this HTTP server would be really helpful in demonstrating/exercising the
intended behavior. 

Re: [PATCH v4 7/8] test-http-server: add simple authentication

[Matthew John Cheetham]

On 2022-12-14 15:23, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
>> +{
>> +	enum auth_result result = AUTH_UNKNOWN;
>> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
>> +	struct auth_module *mod;
>> +
>> +	struct string_list_item *hdr;
>> +	struct string_list_item *token;
>> +	const char *v;
>> +	struct strbuf **split = NULL;
>> +	int i;
>> +	char *challenge;
>> +
>> +	/*
>> +	 * Check all auth modules and try to validate the request.
>> +	 * The first module that matches a valid token approves the request.
>> +	 * If no module is found, or if there is no valid token, then 401 error.
>> +	 * Otherwise, only permit the request if anonymous auth is enabled.
>> +	 */
>> +	for_each_string_list_item(hdr, &req->header_list) {
>> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
> 
> Is only one "Authorization:" header allowed? If so, adding a 'break;' at the
> end of this if-statement would make that clearer. If not, what's the
> expected allow/deny behavior if e.g. one header is ALLOW'd by one auth
> module, and another header is DENY'd by a different auth module?

Yes, only one Authorization header *should* be passed.. but the RFCs are not very
explicit about that. The test server supports multiple, but will `ALLOW` or `DENY`
based on the first matching auth scheme (module).

> 
>> +			split = strbuf_split_str(v, ' ', 2);
>> +			if (!split[0] || !split[1]) continue;
>> +
>> +			/* trim trailing space ' ' */
>> +			strbuf_setlen(split[0], split[0]->len - 1);
>> +
>> +			mod = get_auth_module(split[0]->buf);
>> +			if (mod) {
>> +				result = AUTH_DENY;
>> +
>> +				for_each_string_list_item(token, mod->tokens) {
>> +					if (!strcmp(split[1]->buf, token->string)) {
>> +						result = AUTH_ALLOW;
>> +						break;
>> +					}
>> +				}
>> +
>> +				goto done;
>> +			}
>> +		}
>> +	}
>> +
>> +done:
>> +	switch (result) {
>> +	case AUTH_ALLOW:
>> +		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
>> +		*user = "VALID_TEST_USER";
>> +		*wr = WR_OK;
>> +		break;
>> +
>> +	case AUTH_DENY:
>> +		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
>> +		/* fall-through */
>> +
>> +	case AUTH_UNKNOWN:
>> +		if (result != AUTH_DENY && allow_anonymous)
>> +			break;
> 
> I think this just needs to be 'if (allow_anonymous)' - we already know
> 'result' is 'AUTH_UNKNOWN' once we reach this block.

Note that `AUTH_DENY` falls-through to the `AUTH_UNKNOWN` case.
The only time we *DON'T* want to output the auth challenge response headers is
when there was no challenge provided (`AUTH_UNKNOWN`) *and* we are permitting
anonymous users.

  result      | allow_anoymous | Output Challenge?
---------------------------------------------------
 AUTH_DENY    |       1        |       Yes
 AUTH_DENY    |       0        |       Yes
 AUTH_UNKNOWN |       1        |       No
 AUTH_UNKNOWN |       0        |       Yes

>> +		for (i = 0; i < auth_modules_nr; i++) {
>> +			mod = auth_modules[i];
>> +			if (mod->challenge_params)
>> +				challenge = xstrfmt("WWW-Authenticate: %s %s",
>> +						    mod->scheme,
>> +						    mod->challenge_params);
>> +			else
>> +				challenge = xstrfmt("WWW-Authenticate: %s",
>> +						    mod->scheme);
>> +			string_list_append(&hdrs, challenge);
>> +		}
>> +		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
>> +	}
>> +
>> +	strbuf_list_free(split);
>> +	string_list_clear(&hdrs, 0);
>> +
>> +	return result == AUTH_ALLOW ||
>> +	      (result == AUTH_UNKNOWN && allow_anonymous);
> 
> So if a user is explicitly denied, even with 'allow_anonymous', this fails?
> Is there a test case that uses that behavior and/or is that standard auth
> behavior? Otherwise, it'd be simpler to skip the 'is_authed()' check (in
> 'dispatch()') altogether if 'allow_anonymous' is enabled.

If the user is being denied by a module we should always deny access.

Admittedly, for this simple authentication scenario it's kind of silly to deny
a user who is trying to identify themselves, but permit an anoymous user.
However, if this was an authorization failure then denying a user based on their
token may be totally valid. Right now, we're only concerned about authentication
and not authorization, so I could move this check to `dispatch()` if you feel
strongly about it.

>> +}
>> +
>>  static enum worker_result dispatch(struct req *req)
>>  {
>> +	enum worker_result wr = WR_OK;
>> +	const char *user = NULL;
>> +
>> +	if (!is_authed(req, &user, &wr))
>> +		return wr;
>> +
>>  	if (is_git_request(req))
>> -		return do__git(req, NULL);
>> +		return do__git(req, user);
>>  
>>  	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>>  			       WR_OK | WR_HANGUP);
>> @@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)
>>  	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
>>  	int worker_mode = 0;
>>  	int i;
>> +	struct auth_module *mod = NULL;
>>  
>>  	trace2_cmd_name("test-http-server");
>>  	setup_git_directory_gently(NULL);
>> @@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)
>>  			pid_file = v;
>>  			continue;
>>  		}
>> +		if (skip_prefix(arg, "--allow-anonymous", &v)) {
>> +			allow_anonymous = 1;
>> +			continue;
>> +		}
>> +		if (skip_prefix(arg, "--auth=", &v)) {
> ...
> 
>> +		}
>> +		if (skip_prefix(arg, "--auth-token=", &v)) {
>> +			struct strbuf **p = strbuf_split_str(v, ':', 2);
>> +			if (!p[0]) {
>> +				error("invalid argument '%s'", v);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			if (!p[1]) {
>> +				error("missing token value '%s'\n", v);
>> +				usage(test_http_auth_usage);
>> +			}
>> +
>> +			/* trim trailing ':' */
>> +			strbuf_setlen(p[0], p[0]->len - 1);
>> +
>> +			mod = get_auth_module(p[0]->buf);
>> +			if (!mod) {
>> +				error("auth scheme not defined '%s'\n", p[0]->buf);
>> +				usage(test_http_auth_usage);
>> +			}
> 
> Does this mean that '--auth' needs to be specified before '--auth-token' to
> avoid the "auth scheme not defined" error? If so, this could be made less
> fragile by just setting the string value of the arg in this 'if()' block,
> then processing the value after the option-parsing loop.

Yes, `--auth` needs to come first and 'setup' the module and challenge.

>> +
>> +			string_list_append(mod->tokens, p[1]->buf);
>> +			strbuf_list_free(p);
>> +			continue;
>> +		}
>>  
>>  		fprintf(stderr, "error: unknown argument '%s'\n", arg);
>>  		usage(test_http_auth_usage);
> 
> I think a test (in this patch) showing how the auth headers are handled by
> this HTTP server would be really helpful in demonstrating/exercising the
> intended behavior. 
> 

Thanks,
Matthew

[PATCH v4 8/8] t5556: add HTTP authentication tests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a series of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-credential-helper-replay.sh |  14 +++
 t/t5556-http-auth.sh                      | 120 +++++++++++++++++++++-
 2 files changed, 133 insertions(+), 1 deletion(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh

diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
new file mode 100755
index 00000000000..03e5e63dad6
--- /dev/null
+++ b/t/helper/test-credential-helper-replay.sh
@@ -0,0 +1,14 @@
+cmd=$1
+teefile=$cmd-actual.cred
+catfile=$cmd-response.cred
+rm -f $teefile
+while read line;
+do
+	if test -z "$line"; then
+		break;
+	fi
+	echo "$line" >> $teefile
+done
+if test "$cmd" = "get"; then
+	cat $catfile
+fi
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 78da151f122..541fa32bd77 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -26,6 +26,8 @@ PID_FILE="$(pwd)"/pid-file.pid
 SERVER_LOG="$(pwd)"/OUT.server.log
 
 PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
+	&& export CREDENTIAL_HELPER
 
 test_expect_success 'setup repos' '
 	test_create_repo "$REPO_DIR" &&
@@ -91,7 +93,8 @@ start_http_server () {
 
 per_test_cleanup () {
 	stop_http_server &&
-	rm -f OUT.*
+	rm -f OUT.* &&
+	rm -f *.cred
 }
 
 test_expect_success 'http auth anonymous no challenge' '
@@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '
 	git ls-remote $ORIGIN_URL
 '
 
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	start_http_server \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=basic:$USERPASS64 &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	start_http_server \
+		--auth=foobar:alg=test\ widget=1 \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=basic:$USERPASS64 &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+	start_http_server \
+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
+		--auth=basic:realm=\"example.com\" \
+		--auth-token=basic:$USERPASS64 &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >erase-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	EOF
+
+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp erase-expected.cred erase-actual.cred
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v4 8/8] t5556: add HTTP authentication tests

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Add a series of tests to exercise the HTTP authentication header parsing
> and the interop with credential helpers. Credential helpers will receive
> WWW-Authenticate information in credential requests.

A general comment about this series - the way you have the patches organized
means that the "feature" content you're trying to integrate (the first two
patches) is contextually separated from these tests. For people that
learn/understand code via examples in tests, this makes it really difficult
to understand what's going on. To avoid that, I think you could rearrange
the patches pretty easily:

1. test-http-server: add stub HTTP server test helper (prev. patch 3)
  - t5556 could be introduced here with the basic "anonymous" test in patch
    6, but marked 'test_expect_failure'.
2. test-http-server: add HTTP error response function (prev. patch 4)
3. test-http-server: add HTTP request parsing (prev. patch 5)
4. test-http-server: pass Git requests to http-backend (prev. patch 6)
5. test-http-server: add simple authentication (prev. patch 7)
6. http: read HTTP WWW-Authenticate response headers (prev. patch 1)
7. credential: add WWW-Authenticate header to cred requests (prev patch 2)
  - Some/all of the tests from the current patch (patch 8) could be squashed
    into this one so that the tests exist directly alongside the new
    functionality they're testing.

> 
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  t/helper/test-credential-helper-replay.sh |  14 +++
>  t/t5556-http-auth.sh                      | 120 +++++++++++++++++++++-
>  2 files changed, 133 insertions(+), 1 deletion(-)
>  create mode 100755 t/helper/test-credential-helper-replay.sh
> 
> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
> new file mode 100755
> index 00000000000..03e5e63dad6
> --- /dev/null
> +++ b/t/helper/test-credential-helper-replay.sh

I'm not sure a 't/helper' file is the right place for this - it's a pretty
simple shell script, but it defines a lot of information (namely 'teefile',
'catfile') that is otherwise unexplained in 't5556'. 

What about something like 'lib-rebase.sh' and its 'set_fake_editor()'? You
could create a similar test lib ('lib-credential-helper.sh') and wrapper
function (' that writes out a custom credential helper. Something like
'set_fake_credential_helper()' could also take 'teefile' and 'catfile' as
arguments, making their names more transparent to 't5556'.

> @@ -0,0 +1,14 @@
> +cmd=$1
> +teefile=$cmd-actual.cred
> +catfile=$cmd-response.cred
> +rm -f $teefile
> +while read line;
> +do
> +	if test -z "$line"; then
> +		break;
> +	fi
> +	echo "$line" >> $teefile
> +done
> +if test "$cmd" = "get"; then
> +	cat $catfile
> +fi
> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
> index 78da151f122..541fa32bd77 100755
> --- a/t/t5556-http-auth.sh
> +++ b/t/t5556-http-auth.sh
> @@ -26,6 +26,8 @@ PID_FILE="$(pwd)"/pid-file.pid
>  SERVER_LOG="$(pwd)"/OUT.server.log
>  
>  PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
> +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
> +	&& export CREDENTIAL_HELPER

I see - this is how you connect the "test" credential helper to the HTTP
server and header parsing (as implemented in patches 1 & 2), so that the
results can be compared for correctness.

nit: you can just 'export CREDENTIAL_HELPER="..."', rather than breaking it
into two lines. You also shouldn't need to 'export' at all - the value will
be set in the context of the test.

>  
>  test_expect_success 'setup repos' '
>  	test_create_repo "$REPO_DIR" &&
> @@ -91,7 +93,8 @@ start_http_server () {
>  
>  per_test_cleanup () {
>  	stop_http_server &&
> -	rm -f OUT.*
> +	rm -f OUT.* &&
> +	rm -f *.cred
>  }
>  
>  test_expect_success 'http auth anonymous no challenge' '
> @@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '
>  	git ls-remote $ORIGIN_URL
>  '
>  
> +test_expect_success 'http auth www-auth headers to credential helper basic valid' '

...

> +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '

...

> +test_expect_success 'http auth www-auth headers to credential helper invalid' '

These tests all look good. That said, is there any way to test more
bizarre/edge cases (headers too long to fit on one line, headers that end
with a long string of whitespace, etc.)?

Re: [PATCH v4 8/8] t5556: add HTTP authentication tests

[Junio C Hamano]

Victoria Dye <vdye@github.com> writes:

> A general comment about this series - the way you have the patches organized
> means that the "feature" content you're trying to integrate (the first two
> patches) is contextually separated from these tests. For people that
> learn/understand code via examples in tests, this makes it really difficult
> to understand what's going on. To avoid that, I think you could rearrange
> the patches pretty easily:
> ...

Thanks for a thorough review of the entire series, with concrete
suggestions for improvements with encouragements sprinkled in.

Very much appreciated.

Re: [PATCH v4 8/8] t5556: add HTTP authentication tests

[Matthew John Cheetham]

On 2022-12-14 16:21, Junio C Hamano wrote:

> Victoria Dye <vdye@github.com> writes:
> 
>> A general comment about this series - the way you have the patches organized
>> means that the "feature" content you're trying to integrate (the first two
>> patches) is contextually separated from these tests. For people that
>> learn/understand code via examples in tests, this makes it really difficult
>> to understand what's going on. To avoid that, I think you could rearrange
>> the patches pretty easily:
>> ...
> 
> Thanks for a thorough review of the entire series, with concrete
> suggestions for improvements with encouragements sprinkled in.
> 
> Very much appreciated.
> 

Yes! Thank you Victoria for the detailed and thorough review.
I also too very much appreciate it :-)

Thanks,
Matthew

Re: [PATCH v4 8/8] t5556: add HTTP authentication tests

[Matthew John Cheetham]

On 2022-12-14 15:48, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add a series of tests to exercise the HTTP authentication header parsing
>> and the interop with credential helpers. Credential helpers will receive
>> WWW-Authenticate information in credential requests.
> 
> A general comment about this series - the way you have the patches organized
> means that the "feature" content you're trying to integrate (the first two
> patches) is contextually separated from these tests. For people that
> learn/understand code via examples in tests, this makes it really difficult
> to understand what's going on. To avoid that, I think you could rearrange
> the patches pretty easily:
> 
> 1. test-http-server: add stub HTTP server test helper (prev. patch 3)
>   - t5556 could be introduced here with the basic "anonymous" test in patch
>     6, but marked 'test_expect_failure'.
> 2. test-http-server: add HTTP error response function (prev. patch 4)
> 3. test-http-server: add HTTP request parsing (prev. patch 5)
> 4. test-http-server: pass Git requests to http-backend (prev. patch 6)
> 5. test-http-server: add simple authentication (prev. patch 7)
> 6. http: read HTTP WWW-Authenticate response headers (prev. patch 1)
> 7. credential: add WWW-Authenticate header to cred requests (prev patch 2)
>   - Some/all of the tests from the current patch (patch 8) could be squashed
>     into this one so that the tests exist directly alongside the new
>     functionality they're testing.


I think that order make sense - I'll rearrange for my next iteration.
Thanks!

>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  t/helper/test-credential-helper-replay.sh |  14 +++
>>  t/t5556-http-auth.sh                      | 120 +++++++++++++++++++++-
>>  2 files changed, 133 insertions(+), 1 deletion(-)
>>  create mode 100755 t/helper/test-credential-helper-replay.sh
>>
>> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
>> new file mode 100755
>> index 00000000000..03e5e63dad6
>> --- /dev/null
>> +++ b/t/helper/test-credential-helper-replay.sh
> 
> I'm not sure a 't/helper' file is the right place for this - it's a pretty
> simple shell script, but it defines a lot of information (namely 'teefile',
> 'catfile') that is otherwise unexplained in 't5556'. 
> 
> What about something like 'lib-rebase.sh' and its 'set_fake_editor()'? You
> could create a similar test lib ('lib-credential-helper.sh') and wrapper
> function (' that writes out a custom credential helper. Something like
> 'set_fake_credential_helper()' could also take 'teefile' and 'catfile' as
> arguments, making their names more transparent to 't5556'.

The `lib-rebase.sh` script sets the fake editor by setting an environment
variable (from what I can see). Credential helpers can only be set via config
or command-line arg. Would it be easier to move writing of the test credential
helper script to the t5556 test script setup?

>> @@ -0,0 +1,14 @@
>> +cmd=$1
>> +teefile=$cmd-actual.cred
>> +catfile=$cmd-response.cred
>> +rm -f $teefile
>> +while read line;
>> +do
>> +	if test -z "$line"; then
>> +		break;
>> +	fi
>> +	echo "$line" >> $teefile
>> +done
>> +if test "$cmd" = "get"; then
>> +	cat $catfile
>> +fi
>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
>> index 78da151f122..541fa32bd77 100755
>> --- a/t/t5556-http-auth.sh
>> +++ b/t/t5556-http-auth.sh
>> @@ -26,6 +26,8 @@ PID_FILE="$(pwd)"/pid-file.pid
>>  SERVER_LOG="$(pwd)"/OUT.server.log
>>  
>>  PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
>> +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
>> +	&& export CREDENTIAL_HELPER
> 
> I see - this is how you connect the "test" credential helper to the HTTP
> server and header parsing (as implemented in patches 1 & 2), so that the
> results can be compared for correctness.
> 
> nit: you can just 'export CREDENTIAL_HELPER="..."', rather than breaking it
> into two lines. You also shouldn't need to 'export' at all - the value will
> be set in the context of the test.

I tried this originally, but got errors from one of the environments in CI that
this was not portable.

>>  
>>  test_expect_success 'setup repos' '
>>  	test_create_repo "$REPO_DIR" &&
>> @@ -91,7 +93,8 @@ start_http_server () {
>>  
>>  per_test_cleanup () {
>>  	stop_http_server &&
>> -	rm -f OUT.*
>> +	rm -f OUT.* &&
>> +	rm -f *.cred
>>  }
>>  
>>  test_expect_success 'http auth anonymous no challenge' '
>> @@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '
>>  	git ls-remote $ORIGIN_URL
>>  '
>>  
>> +test_expect_success 'http auth www-auth headers to credential helper basic valid' '
> 
> ...
> 
>> +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
> 
> ...
> 
>> +test_expect_success 'http auth www-auth headers to credential helper invalid' '
> 
> These tests all look good. That said, is there any way to test more
> bizarre/edge cases (headers too long to fit on one line, headers that end
> with a long string of whitespace, etc.)?
> 

Thanks,
Matthew

[PATCH v5 00/10] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server sharing code
with git-daemon, with simple authentication configurable by a config file.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.

Matthew John Cheetham (10):
  daemon: libify socket setup and option functions
  daemon: libify child process handling functions
  daemon: rename some esoteric/laboured terminology
  test-http-server: add stub HTTP server test helper
  test-http-server: add HTTP error response function
  test-http-server: add simple authentication
  http: replace unsafe size_t multiplication with st_mult
  strvec: expose strvec_push_nodup for external use
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt          |  19 +-
 Makefile                                  |   2 +
 contrib/buildsystems/CMakeLists.txt       |  11 +-
 credential.c                              |  13 +
 credential.h                              |  15 +
 daemon-utils.c                            | 286 +++++++
 daemon-utils.h                            |  38 +
 daemon.c                                  | 306 +------
 http.c                                    | 102 ++-
 strvec.c                                  |   2 +-
 strvec.h                                  |   3 +
 t/helper/.gitignore                       |   1 +
 t/helper/test-credential-helper-replay.sh |  14 +
 t/helper/test-http-server.c               | 920 ++++++++++++++++++++++
 t/t5556-http-auth.sh                      | 372 +++++++++
 15 files changed, 1801 insertions(+), 303 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h
 create mode 100755 t/helper/test-credential-helper-replay.sh
 create mode 100644 t/helper/test-http-server.c
 create mode 100755 t/t5556-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v4:

  -:  ----------- >  1:  74b0de14185 daemon: libify socket setup and option functions
  -:  ----------- >  2:  bc972fc8d3d daemon: libify child process handling functions
  -:  ----------- >  3:  8f176d5955d daemon: rename some esoteric/laboured terminology
  3:  07a1845ea56 !  4:  706fb3781bd test-http-server: add stub HTTP server test helper
     @@ Commit message
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Makefile ##
     -@@ Makefile: else
     - 	endif
     - 	BASIC_CFLAGS += $(CURL_CFLAGS)
     +@@ Makefile: TEST_BUILTINS_OBJS += test-xml-encode.o
     + # Do not add more tests here unless they have extra dependencies. Add
     + # them in TEST_BUILTINS_OBJS above.
     + TEST_PROGRAMS_NEED_X += test-fake-ssh
     ++TEST_PROGRAMS_NEED_X += test-http-server
     + TEST_PROGRAMS_NEED_X += test-tool
       
     -+	TEST_PROGRAMS_NEED_X += test-http-server
     -+
     - 	REMOTE_CURL_PRIMARY = git-remote-http$X
     - 	REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
     - 	REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
     + TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
      
       ## contrib/buildsystems/CMakeLists.txt ##
     +@@ contrib/buildsystems/CMakeLists.txt: if(BUILD_TESTING)
     + add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
     + target_link_libraries(test-fake-ssh common-main)
     + 
     ++add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
     ++target_link_libraries(test-http-server common-main)
     ++
     + #reftable-tests
     + parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
     + list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
     +@@ contrib/buildsystems/CMakeLists.txt: if(MSVC)
     + 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
     + 	set_target_properties(test-fake-ssh test-tool
     + 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
     ++
     ++	set_target_properties(test-http-server
     ++			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
     ++	set_target_properties(test-http-server
     ++			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
     + endif()
     + 
     + #wrapper scripts
      @@ contrib/buildsystems/CMakeLists.txt: set(wrapper_scripts
     - set(wrapper_test_scripts
     - 	test-fake-ssh test-tool)
     + 	git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
       
     -+if(CURL_FOUND)
     -+       list(APPEND wrapper_test_scripts test-http-server)
     -+
     -+       add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
     -+       target_link_libraries(test-http-server common-main)
     -+
     -+       if(MSVC)
     -+               set_target_properties(test-http-server
     -+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
     -+               set_target_properties(test-http-server
     -+                                       PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
     -+       endif()
     -+endif()
     + set(wrapper_test_scripts
     +-	test-fake-ssh test-tool)
     +-
     ++	test-http-server test-fake-ssh test-tool)
       
       foreach(script ${wrapper_scripts})
       	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
     @@ t/helper/.gitignore
      
       ## t/helper/test-http-server.c (new) ##
      @@
     ++#include "daemon-utils.h"
      +#include "config.h"
      +#include "run-command.h"
      +#include "strbuf.h"
     @@ t/helper/test-http-server.c (new)
      +
      +static const char test_http_auth_usage[] =
      +"http-server [--verbose]\n"
     -+"           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
     ++"           [--timeout=<n>] [--max-connections=<n>]\n"
      +"           [--reuseaddr] [--pid-file=<file>]\n"
      +"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
      +;
      +
     -+/* Timeout, and initial timeout */
      +static unsigned int timeout;
     -+static unsigned int init_timeout;
      +
      +static void logreport(const char *label, const char *err, va_list params)
      +{
     @@ t/helper/test-http-server.c (new)
      +	va_end(params);
      +}
      +
     -+static void set_keep_alive(int sockfd)
     -+{
     -+	int ka = 1;
     -+
     -+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
     -+		if (errno != ENOTSOCK)
     -+			logerror("unable to set SO_KEEPALIVE on socket: %s",
     -+				strerror(errno));
     -+	}
     -+}
     -+
      +/*
      + * The code in this section is used by "worker" instances to service
      + * a single connection from a client.  The worker talks to the client
     @@ t/helper/test-http-server.c (new)
      +	 * Close the socket and clean up.  Does not imply an error.
      +	 */
      +	WR_HANGUP   = 1<<1,
     -+
     -+	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
      +};
      +
      +static enum worker_result worker(void)
     @@ t/helper/test-http-server.c (new)
      +	if (client_addr)
      +		loginfo("Connection from %s:%s", client_addr, client_port);
      +
     -+	set_keep_alive(0);
     ++	set_keep_alive(0, logerror);
      +
      +	while (1) {
     -+		if (write_in_full(1, response, strlen(response)) < 0) {
     ++		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
      +			logerror("unable to write response");
      +			wr = WR_IO_ERROR;
      +		}
      +
     -+		if (wr & WR_STOP_THE_MUSIC)
     ++		if (wr != WR_OK)
      +			break;
      +	}
      +
     -+	close(0);
     -+	close(1);
     ++	close(STDIN_FILENO);
     ++	close(STDOUT_FILENO);
      +
      +	return !!(wr & WR_IO_ERROR);
      +}
      +
     -+/*
     -+ * This section contains the listener and child-process management
     -+ * code used by the primary instance to accept incoming connections
     -+ * and dispatch them to async child process "worker" instances.
     -+ */
     -+
     -+static int addrcmp(const struct sockaddr_storage *s1,
     -+		   const struct sockaddr_storage *s2)
     -+{
     -+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
     -+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
     -+
     -+	if (sa1->sa_family != sa2->sa_family)
     -+		return sa1->sa_family - sa2->sa_family;
     -+	if (sa1->sa_family == AF_INET)
     -+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
     -+		    &((struct sockaddr_in *)s2)->sin_addr,
     -+		    sizeof(struct in_addr));
     -+#ifndef NO_IPV6
     -+	if (sa1->sa_family == AF_INET6)
     -+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
     -+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
     -+		    sizeof(struct in6_addr));
     -+#endif
     -+	return 0;
     -+}
     -+
      +static int max_connections = 32;
      +
      +static unsigned int live_children;
      +
     -+static struct child {
     -+	struct child *next;
     -+	struct child_process cld;
     -+	struct sockaddr_storage address;
     -+} *firstborn;
     -+
     -+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
     -+{
     -+	struct child *newborn, **cradle;
     -+
     -+	newborn = xcalloc(1, sizeof(*newborn));
     -+	live_children++;
     -+	memcpy(&newborn->cld, cld, sizeof(*cld));
     -+	memcpy(&newborn->address, addr, addrlen);
     -+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
     -+		if (!addrcmp(&(*cradle)->address, &newborn->address))
     -+			break;
     -+	newborn->next = *cradle;
     -+	*cradle = newborn;
     -+}
     -+
     -+/*
     -+ * This gets called if the number of connections grows
     -+ * past "max_connections".
     -+ *
     -+ * We kill the newest connection from a duplicate IP.
     -+ */
     -+static void kill_some_child(void)
     -+{
     -+	const struct child *blanket, *next;
     -+
     -+	if (!(blanket = firstborn))
     -+		return;
     -+
     -+	for (; (next = blanket->next); blanket = next)
     -+		if (!addrcmp(&blanket->address, &next->address)) {
     -+			kill(blanket->cld.pid, SIGTERM);
     -+			break;
     -+		}
     -+}
     -+
     -+static void check_dead_children(void)
     -+{
     -+	int status;
     -+	pid_t pid;
     -+
     -+	struct child **cradle, *blanket;
     -+	for (cradle = &firstborn; (blanket = *cradle);)
     -+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
     -+			const char *dead = "";
     -+			if (status)
     -+				dead = " (with error)";
     -+			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
     -+
     -+			/* remove the child */
     -+			*cradle = blanket->next;
     -+			live_children--;
     -+			child_process_clear(&blanket->cld);
     -+			free(blanket);
     -+		} else
     -+			cradle = &blanket->next;
     -+}
     ++static struct child *first_child;
      +
      +static struct strvec cld_argv = STRVEC_INIT;
      +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
     @@ t/helper/test-http-server.c (new)
      +	struct child_process cld = CHILD_PROCESS_INIT;
      +
      +	if (max_connections && live_children >= max_connections) {
     -+		kill_some_child();
     ++		kill_some_child(first_child);
      +		sleep(1);  /* give it some time to die */
     -+		check_dead_children();
     ++		check_dead_children(first_child, &live_children, loginfo);
      +		if (live_children >= max_connections) {
      +			close(incoming);
      +			logerror("Too many children, dropping connection");
     @@ t/helper/test-http-server.c (new)
      +	else if (start_command(&cld))
      +		logerror("unable to fork");
      +	else
     -+		add_child(&cld, addr, addrlen);
     ++		add_child(&cld, addr, addrlen, first_child, &live_children);
      +}
      +
      +static void child_handler(int signo)
     @@ t/helper/test-http-server.c (new)
      +	signal(SIGCHLD, child_handler);
      +}
      +
     -+static int set_reuse_addr(int sockfd)
     -+{
     -+	int on = 1;
     -+
     -+	if (!reuseaddr)
     -+		return 0;
     -+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
     -+			  &on, sizeof(on));
     -+}
     -+
     -+struct socketlist {
     -+	int *list;
     -+	size_t nr;
     -+	size_t alloc;
     -+};
     -+
     -+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
     -+{
     -+#ifdef NO_IPV6
     -+	static char ip[INET_ADDRSTRLEN];
     -+#else
     -+	static char ip[INET6_ADDRSTRLEN];
     -+#endif
     -+
     -+	switch (family) {
     -+#ifndef NO_IPV6
     -+	case AF_INET6:
     -+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
     -+		break;
     -+#endif
     -+	case AF_INET:
     -+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
     -+		break;
     -+	default:
     -+		xsnprintf(ip, sizeof(ip), "<unknown>");
     -+	}
     -+	return ip;
     -+}
     -+
     -+#ifndef NO_IPV6
     -+
     -+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
     -+{
     -+	int socknum = 0;
     -+	char pbuf[NI_MAXSERV];
     -+	struct addrinfo hints, *ai0, *ai;
     -+	int gai;
     -+	long flags;
     -+
     -+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
     -+	memset(&hints, 0, sizeof(hints));
     -+	hints.ai_family = AF_UNSPEC;
     -+	hints.ai_socktype = SOCK_STREAM;
     -+	hints.ai_protocol = IPPROTO_TCP;
     -+	hints.ai_flags = AI_PASSIVE;
     -+
     -+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
     -+	if (gai) {
     -+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
     -+		return 0;
     -+	}
     -+
     -+	for (ai = ai0; ai; ai = ai->ai_next) {
     -+		int sockfd;
     -+
     -+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
     -+		if (sockfd < 0)
     -+			continue;
     -+		if (sockfd >= FD_SETSIZE) {
     -+			logerror("Socket descriptor too large");
     -+			close(sockfd);
     -+			continue;
     -+		}
     -+
     -+#ifdef IPV6_V6ONLY
     -+		if (ai->ai_family == AF_INET6) {
     -+			int on = 1;
     -+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
     -+				   &on, sizeof(on));
     -+			/* Note: error is not fatal */
     -+		}
     -+#endif
     -+
     -+		if (set_reuse_addr(sockfd)) {
     -+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
     -+			close(sockfd);
     -+			continue;
     -+		}
     -+
     -+		set_keep_alive(sockfd);
     -+
     -+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
     -+			logerror("Could not bind to %s: %s",
     -+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
     -+				 strerror(errno));
     -+			close(sockfd);
     -+			continue;	/* not fatal */
     -+		}
     -+		if (listen(sockfd, 5) < 0) {
     -+			logerror("Could not listen to %s: %s",
     -+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
     -+				 strerror(errno));
     -+			close(sockfd);
     -+			continue;	/* not fatal */
     -+		}
     -+
     -+		flags = fcntl(sockfd, F_GETFD, 0);
     -+		if (flags >= 0)
     -+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
     -+
     -+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
     -+		socklist->list[socklist->nr++] = sockfd;
     -+		socknum++;
     -+	}
     -+
     -+	freeaddrinfo(ai0);
     -+
     -+	return socknum;
     -+}
     -+
     -+#else /* NO_IPV6 */
     -+
     -+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
     -+{
     -+	struct sockaddr_in sin;
     -+	int sockfd;
     -+	long flags;
     -+
     -+	memset(&sin, 0, sizeof sin);
     -+	sin.sin_family = AF_INET;
     -+	sin.sin_port = htons(listen_port);
     -+
     -+	if (listen_addr) {
     -+		/* Well, host better be an IP address here. */
     -+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
     -+			return 0;
     -+	} else {
     -+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
     -+	}
     -+
     -+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
     -+	if (sockfd < 0)
     -+		return 0;
     -+
     -+	if (set_reuse_addr(sockfd)) {
     -+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
     -+		close(sockfd);
     -+		return 0;
     -+	}
     -+
     -+	set_keep_alive(sockfd);
     -+
     -+	if (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {
     -+		logerror("Could not bind to %s: %s",
     -+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
     -+			 strerror(errno));
     -+		close(sockfd);
     -+		return 0;
     -+	}
     -+
     -+	if (listen(sockfd, 5) < 0) {
     -+		logerror("Could not listen to %s: %s",
     -+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
     -+			 strerror(errno));
     -+		close(sockfd);
     -+		return 0;
     -+	}
     -+
     -+	flags = fcntl(sockfd, F_GETFD, 0);
     -+	if (flags >= 0)
     -+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
     -+
     -+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
     -+	socklist->list[socklist->nr++] = sockfd;
     -+	return 1;
     -+}
     -+
     -+#endif
     -+
     -+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
     -+{
     -+	if (!listen_addr->nr)
     -+		setup_named_sock("127.0.0.1", listen_port, socklist);
     -+	else {
     -+		int i, socknum;
     -+		for (i = 0; i < listen_addr->nr; i++) {
     -+			socknum = setup_named_sock(listen_addr->items[i].string,
     -+						   listen_port, socklist);
     -+
     -+			if (socknum == 0)
     -+				logerror("unable to allocate any listen sockets for host %s on port %u",
     -+					 listen_addr->items[i].string, listen_port);
     -+		}
     -+	}
     -+}
     -+
      +static int service_loop(struct socketlist *socklist)
      +{
      +	struct pollfd *pfd;
     @@ t/helper/test-http-server.c (new)
      +		int nr_ready;
      +		int timeout = (pid_file ? 100 : -1);
      +
     -+		check_dead_children();
     ++		check_dead_children(first_child, &live_children, loginfo);
      +
      +		nr_ready = poll(pfd, socklist->nr, timeout);
      +		if (nr_ready < 0) {
     @@ t/helper/test-http-server.c (new)
      +{
      +	struct socketlist socklist = { NULL, 0, 0 };
      +
     -+	socksetup(listen_addr, listen_port, &socklist);
     ++	socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
      +	if (socklist.nr == 0)
      +		die("unable to allocate any listen sockets on port %u",
      +		    listen_port);
     @@ t/helper/test-http-server.c (new)
      +	int i;
      +
      +	trace2_cmd_name("test-http-server");
     ++	trace2_cmd_list_config();
     ++	trace2_cmd_list_env_vars();
      +	setup_git_directory_gently(NULL);
      +
      +	for (i = 1; i < argc; i++) {
     @@ t/helper/test-http-server.c (new)
      +			timeout = atoi(v);
      +			continue;
      +		}
     -+		if (skip_prefix(arg, "--init-timeout=", &v)) {
     -+			init_timeout = atoi(v);
     -+			continue;
     -+		}
      +		if (skip_prefix(arg, "--max-connections=", &v)) {
      +			max_connections = atoi(v);
      +			if (max_connections < 0)
  5:  5c4e36e23ee !  5:  6f66bf146b4 test-http-server: add HTTP request parsing
     @@ Metadata
      Author: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Commit message ##
     -    test-http-server: add HTTP request parsing
     +    test-http-server: add HTTP error response function
      
     -    Add ability to parse HTTP requests to the test-http-server test helper.
     +    Introduce a function to the test-http-server test helper to write more
     +    full and valid HTTP error responses, including all the standard response
     +    headers like `Server` and `Date`.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## t/helper/test-http-server.c ##
      @@ t/helper/test-http-server.c: enum worker_result {
     - 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
     + 	WR_HANGUP   = 1<<1,
       };
       
      +/*
     @@ t/helper/test-http-server.c: enum worker_result {
      +	string_list_clear(&req->header_list, 0);
      +}
      +
     - static enum worker_result send_http_error(
     - 	int fd,
     - 	int http_code, const char *http_code_name,
     -@@ t/helper/test-http-server.c: done:
     - 	return wr;
     - }
     - 
     ++static enum worker_result send_http_error(
     ++	int fd,
     ++	int http_code, const char *http_code_name,
     ++	int retry_after_seconds, struct string_list *response_headers,
     ++	enum worker_result wr_in)
     ++{
     ++	struct strbuf response_header = STRBUF_INIT;
     ++	struct strbuf response_content = STRBUF_INIT;
     ++	struct string_list_item *h;
     ++	enum worker_result wr;
     ++
     ++	strbuf_addf(&response_content, "Error: %d %s\r\n",
     ++		    http_code, http_code_name);
     ++	if (retry_after_seconds > 0)
     ++		strbuf_addf(&response_content, "Retry-After: %d\r\n",
     ++			    retry_after_seconds);
     ++
     ++	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
     ++	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
     ++	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
     ++	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
     ++	if (retry_after_seconds > 0)
     ++		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
     ++	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
     ++	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
     ++	if (response_headers)
     ++		for_each_string_list_item(h, response_headers)
     ++			strbuf_addf(&response_header, "%s\r\n", h->string);
     ++	strbuf_addstr(&response_header, "\r\n");
     ++
     ++	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
     ++		logerror("unable to write response header");
     ++		wr = WR_IO_ERROR;
     ++		goto done;
     ++	}
     ++
     ++	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
     ++		logerror("unable to write response content body");
     ++		wr = WR_IO_ERROR;
     ++		goto done;
     ++	}
     ++
     ++	wr = wr_in;
     ++
     ++done:
     ++	strbuf_release(&response_header);
     ++	strbuf_release(&response_content);
     ++
     ++	return wr;
     ++}
     ++
      +/*
      + * Read the HTTP request up to the start of the optional message-body.
      + * We do this byte-by-byte because we have keep-alive turned on and
     @@ t/helper/test-http-server.c: done:
      +		hp = strbuf_detach(&h, NULL);
      +		string_list_append(&req->header_list, hp);
      +
     -+		/* store common request headers separately */
     ++		/* also store common request headers as struct req members */
      +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
      +			req->content_type = hv;
      +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
     @@ t/helper/test-http-server.c: done:
      +	return result;
      +}
      +
     ++static int is_git_request(struct req *req)
     ++{
     ++	static regex_t *smart_http_regex;
     ++	static int initialized;
     ++
     ++	if (!initialized) {
     ++		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
     ++		/*
     ++		 * This regular expression matches all dumb and smart HTTP
     ++		 * requests that are currently in use, and defined in
     ++		 * Documentation/gitprotocol-http.txt.
     ++		 *
     ++		 */
     ++		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
     ++			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
     ++			    REG_EXTENDED)) {
     ++			warning("could not compile smart HTTP regex");
     ++			smart_http_regex = NULL;
     ++		}
     ++		initialized = 1;
     ++	}
     ++
     ++	return smart_http_regex &&
     ++		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
     ++}
     ++
     ++static enum worker_result do__git(struct req *req)
     ++{
     ++	const char *ok = "HTTP/1.1 200 OK\r\n";
     ++	struct child_process cp = CHILD_PROCESS_INIT;
     ++	int res;
     ++
     ++	/*
     ++	 * Note that we always respond with a 200 OK response even if the
     ++	 * http-backend process exits with an error. This helper is intended
     ++	 * only to be used to exercise the HTTP auth handling in the Git client,
     ++	 * and specifically around authentication (not handled by http-backend).
     ++	 *
     ++	 * If we wanted to respond with a more 'valid' HTTP response status then
     ++	 * we'd need to buffer the output of http-backend, wait for and grok the
     ++	 * exit status of the process, then write the HTTP status line followed
     ++	 * by the http-backend output. This is outside of the scope of this test
     ++	 * helper's use at time of writing.
     ++	 *
     ++	 * The important auth responses (401) we are handling prior to getting
     ++	 * to this point.
     ++	 */
     ++	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
     ++		return error(_("could not send '%s'"), ok);
     ++
     ++	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     ++	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     ++			req->uri_path.buf);
     ++	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
     ++	if (req->query_args.len)
     ++		strvec_pushf(&cp.env, "QUERY_STRING=%s",
     ++				req->query_args.buf);
     ++	if (req->content_type)
     ++		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
     ++				req->content_type);
     ++	if (req->content_length >= 0)
     ++		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
     ++				(intmax_t)req->content_length);
     ++	cp.git_cmd = 1;
     ++	strvec_push(&cp.args, "http-backend");
     ++	res = run_command(&cp);
     ++	close(STDOUT_FILENO);
     ++	close(STDIN_FILENO);
     ++	return !!res;
     ++}
     ++
      +static enum worker_result dispatch(struct req *req)
      +{
     -+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
     ++	if (is_git_request(req))
     ++		return do__git(req);
     ++
     ++	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
      +			       WR_OK | WR_HANGUP);
      +}
      +
       static enum worker_result worker(void)
       {
     +-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
      +	struct req req = REQ__INIT;
       	char *client_addr = getenv("REMOTE_ADDR");
       	char *client_port = getenv("REMOTE_PORT");
       	enum worker_result wr = WR_OK;
      @@ t/helper/test-http-server.c: static enum worker_result worker(void)
     - 	set_keep_alive(0);
     + 	set_keep_alive(0, logerror);
       
       	while (1) {
     --		wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
     --			WR_OK | WR_HANGUP);
     +-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
     +-			logerror("unable to write response");
     +-			wr = WR_IO_ERROR;
     +-		}
      +		req__release(&req);
      +
     -+		alarm(init_timeout ? init_timeout : timeout);
     ++		alarm(timeout);
      +		wr = req__read(&req, 0);
      +		alarm(0);
      +
     -+		if (wr & WR_STOP_THE_MUSIC)
     ++		if (wr != WR_OK)
      +			break;
     -+
     + 
      +		wr = dispatch(&req);
     - 		if (wr & WR_STOP_THE_MUSIC)
     + 		if (wr != WR_OK)
       			break;
       	}
     +
     + ## t/t5556-http-auth.sh (new) ##
     +@@
     ++#!/bin/sh
     ++
     ++test_description='test http auth header and credential helper interop'
     ++
     ++TEST_NO_CREATE_REPO=1
     ++. ./test-lib.sh
     ++
     ++test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
     ++
     ++# Setup a repository
     ++#
     ++REPO_DIR="$TRASH_DIRECTORY"/repo
     ++
     ++# Setup some lookback URLs where test-http-server will be listening.
     ++# We will spawn it directly inside the repo directory, so we avoid
     ++# any need to configure directory mappings etc - we only serve this
     ++# repository from the root '/' of the server.
     ++#
     ++HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
     ++ORIGIN_URL=http://$HOST_PORT/
     ++
     ++# The pid-file is created by test-http-server when it starts.
     ++# The server will shutdown if/when we delete it (this is easier than
     ++# killing it by PID).
     ++#
     ++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
     ++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     ++
     ++PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     ++
     ++test_expect_success 'setup repos' '
     ++	test_create_repo "$REPO_DIR" &&
     ++	git -C "$REPO_DIR" branch -M main
     ++'
     ++
     ++stop_http_server () {
     ++	if ! test -f "$PID_FILE"
     ++	then
     ++		return 0
     ++	fi
     ++	#
     ++	# The server will shutdown automatically when we delete the pid-file.
     ++	#
     ++	rm -f "$PID_FILE"
     ++	#
     ++	# Give it a few seconds to shutdown (mainly to completely release the
     ++	# port before the next test start another instance and it attempts to
     ++	# bind to it).
     ++	#
     ++	for k in 0 1 2 3 4
     ++	do
     ++		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
     ++		then
     ++			return 0
     ++		fi
     ++		sleep 1
     ++	done
     ++
     ++	echo "stop_http_server: timeout waiting for server shutdown"
     ++	return 1
     ++}
     ++
     ++start_http_server () {
     ++	#
     ++	# Launch our server into the background in repo_dir.
     ++	#
     ++	(
     ++		cd "$REPO_DIR"
     ++		test-http-server --verbose \
     ++			--listen=127.0.0.1 \
     ++			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
     ++			--reuseaddr \
     ++			--pid-file="$PID_FILE" \
     ++			"$@" \
     ++			2>"$SERVER_LOG" &
     ++	)
     ++	#
     ++	# Give it a few seconds to get started.
     ++	#
     ++	for k in 0 1 2 3 4
     ++	do
     ++		if test -f "$PID_FILE"
     ++		then
     ++			return 0
     ++		fi
     ++		sleep 1
     ++	done
     ++
     ++	echo "start_http_server: timeout waiting for server startup"
     ++	return 1
     ++}
     ++
     ++per_test_cleanup () {
     ++	stop_http_server &&
     ++	rm -f OUT.*
     ++}
     ++
     ++test_expect_success 'http auth anonymous no challenge' '
     ++	test_when_finished "per_test_cleanup" &&
     ++	start_http_server &&
     ++
     ++	# Attempt to read from a protected repository
     ++	git ls-remote $ORIGIN_URL
     ++'
     ++
     ++test_done
  7:  794256754c1 !  6:  c3c3d17a688 test-http-server: add simple authentication
     @@ Commit message
      
          Add simple authentication to the test-http-server test helper.
          Authentication schemes and sets of valid tokens can be specified via
     -    command-line arguments. Incoming requests are compared against the set
     -    of valid schemes and tokens and only approved if a matching token is
     -    found, or if no auth was provided and anonymous auth is enabled.
     +    a configuration file (in the normal gitconfig file format).
     +    Incoming requests are compared against the set of valid schemes and
     +    tokens and only approved if a matching token is found, or if no auth
     +    was provided and anonymous auth is enabled.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## t/helper/test-http-server.c ##
     +@@
     + #include "version.h"
     + #include "dir.h"
     + #include "date.h"
     ++#include "config.h"
     + 
     + #define TR2_CAT "test-http-server"
     + 
      @@ t/helper/test-http-server.c: static const char test_http_auth_usage[] =
     - "           [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
     + "           [--timeout=<n>] [--max-connections=<n>]\n"
       "           [--reuseaddr] [--pid-file=<file>]\n"
       "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
     -+"           [--anonymous-allowed]\n"
     -+"           [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
     ++"           [--auth-config=<file>]\n"
       ;
       
     - /* Timeout, and initial timeout */
     -@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req, const char *user)
     + static unsigned int timeout;
     +@@ t/helper/test-http-server.c: static int is_git_request(struct req *req)
     + 		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
     + }
     + 
     +-static enum worker_result do__git(struct req *req)
     ++static enum worker_result do__git(struct req *req, const char *user)
     + {
     + 	const char *ok = "HTTP/1.1 200 OK\r\n";
     + 	struct child_process cp = CHILD_PROCESS_INIT;
     +@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
     + 	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
     + 		return error(_("could not send '%s'"), ok);
     + 
     ++	if (user)
     ++		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
     ++
     + 	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     + 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     + 			req->uri_path.buf);
     +@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
       	return !!res;
       }
       
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +static struct auth_module **auth_modules = NULL;
      +static size_t auth_modules_nr = 0;
      +static size_t auth_modules_alloc = 0;
     ++static struct strvec extra_headers = STRVEC_INIT;
     ++
     ++static struct auth_module *create_auth_module(const char *scheme,
     ++					      const char *challenge)
     ++{
     ++	struct auth_module *mod = xmalloc(sizeof(struct auth_module));
     ++	mod->scheme = xstrdup(scheme);
     ++	mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
     ++	CALLOC_ARRAY(mod->tokens, 1);
     ++	string_list_init_dup(mod->tokens);
     ++	return mod;
     ++}
      +
      +static struct auth_module *get_auth_module(const char *scheme)
      +{
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +	return NULL;
      +}
      +
     -+static void add_auth_module(struct auth_module *mod)
     ++static int add_auth_module(struct auth_module *mod)
      +{
     ++	if (get_auth_module(mod->scheme))
     ++		return error("duplicate auth scheme '%s'\n", mod->scheme);
     ++
      +	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
      +	auth_modules[auth_modules_nr++] = mod;
     ++
     ++	return 0;
      +}
      +
      +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +
      +	/*
      +	 * Check all auth modules and try to validate the request.
     -+	 * The first module that matches a valid token approves the request.
     ++	 * The first Authorization header that matches a known auth module
     ++	 * scheme will be consulted to either approve or deny the request.
      +	 * If no module is found, or if there is no valid token, then 401 error.
      +	 * Otherwise, only permit the request if anonymous auth is enabled.
     ++	 * It's atypical for user agents/clients to send multiple Authorization
     ++	 * headers, but not explicitly forbidden or defined.
      +	 */
      +	for_each_string_list_item(hdr, &req->header_list) {
      +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +	case AUTH_UNKNOWN:
      +		if (result != AUTH_DENY && allow_anonymous)
      +			break;
     ++
      +		for (i = 0; i < auth_modules_nr; i++) {
      +			mod = auth_modules[i];
      +			if (mod->challenge_params)
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +						    mod->scheme);
      +			string_list_append(&hdrs, challenge);
      +		}
     -+		*wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
     ++
     ++		for (i = 0; i < extra_headers.nr; i++)
     ++			string_list_append(&hdrs, extra_headers.v[i]);
     ++
     ++		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
     ++				      &hdrs, *wr);
      +	}
      +
      +	strbuf_list_free(split);
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +	return result == AUTH_ALLOW ||
      +	      (result == AUTH_UNKNOWN && allow_anonymous);
      +}
     ++
     ++static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
     ++{
     ++	struct strbuf **p = strbuf_split_str(str, ':', 2);
     ++
     ++	if (!p[0])
     ++		return -1;
     ++
     ++	/* trim trailing ':' */
     ++	if (p[1])
     ++		strbuf_setlen(p[0], p[0]->len - 1);
     ++
     ++	if (required_val && !p[1])
     ++		return -1;
     ++
     ++	*scheme = strbuf_detach(p[0], NULL);
     ++
     ++	if (p[1])
     ++		*val = strbuf_detach(p[1], NULL);
     ++
     ++	strbuf_list_free(p);
     ++	return 0;
     ++}
     ++
     ++static int read_auth_config(const char *name, const char *val, void *data)
     ++{
     ++	int ret = 0;
     ++	char *scheme = NULL;
     ++	char *token = NULL;
     ++	char *challenge = NULL;
     ++	struct auth_module *mod = NULL;
     ++
     ++	if (!strcmp(name, "auth.challenge")) {
     ++		if (split_auth_param(val, &scheme, &challenge, 0)) {
     ++			ret = error("invalid auth challenge '%s'", val);
     ++			goto cleanup;
     ++		}
     ++
     ++		mod = create_auth_module(scheme, challenge);
     ++		if (add_auth_module(mod)) {
     ++			ret = error("failed to add auth module '%s'", val);
     ++			goto cleanup;
     ++		}
     ++	}
     ++	if (!strcmp(name, "auth.token")) {
     ++		if (split_auth_param(val, &scheme, &token, 1)) {
     ++			ret = error("invalid auth token '%s'", val);
     ++			goto cleanup;
     ++		}
     ++
     ++		mod = get_auth_module(scheme);
     ++		if (!mod) {
     ++			ret = error("auth scheme not defined '%s'\n", scheme);
     ++			goto cleanup;
     ++		}
     ++
     ++		string_list_append(mod->tokens, token);
     ++	}
     ++	if (!strcmp(name, "auth.allowanonymous")) {
     ++		allow_anonymous = git_config_bool(name, val);
     ++	}
     ++	if (!strcmp(name, "auth.extraheader")) {
     ++		strvec_push(&extra_headers, val);
     ++	}
     ++
     ++cleanup:
     ++	free(scheme);
     ++	free(token);
     ++	free(challenge);
     ++
     ++	return ret;
     ++}
      +
       static enum worker_result dispatch(struct req *req)
       {
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
      +		return wr;
      +
       	if (is_git_request(req))
     --		return do__git(req, NULL);
     +-		return do__git(req);
      +		return do__git(req, user);
       
     - 	return send_http_error(1, 501, "Not Implemented", -1, NULL,
     + 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
       			       WR_OK | WR_HANGUP);
     -@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
     - 	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
     - 	int worker_mode = 0;
     - 	int i;
     -+	struct auth_module *mod = NULL;
     - 
     - 	trace2_cmd_name("test-http-server");
     - 	setup_git_directory_gently(NULL);
      @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
       			pid_file = v;
       			continue;
       		}
     -+		if (skip_prefix(arg, "--allow-anonymous", &v)) {
     -+			allow_anonymous = 1;
     -+			continue;
     -+		}
     -+		if (skip_prefix(arg, "--auth=", &v)) {
     -+			struct strbuf **p = strbuf_split_str(v, ':', 2);
     -+
     -+			if (!p[0]) {
     -+				error("invalid argument '%s'", v);
     ++		if (skip_prefix(arg, "--auth-config=", &v)) {
     ++			if (!strlen(v)) {
     ++				error("invalid argument - missing file path");
      +				usage(test_http_auth_usage);
      +			}
      +
     -+			/* trim trailing ':' */
     -+			if (p[1])
     -+				strbuf_setlen(p[0], p[0]->len - 1);
     -+
     -+			if (get_auth_module(p[0]->buf)) {
     -+				error("duplicate auth scheme '%s'\n", p[0]->buf);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			mod = xmalloc(sizeof(struct auth_module));
     -+			mod->scheme = xstrdup(p[0]->buf);
     -+			mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
     -+			CALLOC_ARRAY(mod->tokens, 1);
     -+			string_list_init_dup(mod->tokens);
     -+
     -+			add_auth_module(mod);
     -+
     -+			strbuf_list_free(p);
     -+			continue;
     -+		}
     -+		if (skip_prefix(arg, "--auth-token=", &v)) {
     -+			struct strbuf **p = strbuf_split_str(v, ':', 2);
     -+			if (!p[0]) {
     -+				error("invalid argument '%s'", v);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			if (!p[1]) {
     -+				error("missing token value '%s'\n", v);
     -+				usage(test_http_auth_usage);
     -+			}
     -+
     -+			/* trim trailing ':' */
     -+			strbuf_setlen(p[0], p[0]->len - 1);
     -+
     -+			mod = get_auth_module(p[0]->buf);
     -+			if (!mod) {
     -+				error("auth scheme not defined '%s'\n", p[0]->buf);
     ++			if (git_config_from_file(read_auth_config, v, NULL)) {
     ++				error("failed to read auth config file '%s'", v);
      +				usage(test_http_auth_usage);
      +			}
      +
     -+			string_list_append(mod->tokens, p[1]->buf);
     -+			strbuf_list_free(p);
      +			continue;
      +		}
       
  -:  ----------- >  7:  9c4d25945dd http: replace unsafe size_t multiplication with st_mult
  -:  ----------- >  8:  65a620b08ef strvec: expose strvec_push_nodup for external use
  1:  b5b56ccd941 !  9:  bcfec529d95 http: read HTTP WWW-Authenticate response headers
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
       
      +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
      +{
     -+	size_t size = eltsize * nmemb;
     ++	size_t size = st_mult(eltsize, nmemb);
      +	struct strvec *values = &http_auth.wwwauth_headers;
      +	struct strbuf buf = STRBUF_INIT;
      +	const char *val;
     -+	const char *z = NULL;
      +
      +	/*
      +	 * Header lines may not come NULL-terminated from libcurl so we must
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * This line could be a continuation of the previously matched header
      +	 * field. If this is the case then we should append this value to the
      +	 * end of the previously consumed value.
     ++	 * Continuation lines start with at least one whitespace, maybe more,
     ++	 * so we should collapse these down to a single SP (valid per the spec).
      +	 */
      +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
     -+		const char **v = values->v + values->nr - 1;
     -+		char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
     ++		/* Trim leading whitespace from this continuation hdr line. */
     ++		strbuf_ltrim(&buf);
      +
     -+		free((void*)*v);
     -+		*v = append;
     ++		/*
     ++		 * At this point we should always have at least one existing
     ++		 * value, even if it is empty. Do not bother appending the new
     ++		 * value if this continuation header is itself empty.
     ++		 */
     ++		if (!values->nr) {
     ++			BUG("should have at least one existing header value");
     ++		} else if (buf.len) {
     ++			const char *prev = values->v[values->nr - 1];
     ++			struct strbuf append = STRBUF_INIT;
     ++			strbuf_addstr(&append, prev);
     ++
     ++			/* Join two non-empty values with a single space. */
     ++			if (append.len)
     ++				strbuf_addch(&append, ' ');
     ++
     ++			strbuf_addbuf(&append, &buf);
     ++
     ++			strvec_pop(values);
     ++			strvec_push_nodup(values, strbuf_detach(&append, NULL));
     ++		}
      +
      +		goto exit;
      +	}
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * We only care about the last HTTP request response's headers so clear
      +	 * the existing array.
      +	 */
     -+	if (skip_iprefix(buf.buf, "http/", &z))
     ++	if (istarts_with(buf.buf, "http/"))
      +		strvec_clear(values);
      +
      +exit:
  2:  d02875dda7c <  -:  ----------- credential: add WWW-Authenticate header to cred requests
  4:  98dd286db7c <  -:  ----------- test-http-server: add HTTP error response function
  6:  0a0f4fd10c8 <  -:  ----------- test-http-server: pass Git requests to http-backend
  8:  8ecf6383522 ! 10:  af66d2d2ede t5556: add HTTP authentication tests
     @@ Metadata
      Author: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Commit message ##
     -    t5556: add HTTP authentication tests
     +    credential: add WWW-Authenticate header to cred requests
      
     -    Add a series of tests to exercise the HTTP authentication header parsing
     +    Add the value of the WWW-Authenticate response header to credential
     +    requests. Credential helpers that understand and support HTTP
     +    authentication and authorization can use this standard header (RFC 2616
     +    Section 14.47 [1]) to generate valid credentials.
     +
     +    WWW-Authenticate headers can contain information pertaining to the
     +    authority, authentication mechanism, or extra parameters/scopes that are
     +    required.
     +
     +    The current I/O format for credential helpers only allows for unique
     +    names for properties/attributes, so in order to transmit multiple header
     +    values (with a specific order) we introduce a new convention whereby a
     +    C-style array syntax is used in the property name to denote multiple
     +    ordered values for the same property.
     +
     +    In this case we send multiple `wwwauth[]` properties where the order
     +    that the repeated attributes appear in the conversation reflects the
     +    order that the WWW-Authenticate headers appeared in the HTTP response.
     +
     +    Add a set of tests to exercise the HTTP authentication header parsing
          and the interop with credential helpers. Credential helpers will receive
          WWW-Authenticate information in credential requests.
      
     +    [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
     +
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
     + ## Documentation/git-credential.txt ##
     +@@ Documentation/git-credential.txt: separated by an `=` (equals) sign, followed by a newline.
     + The key may contain any bytes except `=`, newline, or NUL. The value may
     + contain any bytes except newline or NUL.
     + 
     +-In both cases, all bytes are treated as-is (i.e., there is no quoting,
     ++Attributes with keys that end with C-style array brackets `[]` can have
     ++multiple values. Each instance of a multi-valued attribute forms an
     ++ordered list of values - the order of the repeated attributes defines
     ++the order of the values. An empty multi-valued attribute (`key[]=\n`)
     ++acts to clear any previous entries and reset the list.
     ++
     ++In all cases, all bytes are treated as-is (i.e., there is no quoting,
     + and one cannot transmit a value with newline or NUL in it). The list of
     + attributes is terminated by a blank line or end-of-file.
     + 
     +@@ Documentation/git-credential.txt: empty string.
     + Components which are missing from the URL (e.g., there is no
     + username in the example above) will be left unset.
     + 
     ++`wwwauth[]`::
     ++
     ++	When an HTTP response is received by Git that includes one or more
     ++	'WWW-Authenticate' authentication headers, these will be passed by Git
     ++	to credential helpers.
     +++
     ++Each 'WWW-Authenticate' header value is passed as a multi-valued
     ++attribute 'wwwauth[]', where the order of the attributes is the same as
     ++they appear in the HTTP response. This attribute is 'one-way' from Git
     ++to pass additional information to credential helpers.
     ++
     + Unrecognised attributes are silently discarded.
     + 
     + GIT
     +
     + ## credential.c ##
     +@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
     + 	fprintf(fp, "%s=%s\n", key, value);
     + }
     + 
     ++static void credential_write_strvec(FILE *fp, const char *key,
     ++				    const struct strvec *vec)
     ++{
     ++	int i = 0;
     ++	const char *full_key = xstrfmt("%s[]", key);
     ++	for (; i < vec->nr; i++) {
     ++		credential_write_item(fp, full_key, vec->v[i], 0);
     ++	}
     ++	free((void*)full_key);
     ++}
     ++
     + void credential_write(const struct credential *c, FILE *fp)
     + {
     + 	credential_write_item(fp, "protocol", c->protocol, 1);
     +@@ credential.c: void credential_write(const struct credential *c, FILE *fp)
     + 	credential_write_item(fp, "path", c->path, 0);
     + 	credential_write_item(fp, "username", c->username, 0);
     + 	credential_write_item(fp, "password", c->password, 0);
     ++	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
     + }
     + 
     + static int run_credential_helper(struct credential *c,
     +
       ## t/helper/test-credential-helper-replay.sh (new) ##
      @@
      +cmd=$1
     @@ t/helper/test-credential-helper-replay.sh (new)
      +fi
      
       ## t/t5556-http-auth.sh ##
     -@@ t/t5556-http-auth.sh: PID_FILE="$(pwd)"/pid-file.pid
     - SERVER_LOG="$(pwd)"/OUT.server.log
     +@@ t/t5556-http-auth.sh: PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
     + SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
       
       PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
      +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
     @@ t/t5556-http-auth.sh: start_http_server () {
       	stop_http_server &&
      -	rm -f OUT.*
      +	rm -f OUT.* &&
     -+	rm -f *.cred
     ++	rm -f *.cred &&
     ++	rm -f auth.config
       }
       
       test_expect_success 'http auth anonymous no challenge' '
     -@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
     + 	test_when_finished "per_test_cleanup" &&
     +-	start_http_server &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    allowAnonymous = true
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     + 
     + 	# Attempt to read from a protected repository
       	git ls-remote $ORIGIN_URL
       '
       
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
      +	export USERPASS64 &&
      +
     -+	start_http_server \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=basic:$USERPASS64 &&
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	test_cmp store-expected.cred store-actual.cred
      +'
      +
     ++test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
     ++	test_when_finished "per_test_cleanup" &&
     ++	# base64("alice:secret-passwd")
     ++	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     ++	export USERPASS64 &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	    extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     ++
     ++	cat >get-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	wwwauth[]=basic realm="example.com"
     ++	wwwauth[]=bEaRer auThoRiTy="id.example.com"
     ++	EOF
     ++
     ++	cat >store-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	cat >get-response.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     ++
     ++	test_cmp get-expected.cred get-actual.cred &&
     ++	test_cmp store-expected.cred store-actual.cred
     ++'
     ++
     ++test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
     ++	test_when_finished "per_test_cleanup" &&
     ++	# base64("alice:secret-passwd")
     ++	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     ++	export USERPASS64 &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     ++
     ++	cat >get-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     ++	wwwauth[]=basic realm="example.com"
     ++	EOF
     ++
     ++	cat >store-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	cat >get-response.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     ++
     ++	test_cmp get-expected.cred get-actual.cred &&
     ++	test_cmp store-expected.cred store-actual.cred
     ++'
     ++
     ++test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
     ++	test_when_finished "per_test_cleanup" &&
     ++	# base64("alice:secret-passwd")
     ++	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     ++	export USERPASS64 &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	    extraheader = "WWW-Authenticate:"
     ++	    extraheader = " "
     ++	    extraheader = " bearer authority=\"id.example.com\""
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     ++
     ++	cat >get-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	wwwauth[]=basic realm="example.com"
     ++	wwwauth[]=bearer authority="id.example.com"
     ++	EOF
     ++
     ++	cat >store-expected.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	cat >get-response.cred <<-EOF &&
     ++	protocol=http
     ++	host=$HOST_PORT
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++
     ++	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     ++
     ++	test_cmp get-expected.cred get-actual.cred &&
     ++	test_cmp store-expected.cred store-actual.cred
     ++'
     ++
      +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
      +	test_when_finished "per_test_cleanup" &&
      +	# base64("alice:secret-passwd")
      +	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
      +	export USERPASS64 &&
      +
     -+	start_http_server \
     -+		--auth=foobar:alg=test\ widget=1 \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=basic:$USERPASS64 &&
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = "foobar:alg=test widget=1"
     ++	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http
     @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
      +	# base64("alice:secret-passwd")
      +	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
      +	export USERPASS64 &&
     -+	start_http_server \
     -+		--auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
     -+		--auth=basic:realm=\"example.com\" \
     -+		--auth-token=basic:$USERPASS64 &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     ++	    challenge = basic:realm=\"example.com\"
     ++	    token = basic:$USERPASS64
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	cat >get-expected.cred <<-EOF &&
      +	protocol=http

-- 
gitgitgadget

[PATCH v5 01/10] daemon: libify socket setup and option functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions for setting up listening sockets and keep-alive options
from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
dependencies on global state by inlining the behaviour at the callsites
for all libified functions.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile       |   1 +
 daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h |  23 ++++++
 daemon.c       | 214 +------------------------------------------------
 4 files changed, 237 insertions(+), 210 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h

diff --git a/Makefile b/Makefile
index b258fdbed86..2654094dbb5 100644
--- a/Makefile
+++ b/Makefile
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o
 LIB_OBJS += csum-file.o
 LIB_OBJS += ctype.o
 LIB_OBJS += date.o
+LIB_OBJS += daemon-utils.o
 LIB_OBJS += decorate.o
 LIB_OBJS += delta-islands.o
 LIB_OBJS += diagnose.o
diff --git a/daemon-utils.c b/daemon-utils.c
new file mode 100644
index 00000000000..b96b55962db
--- /dev/null
+++ b/daemon-utils.c
@@ -0,0 +1,209 @@
+#include "cache.h"
+#include "daemon-utils.h"
+
+void set_keep_alive(int sockfd, log_fn logerror)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (reuseaddr && set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd, logerror);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (reuseaddr && set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd, logerror);
+
+	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror)
+{
+	if (!listen_addr->nr)
+		setup_named_sock(NULL, listen_port, socklist, reuseaddr,
+				 logerror);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist, reuseaddr,
+						   logerror);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
diff --git a/daemon-utils.h b/daemon-utils.h
new file mode 100644
index 00000000000..6710a2a6dc0
--- /dev/null
+++ b/daemon-utils.h
@@ -0,0 +1,23 @@
+#ifndef DAEMON_UTILS_H
+#define DAEMON_UTILS_H
+
+#include "git-compat-util.h"
+#include "string-list.h"
+
+typedef void (*log_fn)(const char *msg, ...);
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+/* Enable sending of keep-alive messages on the socket. */
+void set_keep_alive(int sockfd, log_fn logerror);
+
+/* Setup a number of sockets to listen on the provided addresses. */
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror);
+
+#endif
diff --git a/daemon.c b/daemon.c
index 0ae7d12b5c1..1ed4e705680 100644
--- a/daemon.c
+++ b/daemon.c
@@ -1,9 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "daemon-utils.h"
 #include "pkt-line.h"
 #include "run-command.h"
 #include "strbuf.h"
-#include "string-list.h"
 
 #ifdef NO_INITGROUPS
 #define initgroups(x, y) (0) /* nothing */
@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)
 	strbuf_release(&hi->tcp_port);
 }
 
-static void set_keep_alive(int sockfd)
-{
-	int ka = 1;
-
-	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
-		if (errno != ENOTSOCK)
-			logerror("unable to set SO_KEEPALIVE on socket: %s",
-				strerror(errno));
-	}
-}
-
 static int execute(void)
 {
 	char *line = packet_buffer;
@@ -759,7 +748,7 @@ static int execute(void)
 	if (addr)
 		loginfo("Connection from %s:%s", addr, port);
 
-	set_keep_alive(0);
+	set_keep_alive(0, logerror);
 	alarm(init_timeout ? init_timeout : timeout);
 	pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
 	alarm(0);
@@ -938,202 +927,6 @@ static void child_handler(int signo)
 	signal(SIGCHLD, child_handler);
 }
 
-static int set_reuse_addr(int sockfd)
-{
-	int on = 1;
-
-	if (!reuseaddr)
-		return 0;
-	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
-			  &on, sizeof(on));
-}
-
-struct socketlist {
-	int *list;
-	size_t nr;
-	size_t alloc;
-};
-
-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
-{
-#ifdef NO_IPV6
-	static char ip[INET_ADDRSTRLEN];
-#else
-	static char ip[INET6_ADDRSTRLEN];
-#endif
-
-	switch (family) {
-#ifndef NO_IPV6
-	case AF_INET6:
-		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
-		break;
-#endif
-	case AF_INET:
-		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
-		break;
-	default:
-		xsnprintf(ip, sizeof(ip), "<unknown>");
-	}
-	return ip;
-}
-
-#ifndef NO_IPV6
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	int socknum = 0;
-	char pbuf[NI_MAXSERV];
-	struct addrinfo hints, *ai0, *ai;
-	int gai;
-	long flags;
-
-	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
-	memset(&hints, 0, sizeof(hints));
-	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = SOCK_STREAM;
-	hints.ai_protocol = IPPROTO_TCP;
-	hints.ai_flags = AI_PASSIVE;
-
-	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
-	if (gai) {
-		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
-		return 0;
-	}
-
-	for (ai = ai0; ai; ai = ai->ai_next) {
-		int sockfd;
-
-		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
-		if (sockfd < 0)
-			continue;
-		if (sockfd >= FD_SETSIZE) {
-			logerror("Socket descriptor too large");
-			close(sockfd);
-			continue;
-		}
-
-#ifdef IPV6_V6ONLY
-		if (ai->ai_family == AF_INET6) {
-			int on = 1;
-			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
-				   &on, sizeof(on));
-			/* Note: error is not fatal */
-		}
-#endif
-
-		if (set_reuse_addr(sockfd)) {
-			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-			close(sockfd);
-			continue;
-		}
-
-		set_keep_alive(sockfd);
-
-		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
-			logerror("Could not bind to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-		if (listen(sockfd, 5) < 0) {
-			logerror("Could not listen to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-
-		flags = fcntl(sockfd, F_GETFD, 0);
-		if (flags >= 0)
-			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-		socklist->list[socklist->nr++] = sockfd;
-		socknum++;
-	}
-
-	freeaddrinfo(ai0);
-
-	return socknum;
-}
-
-#else /* NO_IPV6 */
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	struct sockaddr_in sin;
-	int sockfd;
-	long flags;
-
-	memset(&sin, 0, sizeof sin);
-	sin.sin_family = AF_INET;
-	sin.sin_port = htons(listen_port);
-
-	if (listen_addr) {
-		/* Well, host better be an IP address here. */
-		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
-			return 0;
-	} else {
-		sin.sin_addr.s_addr = htonl(INADDR_ANY);
-	}
-
-	sockfd = socket(AF_INET, SOCK_STREAM, 0);
-	if (sockfd < 0)
-		return 0;
-
-	if (set_reuse_addr(sockfd)) {
-		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	set_keep_alive(sockfd);
-
-	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
-		logerror("Could not bind to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	if (listen(sockfd, 5) < 0) {
-		logerror("Could not listen to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	flags = fcntl(sockfd, F_GETFD, 0);
-	if (flags >= 0)
-		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-	socklist->list[socklist->nr++] = sockfd;
-	return 1;
-}
-
-#endif
-
-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	if (!listen_addr->nr)
-		setup_named_sock(NULL, listen_port, socklist);
-	else {
-		int i, socknum;
-		for (i = 0; i < listen_addr->nr; i++) {
-			socknum = setup_named_sock(listen_addr->items[i].string,
-						   listen_port, socklist);
-
-			if (socknum == 0)
-				logerror("unable to allocate any listen sockets for host %s on port %u",
-					 listen_addr->items[i].string, listen_port);
-		}
-	}
-}
-
 static int service_loop(struct socketlist *socklist)
 {
 	struct pollfd *pfd;
@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,
 {
 	struct socketlist socklist = { NULL, 0, 0 };
 
-	socksetup(listen_addr, listen_port, &socklist);
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr,
+		  logerror);
 	if (socklist.nr == 0)
 		die("unable to allocate any listen sockets on port %u",
 		    listen_port);
-- 
gitgitgadget

Re: [PATCH v5 01/10] daemon: libify socket setup and option functions

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Extract functions for setting up listening sockets and keep-alive options
> from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
> dependencies on global state by inlining the behaviour at the callsites
> for all libified functions.

Thanks for making this change, the reduced code duplication should make the
common daemon-related code more maintainable.

For reference, I used 

'git blame -s -b -C -C -C master..<this patch> -- daemon-utils.c' 

to help identify which lines in 'daemon-utils.c' were changed from their
original implementation in 'daemon.c'. I'll try to rearrange the diff to
show those differences more directly.

The first main change I see is that 'logerror' and 'reuseaddr' are changed
from global references to arguments in 'set_keep_alive()',
'setup_named_sock()' (same for 'NO_IPV6' defined and undefined), and
'socksetup()':

> -static void set_keep_alive(int sockfd)
> +void set_keep_alive(int sockfd, log_fn logerror)

> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
> +static int setup_named_sock(char *listen_addr, int listen_port,
> +			    struct socketlist *socklist, int reuseaddr,
> +			    log_fn logerror)

> -static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
> +void socksetup(struct string_list *listen_addr, int listen_port,
> +	       struct socketlist *socklist, int reuseaddr,
> +	       log_fn logerror)

The external calls in 'daemon.c' to 'set_keep_alive()' and 'socksetup()' are
updated to pass the  'logerror()' function and global 'reuseaddr' as
arguments, so there isn't any change in behavior.

> @@ -759,7 +748,7 @@ static int execute(void)
>  	if (addr)
>  		loginfo("Connection from %s:%s", addr, port);
>  
> -	set_keep_alive(0);
> +	set_keep_alive(0, logerror);
>  	alarm(init_timeout ? init_timeout : timeout);
>  	pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
>  	alarm(0);
> @@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,
>  {
>  	struct socketlist socklist = { NULL, 0, 0 };
>  
> -	socksetup(listen_addr, listen_port, &socklist);
> +	socksetup(listen_addr, listen_port, &socklist, reuseaddr,
> +		  logerror);
>  	if (socklist.nr == 0)
>  		die("unable to allocate any listen sockets on port %u",
>  		    listen_port);

The other notable change is moving the 'if (!reusaddr) return 0' block in
'set_reuse_addr()' to its callers in both 'setup_named_sock()'s:

> +#ifndef NO_IPV6
> +
> +static int setup_named_sock(char *listen_addr, int listen_port,
> +			    struct socketlist *socklist, int reuseaddr,
> +			    log_fn logerror)
> +{
...
> +		if (reuseaddr && set_reuse_addr(sockfd)) {
> +			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
> +			close(sockfd);
> +			continue;
> +		}
...
> +}
> +
> +#else /* NO_IPV6 */
> +
> +static int setup_named_sock(char *listen_addr, int listen_port,
> +			    struct socketlist *socklist, int reuseaddr,
> +			    log_fn logerror)
> +{
...
> +	if (reuseaddr && set_reuse_addr(sockfd)) {
> +		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
> +		close(sockfd);
> +		return 0;
> +	}
...
> +}
> +
> +#endif

Where, previously, that region looked like:

> -#ifndef NO_IPV6
> -
> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
> -{
...
> -		if (set_reuse_addr(sockfd)) {
> -			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
> -			close(sockfd);
> -			continue;
> -		}
...
> -}
> -
> -#else /* NO_IPV6 */
> -
> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
> -{
...
> -	if (set_reuse_addr(sockfd)) {
> -		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
> -		close(sockfd);
> -		return 0;
> -	}
...
> -}
> -
> -#endif

'reuseaddr' is passed into 'setup_named_sock()' from 'socksetup()' calls in
'daemon.c', so this also won't result in changed behavior.

Otherwise, you only expose functions & types that are called in 'daemon.c'
(the rest are still static), and everything else is a verbatim copy. Looks
good to me!

Re: [PATCH v5 01/10] daemon: libify socket setup and option functions

[Derrick Stolee]

On 1/12/2023 2:35 PM, Victoria Dye wrote:
> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Extract functions for setting up listening sockets and keep-alive options
>> from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
>> dependencies on global state by inlining the behaviour at the callsites
>> for all libified functions.
> 
> Thanks for making this change, the reduced code duplication should make the
> common daemon-related code more maintainable.
> 
> For reference, I used 
> 
> 'git blame -s -b -C -C -C master..<this patch> -- daemon-utils.c' 
> 
> to help identify which lines in 'daemon-utils.c' were changed from their
> original implementation in 'daemon.c'.

Neat trick! Thanks for sharing. Using --color-moved was giving similar
results, but with a lot more tracking back-and-forth to see what the
differences were.

I agree with your assessment on this patch that the differences are
valid, safe, and desired.

Thanks,
-Stolee

[PATCH v5 02/10] daemon: libify child process handling functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions and structures for managing child processes started
from the parent daemon-like process from `daemon.c` to the new shared
`daemon-utils.{c,h}` files.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h | 15 ++++++++
 daemon.c       | 92 +++-----------------------------------------------
 3 files changed, 97 insertions(+), 87 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index b96b55962db..3804bc60973 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 		}
 	}
 }
+
+static int addrcmp(const struct sockaddr_storage *s1,
+    const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn , unsigned int *live_children)
+{
+	struct child *newborn, **cradle;
+
+	CALLOC_ARRAY(newborn, 1);
+	live_children++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+void kill_some_child(struct child *firstborn)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			if (loginfo) {
+				const char *dead = "";
+				if (status)
+					dead = " (with error)";
+				loginfo("[%"PRIuMAX"] Disconnected%s",
+					(uintmax_t)pid, dead);
+			}
+
+			/* remove the child */
+			*cradle = blanket->next;
+			live_children--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
diff --git a/daemon-utils.h b/daemon-utils.h
index 6710a2a6dc0..fe8d9d05256 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -2,6 +2,7 @@
 #define DAEMON_UTILS_H
 
 #include "git-compat-util.h"
+#include "run-command.h"
 #include "string-list.h"
 
 typedef void (*log_fn)(const char *msg, ...);
@@ -20,4 +21,18 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 	       struct socketlist *socklist, int reuseaddr,
 	       log_fn logerror);
 
+struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+};
+
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn, unsigned int *live_children);
+
+void kill_some_child(struct child *firstborn);
+
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo);
+
 #endif
diff --git a/daemon.c b/daemon.c
index 1ed4e705680..ec3b407ecbc 100644
--- a/daemon.c
+++ b/daemon.c
@@ -785,93 +785,11 @@ static int execute(void)
 	return -1;
 }
 
-static int addrcmp(const struct sockaddr_storage *s1,
-    const struct sockaddr_storage *s2)
-{
-	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
-	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
-
-	if (sa1->sa_family != sa2->sa_family)
-		return sa1->sa_family - sa2->sa_family;
-	if (sa1->sa_family == AF_INET)
-		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
-		    &((struct sockaddr_in *)s2)->sin_addr,
-		    sizeof(struct in_addr));
-#ifndef NO_IPV6
-	if (sa1->sa_family == AF_INET6)
-		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
-		    &((struct sockaddr_in6 *)s2)->sin6_addr,
-		    sizeof(struct in6_addr));
-#endif
-	return 0;
-}
-
 static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child {
-	struct child *next;
-	struct child_process cld;
-	struct sockaddr_storage address;
-} *firstborn;
-
-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
-{
-	struct child *newborn, **cradle;
-
-	CALLOC_ARRAY(newborn, 1);
-	live_children++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
-			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
-}
-
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
-static void kill_some_child(void)
-{
-	const struct child *blanket, *next;
-
-	if (!(blanket = firstborn))
-		return;
-
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
-			break;
-		}
-}
-
-static void check_dead_children(void)
-{
-	int status;
-	pid_t pid;
-
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
-			const char *dead = "";
-			if (status)
-				dead = " (with error)";
-			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
-
-			/* remove the child */
-			*cradle = blanket->next;
-			live_children--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
-		} else
-			cradle = &blanket->next;
-}
+static struct child *firstborn;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child();
+		kill_some_child(firstborn);
 		sleep(1);  /* give it some time to die */
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen);
+		add_child(&cld, addr, addrlen, firstborn, &live_children);
 }
 
 static void child_handler(int signo)
@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

Re: [PATCH v5 02/10] daemon: libify child process handling functions

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Extract functions and structures for managing child processes started
> from the parent daemon-like process from `daemon.c` to the new shared
> `daemon-utils.{c,h}` files.

As with patch 1, it looks like the main changes here are changing global
references to function arguments. Specifically, those variables are
'firstborn', 'live_children', and 'loginfo':

> -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
> +	       struct child *firstborn , unsigned int *live_children)

> -static void kill_some_child(void)
> +void kill_some_child(struct child *firstborn)

> -static void check_dead_children(void)
> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
> +			 log_fn loginfo)

Those values are provided by the callers in 'daemon.c'. The major change
here is that 'live_children' is passed as a pointer, since its value is
updated by  difference is passing 'live_children' as a pointer, since its
value is updated by 'check_dead_children()' and 'add_child()':

> @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>  	struct child_process cld = CHILD_PROCESS_INIT;
>  
>  	if (max_connections && live_children >= max_connections) {
> -		kill_some_child();
> +		kill_some_child(firstborn);
>  		sleep(1);  /* give it some time to die */
> -		check_dead_children();
> +		check_dead_children(firstborn, &live_children, loginfo);
>  		if (live_children >= max_connections) {
>  			close(incoming);
>  			logerror("Too many children, dropping connection");
> @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>  	if (start_command(&cld))
>  		logerror("unable to fork");
>  	else
> -		add_child(&cld, addr, addrlen);
> +		add_child(&cld, addr, addrlen, firstborn, &live_children);
>  }
>  
>  static void child_handler(int signo)
> @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
>  	for (;;) {
>  		int i;
>  
> -		check_dead_children();
> +		check_dead_children(firstborn, &live_children, loginfo);
>  
>  		if (poll(pfd, socklist->nr, -1) < 0) {
>  			if (errno != EINTR) {

However, I think that change to 'live_children' may have caused a bug. In
'check_dead_children()', you decrement the 'live_children' *pointer*. That
changes its address, not its value:

> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
> +			 log_fn loginfo)
> +{
...
> +			live_children--;
...
> +}

Same thing in 'add_child()':

> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
> +	       struct child *firstborn , unsigned int *live_children)
> +{
...
> +	live_children++;
...
> +}

These should be changed to '(*live_children)--' and '(*live_children)++',
respectively.

There's also one minor functional change in 'check_dead_children()', where
an 'if (loginfo)' check is added guarding the call to 'loginfo()':

> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
> +			 log_fn loginfo)
> +{
...
> +			if (loginfo) {
> +				const char *dead = "";
> +				if (status)
> +					dead = " (with error)";
> +				loginfo("[%"PRIuMAX"] Disconnected%s",
> +					(uintmax_t)pid, dead);
> +			}
...
> +}

I'm guessing this is done because a caller later in the series won't provide
a 'loginfo', but if that's the case, it would help to note that in this
patch's commit message.

The one other thing I noticed is that you removed the function documentation
for 'kill_some_child()':

> -/*
> - * This gets called if the number of connections grows
> - * past "max_connections".
> - *
> - * We kill the newest connection from a duplicate IP.
> - */

Is there a reason why you removed it? Otherwise, it should be added back in
- probably in 'daemon-utils.h'?

Everything else here looks good.

Re: [PATCH v5 02/10] daemon: libify child process handling functions

[Matthew John Cheetham]

On 2023-01-12 11:35, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Extract functions and structures for managing child processes started
>> from the parent daemon-like process from `daemon.c` to the new shared
>> `daemon-utils.{c,h}` files.
> 
> As with patch 1, it looks like the main changes here are changing global
> references to function arguments. Specifically, those variables are
> 'firstborn', 'live_children', and 'loginfo':
> 
>> -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
>> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
>> +	       struct child *firstborn , unsigned int *live_children)
> 
>> -static void kill_some_child(void)
>> +void kill_some_child(struct child *firstborn)
> 
>> -static void check_dead_children(void)
>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
>> +			 log_fn loginfo)
> 
> Those values are provided by the callers in 'daemon.c'. The major change
> here is that 'live_children' is passed as a pointer, since its value is
> updated by  difference is passing 'live_children' as a pointer, since its
> value is updated by 'check_dead_children()' and 'add_child()':
> 
>> @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>>  	struct child_process cld = CHILD_PROCESS_INIT;
>>  
>>  	if (max_connections && live_children >= max_connections) {
>> -		kill_some_child();
>> +		kill_some_child(firstborn);
>>  		sleep(1);  /* give it some time to die */
>> -		check_dead_children();
>> +		check_dead_children(firstborn, &live_children, loginfo);
>>  		if (live_children >= max_connections) {
>>  			close(incoming);
>>  			logerror("Too many children, dropping connection");
>> @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
>>  	if (start_command(&cld))
>>  		logerror("unable to fork");
>>  	else
>> -		add_child(&cld, addr, addrlen);
>> +		add_child(&cld, addr, addrlen, firstborn, &live_children);
>>  }
>>  
>>  static void child_handler(int signo)
>> @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
>>  	for (;;) {
>>  		int i;
>>  
>> -		check_dead_children();
>> +		check_dead_children(firstborn, &live_children, loginfo);
>>  
>>  		if (poll(pfd, socklist->nr, -1) < 0) {
>>  			if (errno != EINTR) {
> 
> However, I think that change to 'live_children' may have caused a bug. In
> 'check_dead_children()', you decrement the 'live_children' *pointer*. That
> changes its address, not its value:
> 
>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
>> +			 log_fn loginfo)
>> +{
> ...
>> +			live_children--;
> ...
>> +}
> 
> Same thing in 'add_child()':
> 
>> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
>> +	       struct child *firstborn , unsigned int *live_children)
>> +{
> ...
>> +	live_children++;
> ...
>> +}
> 
> These should be changed to '(*live_children)--' and '(*live_children)++',
> respectively.

Ah! You are correct; my bad. Will correct this in v6.

> There's also one minor functional change in 'check_dead_children()', where
> an 'if (loginfo)' check is added guarding the call to 'loginfo()':
> 
>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,
>> +			 log_fn loginfo)
>> +{
> ...
>> +			if (loginfo) {
>> +				const char *dead = "";
>> +				if (status)
>> +					dead = " (with error)";
>> +				loginfo("[%"PRIuMAX"] Disconnected%s",
>> +					(uintmax_t)pid, dead);
>> +			}
> ...
>> +}
> 
> I'm guessing this is done because a caller later in the series won't provide
> a 'loginfo', but if that's the case, it would help to note that in this
> patch's commit message.

Will call this out in the commit message in v6.

> The one other thing I noticed is that you removed the function documentation
> for 'kill_some_child()':
> 
>> -/*
>> - * This gets called if the number of connections grows
>> - * past "max_connections".
>> - *
>> - * We kill the newest connection from a duplicate IP.
>> - */
> 
> Is there a reason why you removed it? Otherwise, it should be added back in
> - probably in 'daemon-utils.h'?

I removed it initially as it was referencing things like `max_connections`
which no longer existed in the context of `daemon-utils.{c,h}`.

Next iteration I can restore the spirit of the comment, that this should be
called when the maximimum number of connections has been reached, in order
to kill the newest connection from a duplicate IP.

> Everything else here looks good.

Thanks,
Matthew

[PATCH v5 03/10] daemon: rename some esoteric/laboured terminology

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rename some of the variables and function arguments used to manage child
processes. The existing names are esoteric; stretching an analogy too
far to the point of being confusing to understand.

Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket"
to "current" and "cradle" to "ptr".

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 46 +++++++++++++++++++++++-----------------------
 daemon-utils.h |  6 +++---
 daemon.c       | 10 +++++-----
 3 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index 3804bc60973..190da01aea9 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,
 }
 
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn , unsigned int *live_children)
+	       struct child *first, unsigned int *live_children)
 {
-	struct child *newborn, **cradle;
+	struct child *new_cld, **current;
 
-	CALLOC_ARRAY(newborn, 1);
+	CALLOC_ARRAY(new_cld, 1);
 	live_children++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
+	memcpy(&new_cld->cld, cld, sizeof(*cld));
+	memcpy(&new_cld->address, addr, addrlen);
+	for (current = &first; *current; current = &(*current)->next)
+		if (!addrcmp(&(*current)->address, &new_cld->address))
 			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
+	new_cld->next = *current;
+	*current = new_cld;
 }
 
-void kill_some_child(struct child *firstborn)
+void kill_some_child(struct child *first)
 {
-	const struct child *blanket, *next;
+	const struct child *current, *next;
 
-	if (!(blanket = firstborn))
+	if (!(current = first))
 		return;
 
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
+	for (; (next = current->next); current = next)
+		if (!addrcmp(&current->address, &next->address)) {
+			kill(current->cld.pid, SIGTERM);
 			break;
 		}
 }
 
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first, unsigned int *live_children,
 			 log_fn loginfo)
 {
 	int status;
 	pid_t pid;
 
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+	struct child **ptr, *current;
+	for (ptr = &first; (current = *ptr);)
+		if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
 			if (loginfo) {
 				const char *dead = "";
 				if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,
 			}
 
 			/* remove the child */
-			*cradle = blanket->next;
+			*ptr = current->next;
 			live_children--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
+			child_process_clear(&current->cld);
+			free(current);
 		} else
-			cradle = &blanket->next;
+			ptr = &current->next;
 }
diff --git a/daemon-utils.h b/daemon-utils.h
index fe8d9d05256..e87bc7b9567 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -28,11 +28,11 @@ struct child {
 };
 
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn, unsigned int *live_children);
+	       struct child *first, unsigned int *live_children);
 
-void kill_some_child(struct child *firstborn);
+void kill_some_child(struct child *first);
 
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first, unsigned int *live_children,
 			 log_fn loginfo);
 
 #endif
diff --git a/daemon.c b/daemon.c
index ec3b407ecbc..d3e7d81de18 100644
--- a/daemon.c
+++ b/daemon.c
@@ -789,7 +789,7 @@ static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child *firstborn;
+static struct child *first_child;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child(firstborn);
+		kill_some_child(first_child);
 		sleep(1);  /* give it some time to die */
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen, firstborn, &live_children);
+		add_child(&cld, addr, addrlen, first_child, &live_children);
 }
 
 static void child_handler(int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Rename some of the variables and function arguments used to manage child
> processes. The existing names are esoteric; stretching an analogy too
> far to the point of being confusing to understand.
> 
> Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket"
> to "current" and "cradle" to "ptr".

Thanks for this, I agree that the new names make the code much easier to
read.

> diff --git a/daemon.c b/daemon.c
> index ec3b407ecbc..d3e7d81de18 100644
> --- a/daemon.c
> +++ b/daemon.c
> @@ -789,7 +789,7 @@ static int max_connections = 32;
>  
>  static unsigned int live_children;
>  
> -static struct child *firstborn;
> +static struct child *first_child;

minor nit: you changed "firstborn" to "first" in 'daemon-utils.c' (aligning
with the commit message), but it's "first_child" here. If you end up
re-rolling, it would be nice to make the names consistent across both files
(could be 'first', 'first_child', 'first_cld', or anything really).

Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology

[Matthew John Cheetham]

On 2023-01-12 11:44, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Rename some of the variables and function arguments used to manage child
>> processes. The existing names are esoteric; stretching an analogy too
>> far to the point of being confusing to understand.
>>
>> Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket"
>> to "current" and "cradle" to "ptr".
> 
> Thanks for this, I agree that the new names make the code much easier to
> read.
> 
>> diff --git a/daemon.c b/daemon.c
>> index ec3b407ecbc..d3e7d81de18 100644
>> --- a/daemon.c
>> +++ b/daemon.c
>> @@ -789,7 +789,7 @@ static int max_connections = 32;
>>  
>>  static unsigned int live_children;
>>  
>> -static struct child *firstborn;
>> +static struct child *first_child;
> 
> minor nit: you changed "firstborn" to "first" in 'daemon-utils.c' (aligning
> with the commit message), but it's "first_child" here. If you end up
> re-rolling, it would be nice to make the names consistent across both files
> (could be 'first', 'first_child', 'first_cld', or anything really).
> 

Fair point. There's no technical reason to keep them named differently between
the 'libified' functions, and the actual variables for the callers.
I shall align these to `first_child` in the next iteration.

Thanks,
Matthew

[PATCH v5 04/10] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.

Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                            |   1 +
 contrib/buildsystems/CMakeLists.txt |  11 +-
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 385 ++++++++++++++++++++++++++++
 4 files changed, 396 insertions(+), 2 deletions(-)
 create mode 100644 t/helper/test-http-server.c

diff --git a/Makefile b/Makefile
index 2654094dbb5..3cd61c792ac 100644
--- a/Makefile
+++ b/Makefile
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o
 # Do not add more tests here unless they have extra dependencies. Add
 # them in TEST_BUILTINS_OBJS above.
 TEST_PROGRAMS_NEED_X += test-fake-ssh
+TEST_PROGRAMS_NEED_X += test-http-server
 TEST_PROGRAMS_NEED_X += test-tool
 
 TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..5d949dcb16c 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -961,6 +961,9 @@ if(BUILD_TESTING)
 add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
 target_link_libraries(test-fake-ssh common-main)
 
+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+target_link_libraries(test-http-server common-main)
+
 #reftable-tests
 parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
 list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC)
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
 	set_target_properties(test-fake-ssh test-tool
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
 endif()
 
 #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts
 	git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
 
 set(wrapper_test_scripts
-	test-fake-ssh test-tool)
-
+	test-http-server test-fake-ssh test-tool)
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..11071b1dd89
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,385 @@
+#include "daemon-utils.h"
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+static unsigned int timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client.  The worker talks to the client
+ * on 0 and 1.
+ */
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK       = 0,
+
+	/*
+	 * Various errors while processing the request and/or the response.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_IO_ERROR = 1<<0,
+
+	/*
+	 * Close the socket and clean up.  Does not imply an error.
+	 */
+	WR_HANGUP   = 1<<1,
+};
+
+static enum worker_result worker(void)
+{
+	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0, logerror);
+
+	while (1) {
+		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+			logerror("unable to write response");
+			wr = WR_IO_ERROR;
+		}
+
+		if (wr != WR_OK)
+			break;
+	}
+
+	close(STDIN_FILENO);
+	close(STDOUT_FILENO);
+
+	return !!(wr & WR_IO_ERROR);
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child *first_child;
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child(first_child);
+		sleep(1);  /* give it some time to die */
+		check_dead_children(first_child, &live_children, loginfo);
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen, first_child, &live_children);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children(first_child, &live_children, loginfo);
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances.  So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+
+	trace2_cmd_name("test-http-server");
+	trace2_cmd_list_config();
+	trace2_cmd_list_env_vars();
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
-- 
gitgitgadget

Re: [PATCH v5 04/10] test-http-server: add stub HTTP server test helper

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Introduce a mini HTTP server helper that in the future will be enhanced
> to provide a frontend for the git-http-backend, with support for
> arbitrary authentication schemes.
> 
> Right now, test-http-server is a pared-down copy of the git-daemon that
> always returns a 501 Not Implemented response to all callers.

Between your earlier response [1] and this iteration of the patch, all of
the comments from my previous review [2] have been addressed. The changes to
drop the dependency on cURL also look correct to me. Thanks!

[1] https://lore.kernel.org/git/AS2PR03MB98150C33F9704D2CA10A2EF9C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/
[2] https://lore.kernel.org/git/752da6b2-9c75-0f68-e507-cca02bf918ca@github.com/

> 
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  Makefile                            |   1 +
>  contrib/buildsystems/CMakeLists.txt |  11 +-
>  t/helper/.gitignore                 |   1 +
>  t/helper/test-http-server.c         | 385 ++++++++++++++++++++++++++++
>  4 files changed, 396 insertions(+), 2 deletions(-)
>  create mode 100644 t/helper/test-http-server.c

[PATCH v5 05/10] test-http-server: add HTTP error response function

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 303 +++++++++++++++++++++++++++++++++++-
 t/t5556-http-auth.sh        | 106 +++++++++++++
 2 files changed, 404 insertions(+), 5 deletions(-)
 create mode 100755 t/t5556-http-auth.sh

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 11071b1dd89..67bc16354a1 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -83,9 +83,297 @@ enum worker_result {
 	WR_HANGUP   = 1<<1,
 };
 
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	ssize_t content_length;
+};
+
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = -1 \
+	}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
+static enum worker_result send_http_error(
+	int fd,
+	int http_code, const char *http_code_name,
+	int retry_after_seconds, struct string_list *response_headers,
+	enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n",
+		    http_code, http_code_name);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_OK | WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* also store common request headers as struct req members */
+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
+			req->content_length = strtol(hv, &hp, 10);
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->content_length >= 0)
+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		/*
+		 * This regular expression matches all dumb and smart HTTP
+		 * requests that are currently in use, and defined in
+		 * Documentation/gitprotocol-http.txt.
+		 *
+		 */
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	/*
+	 * Note that we always respond with a 200 OK response even if the
+	 * http-backend process exits with an error. This helper is intended
+	 * only to be used to exercise the HTTP auth handling in the Git client,
+	 * and specifically around authentication (not handled by http-backend).
+	 *
+	 * If we wanted to respond with a more 'valid' HTTP response status then
+	 * we'd need to buffer the output of http-backend, wait for and grok the
+	 * exit status of the process, then write the HTTP status line followed
+	 * by the http-backend output. This is outside of the scope of this test
+	 * helper's use at time of writing.
+	 *
+	 * The important auth responses (401) we are handling prior to getting
+	 * to this point.
+	 */
+	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+			req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
+				req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
+				req->content_type);
+	if (req->content_length >= 0)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
+				(intmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(STDOUT_FILENO);
+	close(STDIN_FILENO);
+	return !!res;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	if (is_git_request(req))
+		return do__git(req);
+
+	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+			       WR_OK | WR_HANGUP);
+}
+
 static enum worker_result worker(void)
 {
-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	struct req req = REQ__INIT;
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -96,11 +384,16 @@ static enum worker_result worker(void)
 	set_keep_alive(0, logerror);
 
 	while (1) {
-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
-			logerror("unable to write response");
-			wr = WR_IO_ERROR;
-		}
+		req__release(&req);
+
+		alarm(timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr != WR_OK)
+			break;
 
+		wr = dispatch(&req);
 		if (wr != WR_OK)
 			break;
 	}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..65105a5a6a9
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,106 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
+REPO_DIR="$TRASH_DIRECTORY"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
+per_test_cleanup () {
+	stop_http_server &&
+	rm -f OUT.*
+}
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+	start_http_server &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v5 05/10] test-http-server: add HTTP error response function

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Introduce a function to the test-http-server test helper to write more
> full and valid HTTP error responses, including all the standard response
> headers like `Server` and `Date`.

It took me a second to figure out, but this patch combines the content of
patches 4, 5, and 6 from the last iteration. After squashing those three
patches from v4 together locally, the range-diff is actually pretty simple
(see below). 

Out of curiosity, why did you combine those patches? I don't feel strongly
about changing it, but the smaller, incremental patches in the previous
version were a bit easier to review.

In any case, this version addresses my feedback from [1], [2], and [3] - the
explanatory comments are particularly helpful. Thanks!

[1] https://lore.kernel.org/git/7b7d1059-cecf-744d-6927-b41963b9e5a8@github.com/
[2] https://lore.kernel.org/git/e957d4f4-fa94-7a68-f378-38e6ed131244@github.com/
[3] https://lore.kernel.org/git/f99c381c-1d30-7c95-6158-cecd5321dafd@github.com/

Range diff v4 (patches 4-6, squashed) vs. v5 (this patch)

4:  127827637e !  5:  6f66bf146b test-http-server: add HTTP error response function
    @@ Commit message
     
      ## t/helper/test-http-server.c ##
     @@ t/helper/test-http-server.c: enum worker_result {
    - 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
    + 	WR_HANGUP   = 1<<1,
      };
      
     +/*
    @@ t/helper/test-http-server.c: enum worker_result {
     +		hp = strbuf_detach(&h, NULL);
     +		string_list_append(&req->header_list, hp);
     +
    -+		/* store common request headers separately */
    ++		/* also store common request headers as struct req members */
     +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
     +			req->content_type = hv;
     +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
    @@ t/helper/test-http-server.c: enum worker_result {
     +
     +	if (!initialized) {
     +		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
    ++		/*
    ++		 * This regular expression matches all dumb and smart HTTP
    ++		 * requests that are currently in use, and defined in
    ++		 * Documentation/gitprotocol-http.txt.
    ++		 *
    ++		 */
     +		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
     +			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
     +			    REG_EXTENDED)) {
    @@ t/helper/test-http-server.c: enum worker_result {
     +		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
     +}
     +
    -+static enum worker_result do__git(struct req *req, const char *user)
    ++static enum worker_result do__git(struct req *req)
     +{
     +	const char *ok = "HTTP/1.1 200 OK\r\n";
     +	struct child_process cp = CHILD_PROCESS_INIT;
     +	int res;
     +
    -+	if (write(1, ok, strlen(ok)) < 0)
    ++	/*
    ++	 * Note that we always respond with a 200 OK response even if the
    ++	 * http-backend process exits with an error. This helper is intended
    ++	 * only to be used to exercise the HTTP auth handling in the Git client,
    ++	 * and specifically around authentication (not handled by http-backend).
    ++	 *
    ++	 * If we wanted to respond with a more 'valid' HTTP response status then
    ++	 * we'd need to buffer the output of http-backend, wait for and grok the
    ++	 * exit status of the process, then write the HTTP status line followed
    ++	 * by the http-backend output. This is outside of the scope of this test
    ++	 * helper's use at time of writing.
    ++	 *
    ++	 * The important auth responses (401) we are handling prior to getting
    ++	 * to this point.
    ++	 */
    ++	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
     +		return error(_("could not send '%s'"), ok);
     +
    -+	if (user)
    -+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
    -+
     +	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     +	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     +			req->uri_path.buf);
    @@ t/helper/test-http-server.c: enum worker_result {
     +	cp.git_cmd = 1;
     +	strvec_push(&cp.args, "http-backend");
     +	res = run_command(&cp);
    -+	close(1);
    -+	close(0);
    ++	close(STDOUT_FILENO);
    ++	close(STDIN_FILENO);
     +	return !!res;
     +}
     +
     +static enum worker_result dispatch(struct req *req)
     +{
     +	if (is_git_request(req))
    -+		return do__git(req, NULL);
    ++		return do__git(req);
     +
    -+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
    ++	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
     +			       WR_OK | WR_HANGUP);
     +}
     +
    @@ t/helper/test-http-server.c: enum worker_result {
      	char *client_port = getenv("REMOTE_PORT");
      	enum worker_result wr = WR_OK;
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
    - 	set_keep_alive(0);
    + 	set_keep_alive(0, logerror);
      
      	while (1) {
    --		if (write_in_full(1, response, strlen(response)) < 0) {
    +-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
     -			logerror("unable to write response");
     -			wr = WR_IO_ERROR;
     -		}
     +		req__release(&req);
     +
    -+		alarm(init_timeout ? init_timeout : timeout);
    ++		alarm(timeout);
     +		wr = req__read(&req, 0);
     +		alarm(0);
     +
    -+		if (wr & WR_STOP_THE_MUSIC)
    ++		if (wr != WR_OK)
     +			break;
      
     +		wr = dispatch(&req);
    - 		if (wr & WR_STOP_THE_MUSIC)
    + 		if (wr != WR_OK)
      			break;
      	}
     
    @@ t/t5556-http-auth.sh (new)
     +
     +test_description='test http auth header and credential helper interop'
     +
    ++TEST_NO_CREATE_REPO=1
     +. ./test-lib.sh
     +
     +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
     +
     +# Setup a repository
     +#
    -+REPO_DIR="$(pwd)"/repo
    ++REPO_DIR="$TRASH_DIRECTORY"/repo
     +
     +# Setup some lookback URLs where test-http-server will be listening.
     +# We will spawn it directly inside the repo directory, so we avoid
    @@ t/t5556-http-auth.sh (new)
     +# The server will shutdown if/when we delete it (this is easier than
     +# killing it by PID).
     +#
    -+PID_FILE="$(pwd)"/pid-file.pid
    -+SERVER_LOG="$(pwd)"/OUT.server.log
    ++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
    ++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     +
     +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     +
    @@ t/t5556-http-auth.sh (new)
     +
     +test_expect_success 'http auth anonymous no challenge' '
     +	test_when_finished "per_test_cleanup" &&
    -+	start_http_server --allow-anonymous &&
    ++	start_http_server &&
     +
     +	# Attempt to read from a protected repository
     +	git ls-remote $ORIGIN_URL

Re: [PATCH v5 05/10] test-http-server: add HTTP error response function

[Matthew John Cheetham]

On 2023-01-12 12:35, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Introduce a function to the test-http-server test helper to write more
>> full and valid HTTP error responses, including all the standard response
>> headers like `Server` and `Date`.
> 
> It took me a second to figure out, but this patch combines the content of
> patches 4, 5, and 6 from the last iteration. After squashing those three
> patches from v4 together locally, the range-diff is actually pretty simple
> (see below). 
> 
> Out of curiosity, why did you combine those patches? I don't feel strongly
> about changing it, but the smaller, incremental patches in the previous
> version were a bit easier to review.

This is my mistake. I didn't intend to do this, and agree splitting them is
easier to grok. I will restore this! My apologies! :-(

> In any case, this version addresses my feedback from [1], [2], and [3] - the
> explanatory comments are particularly helpful. Thanks!
> 
> [1] https://lore.kernel.org/git/7b7d1059-cecf-744d-6927-b41963b9e5a8@github.com/
> [2] https://lore.kernel.org/git/e957d4f4-fa94-7a68-f378-38e6ed131244@github.com/
> [3] https://lore.kernel.org/git/f99c381c-1d30-7c95-6158-cecd5321dafd@github.com/
> 
> Range diff v4 (patches 4-6, squashed) vs. v5 (this patch)
> 
> 4:  127827637e !  5:  6f66bf146b test-http-server: add HTTP error response function
>     @@ Commit message
>      
>       ## t/helper/test-http-server.c ##
>      @@ t/helper/test-http-server.c: enum worker_result {
>     - 	WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
>     + 	WR_HANGUP   = 1<<1,
>       };
>       
>      +/*
>     @@ t/helper/test-http-server.c: enum worker_result {
>      +		hp = strbuf_detach(&h, NULL);
>      +		string_list_append(&req->header_list, hp);
>      +
>     -+		/* store common request headers separately */
>     ++		/* also store common request headers as struct req members */
>      +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
>      +			req->content_type = hv;
>      +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
>     @@ t/helper/test-http-server.c: enum worker_result {
>      +
>      +	if (!initialized) {
>      +		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
>     ++		/*
>     ++		 * This regular expression matches all dumb and smart HTTP
>     ++		 * requests that are currently in use, and defined in
>     ++		 * Documentation/gitprotocol-http.txt.
>     ++		 *
>     ++		 */
>      +		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
>      +			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
>      +			    REG_EXTENDED)) {
>     @@ t/helper/test-http-server.c: enum worker_result {
>      +		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
>      +}
>      +
>     -+static enum worker_result do__git(struct req *req, const char *user)
>     ++static enum worker_result do__git(struct req *req)
>      +{
>      +	const char *ok = "HTTP/1.1 200 OK\r\n";
>      +	struct child_process cp = CHILD_PROCESS_INIT;
>      +	int res;
>      +
>     -+	if (write(1, ok, strlen(ok)) < 0)
>     ++	/*
>     ++	 * Note that we always respond with a 200 OK response even if the
>     ++	 * http-backend process exits with an error. This helper is intended
>     ++	 * only to be used to exercise the HTTP auth handling in the Git client,
>     ++	 * and specifically around authentication (not handled by http-backend).
>     ++	 *
>     ++	 * If we wanted to respond with a more 'valid' HTTP response status then
>     ++	 * we'd need to buffer the output of http-backend, wait for and grok the
>     ++	 * exit status of the process, then write the HTTP status line followed
>     ++	 * by the http-backend output. This is outside of the scope of this test
>     ++	 * helper's use at time of writing.
>     ++	 *
>     ++	 * The important auth responses (401) we are handling prior to getting
>     ++	 * to this point.
>     ++	 */
>     ++	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
>      +		return error(_("could not send '%s'"), ok);
>      +
>     -+	if (user)
>     -+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
>     -+
>      +	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
>      +	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
>      +			req->uri_path.buf);
>     @@ t/helper/test-http-server.c: enum worker_result {
>      +	cp.git_cmd = 1;
>      +	strvec_push(&cp.args, "http-backend");
>      +	res = run_command(&cp);
>     -+	close(1);
>     -+	close(0);
>     ++	close(STDOUT_FILENO);
>     ++	close(STDIN_FILENO);
>      +	return !!res;
>      +}
>      +
>      +static enum worker_result dispatch(struct req *req)
>      +{
>      +	if (is_git_request(req))
>     -+		return do__git(req, NULL);
>     ++		return do__git(req);
>      +
>     -+	return send_http_error(1, 501, "Not Implemented", -1, NULL,
>     ++	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
>      +			       WR_OK | WR_HANGUP);
>      +}
>      +
>     @@ t/helper/test-http-server.c: enum worker_result {
>       	char *client_port = getenv("REMOTE_PORT");
>       	enum worker_result wr = WR_OK;
>      @@ t/helper/test-http-server.c: static enum worker_result worker(void)
>     - 	set_keep_alive(0);
>     + 	set_keep_alive(0, logerror);
>       
>       	while (1) {
>     --		if (write_in_full(1, response, strlen(response)) < 0) {
>     +-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
>      -			logerror("unable to write response");
>      -			wr = WR_IO_ERROR;
>      -		}
>      +		req__release(&req);
>      +
>     -+		alarm(init_timeout ? init_timeout : timeout);
>     ++		alarm(timeout);
>      +		wr = req__read(&req, 0);
>      +		alarm(0);
>      +
>     -+		if (wr & WR_STOP_THE_MUSIC)
>     ++		if (wr != WR_OK)
>      +			break;
>       
>      +		wr = dispatch(&req);
>     - 		if (wr & WR_STOP_THE_MUSIC)
>     + 		if (wr != WR_OK)
>       			break;
>       	}
>      
>     @@ t/t5556-http-auth.sh (new)
>      +
>      +test_description='test http auth header and credential helper interop'
>      +
>     ++TEST_NO_CREATE_REPO=1
>      +. ./test-lib.sh
>      +
>      +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
>      +
>      +# Setup a repository
>      +#
>     -+REPO_DIR="$(pwd)"/repo
>     ++REPO_DIR="$TRASH_DIRECTORY"/repo
>      +
>      +# Setup some lookback URLs where test-http-server will be listening.
>      +# We will spawn it directly inside the repo directory, so we avoid
>     @@ t/t5556-http-auth.sh (new)
>      +# The server will shutdown if/when we delete it (this is easier than
>      +# killing it by PID).
>      +#
>     -+PID_FILE="$(pwd)"/pid-file.pid
>     -+SERVER_LOG="$(pwd)"/OUT.server.log
>     ++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
>     ++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
>      +
>      +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
>      +
>     @@ t/t5556-http-auth.sh (new)
>      +
>      +test_expect_success 'http auth anonymous no challenge' '
>      +	test_when_finished "per_test_cleanup" &&
>     -+	start_http_server --allow-anonymous &&
>     ++	start_http_server &&
>      +
>      +	# Attempt to read from a protected repository
>      +	git ls-remote $ORIGIN_URL
> 

[PATCH v5 06/10] test-http-server: add simple authentication

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
a configuration file (in the normal gitconfig file format).
Incoming requests are compared against the set of valid schemes and
tokens and only approved if a matching token is found, or if no auth
was provided and anonymous auth is enabled.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 246 +++++++++++++++++++++++++++++++++++-
 1 file changed, 244 insertions(+), 2 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 67bc16354a1..dcc326c8652 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -7,6 +7,7 @@
 #include "version.h"
 #include "dir.h"
 #include "date.h"
+#include "config.h"
 
 #define TR2_CAT "test-http-server"
 
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =
 "           [--timeout=<n>] [--max-connections=<n>]\n"
 "           [--reuseaddr] [--pid-file=<file>]\n"
 "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--auth-config=<file>]\n"
 ;
 
 static unsigned int timeout;
@@ -317,7 +319,7 @@ static int is_git_request(struct req *req)
 		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
 }
 
-static enum worker_result do__git(struct req *req)
+static enum worker_result do__git(struct req *req, const char *user)
 {
 	const char *ok = "HTTP/1.1 200 OK\r\n";
 	struct child_process cp = CHILD_PROCESS_INIT;
@@ -341,6 +343,9 @@ static enum worker_result do__git(struct req *req)
 	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
 		return error(_("could not send '%s'"), ok);
 
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
 	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
 			req->uri_path.buf);
@@ -362,10 +367,234 @@ static enum worker_result do__git(struct req *req)
 	return !!res;
 }
 
+enum auth_result {
+	/* No auth module matches the request. */
+	AUTH_UNKNOWN = 0,
+
+	/* Auth module denied the request. */
+	AUTH_DENY = 1,
+
+	/* Auth module successfully validated the request. */
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	char *scheme;
+	char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+static struct strvec extra_headers = STRVEC_INIT;
+
+static struct auth_module *create_auth_module(const char *scheme,
+					      const char *challenge)
+{
+	struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+	mod->scheme = xstrdup(scheme);
+	mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
+	CALLOC_ARRAY(mod->tokens, 1);
+	string_list_init_dup(mod->tokens);
+	return mod;
+}
+
+static struct auth_module *get_auth_module(const char *scheme)
+{
+	int i;
+	struct auth_module *mod;
+	for (i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme))
+			return mod;
+	}
+
+	return NULL;
+}
+
+static int add_auth_module(struct auth_module *mod)
+{
+	if (get_auth_module(mod->scheme))
+		return error("duplicate auth scheme '%s'\n", mod->scheme);
+
+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+	auth_modules[auth_modules_nr++] = mod;
+
+	return 0;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/*
+	 * Check all auth modules and try to validate the request.
+	 * The first Authorization header that matches a known auth module
+	 * scheme will be consulted to either approve or deny the request.
+	 * If no module is found, or if there is no valid token, then 401 error.
+	 * Otherwise, only permit the request if anonymous auth is enabled.
+	 * It's atypical for user agents/clients to send multiple Authorization
+	 * headers, but not explicitly forbidden or defined.
+	 */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (!split[0] || !split[1]) continue;
+
+			/* trim trailing space ' ' */
+			strbuf_setlen(split[0], split[0]->len - 1);
+
+			mod = get_auth_module(split[0]->buf);
+			if (mod) {
+				result = AUTH_DENY;
+
+				for_each_string_list_item(token, mod->tokens) {
+					if (!strcmp(split[1]->buf, token->string)) {
+						result = AUTH_ALLOW;
+						break;
+					}
+				}
+
+				goto done;
+			}
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (result != AUTH_DENY && allow_anonymous)
+			break;
+
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+
+		for (i = 0; i < extra_headers.nr; i++)
+			string_list_append(&hdrs, extra_headers.v[i]);
+
+		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
+				      &hdrs, *wr);
+	}
+
+	strbuf_list_free(split);
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
+static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
+{
+	struct strbuf **p = strbuf_split_str(str, ':', 2);
+
+	if (!p[0])
+		return -1;
+
+	/* trim trailing ':' */
+	if (p[1])
+		strbuf_setlen(p[0], p[0]->len - 1);
+
+	if (required_val && !p[1])
+		return -1;
+
+	*scheme = strbuf_detach(p[0], NULL);
+
+	if (p[1])
+		*val = strbuf_detach(p[1], NULL);
+
+	strbuf_list_free(p);
+	return 0;
+}
+
+static int read_auth_config(const char *name, const char *val, void *data)
+{
+	int ret = 0;
+	char *scheme = NULL;
+	char *token = NULL;
+	char *challenge = NULL;
+	struct auth_module *mod = NULL;
+
+	if (!strcmp(name, "auth.challenge")) {
+		if (split_auth_param(val, &scheme, &challenge, 0)) {
+			ret = error("invalid auth challenge '%s'", val);
+			goto cleanup;
+		}
+
+		mod = create_auth_module(scheme, challenge);
+		if (add_auth_module(mod)) {
+			ret = error("failed to add auth module '%s'", val);
+			goto cleanup;
+		}
+	}
+	if (!strcmp(name, "auth.token")) {
+		if (split_auth_param(val, &scheme, &token, 1)) {
+			ret = error("invalid auth token '%s'", val);
+			goto cleanup;
+		}
+
+		mod = get_auth_module(scheme);
+		if (!mod) {
+			ret = error("auth scheme not defined '%s'\n", scheme);
+			goto cleanup;
+		}
+
+		string_list_append(mod->tokens, token);
+	}
+	if (!strcmp(name, "auth.allowanonymous")) {
+		allow_anonymous = git_config_bool(name, val);
+	}
+	if (!strcmp(name, "auth.extraheader")) {
+		strvec_push(&extra_headers, val);
+	}
+
+cleanup:
+	free(scheme);
+	free(token);
+	free(challenge);
+
+	return ret;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
 	if (is_git_request(req))
-		return do__git(req);
+		return do__git(req, user);
 
 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
@@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv)
 			pid_file = v;
 			continue;
 		}
+		if (skip_prefix(arg, "--auth-config=", &v)) {
+			if (!strlen(v)) {
+				error("invalid argument - missing file path");
+				usage(test_http_auth_usage);
+			}
+
+			if (git_config_from_file(read_auth_config, v, NULL)) {
+				error("failed to read auth config file '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			continue;
+		}
 
 		fprintf(stderr, "error: unknown argument '%s'\n", arg);
 		usage(test_http_auth_usage);
-- 
gitgitgadget

Re: [PATCH v5 06/10] test-http-server: add simple authentication

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> +static struct auth_module *create_auth_module(const char *scheme,
> +					      const char *challenge)
> +{
> +	struct auth_module *mod = xmalloc(sizeof(struct auth_module));
> +	mod->scheme = xstrdup(scheme);
> +	mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
> +	CALLOC_ARRAY(mod->tokens, 1);
> +	string_list_init_dup(mod->tokens);
> +	return mod;
> +}

> +
> +static int add_auth_module(struct auth_module *mod)
> +{
> +	if (get_auth_module(mod->scheme))
> +		return error("duplicate auth scheme '%s'\n", mod->scheme);
> +
> +	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
> +	auth_modules[auth_modules_nr++] = mod;
> +
> +	return 0;
> +}

> +static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
> +{
> +	struct strbuf **p = strbuf_split_str(str, ':', 2);
> +
> +	if (!p[0])
> +		return -1;
> +
> +	/* trim trailing ':' */
> +	if (p[1])
> +		strbuf_setlen(p[0], p[0]->len - 1);
> +
> +	if (required_val && !p[1])
> +		return -1;
> +
> +	*scheme = strbuf_detach(p[0], NULL);
> +
> +	if (p[1])
> +		*val = strbuf_detach(p[1], NULL);
> +
> +	strbuf_list_free(p);
> +	return 0;
> +}

There's nothing really *new* in these functions in this iteration, just code
moved from the option parsing/handling in 'cmd_main()' into dedicated
functions. Looks good!

> +	switch (result) {
> +	case AUTH_ALLOW:
> +		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
> +		*user = "VALID_TEST_USER";
> +		*wr = WR_OK;
> +		break;
> +
> +	case AUTH_DENY:
> +		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
> +		/* fall-through */
> +
> +	case AUTH_UNKNOWN:
> +		if (result != AUTH_DENY && allow_anonymous)
> +			break;

I completely missed the "fall-through" comment in my last review [1], as you
kindly pointed out [2]. ;) Given that, this makes sense to me.

[1] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/
[2] https://lore.kernel.org/git/AS2PR03MB981593EB3382F9738D2CA3D7C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/

> +
> +		for (i = 0; i < auth_modules_nr; i++) {
> +			mod = auth_modules[i];
> +			if (mod->challenge_params)
> +				challenge = xstrfmt("WWW-Authenticate: %s %s",
> +						    mod->scheme,
> +						    mod->challenge_params);
> +			else
> +				challenge = xstrfmt("WWW-Authenticate: %s",
> +						    mod->scheme);
> +			string_list_append(&hdrs, challenge);
> +		}
> +
> +		for (i = 0; i < extra_headers.nr; i++)
> +			string_list_append(&hdrs, extra_headers.v[i]);
> +
> +		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
> +				      &hdrs, *wr);

The "extra_headers" configuration is new, and helps make the test server
more flexible. 

> +static int read_auth_config(const char *name, const char *val, void *data)
> +{
> +	int ret = 0;
> +	char *scheme = NULL;
> +	char *token = NULL;
> +	char *challenge = NULL;
> +	struct auth_module *mod = NULL;
> +
> +	if (!strcmp(name, "auth.challenge")) {
> +		if (split_auth_param(val, &scheme, &challenge, 0)) {
> +			ret = error("invalid auth challenge '%s'", val);
> +			goto cleanup;
> +		}
> +
> +		mod = create_auth_module(scheme, challenge);
> +		if (add_auth_module(mod)) {
> +			ret = error("failed to add auth module '%s'", val);
> +			goto cleanup;
> +		}
> +	}
> +	if (!strcmp(name, "auth.token")) {
> +		if (split_auth_param(val, &scheme, &token, 1)) {
> +			ret = error("invalid auth token '%s'", val);
> +			goto cleanup;
> +		}
> +
> +		mod = get_auth_module(scheme);
> +		if (!mod) {
> +			ret = error("auth scheme not defined '%s'\n", scheme);
> +			goto cleanup;
> +		}
> +
> +		string_list_append(mod->tokens, token);
> +	}

I don't think this addresses the implicit option ordering requirement noted
in [3]; instead of needing '--auth' before '--auth-token', this now needs
'auth.challenge' before 'auth.token' in the config file. While I'd prefer it
if this could be rearranged so that the auth setup happens after all config
parsing (so the order doesn't matter), if you want to leave it as-is please
add a comment somewhere in this file explaining that requirement and/or add
a note to the "auth scheme not defined" error message.  

[3] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/

> +	if (!strcmp(name, "auth.allowanonymous")) {
> +		allow_anonymous = git_config_bool(name, val);
> +	}
> +	if (!strcmp(name, "auth.extraheader")) {
> +		strvec_push(&extra_headers, val);
> +	}

Is it worth printing a warning if the option found isn't any of the above?
Something like "ignoring <config option>". This is a test helper, so
user-friendliness isn't quite as important as it is for builtins, but the
warning might be helpful to developers trying to use it in the future.

> +
> +cleanup:
> +	free(scheme);
> +	free(token);
> +	free(challenge);
> +
> +	return ret;
> +}
> +
>  static enum worker_result dispatch(struct req *req)
>  {
> +	enum worker_result wr = WR_OK;
> +	const char *user = NULL;
> +
> +	if (!is_authed(req, &user, &wr))
> +		return wr;
> +
>  	if (is_git_request(req))
> -		return do__git(req);
> +		return do__git(req, user);
>  
>  	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
>  			       WR_OK | WR_HANGUP);
> @@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv)
>  			pid_file = v;
>  			continue;
>  		}
> +		if (skip_prefix(arg, "--auth-config=", &v)) {
> +			if (!strlen(v)) {
> +				error("invalid argument - missing file path");
> +				usage(test_http_auth_usage);
> +			}
> +
> +			if (git_config_from_file(read_auth_config, v, NULL)) {
> +				error("failed to read auth config file '%s'", v);
> +				usage(test_http_auth_usage);
> +			}
> +
> +			continue;
> +		}
>  
>  		fprintf(stderr, "error: unknown argument '%s'\n", arg);
>  		usage(test_http_auth_usage);

Re: [PATCH v5 06/10] test-http-server: add simple authentication

[Junio C Hamano]

Victoria Dye <vdye@github.com> writes:

>> +static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
>> +{
>> ...
>> +}
>
> There's nothing really *new* in these functions in this iteration, just code
> moved from the option parsing/handling in 'cmd_main()' into dedicated
> functions. Looks good!

> ...
>
> I completely missed the "fall-through" comment in my last review [1], as you
> kindly pointed out [2]. ;) Given that, this makes sense to me.

>> +		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
>> +				      &hdrs, *wr);
>
> The "extra_headers" configuration is new, and helps make the test server
> more flexible. 

This is not limited to this single review message, but it is good to
see "this part of the patch is good because ..." explicitly stated.
I wish more people did so, in addition to pointing out what needs to
be improved.

Thanks.

Re: [PATCH v5 06/10] test-http-server: add simple authentication

[Matthew John Cheetham]

On 2023-01-13 10:10, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>> +static int read_auth_config(const char *name, const char *val, void *data)
>> +{
>> +	int ret = 0;
>> +	char *scheme = NULL;
>> +	char *token = NULL;
>> +	char *challenge = NULL;
>> +	struct auth_module *mod = NULL;
>> +
>> +	if (!strcmp(name, "auth.challenge")) {
>> +		if (split_auth_param(val, &scheme, &challenge, 0)) {
>> +			ret = error("invalid auth challenge '%s'", val);
>> +			goto cleanup;
>> +		}
>> +
>> +		mod = create_auth_module(scheme, challenge);
>> +		if (add_auth_module(mod)) {
>> +			ret = error("failed to add auth module '%s'", val);
>> +			goto cleanup;
>> +		}
>> +	}
>> +	if (!strcmp(name, "auth.token")) {
>> +		if (split_auth_param(val, &scheme, &token, 1)) {
>> +			ret = error("invalid auth token '%s'", val);
>> +			goto cleanup;
>> +		}
>> +
>> +		mod = get_auth_module(scheme);
>> +		if (!mod) {
>> +			ret = error("auth scheme not defined '%s'\n", scheme);
>> +			goto cleanup;
>> +		}
>> +
>> +		string_list_append(mod->tokens, token);
>> +	}
> 
> I don't think this addresses the implicit option ordering requirement noted
> in [3]; instead of needing '--auth' before '--auth-token', this now needs
> 'auth.challenge' before 'auth.token' in the config file. While I'd prefer it
> if this could be rearranged so that the auth setup happens after all config
> parsing (so the order doesn't matter), if you want to leave it as-is please
> add a comment somewhere in this file explaining that requirement and/or add
> a note to the "auth scheme not defined" error message.  
> 
> [3] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/
> 
>> +	if (!strcmp(name, "auth.allowanonymous")) {
>> +		allow_anonymous = git_config_bool(name, val);
>> +	}
>> +	if (!strcmp(name, "auth.extraheader")) {
>> +		strvec_push(&extra_headers, val);
>> +	}
> 
> Is it worth printing a warning if the option found isn't any of the above?
> Something like "ignoring <config option>". This is a test helper, so
> user-friendliness isn't quite as important as it is for builtins, but the
> warning might be helpful to developers trying to use it in the future.

I tried this suggestion of adding a warning, but it felt wrong. You are correct
in the first instance that it should really "just work" when specified in any
order. Watch for the next iteration where I'll make it such you can specify them
in any order :-)

Thanks,
Matthew

[PATCH v5 07/10] http: replace unsafe size_t multiplication with st_mult

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Replace direct multiplication of two size_t parameters in curl response
stream handling callback functions with `st_mult` to guard against
overflows.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/http.c b/http.c
index 8a5ba3f4776..a2a80318bb2 100644
--- a/http.c
+++ b/http.c
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
 
 size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct buffer *buffer = buffer_;
 
 	if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
 
 size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct strbuf *buffer = buffer_;
 
 	strbuf_add(buffer, ptr, size);
-- 
gitgitgadget

[PATCH v5 08/10] strvec: expose strvec_push_nodup for external use

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Remove the static modifier from the existing `strvec_push_nodup`
function and define the function is `strvec.h` to make it available for
other callers, making it now possible to append to a `struct strvec`
array without duplication.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 strvec.c | 2 +-
 strvec.h | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/strvec.c b/strvec.c
index 61a76ce6cb9..26e8751cae0 100644
--- a/strvec.c
+++ b/strvec.c
@@ -10,7 +10,7 @@ void strvec_init(struct strvec *array)
 	memcpy(array, &blank, sizeof(*array));
 }
 
-static void strvec_push_nodup(struct strvec *array, const char *value)
+void strvec_push_nodup(struct strvec *array, const char *value)
 {
 	if (array->v == empty_strvec)
 		array->v = NULL;
diff --git a/strvec.h b/strvec.h
index 9f55c8766ba..5d61dd73680 100644
--- a/strvec.h
+++ b/strvec.h
@@ -43,6 +43,9 @@ struct strvec {
  */
 void strvec_init(struct strvec *);
 
+/* Push a string onto the end of the array without copying. */
+void strvec_push_nodup(struct strvec *array, const char *value);
+
 /* Push a copy of a string onto the end of the array. */
 const char *strvec_push(struct strvec *, const char *);
 
-- 
gitgitgadget

[PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 ++++++++
 http.c       | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 114 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index a2a80318bb2..10882034145 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,102 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = st_mult(eltsize, nmemb);
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 * Continuation lines start with at least one whitespace, maybe more,
+	 * so we should collapse these down to a single SP (valid per the spec).
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		/* Trim leading whitespace from this continuation hdr line. */
+		strbuf_ltrim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			const char *prev = values->v[values->nr - 1];
+			struct strbuf append = STRBUF_INIT;
+			strbuf_addstr(&append, prev);
+
+			/* Join two non-empty values with a single space. */
+			if (append.len)
+				strbuf_addch(&append, ' ');
+
+			strbuf_addbuf(&append, &buf);
+
+			strvec_pop(values);
+			strvec_push_nodup(values, strbuf_detach(&append, NULL));
+		}
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (istarts_with(buf.buf, "http/"))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1960,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> [...]
> +		} else if (buf.len) {
> +			const char *prev = values->v[values->nr - 1];
> +			struct strbuf append = STRBUF_INIT;
> +			strbuf_addstr(&append, prev);
> +
> +			/* Join two non-empty values with a single space. */
> +			if (append.len)
> +				strbuf_addch(&append, ' ');
> +
> +			strbuf_addbuf(&append, &buf);
> +
> +			strvec_pop(values);
> +			strvec_push_nodup(values, strbuf_detach(&append, NULL));
> +		}
> +

I've written something like the strvec_push_nodup() patch that preceded
this myself for similar reasons, and as recently noted in [1] I think
such a thing (although I implemented a different interface) might be
useful in general.

But this really doesn't seem like a good justification for adding this
new API. Let's instead do:

	} else if (buf.len) {
		const char *prev = values->v[values->nr - 1];
		/* Join two non-empty values with a single space. */
		const char *const sp = *prev ? " " : ""

		strvec_pop(values);
		strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
	}

There may be cases where a public strvec_push_nodup() simplifies things,
but this doesn't seem like such a case, just use strvec_pushf() directly
instead, and skip the strbuf & strbuf_detach().

I haven't compiled/tested the above, so there may e.g. be a typo in
there. But I think the general concept should work in this case.

1. https://lore.kernel.org/git/RFC-cover-0.5-00000000000-20221215T090226Z-avarab@gmail.com/

Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2023-01-12 00:41, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>> [...]
>> +		} else if (buf.len) {
>> +			const char *prev = values->v[values->nr - 1];
>> +			struct strbuf append = STRBUF_INIT;
>> +			strbuf_addstr(&append, prev);
>> +
>> +			/* Join two non-empty values with a single space. */
>> +			if (append.len)
>> +				strbuf_addch(&append, ' ');
>> +
>> +			strbuf_addbuf(&append, &buf);
>> +
>> +			strvec_pop(values);
>> +			strvec_push_nodup(values, strbuf_detach(&append, NULL));
>> +		}
>> +
> 
> I've written something like the strvec_push_nodup() patch that preceded
> this myself for similar reasons, and as recently noted in [1] I think
> such a thing (although I implemented a different interface) might be
> useful in general.

A fair point, and reading [1] I see there's some concerns about making the
strvec interface more complicated w.r.t. ownership vs saving a `xstrdup`.
In light of this, I'll drop the commit to add `strvec_push_nodup`.

> But this really doesn't seem like a good justification for adding this
> new API. Let's instead do:
> 
> 	} else if (buf.len) {
> 		const char *prev = values->v[values->nr - 1];
> 		/* Join two non-empty values with a single space. */
> 		const char *const sp = *prev ? " " : ""
> 
> 		strvec_pop(values);
> 		strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
> 	}
> 
> There may be cases where a public strvec_push_nodup() simplifies things,
> but this doesn't seem like such a case, just use strvec_pushf() directly
> instead, and skip the strbuf & strbuf_detach().
> 
> I haven't compiled/tested the above, so there may e.g. be a typo in
> there. But I think the general concept should work in this case.
> 
> 1. https://lore.kernel.org/git/RFC-cover-0.5-00000000000-20221215T090226Z-avarab@gmail.com/


There's a bug in your suggestion. We're `strvec_pop`-ing from the array
which also frees the previous value that we want to use to append to in
the next call to `strvec_pushf`. We need to keep a copy of the previous
header value around.

This should work instead (adding an `xstrdup` and `free`):

	} else if (buf.len) {
		char *prev = xstrdup(values->v[values->nr - 1]);

		/* Join two non-empty values with a single space. */
		const char *const sp = *prev ? " " : "";

		strvec_pop(values);
		strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
		free(prev);
	}

Thanks,
Matthew

[PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt          |  19 +-
 credential.c                              |  12 +
 t/helper/test-credential-helper-replay.sh |  14 ++
 t/t5556-http-auth.sh                      | 270 +++++++++++++++++++++-
 4 files changed, 312 insertions(+), 3 deletions(-)
 create mode 100755 t/helper/test-credential-helper-replay.sh

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..8a3ad6c0ae2 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	int i = 0;
+	const char *full_key = xstrfmt("%s[]", key);
+	for (; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free((void*)full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh
new file mode 100755
index 00000000000..03e5e63dad6
--- /dev/null
+++ b/t/helper/test-credential-helper-replay.sh
@@ -0,0 +1,14 @@
+cmd=$1
+teefile=$cmd-actual.cred
+catfile=$cmd-response.cred
+rm -f $teefile
+while read line;
+do
+	if test -z "$line"; then
+		break;
+	fi
+	echo "$line" >> $teefile
+done
+if test "$cmd" = "get"; then
+	cat $catfile
+fi
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 65105a5a6a9..a8dbee6ca40 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -27,6 +27,8 @@ PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
 SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
 
 PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
+	&& export CREDENTIAL_HELPER
 
 test_expect_success 'setup repos' '
 	test_create_repo "$REPO_DIR" &&
@@ -92,15 +94,279 @@ start_http_server () {
 
 per_test_cleanup () {
 	stop_http_server &&
-	rm -f OUT.*
+	rm -f OUT.* &&
+	rm -f *.cred &&
+	rm -f auth.config
 }
 
 test_expect_success 'http auth anonymous no challenge' '
 	test_when_finished "per_test_cleanup" &&
-	start_http_server &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    allowAnonymous = true
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
 
 	# Attempt to read from a protected repository
 	git ls-remote $ORIGIN_URL
 '
 
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	    extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bEaRer auThoRiTy="id.example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	    extraheader = "WWW-Authenticate:"
+	    extraheader = " "
+	    extraheader = " bearer authority=\"id.example.com\""
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bearer authority="id.example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = "foobar:alg=test widget=1"
+	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >store-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+	    challenge = basic:realm=\"example.com\"
+	    token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	cat >get-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >erase-expected.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	cat >get-response.cred <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	EOF
+
+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
+
+	test_cmp get-expected.cred get-actual.cred &&
+	test_cmp erase-expected.cred erase-actual.cred
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> [...]
> +static void credential_write_strvec(FILE *fp, const char *key,
> +				    const struct strvec *vec)
> +{
> +	int i = 0;
> +	const char *full_key = xstrfmt("%s[]", key);
> +	for (; i < vec->nr; i++) {
> +		credential_write_item(fp, full_key, vec->v[i], 0);

Style: Don't mismatch types if there's no good reason. Use "size_t i" here, also let's do:

	for (size_t i = 0; ....

I.e. no reason to declare it earlier.

> +	}
> +	free((void*)full_key);

Just don't add a "const" to that "full_key" and skip the cast with
free() here.

> +++ b/t/helper/test-credential-helper-replay.sh

I see to my surprise that we have one existing *.sh helper in that
directory, but in any case...

> @@ -0,0 +1,14 @@
> +cmd=$1
> +teefile=$cmd-actual.cred
> +catfile=$cmd-response.cred
> +rm -f $teefile
> +while read line;
> +do
> +	if test -z "$line"; then
> +		break;
> +	fi
> +	echo "$line" >> $teefile
> +done

It looks like you're re-inventing "sed" here, isn't this whole loop just

	sed -n -e '/^$/q' -n 'p'

And then you can skip the "rm" before, as you could just clobber the
thing.

> +if test "$cmd" = "get"; then
> +	cat $catfile
> +fi
> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
> index 65105a5a6a9..a8dbee6ca40 100755
> --- a/t/t5556-http-auth.sh
> +++ b/t/t5556-http-auth.sh
> @@ -27,6 +27,8 @@ PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
>  SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
>  
>  PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
> +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
> +	&& export CREDENTIAL_HELPER

...(continued from above): Let's just use write_script() here or
whatever, i.e. no reason to make this a global script, it's just used in
this one test, so it can set it up.
>  
>  test_expect_success 'setup repos' '
>  	test_create_repo "$REPO_DIR" &&
> @@ -92,15 +94,279 @@ start_http_server () {
>  
>  per_test_cleanup () {
>  	stop_http_server &&
> -	rm -f OUT.*
> +	rm -f OUT.* &&
> +	rm -f *.cred &&
> +	rm -f auth.config
>  }
>  
>  test_expect_success 'http auth anonymous no challenge' '
>  	test_when_finished "per_test_cleanup" &&
> -	start_http_server &&
> +
> +	cat >auth.config <<-EOF &&
> +	[auth]
> +	    allowAnonymous = true

Mixed tab/space. Use "\t" not 4x " " (ditto below).

Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2023-01-12 00:48, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>> [...]
>> +static void credential_write_strvec(FILE *fp, const char *key,
>> +				    const struct strvec *vec)
>> +{
>> +	int i = 0;
>> +	const char *full_key = xstrfmt("%s[]", key);
>> +	for (; i < vec->nr; i++) {
>> +		credential_write_item(fp, full_key, vec->v[i], 0);
> 
> Style: Don't mismatch types if there's no good reason. Use "size_t i" here, also let's do:
> 
> 	for (size_t i = 0; ....
> 
> I.e. no reason to declare it earlier.
> 
>> +	}
>> +	free((void*)full_key);
> 
> Just don't add a "const" to that "full_key" and skip the cast with
> free() here.

Both good points! Thanks - will take this onboard in next iteration.

>> +++ b/t/helper/test-credential-helper-replay.sh
> 
> I see to my surprise that we have one existing *.sh helper in that
> directory, but in any case...
> 
>> @@ -0,0 +1,14 @@
>> +cmd=$1
>> +teefile=$cmd-actual.cred
>> +catfile=$cmd-response.cred
>> +rm -f $teefile
>> +while read line;
>> +do
>> +	if test -z "$line"; then
>> +		break;
>> +	fi
>> +	echo "$line" >> $teefile
>> +done
> 
> It looks like you're re-inventing "sed" here, isn't this whole loop just
> 
> 	sed -n -e '/^$/q' -n 'p'

True; `sed -n -e '/^$/q' -e 'p'` is equivalent here.

> And then you can skip the "rm" before, as you could just clobber the
> thing.
> 
>> +if test "$cmd" = "get"; then
>> +	cat $catfile
>> +fi
>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
>> index 65105a5a6a9..a8dbee6ca40 100755
>> --- a/t/t5556-http-auth.sh
>> +++ b/t/t5556-http-auth.sh
>> @@ -27,6 +27,8 @@ PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
>>  SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
>>  
>>  PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
>> +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
>> +	&& export CREDENTIAL_HELPER
> 
> ...(continued from above): Let's just use write_script() here or
> whatever, i.e. no reason to make this a global script, it's just used in
> this one test, so it can set it up.

In the next iteration I will move to using write_script; thanks!

>>  test_expect_success 'setup repos' '
>>  	test_create_repo "$REPO_DIR" &&
>> @@ -92,15 +94,279 @@ start_http_server () {
>>  
>>  per_test_cleanup () {
>>  	stop_http_server &&
>> -	rm -f OUT.*
>> +	rm -f OUT.* &&
>> +	rm -f *.cred &&
>> +	rm -f auth.config
>>  }
>>  
>>  test_expect_success 'http auth anonymous no challenge' '
>>  	test_when_finished "per_test_cleanup" &&
>> -	start_http_server &&
>> +
>> +	cat >auth.config <<-EOF &&
>> +	[auth]
>> +	    allowAnonymous = true
> 
> Mixed tab/space. Use "\t" not 4x " " (ditto below).

Sure!

Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests

[Derrick Stolee]

On 1/11/2023 5:13 PM, Matthew John Cheetham via GitGitGadget wrote:

> +static void credential_write_strvec(FILE *fp, const char *key,
> +				    const struct strvec *vec)
> +{
> +	int i = 0;
> +	const char *full_key = xstrfmt("%s[]", key);
> +	for (; i < vec->nr; i++) {

style nit: use "int i;" and "for (i = 0; ..."

>  test_expect_success 'http auth anonymous no challenge' '
>  	test_when_finished "per_test_cleanup" &&
> -	start_http_server &&
> +
> +	cat >auth.config <<-EOF &&
> +	[auth]
> +	    allowAnonymous = true
> +	EOF
> +
> +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&

I see that you added auth.allowAnonymous and --auth-config options
in Patch 6, so perhaps this test change could move to that patch.

Thanks,
-Stolee

Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2023-01-12 12:41, Derrick Stolee wrote:

> On 1/11/2023 5:13 PM, Matthew John Cheetham via GitGitGadget wrote:
> 
>> +static void credential_write_strvec(FILE *fp, const char *key,
>> +				    const struct strvec *vec)
>> +{
>> +	int i = 0;
>> +	const char *full_key = xstrfmt("%s[]", key);
>> +	for (; i < vec->nr; i++) {
> 
> style nit: use "int i;" and "for (i = 0; ..."

Thanks for pointing this out; I missed that C99 style for-loops
were allowed now. As Ævar pointed out, this should also be `size_t`
and not `int`.

>>  test_expect_success 'http auth anonymous no challenge' '
>>  	test_when_finished "per_test_cleanup" &&
>> -	start_http_server &&
>> +
>> +	cat >auth.config <<-EOF &&
>> +	[auth]
>> +	    allowAnonymous = true
>> +	EOF
>> +
>> +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
> 
> I see that you added auth.allowAnonymous and --auth-config options
> in Patch 6, so perhaps this test change could move to that patch.

Good point; will update on reroll.

> Thanks,
> -Stolee

[PATCH v6 00/12] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server sharing code
with git-daemon, with simple authentication configurable by a config file.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.

Matthew John Cheetham (12):
  daemon: libify socket setup and option functions
  daemon: libify child process handling functions
  daemon: rename some esoteric/laboured terminology
  test-http-server: add stub HTTP server test helper
  test-http-server: add HTTP error response function
  test-http-server: add HTTP request parsing
  test-http-server: pass Git requests to http-backend
  test-http-server: add simple authentication
  test-http-server: add sending of arbitrary headers
  http: replace unsafe size_t multiplication with st_mult
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt    |  19 +-
 Makefile                            |   2 +
 contrib/buildsystems/CMakeLists.txt |  11 +-
 credential.c                        |  12 +
 credential.h                        |  15 +
 daemon-utils.c                      | 286 +++++++++
 daemon-utils.h                      |  55 ++
 daemon.c                            | 306 +---------
 http.c                              |  98 ++-
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 910 ++++++++++++++++++++++++++++
 t/lib-credential-helper.sh          |  27 +
 t/t5556-http-auth.sh                | 398 ++++++++++++
 13 files changed, 1838 insertions(+), 302 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h
 create mode 100644 t/helper/test-http-server.c
 create mode 100644 t/lib-credential-helper.sh
 create mode 100755 t/t5556-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v5:

  1:  74b0de14185 =  1:  74b0de14185 daemon: libify socket setup and option functions
  2:  bc972fc8d3d !  2:  b6ba344a671 daemon: libify child process handling functions
     @@ Commit message
          from the parent daemon-like process from `daemon.c` to the new shared
          `daemon-utils.{c,h}` files.
      
     +    One minor functional change is introduced to `check_dead_children()`
     +    where the logging of a dead/disconnected child is now optional. With the
     +    'libification' of these functions we extract the call to `loginfo` to a
     +    call to a function pointer, and guard the log message creation and
     +    logging behind a `NULL` check. Callers can now skip logging by passing
     +    `NULL` as the `log_fn loginfo` argument.
     +    The behaviour of callers in `daemon.c` remains the same (save one extra
     +    NULL check)  however as a pointer to `loginfo` is always passed.
     +
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## daemon-utils.c ##
     @@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,
      +	struct child *newborn, **cradle;
      +
      +	CALLOC_ARRAY(newborn, 1);
     -+	live_children++;
     ++	(*live_children)++;
      +	memcpy(&newborn->cld, cld, sizeof(*cld));
      +	memcpy(&newborn->address, addr, addrlen);
      +	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
     @@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,
      +
      +			/* remove the child */
      +			*cradle = blanket->next;
     -+			live_children--;
     ++			(*live_children)--;
      +			child_process_clear(&blanket->cld);
      +			free(blanket);
      +		} else
     @@ daemon-utils.h: void socksetup(struct string_list *listen_addr, int listen_port,
      +	struct sockaddr_storage address;
      +};
      +
     ++/*
     ++ * Add the child_process to the set of children and increment the number of
     ++ * live children.
     ++ */
      +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
      +	       struct child *firstborn, unsigned int *live_children);
      +
     ++/*
     ++ * Kill the newest connection from a duplicate IP.
     ++ *
     ++ * This function should be called if the number of connections grows
     ++ * past the maximum number of allowed connections.
     ++ */
      +void kill_some_child(struct child *firstborn);
      +
     ++/*
     ++ * Check for children that have disconnected and remove them from the
     ++ * active set, decrementing the number of live children.
     ++ *
     ++ * Optionally log the child PID that disconnected by passing a loginfo
     ++ * function.
     ++ */
      +void check_dead_children(struct child *firstborn, unsigned int *live_children,
      +			 log_fn loginfo);
      +
  3:  8f176d5955d !  3:  9967401c972 daemon: rename some esoteric/laboured terminology
     @@ Commit message
          processes. The existing names are esoteric; stretching an analogy too
          far to the point of being confusing to understand.
      
     -    Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket"
     +    Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket"
          to "current" and "cradle" to "ptr".
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
     @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
       
       void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
      -	       struct child *firstborn , unsigned int *live_children)
     -+	       struct child *first, unsigned int *live_children)
     ++	       struct child *first_child, unsigned int *live_children)
       {
      -	struct child *newborn, **cradle;
      +	struct child *new_cld, **current;
       
      -	CALLOC_ARRAY(newborn, 1);
      +	CALLOC_ARRAY(new_cld, 1);
     - 	live_children++;
     + 	(*live_children)++;
      -	memcpy(&newborn->cld, cld, sizeof(*cld));
      -	memcpy(&newborn->address, addr, addrlen);
      -	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
      -		if (!addrcmp(&(*cradle)->address, &newborn->address))
      +	memcpy(&new_cld->cld, cld, sizeof(*cld));
      +	memcpy(&new_cld->address, addr, addrlen);
     -+	for (current = &first; *current; current = &(*current)->next)
     ++	for (current = &first_child; *current; current = &(*current)->next)
      +		if (!addrcmp(&(*current)->address, &new_cld->address))
       			break;
      -	newborn->next = *cradle;
     @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
       }
       
      -void kill_some_child(struct child *firstborn)
     -+void kill_some_child(struct child *first)
     ++void kill_some_child(struct child *first_child)
       {
      -	const struct child *blanket, *next;
      +	const struct child *current, *next;
       
      -	if (!(blanket = firstborn))
     -+	if (!(current = first))
     ++	if (!(current = first_child))
       		return;
       
      -	for (; (next = blanket->next); blanket = next)
     @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
       }
       
      -void check_dead_children(struct child *firstborn, unsigned int *live_children,
     -+void check_dead_children(struct child *first, unsigned int *live_children,
     ++void check_dead_children(struct child *first_child, unsigned int *live_children,
       			 log_fn loginfo)
       {
       	int status;
     @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
      -	for (cradle = &firstborn; (blanket = *cradle);)
      -		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
      +	struct child **ptr, *current;
     -+	for (ptr = &first; (current = *ptr);)
     ++	for (ptr = &first_child; (current = *ptr);)
      +		if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
       			if (loginfo) {
       				const char *dead = "";
     @@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *
       			/* remove the child */
      -			*cradle = blanket->next;
      +			*ptr = current->next;
     - 			live_children--;
     + 			(*live_children)--;
      -			child_process_clear(&blanket->cld);
      -			free(blanket);
      +			child_process_clear(&current->cld);
     @@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *
      
       ## daemon-utils.h ##
      @@ daemon-utils.h: struct child {
     - };
     - 
     +  * live children.
     +  */
       void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
      -	       struct child *firstborn, unsigned int *live_children);
     -+	       struct child *first, unsigned int *live_children);
     - 
     ++	       struct child *first_child, unsigned int *live_children);
     + 
     + /*
     +  * Kill the newest connection from a duplicate IP.
     +@@ daemon-utils.h: void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl
     +  * This function should be called if the number of connections grows
     +  * past the maximum number of allowed connections.
     +  */
      -void kill_some_child(struct child *firstborn);
     -+void kill_some_child(struct child *first);
     - 
     ++void kill_some_child(struct child *first_child);
     + 
     + /*
     +  * Check for children that have disconnected and remove them from the
     +@@ daemon-utils.h: void kill_some_child(struct child *firstborn);
     +  * Optionally log the child PID that disconnected by passing a loginfo
     +  * function.
     +  */
      -void check_dead_children(struct child *firstborn, unsigned int *live_children,
     -+void check_dead_children(struct child *first, unsigned int *live_children,
     ++void check_dead_children(struct child *first_child, unsigned int *live_children,
       			 log_fn loginfo);
       
       #endif
  4:  706fb3781bd =  4:  d6e5e8825e8 test-http-server: add stub HTTP server test helper
  -:  ----------- >  5:  79805f042b9 test-http-server: add HTTP error response function
  5:  6f66bf146b4 !  6:  252098db219 test-http-server: add HTTP error response function
     @@ Metadata
      Author: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Commit message ##
     -    test-http-server: add HTTP error response function
     +    test-http-server: add HTTP request parsing
      
     -    Introduce a function to the test-http-server test helper to write more
     -    full and valid HTTP error responses, including all the standard response
     -    headers like `Server` and `Date`.
     +    Add ability to parse HTTP requests to the test-http-server test helper.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
     @@ t/helper/test-http-server.c: enum worker_result {
      +	string_list_clear(&req->header_list, 0);
      +}
      +
     -+static enum worker_result send_http_error(
     -+	int fd,
     -+	int http_code, const char *http_code_name,
     -+	int retry_after_seconds, struct string_list *response_headers,
     -+	enum worker_result wr_in)
     -+{
     -+	struct strbuf response_header = STRBUF_INIT;
     -+	struct strbuf response_content = STRBUF_INIT;
     -+	struct string_list_item *h;
     -+	enum worker_result wr;
     -+
     -+	strbuf_addf(&response_content, "Error: %d %s\r\n",
     -+		    http_code, http_code_name);
     -+	if (retry_after_seconds > 0)
     -+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
     -+			    retry_after_seconds);
     -+
     -+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
     -+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
     -+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
     -+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
     -+	if (retry_after_seconds > 0)
     -+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
     -+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
     -+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
     -+	if (response_headers)
     -+		for_each_string_list_item(h, response_headers)
     -+			strbuf_addf(&response_header, "%s\r\n", h->string);
     -+	strbuf_addstr(&response_header, "\r\n");
     -+
     -+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
     -+		logerror("unable to write response header");
     -+		wr = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
     -+		logerror("unable to write response content body");
     -+		wr = WR_IO_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	wr = wr_in;
     -+
     -+done:
     -+	strbuf_release(&response_header);
     -+	strbuf_release(&response_content);
     -+
     -+	return wr;
     -+}
     -+
     + static enum worker_result send_http_error(
     + 	int fd,
     + 	int http_code, const char *http_code_name,
     +@@ t/helper/test-http-server.c: done:
     + 	return wr;
     + }
     + 
      +/*
      + * Read the HTTP request up to the start of the optional message-body.
      + * We do this byte-by-byte because we have keep-alive turned on and
     @@ t/helper/test-http-server.c: enum worker_result {
      +	return result;
      +}
      +
     -+static int is_git_request(struct req *req)
     -+{
     -+	static regex_t *smart_http_regex;
     -+	static int initialized;
     -+
     -+	if (!initialized) {
     -+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
     -+		/*
     -+		 * This regular expression matches all dumb and smart HTTP
     -+		 * requests that are currently in use, and defined in
     -+		 * Documentation/gitprotocol-http.txt.
     -+		 *
     -+		 */
     -+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
     -+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
     -+			    REG_EXTENDED)) {
     -+			warning("could not compile smart HTTP regex");
     -+			smart_http_regex = NULL;
     -+		}
     -+		initialized = 1;
     -+	}
     -+
     -+	return smart_http_regex &&
     -+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
     -+}
     -+
     -+static enum worker_result do__git(struct req *req)
     -+{
     -+	const char *ok = "HTTP/1.1 200 OK\r\n";
     -+	struct child_process cp = CHILD_PROCESS_INIT;
     -+	int res;
     -+
     -+	/*
     -+	 * Note that we always respond with a 200 OK response even if the
     -+	 * http-backend process exits with an error. This helper is intended
     -+	 * only to be used to exercise the HTTP auth handling in the Git client,
     -+	 * and specifically around authentication (not handled by http-backend).
     -+	 *
     -+	 * If we wanted to respond with a more 'valid' HTTP response status then
     -+	 * we'd need to buffer the output of http-backend, wait for and grok the
     -+	 * exit status of the process, then write the HTTP status line followed
     -+	 * by the http-backend output. This is outside of the scope of this test
     -+	 * helper's use at time of writing.
     -+	 *
     -+	 * The important auth responses (401) we are handling prior to getting
     -+	 * to this point.
     -+	 */
     -+	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
     -+		return error(_("could not send '%s'"), ok);
     -+
     -+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     -+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     -+			req->uri_path.buf);
     -+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
     -+	if (req->query_args.len)
     -+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
     -+				req->query_args.buf);
     -+	if (req->content_type)
     -+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
     -+				req->content_type);
     -+	if (req->content_length >= 0)
     -+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
     -+				(intmax_t)req->content_length);
     -+	cp.git_cmd = 1;
     -+	strvec_push(&cp.args, "http-backend");
     -+	res = run_command(&cp);
     -+	close(STDOUT_FILENO);
     -+	close(STDIN_FILENO);
     -+	return !!res;
     -+}
     -+
      +static enum worker_result dispatch(struct req *req)
      +{
     -+	if (is_git_request(req))
     -+		return do__git(req);
     -+
      +	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
      +			       WR_OK | WR_HANGUP);
      +}
      +
       static enum worker_result worker(void)
       {
     --	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
      +	struct req req = REQ__INIT;
       	char *client_addr = getenv("REMOTE_ADDR");
       	char *client_port = getenv("REMOTE_PORT");
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
       	set_keep_alive(0, logerror);
       
       	while (1) {
     --		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
     --			logerror("unable to write response");
     --			wr = WR_IO_ERROR;
     --		}
     +-		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
     +-				     NULL, WR_OK | WR_HANGUP);
      +		req__release(&req);
      +
      +		alarm(timeout);
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
       		if (wr != WR_OK)
       			break;
       	}
     -
     - ## t/t5556-http-auth.sh (new) ##
     -@@
     -+#!/bin/sh
     -+
     -+test_description='test http auth header and credential helper interop'
     -+
     -+TEST_NO_CREATE_REPO=1
     -+. ./test-lib.sh
     -+
     -+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
     -+
     -+# Setup a repository
     -+#
     -+REPO_DIR="$TRASH_DIRECTORY"/repo
     -+
     -+# Setup some lookback URLs where test-http-server will be listening.
     -+# We will spawn it directly inside the repo directory, so we avoid
     -+# any need to configure directory mappings etc - we only serve this
     -+# repository from the root '/' of the server.
     -+#
     -+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
     -+ORIGIN_URL=http://$HOST_PORT/
     -+
     -+# The pid-file is created by test-http-server when it starts.
     -+# The server will shutdown if/when we delete it (this is easier than
     -+# killing it by PID).
     -+#
     -+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
     -+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     -+
     -+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     -+
     -+test_expect_success 'setup repos' '
     -+	test_create_repo "$REPO_DIR" &&
     -+	git -C "$REPO_DIR" branch -M main
     -+'
     -+
     -+stop_http_server () {
     -+	if ! test -f "$PID_FILE"
     -+	then
     -+		return 0
     -+	fi
     -+	#
     -+	# The server will shutdown automatically when we delete the pid-file.
     -+	#
     -+	rm -f "$PID_FILE"
     -+	#
     -+	# Give it a few seconds to shutdown (mainly to completely release the
     -+	# port before the next test start another instance and it attempts to
     -+	# bind to it).
     -+	#
     -+	for k in 0 1 2 3 4
     -+	do
     -+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
     -+		then
     -+			return 0
     -+		fi
     -+		sleep 1
     -+	done
     -+
     -+	echo "stop_http_server: timeout waiting for server shutdown"
     -+	return 1
     -+}
     -+
     -+start_http_server () {
     -+	#
     -+	# Launch our server into the background in repo_dir.
     -+	#
     -+	(
     -+		cd "$REPO_DIR"
     -+		test-http-server --verbose \
     -+			--listen=127.0.0.1 \
     -+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
     -+			--reuseaddr \
     -+			--pid-file="$PID_FILE" \
     -+			"$@" \
     -+			2>"$SERVER_LOG" &
     -+	)
     -+	#
     -+	# Give it a few seconds to get started.
     -+	#
     -+	for k in 0 1 2 3 4
     -+	do
     -+		if test -f "$PID_FILE"
     -+		then
     -+			return 0
     -+		fi
     -+		sleep 1
     -+	done
     -+
     -+	echo "start_http_server: timeout waiting for server startup"
     -+	return 1
     -+}
     -+
     -+per_test_cleanup () {
     -+	stop_http_server &&
     -+	rm -f OUT.*
     -+}
     -+
     -+test_expect_success 'http auth anonymous no challenge' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	start_http_server &&
     -+
     -+	# Attempt to read from a protected repository
     -+	git ls-remote $ORIGIN_URL
     -+'
     -+
     -+test_done
  -:  ----------- >  7:  ab06ac9b965 test-http-server: pass Git requests to http-backend
  6:  c3c3d17a688 !  8:  a1ff55dd6e2 test-http-server: add simple authentication
     @@ Commit message
          tokens and only approved if a matching token is found, or if no auth
          was provided and anonymous auth is enabled.
      
     +    Configuration for auth includes a simple set of three options:
     +
     +    [auth]
     +            challenge = <scheme>[:<challenge_params>]
     +            token = <scheme>:[<token>]*
     +            allowAnonymous = <bool>
     +
     +    `auth.challenge` allows you define what authentication schemes, and
     +    optional challenge parameters the server should use. Scheme names are
     +    unique and subsequently specified challenge parameters in the config
     +    file will replace previously specified ones.
     +
     +    `auth.token` allows you to define the set of value token values for an
     +    authentication scheme. This is a multi-var and each entry in the
     +    config file will append to the set of valid tokens for that scheme.
     +    Specifying an empty token value will clear the list of tokens so far for
     +    that scheme, i.e. `token = <scheme>:`.
     +
     +    `auth.allowAnonymous` controls whether or not unauthenticated requests
     +    (those without any `Authorization` headers) should succeed or not, and
     +    trigger a 401 Unauthorized response.
     +
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## t/helper/test-http-server.c ##
     @@ t/helper/test-http-server.c: static int is_git_request(struct req *req)
       	const char *ok = "HTTP/1.1 200 OK\r\n";
       	struct child_process cp = CHILD_PROCESS_INIT;
      @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
     + 	 * exit status of the process, then write the HTTP status line followed
     + 	 * by the http-backend output. This is outside of the scope of this test
     + 	 * helper's use at time of writing.
     ++	 *
     ++	 * The important auth responses (401) we are handling prior to getting
     ++	 * to this point.
     + 	 */
       	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
       		return error(_("could not send '%s'"), ok);
       
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +static struct auth_module **auth_modules = NULL;
      +static size_t auth_modules_nr = 0;
      +static size_t auth_modules_alloc = 0;
     -+static struct strvec extra_headers = STRVEC_INIT;
      +
     -+static struct auth_module *create_auth_module(const char *scheme,
     -+					      const char *challenge)
     -+{
     -+	struct auth_module *mod = xmalloc(sizeof(struct auth_module));
     -+	mod->scheme = xstrdup(scheme);
     -+	mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
     -+	CALLOC_ARRAY(mod->tokens, 1);
     -+	string_list_init_dup(mod->tokens);
     -+	return mod;
     -+}
     -+
     -+static struct auth_module *get_auth_module(const char *scheme)
     ++static struct auth_module *get_auth_module(const char *scheme, int create)
      +{
      +	int i;
      +	struct auth_module *mod;
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +			return mod;
      +	}
      +
     -+	return NULL;
     -+}
     ++	if (create) {
     ++		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
     ++		mod->scheme = xstrdup(scheme);
     ++		mod->challenge_params = NULL;
     ++		CALLOC_ARRAY(mod->tokens, 1);
     ++		string_list_init_dup(mod->tokens);
      +
     -+static int add_auth_module(struct auth_module *mod)
     -+{
     -+	if (get_auth_module(mod->scheme))
     -+		return error("duplicate auth scheme '%s'\n", mod->scheme);
     ++		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
     ++		auth_modules[auth_modules_nr++] = mod;
      +
     -+	ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
     -+	auth_modules[auth_modules_nr++] = mod;
     ++		return mod;
     ++	}
      +
     -+	return 0;
     ++	return NULL;
      +}
      +
      +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +			/* trim trailing space ' ' */
      +			strbuf_setlen(split[0], split[0]->len - 1);
      +
     -+			mod = get_auth_module(split[0]->buf);
     ++			mod = get_auth_module(split[0]->buf, 0);
      +			if (mod) {
      +				result = AUTH_DENY;
      +
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +			string_list_append(&hdrs, challenge);
      +		}
      +
     -+		for (i = 0; i < extra_headers.nr; i++)
     -+			string_list_append(&hdrs, extra_headers.v[i]);
     -+
      +		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
      +				      &hdrs, *wr);
      +	}
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +	      (result == AUTH_UNKNOWN && allow_anonymous);
      +}
      +
     -+static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
     ++static int split_auth_param(const char *str, char **scheme, char **val)
      +{
      +	struct strbuf **p = strbuf_split_str(str, ':', 2);
      +
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +		return -1;
      +
      +	/* trim trailing ':' */
     -+	if (p[1])
     ++	if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
      +		strbuf_setlen(p[0], p[0]->len - 1);
      +
     -+	if (required_val && !p[1])
     -+		return -1;
     -+
      +	*scheme = strbuf_detach(p[0], NULL);
      +
      +	if (p[1])
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +	struct auth_module *mod = NULL;
      +
      +	if (!strcmp(name, "auth.challenge")) {
     -+		if (split_auth_param(val, &scheme, &challenge, 0)) {
     ++		if (split_auth_param(val, &scheme, &challenge)) {
      +			ret = error("invalid auth challenge '%s'", val);
      +			goto cleanup;
      +		}
      +
     -+		mod = create_auth_module(scheme, challenge);
     -+		if (add_auth_module(mod)) {
     -+			ret = error("failed to add auth module '%s'", val);
     -+			goto cleanup;
     -+		}
     -+	}
     -+	if (!strcmp(name, "auth.token")) {
     -+		if (split_auth_param(val, &scheme, &token, 1)) {
     -+			ret = error("invalid auth token '%s'", val);
     -+			goto cleanup;
     -+		}
     ++		mod = get_auth_module(scheme, 1);
      +
     -+		mod = get_auth_module(scheme);
     -+		if (!mod) {
     -+			ret = error("auth scheme not defined '%s'\n", scheme);
     ++		/* Replace any existing challenge parameters */
     ++		free(mod->challenge_params);
     ++		mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
     ++	} else if (!strcmp(name, "auth.token")) {
     ++		if (split_auth_param(val, &scheme, &token)) {
     ++			ret = error("invalid auth token '%s'", val);
      +			goto cleanup;
      +		}
      +
     -+		string_list_append(mod->tokens, token);
     -+	}
     -+	if (!strcmp(name, "auth.allowanonymous")) {
     ++		mod = get_auth_module(scheme, 1);
     ++
     ++		/*
     ++		 * Append to set of valid tokens unless an empty token value
     ++		 * is provided, then clear the existing list.
     ++		 */
     ++		if (token)
     ++			string_list_append(mod->tokens, token);
     ++		else
     ++			string_list_clear(mod->tokens, 1);
     ++	} else if (!strcmp(name, "auth.allowanonymous")) {
      +		allow_anonymous = git_config_bool(name, val);
     -+	}
     -+	if (!strcmp(name, "auth.extraheader")) {
     -+		strvec_push(&extra_headers, val);
     ++	} else {
     ++		warning("unknown auth config '%s'", name);
      +	}
      +
      +cleanup:
     @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
       
       		fprintf(stderr, "error: unknown argument '%s'\n", arg);
       		usage(test_http_auth_usage);
     +
     + ## t/t5556-http-auth.sh ##
     +@@ t/t5556-http-auth.sh: per_test_cleanup () {
     + 	rm -f OUT.*
     + }
     + 
     ++test_expect_success CURL 'http auth server auth config' '
     ++	#test_when_finished "per_test_cleanup" &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++		challenge = no-params
     ++		challenge = with-params:foo=\"bar\" p=1
     ++		challenge = with-params:foo=\"replaced\" q=1
     ++
     ++		token = no-explicit-challenge:valid-token
     ++		token = no-explicit-challenge:also-valid
     ++		token = reset-tokens:these-tokens
     ++		token = reset-tokens:will-be-reset
     ++		token = reset-tokens:
     ++		token = reset-tokens:the-only-valid-one
     ++
     ++		allowAnonymous = false
     ++	EOF
     ++
     ++	cat >OUT.expected <<-EOF &&
     ++	WWW-Authenticate: no-params
     ++	WWW-Authenticate: with-params foo="replaced" q=1
     ++	WWW-Authenticate: no-explicit-challenge
     ++	WWW-Authenticate: reset-tokens
     ++
     ++	Error: 401 Unauthorized
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     ++
     ++	curl --include $ORIGIN_URL >OUT.curl &&
     ++	tr -d "\r" <OUT.curl | sed -n "/WWW-Authenticate/,\$p" >OUT.actual &&
     ++
     ++	test_cmp OUT.expected OUT.actual
     ++'
     ++
     + test_expect_success 'http auth anonymous no challenge' '
     + 	test_when_finished "per_test_cleanup" &&
     + 
     +-	start_http_server &&
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++		allowAnonymous = true
     ++	EOF
     ++
     ++	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     + 
     + 	# Attempt to read from a protected repository
     + 	git ls-remote $ORIGIN_URL
  -:  ----------- >  9:  76125cdf239 test-http-server: add sending of arbitrary headers
  7:  9c4d25945dd = 10:  cc9a220ed1f http: replace unsafe size_t multiplication with st_mult
  8:  65a620b08ef <  -:  ----------- strvec: expose strvec_push_nodup for external use
  9:  bcfec529d95 ! 11:  bc1ac8d3eb3 http: read HTTP WWW-Authenticate response headers
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +		if (!values->nr) {
      +			BUG("should have at least one existing header value");
      +		} else if (buf.len) {
     -+			const char *prev = values->v[values->nr - 1];
     -+			struct strbuf append = STRBUF_INIT;
     -+			strbuf_addstr(&append, prev);
     ++			char *prev = xstrdup(values->v[values->nr - 1]);
      +
      +			/* Join two non-empty values with a single space. */
     -+			if (append.len)
     -+				strbuf_addch(&append, ' ');
     -+
     -+			strbuf_addbuf(&append, &buf);
     ++			const char *const sp = *prev ? " " : "";
      +
      +			strvec_pop(values);
     -+			strvec_push_nodup(values, strbuf_detach(&append, NULL));
     ++			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
     ++			free(prev);
      +		}
      +
      +		goto exit;
 10:  af66d2d2ede ! 12:  7c8229f0b11 credential: add WWW-Authenticate header to cred requests
     @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const
      +static void credential_write_strvec(FILE *fp, const char *key,
      +				    const struct strvec *vec)
      +{
     -+	int i = 0;
     -+	const char *full_key = xstrfmt("%s[]", key);
     -+	for (; i < vec->nr; i++) {
     ++	char *full_key = xstrfmt("%s[]", key);
     ++	for (size_t i = 0; i < vec->nr; i++) {
      +		credential_write_item(fp, full_key, vec->v[i], 0);
      +	}
     -+	free((void*)full_key);
     ++	free(full_key);
      +}
      +
       void credential_write(const struct credential *c, FILE *fp)
     @@ credential.c: void credential_write(const struct credential *c, FILE *fp)
       
       static int run_credential_helper(struct credential *c,
      
     - ## t/helper/test-credential-helper-replay.sh (new) ##
     + ## t/lib-credential-helper.sh (new) ##
      @@
     -+cmd=$1
     -+teefile=$cmd-actual.cred
     -+catfile=$cmd-response.cred
     -+rm -f $teefile
     -+while read line;
     -+do
     -+	if test -z "$line"; then
     -+		break;
     -+	fi
     -+	echo "$line" >> $teefile
     -+done
     -+if test "$cmd" = "get"; then
     -+	cat $catfile
     -+fi
     ++setup_credential_helper() {
     ++	test_expect_success 'setup credential helper' '
     ++		CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
     ++		export CREDENTIAL_HELPER &&
     ++		echo $CREDENTIAL_HELPER &&
     ++
     ++		write_script "$CREDENTIAL_HELPER" <<-\EOF
     ++		cmd=$1
     ++		teefile=$cmd-query.cred
     ++		catfile=$cmd-reply.cred
     ++		sed -n -e "/^$/q" -e "p" >> $teefile
     ++		if test "$cmd" = "get"; then
     ++			cat $catfile
     ++		fi
     ++		EOF
     ++	'
     ++}
     ++
     ++set_credential_reply() {
     ++	cat >"$TRASH_DIRECTORY/$1-reply.cred"
     ++}
     ++
     ++expect_credential_query() {
     ++	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
     ++	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
     ++		 "$TRASH_DIRECTORY/$1-query.cred"
     ++}
      
       ## t/t5556-http-auth.sh ##
     -@@ t/t5556-http-auth.sh: PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
     - SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     +@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
     + 
     + TEST_NO_CREATE_REPO=1
     + . ./test-lib.sh
     ++. "$TEST_DIRECTORY"/lib-credential-helper.sh
       
     - PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     -+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
     -+	&& export CREDENTIAL_HELPER
     + test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
       
     - test_expect_success 'setup repos' '
     - 	test_create_repo "$REPO_DIR" &&
     +@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
     + 	git -C "$REPO_DIR" branch -M main
     + '
     + 
     ++setup_credential_helper
     ++
     + stop_http_server () {
     + 	if ! test -f "$PID_FILE"
     + 	then
      @@ t/t5556-http-auth.sh: start_http_server () {
       
       per_test_cleanup () {
     @@ t/t5556-http-auth.sh: start_http_server () {
      +	rm -f auth.config
       }
       
     - test_expect_success 'http auth anonymous no challenge' '
     - 	test_when_finished "per_test_cleanup" &&
     --	start_http_server &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+	    allowAnonymous = true
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     - 
     - 	# Attempt to read from a protected repository
     + test_expect_success CURL 'http auth server auth config' '
     +@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
       	git ls-remote $ORIGIN_URL
       '
       
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >store-expected.cred <<-EOF &&
     ++	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	wwwauth[]=basic realm="example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query store <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
      +'
      +
      +test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     -+	    extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
     ++		extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     -+	wwwauth[]=bEaRer auThoRiTy="id.example.com"
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >store-expected.cred <<-EOF &&
     ++	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	wwwauth[]=basic realm="example.com"
     ++	wwwauth[]=bEaRer auThoRiTy="id.example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query store <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
      +'
      +
      +test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     ++		challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >store-expected.cred <<-EOF &&
     ++	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     ++	wwwauth[]=basic realm="example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query store <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
      +'
      +
      +test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     -+	    extraheader = "WWW-Authenticate:"
     -+	    extraheader = " "
     -+	    extraheader = " bearer authority=\"id.example.com\""
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
     ++		extraheader = "WWW-Authenticate:"
     ++		extraheader = " "
     ++		extraheader = " bearer authority=\"id.example.com\""
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     -+	wwwauth[]=bearer authority="id.example.com"
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >store-expected.cred <<-EOF &&
     ++	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	wwwauth[]=basic realm="example.com"
     ++	wwwauth[]=bearer authority="id.example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query store <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
      +'
      +
      +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = "foobar:alg=test widget=1"
     -+	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     ++		challenge = "foobar:alg=test widget=1"
     ++		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=foobar alg=test widget=1
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >store-expected.cred <<-EOF &&
     ++	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	wwwauth[]=foobar alg=test widget=1
     ++	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     ++	wwwauth[]=basic realm="example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query store <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
     -+
     -+	git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp store-expected.cred store-actual.cred
      +'
      +
      +test_expect_success 'http auth www-auth headers to credential helper invalid' '
     @@ t/t5556-http-auth.sh: start_http_server () {
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     -+	    challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     -+	    challenge = basic:realm=\"example.com\"
     -+	    token = basic:$USERPASS64
     ++		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     ++		challenge = basic:realm=\"example.com\"
     ++		token = basic:$USERPASS64
      +	EOF
      +
      +	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
     -+	cat >get-expected.cred <<-EOF &&
     ++	set_credential_reply get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	username=alice
     ++	password=invalid-passwd
      +	EOF
      +
     -+	cat >erase-expected.cred <<-EOF &&
     ++	test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++
     ++	expect_credential_query get <<-EOF &&
      +	protocol=http
      +	host=$HOST_PORT
     -+	username=alice
     -+	password=invalid-passwd
      +	wwwauth[]=bearer authority="id.example.com" q=1 p=0
      +	wwwauth[]=basic realm="example.com"
      +	EOF
      +
     -+	cat >get-response.cred <<-EOF &&
     ++	expect_credential_query erase <<-EOF
      +	protocol=http
      +	host=$HOST_PORT
      +	username=alice
      +	password=invalid-passwd
     ++	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     ++	wwwauth[]=basic realm="example.com"
      +	EOF
     -+
     -+	test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
     -+
     -+	test_cmp get-expected.cred get-actual.cred &&
     -+	test_cmp erase-expected.cred erase-actual.cred
      +'
      +
       test_done

-- 
gitgitgadget

[PATCH v6 01/12] daemon: libify socket setup and option functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions for setting up listening sockets and keep-alive options
from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
dependencies on global state by inlining the behaviour at the callsites
for all libified functions.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile       |   1 +
 daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h |  23 ++++++
 daemon.c       | 214 +------------------------------------------------
 4 files changed, 237 insertions(+), 210 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h

diff --git a/Makefile b/Makefile
index b258fdbed86..2654094dbb5 100644
--- a/Makefile
+++ b/Makefile
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o
 LIB_OBJS += csum-file.o
 LIB_OBJS += ctype.o
 LIB_OBJS += date.o
+LIB_OBJS += daemon-utils.o
 LIB_OBJS += decorate.o
 LIB_OBJS += delta-islands.o
 LIB_OBJS += diagnose.o
diff --git a/daemon-utils.c b/daemon-utils.c
new file mode 100644
index 00000000000..b96b55962db
--- /dev/null
+++ b/daemon-utils.c
@@ -0,0 +1,209 @@
+#include "cache.h"
+#include "daemon-utils.h"
+
+void set_keep_alive(int sockfd, log_fn logerror)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (reuseaddr && set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd, logerror);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (reuseaddr && set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd, logerror);
+
+	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror)
+{
+	if (!listen_addr->nr)
+		setup_named_sock(NULL, listen_port, socklist, reuseaddr,
+				 logerror);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist, reuseaddr,
+						   logerror);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
diff --git a/daemon-utils.h b/daemon-utils.h
new file mode 100644
index 00000000000..6710a2a6dc0
--- /dev/null
+++ b/daemon-utils.h
@@ -0,0 +1,23 @@
+#ifndef DAEMON_UTILS_H
+#define DAEMON_UTILS_H
+
+#include "git-compat-util.h"
+#include "string-list.h"
+
+typedef void (*log_fn)(const char *msg, ...);
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+/* Enable sending of keep-alive messages on the socket. */
+void set_keep_alive(int sockfd, log_fn logerror);
+
+/* Setup a number of sockets to listen on the provided addresses. */
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror);
+
+#endif
diff --git a/daemon.c b/daemon.c
index 0ae7d12b5c1..1ed4e705680 100644
--- a/daemon.c
+++ b/daemon.c
@@ -1,9 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "daemon-utils.h"
 #include "pkt-line.h"
 #include "run-command.h"
 #include "strbuf.h"
-#include "string-list.h"
 
 #ifdef NO_INITGROUPS
 #define initgroups(x, y) (0) /* nothing */
@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)
 	strbuf_release(&hi->tcp_port);
 }
 
-static void set_keep_alive(int sockfd)
-{
-	int ka = 1;
-
-	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
-		if (errno != ENOTSOCK)
-			logerror("unable to set SO_KEEPALIVE on socket: %s",
-				strerror(errno));
-	}
-}
-
 static int execute(void)
 {
 	char *line = packet_buffer;
@@ -759,7 +748,7 @@ static int execute(void)
 	if (addr)
 		loginfo("Connection from %s:%s", addr, port);
 
-	set_keep_alive(0);
+	set_keep_alive(0, logerror);
 	alarm(init_timeout ? init_timeout : timeout);
 	pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
 	alarm(0);
@@ -938,202 +927,6 @@ static void child_handler(int signo)
 	signal(SIGCHLD, child_handler);
 }
 
-static int set_reuse_addr(int sockfd)
-{
-	int on = 1;
-
-	if (!reuseaddr)
-		return 0;
-	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
-			  &on, sizeof(on));
-}
-
-struct socketlist {
-	int *list;
-	size_t nr;
-	size_t alloc;
-};
-
-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
-{
-#ifdef NO_IPV6
-	static char ip[INET_ADDRSTRLEN];
-#else
-	static char ip[INET6_ADDRSTRLEN];
-#endif
-
-	switch (family) {
-#ifndef NO_IPV6
-	case AF_INET6:
-		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
-		break;
-#endif
-	case AF_INET:
-		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
-		break;
-	default:
-		xsnprintf(ip, sizeof(ip), "<unknown>");
-	}
-	return ip;
-}
-
-#ifndef NO_IPV6
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	int socknum = 0;
-	char pbuf[NI_MAXSERV];
-	struct addrinfo hints, *ai0, *ai;
-	int gai;
-	long flags;
-
-	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
-	memset(&hints, 0, sizeof(hints));
-	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = SOCK_STREAM;
-	hints.ai_protocol = IPPROTO_TCP;
-	hints.ai_flags = AI_PASSIVE;
-
-	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
-	if (gai) {
-		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
-		return 0;
-	}
-
-	for (ai = ai0; ai; ai = ai->ai_next) {
-		int sockfd;
-
-		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
-		if (sockfd < 0)
-			continue;
-		if (sockfd >= FD_SETSIZE) {
-			logerror("Socket descriptor too large");
-			close(sockfd);
-			continue;
-		}
-
-#ifdef IPV6_V6ONLY
-		if (ai->ai_family == AF_INET6) {
-			int on = 1;
-			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
-				   &on, sizeof(on));
-			/* Note: error is not fatal */
-		}
-#endif
-
-		if (set_reuse_addr(sockfd)) {
-			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-			close(sockfd);
-			continue;
-		}
-
-		set_keep_alive(sockfd);
-
-		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
-			logerror("Could not bind to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-		if (listen(sockfd, 5) < 0) {
-			logerror("Could not listen to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-
-		flags = fcntl(sockfd, F_GETFD, 0);
-		if (flags >= 0)
-			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-		socklist->list[socklist->nr++] = sockfd;
-		socknum++;
-	}
-
-	freeaddrinfo(ai0);
-
-	return socknum;
-}
-
-#else /* NO_IPV6 */
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	struct sockaddr_in sin;
-	int sockfd;
-	long flags;
-
-	memset(&sin, 0, sizeof sin);
-	sin.sin_family = AF_INET;
-	sin.sin_port = htons(listen_port);
-
-	if (listen_addr) {
-		/* Well, host better be an IP address here. */
-		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
-			return 0;
-	} else {
-		sin.sin_addr.s_addr = htonl(INADDR_ANY);
-	}
-
-	sockfd = socket(AF_INET, SOCK_STREAM, 0);
-	if (sockfd < 0)
-		return 0;
-
-	if (set_reuse_addr(sockfd)) {
-		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	set_keep_alive(sockfd);
-
-	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
-		logerror("Could not bind to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	if (listen(sockfd, 5) < 0) {
-		logerror("Could not listen to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	flags = fcntl(sockfd, F_GETFD, 0);
-	if (flags >= 0)
-		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-	socklist->list[socklist->nr++] = sockfd;
-	return 1;
-}
-
-#endif
-
-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	if (!listen_addr->nr)
-		setup_named_sock(NULL, listen_port, socklist);
-	else {
-		int i, socknum;
-		for (i = 0; i < listen_addr->nr; i++) {
-			socknum = setup_named_sock(listen_addr->items[i].string,
-						   listen_port, socklist);
-
-			if (socknum == 0)
-				logerror("unable to allocate any listen sockets for host %s on port %u",
-					 listen_addr->items[i].string, listen_port);
-		}
-	}
-}
-
 static int service_loop(struct socketlist *socklist)
 {
 	struct pollfd *pfd;
@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,
 {
 	struct socketlist socklist = { NULL, 0, 0 };
 
-	socksetup(listen_addr, listen_port, &socklist);
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr,
+		  logerror);
 	if (socklist.nr == 0)
 		die("unable to allocate any listen sockets on port %u",
 		    listen_port);
-- 
gitgitgadget

[PATCH v6 02/12] daemon: libify child process handling functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions and structures for managing child processes started
from the parent daemon-like process from `daemon.c` to the new shared
`daemon-utils.{c,h}` files.

One minor functional change is introduced to `check_dead_children()`
where the logging of a dead/disconnected child is now optional. With the
'libification' of these functions we extract the call to `loginfo` to a
call to a function pointer, and guard the log message creation and
logging behind a `NULL` check. Callers can now skip logging by passing
`NULL` as the `log_fn loginfo` argument.
The behaviour of callers in `daemon.c` remains the same (save one extra
NULL check)  however as a pointer to `loginfo` is always passed.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h | 32 ++++++++++++++++++
 daemon.c       | 92 +++-----------------------------------------------
 3 files changed, 114 insertions(+), 87 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index b96b55962db..8506664b440 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 		}
 	}
 }
+
+static int addrcmp(const struct sockaddr_storage *s1,
+    const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn , unsigned int *live_children)
+{
+	struct child *newborn, **cradle;
+
+	CALLOC_ARRAY(newborn, 1);
+	(*live_children)++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+void kill_some_child(struct child *firstborn)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			if (loginfo) {
+				const char *dead = "";
+				if (status)
+					dead = " (with error)";
+				loginfo("[%"PRIuMAX"] Disconnected%s",
+					(uintmax_t)pid, dead);
+			}
+
+			/* remove the child */
+			*cradle = blanket->next;
+			(*live_children)--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
diff --git a/daemon-utils.h b/daemon-utils.h
index 6710a2a6dc0..97e5cae20b8 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -2,6 +2,7 @@
 #define DAEMON_UTILS_H
 
 #include "git-compat-util.h"
+#include "run-command.h"
 #include "string-list.h"
 
 typedef void (*log_fn)(const char *msg, ...);
@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 	       struct socketlist *socklist, int reuseaddr,
 	       log_fn logerror);
 
+struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+};
+
+/*
+ * Add the child_process to the set of children and increment the number of
+ * live children.
+ */
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn, unsigned int *live_children);
+
+/*
+ * Kill the newest connection from a duplicate IP.
+ *
+ * This function should be called if the number of connections grows
+ * past the maximum number of allowed connections.
+ */
+void kill_some_child(struct child *firstborn);
+
+/*
+ * Check for children that have disconnected and remove them from the
+ * active set, decrementing the number of live children.
+ *
+ * Optionally log the child PID that disconnected by passing a loginfo
+ * function.
+ */
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo);
+
 #endif
diff --git a/daemon.c b/daemon.c
index 1ed4e705680..ec3b407ecbc 100644
--- a/daemon.c
+++ b/daemon.c
@@ -785,93 +785,11 @@ static int execute(void)
 	return -1;
 }
 
-static int addrcmp(const struct sockaddr_storage *s1,
-    const struct sockaddr_storage *s2)
-{
-	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
-	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
-
-	if (sa1->sa_family != sa2->sa_family)
-		return sa1->sa_family - sa2->sa_family;
-	if (sa1->sa_family == AF_INET)
-		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
-		    &((struct sockaddr_in *)s2)->sin_addr,
-		    sizeof(struct in_addr));
-#ifndef NO_IPV6
-	if (sa1->sa_family == AF_INET6)
-		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
-		    &((struct sockaddr_in6 *)s2)->sin6_addr,
-		    sizeof(struct in6_addr));
-#endif
-	return 0;
-}
-
 static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child {
-	struct child *next;
-	struct child_process cld;
-	struct sockaddr_storage address;
-} *firstborn;
-
-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
-{
-	struct child *newborn, **cradle;
-
-	CALLOC_ARRAY(newborn, 1);
-	live_children++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
-			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
-}
-
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
-static void kill_some_child(void)
-{
-	const struct child *blanket, *next;
-
-	if (!(blanket = firstborn))
-		return;
-
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
-			break;
-		}
-}
-
-static void check_dead_children(void)
-{
-	int status;
-	pid_t pid;
-
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
-			const char *dead = "";
-			if (status)
-				dead = " (with error)";
-			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
-
-			/* remove the child */
-			*cradle = blanket->next;
-			live_children--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
-		} else
-			cradle = &blanket->next;
-}
+static struct child *firstborn;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child();
+		kill_some_child(firstborn);
 		sleep(1);  /* give it some time to die */
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen);
+		add_child(&cld, addr, addrlen, firstborn, &live_children);
 }
 
 static void child_handler(int signo)
@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

[PATCH v6 03/12] daemon: rename some esoteric/laboured terminology

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rename some of the variables and function arguments used to manage child
processes. The existing names are esoteric; stretching an analogy too
far to the point of being confusing to understand.

Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket"
to "current" and "cradle" to "ptr".

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 46 +++++++++++++++++++++++-----------------------
 daemon-utils.h |  6 +++---
 daemon.c       | 10 +++++-----
 3 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index 8506664b440..f23ea35ed7b 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,
 }
 
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn , unsigned int *live_children)
+	       struct child *first_child, unsigned int *live_children)
 {
-	struct child *newborn, **cradle;
+	struct child *new_cld, **current;
 
-	CALLOC_ARRAY(newborn, 1);
+	CALLOC_ARRAY(new_cld, 1);
 	(*live_children)++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
+	memcpy(&new_cld->cld, cld, sizeof(*cld));
+	memcpy(&new_cld->address, addr, addrlen);
+	for (current = &first_child; *current; current = &(*current)->next)
+		if (!addrcmp(&(*current)->address, &new_cld->address))
 			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
+	new_cld->next = *current;
+	*current = new_cld;
 }
 
-void kill_some_child(struct child *firstborn)
+void kill_some_child(struct child *first_child)
 {
-	const struct child *blanket, *next;
+	const struct child *current, *next;
 
-	if (!(blanket = firstborn))
+	if (!(current = first_child))
 		return;
 
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
+	for (; (next = current->next); current = next)
+		if (!addrcmp(&current->address, &next->address)) {
+			kill(current->cld.pid, SIGTERM);
 			break;
 		}
 }
 
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
 			 log_fn loginfo)
 {
 	int status;
 	pid_t pid;
 
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+	struct child **ptr, *current;
+	for (ptr = &first_child; (current = *ptr);)
+		if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
 			if (loginfo) {
 				const char *dead = "";
 				if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,
 			}
 
 			/* remove the child */
-			*cradle = blanket->next;
+			*ptr = current->next;
 			(*live_children)--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
+			child_process_clear(&current->cld);
+			free(current);
 		} else
-			cradle = &blanket->next;
+			ptr = &current->next;
 }
diff --git a/daemon-utils.h b/daemon-utils.h
index 97e5cae20b8..c866e9c9a4e 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -32,7 +32,7 @@ struct child {
  * live children.
  */
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn, unsigned int *live_children);
+	       struct child *first_child, unsigned int *live_children);
 
 /*
  * Kill the newest connection from a duplicate IP.
@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl
  * This function should be called if the number of connections grows
  * past the maximum number of allowed connections.
  */
-void kill_some_child(struct child *firstborn);
+void kill_some_child(struct child *first_child);
 
 /*
  * Check for children that have disconnected and remove them from the
@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);
  * Optionally log the child PID that disconnected by passing a loginfo
  * function.
  */
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
 			 log_fn loginfo);
 
 #endif
diff --git a/daemon.c b/daemon.c
index ec3b407ecbc..d3e7d81de18 100644
--- a/daemon.c
+++ b/daemon.c
@@ -789,7 +789,7 @@ static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child *firstborn;
+static struct child *first_child;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child(firstborn);
+		kill_some_child(first_child);
 		sleep(1);  /* give it some time to die */
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen, firstborn, &live_children);
+		add_child(&cld, addr, addrlen, first_child, &live_children);
 }
 
 static void child_handler(int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

[PATCH v6 04/12] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.

Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                            |   1 +
 contrib/buildsystems/CMakeLists.txt |  11 +-
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 385 ++++++++++++++++++++++++++++
 4 files changed, 396 insertions(+), 2 deletions(-)
 create mode 100644 t/helper/test-http-server.c

diff --git a/Makefile b/Makefile
index 2654094dbb5..3cd61c792ac 100644
--- a/Makefile
+++ b/Makefile
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o
 # Do not add more tests here unless they have extra dependencies. Add
 # them in TEST_BUILTINS_OBJS above.
 TEST_PROGRAMS_NEED_X += test-fake-ssh
+TEST_PROGRAMS_NEED_X += test-http-server
 TEST_PROGRAMS_NEED_X += test-tool
 
 TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..5d949dcb16c 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -961,6 +961,9 @@ if(BUILD_TESTING)
 add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
 target_link_libraries(test-fake-ssh common-main)
 
+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+target_link_libraries(test-http-server common-main)
+
 #reftable-tests
 parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
 list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC)
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
 	set_target_properties(test-fake-ssh test-tool
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
 endif()
 
 #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts
 	git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
 
 set(wrapper_test_scripts
-	test-fake-ssh test-tool)
-
+	test-http-server test-fake-ssh test-tool)
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..11071b1dd89
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,385 @@
+#include "daemon-utils.h"
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+static unsigned int timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client.  The worker talks to the client
+ * on 0 and 1.
+ */
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK       = 0,
+
+	/*
+	 * Various errors while processing the request and/or the response.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_IO_ERROR = 1<<0,
+
+	/*
+	 * Close the socket and clean up.  Does not imply an error.
+	 */
+	WR_HANGUP   = 1<<1,
+};
+
+static enum worker_result worker(void)
+{
+	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0, logerror);
+
+	while (1) {
+		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+			logerror("unable to write response");
+			wr = WR_IO_ERROR;
+		}
+
+		if (wr != WR_OK)
+			break;
+	}
+
+	close(STDIN_FILENO);
+	close(STDOUT_FILENO);
+
+	return !!(wr & WR_IO_ERROR);
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child *first_child;
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child(first_child);
+		sleep(1);  /* give it some time to die */
+		check_dead_children(first_child, &live_children, loginfo);
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen, first_child, &live_children);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children(first_child, &live_children, loginfo);
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances.  So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+
+	trace2_cmd_name("test-http-server");
+	trace2_cmd_list_config();
+	trace2_cmd_list_env_vars();
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
-- 
gitgitgadget

Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> [...]
> +enum worker_result {
> +	/*
> +	 * Operation successful.
> +	 * Caller *might* keep the socket open and allow keep-alive.
> +	 */
> +	WR_OK       = 0,
> [...]
> +	enum worker_result wr = WR_OK;
> +
> +	if (client_addr)
> +		loginfo("Connection from %s:%s", client_addr, client_port);
> +
> +	set_keep_alive(0, logerror);
> +
> +	while (1) {
> +		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
> +			logerror("unable to write response");
> +			wr = WR_IO_ERROR;
> +		}
> +
> +		if (wr != WR_OK)
> +			break;
> +	}
> +
> +	close(STDIN_FILENO);
> +	close(STDOUT_FILENO);
> +
> +	return !!(wr & WR_IO_ERROR);
> +}

We have cases where we assign "0" to a bitfield-looking structure like
this, but only in cases where we're planning to use it as a boolean too.

Or, in other cases where we want some to be explicitly <-1.

Here though we're adding a mixed "OK" and error use, which seems a bit
odd. Shouldn't we pick one or the other?

So far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we
also use the bitfield-looking thing as a return value from main()....

Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham]

On 2023-01-18 03:04, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>> [...]
>> +enum worker_result {
>> +	/*
>> +	 * Operation successful.
>> +	 * Caller *might* keep the socket open and allow keep-alive.
>> +	 */
>> +	WR_OK       = 0,
>> [...]
>> +	enum worker_result wr = WR_OK;
>> +
>> +	if (client_addr)
>> +		loginfo("Connection from %s:%s", client_addr, client_port);
>> +
>> +	set_keep_alive(0, logerror);
>> +
>> +	while (1) {
>> +		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
>> +			logerror("unable to write response");
>> +			wr = WR_IO_ERROR;
>> +		}
>> +
>> +		if (wr != WR_OK)
>> +			break;
>> +	}
>> +
>> +	close(STDIN_FILENO);
>> +	close(STDOUT_FILENO);
>> +
>> +	return !!(wr & WR_IO_ERROR);
>> +}
> 
> We have cases where we assign "0" to a bitfield-looking structure like
> this, but only in cases where we're planning to use it as a boolean too.
> 
> Or, in other cases where we want some to be explicitly <-1.
> 
> Here though we're adding a mixed "OK" and error use, which seems a bit
> odd. Shouldn't we pick one or the other?

You make a fair point about bitfields vs simple integer values. This was a
holdover from previous early hacking on this work where I had the bitfield
serve as a way to communicate the aspects of "does this count as an error?"
and "should we close the connection?".

Upon second thought, I think just simple integer values would be fine as
really only an "OK" and "HANGUP" are non-errors (the latter being the case
that the client gracefully ended the connection without an error and we
should exit).

Check for my next iteration for a rework on these `worker_result` values.

> So far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we
> also use the bitfield-looking thing as a return value from main()....

We don't use the `enum worker_result` values as a return from `main`. We only
ever return 0 or 1 as we `return worker()` from `main`, and the only `return`
from `worker()` is `!!(wr & WR_IO_ERROR)` - 1 if we have `WR_IO_ERROR` set,
otherwise 0.

Thanks,
Matthew

[PATCH v6 05/12] test-http-server: add HTTP error response function

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----
 1 file changed, 53 insertions(+), 5 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 11071b1dd89..6cdac223a55 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -83,9 +83,59 @@ enum worker_result {
 	WR_HANGUP   = 1<<1,
 };
 
+static enum worker_result send_http_error(
+	int fd,
+	int http_code, const char *http_code_name,
+	int retry_after_seconds, struct string_list *response_headers,
+	enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n",
+		    http_code, http_code_name);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_IO_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
 static enum worker_result worker(void)
 {
-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -96,10 +146,8 @@ static enum worker_result worker(void)
 	set_keep_alive(0, logerror);
 
 	while (1) {
-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
-			logerror("unable to write response");
-			wr = WR_IO_ERROR;
-		}
+		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
+				     NULL, WR_OK | WR_HANGUP);
 
 		if (wr != WR_OK)
 			break;
-- 
gitgitgadget

Re: [PATCH v6 05/12] test-http-server: add HTTP error response function

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Introduce a function to the test-http-server test helper to write more
> full and valid HTTP error responses, including all the standard response
> headers like `Server` and `Date`.
>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----
>  1 file changed, 53 insertions(+), 5 deletions(-)
>
> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> index 11071b1dd89..6cdac223a55 100644
> --- a/t/helper/test-http-server.c
> +++ b/t/helper/test-http-server.c
> @@ -83,9 +83,59 @@ enum worker_result {
>  	WR_HANGUP   = 1<<1,
>  };

...okey, this is the commit that makes use of WR_HANGUP. Whatever else
we do, let's then squash that addition into this change.

> +static enum worker_result send_http_error(
> +	int fd,
> +	int http_code, const char *http_code_name,
> +	int retry_after_seconds, struct string_list *response_headers,
> +	enum worker_result wr_in)

In general in this series you are mis-indenting argument lists. Our
usual style is to wrap at 79 characters, then to align (with tabs and
spaces) with the "(".

So in this case:

static enum worker_result send_http_error(int fd, int http_code,
					  const char *http_code_name,
					  int retry_after_seconds,
					  struct string_list *response_headers,
					  enum worker_result wr_in)

> +{
> +	struct strbuf response_header = STRBUF_INIT;
> +	struct strbuf response_content = STRBUF_INIT;
> +	struct string_list_item *h;
> +	enum worker_result wr;
> +
> +	strbuf_addf(&response_content, "Error: %d %s\r\n",
> +		    http_code, http_code_name);


Ditto here, where "http_code" should go on the preceding line...

> +	if (retry_after_seconds > 0)
> +		strbuf_addf(&response_content, "Retry-After: %d\r\n",
> +			    retry_after_seconds);
> +
> +	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);

...and here there's a lack of such wrapping...

> +	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
> +	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
> +	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
> +	if (retry_after_seconds > 0)
> +		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
> +	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
> +	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));

...here you're adding strange whitespace at the start of an argument list...

> +	if (response_headers)
> +		for_each_string_list_item(h, response_headers)
> +			strbuf_addf(&response_header, "%s\r\n", h->string);
> +	strbuf_addstr(&response_header, "\r\n");

To comment on the code a bit, this whole thing would be more readable
IMO if your own headers were also a "struct string_list". Yes we'd waste
a bit more memory, but in this case that's fine..

I.e. don't add the "\r\n" every time, just:

	string_list_append(&headers, "Cache-Control: private");

etc.

Then at the end you'd do e.g.:

	add_headers(&buf, &headers);
	if (response_headers)
		add_headers(&buf, response_headers);

Where the add_headers() is a trivial "static" helper which does that
for_each_string_list_item() loop above.

>  	while (1) {
> -		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
> -			logerror("unable to write response");
> -			wr = WR_IO_ERROR;
> -		}
> +		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
> +				     NULL, WR_OK | WR_HANGUP);

This *does* use correct wrapping & indenation for a continuing argument
list.

Re: [PATCH v6 05/12] test-http-server: add HTTP error response function

[Matthew John Cheetham]

On 2023-01-18 03:07, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Introduce a function to the test-http-server test helper to write more
>> full and valid HTTP error responses, including all the standard response
>> headers like `Server` and `Date`.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----
>>  1 file changed, 53 insertions(+), 5 deletions(-)
>>
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> index 11071b1dd89..6cdac223a55 100644
>> --- a/t/helper/test-http-server.c
>> +++ b/t/helper/test-http-server.c
>> @@ -83,9 +83,59 @@ enum worker_result {
>>  	WR_HANGUP   = 1<<1,
>>  };
> 
> ...okey, this is the commit that makes use of WR_HANGUP. Whatever else
> we do, let's then squash that addition into this change.
> 
>> +static enum worker_result send_http_error(
>> +	int fd,
>> +	int http_code, const char *http_code_name,
>> +	int retry_after_seconds, struct string_list *response_headers,
>> +	enum worker_result wr_in)
> 
> In general in this series you are mis-indenting argument lists. Our
> usual style is to wrap at 79 characters, then to align (with tabs and
> spaces) with the "(".
> 
> So in this case:
> 
> static enum worker_result send_http_error(int fd, int http_code,
> 					  const char *http_code_name,
> 					  int retry_after_seconds,
> 					  struct string_list *response_headers,
> 					  enum worker_result wr_in)
> 
>> +{
>> +	struct strbuf response_header = STRBUF_INIT;
>> +	struct strbuf response_content = STRBUF_INIT;
>> +	struct string_list_item *h;
>> +	enum worker_result wr;
>> +
>> +	strbuf_addf(&response_content, "Error: %d %s\r\n",
>> +		    http_code, http_code_name);
> 
> 
> Ditto here, where "http_code" should go on the preceding line...
> 
>> +	if (retry_after_seconds > 0)
>> +		strbuf_addf(&response_content, "Retry-After: %d\r\n",
>> +			    retry_after_seconds);
>> +
>> +	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
> 
> ...and here there's a lack of such wrapping...
> 
>> +	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
>> +	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
>> +	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
>> +	if (retry_after_seconds > 0)
>> +		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
>> +	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
>> +	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
> 
> ...here you're adding strange whitespace at the start of an argument list...
> 
>> +	if (response_headers)
>> +		for_each_string_list_item(h, response_headers)
>> +			strbuf_addf(&response_header, "%s\r\n", h->string);
>> +	strbuf_addstr(&response_header, "\r\n");

Argh! Thanks again for catching these. I shall address them.

> To comment on the code a bit, this whole thing would be more readable
> IMO if your own headers were also a "struct string_list". Yes we'd waste
> a bit more memory, but in this case that's fine..
> 
> I.e. don't add the "\r\n" every time, just:
> 
> 	string_list_append(&headers, "Cache-Control: private");
> 
> etc.
> 
> Then at the end you'd do e.g.:
> 
> 	add_headers(&buf, &headers);
> 	if (response_headers)
> 		add_headers(&buf, response_headers);
> 
> Where the add_headers() is a trivial "static" helper which does that
> for_each_string_list_item() loop above.

In reality this only helps simplify the code in the case of a simple static
header like "Cache-Control: private". There's no `string_list_appendf` or
similar where I need to append a header that contains dynamic information
(date, content length, etc).

Building the `strbuf` directly, and specifying the CRLF seems a lot easier IMO.

>>  	while (1) {
>> -		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
>> -			logerror("unable to write response");
>> -			wr = WR_IO_ERROR;
>> -		}
>> +		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
>> +				     NULL, WR_OK | WR_HANGUP);
> 
> This *does* use correct wrapping & indenation for a continuing argument
> list.


Thanks,
Matthew

[PATCH v6 06/12] test-http-server: add HTTP request parsing

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add ability to parse HTTP requests to the test-http-server test helper.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-
 1 file changed, 173 insertions(+), 2 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 6cdac223a55..36f4a54fe6d 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -83,6 +83,42 @@ enum worker_result {
 	WR_HANGUP   = 1<<1,
 };
 
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	ssize_t content_length;
+};
+
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = -1 \
+	}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
 static enum worker_result send_http_error(
 	int fd,
 	int http_code, const char *http_code_name,
@@ -134,8 +170,136 @@ done:
 	return wr;
 }
 
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_OK | WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_IO_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* also store common request headers as struct req members */
+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
+			req->content_length = strtol(hv, &hp, 10);
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->content_length >= 0)
+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+			       WR_OK | WR_HANGUP);
+}
+
 static enum worker_result worker(void)
 {
+	struct req req = REQ__INIT;
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -146,9 +310,16 @@ static enum worker_result worker(void)
 	set_keep_alive(0, logerror);
 
 	while (1) {
-		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-				     NULL, WR_OK | WR_HANGUP);
+		req__release(&req);
+
+		alarm(timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr != WR_OK)
+			break;
 
+		wr = dispatch(&req);
 		if (wr != WR_OK)
 			break;
 	}
-- 
gitgitgadget

Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Add ability to parse HTTP requests to the test-http-server test helper.
>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 173 insertions(+), 2 deletions(-)
>
> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
> index 6cdac223a55..36f4a54fe6d 100644
> --- a/t/helper/test-http-server.c
> +++ b/t/helper/test-http-server.c
> @@ -83,6 +83,42 @@ enum worker_result {
>  	WR_HANGUP   = 1<<1,
>  };
>  
> +/*
> + * Fields from a parsed HTTP request.
> + */
> +struct req {
> +	struct strbuf start_line;
> +
> +	const char *method;
> +	const char *http_version;
> +
> +	struct strbuf uri_path;
> +	struct strbuf query_args;
> +
> +	struct string_list header_list;
> +	const char *content_type;
> +	ssize_t content_length;
> +};
> +
> +#define REQ__INIT { \
> +	.start_line = STRBUF_INIT, \
> +	.uri_path = STRBUF_INIT, \
> +	.query_args = STRBUF_INIT, \
> +	.header_list = STRING_LIST_INIT_NODUP, \
> +	.content_type = NULL, \
> +	.content_length = -1 \
> +	}

Style nit: Don't indent the trailing "}", and add a "," after the last
"content_length" item.

We omit the comma by convention when there really should not be another
item, such as when we have a "NULL" terminator, here though we might add
a struct element at the end, so...

> +static enum worker_result req__read(struct req *req, int fd)
> +{
> +	struct strbuf h = STRBUF_INIT;
> +	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
> +	int nr_start_line_fields;
> +	const char *uri_target;
> +	const char *query;
> +	char *hp;
> +	const char *hv;
> +
> +	enum worker_result result = WR_OK;
> +
> +	/*
> +	 * Read line 0 of the request and split it into component parts:
> +	 *
> +	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
> +	 *
> +	 */
> +	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
> +		result = WR_OK | WR_HANGUP;
> +		goto done;
> +	}
> +
> +	strbuf_trim_trailing_newline(&req->start_line);
> +
> +	nr_start_line_fields = string_list_split(&start_line_fields,
> +						 req->start_line.buf,
> +						 ' ', -1);
> +	if (nr_start_line_fields != 3) {
> +		logerror("could not parse request start-line '%s'",
> +			 req->start_line.buf);
> +		result = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	req->method = xstrdup(start_line_fields.items[0].string);
> +	req->http_version = xstrdup(start_line_fields.items[2].string);
> +
> +	uri_target = start_line_fields.items[1].string;
> +
> +	if (strcmp(req->http_version, "HTTP/1.1")) {
> +		logerror("unsupported version '%s' (expecting HTTP/1.1)",
> +			 req->http_version);
> +		result = WR_IO_ERROR;
> +		goto done;
> +	}
> +
> +	query = strchr(uri_target, '?');
> +
> +	if (query) {
> +		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
> +		strbuf_addstr(&req->query_args, query + 1);
> +	} else {
> +		strbuf_addstr(&req->uri_path, uri_target);
> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
> +	}
> +
> +	/*
> +	 * Read the set of HTTP headers into a string-list.
> +	 */
> +	while (1) {
> +		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
> +			goto done;
> +		strbuf_trim_trailing_newline(&h);
> +
> +		if (!h.len)
> +			goto done; /* a blank line ends the header */
> +
> +		hp = strbuf_detach(&h, NULL);
> +		string_list_append(&req->header_list, hp);
> +
> +		/* also store common request headers as struct req members */
> +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
> +			req->content_type = hv;
> +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
> +			req->content_length = strtol(hv, &hp, 10);

In POSIX the "ssize_t" is not a "this is the unsigned size_t", but can
be a much smaller integer type (although in practice it tends to be the
signed version of "size_t".

But this seems like a potential overflow trap as a result, but sometimes
we need to live with "ssize_t".

However, in this case it seems like we don't, as it seems the only
reason you init'd this to -1 and then...

> +	if (trace2_is_enabled()) {
> +		struct string_list_item *item;
> +		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
> +		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
> +		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
> +		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
> +		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
> +		if (req->content_length >= 0)
> +			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);

...use that ">= 0" is to keep the state of "did I assign to this above?

So firstly, shouldn't we error or something on a "Content-Length: 0",
and aside from that wouldn't we just have a "int have_content_length =
0" in this function that we'd then flip to 1?

Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing

[Matthew John Cheetham]

On 2023-01-18 03:14, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add ability to parse HTTP requests to the test-http-server test helper.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-
>>  1 file changed, 173 insertions(+), 2 deletions(-)
>>
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> index 6cdac223a55..36f4a54fe6d 100644
>> --- a/t/helper/test-http-server.c
>> +++ b/t/helper/test-http-server.c
>> @@ -83,6 +83,42 @@ enum worker_result {
>>  	WR_HANGUP   = 1<<1,
>>  };
>>  
>> +/*
>> + * Fields from a parsed HTTP request.
>> + */
>> +struct req {
>> +	struct strbuf start_line;
>> +
>> +	const char *method;
>> +	const char *http_version;
>> +
>> +	struct strbuf uri_path;
>> +	struct strbuf query_args;
>> +
>> +	struct string_list header_list;
>> +	const char *content_type;
>> +	ssize_t content_length;
>> +};
>> +
>> +#define REQ__INIT { \
>> +	.start_line = STRBUF_INIT, \
>> +	.uri_path = STRBUF_INIT, \
>> +	.query_args = STRBUF_INIT, \
>> +	.header_list = STRING_LIST_INIT_NODUP, \
>> +	.content_type = NULL, \
>> +	.content_length = -1 \
>> +	}
> 
> Style nit: Don't indent the trailing "}", and add a "," after the last
> "content_length" item.
> 
> We omit the comma by convention when there really should not be another
> item, such as when we have a "NULL" terminator, here though we might add
> a struct element at the end, so...

Sure.

>> +static enum worker_result req__read(struct req *req, int fd)
>> +{
>> +	struct strbuf h = STRBUF_INIT;
>> +	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
>> +	int nr_start_line_fields;
>> +	const char *uri_target;
>> +	const char *query;
>> +	char *hp;
>> +	const char *hv;
>> +
>> +	enum worker_result result = WR_OK;
>> +
>> +	/*
>> +	 * Read line 0 of the request and split it into component parts:
>> +	 *
>> +	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
>> +	 *
>> +	 */
>> +	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
>> +		result = WR_OK | WR_HANGUP;
>> +		goto done;
>> +	}
>> +
>> +	strbuf_trim_trailing_newline(&req->start_line);
>> +
>> +	nr_start_line_fields = string_list_split(&start_line_fields,
>> +						 req->start_line.buf,
>> +						 ' ', -1);
>> +	if (nr_start_line_fields != 3) {
>> +		logerror("could not parse request start-line '%s'",
>> +			 req->start_line.buf);
>> +		result = WR_IO_ERROR;
>> +		goto done;
>> +	}
>> +
>> +	req->method = xstrdup(start_line_fields.items[0].string);
>> +	req->http_version = xstrdup(start_line_fields.items[2].string);
>> +
>> +	uri_target = start_line_fields.items[1].string;
>> +
>> +	if (strcmp(req->http_version, "HTTP/1.1")) {
>> +		logerror("unsupported version '%s' (expecting HTTP/1.1)",
>> +			 req->http_version);
>> +		result = WR_IO_ERROR;
>> +		goto done;
>> +	}
>> +
>> +	query = strchr(uri_target, '?');
>> +
>> +	if (query) {
>> +		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
>> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
>> +		strbuf_addstr(&req->query_args, query + 1);
>> +	} else {
>> +		strbuf_addstr(&req->uri_path, uri_target);
>> +		strbuf_trim_trailing_dir_sep(&req->uri_path);
>> +	}
>> +
>> +	/*
>> +	 * Read the set of HTTP headers into a string-list.
>> +	 */
>> +	while (1) {
>> +		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
>> +			goto done;
>> +		strbuf_trim_trailing_newline(&h);
>> +
>> +		if (!h.len)
>> +			goto done; /* a blank line ends the header */
>> +
>> +		hp = strbuf_detach(&h, NULL);
>> +		string_list_append(&req->header_list, hp);
>> +
>> +		/* also store common request headers as struct req members */
>> +		if (skip_prefix(hp, "Content-Type: ", &hv)) {
>> +			req->content_type = hv;
>> +		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
>> +			req->content_length = strtol(hv, &hp, 10);
> 
> In POSIX the "ssize_t" is not a "this is the unsigned size_t", but can
> be a much smaller integer type (although in practice it tends to be the
> signed version of "size_t".
> 
> But this seems like a potential overflow trap as a result, but sometimes
> we need to live with "ssize_t".
> 
> However, in this case it seems like we don't, as it seems the only
> reason you init'd this to -1 and then...
> 
>> +	if (trace2_is_enabled()) {
>> +		struct string_list_item *item;
>> +		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
>> +		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
>> +		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
>> +		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
>> +		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
>> +		if (req->content_length >= 0)
>> +			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
> 
> ...use that ">= 0" is to keep the state of "did I assign to this above?
> 
> So firstly, shouldn't we error or something on a "Content-Length: 0",
> and aside from that wouldn't we just have a "int have_content_length =
> 0" in this function that we'd then flip to 1?

It seems like the perfect type for such a non-zero-or-error size value; from
POSIX specifications[1]:

> ...
> size_t
>	Used for sizes of objects.
> ssize_t
>	Used for a count of bytes or an error indication.
> ...

But you're probably right here that `ssize_t` isn't that suitable in practice
due to the comically low minimum size of the `SSIZE_MAX` (2^15 I believe).

RFC 9110 §8.6 [2] addresses the `Content-Length` HTTP header and says that its
value should be non-negative, but also have no upper bound; we're gonna have to
set at least some practical limit.

Libcurl handles this by writing it's own parsing function that's good up to
a max 64-bit integer value [3][4].

Given this is for a test helper and only going to be receiving data from tests,
I propose just using something like `uintmax_t` and storing a bit with
`unsigned has_content_length:1;` to show if we actually got a header in the
request or not.

Thanks,
Matthew

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html
[2] https://www.rfc-editor.org/rfc/rfc9110.html#section-8.6
[3] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/http.c#L3348-L3349
[4] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/strtoofft.c#L214-L218

[PATCH v6 07/12] test-http-server: pass Git requests to http-backend

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Teach the test-http-sever test helper to forward Git requests to the
`git-http-backend`.

Introduce a new test script t5556-http-auth.sh that spins up the test
HTTP server and attempts an `ls-remote` on the served repository,
without any authentication.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c |  71 ++++++++++++++++++++++++
 t/t5556-http-auth.sh        | 107 ++++++++++++++++++++++++++++++++++++
 2 files changed, 178 insertions(+)
 create mode 100755 t/t5556-http-auth.sh

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 36f4a54fe6d..ae17c738259 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -291,8 +291,79 @@ done:
 	return result;
 }
 
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		/*
+		 * This regular expression matches all dumb and smart HTTP
+		 * requests that are currently in use, and defined in
+		 * Documentation/gitprotocol-http.txt.
+		 *
+		 */
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	/*
+	 * Note that we always respond with a 200 OK response even if the
+	 * http-backend process exits with an error. This helper is intended
+	 * only to be used to exercise the HTTP auth handling in the Git client,
+	 * and specifically around authentication (not handled by http-backend).
+	 *
+	 * If we wanted to respond with a more 'valid' HTTP response status then
+	 * we'd need to buffer the output of http-backend, wait for and grok the
+	 * exit status of the process, then write the HTTP status line followed
+	 * by the http-backend output. This is outside of the scope of this test
+	 * helper's use at time of writing.
+	 */
+	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+			req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
+				req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
+				req->content_type);
+	if (req->content_length >= 0)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
+				(intmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(STDOUT_FILENO);
+	close(STDIN_FILENO);
+	return !!res;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	if (is_git_request(req))
+		return do__git(req);
+
 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
 }
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..ce1abffa6aa
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,107 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
+REPO_DIR="$TRASH_DIRECTORY"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
+per_test_cleanup () {
+	stop_http_server &&
+	rm -f OUT.*
+}
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+
+	start_http_server &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
+test_done
-- 
gitgitgadget

[PATCH v6 08/12] test-http-server: add simple authentication

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
a configuration file (in the normal gitconfig file format).
Incoming requests are compared against the set of valid schemes and
tokens and only approved if a matching token is found, or if no auth
was provided and anonymous auth is enabled.

Configuration for auth includes a simple set of three options:

[auth]
	challenge = <scheme>[:<challenge_params>]
	token = <scheme>:[<token>]*
	allowAnonymous = <bool>

`auth.challenge` allows you define what authentication schemes, and
optional challenge parameters the server should use. Scheme names are
unique and subsequently specified challenge parameters in the config
file will replace previously specified ones.

`auth.token` allows you to define the set of value token values for an
authentication scheme. This is a multi-var and each entry in the
config file will append to the set of valid tokens for that scheme.
Specifying an empty token value will clear the list of tokens so far for
that scheme, i.e. `token = <scheme>:`.

`auth.allowAnonymous` controls whether or not unauthenticated requests
(those without any `Authorization` headers) should succeed or not, and
trigger a 401 Unauthorized response.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 233 +++++++++++++++++++++++++++++++++++-
 t/t5556-http-auth.sh        |  43 ++++++-
 2 files changed, 273 insertions(+), 3 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index ae17c738259..691fbfb51d6 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -7,6 +7,7 @@
 #include "version.h"
 #include "dir.h"
 #include "date.h"
+#include "config.h"
 
 #define TR2_CAT "test-http-server"
 
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =
 "           [--timeout=<n>] [--max-connections=<n>]\n"
 "           [--reuseaddr] [--pid-file=<file>]\n"
 "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--auth-config=<file>]\n"
 ;
 
 static unsigned int timeout;
@@ -317,7 +319,7 @@ static int is_git_request(struct req *req)
 		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
 }
 
-static enum worker_result do__git(struct req *req)
+static enum worker_result do__git(struct req *req, const char *user)
 {
 	const char *ok = "HTTP/1.1 200 OK\r\n";
 	struct child_process cp = CHILD_PROCESS_INIT;
@@ -334,10 +336,16 @@ static enum worker_result do__git(struct req *req)
 	 * exit status of the process, then write the HTTP status line followed
 	 * by the http-backend output. This is outside of the scope of this test
 	 * helper's use at time of writing.
+	 *
+	 * The important auth responses (401) we are handling prior to getting
+	 * to this point.
 	 */
 	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
 		return error(_("could not send '%s'"), ok);
 
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
 	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
 			req->uri_path.buf);
@@ -359,10 +367,218 @@ static enum worker_result do__git(struct req *req)
 	return !!res;
 }
 
+enum auth_result {
+	/* No auth module matches the request. */
+	AUTH_UNKNOWN = 0,
+
+	/* Auth module denied the request. */
+	AUTH_DENY = 1,
+
+	/* Auth module successfully validated the request. */
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	char *scheme;
+	char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(const char *scheme, int create)
+{
+	int i;
+	struct auth_module *mod;
+	for (i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme))
+			return mod;
+	}
+
+	if (create) {
+		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+		mod->scheme = xstrdup(scheme);
+		mod->challenge_params = NULL;
+		CALLOC_ARRAY(mod->tokens, 1);
+		string_list_init_dup(mod->tokens);
+
+		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+		auth_modules[auth_modules_nr++] = mod;
+
+		return mod;
+	}
+
+	return NULL;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/*
+	 * Check all auth modules and try to validate the request.
+	 * The first Authorization header that matches a known auth module
+	 * scheme will be consulted to either approve or deny the request.
+	 * If no module is found, or if there is no valid token, then 401 error.
+	 * Otherwise, only permit the request if anonymous auth is enabled.
+	 * It's atypical for user agents/clients to send multiple Authorization
+	 * headers, but not explicitly forbidden or defined.
+	 */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (!split[0] || !split[1]) continue;
+
+			/* trim trailing space ' ' */
+			strbuf_setlen(split[0], split[0]->len - 1);
+
+			mod = get_auth_module(split[0]->buf, 0);
+			if (mod) {
+				result = AUTH_DENY;
+
+				for_each_string_list_item(token, mod->tokens) {
+					if (!strcmp(split[1]->buf, token->string)) {
+						result = AUTH_ALLOW;
+						break;
+					}
+				}
+
+				goto done;
+			}
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (result != AUTH_DENY && allow_anonymous)
+			break;
+
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+
+		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
+				      &hdrs, *wr);
+	}
+
+	strbuf_list_free(split);
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
+static int split_auth_param(const char *str, char **scheme, char **val)
+{
+	struct strbuf **p = strbuf_split_str(str, ':', 2);
+
+	if (!p[0])
+		return -1;
+
+	/* trim trailing ':' */
+	if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
+		strbuf_setlen(p[0], p[0]->len - 1);
+
+	*scheme = strbuf_detach(p[0], NULL);
+
+	if (p[1])
+		*val = strbuf_detach(p[1], NULL);
+
+	strbuf_list_free(p);
+	return 0;
+}
+
+static int read_auth_config(const char *name, const char *val, void *data)
+{
+	int ret = 0;
+	char *scheme = NULL;
+	char *token = NULL;
+	char *challenge = NULL;
+	struct auth_module *mod = NULL;
+
+	if (!strcmp(name, "auth.challenge")) {
+		if (split_auth_param(val, &scheme, &challenge)) {
+			ret = error("invalid auth challenge '%s'", val);
+			goto cleanup;
+		}
+
+		mod = get_auth_module(scheme, 1);
+
+		/* Replace any existing challenge parameters */
+		free(mod->challenge_params);
+		mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
+	} else if (!strcmp(name, "auth.token")) {
+		if (split_auth_param(val, &scheme, &token)) {
+			ret = error("invalid auth token '%s'", val);
+			goto cleanup;
+		}
+
+		mod = get_auth_module(scheme, 1);
+
+		/*
+		 * Append to set of valid tokens unless an empty token value
+		 * is provided, then clear the existing list.
+		 */
+		if (token)
+			string_list_append(mod->tokens, token);
+		else
+			string_list_clear(mod->tokens, 1);
+	} else if (!strcmp(name, "auth.allowanonymous")) {
+		allow_anonymous = git_config_bool(name, val);
+	} else {
+		warning("unknown auth config '%s'", name);
+	}
+
+cleanup:
+	free(scheme);
+	free(token);
+	free(challenge);
+
+	return ret;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
 	if (is_git_request(req))
-		return do__git(req);
+		return do__git(req, user);
 
 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
 			       WR_OK | WR_HANGUP);
@@ -621,6 +837,19 @@ int cmd_main(int argc, const char **argv)
 			pid_file = v;
 			continue;
 		}
+		if (skip_prefix(arg, "--auth-config=", &v)) {
+			if (!strlen(v)) {
+				error("invalid argument - missing file path");
+				usage(test_http_auth_usage);
+			}
+
+			if (git_config_from_file(read_auth_config, v, NULL)) {
+				error("failed to read auth config file '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			continue;
+		}
 
 		fprintf(stderr, "error: unknown argument '%s'\n", arg);
 		usage(test_http_auth_usage);
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index ce1abffa6aa..cb5562a41bf 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -95,10 +95,51 @@ per_test_cleanup () {
 	rm -f OUT.*
 }
 
+test_expect_success CURL 'http auth server auth config' '
+	#test_when_finished "per_test_cleanup" &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = no-params
+		challenge = with-params:foo=\"bar\" p=1
+		challenge = with-params:foo=\"replaced\" q=1
+
+		token = no-explicit-challenge:valid-token
+		token = no-explicit-challenge:also-valid
+		token = reset-tokens:these-tokens
+		token = reset-tokens:will-be-reset
+		token = reset-tokens:
+		token = reset-tokens:the-only-valid-one
+
+		allowAnonymous = false
+	EOF
+
+	cat >OUT.expected <<-EOF &&
+	WWW-Authenticate: no-params
+	WWW-Authenticate: with-params foo="replaced" q=1
+	WWW-Authenticate: no-explicit-challenge
+	WWW-Authenticate: reset-tokens
+
+	Error: 401 Unauthorized
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	curl --include $ORIGIN_URL >OUT.curl &&
+	tr -d "\r" <OUT.curl | sed -n "/WWW-Authenticate/,\$p" >OUT.actual &&
+
+	test_cmp OUT.expected OUT.actual
+'
+
 test_expect_success 'http auth anonymous no challenge' '
 	test_when_finished "per_test_cleanup" &&
 
-	start_http_server &&
+	cat >auth.config <<-EOF &&
+	[auth]
+		allowAnonymous = true
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
 
 	# Attempt to read from a protected repository
 	git ls-remote $ORIGIN_URL
-- 
gitgitgadget

Re: [PATCH v6 08/12] test-http-server: add simple authentication

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:


> +static struct auth_module *get_auth_module(const char *scheme, int create)
> +{
> +	int i;
> +	struct auth_module *mod;
> +	for (i = 0; i < auth_modules_nr; i++) {

We can use "for (size_t i = 0" syntax now, let's do that here to not mix
"size_t" and "int" types needlessly.

> +	if (create) {
> +		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
> +		mod->scheme = xstrdup(scheme);
> +		mod->challenge_params = NULL;
> +		CALLOC_ARRAY(mod->tokens, 1);
> +		string_list_init_dup(mod->tokens);

Don't use CALLOC_ARRAY() if you're then going to use
string_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We
don't need to set the memory to 0, only to overwrite it entirely again.

> +		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
> +		auth_modules[auth_modules_nr++] = mod;

I have not looked at the whole context here, but instead of:

	struct auth_module {
		char *scheme;
		char *challenge_params;
		struct string_list *tokens;
	};

Why not:

	struct auth_module {
		char *challenge_params;
		struct string_list *tokens;
	};

Then you could use a "struct string_list" for this, make the "scheme" be
the "string" member, and stick the remaining two fields in the "util",
and thus save yourself the manual memory management etc.

> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
> +{
> +	enum auth_result result = AUTH_UNKNOWN;
> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
> +	struct auth_module *mod;
> +
> +	struct string_list_item *hdr;
> +	struct string_list_item *token;
> +	const char *v;
> +	struct strbuf **split = NULL;
> +	int i;
> +	char *challenge;
> +
> +	/*
> +	 * Check all auth modules and try to validate the request.
> +	 * The first Authorization header that matches a known auth module
> +	 * scheme will be consulted to either approve or deny the request.
> +	 * If no module is found, or if there is no valid token, then 401 error.
> +	 * Otherwise, only permit the request if anonymous auth is enabled.
> +	 * It's atypical for user agents/clients to send multiple Authorization
> +	 * headers, but not explicitly forbidden or defined.
> +	 */
> +	for_each_string_list_item(hdr, &req->header_list) {
> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
> +			split = strbuf_split_str(v, ' ', 2);
> +			if (!split[0] || !split[1]) continue;
> +
> +			/* trim trailing space ' ' */
> +			strbuf_setlen(split[0], split[0]->len - 1);
> +
> +			mod = get_auth_module(split[0]->buf, 0);
> +			if (mod) {
> +				result = AUTH_DENY;
> +
> +				for_each_string_list_item(token, mod->tokens) {
> +					if (!strcmp(split[1]->buf, token->string)) {
> +						result = AUTH_ALLOW;
> +						break;
> +					}
> +				}
> +
> +				goto done;

Sometimes we need a strbuf_split_str, but in this case couldn't you use
the in-place "struct string_list" variant of that instead, and just
carry a "size_t len" here for it, which you'd then pass to
get_auth_module() (which this commit adds)?

Also, you "split" in the loop, but...

> +	strbuf_list_free(split);
...only free() the last one here, isn't this leaking?

> +static int split_auth_param(const char *str, char **scheme, char **val)
> +{
> +	struct strbuf **p = strbuf_split_str(str, ':', 2);
> +
> +	if (!p[0])
> +		return -1;
> +
> +	/* trim trailing ':' */
> +	if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')

Don't compare unsigned length fields to "> 0", just do "if (len &&
....)".

Also, maybe I'm just groggy today, but how do we have a trailing ":" if
we just split on ":", and with a limit such that...

> +	if (p[1])
> +		*val = strbuf_detach(p[1], NULL);

...we have an item after that?


> +static int read_auth_config(const char *name, const char *val, void *data)
> +{
> +	int ret = 0;
> +	char *scheme = NULL;

Don't init this to NULL, instead the split_auth_param() return value
should be trusted, the compiler will then help us catch errors, no?

> +	char *token = NULL;
> +	char *challenge = NULL;

In this case it *is* needed though, as the function will return
non-errors, but *maybe* give us the second out parameter.

For such a function though, isn't just assigning "*second_param = NULL"
at the start of it less of a "running with scissors" pattern?

> +	struct auth_module *mod = NULL;

This NULL assignment can be dropped, we assign to it below
unconditionally before using it.

> +
> +	if (!strcmp(name, "auth.challenge")) {
> +		if (split_auth_param(val, &scheme, &challenge)) {
> +			ret = error("invalid auth challenge '%s'", val);
> +			goto cleanup;
> +		}
> +
> +		mod = get_auth_module(scheme, 1);
> +
> +		/* Replace any existing challenge parameters */
> +		free(mod->challenge_params);
> +		mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
> +	} else if (!strcmp(name, "auth.token")) {
> +		if (split_auth_param(val, &scheme, &token)) {
> +			ret = error("invalid auth token '%s'", val);
> +			goto cleanup;
> +		}
> +
> +		mod = get_auth_module(scheme, 1);
> +
> +		/*
> +		 * Append to set of valid tokens unless an empty token value
> +		 * is provided, then clear the existing list.
> +		 */
> +		if (token)
> +			string_list_append(mod->tokens, token);
> +		else
> +			string_list_clear(mod->tokens, 1);
> +	} else if (!strcmp(name, "auth.allowanonymous")) {
> +		allow_anonymous = git_config_bool(name, val);
> +	} else {
> +		warning("unknown auth config '%s'", name);
> +	}
> +
> +cleanup:
> +	free(scheme);
> +	free(token);
> +	free(challenge);
> +
> +	return ret;
> +}
> +

Re: [PATCH v6 08/12] test-http-server: add simple authentication

[Matthew John Cheetham]

On 2023-01-18 03:21, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
> 
>> +static struct auth_module *get_auth_module(const char *scheme, int create)
>> +{
>> +	int i;
>> +	struct auth_module *mod;
>> +	for (i = 0; i < auth_modules_nr; i++) {
> 
> We can use "for (size_t i = 0" syntax now, let's do that here to not mix
> "size_t" and "int" types needlessly.

Yep!

>> +	if (create) {
>> +		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
>> +		mod->scheme = xstrdup(scheme);
>> +		mod->challenge_params = NULL;
>> +		CALLOC_ARRAY(mod->tokens, 1);
>> +		string_list_init_dup(mod->tokens);
> 
> Don't use CALLOC_ARRAY() if you're then going to use
> string_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We
> don't need to set the memory to 0, only to overwrite it entirely again.

Sure.

>> +		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
>> +		auth_modules[auth_modules_nr++] = mod;
> 
> I have not looked at the whole context here, but instead of:
> 
> 	struct auth_module {
> 		char *scheme;
> 		char *challenge_params;
> 		struct string_list *tokens;
> 	};
> 
> Why not:
> 
> 	struct auth_module {
> 		char *challenge_params;
> 		struct string_list *tokens;
> 	};
> 
> Then you could use a "struct string_list" for this, make the "scheme" be
> the "string" member, and stick the remaining two fields in the "util",
> and thus save yourself the manual memory management etc.

I looked at this, but this then means being more careful when looping over
different `struct auth_module`s to keep the current 'scheme' and `*mod` in
sync/together. Just feels like overkill right now.

>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
>> +{
>> +	enum auth_result result = AUTH_UNKNOWN;
>> +	struct string_list hdrs = STRING_LIST_INIT_NODUP;
>> +	struct auth_module *mod;
>> +
>> +	struct string_list_item *hdr;
>> +	struct string_list_item *token;
>> +	const char *v;
>> +	struct strbuf **split = NULL;
>> +	int i;
>> +	char *challenge;
>> +
>> +	/*
>> +	 * Check all auth modules and try to validate the request.
>> +	 * The first Authorization header that matches a known auth module
>> +	 * scheme will be consulted to either approve or deny the request.
>> +	 * If no module is found, or if there is no valid token, then 401 error.
>> +	 * Otherwise, only permit the request if anonymous auth is enabled.
>> +	 * It's atypical for user agents/clients to send multiple Authorization
>> +	 * headers, but not explicitly forbidden or defined.
>> +	 */
>> +	for_each_string_list_item(hdr, &req->header_list) {
>> +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
>> +			split = strbuf_split_str(v, ' ', 2);
>> +			if (!split[0] || !split[1]) continue;
>> +
>> +			/* trim trailing space ' ' */
>> +			strbuf_setlen(split[0], split[0]->len - 1);
>> +
>> +			mod = get_auth_module(split[0]->buf, 0);
>> +			if (mod) {
>> +				result = AUTH_DENY;
>> +
>> +				for_each_string_list_item(token, mod->tokens) {
>> +					if (!strcmp(split[1]->buf, token->string)) {
>> +						result = AUTH_ALLOW;
>> +						break;
>> +					}
>> +				}
>> +
>> +				goto done;
> 
> Sometimes we need a strbuf_split_str, but in this case couldn't you use
> the in-place "struct string_list" variant of that instead, and just
> carry a "size_t len" here for it, which you'd then pass to
> get_auth_module() (which this commit adds)?

`get_auth_module` taking a scheme name as parameter is a more sensible, IMO,
than a `string_list` or `string_list_item` and an offset. Given this is a test
helper, performance also isn't a priority. Readability wins here I think.

> Also, you "split" in the loop, but...
> 
>> +	strbuf_list_free(split);
> ...only free() the last one here, isn't this leaking?

Yes, it is. Will fix in next iteration.

>> +static int split_auth_param(const char *str, char **scheme, char **val)
>> +{
>> +	struct strbuf **p = strbuf_split_str(str, ':', 2);
>> +
>> +	if (!p[0])
>> +		return -1;
>> +
>> +	/* trim trailing ':' */
>> +	if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
> 
> Don't compare unsigned length fields to "> 0", just do "if (len &&
> ....)".

Sure!

> Also, maybe I'm just groggy today, but how do we have a trailing ":" if
> we just split on ":", and with a limit such that...
> 
>> +	if (p[1])
>> +		*val = strbuf_detach(p[1], NULL);
> 
> ...we have an item after that?

Because that's how the `strbuf_split_str` function works. The comments
in the header file even call that out. "The substrings include the
terminator". From strbuf.h:

/**
 * Split str (of length slen) at the specified terminator character.
 * Return a null-terminated array of pointers to strbuf objects
 * holding the substrings.  The substrings include the terminator,
 * except for the last substring, which might be unterminated if the
 * original string did not end with a terminator. [cut] ...
   ...
 */
struct strbuf **strbuf_split_buf(const char *str, size_t len,
				 int terminator, int max);

>> +static int read_auth_config(const char *name, const char *val, void *data)
>> +{
>> +	int ret = 0;
>> +	char *scheme = NULL;
> 
> Don't init this to NULL, instead the split_auth_param() return value
> should be trusted, the compiler will then help us catch errors, no?
> 
>> +	char *token = NULL;
>> +	char *challenge = NULL;
> 
> In this case it *is* needed though, as the function will return
> non-errors, but *maybe* give us the second out parameter.
> 
> For such a function though, isn't just assigning "*second_param = NULL"
> at the start of it less of a "running with scissors" pattern?
> 
>> +	struct auth_module *mod = NULL;
> 
> This NULL assignment can be dropped, we assign to it below
> unconditionally before using it.

All of these variables need to be initialised to NULL because not all
arms of the `if-elseif` chain assign to all of these variables, but
we always `free` all of them at the function exit.

For example,

char *scheme = NULL;
char *token = NULL;
char *challenge = NULL;
...
} else if (!strcmp(name, "auth.allowanonymous")) {
	allow_anonymous = git_config_bool(name, val);
} else {
...
free(scheme);
free(token);
free(challenge);

>> +
>> +	if (!strcmp(name, "auth.challenge")) {
>> +		if (split_auth_param(val, &scheme, &challenge)) {
>> +			ret = error("invalid auth challenge '%s'", val);
>> +			goto cleanup;
>> +		}
>> +
>> +		mod = get_auth_module(scheme, 1);
>> +
>> +		/* Replace any existing challenge parameters */
>> +		free(mod->challenge_params);
>> +		mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
>> +	} else if (!strcmp(name, "auth.token")) {
>> +		if (split_auth_param(val, &scheme, &token)) {
>> +			ret = error("invalid auth token '%s'", val);
>> +			goto cleanup;
>> +		}
>> +
>> +		mod = get_auth_module(scheme, 1);
>> +
>> +		/*
>> +		 * Append to set of valid tokens unless an empty token value
>> +		 * is provided, then clear the existing list.
>> +		 */
>> +		if (token)
>> +			string_list_append(mod->tokens, token);
>> +		else
>> +			string_list_clear(mod->tokens, 1);
>> +	} else if (!strcmp(name, "auth.allowanonymous")) {
>> +		allow_anonymous = git_config_bool(name, val);
>> +	} else {
>> +		warning("unknown auth config '%s'", name);
>> +	}
>> +
>> +cleanup:
>> +	free(scheme);
>> +	free(token);
>> +	free(challenge);
>> +
>> +	return ret;
>> +}
>> +

Thanks,
Matthew

[PATCH v6 09/12] test-http-server: add sending of arbitrary headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the ability to send arbitrary headers in HTTP responses from the
test-http-server. This is useful when we want to test 'malformed'
response message handling.

Add the following option to the server auth config file:

[auth]
	extraHeader = [<value>]*

Each `auth.extraHeader` value will be appended to the response headers
verbatim.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 6 ++++++
 t/t5556-http-auth.sh        | 7 +++++++
 2 files changed, 13 insertions(+)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 691fbfb51d6..cbaee4fc0f4 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -388,6 +388,7 @@ static int allow_anonymous;
 static struct auth_module **auth_modules = NULL;
 static size_t auth_modules_nr = 0;
 static size_t auth_modules_alloc = 0;
+static struct strvec extra_headers = STRVEC_INIT;
 
 static struct auth_module *get_auth_module(const char *scheme, int create)
 {
@@ -489,6 +490,9 @@ done:
 			string_list_append(&hdrs, challenge);
 		}
 
+		for (i = 0; i < extra_headers.nr; i++)
+			string_list_append(&hdrs, extra_headers.v[i]);
+
 		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
 				      &hdrs, *wr);
 	}
@@ -557,6 +561,8 @@ static int read_auth_config(const char *name, const char *val, void *data)
 			string_list_clear(mod->tokens, 1);
 	} else if (!strcmp(name, "auth.allowanonymous")) {
 		allow_anonymous = git_config_bool(name, val);
+	} else if (!strcmp(name, "auth.extraheader")) {
+		strvec_push(&extra_headers, val);
 	} else {
 		warning("unknown auth config '%s'", name);
 	}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index cb5562a41bf..e36107ea95d 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -112,6 +112,10 @@ test_expect_success CURL 'http auth server auth config' '
 		token = reset-tokens:the-only-valid-one
 
 		allowAnonymous = false
+
+		extraHeader = X-Extra-Header: abc
+		extraHeader = X-Extra-Header: 123
+		extraHeader = X-Another: header\twith\twhitespace!
 	EOF
 
 	cat >OUT.expected <<-EOF &&
@@ -119,6 +123,9 @@ test_expect_success CURL 'http auth server auth config' '
 	WWW-Authenticate: with-params foo="replaced" q=1
 	WWW-Authenticate: no-explicit-challenge
 	WWW-Authenticate: reset-tokens
+	X-Extra-Header: abc
+	X-Extra-Header: 123
+	X-Another: header	with	whitespace!
 
 	Error: 401 Unauthorized
 	EOF
-- 
gitgitgadget

[PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Replace direct multiplication of two size_t parameters in curl response
stream handling callback functions with `st_mult` to guard against
overflows.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/http.c b/http.c
index 8a5ba3f4776..a2a80318bb2 100644
--- a/http.c
+++ b/http.c
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
 
 size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct buffer *buffer = buffer_;
 
 	if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
 
 size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct strbuf *buffer = buffer_;
 
 	strbuf_add(buffer, ptr, size);
-- 
gitgitgadget

Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Replace direct multiplication of two size_t parameters in curl response
> stream handling callback functions with `st_mult` to guard against
> overflows.
>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  http.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/http.c b/http.c
> index 8a5ba3f4776..a2a80318bb2 100644
> --- a/http.c
> +++ b/http.c
> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
>  
>  size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  {
> -	size_t size = eltsize * nmemb;
> +	size_t size = st_mult(eltsize, nmemb);
>  	struct buffer *buffer = buffer_;
>  
>  	if (size > buffer->buf.len - buffer->posn)
> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
>  
>  size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  {
> -	size_t size = eltsize * nmemb;
> +	size_t size = st_mult(eltsize, nmemb);
>  	struct strbuf *buffer = buffer_;
>  
>  	strbuf_add(buffer, ptr, size);

This is a really worthwhile fix, but shouldn't this be split into its
own stand-alone patch? It applies on "master", and seems like something
that's a good idea outside of this "test-http-server" topic.

Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult

[Victoria Dye]

Ævar Arnfjörð Bjarmason wrote:
> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Replace direct multiplication of two size_t parameters in curl response
>> stream handling callback functions with `st_mult` to guard against
>> overflows.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>>  http.c | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/http.c b/http.c
>> index 8a5ba3f4776..a2a80318bb2 100644
>> --- a/http.c
>> +++ b/http.c
>> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
>>  
>>  size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>  {
>> -	size_t size = eltsize * nmemb;
>> +	size_t size = st_mult(eltsize, nmemb);
>>  	struct buffer *buffer = buffer_;
>>  
>>  	if (size > buffer->buf.len - buffer->posn)
>> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
>>  
>>  size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>  {
>> -	size_t size = eltsize * nmemb;
>> +	size_t size = st_mult(eltsize, nmemb);
>>  	struct strbuf *buffer = buffer_;
>>  
>>  	strbuf_add(buffer, ptr, size);
> 
> This is a really worthwhile fix, but shouldn't this be split into its
> own stand-alone patch? It applies on "master", and seems like something
> that's a good idea outside of this "test-http-server" topic.

While it's this change *can* stand alone, please keep in mind that
suggestions like this (recommending a series be split and resubmitted) can
be highly disruptive to the in-flight topic and the original contributor.

Monitoring and iterating on multiple series at once is time-consuming for
the contributor and reviewers, and often (although not in this case) it
creates a dependency of one series on another, which comes with a cost to
the maintainer's time. Not to say those recommendations should never be made
(e.g. in a clearly too-long series early in its review cycle, or when
certain patches lead to excessive context switching while reviewing), just
that they should be made more carefully, with consideration for the time of
other contributors.

So, with that in mind, I don't think this patch is critical enough to
separate into an independent submission, and (subjectively) it does not
disrupt the flow of this series.

Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Victoria Dye wrote:

> Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>> 
>>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>>
>>> Replace direct multiplication of two size_t parameters in curl response
>>> stream handling callback functions with `st_mult` to guard against
>>> overflows.
>>>
>>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>>> ---
>>>  http.c | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/http.c b/http.c
>>> index 8a5ba3f4776..a2a80318bb2 100644
>>> --- a/http.c
>>> +++ b/http.c
>>> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
>>>  
>>>  size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>>  {
>>> -	size_t size = eltsize * nmemb;
>>> +	size_t size = st_mult(eltsize, nmemb);
>>>  	struct buffer *buffer = buffer_;
>>>  
>>>  	if (size > buffer->buf.len - buffer->posn)
>>> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
>>>  
>>>  size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>>  {
>>> -	size_t size = eltsize * nmemb;
>>> +	size_t size = st_mult(eltsize, nmemb);
>>>  	struct strbuf *buffer = buffer_;
>>>  
>>>  	strbuf_add(buffer, ptr, size);
>> 
>> This is a really worthwhile fix, but shouldn't this be split into its
>> own stand-alone patch? It applies on "master", and seems like something
>> that's a good idea outside of this "test-http-server" topic.
>
> While it's this change *can* stand alone, please keep in mind that
> suggestions like this (recommending a series be split and resubmitted) can
> be highly disruptive to the in-flight topic and the original contributor.
>
> Monitoring and iterating on multiple series at once is time-consuming for
> the contributor and reviewers, and often (although not in this case) it
> creates a dependency of one series on another, which comes with a cost to
> the maintainer's time. Not to say those recommendations should never be made
> (e.g. in a clearly too-long series early in its review cycle, or when
> certain patches lead to excessive context switching while reviewing), just
> that they should be made more carefully, with consideration for the time of
> other contributors.
>
> So, with that in mind, I don't think this patch is critical enough to
> separate into an independent submission, and (subjectively) it does not
> disrupt the flow of this series.

Yes, I take your general point, it's not always the right thing,
sometimes a while-at-it cleanup is better than a split-out etc.

In this case the split-out seemed like it wouldn't create a dependency
between topics, as the rest of the series didn't rely on the overflow
sanity check being added, it's just a good idea to do it in general.

[PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 +++++++++
 http.c       | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index a2a80318bb2..595c93bc7a3 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = st_mult(eltsize, nmemb);
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 * Continuation lines start with at least one whitespace, maybe more,
+	 * so we should collapse these down to a single SP (valid per the spec).
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		/* Trim leading whitespace from this continuation hdr line. */
+		strbuf_ltrim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (istarts_with(buf.buf, "http/"))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1956,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>

> +	strbuf_add(&buf, ptr, size);
> +
> +	/* Strip the CRLF that should be present at the end of each field */
> +	strbuf_trim_trailing_newline(&buf);
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
> +		while (isspace(*val))
> +			val++;
> +
> +		strvec_push(values, val);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> [...]
> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
> +		/* Trim leading whitespace from this continuation hdr line. */
> +		strbuf_ltrim(&buf);


The mixture of this isspace() loop and then strbuf_ltrim() seems odd,
why not stick with the strbuf API?

I.e. after skip_iprefix() strbuf_splice() the start of the string away,
then use strbuf_ltrim() in the first "if" branch here?

Likewise this is open-coding the "isspace" in strbuf_ltrim() for the
second "if". Maybe run the strbuf_ltrim() unconditionally, save away the
length before, and then:

	if (http_auth.header_is_last_match && prev_len != buf.len) { ...

?

Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2023-01-18 03:42, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
>> +	strbuf_add(&buf, ptr, size);
>> +
>> +	/* Strip the CRLF that should be present at the end of each field */
>> +	strbuf_trim_trailing_newline(&buf);
>> +
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> +		while (isspace(*val))
>> +			val++;
>> +
>> +		strvec_push(values, val);
>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
>> [...]
>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> +		/* Trim leading whitespace from this continuation hdr line. */
>> +		strbuf_ltrim(&buf);
> 
> 
> The mixture of this isspace() loop and then strbuf_ltrim() seems odd,
> why not stick with the strbuf API?
> 
> I.e. after skip_iprefix() strbuf_splice() the start of the string away,
> then use strbuf_ltrim() in the first "if" branch here?

You mean like this?

        size_t size = st_mult(eltsize, nmemb);
        struct strvec *values = &http_auth.wwwauth_headers;
        struct strbuf buf = STRBUF_INIT;
-       const char *val;
 
        /*
         * Header lines may not come NULL-terminated from libcurl so we must
@@ -216,11 +215,11 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
        strbuf_trim_trailing_newline(&buf);
 
        /* Start of a new WWW-Authenticate header */
-       if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
-               while (isspace(*val))
-                       val++;
+       if (istarts_with(buf.buf, "www-authenticate:")) {
+               strbuf_splice(&buf, 0, 17, NULL, 0);
+               strbuf_ltrim(&buf);
 
-               strvec_push(values, val);
+               strvec_push(values, buf.buf);
                http_auth.header_is_last_match = 1;
                goto exit;
        }


I don't particularly like this given we're now introducing the 'magic' number
17 that's the length of `www-authenticate:`, plus `strbuf_splice` is doing
a lot more work moving pieces of memory around rather than just producing
a new starting pointer to the start of the value (skipping leading whitespace).

> Likewise this is open-coding the "isspace" in strbuf_ltrim() for the
> second "if". Maybe run the strbuf_ltrim() unconditionally, save away the
> length before, and then:
> 
> 	if (http_auth.header_is_last_match && prev_len != buf.len) { ...
> 
> ?

The suggestion of trimming and comparing lengths just makes a piece of code
handling a little-known edge case less immediately obvious in its intent in
my opinion. The current implementation of "if starts with a single space"
matches the definition of continuation header lines, rather than re-piecing
together this from "why are we trimming and comparing lengths?"
Perf-wise the current implementation is only adding one extra `isspace`
call which we're potentially about to do in a loop inside of `strbuf_ltrim`
anyway. Plus, the common case will be a single space anyway.

Thanks,
Matthew

[PATCH v6 12/12] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |  11 ++
 t/lib-credential-helper.sh       |  27 ++++
 t/t5556-http-auth.sh             | 245 ++++++++++++++++++++++++++++++-
 4 files changed, 300 insertions(+), 2 deletions(-)
 create mode 100644 t/lib-credential-helper.sh

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..9f39ebc3c7e 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	char *full_key = xstrfmt("%s[]", key);
+	for (size_t i = 0; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free(full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh
new file mode 100644
index 00000000000..8b0e4414234
--- /dev/null
+++ b/t/lib-credential-helper.sh
@@ -0,0 +1,27 @@
+setup_credential_helper() {
+	test_expect_success 'setup credential helper' '
+		CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
+		export CREDENTIAL_HELPER &&
+		echo $CREDENTIAL_HELPER &&
+
+		write_script "$CREDENTIAL_HELPER" <<-\EOF
+		cmd=$1
+		teefile=$cmd-query.cred
+		catfile=$cmd-reply.cred
+		sed -n -e "/^$/q" -e "p" >> $teefile
+		if test "$cmd" = "get"; then
+			cat $catfile
+		fi
+		EOF
+	'
+}
+
+set_credential_reply() {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query() {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index e36107ea95d..79122c611a1 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop'
 
 TEST_NO_CREATE_REPO=1
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-credential-helper.sh
 
 test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
 
@@ -33,6 +34,8 @@ test_expect_success 'setup repos' '
 	git -C "$REPO_DIR" branch -M main
 '
 
+setup_credential_helper
+
 stop_http_server () {
 	if ! test -f "$PID_FILE"
 	then
@@ -92,7 +95,9 @@ start_http_server () {
 
 per_test_cleanup () {
 	stop_http_server &&
-	rm -f OUT.*
+	rm -f OUT.* &&
+	rm -f *.cred &&
+	rm -f auth.config
 }
 
 test_expect_success CURL 'http auth server auth config' '
@@ -152,4 +157,242 @@ test_expect_success 'http auth anonymous no challenge' '
 	git ls-remote $ORIGIN_URL
 '
 
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+		extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bEaRer auThoRiTy="id.example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+		extraheader = "WWW-Authenticate:"
+		extraheader = " "
+		extraheader = " bearer authority=\"id.example.com\""
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bearer authority="id.example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "foobar:alg=test widget=1"
+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	EOF
+
+	test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+'
+
 test_done
-- 
gitgitgadget

[PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server sharing code
with git-daemon, with simple authentication configurable by a config file.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.


Updates in v7
=============

 * Address several whitespace and arg/param list alignment issues.

 * Rethink the test-http-helper worker-mode error and result enum to be more
   simple and more informative to the nature of the error.

 * Use uintmax_t to store the Content-Length of a request in the helper
   test-http-server. Maintain a bit flag to store if we received such a
   header.

 * Return a "400 Bad Request" HTTP response if we fail to parse the request
   in the test-http-server.

 * Add test case to cover request message parsing in test-http-server.

 * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
   get_auth_module.

 * Correctly free the split strbufs created in the header parsing loop in
   test-http-server.

 * Avoid needless comparison > 0 for unsigned types.

 * Always set optional outputs to NULL if not present in test helper config
   value handling.

 * Remove an accidentally commented-out test cleanup line for one test case
   in t5556.

Matthew John Cheetham (12):
  daemon: libify socket setup and option functions
  daemon: libify child process handling functions
  daemon: rename some esoteric/laboured terminology
  test-http-server: add stub HTTP server test helper
  test-http-server: add HTTP error response function
  test-http-server: add HTTP request parsing
  test-http-server: pass Git requests to http-backend
  test-http-server: add simple authentication
  test-http-server: add sending of arbitrary headers
  http: replace unsafe size_t multiplication with st_mult
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt    |  19 +-
 Makefile                            |   2 +
 contrib/buildsystems/CMakeLists.txt |  11 +-
 credential.c                        |  12 +
 credential.h                        |  15 +
 daemon-utils.c                      | 286 +++++++++
 daemon-utils.h                      |  55 ++
 daemon.c                            | 306 +--------
 http.c                              |  98 ++-
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 943 ++++++++++++++++++++++++++++
 t/lib-credential-helper.sh          |  27 +
 t/t5556-http-auth.sh                | 463 ++++++++++++++
 13 files changed, 1936 insertions(+), 302 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h
 create mode 100644 t/helper/test-http-server.c
 create mode 100644 t/lib-credential-helper.sh
 create mode 100755 t/t5556-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v6:

  1:  74b0de14185 =  1:  74b0de14185 daemon: libify socket setup and option functions
  2:  b6ba344a671 =  2:  b6ba344a671 daemon: libify child process handling functions
  3:  9967401c972 =  3:  9967401c972 daemon: rename some esoteric/laboured terminology
  4:  d6e5e8825e8 !  4:  17c890ee108 test-http-server: add stub HTTP server test helper
     @@ t/helper/test-http-server.c (new)
      +
      +/*
      + * The code in this section is used by "worker" instances to service
     -+ * a single connection from a client.  The worker talks to the client
     -+ * on 0 and 1.
     ++ * a single connection from a client. The worker talks to the client
     ++ * on stdin and stdout.
      + */
      +
      +enum worker_result {
     @@ t/helper/test-http-server.c (new)
      +	 * Operation successful.
      +	 * Caller *might* keep the socket open and allow keep-alive.
      +	 */
     -+	WR_OK       = 0,
     ++	WR_OK = 0,
      +
      +	/*
     -+	 * Various errors while processing the request and/or the response.
     ++	 * Fatal error that is not recoverable.
      +	 * Close the socket and clean up.
      +	 * Exit child-process with non-zero status.
      +	 */
     -+	WR_IO_ERROR = 1<<0,
     -+
     -+	/*
     -+	 * Close the socket and clean up.  Does not imply an error.
     -+	 */
     -+	WR_HANGUP   = 1<<1,
     ++	WR_FATAL_ERROR = 1,
      +};
      +
      +static enum worker_result worker(void)
     @@ t/helper/test-http-server.c (new)
      +	while (1) {
      +		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
      +			logerror("unable to write response");
     -+			wr = WR_IO_ERROR;
     ++			wr = WR_FATAL_ERROR;
      +		}
      +
      +		if (wr != WR_OK)
     @@ t/helper/test-http-server.c (new)
      +	close(STDIN_FILENO);
      +	close(STDOUT_FILENO);
      +
     -+	return !!(wr & WR_IO_ERROR);
     ++	/* Only WR_OK should result in a non-zero exit code */
     ++	return wr != WR_OK;
      +}
      +
      +static int max_connections = 32;
  5:  79805f042b9 !  5:  6e70e304cfe test-http-server: add HTTP error response function
     @@ Commit message
      
       ## t/helper/test-http-server.c ##
      @@ t/helper/test-http-server.c: enum worker_result {
     - 	WR_HANGUP   = 1<<1,
     + 	 * Exit child-process with non-zero status.
     + 	 */
     + 	WR_FATAL_ERROR = 1,
     ++
     ++	/*
     ++	 * Close the socket and clean up. Does not imply an error.
     ++	 */
     ++	WR_HANGUP = 2,
       };
       
     -+static enum worker_result send_http_error(
     -+	int fd,
     -+	int http_code, const char *http_code_name,
     -+	int retry_after_seconds, struct string_list *response_headers,
     -+	enum worker_result wr_in)
     ++static enum worker_result send_http_error(int fd, int http_code,
     ++					  const char *http_code_name,
     ++					  int retry_after_seconds,
     ++					  struct string_list *response_headers,
     ++					  enum worker_result wr_in)
      +{
      +	struct strbuf response_header = STRBUF_INIT;
      +	struct strbuf response_content = STRBUF_INIT;
      +	struct string_list_item *h;
      +	enum worker_result wr;
      +
     -+	strbuf_addf(&response_content, "Error: %d %s\r\n",
     -+		    http_code, http_code_name);
     ++	strbuf_addf(&response_content, "Error: %d %s\r\n", http_code,
     ++		    http_code_name);
     ++
      +	if (retry_after_seconds > 0)
      +		strbuf_addf(&response_content, "Retry-After: %d\r\n",
      +			    retry_after_seconds);
      +
     -+	strbuf_addf  (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
     ++	strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code,
     ++		    http_code_name);
      +	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
     -+	strbuf_addstr(&response_header,	"Content-Type: text/plain\r\n");
     -+	strbuf_addf  (&response_header,	"Content-Length: %d\r\n", (int)response_content.len);
     ++	strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
     ++	strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n",
     ++		    (uintmax_t)response_content.len);
     ++
      +	if (retry_after_seconds > 0)
     -+		strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
     -+	strbuf_addf(  &response_header,	"Server: test-http-server/%s\r\n", git_version_string);
     -+	strbuf_addf(  &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
     ++		strbuf_addf(&response_header, "Retry-After: %d\r\n",
     ++			    retry_after_seconds);
     ++
     ++	strbuf_addf(&response_header, "Server: test-http-server/%s\r\n",
     ++		    git_version_string);
     ++	strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0,
     ++		    DATE_MODE(RFC2822)));
     ++
      +	if (response_headers)
      +		for_each_string_list_item(h, response_headers)
      +			strbuf_addf(&response_header, "%s\r\n", h->string);
     @@ t/helper/test-http-server.c: enum worker_result {
      +
      +	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
      +		logerror("unable to write response header");
     -+		wr = WR_IO_ERROR;
     ++		wr = WR_FATAL_ERROR;
      +		goto done;
      +	}
      +
      +	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
      +		logerror("unable to write response content body");
     -+		wr = WR_IO_ERROR;
     ++		wr = WR_FATAL_ERROR;
      +		goto done;
      +	}
      +
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
       	while (1) {
      -		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
      -			logerror("unable to write response");
     --			wr = WR_IO_ERROR;
     +-			wr = WR_FATAL_ERROR;
      -		}
      +		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
     -+				     NULL, WR_OK | WR_HANGUP);
     ++				     NULL, WR_HANGUP);
       
       		if (wr != WR_OK)
       			break;
     +@@ t/helper/test-http-server.c: static enum worker_result worker(void)
     + 	close(STDIN_FILENO);
     + 	close(STDOUT_FILENO);
     + 
     +-	/* Only WR_OK should result in a non-zero exit code */
     +-	return wr != WR_OK;
     ++	/* Only WR_OK and WR_HANGUP should result in a non-zero exit code */
     ++	return wr != WR_OK && wr != WR_HANGUP;
     + }
     + 
     + static int max_connections = 32;
  6:  252098db219 !  6:  43f1cdcbb82 test-http-server: add HTTP request parsing
     @@ Commit message
          test-http-server: add HTTP request parsing
      
          Add ability to parse HTTP requests to the test-http-server test helper.
     +    Introduce `struct req` to store request information including:
     +
     +     * HTTP method & version
     +     * Request path and query parameters
     +     * Headers
     +     * Content type and length (from `Content-Type` and `-Length` headers)
     +
     +    Failure to parse the request results in a 400 Bad Request response to
     +    the client. Note that we're not trying to support all possible requests
     +    here, but just enough to exercise all code under test.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## t/helper/test-http-server.c ##
      @@ t/helper/test-http-server.c: enum worker_result {
     - 	WR_HANGUP   = 1<<1,
     - };
     - 
     + 	 * Close the socket and clean up. Does not imply an error.
     + 	 */
     + 	WR_HANGUP = 2,
     ++
     ++	/*
     ++	 * Unexpected request message or error in request parsing.
     ++	 * Respond with an 400 error. Close the socket and cleanup.
     ++	 * Exit child-process with a non-zero status.
     ++	 */
     ++	WR_CLIENT_ERROR = 3,
     ++};
     ++
      +/*
      + * Fields from a parsed HTTP request.
      + */
     @@ t/helper/test-http-server.c: enum worker_result {
      +
      +	struct string_list header_list;
      +	const char *content_type;
     -+	ssize_t content_length;
     -+};
     -+
     ++	uintmax_t content_length;
     ++	unsigned has_content_length:1;
     + };
     + 
      +#define REQ__INIT { \
      +	.start_line = STRBUF_INIT, \
      +	.uri_path = STRBUF_INIT, \
      +	.query_args = STRBUF_INIT, \
      +	.header_list = STRING_LIST_INIT_NODUP, \
      +	.content_type = NULL, \
     -+	.content_length = -1 \
     -+	}
     ++	.content_length = 0, \
     ++	.has_content_length = 0, \
     ++}
      +
      +static void req__release(struct req *req)
      +{
     @@ t/helper/test-http-server.c: enum worker_result {
      +	string_list_clear(&req->header_list, 0);
      +}
      +
     - static enum worker_result send_http_error(
     - 	int fd,
     - 	int http_code, const char *http_code_name,
     + static enum worker_result send_http_error(int fd, int http_code,
     + 					  const char *http_code_name,
     + 					  int retry_after_seconds,
      @@ t/helper/test-http-server.c: done:
       	return wr;
       }
     @@ t/helper/test-http-server.c: done:
      +	 *
      +	 */
      +	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
     -+		result = WR_OK | WR_HANGUP;
     ++		result = WR_HANGUP;
      +		goto done;
      +	}
      +
     @@ t/helper/test-http-server.c: done:
      +	if (nr_start_line_fields != 3) {
      +		logerror("could not parse request start-line '%s'",
      +			 req->start_line.buf);
     -+		result = WR_IO_ERROR;
     ++		result = WR_CLIENT_ERROR;
      +		goto done;
      +	}
      +
     @@ t/helper/test-http-server.c: done:
      +	if (strcmp(req->http_version, "HTTP/1.1")) {
      +		logerror("unsupported version '%s' (expecting HTTP/1.1)",
      +			 req->http_version);
     -+		result = WR_IO_ERROR;
     ++		result = WR_CLIENT_ERROR;
      +		goto done;
      +	}
      +
     @@ t/helper/test-http-server.c: done:
      +		string_list_append(&req->header_list, hp);
      +
      +		/* also store common request headers as struct req members */
     -+		if (skip_prefix(hp, "Content-Type: ", &hv)) {
     ++		if (skip_iprefix(hp, "Content-Type: ", &hv)) {
      +			req->content_type = hv;
     -+		} else if (skip_prefix(hp, "Content-Length: ", &hv)) {
     -+			req->content_length = strtol(hv, &hp, 10);
     ++		} else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
     ++			/*
     ++			 * Content-Length is always non-negative, but has no
     ++			 * upper bound according to RFC 7230 (§3.3.2).
     ++			 */
     ++			intmax_t len = 0;
     ++			if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
     ++			    len == INTMAX_MAX) {
     ++				logerror("invalid content-length: '%s'", hv);
     ++				result = WR_CLIENT_ERROR;
     ++				goto done;
     ++			}
     ++
     ++			req->content_length = (uintmax_t)len;
     ++			req->has_content_length = 1;
      +		}
      +	}
      +
     @@ t/helper/test-http-server.c: done:
      +		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
      +		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
      +		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
     -+		if (req->content_length >= 0)
     -+			trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
     ++		if (req->has_content_length)
     ++			trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
     ++				      req->content_length);
      +		if (req->content_type)
      +			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
      +		for_each_string_list_item(item, &req->header_list)
     @@ t/helper/test-http-server.c: done:
      +static enum worker_result dispatch(struct req *req)
      +{
      +	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
     -+			       WR_OK | WR_HANGUP);
     ++			       WR_HANGUP);
      +}
      +
       static enum worker_result worker(void)
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
       
       	while (1) {
      -		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
     --				     NULL, WR_OK | WR_HANGUP);
     +-				     NULL, WR_HANGUP);
      +		req__release(&req);
      +
      +		alarm(timeout);
      +		wr = req__read(&req, 0);
      +		alarm(0);
      +
     ++		if (wr == WR_CLIENT_ERROR)
     ++			wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
     ++					     -1, NULL, wr);
     ++
      +		if (wr != WR_OK)
      +			break;
       
     @@ t/helper/test-http-server.c: static enum worker_result worker(void)
       		if (wr != WR_OK)
       			break;
       	}
     +
     + ## t/t5556-http-auth.sh (new) ##
     +@@
     ++#!/bin/sh
     ++
     ++test_description='test http auth header and credential helper interop'
     ++
     ++TEST_NO_CREATE_REPO=1
     ++. ./test-lib.sh
     ++
     ++# Setup a repository
     ++#
     ++REPO_DIR="$TRASH_DIRECTORY"/repo
     ++
     ++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     ++
     ++PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     ++
     ++test_expect_success 'setup repos' '
     ++	test_create_repo "$REPO_DIR" &&
     ++	git -C "$REPO_DIR" branch -M main
     ++'
     ++
     ++run_http_server_worker() {
     ++	(
     ++		cd "$REPO_DIR"
     ++		test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
     ++	)
     ++}
     ++
     ++per_test_cleanup () {
     ++	rm -f OUT.* &&
     ++	rm -f IN.* &&
     ++}
     ++
     ++test_expect_success 'http auth server request parsing' '
     ++	test_when_finished "per_test_cleanup" &&
     ++
     ++	cat >auth.config <<-EOF &&
     ++	[auth]
     ++		allowAnonymous = true
     ++	EOF
     ++
     ++	echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
     ++	echo "HTTP/1.1 200 OK" >OUT.http200 &&
     ++
     ++	cat >IN.http.valid <<-EOF &&
     ++	GET /info/refs HTTP/1.1
     ++	Content-Length: 0
     ++	EOF
     ++
     ++	cat >IN.http.badfirstline <<-EOF &&
     ++	/info/refs GET HTTP
     ++	EOF
     ++
     ++	cat >IN.http.badhttpver <<-EOF &&
     ++	GET /info/refs HTTP/999.9
     ++	EOF
     ++
     ++	cat >IN.http.ltzlen <<-EOF &&
     ++	GET /info/refs HTTP/1.1
     ++	Content-Length: -1
     ++	EOF
     ++
     ++	cat >IN.http.badlen <<-EOF &&
     ++	GET /info/refs HTTP/1.1
     ++	Content-Length: not-a-number
     ++	EOF
     ++
     ++	cat >IN.http.overlen <<-EOF &&
     ++	GET /info/refs HTTP/1.1
     ++	Content-Length: 9223372036854775807
     ++	EOF
     ++
     ++	run_http_server_worker \
     ++		--auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
     ++		| head -n1 >OUT.actual &&
     ++	test_cmp OUT.http200 OUT.actual &&
     ++
     ++	run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
     ++	test_cmp OUT.http400 OUT.actual &&
     ++
     ++	run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
     ++	test_cmp OUT.http400 OUT.actual &&
     ++
     ++	run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
     ++	test_cmp OUT.http400 OUT.actual &&
     ++
     ++	run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
     ++	test_cmp OUT.http400 OUT.actual
     ++'
     ++
     ++test_done
  7:  ab06ac9b965 !  7:  ca9c2787248 test-http-server: pass Git requests to http-backend
     @@ t/helper/test-http-server.c: done:
      +		return error(_("could not send '%s'"), ok);
      +
      +	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     -+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     -+			req->uri_path.buf);
     ++	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
      +	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
      +	if (req->query_args.len)
     -+		strvec_pushf(&cp.env, "QUERY_STRING=%s",
     -+				req->query_args.buf);
     ++		strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf);
      +	if (req->content_type)
     -+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
     -+				req->content_type);
     -+	if (req->content_length >= 0)
     -+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
     -+				(intmax_t)req->content_length);
     ++		strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type);
     ++	if (req->has_content_length)
     ++		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX,
     ++			(uintmax_t)req->content_length);
      +	cp.git_cmd = 1;
      +	strvec_push(&cp.args, "http-backend");
      +	res = run_command(&cp);
     @@ t/helper/test-http-server.c: done:
      +		return do__git(req);
      +
       	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
     - 			       WR_OK | WR_HANGUP);
     + 			       WR_HANGUP);
       }
      
     - ## t/t5556-http-auth.sh (new) ##
     -@@
     -+#!/bin/sh
     -+
     -+test_description='test http auth header and credential helper interop'
     -+
     -+TEST_NO_CREATE_REPO=1
     -+. ./test-lib.sh
     -+
     + ## t/t5556-http-auth.sh ##
     +@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
     + TEST_NO_CREATE_REPO=1
     + . ./test-lib.sh
     + 
      +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
      +
     -+# Setup a repository
     -+#
     -+REPO_DIR="$TRASH_DIRECTORY"/repo
     -+
     + # Setup a repository
     + #
     + REPO_DIR="$TRASH_DIRECTORY"/repo
     + 
      +# Setup some lookback URLs where test-http-server will be listening.
      +# We will spawn it directly inside the repo directory, so we avoid
      +# any need to configure directory mappings etc - we only serve this
     @@ t/t5556-http-auth.sh (new)
      +# killing it by PID).
      +#
      +PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
     -+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     -+
     -+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     -+
     -+test_expect_success 'setup repos' '
     -+	test_create_repo "$REPO_DIR" &&
     -+	git -C "$REPO_DIR" branch -M main
     -+'
     -+
     + SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     + 
     + PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     +@@ t/t5556-http-auth.sh: run_http_server_worker() {
     + 	)
     + }
     + 
      +stop_http_server () {
      +	if ! test -f "$PID_FILE"
      +	then
     @@ t/t5556-http-auth.sh (new)
      +	return 1
      +}
      +
     -+per_test_cleanup () {
     + per_test_cleanup () {
      +	stop_http_server &&
     -+	rm -f OUT.*
     -+}
     + 	rm -f OUT.* &&
     + 	rm -f IN.* &&
     + }
     +@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
     + 	test_cmp OUT.http400 OUT.actual
     + '
     + 
      +
      +test_expect_success 'http auth anonymous no challenge' '
      +	test_when_finished "per_test_cleanup" &&
     @@ t/t5556-http-auth.sh (new)
      +	git ls-remote $ORIGIN_URL
      +'
      +
     -+test_done
     + test_done
  8:  a1ff55dd6e2 !  8:  b8d3e81b553 test-http-server: add simple authentication
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
      +
       	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
     - 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
     - 			req->uri_path.buf);
     + 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
     + 	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
      @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
       	return !!res;
       }
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +
      +static struct auth_module *get_auth_module(const char *scheme, int create)
      +{
     -+	int i;
      +	struct auth_module *mod;
     -+	for (i = 0; i < auth_modules_nr; i++) {
     ++	for (size_t i = 0; i < auth_modules_nr; i++) {
      +		mod = auth_modules[i];
      +		if (!strcasecmp(mod->scheme, scheme))
      +			return mod;
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
      +		mod->scheme = xstrdup(scheme);
      +		mod->challenge_params = NULL;
     -+		CALLOC_ARRAY(mod->tokens, 1);
     ++		ALLOC_ARRAY(mod->tokens, 1);
      +		string_list_init_dup(mod->tokens);
      +
      +		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +	for_each_string_list_item(hdr, &req->header_list) {
      +		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
      +			split = strbuf_split_str(v, ' ', 2);
     -+			if (!split[0] || !split[1]) continue;
     -+
     -+			/* trim trailing space ' ' */
     -+			strbuf_setlen(split[0], split[0]->len - 1);
     -+
     -+			mod = get_auth_module(split[0]->buf, 0);
     -+			if (mod) {
     -+				result = AUTH_DENY;
     -+
     -+				for_each_string_list_item(token, mod->tokens) {
     -+					if (!strcmp(split[1]->buf, token->string)) {
     -+						result = AUTH_ALLOW;
     -+						break;
     ++			if (split[0] && split[1]) {
     ++				/* trim trailing space ' ' */
     ++				strbuf_rtrim(split[0]);
     ++
     ++				mod = get_auth_module(split[0]->buf, 0);
     ++				if (mod) {
     ++					result = AUTH_DENY;
     ++
     ++					for_each_string_list_item(token, mod->tokens) {
     ++						if (!strcmp(split[1]->buf, token->string)) {
     ++							result = AUTH_ALLOW;
     ++							break;
     ++						}
      +					}
     -+				}
      +
     -+				goto done;
     ++					strbuf_list_free(split);
     ++					goto done;
     ++				}
      +			}
     ++
     ++			strbuf_list_free(split);
      +		}
      +	}
      +
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +				      &hdrs, *wr);
      +	}
      +
     -+	strbuf_list_free(split);
      +	string_list_clear(&hdrs, 0);
      +
      +	return result == AUTH_ALLOW ||
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +		return -1;
      +
      +	/* trim trailing ':' */
     -+	if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
     ++	if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
      +		strbuf_setlen(p[0], p[0]->len - 1);
      +
      +	*scheme = strbuf_detach(p[0], NULL);
     -+
     -+	if (p[1])
     -+		*val = strbuf_detach(p[1], NULL);
     ++	*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
      +
      +	strbuf_list_free(p);
      +	return 0;
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +	char *scheme = NULL;
      +	char *token = NULL;
      +	char *challenge = NULL;
     -+	struct auth_module *mod = NULL;
     ++	struct auth_module *mod;
      +
      +	if (!strcmp(name, "auth.challenge")) {
      +		if (split_auth_param(val, &scheme, &challenge)) {
     @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
      +		return do__git(req, user);
       
       	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
     - 			       WR_OK | WR_HANGUP);
     + 			       WR_HANGUP);
      @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
       			pid_file = v;
       			continue;
     @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
      
       ## t/t5556-http-auth.sh ##
      @@ t/t5556-http-auth.sh: per_test_cleanup () {
     - 	rm -f OUT.*
     + 	stop_http_server &&
     + 	rm -f OUT.* &&
     + 	rm -f IN.* &&
     ++	rm -f auth.config
       }
       
     + test_expect_success 'http auth server request parsing' '
     +@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
     + 	test_cmp OUT.http400 OUT.actual
     + '
     + 
      +test_expect_success CURL 'http auth server auth config' '
     -+	#test_when_finished "per_test_cleanup" &&
     ++	test_when_finished "per_test_cleanup" &&
      +
      +	cat >auth.config <<-EOF &&
      +	[auth]
     @@ t/t5556-http-auth.sh: per_test_cleanup () {
      +
      +	test_cmp OUT.expected OUT.actual
      +'
     -+
     + 
       test_expect_success 'http auth anonymous no challenge' '
       	test_when_finished "per_test_cleanup" &&
       
  9:  76125cdf239 =  9:  2f97c94f679 test-http-server: add sending of arbitrary headers
 10:  cc9a220ed1f = 10:  4b1635b3f69 http: replace unsafe size_t multiplication with st_mult
 11:  bc1ac8d3eb3 = 11:  5f5e46038cf http: read HTTP WWW-Authenticate response headers
 12:  7c8229f0b11 ! 12:  09164f77d56 credential: add WWW-Authenticate header to cred requests
     @@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
       
      +setup_credential_helper
      +
     - stop_http_server () {
     - 	if ! test -f "$PID_FILE"
     - 	then
     -@@ t/t5556-http-auth.sh: start_http_server () {
     - 
     - per_test_cleanup () {
     + run_http_server_worker() {
     + 	(
     + 		cd "$REPO_DIR"
     +@@ t/t5556-http-auth.sh: per_test_cleanup () {
       	stop_http_server &&
     --	rm -f OUT.*
     -+	rm -f OUT.* &&
     + 	rm -f OUT.* &&
     + 	rm -f IN.* &&
      +	rm -f *.cred &&
     -+	rm -f auth.config
     + 	rm -f auth.config
       }
       
     - test_expect_success CURL 'http auth server auth config' '
      @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
       	git ls-remote $ORIGIN_URL
       '

-- 
gitgitgadget

[PATCH v7 01/12] daemon: libify socket setup and option functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions for setting up listening sockets and keep-alive options
from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
dependencies on global state by inlining the behaviour at the callsites
for all libified functions.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile       |   1 +
 daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h |  23 ++++++
 daemon.c       | 214 +------------------------------------------------
 4 files changed, 237 insertions(+), 210 deletions(-)
 create mode 100644 daemon-utils.c
 create mode 100644 daemon-utils.h

diff --git a/Makefile b/Makefile
index b258fdbed86..2654094dbb5 100644
--- a/Makefile
+++ b/Makefile
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o
 LIB_OBJS += csum-file.o
 LIB_OBJS += ctype.o
 LIB_OBJS += date.o
+LIB_OBJS += daemon-utils.o
 LIB_OBJS += decorate.o
 LIB_OBJS += delta-islands.o
 LIB_OBJS += diagnose.o
diff --git a/daemon-utils.c b/daemon-utils.c
new file mode 100644
index 00000000000..b96b55962db
--- /dev/null
+++ b/daemon-utils.c
@@ -0,0 +1,209 @@
+#include "cache.h"
+#include "daemon-utils.h"
+
+void set_keep_alive(int sockfd, log_fn logerror)
+{
+	int ka = 1;
+
+	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+		if (errno != ENOTSOCK)
+			logerror("unable to set SO_KEEPALIVE on socket: %s",
+				strerror(errno));
+	}
+}
+
+static int set_reuse_addr(int sockfd)
+{
+	int on = 1;
+
+	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+			  &on, sizeof(on));
+}
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+	static char ip[INET_ADDRSTRLEN];
+#else
+	static char ip[INET6_ADDRSTRLEN];
+#endif
+
+	switch (family) {
+#ifndef NO_IPV6
+	case AF_INET6:
+		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+		break;
+#endif
+	case AF_INET:
+		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+		break;
+	default:
+		xsnprintf(ip, sizeof(ip), "<unknown>");
+	}
+	return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	int socknum = 0;
+	char pbuf[NI_MAXSERV];
+	struct addrinfo hints, *ai0, *ai;
+	int gai;
+	long flags;
+
+	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+	memset(&hints, 0, sizeof(hints));
+	hints.ai_family = AF_UNSPEC;
+	hints.ai_socktype = SOCK_STREAM;
+	hints.ai_protocol = IPPROTO_TCP;
+	hints.ai_flags = AI_PASSIVE;
+
+	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+	if (gai) {
+		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+		return 0;
+	}
+
+	for (ai = ai0; ai; ai = ai->ai_next) {
+		int sockfd;
+
+		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+		if (sockfd < 0)
+			continue;
+		if (sockfd >= FD_SETSIZE) {
+			logerror("Socket descriptor too large");
+			close(sockfd);
+			continue;
+		}
+
+#ifdef IPV6_V6ONLY
+		if (ai->ai_family == AF_INET6) {
+			int on = 1;
+			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+				   &on, sizeof(on));
+			/* Note: error is not fatal */
+		}
+#endif
+
+		if (reuseaddr && set_reuse_addr(sockfd)) {
+			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+			close(sockfd);
+			continue;
+		}
+
+		set_keep_alive(sockfd, logerror);
+
+		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+			logerror("Could not bind to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+		if (listen(sockfd, 5) < 0) {
+			logerror("Could not listen to %s: %s",
+				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+				 strerror(errno));
+			close(sockfd);
+			continue;	/* not fatal */
+		}
+
+		flags = fcntl(sockfd, F_GETFD, 0);
+		if (flags >= 0)
+			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+		socklist->list[socklist->nr++] = sockfd;
+		socknum++;
+	}
+
+	freeaddrinfo(ai0);
+
+	return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+			    struct socketlist *socklist, int reuseaddr,
+			    log_fn logerror)
+{
+	struct sockaddr_in sin;
+	int sockfd;
+	long flags;
+
+	memset(&sin, 0, sizeof sin);
+	sin.sin_family = AF_INET;
+	sin.sin_port = htons(listen_port);
+
+	if (listen_addr) {
+		/* Well, host better be an IP address here. */
+		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+			return 0;
+	} else {
+		sin.sin_addr.s_addr = htonl(INADDR_ANY);
+	}
+
+	sockfd = socket(AF_INET, SOCK_STREAM, 0);
+	if (sockfd < 0)
+		return 0;
+
+	if (reuseaddr && set_reuse_addr(sockfd)) {
+		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	set_keep_alive(sockfd, logerror);
+
+	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
+		logerror("Could not bind to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	if (listen(sockfd, 5) < 0) {
+		logerror("Could not listen to %s: %s",
+			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+			 strerror(errno));
+		close(sockfd);
+		return 0;
+	}
+
+	flags = fcntl(sockfd, F_GETFD, 0);
+	if (flags >= 0)
+		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+	socklist->list[socklist->nr++] = sockfd;
+	return 1;
+}
+
+#endif
+
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror)
+{
+	if (!listen_addr->nr)
+		setup_named_sock(NULL, listen_port, socklist, reuseaddr,
+				 logerror);
+	else {
+		int i, socknum;
+		for (i = 0; i < listen_addr->nr; i++) {
+			socknum = setup_named_sock(listen_addr->items[i].string,
+						   listen_port, socklist, reuseaddr,
+						   logerror);
+
+			if (socknum == 0)
+				logerror("unable to allocate any listen sockets for host %s on port %u",
+					 listen_addr->items[i].string, listen_port);
+		}
+	}
+}
diff --git a/daemon-utils.h b/daemon-utils.h
new file mode 100644
index 00000000000..6710a2a6dc0
--- /dev/null
+++ b/daemon-utils.h
@@ -0,0 +1,23 @@
+#ifndef DAEMON_UTILS_H
+#define DAEMON_UTILS_H
+
+#include "git-compat-util.h"
+#include "string-list.h"
+
+typedef void (*log_fn)(const char *msg, ...);
+
+struct socketlist {
+	int *list;
+	size_t nr;
+	size_t alloc;
+};
+
+/* Enable sending of keep-alive messages on the socket. */
+void set_keep_alive(int sockfd, log_fn logerror);
+
+/* Setup a number of sockets to listen on the provided addresses. */
+void socksetup(struct string_list *listen_addr, int listen_port,
+	       struct socketlist *socklist, int reuseaddr,
+	       log_fn logerror);
+
+#endif
diff --git a/daemon.c b/daemon.c
index 0ae7d12b5c1..1ed4e705680 100644
--- a/daemon.c
+++ b/daemon.c
@@ -1,9 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "daemon-utils.h"
 #include "pkt-line.h"
 #include "run-command.h"
 #include "strbuf.h"
-#include "string-list.h"
 
 #ifdef NO_INITGROUPS
 #define initgroups(x, y) (0) /* nothing */
@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)
 	strbuf_release(&hi->tcp_port);
 }
 
-static void set_keep_alive(int sockfd)
-{
-	int ka = 1;
-
-	if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
-		if (errno != ENOTSOCK)
-			logerror("unable to set SO_KEEPALIVE on socket: %s",
-				strerror(errno));
-	}
-}
-
 static int execute(void)
 {
 	char *line = packet_buffer;
@@ -759,7 +748,7 @@ static int execute(void)
 	if (addr)
 		loginfo("Connection from %s:%s", addr, port);
 
-	set_keep_alive(0);
+	set_keep_alive(0, logerror);
 	alarm(init_timeout ? init_timeout : timeout);
 	pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
 	alarm(0);
@@ -938,202 +927,6 @@ static void child_handler(int signo)
 	signal(SIGCHLD, child_handler);
 }
 
-static int set_reuse_addr(int sockfd)
-{
-	int on = 1;
-
-	if (!reuseaddr)
-		return 0;
-	return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
-			  &on, sizeof(on));
-}
-
-struct socketlist {
-	int *list;
-	size_t nr;
-	size_t alloc;
-};
-
-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
-{
-#ifdef NO_IPV6
-	static char ip[INET_ADDRSTRLEN];
-#else
-	static char ip[INET6_ADDRSTRLEN];
-#endif
-
-	switch (family) {
-#ifndef NO_IPV6
-	case AF_INET6:
-		inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
-		break;
-#endif
-	case AF_INET:
-		inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
-		break;
-	default:
-		xsnprintf(ip, sizeof(ip), "<unknown>");
-	}
-	return ip;
-}
-
-#ifndef NO_IPV6
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	int socknum = 0;
-	char pbuf[NI_MAXSERV];
-	struct addrinfo hints, *ai0, *ai;
-	int gai;
-	long flags;
-
-	xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
-	memset(&hints, 0, sizeof(hints));
-	hints.ai_family = AF_UNSPEC;
-	hints.ai_socktype = SOCK_STREAM;
-	hints.ai_protocol = IPPROTO_TCP;
-	hints.ai_flags = AI_PASSIVE;
-
-	gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
-	if (gai) {
-		logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
-		return 0;
-	}
-
-	for (ai = ai0; ai; ai = ai->ai_next) {
-		int sockfd;
-
-		sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
-		if (sockfd < 0)
-			continue;
-		if (sockfd >= FD_SETSIZE) {
-			logerror("Socket descriptor too large");
-			close(sockfd);
-			continue;
-		}
-
-#ifdef IPV6_V6ONLY
-		if (ai->ai_family == AF_INET6) {
-			int on = 1;
-			setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
-				   &on, sizeof(on));
-			/* Note: error is not fatal */
-		}
-#endif
-
-		if (set_reuse_addr(sockfd)) {
-			logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-			close(sockfd);
-			continue;
-		}
-
-		set_keep_alive(sockfd);
-
-		if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
-			logerror("Could not bind to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-		if (listen(sockfd, 5) < 0) {
-			logerror("Could not listen to %s: %s",
-				 ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-				 strerror(errno));
-			close(sockfd);
-			continue;	/* not fatal */
-		}
-
-		flags = fcntl(sockfd, F_GETFD, 0);
-		if (flags >= 0)
-			fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-		ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-		socklist->list[socklist->nr++] = sockfd;
-		socknum++;
-	}
-
-	freeaddrinfo(ai0);
-
-	return socknum;
-}
-
-#else /* NO_IPV6 */
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	struct sockaddr_in sin;
-	int sockfd;
-	long flags;
-
-	memset(&sin, 0, sizeof sin);
-	sin.sin_family = AF_INET;
-	sin.sin_port = htons(listen_port);
-
-	if (listen_addr) {
-		/* Well, host better be an IP address here. */
-		if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
-			return 0;
-	} else {
-		sin.sin_addr.s_addr = htonl(INADDR_ANY);
-	}
-
-	sockfd = socket(AF_INET, SOCK_STREAM, 0);
-	if (sockfd < 0)
-		return 0;
-
-	if (set_reuse_addr(sockfd)) {
-		logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	set_keep_alive(sockfd);
-
-	if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
-		logerror("Could not bind to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	if (listen(sockfd, 5) < 0) {
-		logerror("Could not listen to %s: %s",
-			 ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-			 strerror(errno));
-		close(sockfd);
-		return 0;
-	}
-
-	flags = fcntl(sockfd, F_GETFD, 0);
-	if (flags >= 0)
-		fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
-	ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-	socklist->list[socklist->nr++] = sockfd;
-	return 1;
-}
-
-#endif
-
-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
-{
-	if (!listen_addr->nr)
-		setup_named_sock(NULL, listen_port, socklist);
-	else {
-		int i, socknum;
-		for (i = 0; i < listen_addr->nr; i++) {
-			socknum = setup_named_sock(listen_addr->items[i].string,
-						   listen_port, socklist);
-
-			if (socknum == 0)
-				logerror("unable to allocate any listen sockets for host %s on port %u",
-					 listen_addr->items[i].string, listen_port);
-		}
-	}
-}
-
 static int service_loop(struct socketlist *socklist)
 {
 	struct pollfd *pfd;
@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,
 {
 	struct socketlist socklist = { NULL, 0, 0 };
 
-	socksetup(listen_addr, listen_port, &socklist);
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr,
+		  logerror);
 	if (socklist.nr == 0)
 		die("unable to allocate any listen sockets on port %u",
 		    listen_port);
-- 
gitgitgadget

[PATCH v7 02/12] daemon: libify child process handling functions

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Extract functions and structures for managing child processes started
from the parent daemon-like process from `daemon.c` to the new shared
`daemon-utils.{c,h}` files.

One minor functional change is introduced to `check_dead_children()`
where the logging of a dead/disconnected child is now optional. With the
'libification' of these functions we extract the call to `loginfo` to a
call to a function pointer, and guard the log message creation and
logging behind a `NULL` check. Callers can now skip logging by passing
`NULL` as the `log_fn loginfo` argument.
The behaviour of callers in `daemon.c` remains the same (save one extra
NULL check)  however as a pointer to `loginfo` is always passed.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++
 daemon-utils.h | 32 ++++++++++++++++++
 daemon.c       | 92 +++-----------------------------------------------
 3 files changed, 114 insertions(+), 87 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index b96b55962db..8506664b440 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 		}
 	}
 }
+
+static int addrcmp(const struct sockaddr_storage *s1,
+    const struct sockaddr_storage *s2)
+{
+	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+	if (sa1->sa_family != sa2->sa_family)
+		return sa1->sa_family - sa2->sa_family;
+	if (sa1->sa_family == AF_INET)
+		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+		    &((struct sockaddr_in *)s2)->sin_addr,
+		    sizeof(struct in_addr));
+#ifndef NO_IPV6
+	if (sa1->sa_family == AF_INET6)
+		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+		    &((struct sockaddr_in6 *)s2)->sin6_addr,
+		    sizeof(struct in6_addr));
+#endif
+	return 0;
+}
+
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn , unsigned int *live_children)
+{
+	struct child *newborn, **cradle;
+
+	CALLOC_ARRAY(newborn, 1);
+	(*live_children)++;
+	memcpy(&newborn->cld, cld, sizeof(*cld));
+	memcpy(&newborn->address, addr, addrlen);
+	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+		if (!addrcmp(&(*cradle)->address, &newborn->address))
+			break;
+	newborn->next = *cradle;
+	*cradle = newborn;
+}
+
+void kill_some_child(struct child *firstborn)
+{
+	const struct child *blanket, *next;
+
+	if (!(blanket = firstborn))
+		return;
+
+	for (; (next = blanket->next); blanket = next)
+		if (!addrcmp(&blanket->address, &next->address)) {
+			kill(blanket->cld.pid, SIGTERM);
+			break;
+		}
+}
+
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo)
+{
+	int status;
+	pid_t pid;
+
+	struct child **cradle, *blanket;
+	for (cradle = &firstborn; (blanket = *cradle);)
+		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+			if (loginfo) {
+				const char *dead = "";
+				if (status)
+					dead = " (with error)";
+				loginfo("[%"PRIuMAX"] Disconnected%s",
+					(uintmax_t)pid, dead);
+			}
+
+			/* remove the child */
+			*cradle = blanket->next;
+			(*live_children)--;
+			child_process_clear(&blanket->cld);
+			free(blanket);
+		} else
+			cradle = &blanket->next;
+}
diff --git a/daemon-utils.h b/daemon-utils.h
index 6710a2a6dc0..97e5cae20b8 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -2,6 +2,7 @@
 #define DAEMON_UTILS_H
 
 #include "git-compat-util.h"
+#include "run-command.h"
 #include "string-list.h"
 
 typedef void (*log_fn)(const char *msg, ...);
@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port,
 	       struct socketlist *socklist, int reuseaddr,
 	       log_fn logerror);
 
+struct child {
+	struct child *next;
+	struct child_process cld;
+	struct sockaddr_storage address;
+};
+
+/*
+ * Add the child_process to the set of children and increment the number of
+ * live children.
+ */
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+	       struct child *firstborn, unsigned int *live_children);
+
+/*
+ * Kill the newest connection from a duplicate IP.
+ *
+ * This function should be called if the number of connections grows
+ * past the maximum number of allowed connections.
+ */
+void kill_some_child(struct child *firstborn);
+
+/*
+ * Check for children that have disconnected and remove them from the
+ * active set, decrementing the number of live children.
+ *
+ * Optionally log the child PID that disconnected by passing a loginfo
+ * function.
+ */
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+			 log_fn loginfo);
+
 #endif
diff --git a/daemon.c b/daemon.c
index 1ed4e705680..ec3b407ecbc 100644
--- a/daemon.c
+++ b/daemon.c
@@ -785,93 +785,11 @@ static int execute(void)
 	return -1;
 }
 
-static int addrcmp(const struct sockaddr_storage *s1,
-    const struct sockaddr_storage *s2)
-{
-	const struct sockaddr *sa1 = (const struct sockaddr*) s1;
-	const struct sockaddr *sa2 = (const struct sockaddr*) s2;
-
-	if (sa1->sa_family != sa2->sa_family)
-		return sa1->sa_family - sa2->sa_family;
-	if (sa1->sa_family == AF_INET)
-		return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
-		    &((struct sockaddr_in *)s2)->sin_addr,
-		    sizeof(struct in_addr));
-#ifndef NO_IPV6
-	if (sa1->sa_family == AF_INET6)
-		return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
-		    &((struct sockaddr_in6 *)s2)->sin6_addr,
-		    sizeof(struct in6_addr));
-#endif
-	return 0;
-}
-
 static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child {
-	struct child *next;
-	struct child_process cld;
-	struct sockaddr_storage address;
-} *firstborn;
-
-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
-{
-	struct child *newborn, **cradle;
-
-	CALLOC_ARRAY(newborn, 1);
-	live_children++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
-			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
-}
-
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
-static void kill_some_child(void)
-{
-	const struct child *blanket, *next;
-
-	if (!(blanket = firstborn))
-		return;
-
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
-			break;
-		}
-}
-
-static void check_dead_children(void)
-{
-	int status;
-	pid_t pid;
-
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
-			const char *dead = "";
-			if (status)
-				dead = " (with error)";
-			loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
-
-			/* remove the child */
-			*cradle = blanket->next;
-			live_children--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
-		} else
-			cradle = &blanket->next;
-}
+static struct child *firstborn;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child();
+		kill_some_child(firstborn);
 		sleep(1);  /* give it some time to die */
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen);
+		add_child(&cld, addr, addrlen, firstborn, &live_children);
 }
 
 static void child_handler(int signo)
@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children();
+		check_dead_children(firstborn, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

[PATCH v7 03/12] daemon: rename some esoteric/laboured terminology

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Rename some of the variables and function arguments used to manage child
processes. The existing names are esoteric; stretching an analogy too
far to the point of being confusing to understand.

Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket"
to "current" and "cradle" to "ptr".

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 daemon-utils.c | 46 +++++++++++++++++++++++-----------------------
 daemon-utils.h |  6 +++---
 daemon.c       | 10 +++++-----
 3 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/daemon-utils.c b/daemon-utils.c
index 8506664b440..f23ea35ed7b 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,
 }
 
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn , unsigned int *live_children)
+	       struct child *first_child, unsigned int *live_children)
 {
-	struct child *newborn, **cradle;
+	struct child *new_cld, **current;
 
-	CALLOC_ARRAY(newborn, 1);
+	CALLOC_ARRAY(new_cld, 1);
 	(*live_children)++;
-	memcpy(&newborn->cld, cld, sizeof(*cld));
-	memcpy(&newborn->address, addr, addrlen);
-	for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-		if (!addrcmp(&(*cradle)->address, &newborn->address))
+	memcpy(&new_cld->cld, cld, sizeof(*cld));
+	memcpy(&new_cld->address, addr, addrlen);
+	for (current = &first_child; *current; current = &(*current)->next)
+		if (!addrcmp(&(*current)->address, &new_cld->address))
 			break;
-	newborn->next = *cradle;
-	*cradle = newborn;
+	new_cld->next = *current;
+	*current = new_cld;
 }
 
-void kill_some_child(struct child *firstborn)
+void kill_some_child(struct child *first_child)
 {
-	const struct child *blanket, *next;
+	const struct child *current, *next;
 
-	if (!(blanket = firstborn))
+	if (!(current = first_child))
 		return;
 
-	for (; (next = blanket->next); blanket = next)
-		if (!addrcmp(&blanket->address, &next->address)) {
-			kill(blanket->cld.pid, SIGTERM);
+	for (; (next = current->next); current = next)
+		if (!addrcmp(&current->address, &next->address)) {
+			kill(current->cld.pid, SIGTERM);
 			break;
 		}
 }
 
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
 			 log_fn loginfo)
 {
 	int status;
 	pid_t pid;
 
-	struct child **cradle, *blanket;
-	for (cradle = &firstborn; (blanket = *cradle);)
-		if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+	struct child **ptr, *current;
+	for (ptr = &first_child; (current = *ptr);)
+		if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
 			if (loginfo) {
 				const char *dead = "";
 				if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,
 			}
 
 			/* remove the child */
-			*cradle = blanket->next;
+			*ptr = current->next;
 			(*live_children)--;
-			child_process_clear(&blanket->cld);
-			free(blanket);
+			child_process_clear(&current->cld);
+			free(current);
 		} else
-			cradle = &blanket->next;
+			ptr = &current->next;
 }
diff --git a/daemon-utils.h b/daemon-utils.h
index 97e5cae20b8..c866e9c9a4e 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -32,7 +32,7 @@ struct child {
  * live children.
  */
 void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
-	       struct child *firstborn, unsigned int *live_children);
+	       struct child *first_child, unsigned int *live_children);
 
 /*
  * Kill the newest connection from a duplicate IP.
@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl
  * This function should be called if the number of connections grows
  * past the maximum number of allowed connections.
  */
-void kill_some_child(struct child *firstborn);
+void kill_some_child(struct child *first_child);
 
 /*
  * Check for children that have disconnected and remove them from the
@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);
  * Optionally log the child PID that disconnected by passing a loginfo
  * function.
  */
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
 			 log_fn loginfo);
 
 #endif
diff --git a/daemon.c b/daemon.c
index ec3b407ecbc..d3e7d81de18 100644
--- a/daemon.c
+++ b/daemon.c
@@ -789,7 +789,7 @@ static int max_connections = 32;
 
 static unsigned int live_children;
 
-static struct child *firstborn;
+static struct child *first_child;
 
 static struct strvec cld_argv = STRVEC_INIT;
 static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	struct child_process cld = CHILD_PROCESS_INIT;
 
 	if (max_connections && live_children >= max_connections) {
-		kill_some_child(firstborn);
+		kill_some_child(first_child);
 		sleep(1);  /* give it some time to die */
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 		if (live_children >= max_connections) {
 			close(incoming);
 			logerror("Too many children, dropping connection");
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
 	if (start_command(&cld))
 		logerror("unable to fork");
 	else
-		add_child(&cld, addr, addrlen, firstborn, &live_children);
+		add_child(&cld, addr, addrlen, first_child, &live_children);
 }
 
 static void child_handler(int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
 	for (;;) {
 		int i;
 
-		check_dead_children(firstborn, &live_children, loginfo);
+		check_dead_children(first_child, &live_children, loginfo);
 
 		if (poll(pfd, socklist->nr, -1) < 0) {
 			if (errno != EINTR) {
-- 
gitgitgadget

[PATCH v7 04/12] test-http-server: add stub HTTP server test helper

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.

Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Makefile                            |   1 +
 contrib/buildsystems/CMakeLists.txt |  11 +-
 t/helper/.gitignore                 |   1 +
 t/helper/test-http-server.c         | 381 ++++++++++++++++++++++++++++
 4 files changed, 392 insertions(+), 2 deletions(-)
 create mode 100644 t/helper/test-http-server.c

diff --git a/Makefile b/Makefile
index 2654094dbb5..3cd61c792ac 100644
--- a/Makefile
+++ b/Makefile
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o
 # Do not add more tests here unless they have extra dependencies. Add
 # them in TEST_BUILTINS_OBJS above.
 TEST_PROGRAMS_NEED_X += test-fake-ssh
+TEST_PROGRAMS_NEED_X += test-http-server
 TEST_PROGRAMS_NEED_X += test-tool
 
 TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..5d949dcb16c 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -961,6 +961,9 @@ if(BUILD_TESTING)
 add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
 target_link_libraries(test-fake-ssh common-main)
 
+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+target_link_libraries(test-http-server common-main)
+
 #reftable-tests
 parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
 list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC)
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
 	set_target_properties(test-fake-ssh test-tool
 				PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+	set_target_properties(test-http-server
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
 endif()
 
 #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts
 	git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
 
 set(wrapper_test_scripts
-	test-fake-ssh test-tool)
-
+	test-http-server test-fake-ssh test-tool)
 
 foreach(script ${wrapper_scripts})
 	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
 /test-tool
 /test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..6e9a1c479ce
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,381 @@
+#include "daemon-utils.h"
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+"           [--timeout=<n>] [--max-connections=<n>]\n"
+"           [--reuseaddr] [--pid-file=<file>]\n"
+"           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+static unsigned int timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+	struct strbuf msg = STRBUF_INIT;
+
+	strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+	strbuf_vaddf(&msg, err, params);
+	strbuf_addch(&msg, '\n');
+
+	fwrite(msg.buf, sizeof(char), msg.len, stderr);
+	fflush(stderr);
+
+	strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+	va_list params;
+	va_start(params, err);
+	logreport("error", err, params);
+	va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+	va_list params;
+	if (!verbose)
+		return;
+	va_start(params, err);
+	logreport("info", err, params);
+	va_end(params);
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client. The worker talks to the client
+ * on stdin and stdout.
+ */
+
+enum worker_result {
+	/*
+	 * Operation successful.
+	 * Caller *might* keep the socket open and allow keep-alive.
+	 */
+	WR_OK = 0,
+
+	/*
+	 * Fatal error that is not recoverable.
+	 * Close the socket and clean up.
+	 * Exit child-process with non-zero status.
+	 */
+	WR_FATAL_ERROR = 1,
+};
+
+static enum worker_result worker(void)
+{
+	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+	char *client_addr = getenv("REMOTE_ADDR");
+	char *client_port = getenv("REMOTE_PORT");
+	enum worker_result wr = WR_OK;
+
+	if (client_addr)
+		loginfo("Connection from %s:%s", client_addr, client_port);
+
+	set_keep_alive(0, logerror);
+
+	while (1) {
+		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+			logerror("unable to write response");
+			wr = WR_FATAL_ERROR;
+		}
+
+		if (wr != WR_OK)
+			break;
+	}
+
+	close(STDIN_FILENO);
+	close(STDOUT_FILENO);
+
+	/* Only WR_OK should result in a non-zero exit code */
+	return wr != WR_OK;
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child *first_child;
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+	struct child_process cld = CHILD_PROCESS_INIT;
+
+	if (max_connections && live_children >= max_connections) {
+		kill_some_child(first_child);
+		sleep(1);  /* give it some time to die */
+		check_dead_children(first_child, &live_children, loginfo);
+		if (live_children >= max_connections) {
+			close(incoming);
+			logerror("Too many children, dropping connection");
+			return;
+		}
+	}
+
+	if (addr->sa_family == AF_INET) {
+		char buf[128] = "";
+		struct sockaddr_in *sin_addr = (void *) addr;
+		inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+	} else if (addr->sa_family == AF_INET6) {
+		char buf[128] = "";
+		struct sockaddr_in6 *sin6_addr = (void *) addr;
+		inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+		strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+		strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+				 ntohs(sin6_addr->sin6_port));
+#endif
+	}
+
+	strvec_pushv(&cld.args, cld_argv.v);
+	cld.in = incoming;
+	cld.out = dup(incoming);
+
+	if (cld.out < 0)
+		logerror("could not dup() `incoming`");
+	else if (start_command(&cld))
+		logerror("unable to fork");
+	else
+		add_child(&cld, addr, addrlen, first_child, &live_children);
+}
+
+static void child_handler(int signo)
+{
+	/*
+	 * Otherwise empty handler because systemcalls will get interrupted
+	 * upon signal receipt
+	 * SysV needs the handler to be rearmed
+	 */
+	signal(SIGCHLD, child_handler);
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+	struct pollfd *pfd;
+	int i;
+
+	CALLOC_ARRAY(pfd, socklist->nr);
+
+	for (i = 0; i < socklist->nr; i++) {
+		pfd[i].fd = socklist->list[i];
+		pfd[i].events = POLLIN;
+	}
+
+	signal(SIGCHLD, child_handler);
+
+	for (;;) {
+		int i;
+		int nr_ready;
+		int timeout = (pid_file ? 100 : -1);
+
+		check_dead_children(first_child, &live_children, loginfo);
+
+		nr_ready = poll(pfd, socklist->nr, timeout);
+		if (nr_ready < 0) {
+			if (errno != EINTR) {
+				logerror("Poll failed, resuming: %s",
+				      strerror(errno));
+				sleep(1);
+			}
+			continue;
+		}
+		else if (nr_ready == 0) {
+			/*
+			 * If we have a pid_file, then we watch it.
+			 * If someone deletes it, we shutdown the service.
+			 * The shell scripts in the test suite will use this.
+			 */
+			if (!pid_file || file_exists(pid_file))
+				continue;
+			goto shutdown;
+		}
+
+		for (i = 0; i < socklist->nr; i++) {
+			if (pfd[i].revents & POLLIN) {
+				union {
+					struct sockaddr sa;
+					struct sockaddr_in sai;
+#ifndef NO_IPV6
+					struct sockaddr_in6 sai6;
+#endif
+				} ss;
+				socklen_t sslen = sizeof(ss);
+				int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+				if (incoming < 0) {
+					switch (errno) {
+					case EAGAIN:
+					case EINTR:
+					case ECONNABORTED:
+						continue;
+					default:
+						die_errno("accept returned");
+					}
+				}
+				handle(incoming, &ss.sa, sslen);
+			}
+		}
+	}
+
+shutdown:
+	loginfo("Starting graceful shutdown (pid-file gone)");
+	for (i = 0; i < socklist->nr; i++)
+		close(socklist->list[i]);
+
+	return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+	struct socketlist socklist = { NULL, 0, 0 };
+
+	socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
+	if (socklist.nr == 0)
+		die("unable to allocate any listen sockets on port %u",
+		    listen_port);
+
+	loginfo("Ready to rumble");
+
+	/*
+	 * Wait to create the pid-file until we've setup the sockets
+	 * and are open for business.
+	 */
+	if (pid_file)
+		write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+	return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances.  So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+	int listen_port = 0;
+	struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+	int worker_mode = 0;
+	int i;
+
+	trace2_cmd_name("test-http-server");
+	trace2_cmd_list_config();
+	trace2_cmd_list_env_vars();
+	setup_git_directory_gently(NULL);
+
+	for (i = 1; i < argc; i++) {
+		const char *arg = argv[i];
+		const char *v;
+
+		if (skip_prefix(arg, "--listen=", &v)) {
+			string_list_append(&listen_addr, xstrdup_tolower(v));
+			continue;
+		}
+		if (skip_prefix(arg, "--port=", &v)) {
+			char *end;
+			unsigned long n;
+			n = strtoul(v, &end, 0);
+			if (*v && !*end) {
+				listen_port = n;
+				continue;
+			}
+		}
+		if (!strcmp(arg, "--worker")) {
+			worker_mode = 1;
+			trace2_cmd_mode("worker");
+			continue;
+		}
+		if (!strcmp(arg, "--verbose")) {
+			verbose = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--timeout=", &v)) {
+			timeout = atoi(v);
+			continue;
+		}
+		if (skip_prefix(arg, "--max-connections=", &v)) {
+			max_connections = atoi(v);
+			if (max_connections < 0)
+				max_connections = 0; /* unlimited */
+			continue;
+		}
+		if (!strcmp(arg, "--reuseaddr")) {
+			reuseaddr = 1;
+			continue;
+		}
+		if (skip_prefix(arg, "--pid-file=", &v)) {
+			pid_file = v;
+			continue;
+		}
+
+		fprintf(stderr, "error: unknown argument '%s'\n", arg);
+		usage(test_http_auth_usage);
+	}
+
+	/* avoid splitting a message in the middle */
+	setvbuf(stderr, NULL, _IOFBF, 4096);
+
+	if (listen_port == 0)
+		listen_port = DEFAULT_GIT_PORT;
+
+	/*
+	 * If no --listen=<addr> args are given, the setup_named_sock()
+	 * code will use receive a NULL address and set INADDR_ANY.
+	 * This exposes both internal and external interfaces on the
+	 * port.
+	 *
+	 * Disallow that and default to the internal-use-only loopback
+	 * address.
+	 */
+	if (!listen_addr.nr)
+		string_list_append(&listen_addr, "127.0.0.1");
+
+	/*
+	 * worker_mode is set in our own child process instances
+	 * (that are bound to a connected socket from a client).
+	 */
+	if (worker_mode)
+		return worker();
+
+	/*
+	 * `cld_argv` is a bit of a clever hack. The top-level instance
+	 * of test-http-server does the normal bind/listen/accept stuff.
+	 * For each incoming socket, the top-level process spawns
+	 * a child instance of test-http-server *WITH* the additional
+	 * `--worker` argument. This causes the child to set `worker_mode`
+	 * and immediately call `worker()` using the connected socket (and
+	 * without the usual need for fork() or threads).
+	 *
+	 * The magic here is made possible because `cld_argv` is static
+	 * and handle() (called by service_loop()) knows about it.
+	 */
+	strvec_push(&cld_argv, argv[0]);
+	strvec_push(&cld_argv, "--worker");
+	for (i = 1; i < argc; ++i)
+		strvec_push(&cld_argv, argv[i]);
+
+	/*
+	 * Setup primary instance to listen for connections.
+	 */
+	return serve(&listen_addr, listen_port);
+}
-- 
gitgitgadget

Re: [PATCH v7 04/12] test-http-server: add stub HTTP server test helper

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:42PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> Introduce a mini HTTP server helper that in the future will be enhanced
> to provide a frontend for the git-http-backend, with support for
> arbitrary authentication schemes.
> 
> Right now, test-http-server is a pared-down copy of the git-daemon that
> always returns a 501 Not Implemented response to all callers.

This may be a dumb question, but I didn't see it raised or answered in
the cover letter or earlier in the thread: what does this custom server
give us that our current use of apache in the tests does not?

I'd imagine the answer is along the lines of: configuring apache to
respond to auth in the way we'd like is hard and/or impossible. And if
so, and if it's just "hard", I'd ask "how hard?".

Because I see a few downsides to introducing a custom server here:

  1. It may or may not behave like real-world servers, which makes the
     test slightly less good. Not that apache can claim to cover all
     real-world behavior, but it's probably closer to reality (and that
     has flushed out interesting bugs and behaviors before).

  2. It's a non-trivial amount of code, doing tricky things like
     daemonizing, socket setup and I/O, pidfiles, and so on.  For
     example, it handles multiple listen addresses, and ipv6. Do we
     really need that? And we take shortcuts around things like CGI
     output buffering. I do see that you tried to reuse some existing
     code, but...

  3. You're reusing parts of git-daemon, which I personally consider to
     be one of the absolute low-points of code quality inside git.git. I
     know that's a subjective statement, but my experience running it
     within GitHub was that there were a lot of rough edges, and we
     ended up rewriting several parts of it. In particular, the
     child-handling is inefficient (I seem to recall that it's quadratic
     in several places) and has odd behaviors (its kill_some_child() is
     basically nonsense, and can starve requests). Probably none of that
     matters for your use case in tests, which is likely doing one
     request at a time.

     But then I'd wonder: do we really need those bits at all, then? In
     fact, would it be sufficient to write the server to handle one
     request at a time, without spawning a worker child at all?

I dunno. I know I am showing up to review quite late in the life of this
patch series, and that probably makes me a bad person to start the
review with "and could you re-do the whole test infrastructure". So if
you want to tell me to get lost, I'd understand. But I had hoped that
one day we could just delete all of daemon.c, and this moves in the
opposite direction.

I think my order of preference (if you care ;) ) is:

  1. Can we do it with apache?

  2. If not, could we do it with a trivial application of some existing
     http server framework? I know that may mean extra dependencies, but
     there's a lot of perl in the test suite already, and it doesn't
     seem too terrible to me to require it for these tests.

  3. If not, can we make the http-server code even more minimal?

>  Makefile                            |   1 +
>  contrib/buildsystems/CMakeLists.txt |  11 +-
>  t/helper/.gitignore                 |   1 +
>  t/helper/test-http-server.c         | 381 ++++++++++++++++++++++++++++
>  4 files changed, 392 insertions(+), 2 deletions(-)
>  create mode 100644 t/helper/test-http-server.c

If we do use this code, here are a few small bits I noticed:

> +static void child_handler(int signo)
> +{
> +	/*
> +	 * Otherwise empty handler because systemcalls will get interrupted
> +	 * upon signal receipt
> +	 * SysV needs the handler to be rearmed
> +	 */
> +	signal(SIGCHLD, child_handler);
> +}

daemon.c has this, too. If we're going to share its child-handling code,
should it maybe just handle this part, too?

> +static int service_loop(struct socketlist *socklist)
> +{
> +	struct pollfd *pfd;
> +	int i;
> +
> +	CALLOC_ARRAY(pfd, socklist->nr);

(Actually, Coverity noticed this, not me).

This pfd is never freed. I know this is copied from daemon.c, but in
that file we never return from the function. Here you do break out of
the loop and try to clean up; you'd want to free(pfd) there.

> +	for (;;) {
> +		int i;
> +		int nr_ready;
> +		int timeout = (pid_file ? 100 : -1);
> +
> +		check_dead_children(first_child, &live_children, loginfo);
> +
> +		nr_ready = poll(pfd, socklist->nr, timeout);
> +		if (nr_ready < 0) {
> +			if (errno != EINTR) {
> +				logerror("Poll failed, resuming: %s",
> +				      strerror(errno));
> +				sleep(1);
> +			}
> +			continue;
> +		}
> +		else if (nr_ready == 0) {
> +			/*
> +			 * If we have a pid_file, then we watch it.
> +			 * If someone deletes it, we shutdown the service.
> +			 * The shell scripts in the test suite will use this.
> +			 */
> +			if (!pid_file || file_exists(pid_file))
> +				continue;
> +			goto shutdown;
> +		}

I wondered how this would work, since removal of the pid file won't
trigger poll(). But it looks like you set the timeout unconditionally in
that case, so we're effectively polling for its removal every 100ms.
It's not beautiful, but it should work reliably.

That also made me wonder about this timeout:

> +		if (skip_prefix(arg, "--timeout=", &v)) {
> +			timeout = atoi(v);
> +			continue;
> +		}

but it is not used. The "timeout" in service_loop shadows the global,
and nobody ever looks at the global (however, it looks like a later
patch adds an alarm() which uses it).

> +		if (skip_prefix(arg, "--max-connections=", &v)) {
> +			max_connections = atoi(v);
> +			if (max_connections < 0)
> +				max_connections = 0; /* unlimited */
> +			continue;
> +		}

I don't think any caller ever uses --max-connections, though. This could
be dropped, and that would simplify service_loop a bit.

> +	if (listen_port == 0)
> +		listen_port = DEFAULT_GIT_PORT;

That's a funny default. Surely "80" or even "8080" would make more
sense. But really, since our purpose is tests, isn't it a
misconfiguration if the test does not tell us which port (which is
generally dynamic based on the test number), and we should bail?

> +	/*
> +	 * If no --listen=<addr> args are given, the setup_named_sock()
> +	 * code will use receive a NULL address and set INADDR_ANY.
> +	 * This exposes both internal and external interfaces on the
> +	 * port.
> +	 *
> +	 * Disallow that and default to the internal-use-only loopback
> +	 * address.
> +	 */
> +	if (!listen_addr.nr)
> +		string_list_append(&listen_addr, "127.0.0.1");

Likewise, it seems like you could probably ditch --listen entirely, and
just always listen on 127.0.0.1, for the purposes of the tests.

-Peff

[PATCH v7 05/12] test-http-server: add HTTP error response function

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 76 +++++++++++++++++++++++++++++++++----
 1 file changed, 69 insertions(+), 7 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 6e9a1c479ce..7ca4ddc7999 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -76,11 +76,75 @@ enum worker_result {
 	 * Exit child-process with non-zero status.
 	 */
 	WR_FATAL_ERROR = 1,
+
+	/*
+	 * Close the socket and clean up. Does not imply an error.
+	 */
+	WR_HANGUP = 2,
 };
 
+static enum worker_result send_http_error(int fd, int http_code,
+					  const char *http_code_name,
+					  int retry_after_seconds,
+					  struct string_list *response_headers,
+					  enum worker_result wr_in)
+{
+	struct strbuf response_header = STRBUF_INIT;
+	struct strbuf response_content = STRBUF_INIT;
+	struct string_list_item *h;
+	enum worker_result wr;
+
+	strbuf_addf(&response_content, "Error: %d %s\r\n", http_code,
+		    http_code_name);
+
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_content, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code,
+		    http_code_name);
+	strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+	strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
+	strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n",
+		    (uintmax_t)response_content.len);
+
+	if (retry_after_seconds > 0)
+		strbuf_addf(&response_header, "Retry-After: %d\r\n",
+			    retry_after_seconds);
+
+	strbuf_addf(&response_header, "Server: test-http-server/%s\r\n",
+		    git_version_string);
+	strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0,
+		    DATE_MODE(RFC2822)));
+
+	if (response_headers)
+		for_each_string_list_item(h, response_headers)
+			strbuf_addf(&response_header, "%s\r\n", h->string);
+	strbuf_addstr(&response_header, "\r\n");
+
+	if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+		logerror("unable to write response header");
+		wr = WR_FATAL_ERROR;
+		goto done;
+	}
+
+	if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+		logerror("unable to write response content body");
+		wr = WR_FATAL_ERROR;
+		goto done;
+	}
+
+	wr = wr_in;
+
+done:
+	strbuf_release(&response_header);
+	strbuf_release(&response_content);
+
+	return wr;
+}
+
 static enum worker_result worker(void)
 {
-	const char *response = "HTTP/1.1 501 Not Implemented\r\n";
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -91,10 +155,8 @@ static enum worker_result worker(void)
 	set_keep_alive(0, logerror);
 
 	while (1) {
-		if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
-			logerror("unable to write response");
-			wr = WR_FATAL_ERROR;
-		}
+		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
+				     NULL, WR_HANGUP);
 
 		if (wr != WR_OK)
 			break;
@@ -103,8 +165,8 @@ static enum worker_result worker(void)
 	close(STDIN_FILENO);
 	close(STDOUT_FILENO);
 
-	/* Only WR_OK should result in a non-zero exit code */
-	return wr != WR_OK;
+	/* Only WR_OK and WR_HANGUP should result in a non-zero exit code */
+	return wr != WR_OK && wr != WR_HANGUP;
 }
 
 static int max_connections = 32;
-- 
gitgitgadget

[PATCH v7 06/12] test-http-server: add HTTP request parsing

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add ability to parse HTTP requests to the test-http-server test helper.
Introduce `struct req` to store request information including:

 * HTTP method & version
 * Request path and query parameters
 * Headers
 * Content type and length (from `Content-Type` and `-Length` headers)

Failure to parse the request results in a 400 Bad Request response to
the client. Note that we're not trying to support all possible requests
here, but just enough to exercise all code under test.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 202 +++++++++++++++++++++++++++++++++++-
 t/t5556-http-auth.sh        |  90 ++++++++++++++++
 2 files changed, 290 insertions(+), 2 deletions(-)
 create mode 100755 t/t5556-http-auth.sh

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 7ca4ddc7999..900f5733cc1 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -81,8 +81,53 @@ enum worker_result {
 	 * Close the socket and clean up. Does not imply an error.
 	 */
 	WR_HANGUP = 2,
+
+	/*
+	 * Unexpected request message or error in request parsing.
+	 * Respond with an 400 error. Close the socket and cleanup.
+	 * Exit child-process with a non-zero status.
+	 */
+	WR_CLIENT_ERROR = 3,
+};
+
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+	struct strbuf start_line;
+
+	const char *method;
+	const char *http_version;
+
+	struct strbuf uri_path;
+	struct strbuf query_args;
+
+	struct string_list header_list;
+	const char *content_type;
+	uintmax_t content_length;
+	unsigned has_content_length:1;
 };
 
+#define REQ__INIT { \
+	.start_line = STRBUF_INIT, \
+	.uri_path = STRBUF_INIT, \
+	.query_args = STRBUF_INIT, \
+	.header_list = STRING_LIST_INIT_NODUP, \
+	.content_type = NULL, \
+	.content_length = 0, \
+	.has_content_length = 0, \
+}
+
+static void req__release(struct req *req)
+{
+	strbuf_release(&req->start_line);
+
+	strbuf_release(&req->uri_path);
+	strbuf_release(&req->query_args);
+
+	string_list_clear(&req->header_list, 0);
+}
+
 static enum worker_result send_http_error(int fd, int http_code,
 					  const char *http_code_name,
 					  int retry_after_seconds,
@@ -143,8 +188,150 @@ done:
 	return wr;
 }
 
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+	struct strbuf h = STRBUF_INIT;
+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+	int nr_start_line_fields;
+	const char *uri_target;
+	const char *query;
+	char *hp;
+	const char *hv;
+
+	enum worker_result result = WR_OK;
+
+	/*
+	 * Read line 0 of the request and split it into component parts:
+	 *
+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
+	 *
+	 */
+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+		result = WR_HANGUP;
+		goto done;
+	}
+
+	strbuf_trim_trailing_newline(&req->start_line);
+
+	nr_start_line_fields = string_list_split(&start_line_fields,
+						 req->start_line.buf,
+						 ' ', -1);
+	if (nr_start_line_fields != 3) {
+		logerror("could not parse request start-line '%s'",
+			 req->start_line.buf);
+		result = WR_CLIENT_ERROR;
+		goto done;
+	}
+
+	req->method = xstrdup(start_line_fields.items[0].string);
+	req->http_version = xstrdup(start_line_fields.items[2].string);
+
+	uri_target = start_line_fields.items[1].string;
+
+	if (strcmp(req->http_version, "HTTP/1.1")) {
+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
+			 req->http_version);
+		result = WR_CLIENT_ERROR;
+		goto done;
+	}
+
+	query = strchr(uri_target, '?');
+
+	if (query) {
+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+		strbuf_addstr(&req->query_args, query + 1);
+	} else {
+		strbuf_addstr(&req->uri_path, uri_target);
+		strbuf_trim_trailing_dir_sep(&req->uri_path);
+	}
+
+	/*
+	 * Read the set of HTTP headers into a string-list.
+	 */
+	while (1) {
+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+			goto done;
+		strbuf_trim_trailing_newline(&h);
+
+		if (!h.len)
+			goto done; /* a blank line ends the header */
+
+		hp = strbuf_detach(&h, NULL);
+		string_list_append(&req->header_list, hp);
+
+		/* also store common request headers as struct req members */
+		if (skip_iprefix(hp, "Content-Type: ", &hv)) {
+			req->content_type = hv;
+		} else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
+			/*
+			 * Content-Length is always non-negative, but has no
+			 * upper bound according to RFC 7230 (§3.3.2).
+			 */
+			intmax_t len = 0;
+			if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
+			    len == INTMAX_MAX) {
+				logerror("invalid content-length: '%s'", hv);
+				result = WR_CLIENT_ERROR;
+				goto done;
+			}
+
+			req->content_length = (uintmax_t)len;
+			req->has_content_length = 1;
+		}
+	}
+
+	/*
+	 * We do not attempt to read the <message-body>, if it exists.
+	 * We let our caller read/chunk it in as appropriate.
+	 */
+
+done:
+	string_list_clear(&start_line_fields, 0);
+
+	/*
+	 * This is useful for debugging the request, but very noisy.
+	 */
+	if (trace2_is_enabled()) {
+		struct string_list_item *item;
+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+		if (req->has_content_length)
+			trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
+				      req->content_length);
+		if (req->content_type)
+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+		for_each_string_list_item(item, &req->header_list)
+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+	}
+
+	return result;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+			       WR_HANGUP);
+}
+
 static enum worker_result worker(void)
 {
+	struct req req = REQ__INIT;
 	char *client_addr = getenv("REMOTE_ADDR");
 	char *client_port = getenv("REMOTE_PORT");
 	enum worker_result wr = WR_OK;
@@ -155,9 +342,20 @@ static enum worker_result worker(void)
 	set_keep_alive(0, logerror);
 
 	while (1) {
-		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-				     NULL, WR_HANGUP);
+		req__release(&req);
+
+		alarm(timeout);
+		wr = req__read(&req, 0);
+		alarm(0);
+
+		if (wr == WR_CLIENT_ERROR)
+			wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
+					     -1, NULL, wr);
+
+		if (wr != WR_OK)
+			break;
 
+		wr = dispatch(&req);
 		if (wr != WR_OK)
 			break;
 	}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..06efc85ca53
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,90 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+# Setup a repository
+#
+REPO_DIR="$TRASH_DIRECTORY"/repo
+
+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+	test_create_repo "$REPO_DIR" &&
+	git -C "$REPO_DIR" branch -M main
+'
+
+run_http_server_worker() {
+	(
+		cd "$REPO_DIR"
+		test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
+	)
+}
+
+per_test_cleanup () {
+	rm -f OUT.* &&
+	rm -f IN.* &&
+}
+
+test_expect_success 'http auth server request parsing' '
+	test_when_finished "per_test_cleanup" &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		allowAnonymous = true
+	EOF
+
+	echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
+	echo "HTTP/1.1 200 OK" >OUT.http200 &&
+
+	cat >IN.http.valid <<-EOF &&
+	GET /info/refs HTTP/1.1
+	Content-Length: 0
+	EOF
+
+	cat >IN.http.badfirstline <<-EOF &&
+	/info/refs GET HTTP
+	EOF
+
+	cat >IN.http.badhttpver <<-EOF &&
+	GET /info/refs HTTP/999.9
+	EOF
+
+	cat >IN.http.ltzlen <<-EOF &&
+	GET /info/refs HTTP/1.1
+	Content-Length: -1
+	EOF
+
+	cat >IN.http.badlen <<-EOF &&
+	GET /info/refs HTTP/1.1
+	Content-Length: not-a-number
+	EOF
+
+	cat >IN.http.overlen <<-EOF &&
+	GET /info/refs HTTP/1.1
+	Content-Length: 9223372036854775807
+	EOF
+
+	run_http_server_worker \
+		--auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
+		| head -n1 >OUT.actual &&
+	test_cmp OUT.http200 OUT.actual &&
+
+	run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
+	test_cmp OUT.http400 OUT.actual &&
+
+	run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
+	test_cmp OUT.http400 OUT.actual &&
+
+	run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
+	test_cmp OUT.http400 OUT.actual &&
+
+	run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
+	test_cmp OUT.http400 OUT.actual
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v7 06/12] test-http-server: add HTTP request parsing

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:44PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> +#define REQ__INIT { \
> +     .start_line = STRBUF_INIT, \
> +     .uri_path = STRBUF_INIT, \
> +     .query_args = STRBUF_INIT, \
> +     .header_list = STRING_LIST_INIT_NODUP, \
> +     .content_type = NULL, \
> +     .content_length = 0, \
> +     .has_content_length = 0, \
> +}

We declare header_list as nodup, but later we put actual duplicated
strings in it:

> +             hp = strbuf_detach(&h, NULL);
> +             string_list_append(&req->header_list, hp);

So later when we free it:

> +static void req__release(struct req *req)
> +{
> +     strbuf_release(&req->start_line);
> +
> +     strbuf_release(&req->uri_path);
> +     strbuf_release(&req->query_args);
> +
> +     string_list_clear(&req->header_list, 0);
> +}

the strings will be leaked. There are a lot of solutions here, including
setting strdup_strings right before freeing. But it's probably
reasonable to just use INIT_DUP, and then when storing, just do:

  string_list_append(&req->header_list, h.buf);

Since "h" is filled by strbuf_getwholeline(), there's no need to erase
the contents. It should reset the buffer itself (and so you end up
re-using the same buffer, rather than freeing it for each loop).  You
will have to remember to strbuf_release() after the loop, though.

The leak isn't very big, and we hold onto it until the process ends
anyway, but it will probably cause the leak-detector to complain.

(Yet another solution would just be to dump the trace as we parse the
headers, rather than holding them, since that appears to be the only use
of header_list).

> +	/*
> +	 * Read the set of HTTP headers into a string-list.
> +	 */
> +	while (1) {
> +		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
> +			goto done;
> +		strbuf_trim_trailing_newline(&h);
> +
> +		if (!h.len)
> +			goto done; /* a blank line ends the header */
> +
> +		hp = strbuf_detach(&h, NULL);
> +		string_list_append(&req->header_list, hp);
> +
> +		/* also store common request headers as struct req members */
> +		if (skip_iprefix(hp, "Content-Type: ", &hv)) {
> +			req->content_type = hv;

I think this is stricter than necessary. The whitespace after the colon
is optional, but can also be longer than just one space (or could be a
tab). It's probably OK to be picky here since this is just for tests,
but we'd want to make sure we're not this picky on the client side.

> +		} else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
> +			/*
> +			 * Content-Length is always non-negative, but has no
> +			 * upper bound according to RFC 7230 (§3.3.2).
> +			 */
> +			intmax_t len = 0;
> +			if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
> +			    len == INTMAX_MAX) {
> +				logerror("invalid content-length: '%s'", hv);
> +				result = WR_CLIENT_ERROR;
> +				goto done;
> +			}

We usually avoid sscanf because it's error-checking sucks. For example,
this will accept "123.garbage", but you can't tell because you have no
clue how far it got.  Something like strtoimax() is better. It probably
doesn't matter much since this is test code, though I do think in the
long run it would be nice to add scanf(), etc, to our list of banned
functions (there are one or two other uses currently, though, so that
isn't imminent).

-Peff

[PATCH v7 07/12] test-http-server: pass Git requests to http-backend

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Teach the test-http-sever test helper to forward Git requests to the
`git-http-backend`.

Introduce a new test script t5556-http-auth.sh that spins up the test
HTTP server and attempts an `ls-remote` on the served repository,
without any authentication.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 68 ++++++++++++++++++++++++++++++
 t/t5556-http-auth.sh        | 83 +++++++++++++++++++++++++++++++++++++
 2 files changed, 151 insertions(+)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 900f5733cc1..4191daf3c64 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -323,8 +323,76 @@ done:
 	return result;
 }
 
+static int is_git_request(struct req *req)
+{
+	static regex_t *smart_http_regex;
+	static int initialized;
+
+	if (!initialized) {
+		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+		/*
+		 * This regular expression matches all dumb and smart HTTP
+		 * requests that are currently in use, and defined in
+		 * Documentation/gitprotocol-http.txt.
+		 *
+		 */
+		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
+			    REG_EXTENDED)) {
+			warning("could not compile smart HTTP regex");
+			smart_http_regex = NULL;
+		}
+		initialized = 1;
+	}
+
+	return smart_http_regex &&
+		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req)
+{
+	const char *ok = "HTTP/1.1 200 OK\r\n";
+	struct child_process cp = CHILD_PROCESS_INIT;
+	int res;
+
+	/*
+	 * Note that we always respond with a 200 OK response even if the
+	 * http-backend process exits with an error. This helper is intended
+	 * only to be used to exercise the HTTP auth handling in the Git client,
+	 * and specifically around authentication (not handled by http-backend).
+	 *
+	 * If we wanted to respond with a more 'valid' HTTP response status then
+	 * we'd need to buffer the output of http-backend, wait for and grok the
+	 * exit status of the process, then write the HTTP status line followed
+	 * by the http-backend output. This is outside of the scope of this test
+	 * helper's use at time of writing.
+	 */
+	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+		return error(_("could not send '%s'"), ok);
+
+	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+	if (req->query_args.len)
+		strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf);
+	if (req->content_type)
+		strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type);
+	if (req->has_content_length)
+		strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX,
+			(uintmax_t)req->content_length);
+	cp.git_cmd = 1;
+	strvec_push(&cp.args, "http-backend");
+	res = run_command(&cp);
+	close(STDOUT_FILENO);
+	close(STDIN_FILENO);
+	return !!res;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	if (is_git_request(req))
+		return do__git(req);
+
 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
 			       WR_HANGUP);
 }
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 06efc85ca53..c0a47ce342b 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -5,10 +5,25 @@ test_description='test http auth header and credential helper interop'
 TEST_NO_CREATE_REPO=1
 . ./test-lib.sh
 
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
 # Setup a repository
 #
 REPO_DIR="$TRASH_DIRECTORY"/repo
 
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
 SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
 
 PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
@@ -25,7 +40,65 @@ run_http_server_worker() {
 	)
 }
 
+stop_http_server () {
+	if ! test -f "$PID_FILE"
+	then
+		return 0
+	fi
+	#
+	# The server will shutdown automatically when we delete the pid-file.
+	#
+	rm -f "$PID_FILE"
+	#
+	# Give it a few seconds to shutdown (mainly to completely release the
+	# port before the next test start another instance and it attempts to
+	# bind to it).
+	#
+	for k in 0 1 2 3 4
+	do
+		if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "stop_http_server: timeout waiting for server shutdown"
+	return 1
+}
+
+start_http_server () {
+	#
+	# Launch our server into the background in repo_dir.
+	#
+	(
+		cd "$REPO_DIR"
+		test-http-server --verbose \
+			--listen=127.0.0.1 \
+			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+			--reuseaddr \
+			--pid-file="$PID_FILE" \
+			"$@" \
+			2>"$SERVER_LOG" &
+	)
+	#
+	# Give it a few seconds to get started.
+	#
+	for k in 0 1 2 3 4
+	do
+		if test -f "$PID_FILE"
+		then
+			return 0
+		fi
+		sleep 1
+	done
+
+	echo "start_http_server: timeout waiting for server startup"
+	return 1
+}
+
 per_test_cleanup () {
+	stop_http_server &&
 	rm -f OUT.* &&
 	rm -f IN.* &&
 }
@@ -87,4 +160,14 @@ test_expect_success 'http auth server request parsing' '
 	test_cmp OUT.http400 OUT.actual
 '
 
+
+test_expect_success 'http auth anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+
+	start_http_server &&
+
+	# Attempt to read from a protected repository
+	git ls-remote $ORIGIN_URL
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v7 07/12] test-http-server: pass Git requests to http-backend

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:45PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> +static int is_git_request(struct req *req)
> +{
> +	static regex_t *smart_http_regex;
> +	static int initialized;
> +
> +	if (!initialized) {
> +		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
> +		/*
> +		 * This regular expression matches all dumb and smart HTTP
> +		 * requests that are currently in use, and defined in
> +		 * Documentation/gitprotocol-http.txt.
> +		 *
> +		 */
> +		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
> +			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
> +			    REG_EXTENDED)) {
> +			warning("could not compile smart HTTP regex");
> +			smart_http_regex = NULL;
> +		}
> +		initialized = 1;
> +	}
> +
> +	return smart_http_regex &&
> +		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
> +}

Assigning NULL to smart_http_regex leaks the earlier allocation. You
could free it, but I have to wonder why it is on the heap in the first
place. Yes, you check for NULL and return 0 if it failed to compile,
but...why would it? It's hard-coded. And if it does fail, wouldn't you
want to fail immediately and loudly, because it means all of the tests
are broken?

I.e., something like this is a bit simpler:

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 14d170e640..8048ba1636 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -327,28 +327,25 @@ static enum worker_result req__read(struct req *req, int fd)
 
 static int is_git_request(struct req *req)
 {
-	static regex_t *smart_http_regex;
+	static regex_t smart_http_regex;
 	static int initialized;
 
 	if (!initialized) {
-		smart_http_regex = xmalloc(sizeof(*smart_http_regex));
 		/*
 		 * This regular expression matches all dumb and smart HTTP
 		 * requests that are currently in use, and defined in
 		 * Documentation/gitprotocol-http.txt.
 		 *
 		 */
-		if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+		if (regcomp(&smart_http_regex, "^/(HEAD|info/refs|"
 			    "objects/info/[^/]+|git-(upload|receive)-pack)$",
 			    REG_EXTENDED)) {
-			warning("could not compile smart HTTP regex");
-			smart_http_regex = NULL;
+			die("could not compile smart HTTP regex");
 		}
 		initialized = 1;
 	}
 
-	return smart_http_regex &&
-		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+	return !regexec(&smart_http_regex, req->uri_path.buf, 0, NULL, 0);
 }
 
 static enum worker_result do__git(struct req *req, const char *user)

> +start_http_server () {
> +	#
> +	# Launch our server into the background in repo_dir.
> +	#
> +	(
> +		cd "$REPO_DIR"
> +		test-http-server --verbose \
> +			--listen=127.0.0.1 \
> +			--port=$GIT_TEST_HTTP_PROTOCOL_PORT \
> +			--reuseaddr \
> +			--pid-file="$PID_FILE" \
> +			"$@" \
> +			2>"$SERVER_LOG" &
> +	)
> +	#
> +	# Give it a few seconds to get started.
> +	#
> +	for k in 0 1 2 3 4
> +	do
> +		if test -f "$PID_FILE"
> +		then
> +			return 0
> +		fi
> +		sleep 1
> +	done

Yuck. This makes the test take a long time to run, since it will almost
always "sleep 1" each time (and it looks like you bring the server up
and down in several tests). Worse, it's at risk of failing racily if it
ever takes more than 5 seconds to start up. That should be uncommon, I'd
think, but could happen on a heavily loaded system.

There's a race-less solution using fifos in lib-git-daemon.sh, where we
wait for the "ready to rumble" line. It's kind of horrific, but it does
work and is battle-tested.

-Peff

[PATCH v7 08/12] test-http-server: add simple authentication

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
a configuration file (in the normal gitconfig file format).
Incoming requests are compared against the set of valid schemes and
tokens and only approved if a matching token is found, or if no auth
was provided and anonymous auth is enabled.

Configuration for auth includes a simple set of three options:

[auth]
	challenge = <scheme>[:<challenge_params>]
	token = <scheme>:[<token>]*
	allowAnonymous = <bool>

`auth.challenge` allows you define what authentication schemes, and
optional challenge parameters the server should use. Scheme names are
unique and subsequently specified challenge parameters in the config
file will replace previously specified ones.

`auth.token` allows you to define the set of value token values for an
authentication scheme. This is a multi-var and each entry in the
config file will append to the set of valid tokens for that scheme.
Specifying an empty token value will clear the list of tokens so far for
that scheme, i.e. `token = <scheme>:`.

`auth.allowAnonymous` controls whether or not unauthenticated requests
(those without any `Authorization` headers) should succeed or not, and
trigger a 401 Unauthorized response.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 232 +++++++++++++++++++++++++++++++++++-
 t/t5556-http-auth.sh        |  43 ++++++-
 2 files changed, 272 insertions(+), 3 deletions(-)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 4191daf3c64..72c6cca7e5c 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -7,6 +7,7 @@
 #include "version.h"
 #include "dir.h"
 #include "date.h"
+#include "config.h"
 
 #define TR2_CAT "test-http-server"
 
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =
 "           [--timeout=<n>] [--max-connections=<n>]\n"
 "           [--reuseaddr] [--pid-file=<file>]\n"
 "           [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+"           [--auth-config=<file>]\n"
 ;
 
 static unsigned int timeout;
@@ -349,7 +351,7 @@ static int is_git_request(struct req *req)
 		!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
 }
 
-static enum worker_result do__git(struct req *req)
+static enum worker_result do__git(struct req *req, const char *user)
 {
 	const char *ok = "HTTP/1.1 200 OK\r\n";
 	struct child_process cp = CHILD_PROCESS_INIT;
@@ -366,10 +368,16 @@ static enum worker_result do__git(struct req *req)
 	 * exit status of the process, then write the HTTP status line followed
 	 * by the http-backend output. This is outside of the scope of this test
 	 * helper's use at time of writing.
+	 *
+	 * The important auth responses (401) we are handling prior to getting
+	 * to this point.
 	 */
 	if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
 		return error(_("could not send '%s'"), ok);
 
+	if (user)
+		strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
 	strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
 	strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
 	strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
@@ -388,10 +396,217 @@ static enum worker_result do__git(struct req *req)
 	return !!res;
 }
 
+enum auth_result {
+	/* No auth module matches the request. */
+	AUTH_UNKNOWN = 0,
+
+	/* Auth module denied the request. */
+	AUTH_DENY = 1,
+
+	/* Auth module successfully validated the request. */
+	AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+	char *scheme;
+	char *challenge_params;
+	struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(const char *scheme, int create)
+{
+	struct auth_module *mod;
+	for (size_t i = 0; i < auth_modules_nr; i++) {
+		mod = auth_modules[i];
+		if (!strcasecmp(mod->scheme, scheme))
+			return mod;
+	}
+
+	if (create) {
+		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+		mod->scheme = xstrdup(scheme);
+		mod->challenge_params = NULL;
+		ALLOC_ARRAY(mod->tokens, 1);
+		string_list_init_dup(mod->tokens);
+
+		ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+		auth_modules[auth_modules_nr++] = mod;
+
+		return mod;
+	}
+
+	return NULL;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+	enum auth_result result = AUTH_UNKNOWN;
+	struct string_list hdrs = STRING_LIST_INIT_NODUP;
+	struct auth_module *mod;
+
+	struct string_list_item *hdr;
+	struct string_list_item *token;
+	const char *v;
+	struct strbuf **split = NULL;
+	int i;
+	char *challenge;
+
+	/*
+	 * Check all auth modules and try to validate the request.
+	 * The first Authorization header that matches a known auth module
+	 * scheme will be consulted to either approve or deny the request.
+	 * If no module is found, or if there is no valid token, then 401 error.
+	 * Otherwise, only permit the request if anonymous auth is enabled.
+	 * It's atypical for user agents/clients to send multiple Authorization
+	 * headers, but not explicitly forbidden or defined.
+	 */
+	for_each_string_list_item(hdr, &req->header_list) {
+		if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+			split = strbuf_split_str(v, ' ', 2);
+			if (split[0] && split[1]) {
+				/* trim trailing space ' ' */
+				strbuf_rtrim(split[0]);
+
+				mod = get_auth_module(split[0]->buf, 0);
+				if (mod) {
+					result = AUTH_DENY;
+
+					for_each_string_list_item(token, mod->tokens) {
+						if (!strcmp(split[1]->buf, token->string)) {
+							result = AUTH_ALLOW;
+							break;
+						}
+					}
+
+					strbuf_list_free(split);
+					goto done;
+				}
+			}
+
+			strbuf_list_free(split);
+		}
+	}
+
+done:
+	switch (result) {
+	case AUTH_ALLOW:
+		trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+		*user = "VALID_TEST_USER";
+		*wr = WR_OK;
+		break;
+
+	case AUTH_DENY:
+		trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+		/* fall-through */
+
+	case AUTH_UNKNOWN:
+		if (result != AUTH_DENY && allow_anonymous)
+			break;
+
+		for (i = 0; i < auth_modules_nr; i++) {
+			mod = auth_modules[i];
+			if (mod->challenge_params)
+				challenge = xstrfmt("WWW-Authenticate: %s %s",
+						    mod->scheme,
+						    mod->challenge_params);
+			else
+				challenge = xstrfmt("WWW-Authenticate: %s",
+						    mod->scheme);
+			string_list_append(&hdrs, challenge);
+		}
+
+		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
+				      &hdrs, *wr);
+	}
+
+	string_list_clear(&hdrs, 0);
+
+	return result == AUTH_ALLOW ||
+	      (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
+static int split_auth_param(const char *str, char **scheme, char **val)
+{
+	struct strbuf **p = strbuf_split_str(str, ':', 2);
+
+	if (!p[0])
+		return -1;
+
+	/* trim trailing ':' */
+	if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
+		strbuf_setlen(p[0], p[0]->len - 1);
+
+	*scheme = strbuf_detach(p[0], NULL);
+	*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
+
+	strbuf_list_free(p);
+	return 0;
+}
+
+static int read_auth_config(const char *name, const char *val, void *data)
+{
+	int ret = 0;
+	char *scheme = NULL;
+	char *token = NULL;
+	char *challenge = NULL;
+	struct auth_module *mod;
+
+	if (!strcmp(name, "auth.challenge")) {
+		if (split_auth_param(val, &scheme, &challenge)) {
+			ret = error("invalid auth challenge '%s'", val);
+			goto cleanup;
+		}
+
+		mod = get_auth_module(scheme, 1);
+
+		/* Replace any existing challenge parameters */
+		free(mod->challenge_params);
+		mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
+	} else if (!strcmp(name, "auth.token")) {
+		if (split_auth_param(val, &scheme, &token)) {
+			ret = error("invalid auth token '%s'", val);
+			goto cleanup;
+		}
+
+		mod = get_auth_module(scheme, 1);
+
+		/*
+		 * Append to set of valid tokens unless an empty token value
+		 * is provided, then clear the existing list.
+		 */
+		if (token)
+			string_list_append(mod->tokens, token);
+		else
+			string_list_clear(mod->tokens, 1);
+	} else if (!strcmp(name, "auth.allowanonymous")) {
+		allow_anonymous = git_config_bool(name, val);
+	} else {
+		warning("unknown auth config '%s'", name);
+	}
+
+cleanup:
+	free(scheme);
+	free(token);
+	free(challenge);
+
+	return ret;
+}
+
 static enum worker_result dispatch(struct req *req)
 {
+	enum worker_result wr = WR_OK;
+	const char *user = NULL;
+
+	if (!is_authed(req, &user, &wr))
+		return wr;
+
 	if (is_git_request(req))
-		return do__git(req);
+		return do__git(req, user);
 
 	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
 			       WR_HANGUP);
@@ -655,6 +870,19 @@ int cmd_main(int argc, const char **argv)
 			pid_file = v;
 			continue;
 		}
+		if (skip_prefix(arg, "--auth-config=", &v)) {
+			if (!strlen(v)) {
+				error("invalid argument - missing file path");
+				usage(test_http_auth_usage);
+			}
+
+			if (git_config_from_file(read_auth_config, v, NULL)) {
+				error("failed to read auth config file '%s'", v);
+				usage(test_http_auth_usage);
+			}
+
+			continue;
+		}
 
 		fprintf(stderr, "error: unknown argument '%s'\n", arg);
 		usage(test_http_auth_usage);
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index c0a47ce342b..20fd9b09aef 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -101,6 +101,7 @@ per_test_cleanup () {
 	stop_http_server &&
 	rm -f OUT.* &&
 	rm -f IN.* &&
+	rm -f auth.config
 }
 
 test_expect_success 'http auth server request parsing' '
@@ -160,11 +161,51 @@ test_expect_success 'http auth server request parsing' '
 	test_cmp OUT.http400 OUT.actual
 '
 
+test_expect_success CURL 'http auth server auth config' '
+	test_when_finished "per_test_cleanup" &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = no-params
+		challenge = with-params:foo=\"bar\" p=1
+		challenge = with-params:foo=\"replaced\" q=1
+
+		token = no-explicit-challenge:valid-token
+		token = no-explicit-challenge:also-valid
+		token = reset-tokens:these-tokens
+		token = reset-tokens:will-be-reset
+		token = reset-tokens:
+		token = reset-tokens:the-only-valid-one
+
+		allowAnonymous = false
+	EOF
+
+	cat >OUT.expected <<-EOF &&
+	WWW-Authenticate: no-params
+	WWW-Authenticate: with-params foo="replaced" q=1
+	WWW-Authenticate: no-explicit-challenge
+	WWW-Authenticate: reset-tokens
+
+	Error: 401 Unauthorized
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	curl --include $ORIGIN_URL >OUT.curl &&
+	tr -d "\r" <OUT.curl | sed -n "/WWW-Authenticate/,\$p" >OUT.actual &&
+
+	test_cmp OUT.expected OUT.actual
+'
 
 test_expect_success 'http auth anonymous no challenge' '
 	test_when_finished "per_test_cleanup" &&
 
-	start_http_server &&
+	cat >auth.config <<-EOF &&
+	[auth]
+		allowAnonymous = true
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
 
 	# Attempt to read from a protected repository
 	git ls-remote $ORIGIN_URL
-- 
gitgitgadget

Re: [PATCH v7 08/12] test-http-server: add simple authentication

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:46PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> +struct auth_module {
> +	char *scheme;
> +	char *challenge_params;
> +	struct string_list *tokens;
> +};

This is a really minor nit, but: why is "tokens" a pointer? It's always
initialized, so you never need or want to test it for NULL.

That would make this:

> +	if (create) {
> +		struct auth_module *mod = xmalloc(sizeof(struct auth_module));
> +		mod->scheme = xstrdup(scheme);
> +		mod->challenge_params = NULL;
> +		ALLOC_ARRAY(mod->tokens, 1);
> +		string_list_init_dup(mod->tokens);

simplify to:

  string_list_init_dup(&mod->tokens);

and one does not have to wonder why we use ALLOC_ARRAY() there, but not
when allocating the module itself. :)

Likewise you could skip freeing it, but since the memory is held until
program end anyway, that doesn't happen either way.

Certainly what you have won't behave wrong; I'd consider this more like
a coding style thing.

> +	cat >auth.config <<-EOF &&
> +	[auth]
> +		challenge = no-params
> +		challenge = with-params:foo=\"bar\" p=1
> +		challenge = with-params:foo=\"replaced\" q=1
> +
> +		token = no-explicit-challenge:valid-token
> +		token = no-explicit-challenge:also-valid
> +		token = reset-tokens:these-tokens
> +		token = reset-tokens:will-be-reset
> +		token = reset-tokens:
> +		token = reset-tokens:the-only-valid-one
> +
> +		allowAnonymous = false
> +	EOF
> +
> +	cat >OUT.expected <<-EOF &&
> +	WWW-Authenticate: no-params
> +	WWW-Authenticate: with-params foo="replaced" q=1
> +	WWW-Authenticate: no-explicit-challenge
> +	WWW-Authenticate: reset-tokens
> +
> +	Error: 401 Unauthorized
> +	EOF

OK, so I think now we are getting to the interesting part of what your
custom http-server does compared to something like apache. And the
answer so far is: custom WWW-Authenticate lines.

I think we could do that with mod_headers pretty easily. But presumably
we also want to check that we are getting the correct tokens, generate a
401, etc.

I suspect this could all be done as a CGI wrapping git-http-backend. You
can influence the HTTP response code by sending:

   Status: 401 Authorization Required
   WWW-Authenticate: whatever you want

And likewise you can see what the client sends by putting something like
this in apache.conf:

   SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1

and then reading $HTTP_AUTHORIZATION as you like. At that point, it
feels like a simple shell or perl script could then decide whether to
return a 401 or not (and if not, then just exec git-http-backend to do
the rest). And the scripts would be simple enough that you could have
individual scripts to implement various schemes, rather than
implementing this configuration scheme. You can control which script is
run based on the URL; see the way we match /broken_smart/, etc, in
t/lib-httpd/apache.conf.

-Peff

Re: [PATCH v7 08/12] test-http-server: add simple authentication

[Jeff King]

On Thu, Jan 26, 2023 at 05:02:27AM -0500, Jeff King wrote:

> I suspect this could all be done as a CGI wrapping git-http-backend. You
> can influence the HTTP response code by sending:
> 
>    Status: 401 Authorization Required
>    WWW-Authenticate: whatever you want
> 
> And likewise you can see what the client sends by putting something like
> this in apache.conf:
> 
>    SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
> 
> and then reading $HTTP_AUTHORIZATION as you like. At that point, it
> feels like a simple shell or perl script could then decide whether to
> return a 401 or not (and if not, then just exec git-http-backend to do
> the rest). And the scripts would be simple enough that you could have
> individual scripts to implement various schemes, rather than
> implementing this configuration scheme. You can control which script is
> run based on the URL; see the way we match /broken_smart/, etc, in
> t/lib-httpd/apache.conf.

And here's a minimally worked-out example of that approach. It's on top
of your patches so I could use your credential-helper infrastructure in
the test, but the intent is that it would replace all of the test-tool
server patches and be rolled into t5556 as appropriate.

---
 t/lib-httpd.sh              |  1 +
 t/lib-httpd/apache.conf     |  6 ++++++
 t/lib-httpd/custom-auth.sh  | 18 ++++++++++++++++
 t/t5563-simple-http-auth.sh | 42 +++++++++++++++++++++++++++++++++++++
 4 files changed, 67 insertions(+)
 create mode 100644 t/lib-httpd/custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index 608949ea80..ab255bdbc5 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -137,6 +137,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
+	install_script custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 0294739a77..4b2256363f 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_HTTP_EXPORT_ALL
 	SetEnv GIT_PROTOCOL
 </LocationMatch>
+<LocationMatch /custom_auth/>
+	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
+	SetEnv GIT_HTTP_EXPORT_ALL
+	CGIPassAuth on
+</LocationMatch>
 ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
 ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
 ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
+ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/custom-auth.sh b/t/lib-httpd/custom-auth.sh
new file mode 100644
index 0000000000..686895ee8c
--- /dev/null
+++ b/t/lib-httpd/custom-auth.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+
+# Our acceptable auth here is hard-coded, but we could
+# read it from a file provided by individual tests, etc.
+#
+# base64("alice:secret-passwd")
+USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+
+case "$HTTP_AUTHORIZATION" in
+"Basic $USERPASS64")
+	exec "$GIT_EXEC_PATH"/git-http-backend
+	;;
+*)
+	echo 'Status: 401 Auth Required'
+	echo 'WWW-Authenticate: Basic realm="whatever"'
+	echo
+	;;
+esac
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
new file mode 100755
index 0000000000..314f9217e6
--- /dev/null
+++ b/t/t5563-simple-http-auth.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+. "$TEST_DIRECTORY"/lib-credential-helper.sh
+
+start_httpd
+
+setup_credential_helper
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+test_expect_success 'access using custom auth' '
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote \
+		"$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="whatever"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_done
-- 
2.39.1.738.g5e5f8a2714

Re: [PATCH v7 08/12] test-http-server: add simple authentication

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> I suspect this could all be done as a CGI wrapping git-http-backend. You
>> can influence the HTTP response code by sending:
> ...
> And here's a minimally worked-out example of that approach. It's on top
> of your patches so I could use your credential-helper infrastructure in
> the test, but the intent is that it would replace all of the test-tool
> server patches and be rolled into t5556 as appropriate.

Thanks for helping Matthew's topic move forward.  I very much like
seeing apache used for tests in this sample approach, like all the
other http tests we do with apache, instead of a custom server that
we need to ensure that it mimics the real-world use cases and we
have to maintain.

Re: [PATCH v7 08/12] test-http-server: add simple authentication

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:46PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> +static int split_auth_param(const char *str, char **scheme, char **val)
> +{
> +	struct strbuf **p = strbuf_split_str(str, ':', 2);
> +
> +	if (!p[0])
> +		return -1;
> +
> +	/* trim trailing ':' */
> +	if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
> +		strbuf_setlen(p[0], p[0]->len - 1);
> +
> +	*scheme = strbuf_detach(p[0], NULL);
> +	*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
> +
> +	strbuf_list_free(p);
> +	return 0;
> +}

Oh, I forgot one more Coverity-detected problem here when reviewing last
night. The early "return -1" here leaks "p" (there are no strbufs in the
resulting array, but strbuf_split_str() will still have allocated the
array). It needs a call to strbuf_list_free(p) there.

-Peff

[PATCH v7 09/12] test-http-server: add sending of arbitrary headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the ability to send arbitrary headers in HTTP responses from the
test-http-server. This is useful when we want to test 'malformed'
response message handling.

Add the following option to the server auth config file:

[auth]
	extraHeader = [<value>]*

Each `auth.extraHeader` value will be appended to the response headers
verbatim.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/helper/test-http-server.c | 6 ++++++
 t/t5556-http-auth.sh        | 7 +++++++
 2 files changed, 13 insertions(+)

diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 72c6cca7e5c..70bf15c3fa1 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -417,6 +417,7 @@ static int allow_anonymous;
 static struct auth_module **auth_modules = NULL;
 static size_t auth_modules_nr = 0;
 static size_t auth_modules_alloc = 0;
+static struct strvec extra_headers = STRVEC_INIT;
 
 static struct auth_module *get_auth_module(const char *scheme, int create)
 {
@@ -520,6 +521,9 @@ done:
 			string_list_append(&hdrs, challenge);
 		}
 
+		for (i = 0; i < extra_headers.nr; i++)
+			string_list_append(&hdrs, extra_headers.v[i]);
+
 		*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
 				      &hdrs, *wr);
 	}
@@ -585,6 +589,8 @@ static int read_auth_config(const char *name, const char *val, void *data)
 			string_list_clear(mod->tokens, 1);
 	} else if (!strcmp(name, "auth.allowanonymous")) {
 		allow_anonymous = git_config_bool(name, val);
+	} else if (!strcmp(name, "auth.extraheader")) {
+		strvec_push(&extra_headers, val);
 	} else {
 		warning("unknown auth config '%s'", name);
 	}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 20fd9b09aef..2c16c8f72a5 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -178,6 +178,10 @@ test_expect_success CURL 'http auth server auth config' '
 		token = reset-tokens:the-only-valid-one
 
 		allowAnonymous = false
+
+		extraHeader = X-Extra-Header: abc
+		extraHeader = X-Extra-Header: 123
+		extraHeader = X-Another: header\twith\twhitespace!
 	EOF
 
 	cat >OUT.expected <<-EOF &&
@@ -185,6 +189,9 @@ test_expect_success CURL 'http auth server auth config' '
 	WWW-Authenticate: with-params foo="replaced" q=1
 	WWW-Authenticate: no-explicit-challenge
 	WWW-Authenticate: reset-tokens
+	X-Extra-Header: abc
+	X-Extra-Header: 123
+	X-Another: header	with	whitespace!
 
 	Error: 401 Unauthorized
 	EOF
-- 
gitgitgadget

[PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Replace direct multiplication of two size_t parameters in curl response
stream handling callback functions with `st_mult` to guard against
overflows.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 http.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/http.c b/http.c
index 8a5ba3f4776..a2a80318bb2 100644
--- a/http.c
+++ b/http.c
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
 
 size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct buffer *buffer = buffer_;
 
 	if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
 
 size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 {
-	size_t size = eltsize * nmemb;
+	size_t size = st_mult(eltsize, nmemb);
 	struct strbuf *buffer = buffer_;
 
 	strbuf_add(buffer, ptr, size);
-- 
gitgitgadget

Re: [PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:48PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> Replace direct multiplication of two size_t parameters in curl response
> stream handling callback functions with `st_mult` to guard against
> overflows.

Hmm. So part of me says that more overflow detection is better than
less, but...I really doubt this is doing anything, and it feels odd to
me to do overflow checks when there is no allocation.

There are tons of integer multiplications in Git. Our usual strategy is
to try to handle overflow like this when we're about to allocate a
buffer, with the idea that we'll avoid a truncated size (that we may
later fill with too many bytes).

In these cases, we could possibly avoid a weird or wrong result due to
truncation, but I don't see how that is different than most of the rest
of Git. What makes these worth touching?

Moreover...

> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
>  
>  size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  {
> -	size_t size = eltsize * nmemb;
> +	size_t size = st_mult(eltsize, nmemb);
>  	struct strbuf *buffer = buffer_;
>  
>  	strbuf_add(buffer, ptr, size);

The caller is already claiming to have eltsize*nmemb bytes accessible
via "ptr". How did it get such a buffer if that overflows size_t?

> diff --git a/http.c b/http.c
> index 8a5ba3f4776..a2a80318bb2 100644
> --- a/http.c
> +++ b/http.c
> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
>  
>  size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  {
> -	size_t size = eltsize * nmemb;
> +	size_t size = st_mult(eltsize, nmemb);
>  	struct buffer *buffer = buffer_;
>  
>  	if (size > buffer->buf.len - buffer->posn)

Likewise the caller is asking us to fill a buffer that is eltsize*nmemb.
So they must have allocated it already. How can it be bigger than a
size_t?

In practice, of course, these are both coming from curl, and I strongly
suspect that curl always sets "1" for eltsize anyway, since it's working
with bytes. The two fields only exist to conform to the weird fread()
interface for historical reasons.

So I don't think this patch is really hurting much. It just feels like a
weird one-off that makes the code inconsistent. If somebody who was
wanting to write similar code later asks "why is this one st_mult(), but
not other multiplications", I wouldn't have an answer for them.

-Peff

[PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c |  1 +
 credential.h | 15 +++++++++
 http.c       | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index a2a80318bb2..595c93bc7a3 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = st_mult(eltsize, nmemb);
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	strbuf_add(&buf, ptr, size);
+
+	/* Strip the CRLF that should be present at the end of each field */
+	strbuf_trim_trailing_newline(&buf);
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+		while (isspace(*val))
+			val++;
+
+		strvec_push(values, val);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 * Continuation lines start with at least one whitespace, maybe more,
+	 * so we should collapse these down to a single SP (valid per the spec).
+	 */
+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+		/* Trim leading whitespace from this continuation hdr line. */
+		strbuf_ltrim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (istarts_with(buf.buf, "http/"))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1956,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:49PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Read and store the HTTP WWW-Authenticate response headers made for
> a particular request.
> 
> This will allow us to pass important authentication challenge
> information to credential helpers or others that would otherwise have
> been lost.

Makes sense, and the code looks pretty reasonable overall.

A few observations:

> @@ -115,6 +116,19 @@ struct credential {
>  	 */
>  	struct string_list helpers;
>  
> +	/**
> +	 * A `strvec` of WWW-Authenticate header values. Each string
> +	 * is the value of a WWW-Authenticate header in an HTTP response,
> +	 * in the order they were received in the response.
> +	 */
> +	struct strvec wwwauth_headers;
> +
> +	/**
> +	 * Internal use only. Used to keep track of split header fields
> +	 * in order to fold multiple lines into one value.
> +	 */
> +	unsigned header_is_last_match:1;
> +

Stuffing this into a "struct credential" feels a little weird, just
because it's specific to http parsing (especially this internal flag).
And the credential code is seeing full header lines, not broken down at
all.

I guess I would have expected some level of abstraction here between the
credential subsystem and the http subsystem, where the latter is parsing
and then sticking opaque data into the credential to ferry to the
helpers.

But it probably isn't that big a deal either way. Even though there are
non-http credentials, it's not too unreasonable for the credential
system to be aware of http specifically.

> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = st_mult(eltsize, nmemb);

Here's that st_mult() again. Same comment as the previous patch. :)

> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	strbuf_add(&buf, ptr, size);

OK, so we just copy the buffer. I don't think it would be too hard to
handle the buffer as-is, but this does make things a bit easier.  Given
that we're going to immediately throw away the copy for anything except
www-authenticate, we could perhaps wait until we've matched it.  That
does mean trimming the CRLF ourselves and using skip_prefix_mem() to
match the start (you'd want skip_iprefix_mem(), of course, but it
doesn't yet exist; I'll leave that as an exercise).

Maybe not worth it to save a few allocations, as an http request is
already pretty heavyweight. Mostly I flagged it because this is going to
run for every header of every request, even though most requests won't
trigger it at all.

> +	/* Strip the CRLF that should be present at the end of each field */
> +	strbuf_trim_trailing_newline(&buf);
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
> +		while (isspace(*val))
> +			val++;
> +
> +		strvec_push(values, val);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> +	}

OK, this looks correct from my knowledge of the RFCs. I saw something
about isspace() matching newlines, etc, in an earlier thread, but I
think we'd never see a newline here, as we're expecting curl to be
splitting on our behalf.

> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 * Continuation lines start with at least one whitespace, maybe more,
> +	 * so we should collapse these down to a single SP (valid per the spec).
> +	 */
> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
> +		/* Trim leading whitespace from this continuation hdr line. */
> +		strbuf_ltrim(&buf);

OK, makes sense. This will memmove(), which is needlessly inefficient
(we could just advance a pointer), but probably not a big deal in
practice. Using the strbuf functions is a nice simplification.

> +		/*
> +		 * At this point we should always have at least one existing
> +		 * value, even if it is empty. Do not bother appending the new
> +		 * value if this continuation header is itself empty.
> +		 */
> +		if (!values->nr) {
> +			BUG("should have at least one existing header value");
> +		} else if (buf.len) {
> +			char *prev = xstrdup(values->v[values->nr - 1]);
> +
> +			/* Join two non-empty values with a single space. */
> +			const char *const sp = *prev ? " " : "";
> +
> +			strvec_pop(values);
> +			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
> +			free(prev);
> +		}

Likewise here we end up with an extra allocation of "prev", just because
we can't pop/push in the right order. But that's probably OK in
practice, as this is triggering only for the header we care about.

The concatenation itself makes the whole thing quadratic, but unless we
are worried about a malicious server DoS-ing us with a billion
www-authenticate continuations, I think we can disregard that.

-Peff

Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2023-01-26 02:31, Jeff King wrote:

> On Fri, Jan 20, 2023 at 10:08:49PM +0000, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Read and store the HTTP WWW-Authenticate response headers made for
>> a particular request.
>>
>> This will allow us to pass important authentication challenge
>> information to credential helpers or others that would otherwise have
>> been lost.
> 
> Makes sense, and the code looks pretty reasonable overall.
> 
> A few observations:
> 
>> @@ -115,6 +116,19 @@ struct credential {
>>  	 */
>>  	struct string_list helpers;
>>  
>> +	/**
>> +	 * A `strvec` of WWW-Authenticate header values. Each string
>> +	 * is the value of a WWW-Authenticate header in an HTTP response,
>> +	 * in the order they were received in the response.
>> +	 */
>> +	struct strvec wwwauth_headers;
>> +
>> +	/**
>> +	 * Internal use only. Used to keep track of split header fields
>> +	 * in order to fold multiple lines into one value.
>> +	 */
>> +	unsigned header_is_last_match:1;
>> +
> 
> Stuffing this into a "struct credential" feels a little weird, just
> because it's specific to http parsing (especially this internal flag).
> And the credential code is seeing full header lines, not broken down at
> all.
> 
> I guess I would have expected some level of abstraction here between the
> credential subsystem and the http subsystem, where the latter is parsing
> and then sticking opaque data into the credential to ferry to the
> helpers.
> 
> But it probably isn't that big a deal either way. Even though there are
> non-http credentials, it's not too unreasonable for the credential
> system to be aware of http specifically.

I had considered possibly introducing an opaque property-bag style of
'protocol-specific properties' that, for example, http.c would add the
WWW-Authenticate headers to as something like `http.wwwauth[]`.
Other protocols (like smtp:// or cert://) could add their own properties
if they needed or wanted to also.

Thoughts?

>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> +{
>> +	size_t size = st_mult(eltsize, nmemb);
> 
> Here's that st_mult() again. Same comment as the previous patch. :)

Yeah I'm gonna drop this. Your arguments make sense; it's not going to be a
problem in reality :-)

>> +	/*
>> +	 * Header lines may not come NULL-terminated from libcurl so we must
>> +	 * limit all scans to the maximum length of the header line, or leverage
>> +	 * strbufs for all operations.
>> +	 *
>> +	 * In addition, it is possible that header values can be split over
>> +	 * multiple lines as per RFC 2616 (even though this has since been
>> +	 * deprecated in RFC 7230). A continuation header field value is
>> +	 * identified as starting with a space or horizontal tab.
>> +	 *
>> +	 * The formal definition of a header field as given in RFC 2616 is:
>> +	 *
>> +	 *   message-header = field-name ":" [ field-value ]
>> +	 *   field-name     = token
>> +	 *   field-value    = *( field-content | LWS )
>> +	 *   field-content  = <the OCTETs making up the field-value
>> +	 *                    and consisting of either *TEXT or combinations
>> +	 *                    of token, separators, and quoted-string>
>> +	 */
>> +
>> +	strbuf_add(&buf, ptr, size);
> 
> OK, so we just copy the buffer. I don't think it would be too hard to
> handle the buffer as-is, but this does make things a bit easier.  Given
> that we're going to immediately throw away the copy for anything except
> www-authenticate, we could perhaps wait until we've matched it.  That
> does mean trimming the CRLF ourselves and using skip_prefix_mem() to
> match the start (you'd want skip_iprefix_mem(), of course, but it
> doesn't yet exist; I'll leave that as an exercise).

Fair point! I can replace most of these with operations over the curl ptr.

> Maybe not worth it to save a few allocations, as an http request is
> already pretty heavyweight. Mostly I flagged it because this is going to
> run for every header of every request, even though most requests won't
> trigger it at all.
> 
>> +	/* Strip the CRLF that should be present at the end of each field */
>> +	strbuf_trim_trailing_newline(&buf);
>> +
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> +		while (isspace(*val))
>> +			val++;
>> +
>> +		strvec_push(values, val);
>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
>> +	}
> 
> OK, this looks correct from my knowledge of the RFCs. I saw something
> about isspace() matching newlines, etc, in an earlier thread, but I
> think we'd never see a newline here, as we're expecting curl to be
> splitting on our behalf.
> 
>> +	/*
>> +	 * This line could be a continuation of the previously matched header
>> +	 * field. If this is the case then we should append this value to the
>> +	 * end of the previously consumed value.
>> +	 * Continuation lines start with at least one whitespace, maybe more,
>> +	 * so we should collapse these down to a single SP (valid per the spec).
>> +	 */
>> +	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> +		/* Trim leading whitespace from this continuation hdr line. */
>> +		strbuf_ltrim(&buf);
> 
> OK, makes sense. This will memmove(), which is needlessly inefficient
> (we could just advance a pointer), but probably not a big deal in
> practice. Using the strbuf functions is a nice simplification.
> 
>> +		/*
>> +		 * At this point we should always have at least one existing
>> +		 * value, even if it is empty. Do not bother appending the new
>> +		 * value if this continuation header is itself empty.
>> +		 */
>> +		if (!values->nr) {
>> +			BUG("should have at least one existing header value");
>> +		} else if (buf.len) {
>> +			char *prev = xstrdup(values->v[values->nr - 1]);
>> +
>> +			/* Join two non-empty values with a single space. */
>> +			const char *const sp = *prev ? " " : "";
>> +
>> +			strvec_pop(values);
>> +			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
>> +			free(prev);
>> +		}
> 
> Likewise here we end up with an extra allocation of "prev", just because
> we can't pop/push in the right order. But that's probably OK in
> practice, as this is triggering only for the header we care about.
> 
> The concatenation itself makes the whole thing quadratic, but unless we
> are worried about a malicious server DoS-ing us with a billion
> www-authenticate continuations, I think we can disregard that.
> 
> -Peff

Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers

[Jeff King]

On Mon, Feb 06, 2023 at 11:25:49AM -0800, Matthew John Cheetham wrote:

> > I guess I would have expected some level of abstraction here between the
> > credential subsystem and the http subsystem, where the latter is parsing
> > and then sticking opaque data into the credential to ferry to the
> > helpers.
> > 
> > But it probably isn't that big a deal either way. Even though there are
> > non-http credentials, it's not too unreasonable for the credential
> > system to be aware of http specifically.
> 
> I had considered possibly introducing an opaque property-bag style of
> 'protocol-specific properties' that, for example, http.c would add the
> WWW-Authenticate headers to as something like `http.wwwauth[]`.
> Other protocols (like smtp:// or cert://) could add their own properties
> if they needed or wanted to also.
> 
> Thoughts?

At the protocol level, I don't see much point. wwwauth sufficiently
implies "http", and any helper is free to ignore or respect keys as
appropriate to what it can handle. A flat namespace is fine.

Here I was more talking about the internal implementation. Mostly it was
just funky that this internal http state flag was stuck into the
credential struct. I think it could be removed with some minor pain, but
it's probably not too big a deal (the pain at all is only because we are
having to bring this state across multiple curl callbacks).

So let's go with it for now.

-Peff

[PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |  11 ++
 t/lib-credential-helper.sh       |  27 ++++
 t/t5556-http-auth.sh             | 242 +++++++++++++++++++++++++++++++
 4 files changed, 298 insertions(+), 1 deletion(-)
 create mode 100644 t/lib-credential-helper.sh

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..9f39ebc3c7e 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	char *full_key = xstrfmt("%s[]", key);
+	for (size_t i = 0; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free(full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh
new file mode 100644
index 00000000000..8b0e4414234
--- /dev/null
+++ b/t/lib-credential-helper.sh
@@ -0,0 +1,27 @@
+setup_credential_helper() {
+	test_expect_success 'setup credential helper' '
+		CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
+		export CREDENTIAL_HELPER &&
+		echo $CREDENTIAL_HELPER &&
+
+		write_script "$CREDENTIAL_HELPER" <<-\EOF
+		cmd=$1
+		teefile=$cmd-query.cred
+		catfile=$cmd-reply.cred
+		sed -n -e "/^$/q" -e "p" >> $teefile
+		if test "$cmd" = "get"; then
+			cat $catfile
+		fi
+		EOF
+	'
+}
+
+set_credential_reply() {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query() {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 2c16c8f72a5..93b7c178da6 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop'
 
 TEST_NO_CREATE_REPO=1
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-credential-helper.sh
 
 test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
 
@@ -33,6 +34,8 @@ test_expect_success 'setup repos' '
 	git -C "$REPO_DIR" branch -M main
 '
 
+setup_credential_helper
+
 run_http_server_worker() {
 	(
 		cd "$REPO_DIR"
@@ -101,6 +104,7 @@ per_test_cleanup () {
 	stop_http_server &&
 	rm -f OUT.* &&
 	rm -f IN.* &&
+	rm -f *.cred &&
 	rm -f auth.config
 }
 
@@ -218,4 +222,242 @@ test_expect_success 'http auth anonymous no challenge' '
 	git ls-remote $ORIGIN_URL
 '
 
+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+		extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bEaRer auThoRiTy="id.example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+		extraheader = "WWW-Authenticate:"
+		extraheader = " "
+		extraheader = " bearer authority=\"id.example.com\""
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=basic realm="example.com"
+	wwwauth[]=bearer authority="id.example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "foobar:alg=test widget=1"
+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+
+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=foobar alg=test widget=1
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
+	test_when_finished "per_test_cleanup" &&
+	# base64("alice:secret-passwd")
+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+	export USERPASS64 &&
+
+	cat >auth.config <<-EOF &&
+	[auth]
+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
+		challenge = basic:realm=\"example.com\"
+		token = basic:$USERPASS64
+	EOF
+
+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+	set_credential_reply get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	EOF
+
+	test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HOST_PORT
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HOST_PORT
+	username=alice
+	password=invalid-passwd
+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
+	wwwauth[]=basic realm="example.com"
+	EOF
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests

[Jeff King]

On Fri, Jan 20, 2023 at 10:08:50PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
> Add the value of the WWW-Authenticate response header to credential
> requests. Credential helpers that understand and support HTTP
> authentication and authorization can use this standard header (RFC 2616
> Section 14.47 [1]) to generate valid credentials.
> 
> WWW-Authenticate headers can contain information pertaining to the
> authority, authentication mechanism, or extra parameters/scopes that are
> required.

I'm definitely on board with sending these to the helpers. It does feel
a bit weird that we don't parse them at all, and just foist that on the
helpers.

If I understand the RFC correctly, you can have multiple challenges per
header, but also multiple headers. So:

  WWW-Authenticate: Basic realm="foo", OtherAuth realm="bar"
  WWW-Authenticate: YetAnotherScheme some-token

could be normalized as:

  www-auth-challenge=Basic realm="foo"
  www-auth-challenge=OtherAuth realm="bar"
  www-auth-challenge=YetAnotherScheme some-token

which saves each helper from having to do the same work. Likewise, we
can do a _little_ more parsing to get:

  www-auth-basic=realm="foo"
  www-auth-otherauth=realm="bar"
  www-auth-yetanotherscheme=some-token

I don't think we can go beyond there, though, without understanding the
syntax of individual schemes. Which is a shame, as one of the goals of
the credential format was to let the helpers do as little as possible
(so they can't get it wrong!). But helpers are stuck doing things like
handling backslashed double-quotes, soaking up extra whitespace, etc.

I'm not really sure what we expect to see in the real world. I guess for
your purposes, you are working on an already-big helper that is happy to
just get the raw values and process them according to the rfc. I'm just
wondering if there are use cases where somebody might want to do
something with this header, but in a quick shell script kind of way. For
example, my credential config is still:

  [credential "https://github.com"]
  username = peff
  helper = "!f() { test $1 = get && echo password=$(pass ...); }; f"

That's an extreme example, but I'm wondering if there's _anything_
useful somebody would want to do in a similar quick-and-dirty kind of
way. For example, deciding which cred to use based on basic realm, like:

  realm=foo
  while read line; do
    case "$line" in
    www-auth-basic=)
        value=${line#*=}
	# oops, we're just assuming it's realm= here, and we're
	# not handling quotes at all. I think it could technically be
	# realm=foo or realm="foo"
	realm=${value#realm=}
	;;
    esac
  done
  echo password=$(pass "pats-by-realm/$realm")

which could be made a lot easier if we did more parsing (e.g.,
www-auth-basic-realm or something). I dunno. Maybe that is just opening
up a can of worms, as we're stuffing structured data into a linearized
key-value list. The nice thing about your proposal is that Git does not
even have to know anything about these schemes; it's all the problem of
the helper. My biggest fear is just that we'll want to shift that later,
and we'll be stuck with this microformat forever.

> The current I/O format for credential helpers only allows for unique
> names for properties/attributes, so in order to transmit multiple header
> values (with a specific order) we introduce a new convention whereby a
> C-style array syntax is used in the property name to denote multiple
> ordered values for the same property.

I don't know if this is strictly necessary. The semantics of duplicate
keys are not really defined anywhere, and just because the
implementations of current readers happen to replace duplicates for the
current set of keys doesn't mean everything has to. So you could just
define "wwwauth" to behave differently. But I don't mind having a
syntactic marker to indicate this new type.

If you're at all convinced by what I said above, then we also might be
able to get away with having unique keys anyway.

>  Documentation/git-credential.txt |  19 ++-
>  credential.c                     |  11 ++
>  t/lib-credential-helper.sh       |  27 ++++
>  t/t5556-http-auth.sh             | 242 +++++++++++++++++++++++++++++++
>  4 files changed, 298 insertions(+), 1 deletion(-)
>  create mode 100644 t/lib-credential-helper.sh

The patch itself looks pretty reasonable to me.

One small thing I noticed:

> +	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&

As you undoubtedly figured out, the helper path is fed to the shell, so
spaces in the trash directory are a problem. You've solved it here by
adding a layer of double quotes, which handles spaces. But you'd run
into problems if the absolute path that somebody is using for the test
suite has a backslash or a double quote in it.

I don't know how careful we want to be here (or how careful we already
are[1]), but one simple-ish solution is:

  export CREDENTIAL_HELPER
  git -c "credential.helper=!\"\$CREDENTIAL_HELPER\"" ...

I.e., letting the inner shell expand the variable itself. Another option
is to put the helper into $TRASH_DIRECTORY/bin and add that to the
$PATH.

I also wondered if it was worth having setup_credential_helper() just
stick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't
have to keep doing that ugly "-c" invocation. Or if you really want to
have each test enable it, perhaps have set_credential_reply() turn it on
via test_config (which will auto-remove it at the end of the test).

-Peff

[1] Curious, I tried cloning git into this directory:

      mkdir '/tmp/foo/"horrible \"path\"'

    and we do indeed already fail. The first breakage I saw was recent,
    but going further back, it looks like bin-wrappers don't correctly
    handle this case anyway. So maybe that's evidence that nobody would
    do something so ridiculous in practice.

Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2023-01-26 03:25, Jeff King wrote:

> On Fri, Jan 20, 2023 at 10:08:50PM +0000, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add the value of the WWW-Authenticate response header to credential
>> requests. Credential helpers that understand and support HTTP
>> authentication and authorization can use this standard header (RFC 2616
>> Section 14.47 [1]) to generate valid credentials.
>>
>> WWW-Authenticate headers can contain information pertaining to the
>> authority, authentication mechanism, or extra parameters/scopes that are
>> required.
> 
> I'm definitely on board with sending these to the helpers. It does feel
> a bit weird that we don't parse them at all, and just foist that on the
> helpers.
> 
> If I understand the RFC correctly, you can have multiple challenges per
> header, but also multiple headers. So:
> 
>   WWW-Authenticate: Basic realm="foo", OtherAuth realm="bar"
>   WWW-Authenticate: YetAnotherScheme some-token

That is correct. It would be strange that server would respond with a mix
of styles, but I guess it's not forbidden.

> could be normalized as:
> 
>   www-auth-challenge=Basic realm="foo"
>   www-auth-challenge=OtherAuth realm="bar"
>   www-auth-challenge=YetAnotherScheme some-token
> 
> which saves each helper from having to do the same work. Likewise, we
> can do a _little_ more parsing to get:
> 
>   www-auth-basic=realm="foo"
>   www-auth-otherauth=realm="bar"
>   www-auth-yetanotherscheme=some-token
> 
> I don't think we can go beyond there, though, without understanding the
> syntax of individual schemes. Which is a shame, as one of the goals of
> the credential format was to let the helpers do as little as possible
> (so they can't get it wrong!). But helpers are stuck doing things like
> handling backslashed double-quotes, soaking up extra whitespace, etc.

This key format wouldn't make it obviously easier for simple helpers to
understand. Now they no longer have well-known keys but a key prefix.

My overall goal here is to have Git know less about auth, so it treats
all values as totally opaque. The only logic added is around reconstructing
folded headers, which is just HTTP and not auth specific.

> I'm not really sure what we expect to see in the real world. I guess for
> your purposes, you are working on an already-big helper that is happy to
> just get the raw values and process them according to the rfc. I'm just
> wondering if there are use cases where somebody might want to do
> something with this header, but in a quick shell script kind of way. For
> example, my credential config is still:
> 
>   [credential "https://github.com"]
>   username = peff
>   helper = "!f() { test $1 = get && echo password=$(pass ...); }; f"
> 
> That's an extreme example, but I'm wondering if there's _anything_
> useful somebody would want to do in a similar quick-and-dirty kind of
> way. For example, deciding which cred to use based on basic realm, like:
> 
>   realm=foo
>   while read line; do
>     case "$line" in
>     www-auth-basic=)
>         value=${line#*=}
> 	# oops, we're just assuming it's realm= here, and we're
> 	# not handling quotes at all. I think it could technically be
> 	# realm=foo or realm="foo"
> 	realm=${value#realm=}
> 	;;
>     esac
>   done
>   echo password=$(pass "pats-by-realm/$realm")
> 
> which could be made a lot easier if we did more parsing (e.g.,
> www-auth-basic-realm or something). I dunno. Maybe that is just opening
> up a can of worms, as we're stuffing structured data into a linearized
> key-value list. The nice thing about your proposal is that Git does not
> even have to know anything about these schemes; it's all the problem of
> the helper. My biggest fear is just that we'll want to shift that later,
> and we'll be stuck with this microformat forever.

I'm not sure there's such a continuous scale between simple and 'complex'
helpers that would mean there'd be a simple shell script generating
OAuth or DPoP credentials instead of a helper written in a higher-level
language where parsing the headers is one of the simpler challenges faced.

>> The current I/O format for credential helpers only allows for unique
>> names for properties/attributes, so in order to transmit multiple header
>> values (with a specific order) we introduce a new convention whereby a
>> C-style array syntax is used in the property name to denote multiple
>> ordered values for the same property.
> 
> I don't know if this is strictly necessary. The semantics of duplicate
> keys are not really defined anywhere, and just because the
> implementations of current readers happen to replace duplicates for the
> current set of keys doesn't mean everything has to. So you could just
> define "wwwauth" to behave differently. But I don't mind having a
> syntactic marker to indicate this new type.

I had considered another model whereby we forgo the key=value line model,
and hide another format behind the 'final' terminating new-line. However
I thought this would be even more distuptive.

> If you're at all convinced by what I said above, then we also might be
> able to get away with having unique keys anyway.
> 
>>  Documentation/git-credential.txt |  19 ++-
>>  credential.c                     |  11 ++
>>  t/lib-credential-helper.sh       |  27 ++++
>>  t/t5556-http-auth.sh             | 242 +++++++++++++++++++++++++++++++
>>  4 files changed, 298 insertions(+), 1 deletion(-)
>>  create mode 100644 t/lib-credential-helper.sh
> 
> The patch itself looks pretty reasonable to me.
> 
> One small thing I noticed:
> 
>> +	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
> 
> As you undoubtedly figured out, the helper path is fed to the shell, so
> spaces in the trash directory are a problem. You've solved it here by
> adding a layer of double quotes, which handles spaces. But you'd run
> into problems if the absolute path that somebody is using for the test
> suite has a backslash or a double quote in it.
> 
> I don't know how careful we want to be here (or how careful we already
> are[1]), but one simple-ish solution is:
> 
>   export CREDENTIAL_HELPER
>   git -c "credential.helper=!\"\$CREDENTIAL_HELPER\"" ...
> 
> I.e., letting the inner shell expand the variable itself. Another option
> is to put the helper into $TRASH_DIRECTORY/bin and add that to the
> $PATH.
> 
> I also wondered if it was worth having setup_credential_helper() just
> stick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't
> have to keep doing that ugly "-c" invocation. Or if you really want to
> have each test enable it, perhaps have set_credential_reply() turn it on
> via test_config (which will auto-remove it at the end of the test).

Good ideas! I shall try those.

> -Peff
> 
> [1] Curious, I tried cloning git into this directory:
> 
>       mkdir '/tmp/foo/"horrible \"path\"'
> 
>     and we do indeed already fail. The first breakage I saw was recent,
>     but going further back, it looks like bin-wrappers don't correctly
>     handle this case anyway. So maybe that's evidence that nobody would
>     do something so ridiculous in practice.

Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests

[Jeff King]

On Mon, Feb 06, 2023 at 11:18:03AM -0800, Matthew John Cheetham wrote:

> > could be normalized as:
> > 
> >   www-auth-challenge=Basic realm="foo"
> >   www-auth-challenge=OtherAuth realm="bar"
> >   www-auth-challenge=YetAnotherScheme some-token
> > 
> > which saves each helper from having to do the same work. Likewise, we
> > can do a _little_ more parsing to get:
> > 
> >   www-auth-basic=realm="foo"
> >   www-auth-otherauth=realm="bar"
> >   www-auth-yetanotherscheme=some-token
> > 
> > I don't think we can go beyond there, though, without understanding the
> > syntax of individual schemes. Which is a shame, as one of the goals of
> > the credential format was to let the helpers do as little as possible
> > (so they can't get it wrong!). But helpers are stuck doing things like
> > handling backslashed double-quotes, soaking up extra whitespace, etc.
> 
> This key format wouldn't make it obviously easier for simple helpers to
> understand. Now they no longer have well-known keys but a key prefix.

Yes, though I don't think that's particularly complicated to parse.
Either way we're just flattening a tuple of (a, b, c) from "a=b c" to
"a-b=c". The value is in normalizing the syntax, so that helpers don't
have to deal with both "a=b c d e" and ("a=b c", "a=d e") themselves.

Another way to do that normalization would be to have Git convert:

  WWW-Authenticate: Basic realm="foo" OtherAuth realm="bar"

into:

 WWW-Authenticate: Basic realm="foo"
 WWW-Authenticate: OtherAuth realm="bar"

which then becomes (at the credential level):

  www-auth[]=Basic realm="foo"
  www-auth[]=OtherAuth realm="bar"

And likewise to normalize whitespace, etc, so each individual helper
doesn't have to (or risk getting confused/exploited). That said...

> My overall goal here is to have Git know less about auth, so it treats
> all values as totally opaque. The only logic added is around reconstructing
> folded headers, which is just HTTP and not auth specific.

Yeah, in general I agree with the notion that Git is mostly just passing
around opaque tokens. We do have to understand some syntax (like
folding!) at the HTTP level, so I think some syntactic normalization /
simplification is reasonable.

BUT. I think you are right that embedding it into the schema of the
helper protocol is probably bad. If the point is that the two forms of
my Basic / OtherAuth example are semantically equivalent, then we can
always decide later to convert between one and the other as a favor to
helpers. Whereas baking it into the schema is a promise for Git to
always parse and understand the headers.

So let me retract my suggestion, and we can leave "maybe normalize
headers to save helpers some work" as a possible topic for later (if
indeed it ever even becomes a problem in practice).

> >   realm=foo
> >   while read line; do
> >     case "$line" in
> >     www-auth-basic=)
> >         value=${line#*=}
> > 	# oops, we're just assuming it's realm= here, and we're
> > 	# not handling quotes at all. I think it could technically be
> > 	# realm=foo or realm="foo"
> > 	realm=${value#realm=}
> > 	;;
> >     esac
> >   done
> >   echo password=$(pass "pats-by-realm/$realm")
> > 
> > which could be made a lot easier if we did more parsing (e.g.,
> > www-auth-basic-realm or something). I dunno. Maybe that is just opening
> > up a can of worms, as we're stuffing structured data into a linearized
> > key-value list. The nice thing about your proposal is that Git does not
> > even have to know anything about these schemes; it's all the problem of
> > the helper. My biggest fear is just that we'll want to shift that later,
> > and we'll be stuck with this microformat forever.
> 
> I'm not sure there's such a continuous scale between simple and 'complex'
> helpers that would mean there'd be a simple shell script generating
> OAuth or DPoP credentials instead of a helper written in a higher-level
> language where parsing the headers is one of the simpler challenges faced.

For the most part, yeah. I tried to form the above example as something
that was really just relying on "basic", but taking in more information
/ context than we currently provide (and that your patch would provide).
I admit it's a stretch, though. Are there any servers which actually use
a Basic realm to distinguish between two credential's you'd want to
provide? I don't think I've seen one.

(Not to mention that people scripting helpers like this is probably
pretty rare; I do, but you can probably consider me a special case. And
if things got more complicated I'd just turn to Perl anyway. ;) ).

> I had considered another model whereby we forgo the key=value line model,
> and hide another format behind the 'final' terminating new-line. However
> I thought this would be even more distuptive.

Yeah, if we can shoe-horn this into the existing key/value model, that's
much better. The original intent with the final newline is that you
could read multiple credentials in a list, though in the end I don't
recall that we ever used that feature anyway.

-Peff

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> Updates in v6
> =============
> 
>  * Clarify the change to make logging optional in the check_dead_children()
>    function during libification of daemon.c.
> 
>  * Fix missing pointer dereference bugs identified in libification of child
>    process handling functions for daemon.c.
> 
>  * Add doc comments to child process handling function declarations in the
>    daemon-utils.h header.
> 
>  * Align function parameter names with variable names at callsites for
>    libified daemon functions.
> 
>  * Re-split out the test-http-server test helper commits in to smaller
>    patches: error response handling, request parsing, http-backend
>    pass-through, simple authentication, arbitrary header support.
> 
>  * Call out auth configuration file format for test-http-server test helper
>    and supported options in commit messages, as well as a test to exercise
>    and demonstrate these options.
> 
>  * Permit auth.token and auth.challenge to appear in any order; create the
>    struct auth_module just-in-time as options for that scheme are read. This
>    simplifies the configuration authoring of the test-http-server test
>    helper.
> 
>  * Update tests to use auth.allowAnoymous in the patch that introduces the
>    new test helper option.
> 
>  * Drop the strvec_push_nodup() commit and update the implementation of HTTP
>    request header line folding to use xstrdup and strvec_pop and _pushf.
> 
>  * Use size_t instead of int in credential.c when iterating over the struct
>    strvec credential members. Also drop the not required const and cast from
>    the full_key definition and free.
> 
>  * Replace in-tree test-credential-helper-reply.sh test cred helper script
>    with the lib-credential-helper.sh reusable 'lib' test script and shell
>    functions to configure the helper behaviour.
> 
>  * Leverage sed over the while read $line loop in the test credential helper
>    script.
> 
> 
> Updates in v7
> =============
> 
>  * Address several whitespace and arg/param list alignment issues.
> 
>  * Rethink the test-http-helper worker-mode error and result enum to be more
>    simple and more informative to the nature of the error.
> 
>  * Use uintmax_t to store the Content-Length of a request in the helper
>    test-http-server. Maintain a bit flag to store if we received such a
>    header.
> 
>  * Return a "400 Bad Request" HTTP response if we fail to parse the request
>    in the test-http-server.
> 
>  * Add test case to cover request message parsing in test-http-server.
> 
>  * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
>    get_auth_module.
> 
>  * Correctly free the split strbufs created in the header parsing loop in
>    test-http-server.
> 
>  * Avoid needless comparison > 0 for unsigned types.
> 
>  * Always set optional outputs to NULL if not present in test helper config
>    value handling.
> 
>  * Remove an accidentally commented-out test cleanup line for one test case
>    in t5556.
I've re-read the patches in this version; all of my comments from v5 have
been addressed, and the additional updates w.r.t. other reviewer feedback
all look good as well. At this point, I think the series is ready for
'next'.

Thanks!

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Junio C Hamano]

Victoria Dye <vdye@github.com> writes:

> Matthew John Cheetham via GitGitGadget wrote:
>> Updates in v6
>> =============
>> ...
> I've re-read the patches in this version; all of my comments from v5 have
> been addressed, and the additional updates w.r.t. other reviewer feedback
> all look good as well. At this point, I think the series is ready for
> 'next'.
>
> Thanks!

Thanks, both.  Let's merge it down.

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Jeff King]

On Tue, Jan 24, 2023 at 10:03:02AM -0800, Junio C Hamano wrote:

> Victoria Dye <vdye@github.com> writes:
> 
> > Matthew John Cheetham via GitGitGadget wrote:
> >> Updates in v6
> >> =============
> >> ...
> > I've re-read the patches in this version; all of my comments from v5 have
> > been addressed, and the additional updates w.r.t. other reviewer feedback
> > all look good as well. At this point, I think the series is ready for
> > 'next'.
> >
> > Thanks!
> 
> Thanks, both.  Let's merge it down.

Sorry, I'm a bit late to the party, but I left some comments just now
(this topic had been on my review backlog for ages, but I never quite
got to it).

Many of my comments were small bits that could be fixed on top (tiny
leaks, etc). But some of my comments were of the form "no, do it totally
differently". It may simply be too late for those ones, but let's see if
Matthew finds anything compelling in them.

-Peff

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> Thanks, both.  Let's merge it down.
>
> Sorry, I'm a bit late to the party, but I left some comments just now
> (this topic had been on my review backlog for ages, but I never quite
> got to it).
>
> Many of my comments were small bits that could be fixed on top (tiny
> leaks, etc). But some of my comments were of the form "no, do it totally
> differently". It may simply be too late for those ones, but let's see if
> Matthew finds anything compelling in them.

I do not mind reverting the merge to 'next' to have an improved
version.  Your "do we really want to add a custom server based on
questionable codebase whose quality as a test-bed for real world
usage is dubious?" is a valid concern.

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Johannes Schindelin]

Hi Junio & Peff,

On Thu, 26 Jan 2023, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
>
> >> Thanks, both.  Let's merge it down.
> >
> > Sorry, I'm a bit late to the party, but I left some comments just now
> > (this topic had been on my review backlog for ages, but I never quite
> > got to it).
> >
> > Many of my comments were small bits that could be fixed on top (tiny
> > leaks, etc). But some of my comments were of the form "no, do it totally
> > differently". It may simply be too late for those ones, but let's see if
> > Matthew finds anything compelling in them.
>
> I do not mind reverting the merge to 'next' to have an improved
> version.  Your "do we really want to add a custom server based on
> questionable codebase whose quality as a test-bed for real world
> usage is dubious?" is a valid concern.

Except.

Except that this code base would have made for a fine base to potentially
implement an HTTPS-based replacement for the aging and insecure
git-daemon.

That code base (which is hardly as questionable codebase as you make it
sound because it has been in use for years in a slightly different form)
would have had the opportunity to mature in a relatively safe environment:
our test suite. And eventually, once robust enough, it could have been
extended to allow for easy and painless yet secure ad-hoc serving of Git
repositories, addressing the security concerns around git-daemon.

And now that we're throwing out that code we don't have that opportunity,
making the goal to deprecate the git-daemon and replace it by something
that is as easy to set up but talks HTTPS instead much, much harder to
reach.

In addition, it causes a loss of test coverage because Apache is not
available in all the setups where the "questionable" code would have had
no problem being built and validating the credential code.

Windows, for example, will now go completely uncovered in CI regarding the
new code.

Ciao,
Johannes

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Ævar Arnfjörð Bjarmason]

On Thu, Feb 02 2023, Johannes Schindelin wrote:

> Hi Junio & Peff,
>
> On Thu, 26 Jan 2023, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>>
>> >> Thanks, both.  Let's merge it down.
>> >
>> > Sorry, I'm a bit late to the party, but I left some comments just now
>> > (this topic had been on my review backlog for ages, but I never quite
>> > got to it).
>> >
>> > Many of my comments were small bits that could be fixed on top (tiny
>> > leaks, etc). But some of my comments were of the form "no, do it totally
>> > differently". It may simply be too late for those ones, but let's see if
>> > Matthew finds anything compelling in them.
>>
>> I do not mind reverting the merge to 'next' to have an improved
>> version.  Your "do we really want to add a custom server based on
>> questionable codebase whose quality as a test-bed for real world
>> usage is dubious?" is a valid concern.
>
> Except.
>
> Except that this code base would have made for a fine base to potentially
> implement an HTTPS-based replacement for the aging and insecure
> git-daemon.
>
> That code base (which is hardly as questionable codebase as you make it
> sound because it has been in use for years in a slightly different form)
> would have had the opportunity to mature in a relatively safe environment:
> our test suite. And eventually, once robust enough, it could have been
> extended to allow for easy and painless yet secure ad-hoc serving of Git
> repositories, addressing the security concerns around git-daemon.
>
> And now that we're throwing out that code we don't have that opportunity,
> making the goal to deprecate the git-daemon and replace it by something
> that is as easy to set up but talks HTTPS instead much, much harder to
> reach.

There's many reasons for why you almost never see a git:// URL in the
wild anymore.

But if "easy and painless" was synonymous with "built with git" or
"ships with git" as you seem to be using it, surely it would be more
common than doing the same with http or https, which requires an
external server?

So, easy for whom? Just us and our own test suite?

Having read both your reply & Jeff's[1] I don't think you're addressing
the thrust of his argument.

You can share all those goals without the method of getting there
requiring us to start maintaining our own webserver.

> In addition, it causes a loss of test coverage because Apache is not
> available in all the setups where the "questionable" code would have had
> no problem being built and validating the credential code.
>
> Windows, for example, will now go completely uncovered in CI regarding the
> new code.

I have not set up Apache on Windows, but binaries seem to be available
for it[2]. We don't use those now, but is downloading, setting up &
running them in CI really harder than emarking on a project of
maintaining our own webserver, especially we've got an eye towards
non-test suite use?

Even if we think that we'd like to have a webserver built when you "make
git" I don't see why we'd go the NIH route of writing our own. Unlike
the git:// protocol there's a *lot* of implementations of http(s)://
servers.

If we think apache is too heavyweight for whatever reason, can't we add
one of the many light http servers out there to contrib/ use it it from
there?

1. https://lore.kernel.org/git/Y9JA0UCRh7qUqKQI@coredump.intra.peff.net/
2. https://httpd.apache.org/docs/2.4/platform/windows.html

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 2620 bytes --]

Hi Ævar,

On Thu, 2 Feb 2023, Ævar Arnfjörð Bjarmason wrote:

> On Thu, Feb 02 2023, Johannes Schindelin wrote:
>
> > On Thu, 26 Jan 2023, Junio C Hamano wrote:
> >
> >> Jeff King <peff@peff.net> writes:
> >>
> >> >> Thanks, both.  Let's merge it down.
> >> >
> >> > Sorry, I'm a bit late to the party, but I left some comments just now
> >> > (this topic had been on my review backlog for ages, but I never quite
> >> > got to it).
> >> >
> >> > Many of my comments were small bits that could be fixed on top (tiny
> >> > leaks, etc). But some of my comments were of the form "no, do it totally
> >> > differently". It may simply be too late for those ones, but let's see if
> >> > Matthew finds anything compelling in them.
> >>
> >> I do not mind reverting the merge to 'next' to have an improved
> >> version.  Your "do we really want to add a custom server based on
> >> questionable codebase whose quality as a test-bed for real world
> >> usage is dubious?" is a valid concern.
> >
> > Except.
> >
> > Except that this code base would have made for a fine base to potentially
> > implement an HTTPS-based replacement for the aging and insecure
> > git-daemon.
> >
> > That code base (which is hardly as questionable codebase as you make it
> > sound because it has been in use for years in a slightly different form)
> > would have had the opportunity to mature in a relatively safe environment:
> > our test suite. And eventually, once robust enough, it could have been
> > extended to allow for easy and painless yet secure ad-hoc serving of Git
> > repositories, addressing the security concerns around git-daemon.
> >
> > And now that we're throwing out that code we don't have that opportunity,
> > making the goal to deprecate the git-daemon and replace it by something
> > that is as easy to set up but talks HTTPS instead much, much harder to
> > reach.
>
> There's many reasons for why you almost never see a git:// URL in the
> wild anymore.

I am unwilling to accept that statement without any source to back it up.
Thin air is no substitute for reliable evidence.

> But if "easy and painless" was synonymous with "built with git" or
> "ships with git" as you seem to be using it, surely it would be more
> common than doing the same with http or https, which requires an
> external server?

Oh whoa... "requires an external server"?

My entire point was to suggest a way forward for an _internal_ server that
speaks https:// instead of git://.

So I am not suggesting what you seem to have understood me to suggest.

Ciao,
Johannes

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Ævar Arnfjörð Bjarmason]

On Thu, Feb 02 2023, Johannes Schindelin wrote:

> Hi Ævar,
>
> On Thu, 2 Feb 2023, Ævar Arnfjörð Bjarmason wrote:
>
>> On Thu, Feb 02 2023, Johannes Schindelin wrote:
>>
>> > On Thu, 26 Jan 2023, Junio C Hamano wrote:
>> >
>> >> Jeff King <peff@peff.net> writes:
>> >>
>> >> >> Thanks, both.  Let's merge it down.
>> >> >
>> >> > Sorry, I'm a bit late to the party, but I left some comments just now
>> >> > (this topic had been on my review backlog for ages, but I never quite
>> >> > got to it).
>> >> >
>> >> > Many of my comments were small bits that could be fixed on top (tiny
>> >> > leaks, etc). But some of my comments were of the form "no, do it totally
>> >> > differently". It may simply be too late for those ones, but let's see if
>> >> > Matthew finds anything compelling in them.
>> >>
>> >> I do not mind reverting the merge to 'next' to have an improved
>> >> version.  Your "do we really want to add a custom server based on
>> >> questionable codebase whose quality as a test-bed for real world
>> >> usage is dubious?" is a valid concern.
>> >
>> > Except.
>> >
>> > Except that this code base would have made for a fine base to potentially
>> > implement an HTTPS-based replacement for the aging and insecure
>> > git-daemon.
>> >
>> > That code base (which is hardly as questionable codebase as you make it
>> > sound because it has been in use for years in a slightly different form)
>> > would have had the opportunity to mature in a relatively safe environment:
>> > our test suite. And eventually, once robust enough, it could have been
>> > extended to allow for easy and painless yet secure ad-hoc serving of Git
>> > repositories, addressing the security concerns around git-daemon.
>> >
>> > And now that we're throwing out that code we don't have that opportunity,
>> > making the goal to deprecate the git-daemon and replace it by something
>> > that is as easy to set up but talks HTTPS instead much, much harder to
>> > reach.
>>
>> There's many reasons for why you almost never see a git:// URL in the
>> wild anymore.
>
> I am unwilling to accept that statement without any source to back it up.
> Thin air is no substitute for reliable evidence.

Most people exposing git over the Internet use the ssh or http
transport, and our own "git" protocol is relatively obscure.

If you need data I think major hosting sites not offering it, or
deprecating it, is a pretty strong signal, e.g. Microsoft with:
https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git

But if you'll grant me that it's 50/50 git/other protocols (I think it's
a *lot* more lopsided), then clearly combining git with 3rd party server
components isn't the limiting factor on deploying it.

Which is the point I was going for.

>> But if "easy and painless" was synonymous with "built with git" or
>> "ships with git" as you seem to be using it, surely it would be more
>> common than doing the same with http or https, which requires an
>> external server?
>
> Oh whoa... "requires an external server"?
>
> My entire point was to suggest a way forward for an _internal_ server that
> speaks https:// instead of git://.

I understand that.

> So I am not suggesting what you seem to have understood me to suggest.

I wasn't suggesting that, and you seem to have not read my reply to the
end, which should have addressed that.

Briefly, we'd like to be guaranteed to have regcomp() and regexec(), but
did the Git project write its own regex engine?

No, we imported (with some minor tweaks) one from glibc/gawk (whatever
current issues have cropped up with it lately...).

So can't we do the same for a httpd? If it really comes to "we must have
it in-tree"?

It seems to me that there's a continuum here, which is at the very
least:

1) We require an external package (e.g. ssh, or apache/httpd)
2) We require an external package *or* built-in (e.g. our SHA-1
   implementations)
3) We use an external package as-is (sha1dc)
4) We adapt an external codebase, and perma-fork it (git-imap-send,
   although that example also kind of sucks)
5) We write it "in-house" from scratch.

It seems to me from reading the upthread that we're jumping straight
from #1 to #5, and it's not clear to me why that is.

Not even that, we currently have CI tests running Apache on *nix boxes,
but you're suggesting a loss of coverage on Windows 

Is it really harder to just install (or even ship our own package of)
Apache for Windows than it is to embark on PID file handling, logging,
timeout management and the long tail of "80% is easy, the rest is really
hard" of writing our own production-class httpd (as the suggestion is to
have it eventually mature beyond the test suite)?

Maybe, all I'm saying (in trying to mediate the discussion between you
and Jeff) is that it's not obvious to me why that is...

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hi Ævar,

On Mon, 6 Feb 2023, Ævar Arnfjörð Bjarmason wrote:

> we currently have CI tests running Apache on *nix boxes, but you're
> suggesting a loss of coverage on Windows
>
> Is it really harder to just install (or even ship our own package of)
> Apache for Windows than it is to embark on PID file handling, logging,
> timeout management and the long tail of "80% is easy, the rest is really
> hard" of writing our own production-class httpd (as the suggestion is to
> have it eventually mature beyond the test suite)?

Yes, it _is_ that much harder, and it would result in yet more painful
increases of the build times which have really gotten out of hand in the
past year.

Ciao,
Johannes

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Jeff King]

On Thu, Feb 02, 2023 at 11:14:33AM +0100, Johannes Schindelin wrote:

> > I do not mind reverting the merge to 'next' to have an improved
> > version.  Your "do we really want to add a custom server based on
> > questionable codebase whose quality as a test-bed for real world
> > usage is dubious?" is a valid concern.
> 
> Except.
> 
> Except that this code base would have made for a fine base to potentially
> implement an HTTPS-based replacement for the aging and insecure
> git-daemon.

I'm skeptical that it is a good idea for Git to implement a custom http
server from scratch. There are a lot of extended features people would
want in a production-ready server. TLS, HTTP/2, and so on. The code
under discussion is pretty stripped-down. A network service is also a
pretty big attack surface for buffer overflows, etc.

It feels to me like the resources required to make it good enough for
normal users to run would be substantial. And we'd be better off trying
to integrate with an existing project that provides a web server
(whether it's a lightweight server that supports us as a CGI, or a
library that does most of the heavy lifting).

> That code base (which is hardly as questionable codebase as you make it
> sound because it has been in use for years in a slightly different form)

Perhaps I'm being too hard on git-daemon. But my operational experience
with it is that it has several flaws, mostly around the child-management
code. We rewrote that code totally to make it usable at GitHub.

As a concrete example, the parent daemon process will do a linear walk
over all children, calling waitpid() on each one. This makes handling N
children quadratic, and the daemon grinds to a halt when there are many
clients.

> In addition, it causes a loss of test coverage because Apache is not
> available in all the setups where the "questionable" code would have had
> no problem being built and validating the credential code.
> 
> Windows, for example, will now go completely uncovered in CI regarding the
> new code.

I'm sympathetic there, though it's a problem for all of the existing
http code, too. Is there some server option that would be easier to run
everywhere, but that doesn't involve us writing a server from scratch?
Certainly I have no particular love for apache or its configuration
language.

-Peff

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Johannes Schindelin]

Hi Jeff,

On Fri, 3 Feb 2023, Jeff King wrote:

> On Thu, Feb 02, 2023 at 11:14:33AM +0100, Johannes Schindelin wrote:
>
> > > I do not mind reverting the merge to 'next' to have an improved
> > > version.  Your "do we really want to add a custom server based on
> > > questionable codebase whose quality as a test-bed for real world
> > > usage is dubious?" is a valid concern.
> >
> > Except.
> >
> > Except that this code base would have made for a fine base to potentially
> > implement an HTTPS-based replacement for the aging and insecure
> > git-daemon.
>
> I'm skeptical that it is a good idea for Git to implement a custom http
> server from scratch.

To be clear: I never suggested to implement a generic HTTP server.

All I wanted was to have a replacement for `git daemon` that speaks
https:// instead of git://. It does not have to speak to every browser out
there, it only needs to respond well when speaking to Git clients. That is
a much, much smaller surface than "production-ready server, HTTP/2 and so
on".

And while the proposed test helper was not quite complete in that way, and
while it may have had much of the `git daemon` code that you would love to
lose, it would have offered an incremental way forward.

I am afraid that this way forward is now blocked, and we're further away
from dropping that `git daemon` code you wanted to drop than ever.

Ciao,
Johannes

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Jeff King]

On Mon, Mar 27, 2023 at 11:10:40AM +0200, Johannes Schindelin wrote:

> > I'm skeptical that it is a good idea for Git to implement a custom http
> > server from scratch.
> 
> To be clear: I never suggested to implement a generic HTTP server.
> 
> All I wanted was to have a replacement for `git daemon` that speaks
> https:// instead of git://. It does not have to speak to every browser out
> there, it only needs to respond well when speaking to Git clients. That is
> a much, much smaller surface than "production-ready server, HTTP/2 and so
> on".

I guess I don't see the point of having this in our test suite, though.
We do want to test things like HTTP/2, SSL, and so on in our test suite.
So either we have a split in our tests (some use apache, some don't,
which presumably means many tests are still not run on Windows), or this
custom HTTP server eventually grows to do all of those other things.

I can see the utility outside the tests of a quick "let me stand up an
HTTP server to access Git" tool. But even there, I'd be considered with
feature creep as regular users ignore any warnings about its lack of
encryption/robustness, and so on. And it feels like something that could
utilize work already done by others in making a web server. Yes, that's
a new dependency for the tool, but there are a lot of options out there.
Surely one of them is worth building on?

> And while the proposed test helper was not quite complete in that way, and
> while it may have had much of the `git daemon` code that you would love to
> lose, it would have offered an incremental way forward.
> 
> I am afraid that this way forward is now blocked, and we're further away
> from dropping that `git daemon` code you wanted to drop than ever.

I don't see how pushing the same code into an http server helps. If we
could have incrementally improved it there, we could incrementally
improve it in git-daemon, too.

-Peff

[PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[M Hickford]

> Future work
> ===========
> 
> In the future we can further expand the protocol to allow credential helpers
> decide the best authentication scheme. Today credential helpers are still
> only expected to return a username/password pair to Git, meaning the other
> authentication schemes that may be offered still need challenge responses
> sent via a Basic Authorization header. The changes outlined above still
> permit helpers to select and configure an available authentication mode, but
> require the remote for example to unpack a bearer token from a basic
> challenge.
> 
> More careful consideration is required in the handling of custom
> authentication schemes which may not have a username, or may require
> arbitrary additional request header values be set.
> 
> For example imagine a new "FooBar" authentication scheme that is surfaced in
> the following response:
> 
> HTTP/1.1 401 Unauthorized
> WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
> 
> 
> With support for arbitrary authentication schemes, Git would call credential
> helpers with the following over standard input:
> 
> protocol=https
> host=example.com
> wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
> 
> 
> And then an enlightened credential helper could return over standard output:
> 
> protocol=https
> host=example.com
> authtype=FooBar
> username=bob@id.example.com
> password=<FooBar credential>
> header[]=X-FooBar: 12345
> header[]=X-FooBar-Alt: ABCDEF
> 
> 
> Git would be expected to attach this authorization header to the next
> request:
> 
> GET /info/refs?service=git-upload-pack HTTP/1.1
> Host: git.example
> Git-Protocol: version=2
> Authorization: FooBar <FooBar credential>
> X-FooBar: 12345
> X-FooBar-Alt: ABCDEF

Interesting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar). 

Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.

[credential]
    helper = storage  # eg. cache or osxkeychain
    helper = challenge-response  # eg. oauth-dpop

Storage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/pull.1443.git.git.1674914650588.gitgitgadget@gmail.com/T/#u

A monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.

Declaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth

[1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham]

On 2023-01-28 06:28, M Hickford wrote:

>> Future work
>> ===========
>>
>> In the future we can further expand the protocol to allow credential helpers
>> decide the best authentication scheme. Today credential helpers are still
>> only expected to return a username/password pair to Git, meaning the other
>> authentication schemes that may be offered still need challenge responses
>> sent via a Basic Authorization header. The changes outlined above still
>> permit helpers to select and configure an available authentication mode, but
>> require the remote for example to unpack a bearer token from a basic
>> challenge.
>>
>> More careful consideration is required in the handling of custom
>> authentication schemes which may not have a username, or may require
>> arbitrary additional request header values be set.
>>
>> For example imagine a new "FooBar" authentication scheme that is surfaced in
>> the following response:
>>
>> HTTP/1.1 401 Unauthorized
>> WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
>>
>>
>> With support for arbitrary authentication schemes, Git would call credential
>> helpers with the following over standard input:
>>
>> protocol=https
>> host=example.com
>> wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
>>
>>
>> And then an enlightened credential helper could return over standard output:
>>
>> protocol=https
>> host=example.com
>> authtype=FooBar
>> username=bob@id.example.com
>> password=<FooBar credential>
>> header[]=X-FooBar: 12345
>> header[]=X-FooBar-Alt: ABCDEF
>>
>>
>> Git would be expected to attach this authorization header to the next
>> request:
>>
>> GET /info/refs?service=git-upload-pack HTTP/1.1
>> Host: git.example
>> Git-Protocol: version=2
>> Authorization: FooBar <FooBar credential>
>> X-FooBar: 12345
>> X-FooBar-Alt: ABCDEF
> 
> Interesting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar). 

This would be exactly the sort of thing that this would enable. DPoP is one
example where the correct auth response requires more than just a username/
password pair in the Authorization header. We should also be returning
standard headers like 'Authenticate-Info' on 200 OK responses along side the
'store' calls to helpers; they could contain nonces or other important auth
information.

My end goal here is to extend the credential helper protocol such that that
helpers can see more of the initial request challenge, and then modify the
subsequent request directly, or configure and let curl handle it (the latter
part not present in this iteration).

One thing not called out in this doc is really the need for some capability
advertisement between Git and helpers - for example if the curl version in
use supports CURLOPT_XOAUTH2_BEARER for bearer tokens.

> Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.
> 
> [credential]
>     helper = storage  # eg. cache or osxkeychain
>     helper = challenge-response  # eg. oauth-dpop

I think really this sort of thing is where the credential helper protocol
isn't designed for credential-generating helpers in mind, but only simple
storage-only helpers. There is no affinity between get/erase/store commands
meaning one helper may return a credential for another helper to store it.
Not sure if this was ever the intention, over just the need to consult a
list of helpers for a stored credential.

> Storage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/pull.1443.git.git.1674914650588.gitgitgadget@gmail.com/T/#u
> 
> A monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.
> 
> Declaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth
> 
> [1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8

Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers

[Jeff King]

On Wed, Feb 01, 2023 at 12:15:17PM -0800, Matthew John Cheetham wrote:

> > Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.
> > 
> > [credential]
> >     helper = storage  # eg. cache or osxkeychain
> >     helper = challenge-response  # eg. oauth-dpop
> 
> I think really this sort of thing is where the credential helper protocol
> isn't designed for credential-generating helpers in mind, but only simple
> storage-only helpers. There is no affinity between get/erase/store commands
> meaning one helper may return a credential for another helper to store it.
> Not sure if this was ever the intention, over just the need to consult a
> list of helpers for a stored credential.

I actually had envisioned helpers generating credentials. In fact, in
the first iteration of the series, Git did not prompt for passwords at
all! It would depend on git-credential-prompt to do so. But I ended up
folding that in for simplicity.

I could well believe that there is not enough context passed around for
helpers to make good decisions, though. It's both a feature and a bug
that credentials from one helper get passed to another. It's good if you
want to cache a generated credential. It's bad if you don't want
credentials from one helper to leak to another, less-secure one.

So I'm open to improvements that help define and communicate that
context.

-Peff

[PATCH v8 0/3] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I use a small CGI shell script that acts as a
frontend to git-http-backend; simple authentication is configurable by
files.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.


Updates in v7
=============

 * Address several whitespace and arg/param list alignment issues.

 * Rethink the test-http-helper worker-mode error and result enum to be more
   simple and more informative to the nature of the error.

 * Use uintmax_t to store the Content-Length of a request in the helper
   test-http-server. Maintain a bit flag to store if we received such a
   header.

 * Return a "400 Bad Request" HTTP response if we fail to parse the request
   in the test-http-server.

 * Add test case to cover request message parsing in test-http-server.

 * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
   get_auth_module.

 * Correctly free the split strbufs created in the header parsing loop in
   test-http-server.

 * Avoid needless comparison > 0 for unsigned types.

 * Always set optional outputs to NULL if not present in test helper config
   value handling.

 * Remove an accidentally commented-out test cleanup line for one test case
   in t5556.


Updates in v8
=============

 * Drop custom HTTP test helper tool in favour of using a CGI shell script
   and Apache; avoiding the need to implement an HTTP server.

 * Avoid allocations in header reading callback unless we have a header we
   care about; act on the char* from libcurl directly rather than create a
   strbuf for each header.

 * Drop st_mult overflow guarding function in curl callback functions; we're
   not allocating memory based on the resulting value and just adds to
   potential confusion in the future.

Matthew John Cheetham (3):
  t5563: add tests for basic and anoymous HTTP access
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt |  19 +-
 credential.c                     |  12 ++
 credential.h                     |  15 ++
 git-compat-util.h                |  22 +++
 http.c                           | 120 ++++++++++++
 t/lib-httpd.sh                   |   1 +
 t/lib-httpd/apache.conf          |   6 +
 t/lib-httpd/nph-custom-auth.sh   |  42 ++++
 t/t5563-simple-http-auth.sh      | 322 +++++++++++++++++++++++++++++++
 9 files changed, 558 insertions(+), 1 deletion(-)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v8
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v8
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v7:

  1:  74b0de14185 <  -:  ----------- daemon: libify socket setup and option functions
  2:  b6ba344a671 <  -:  ----------- daemon: libify child process handling functions
  3:  9967401c972 <  -:  ----------- daemon: rename some esoteric/laboured terminology
  4:  17c890ee108 <  -:  ----------- test-http-server: add stub HTTP server test helper
  5:  6e70e304cfe <  -:  ----------- test-http-server: add HTTP error response function
  6:  43f1cdcbb82 !  1:  d362f7016d3 test-http-server: add HTTP request parsing
     @@ Metadata
      Author: Matthew John Cheetham <mjcheetham@outlook.com>
      
       ## Commit message ##
     -    test-http-server: add HTTP request parsing
     +    t5563: add tests for basic and anoymous HTTP access
      
     -    Add ability to parse HTTP requests to the test-http-server test helper.
     -    Introduce `struct req` to store request information including:
     +    Add a test showing simple anoymous HTTP access to an unprotected
     +    repository, that results in no credential helper invocations.
     +    Also add a test demonstrating simple basic authentication with
     +    simple credential helper support.
      
     -     * HTTP method & version
     -     * Request path and query parameters
     -     * Headers
     -     * Content type and length (from `Content-Type` and `-Length` headers)
     -
     -    Failure to parse the request results in a 400 Bad Request response to
     -    the client. Note that we're not trying to support all possible requests
     -    here, but just enough to exercise all code under test.
     +    Leverage a no-parsed headers (NPH) CGI script so that we can directly
     +    control the HTTP responses to simulate a multitude of good, bad and ugly
     +    remote server implementations around auth.
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
      
     - ## t/helper/test-http-server.c ##
     -@@ t/helper/test-http-server.c: enum worker_result {
     - 	 * Close the socket and clean up. Does not imply an error.
     - 	 */
     - 	WR_HANGUP = 2,
     -+
     -+	/*
     -+	 * Unexpected request message or error in request parsing.
     -+	 * Respond with an 400 error. Close the socket and cleanup.
     -+	 * Exit child-process with a non-zero status.
     -+	 */
     -+	WR_CLIENT_ERROR = 3,
     -+};
     -+
     -+/*
     -+ * Fields from a parsed HTTP request.
     -+ */
     -+struct req {
     -+	struct strbuf start_line;
     -+
     -+	const char *method;
     -+	const char *http_version;
     -+
     -+	struct strbuf uri_path;
     -+	struct strbuf query_args;
     -+
     -+	struct string_list header_list;
     -+	const char *content_type;
     -+	uintmax_t content_length;
     -+	unsigned has_content_length:1;
     - };
     + ## t/lib-httpd.sh ##
     +@@ t/lib-httpd.sh: prepare_httpd() {
     + 	install_script error-smart-http.sh
     + 	install_script error.sh
     + 	install_script apply-one-time-perl.sh
     ++	install_script nph-custom-auth.sh
       
     -+#define REQ__INIT { \
     -+	.start_line = STRBUF_INIT, \
     -+	.uri_path = STRBUF_INIT, \
     -+	.query_args = STRBUF_INIT, \
     -+	.header_list = STRING_LIST_INIT_NODUP, \
     -+	.content_type = NULL, \
     -+	.content_length = 0, \
     -+	.has_content_length = 0, \
     -+}
     -+
     -+static void req__release(struct req *req)
     -+{
     -+	strbuf_release(&req->start_line);
     -+
     -+	strbuf_release(&req->uri_path);
     -+	strbuf_release(&req->query_args);
     -+
     -+	string_list_clear(&req->header_list, 0);
     -+}
     -+
     - static enum worker_result send_http_error(int fd, int http_code,
     - 					  const char *http_code_name,
     - 					  int retry_after_seconds,
     -@@ t/helper/test-http-server.c: done:
     - 	return wr;
     - }
     - 
     -+/*
     -+ * Read the HTTP request up to the start of the optional message-body.
     -+ * We do this byte-by-byte because we have keep-alive turned on and
     -+ * cannot rely on an EOF.
     -+ *
     -+ * https://tools.ietf.org/html/rfc7230
     -+ *
     -+ * We cannot call die() here because our caller needs to properly
     -+ * respond to the client and/or close the socket before this
     -+ * child exits so that the client doesn't get a connection reset
     -+ * by peer error.
     -+ */
     -+static enum worker_result req__read(struct req *req, int fd)
     -+{
     -+	struct strbuf h = STRBUF_INIT;
     -+	struct string_list start_line_fields = STRING_LIST_INIT_DUP;
     -+	int nr_start_line_fields;
     -+	const char *uri_target;
     -+	const char *query;
     -+	char *hp;
     -+	const char *hv;
     -+
     -+	enum worker_result result = WR_OK;
     -+
     -+	/*
     -+	 * Read line 0 of the request and split it into component parts:
     -+	 *
     -+	 *    <method> SP <uri-target> SP <HTTP-version> CRLF
     -+	 *
     -+	 */
     -+	if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
     -+		result = WR_HANGUP;
     -+		goto done;
     -+	}
     -+
     -+	strbuf_trim_trailing_newline(&req->start_line);
     -+
     -+	nr_start_line_fields = string_list_split(&start_line_fields,
     -+						 req->start_line.buf,
     -+						 ' ', -1);
     -+	if (nr_start_line_fields != 3) {
     -+		logerror("could not parse request start-line '%s'",
     -+			 req->start_line.buf);
     -+		result = WR_CLIENT_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	req->method = xstrdup(start_line_fields.items[0].string);
     -+	req->http_version = xstrdup(start_line_fields.items[2].string);
     -+
     -+	uri_target = start_line_fields.items[1].string;
     -+
     -+	if (strcmp(req->http_version, "HTTP/1.1")) {
     -+		logerror("unsupported version '%s' (expecting HTTP/1.1)",
     -+			 req->http_version);
     -+		result = WR_CLIENT_ERROR;
     -+		goto done;
     -+	}
     -+
     -+	query = strchr(uri_target, '?');
     -+
     -+	if (query) {
     -+		strbuf_add(&req->uri_path, uri_target, (query - uri_target));
     -+		strbuf_trim_trailing_dir_sep(&req->uri_path);
     -+		strbuf_addstr(&req->query_args, query + 1);
     -+	} else {
     -+		strbuf_addstr(&req->uri_path, uri_target);
     -+		strbuf_trim_trailing_dir_sep(&req->uri_path);
     -+	}
     -+
     -+	/*
     -+	 * Read the set of HTTP headers into a string-list.
     -+	 */
     -+	while (1) {
     -+		if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
     -+			goto done;
     -+		strbuf_trim_trailing_newline(&h);
     -+
     -+		if (!h.len)
     -+			goto done; /* a blank line ends the header */
     -+
     -+		hp = strbuf_detach(&h, NULL);
     -+		string_list_append(&req->header_list, hp);
     -+
     -+		/* also store common request headers as struct req members */
     -+		if (skip_iprefix(hp, "Content-Type: ", &hv)) {
     -+			req->content_type = hv;
     -+		} else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
     -+			/*
     -+			 * Content-Length is always non-negative, but has no
     -+			 * upper bound according to RFC 7230 (§3.3.2).
     -+			 */
     -+			intmax_t len = 0;
     -+			if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
     -+			    len == INTMAX_MAX) {
     -+				logerror("invalid content-length: '%s'", hv);
     -+				result = WR_CLIENT_ERROR;
     -+				goto done;
     -+			}
     -+
     -+			req->content_length = (uintmax_t)len;
     -+			req->has_content_length = 1;
     -+		}
     -+	}
     -+
     -+	/*
     -+	 * We do not attempt to read the <message-body>, if it exists.
     -+	 * We let our caller read/chunk it in as appropriate.
     -+	 */
     -+
     -+done:
     -+	string_list_clear(&start_line_fields, 0);
     -+
     -+	/*
     -+	 * This is useful for debugging the request, but very noisy.
     -+	 */
     -+	if (trace2_is_enabled()) {
     -+		struct string_list_item *item;
     -+		trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
     -+		trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
     -+		trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
     -+		trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
     -+		trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
     -+		if (req->has_content_length)
     -+			trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
     -+				      req->content_length);
     -+		if (req->content_type)
     -+			trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
     -+		for_each_string_list_item(item, &req->header_list)
     -+			trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
     -+	}
     -+
     -+	return result;
     -+}
     -+
     -+static enum worker_result dispatch(struct req *req)
     -+{
     -+	return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
     -+			       WR_HANGUP);
     -+}
     -+
     - static enum worker_result worker(void)
     - {
     -+	struct req req = REQ__INIT;
     - 	char *client_addr = getenv("REMOTE_ADDR");
     - 	char *client_port = getenv("REMOTE_PORT");
     - 	enum worker_result wr = WR_OK;
     -@@ t/helper/test-http-server.c: static enum worker_result worker(void)
     - 	set_keep_alive(0, logerror);
     + 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
       
     - 	while (1) {
     --		wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
     --				     NULL, WR_HANGUP);
     -+		req__release(&req);
     +
     + ## t/lib-httpd/apache.conf ##
     +@@ t/lib-httpd/apache.conf: Alias /auth/dumb/ www/auth/dumb/
     + 	SetEnv GIT_HTTP_EXPORT_ALL
     + 	SetEnv GIT_PROTOCOL
     + </LocationMatch>
     ++<LocationMatch /custom_auth/>
     ++	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
     ++	SetEnv GIT_HTTP_EXPORT_ALL
     ++	CGIPassAuth on
     ++</LocationMatch>
     + ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
     + ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
     + ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
     +@@ t/lib-httpd/apache.conf: ScriptAlias /broken_smart/ broken-smart-http.sh/
     + ScriptAlias /error_smart/ error-smart-http.sh/
     + ScriptAlias /error/ error.sh/
     + ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
     ++ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
     + <Directory ${GIT_EXEC_PATH}>
     + 	Options FollowSymlinks
     + </Directory>
     +
     + ## t/lib-httpd/nph-custom-auth.sh (new) ##
     +@@
     ++#!/bin/sh
      +
     -+		alarm(timeout);
     -+		wr = req__read(&req, 0);
     -+		alarm(0);
     ++VALID_CREDS_FILE=custom-auth.valid
     ++CHALLENGE_FILE=custom-auth.challenge
     ++ANONYMOUS_FILE=custom-auth.anonymous
      +
     -+		if (wr == WR_CLIENT_ERROR)
     -+			wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
     -+					     -1, NULL, wr);
     ++#
     ++# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
     ++#
     ++# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
     ++# credential for the current request. Each line in the file is considered a
     ++# valid HTTP Authorization header value. For example:
     ++#
     ++# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++#
     ++# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
     ++# in a 401 response if no valid authentication credentials were included in the
     ++# request. For example:
     ++#
     ++# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
     ++# WWW-Authenticate: Basic realm="example.com"
     ++#
      +
     -+		if (wr != WR_OK)
     -+			break;
     - 
     -+		wr = dispatch(&req);
     - 		if (wr != WR_OK)
     - 			break;
     - 	}
     ++if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
     ++	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
     ++then
     ++	# Note that although git-http-backend returns a status line, it
     ++	# does so using a CGI 'Status' header. Because this script is an
     ++	# No Parsed Headers (NPH) script, we must return a real HTTP
     ++	# status line.
     ++	# This is only a test script, so we don't bother to check for
     ++	# the actual status from git-http-backend and always return 200.
     ++	echo 'HTTP/1.1 200 OK'
     ++	exec "$GIT_EXEC_PATH"/git-http-backend
     ++fi
     ++
     ++echo 'HTTP/1.1 401 Authorization Required'
     ++if test -f "$CHALLENGE_FILE"
     ++then
     ++	cat "$CHALLENGE_FILE"
     ++fi
     ++echo
      
     - ## t/t5556-http-auth.sh (new) ##
     + ## t/t5563-simple-http-auth.sh (new) ##
      @@
      +#!/bin/sh
      +
      +test_description='test http auth header and credential helper interop'
      +
     -+TEST_NO_CREATE_REPO=1
      +. ./test-lib.sh
     -+
     -+# Setup a repository
     -+#
     -+REPO_DIR="$TRASH_DIRECTORY"/repo
     -+
     -+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
     -+
     -+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
     -+
     -+test_expect_success 'setup repos' '
     -+	test_create_repo "$REPO_DIR" &&
     -+	git -C "$REPO_DIR" branch -M main
     ++. "$TEST_DIRECTORY"/lib-httpd.sh
     ++
     ++start_httpd
     ++
     ++test_expect_success 'setup_credential_helper' '
     ++	mkdir -p "$TRASH_DIRECTORY/bin" &&
     ++	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
     ++	export PATH &&
     ++
     ++	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
     ++	write_script "$CREDENTIAL_HELPER" <<-\EOF
     ++	cmd=$1
     ++	teefile=$cmd-query.cred
     ++	catfile=$cmd-reply.cred
     ++	sed -n -e "/^$/q" -e "p" >> $teefile
     ++	if test "$cmd" = "get"; then
     ++		cat $catfile
     ++	fi
     ++	EOF
      +'
      +
     -+run_http_server_worker() {
     -+	(
     -+		cd "$REPO_DIR"
     -+		test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
     -+	)
     ++set_credential_reply() {
     ++	cat >"$TRASH_DIRECTORY/$1-reply.cred"
     ++}
     ++
     ++expect_credential_query() {
     ++	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
     ++	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
     ++		 "$TRASH_DIRECTORY/$1-query.cred"
      +}
      +
      +per_test_cleanup () {
     -+	rm -f OUT.* &&
     -+	rm -f IN.* &&
     ++	rm -f *.cred &&
     ++	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
      +}
      +
     -+test_expect_success 'http auth server request parsing' '
     -+	test_when_finished "per_test_cleanup" &&
     ++test_expect_success 'setup repository' '
     ++	test_commit foo &&
     ++	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
     ++	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
     ++'
      +
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		allowAnonymous = true
     -+	EOF
     ++test_expect_success 'access anonymous no challenge' '
     ++	test_when_finished "per_test_cleanup" &&
     ++	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
     ++'
      +
     -+	echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
     -+	echo "HTTP/1.1 200 OK" >OUT.http200 &&
     ++test_expect_success 'access using basic auth' '
     ++	test_when_finished "per_test_cleanup" &&
      +
     -+	cat >IN.http.valid <<-EOF &&
     -+	GET /info/refs HTTP/1.1
     -+	Content-Length: 0
     ++	set_credential_reply get <<-EOF &&
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
      +
     -+	cat >IN.http.badfirstline <<-EOF &&
     -+	/info/refs GET HTTP
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
      +
     -+	cat >IN.http.badhttpver <<-EOF &&
     -+	GET /info/refs HTTP/999.9
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
     ++	WWW-Authenticate: Basic realm="example.com"
      +	EOF
      +
     -+	cat >IN.http.ltzlen <<-EOF &&
     -+	GET /info/refs HTTP/1.1
     -+	Content-Length: -1
     -+	EOF
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
     -+	cat >IN.http.badlen <<-EOF &&
     -+	GET /info/refs HTTP/1.1
     -+	Content-Length: not-a-number
     ++	expect_credential_query get <<-EOF &&
     ++	protocol=http
     ++	host=$HTTPD_DEST
      +	EOF
      +
     -+	cat >IN.http.overlen <<-EOF &&
     -+	GET /info/refs HTTP/1.1
     -+	Content-Length: 9223372036854775807
     ++	expect_credential_query store <<-EOF
     ++	protocol=http
     ++	host=$HTTPD_DEST
     ++	username=alice
     ++	password=secret-passwd
      +	EOF
     -+
     -+	run_http_server_worker \
     -+		--auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
     -+		| head -n1 >OUT.actual &&
     -+	test_cmp OUT.http200 OUT.actual &&
     -+
     -+	run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
     -+	test_cmp OUT.http400 OUT.actual &&
     -+
     -+	run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
     -+	test_cmp OUT.http400 OUT.actual &&
     -+
     -+	run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
     -+	test_cmp OUT.http400 OUT.actual &&
     -+
     -+	run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
     -+	test_cmp OUT.http400 OUT.actual
      +'
      +
      +test_done
  7:  ca9c2787248 <  -:  ----------- test-http-server: pass Git requests to http-backend
  8:  b8d3e81b553 <  -:  ----------- test-http-server: add simple authentication
  9:  2f97c94f679 <  -:  ----------- test-http-server: add sending of arbitrary headers
 10:  4b1635b3f69 <  -:  ----------- http: replace unsafe size_t multiplication with st_mult
 11:  5f5e46038cf !  2:  cd9a02ba94e http: read HTTP WWW-Authenticate response headers
     @@ credential.h: struct credential {
       
       /* Initialize a credential structure, setting all fields to empty. */
      
     + ## git-compat-util.h ##
     +@@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *prefix,
     + 	return 0;
     + }
     + 
     ++/*
     ++ * Like skip_prefix_mem, but compare case-insensitively. Note that the
     ++ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
     ++ * characters or locale-specific conversions).
     ++ */
     ++static inline int skip_iprefix_mem(const char *buf, size_t len,
     ++				   const char *prefix,
     ++				   const char **out, size_t *outlen)
     ++{
     ++	size_t prefix_len = strlen(prefix);
     ++	if (len < prefix_len)
     ++		return 0;
     ++
     ++	if (!strncasecmp(buf, prefix, prefix_len)){
     ++		*out = buf + prefix_len;
     ++		*outlen = len - prefix_len;
     ++		return 1;
     ++	}
     ++
     ++	return 0;
     ++}
     ++
     + static inline int strtoul_ui(char const *s, int base, unsigned int *result)
     + {
     + 	unsigned long ul;
     +
       ## http.c ##
      @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
       	return nmemb;
       }
       
     ++/*
     ++ * A folded header continuation line starts with at least one single whitespace
     ++ * character. It is not a continuation line if the line is *just* a newline.
     ++ * The RFC for HTTP states that CRLF is the header field line ending, but some
     ++ * servers may use LF only; we accept both.
     ++ */
     ++static inline int is_hdr_continuation(const char *ptr, const size_t size)
     ++{
     ++	/* totally empty line or normal header */
     ++	if (!size || !isspace(*ptr))
     ++		return 0;
     ++
     ++	/* empty line with LF line ending */
     ++	if (size == 1 && ptr[0] == '\n')
     ++		return 0;
     ++
     ++	/* empty line with CRLF line ending */
     ++	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
     ++		return 0;
     ++
     ++	return 1;
     ++}
     ++
      +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
      +{
     -+	size_t size = st_mult(eltsize, nmemb);
     ++	size_t size = eltsize * nmemb;
      +	struct strvec *values = &http_auth.wwwauth_headers;
      +	struct strbuf buf = STRBUF_INIT;
      +	const char *val;
     ++	size_t val_len;
      +
      +	/*
      +	 * Header lines may not come NULL-terminated from libcurl so we must
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 *                    of token, separators, and quoted-string>
      +	 */
      +
     -+	strbuf_add(&buf, ptr, size);
     -+
     -+	/* Strip the CRLF that should be present at the end of each field */
     -+	strbuf_trim_trailing_newline(&buf);
     -+
      +	/* Start of a new WWW-Authenticate header */
     -+	if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
     -+		while (isspace(*val))
     -+			val++;
     ++	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
     ++		strbuf_add(&buf, val, val_len);
      +
     -+		strvec_push(values, val);
     ++		/*
     ++		 * Strip the CRLF that should be present at the end of each
     ++		 * field as well as any trailing or leading whitespace from the
     ++		 * value.
     ++		 */
     ++		strbuf_trim(&buf);
     ++
     ++		strvec_push(values, buf.buf);
      +		http_auth.header_is_last_match = 1;
      +		goto exit;
      +	}
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * This line could be a continuation of the previously matched header
      +	 * field. If this is the case then we should append this value to the
      +	 * end of the previously consumed value.
     -+	 * Continuation lines start with at least one whitespace, maybe more,
     -+	 * so we should collapse these down to a single SP (valid per the spec).
      +	 */
     -+	if (http_auth.header_is_last_match && isspace(*buf.buf)) {
     -+		/* Trim leading whitespace from this continuation hdr line. */
     -+		strbuf_ltrim(&buf);
     ++	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
     ++		/*
     ++		 * Trim the CRLF and any leading or trailing from this line.
     ++		 */
     ++		strbuf_add(&buf, ptr, size);
     ++		strbuf_trim(&buf);
      +
      +		/*
      +		 * At this point we should always have at least one existing
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * We only care about the last HTTP request response's headers so clear
      +	 * the existing array.
      +	 */
     -+	if (istarts_with(buf.buf, "http/"))
     ++	if (!strncasecmp(ptr, "http/", 5))
      +		strvec_clear(values);
      +
      +exit:
 12:  09164f77d56 !  3:  149aedf5501 credential: add WWW-Authenticate header to cred requests
     @@ credential.c: void credential_write(const struct credential *c, FILE *fp)
       
       static int run_credential_helper(struct credential *c,
      
     - ## t/lib-credential-helper.sh (new) ##
     -@@
     -+setup_credential_helper() {
     -+	test_expect_success 'setup credential helper' '
     -+		CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
     -+		export CREDENTIAL_HELPER &&
     -+		echo $CREDENTIAL_HELPER &&
     -+
     -+		write_script "$CREDENTIAL_HELPER" <<-\EOF
     -+		cmd=$1
     -+		teefile=$cmd-query.cred
     -+		catfile=$cmd-reply.cred
     -+		sed -n -e "/^$/q" -e "p" >> $teefile
     -+		if test "$cmd" = "get"; then
     -+			cat $catfile
     -+		fi
     -+		EOF
     -+	'
     -+}
     + ## t/t5563-simple-http-auth.sh ##
     +@@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
     + 	expect_credential_query get <<-EOF &&
     + 	protocol=http
     + 	host=$HTTPD_DEST
     ++	wwwauth[]=Basic realm="example.com"
     ++	EOF
      +
     -+set_credential_reply() {
     -+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
     -+}
     ++	expect_credential_query store <<-EOF
     ++	protocol=http
     ++	host=$HTTPD_DEST
     ++	username=alice
     ++	password=secret-passwd
     ++	EOF
     ++'
      +
     -+expect_credential_query() {
     -+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
     -+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
     -+		 "$TRASH_DIRECTORY/$1-query.cred"
     -+}
     -
     - ## t/t5556-http-auth.sh ##
     -@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
     - 
     - TEST_NO_CREATE_REPO=1
     - . ./test-lib.sh
     -+. "$TEST_DIRECTORY"/lib-credential-helper.sh
     - 
     - test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
     - 
     -@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
     - 	git -C "$REPO_DIR" branch -M main
     - '
     - 
     -+setup_credential_helper
     -+
     - run_http_server_worker() {
     - 	(
     - 		cd "$REPO_DIR"
     -@@ t/t5556-http-auth.sh: per_test_cleanup () {
     - 	stop_http_server &&
     - 	rm -f OUT.* &&
     - 	rm -f IN.* &&
     -+	rm -f *.cred &&
     - 	rm -f auth.config
     - }
     - 
     -@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
     - 	git ls-remote $ORIGIN_URL
     - '
     - 
     -+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
     ++test_expect_success 'access using basic auth invalid credentials' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     ++
     ++	set_credential_reply get <<-EOF &&
     ++	username=baduser
     ++	password=wrong-passwd
      +	EOF
      +
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++	EOF
      +
     -+	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
     ++	WWW-Authenticate: Basic realm="example.com"
      +	EOF
      +
     -+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++	test_config_global credential.helper test-helper &&
     ++	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
      +	expect_credential_query get <<-EOF &&
      +	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=Basic realm="example.com"
      +	EOF
      +
     -+	expect_credential_query store <<-EOF
     ++	expect_credential_query erase <<-EOF
      +	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=secret-passwd
     ++	host=$HTTPD_DEST
     ++	username=baduser
     ++	password=wrong-passwd
     ++	wwwauth[]=Basic realm="example.com"
      +	EOF
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
     ++test_expect_success 'access using basic auth with extra challenges' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     -+		extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +
     -+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++	EOF
     ++
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
     ++	WWW-Authenticate: FooBar param1="value1" param2="value2"
     ++	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
     ++	WWW-Authenticate: Basic realm="example.com"
     ++	EOF
     ++
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
      +	expect_credential_query get <<-EOF &&
      +	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     -+	wwwauth[]=bEaRer auThoRiTy="id.example.com"
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=FooBar param1="value1" param2="value2"
     ++	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
     ++	wwwauth[]=Basic realm="example.com"
      +	EOF
      +
      +	expect_credential_query store <<-EOF
      +	protocol=http
     -+	host=$HOST_PORT
     ++	host=$HTTPD_DEST
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
     ++test_expect_success 'access using basic auth mixed-case wwwauth header name' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = "bearer:authority=\"id.example.com\"\\n    q=1\\n \\t p=0"
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +
     -+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++	EOF
     ++
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
     ++	www-authenticate: foobar param1="value1" param2="value2"
     ++	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
     ++	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
     ++	EOF
     ++
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
      +	expect_credential_query get <<-EOF &&
      +	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=foobar param1="value1" param2="value2"
     ++	wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
     ++	wwwauth[]=baSiC realm="example.com"
      +	EOF
      +
      +	expect_credential_query store <<-EOF
      +	protocol=http
     -+	host=$HOST_PORT
     ++	host=$HTTPD_DEST
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
     ++test_expect_success 'access using basic auth with wwwauth header continuations' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     -+		extraheader = "WWW-Authenticate:"
     -+		extraheader = " "
     -+		extraheader = " bearer authority=\"id.example.com\""
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +
     -+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++	EOF
     ++
     ++	# Note that leading and trailing whitespace is important to correctly
     ++	# simulate a continuation/folded header.
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
     ++	WWW-Authenticate: FooBar param1="value1"
     ++	 param2="value2"
     ++	WWW-Authenticate: Bearer authorize_uri="id.example.com"
     ++	 p=1
     ++	 q=0
     ++	WWW-Authenticate: Basic realm="example.com"
     ++	EOF
     ++
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
      +	expect_credential_query get <<-EOF &&
      +	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=basic realm="example.com"
     -+	wwwauth[]=bearer authority="id.example.com"
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=FooBar param1="value1" param2="value2"
     ++	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
     ++	wwwauth[]=Basic realm="example.com"
      +	EOF
      +
      +	expect_credential_query store <<-EOF
      +	protocol=http
     -+	host=$HOST_PORT
     ++	host=$HTTPD_DEST
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
     ++test_expect_success 'access using basic auth with wwwauth header empty continuations' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = "foobar:alg=test widget=1"
     -+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +
     -+	git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
     ++	EOF
     ++
     ++	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
     ++
     ++	# Note that leading and trailing whitespace is important to correctly
     ++	# simulate a continuation/folded header.
     ++	printf "">$CHALLENGE &&
     ++	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
     ++	printf " \r\n" >>$CHALLENGE &&
     ++	printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
     ++	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
     ++	printf " p=1\r\n" >>$CHALLENGE &&
     ++	printf " \r\n" >>$CHALLENGE &&
     ++	printf " q=0\r\n" >>$CHALLENGE &&
     ++	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
     ++
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
      +
      +	expect_credential_query get <<-EOF &&
      +	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=foobar alg=test widget=1
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=FooBar param1="value1" param2="value2"
     ++	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
     ++	wwwauth[]=Basic realm="example.com"
      +	EOF
      +
      +	expect_credential_query store <<-EOF
      +	protocol=http
     -+	host=$HOST_PORT
     ++	host=$HTTPD_DEST
      +	username=alice
      +	password=secret-passwd
      +	EOF
      +'
      +
     -+test_expect_success 'http auth www-auth headers to credential helper invalid' '
     ++test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
      +	test_when_finished "per_test_cleanup" &&
     -+	# base64("alice:secret-passwd")
     -+	USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
     -+	export USERPASS64 &&
     -+
     -+	cat >auth.config <<-EOF &&
     -+	[auth]
     -+		challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
     -+		challenge = basic:realm=\"example.com\"
     -+		token = basic:$USERPASS64
     -+	EOF
     -+
     -+	start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
      +
      +	set_credential_reply get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
      +	username=alice
     -+	password=invalid-passwd
     ++	password=secret-passwd
      +	EOF
      +
     -+	test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
     -+
     -+	expect_credential_query get <<-EOF &&
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     ++	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
     ++	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
      +
     -+	expect_credential_query erase <<-EOF
     -+	protocol=http
     -+	host=$HOST_PORT
     -+	username=alice
     -+	password=invalid-passwd
     -+	wwwauth[]=bearer authority="id.example.com" q=1 p=0
     -+	wwwauth[]=basic realm="example.com"
     -+	EOF
     -+'
     ++	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
     ++
     ++	# Note that leading and trailing whitespace is important to correctly
     ++	# simulate a continuation/folded header.
     ++	printf "">$CHALLENGE &&
     ++	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
     ++	printf " \r\n" >>$CHALLENGE &&
     ++	printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
     ++	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
      +
     - test_done
     ++	test_config_global credential.helper test-helper &&
     ++	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
     ++
     ++	expect_credential_query get <<-EOF &&
     ++	protocol=http
     ++	host=$HTTPD_DEST
     ++	wwwauth[]=FooBar param1="value1" param2="value2"
     ++	wwwauth[]=Basic realm="example.com"
     + 	EOF
     + 
     + 	expect_credential_query store <<-EOF

-- 
gitgitgadget

[PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a test showing simple anoymous HTTP access to an unprotected
repository, that results in no credential helper invocations.
Also add a test demonstrating simple basic authentication with
simple credential helper support.

Leverage a no-parsed headers (NPH) CGI script so that we can directly
control the HTTP responses to simulate a multitude of good, bad and ugly
remote server implementations around auth.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/lib-httpd.sh                 |  1 +
 t/lib-httpd/apache.conf        |  6 +++
 t/lib-httpd/nph-custom-auth.sh | 42 +++++++++++++++++
 t/t5563-simple-http-auth.sh    | 86 ++++++++++++++++++++++++++++++++++
 4 files changed, 135 insertions(+)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index 608949ea80b..2c49569f675 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -137,6 +137,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
+	install_script nph-custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 0294739a77a..76335cdb24d 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_HTTP_EXPORT_ALL
 	SetEnv GIT_PROTOCOL
 </LocationMatch>
+<LocationMatch /custom_auth/>
+	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
+	SetEnv GIT_HTTP_EXPORT_ALL
+	CGIPassAuth on
+</LocationMatch>
 ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
 ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
 ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
new file mode 100755
index 00000000000..8f851aebac4
--- /dev/null
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+VALID_CREDS_FILE=custom-auth.valid
+CHALLENGE_FILE=custom-auth.challenge
+ANONYMOUS_FILE=custom-auth.anonymous
+
+#
+# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
+#
+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
+# credential for the current request. Each line in the file is considered a
+# valid HTTP Authorization header value. For example:
+#
+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+#
+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
+# in a 401 response if no valid authentication credentials were included in the
+# request. For example:
+#
+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+# WWW-Authenticate: Basic realm="example.com"
+#
+
+if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
+	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
+then
+	# Note that although git-http-backend returns a status line, it
+	# does so using a CGI 'Status' header. Because this script is an
+	# No Parsed Headers (NPH) script, we must return a real HTTP
+	# status line.
+	# This is only a test script, so we don't bother to check for
+	# the actual status from git-http-backend and always return 200.
+	echo 'HTTP/1.1 200 OK'
+	exec "$GIT_EXEC_PATH"/git-http-backend
+fi
+
+echo 'HTTP/1.1 401 Authorization Required'
+if test -f "$CHALLENGE_FILE"
+then
+	cat "$CHALLENGE_FILE"
+fi
+echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
new file mode 100755
index 00000000000..004eac5d1ed
--- /dev/null
+++ b/t/t5563-simple-http-auth.sh
@@ -0,0 +1,86 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup_credential_helper' '
+	mkdir -p "$TRASH_DIRECTORY/bin" &&
+	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
+	export PATH &&
+
+	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
+	write_script "$CREDENTIAL_HELPER" <<-\EOF
+	cmd=$1
+	teefile=$cmd-query.cred
+	catfile=$cmd-reply.cred
+	sed -n -e "/^$/q" -e "p" >> $teefile
+	if test "$cmd" = "get"; then
+		cat $catfile
+	fi
+	EOF
+'
+
+set_credential_reply() {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query() {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
+
+per_test_cleanup () {
+	rm -f *.cred &&
+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
+}
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+test_expect_success 'access anonymous no challenge' '
+	test_when_finished "per_test_cleanup" &&
+	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
+'
+
+test_expect_success 'access using basic auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Add a test showing simple anoymous HTTP access to an unprotected
> repository, that results in no credential helper invocations.
> Also add a test demonstrating simple basic authentication with
> simple credential helper support.
>
> Leverage a no-parsed headers (NPH) CGI script so that we can directly
> control the HTTP responses to simulate a multitude of good, bad and ugly
> remote server implementations around auth.
>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> ---
>  t/lib-httpd.sh                 |  1 +
>  t/lib-httpd/apache.conf        |  6 +++
>  t/lib-httpd/nph-custom-auth.sh | 42 +++++++++++++++++
>  t/t5563-simple-http-auth.sh    | 86 ++++++++++++++++++++++++++++++++++
>  4 files changed, 135 insertions(+)
>  create mode 100755 t/lib-httpd/nph-custom-auth.sh
>  create mode 100755 t/t5563-simple-http-auth.sh
>
> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
> index 608949ea80b..2c49569f675 100644
> --- a/t/lib-httpd.sh
> +++ b/t/lib-httpd.sh
> @@ -137,6 +137,7 @@ prepare_httpd() {
>  	install_script error-smart-http.sh
>  	install_script error.sh
>  	install_script apply-one-time-perl.sh
> +	install_script nph-custom-auth.sh
>  
>  	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
>  
> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
> index 0294739a77a..76335cdb24d 100644
> --- a/t/lib-httpd/apache.conf
> +++ b/t/lib-httpd/apache.conf
> @@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
>  	SetEnv GIT_HTTP_EXPORT_ALL
>  	SetEnv GIT_PROTOCOL
>  </LocationMatch>
> +<LocationMatch /custom_auth/>
> +	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
> +	SetEnv GIT_HTTP_EXPORT_ALL
> +	CGIPassAuth on
> +</LocationMatch>
>  ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
>  ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
>  ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
> @@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
>  ScriptAlias /error_smart/ error-smart-http.sh/
>  ScriptAlias /error/ error.sh/
>  ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
>  <Directory ${GIT_EXEC_PATH}>
>  	Options FollowSymlinks
>  </Directory>
> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
> new file mode 100755
> index 00000000000..8f851aebac4
> --- /dev/null
> +++ b/t/lib-httpd/nph-custom-auth.sh
> @@ -0,0 +1,42 @@
> +#!/bin/sh
> +
> +VALID_CREDS_FILE=custom-auth.valid
> +CHALLENGE_FILE=custom-auth.challenge
> +ANONYMOUS_FILE=custom-auth.anonymous
> +
> +#
> +# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
> +#
> +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
> +# credential for the current request. Each line in the file is considered a
> +# valid HTTP Authorization header value. For example:
> +#
> +# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
> +#
> +# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
> +# in a 401 response if no valid authentication credentials were included in the
> +# request. For example:
> +#
> +# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> +# WWW-Authenticate: Basic realm="example.com"
> +#
> +
> +if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
> +	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")

Rather than "test -f "$f" & grep ... "$f" I think you can just use only
"grep", if the file doesn't exist it'll give you an error.

If you don't want to see that error just pipe it to /dev/null, in case
that's what you were trying to avoid with the "check if it exists
first".

> +echo 'HTTP/1.1 401 Authorization Required'
> +if test -f "$CHALLENGE_FILE"
> +then
> +	cat "$CHALLENGE_FILE"

Maybe the same here, i.e. just:

	cat "$f" 2>/dev/null

> +test_expect_success 'setup_credential_helper' '
> +	mkdir -p "$TRASH_DIRECTORY/bin" &&

The "$TRASH_DIRECTORY" is already created for you, so don't use "-p",
unless something went wrong here..

> +	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
> +	export PATH &&
> +
> +	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
> +	write_script "$CREDENTIAL_HELPER" <<-\EOF
> +	cmd=$1
> +	teefile=$cmd-query.cred
> +	catfile=$cmd-reply.cred
> +	sed -n -e "/^$/q" -e "p" >> $teefile

Style: ">>$f", not ">> $f"

> +	if test "$cmd" = "get"; then

Style: We usually use "\nthen", not "; then".

Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
> index 608949ea80b..2c49569f675 100644
> --- a/t/lib-httpd.sh
> +++ b/t/lib-httpd.sh
> @@ -137,6 +137,7 @@ prepare_httpd() {
>  	install_script error-smart-http.sh
>  	install_script error.sh
>  	install_script apply-one-time-perl.sh
> +	install_script nph-custom-auth.sh
>  
>  	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
>  
> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
> index 0294739a77a..76335cdb24d 100644
> --- a/t/lib-httpd/apache.conf
> +++ b/t/lib-httpd/apache.conf
> @@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
>  	SetEnv GIT_HTTP_EXPORT_ALL
>  	SetEnv GIT_PROTOCOL
>  </LocationMatch>
> +<LocationMatch /custom_auth/>
> +	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
> +	SetEnv GIT_HTTP_EXPORT_ALL
> +	CGIPassAuth on
> +</LocationMatch>
>  ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
>  ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
>  ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
> @@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
>  ScriptAlias /error_smart/ error-smart-http.sh/
>  ScriptAlias /error/ error.sh/
>  ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1

This setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'
script) is nice and straightforward. 

>  <Directory ${GIT_EXEC_PATH}>
>  	Options FollowSymlinks
>  </Directory>
> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
> new file mode 100755
> index 00000000000..8f851aebac4
> --- /dev/null
> +++ b/t/lib-httpd/nph-custom-auth.sh
> @@ -0,0 +1,42 @@
> +#!/bin/sh
> +
> +VALID_CREDS_FILE=custom-auth.valid
> +CHALLENGE_FILE=custom-auth.challenge
> +ANONYMOUS_FILE=custom-auth.anonymous
> +
> +#
> +# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
> +#
> +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
> +# credential for the current request. Each line in the file is considered a
> +# valid HTTP Authorization header value. For example:
> +#
> +# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
> +#
> +# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
> +# in a 401 response if no valid authentication credentials were included in the
> +# request. For example:
> +#
> +# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> +# WWW-Authenticate: Basic realm="example.com"
> +#
> +
> +if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
> +	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")

So there are two cases where you want to return a '200 OK' response:

1. anonymous access is allowed (indicated by $ANONYMOUS_FILE existing)
2. anonymous access is *not* allowed, 'HTTP_AUTHORIZATION' is non-empty, and
   it matches at least one line in $VALID_CREDS_FILE

Does the '$' at the end of "^${HTTP_AUTHORIZATION:-nopenopnope}$" need to be
escaped? I'm guessing it doesn't *need* to be based on the fact that the
tests are passing, but it might be safer to escape it anyway.

I see what you're going for with the "nopenopenope" substitution, but I
think you could be more explicit about requiring that 'HTTP_AUTHORIZATION'
is set without the need for a special invalid value fallback:

    if test -f "$ANONYMOUS_FILE" || (test -n "$HTTP_AUTHORIZATION" && \
    	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE")

Note the addition of '-s' to 'grep' - it seems cleaner than redirecting to
'/dev/null' (as Ævar suggested [1]) while achieving the same result.

[1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/

> +then
> +	# Note that although git-http-backend returns a status line, it
> +	# does so using a CGI 'Status' header. Because this script is an
> +	# No Parsed Headers (NPH) script, we must return a real HTTP
> +	# status line.
> +	# This is only a test script, so we don't bother to check for
> +	# the actual status from git-http-backend and always return 200.
> +	echo 'HTTP/1.1 200 OK'
> +	exec "$GIT_EXEC_PATH"/git-http-backend

I'm not familiar with 'exec', but a cursory look at the documentation shows
that, because this replaces the current shell, it will exit with the code
from 'git-http-backend', so there's no risk of continuing on to print the
'401 Authorization Required' response & challenge handling. 

> +fi
> +
> +echo 'HTTP/1.1 401 Authorization Required'
> +if test -f "$CHALLENGE_FILE"
> +then
> +	cat "$CHALLENGE_FILE"
> +fi

In contrast to Ævar's comments in the review linked earlier, I like having
the explicit 'test -f' (to sort of "self-document" that the challenge is
only issued if $CHALLENGE_FILE exists). I think you're fine keeping this
as-is or changing it, depending on your preference.

> +test_expect_success 'access anonymous no challenge' '
> +	test_when_finished "per_test_cleanup" &&
> +	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
> +	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
> +'
> +
> +test_expect_success 'access using basic auth' '
> +	test_when_finished "per_test_cleanup" &&
> +
> +	set_credential_reply get <<-EOF &&
> +	username=alice
> +	password=secret-passwd
> +	EOF
> +
> +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
> +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
> +	EOF
> +
> +	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
> +	WWW-Authenticate: Basic realm="example.com"
> +	EOF
> +
> +	test_config_global credential.helper test-helper &&
> +	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
> +
> +	expect_credential_query get <<-EOF &&
> +	protocol=http
> +	host=$HTTPD_DEST
> +	EOF
> +
> +	expect_credential_query store <<-EOF
> +	protocol=http
> +	host=$HTTPD_DEST
> +	username=alice
> +	password=secret-passwd
> +	EOF
> +'
> +
> +test_done

And these tests properly exercise the custom auth handling. 

While I wasn't as opposed to the custom HTTP handler as others that have
commented, I do appreciate the relative simplicity of this new Apache setup
and like that it's still pretty easy to test. Nice work!

Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access

[Ævar Arnfjörð Bjarmason]

On Wed, Feb 08 2023, Victoria Dye wrote:

> Matthew John Cheetham via GitGitGadget wrote:
>>  ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
>> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
>
> This setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'
> script) is nice and straightforward. 

*nod*

> [...]
>> +if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
>> +	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
> [...]
>     if test -f "$ANONYMOUS_FILE" || (test -n "$HTTP_AUTHORIZATION" && \
>     	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE")
>
> Note the addition of '-s' to 'grep' - it seems cleaner than redirecting to
> '/dev/null' (as Ævar suggested [1]) while achieving the same result.
>
> [1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/

I wondered if it's in POSIX, turns out it is!:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html

But we don't have any existing use of it, even for things in POSIX it's
often a gamble what the exact semantics are on our long tail of *nix,
e.g. old AIX.

In general I'd think we could just avoid "-s" or piping to "/dev/null"
here, i.e. under "-x" or whatever it's informative to know it doesn't
exist from the stderr, but on second look I think both of us long track
of a larger issue here...
> [...]
>> +fi
>> +
>> +echo 'HTTP/1.1 401 Authorization Required'
>> +if test -f "$CHALLENGE_FILE"
>> +then
>> +	cat "$CHALLENGE_FILE"
>> +fi
>
> In contrast to Ævar's comments in the review linked earlier, I like having
> the explicit 'test -f' (to sort of "self-document" that the challenge is
> only issued if $CHALLENGE_FILE exists). I think you're fine keeping this
> as-is or changing it, depending on your preference.

Looking at this again I think we should just have it be unconditional
here. I.e. it looks like we both assumed that this needs to be a
conditional, but actually every /custom_auth/ test also sets up this
"$CHALLENGE_FILE".

So this "test -f" seems to only serve the purpose of burying an error
under the rug if things have already gone wrong.

But if we're making these requests why are we writing a script that
handles the combination of 3 parameters, and needs to second guess
things? We can just create N urls and N scripts instead. So I tried this
fix-up instead:
	
	diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
	index 7979605d344..c25e3000db0 100644
	--- a/t/lib-httpd.sh
	+++ b/t/lib-httpd.sh
	@@ -141,6 +141,7 @@ prepare_httpd() {
	 	install_script error.sh
	 	install_script apply-one-time-perl.sh
	 	install_script nph-custom-auth.sh
	+	install_script nph-custom-auth-anon.sh
	 
	 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
	 
	diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
	index 2aac922376c..7a63a9169c3 100644
	--- a/t/lib-httpd/apache.conf
	+++ b/t/lib-httpd/apache.conf
	@@ -140,6 +140,7 @@ ScriptAlias /error_smart/ error-smart-http.sh/
	 ScriptAlias /error/ error.sh/
	 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
	 ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
	+ScriptAliasMatch /custom_auth_anon/(.*) nph-custom-auth-anon.sh/$1
	 <Directory ${GIT_EXEC_PATH}>
	 	Options FollowSymlinks
	 </Directory>
	diff --git a/t/lib-httpd/nph-custom-auth-anon.sh b/t/lib-httpd/nph-custom-auth-anon.sh
	new file mode 100755
	index 00000000000..3c7a24fed6b
	--- /dev/null
	+++ b/t/lib-httpd/nph-custom-auth-anon.sh
	@@ -0,0 +1,4 @@
	+#!/bin/sh
	+
	+echo 'HTTP/1.1 200 OK'
	+exec "$GIT_EXEC_PATH"/git-http-backend
	diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
	index 8f851aebac4..e3ee61c8c9e 100755
	--- a/t/lib-httpd/nph-custom-auth.sh
	+++ b/t/lib-httpd/nph-custom-auth.sh
	@@ -1,28 +1,15 @@
	 #!/bin/sh
	 
	+set -e
	+
	 VALID_CREDS_FILE=custom-auth.valid
	-CHALLENGE_FILE=custom-auth.challenge
	-ANONYMOUS_FILE=custom-auth.anonymous
	 
	-#
	-# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
	-#
	 # If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
	 # credential for the current request. Each line in the file is considered a
	 # valid HTTP Authorization header value. For example:
	 #
	 # Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
	-#
	-# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
	-# in a 401 response if no valid authentication credentials were included in the
	-# request. For example:
	-#
	-# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
	-# WWW-Authenticate: Basic realm="example.com"
	-#
	-
	-if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
	-	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
	+if grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE"
	 then
	 	# Note that although git-http-backend returns a status line, it
	 	# does so using a CGI 'Status' header. Because this script is an
	@@ -31,12 +18,15 @@ then
	 	# This is only a test script, so we don't bother to check for
	 	# the actual status from git-http-backend and always return 200.
	 	echo 'HTTP/1.1 200 OK'
	-	exec "$GIT_EXEC_PATH"/git-http-backend
	+	exit 1
	 fi
	 
	+# Output of our challenge file as headers
	+# in a 401 response if no valid authentication credentials were included in the
	+# request. For example:
	+#
	+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
	+# WWW-Authenticate: Basic realm="example.com"
	 echo 'HTTP/1.1 401 Authorization Required'
	-if test -f "$CHALLENGE_FILE"
	-then
	-	cat "$CHALLENGE_FILE"
	-fi
	+cat custom-auth.challenge
	 echo
	diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
	index a7b1e5bd1af..feb8149de8f 100755
	--- a/t/t5563-simple-http-auth.sh
	+++ b/t/t5563-simple-http-auth.sh
	@@ -47,8 +47,7 @@ test_expect_success 'setup repository' '
	 
	 test_expect_success 'access anonymous no challenge' '
	 	test_when_finished "per_test_cleanup" &&
	-	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
	-	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
	+	git ls-remote "$HTTPD_URL/custom_auth_anon/repo.git"
	 '
	 
	 test_expect_success 'access using basic auth' '

I think that's much better, now we just have a 2-line script to handle
this "anon auth" case. Instead of creating a "custom-auth.anonymous"
file to communicate how the remote end should behave, let's just
communicate that by requesting a different URL, one that accepts
anonymous authentication.

I did insert a deliberate bug here, or:

	-	exec "$GIT_EXEC_PATH"/git-http-backend
	+	exit 1

So aside from your "exec" comment it seems both of us missed that this
"exec" does nothing useful, the test will fail if we emit different
headers, but it doesn't matter that we execute the git-http-backend.

Or maybe it does, but the tests aren't good enough to spot the
difference.

The above is a rough WIP, I'm leaving it here for Matthew to follow-up
on. I think it might benefit from being further split-up, i.e. we know
which URLs we expect to fail auth, so if we just had another URL for
"the auth response fails here" we'd have 3x trivial scripts with no
if/else; but maybe that sucks, I didn't try it.

Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham]

On 2023-02-09 03:19, Ævar Arnfjörð Bjarmason wrote:

> 
> On Wed, Feb 08 2023, Victoria Dye wrote:
> 
>> Matthew John Cheetham via GitGitGadget wrote:
>>>  ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
>>> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
>>
>> This setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'
>> script) is nice and straightforward. 
> 
> *nod*
> 
>> [...]
>>> +if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
>>> +	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
>> [...]
>>     if test -f "$ANONYMOUS_FILE" || (test -n "$HTTP_AUTHORIZATION" && \
>>     	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE")
>>
>> Note the addition of '-s' to 'grep' - it seems cleaner than redirecting to
>> '/dev/null' (as Ævar suggested [1]) while achieving the same result.
>>
>> [1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/
> 
> I wondered if it's in POSIX, turns out it is!:
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html
> 
> But we don't have any existing use of it, even for things in POSIX it's
> often a gamble what the exact semantics are on our long tail of *nix,
> e.g. old AIX.
> 
> In general I'd think we could just avoid "-s" or piping to "/dev/null"
> here, i.e. under "-x" or whatever it's informative to know it doesn't
> exist from the stderr, but on second look I think both of us long track
> of a larger issue here...
>> [...]
>>> +fi
>>> +
>>> +echo 'HTTP/1.1 401 Authorization Required'
>>> +if test -f "$CHALLENGE_FILE"
>>> +then
>>> +	cat "$CHALLENGE_FILE"
>>> +fi
>>
>> In contrast to Ævar's comments in the review linked earlier, I like having
>> the explicit 'test -f' (to sort of "self-document" that the challenge is
>> only issued if $CHALLENGE_FILE exists). I think you're fine keeping this
>> as-is or changing it, depending on your preference.
> 
> Looking at this again I think we should just have it be unconditional
> here. I.e. it looks like we both assumed that this needs to be a
> conditional, but actually every /custom_auth/ test also sets up this
> "$CHALLENGE_FILE".
> 
> So this "test -f" seems to only serve the purpose of burying an error
> under the rug if things have already gone wrong.
> 
> But if we're making these requests why are we writing a script that
> handles the combination of 3 parameters, and needs to second guess
> things? We can just create N urls and N scripts instead. So I tried this
> fix-up instead:
> 	
> 	diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
> 	index 7979605d344..c25e3000db0 100644
> 	--- a/t/lib-httpd.sh
> 	+++ b/t/lib-httpd.sh
> 	@@ -141,6 +141,7 @@ prepare_httpd() {
> 	 	install_script error.sh
> 	 	install_script apply-one-time-perl.sh
> 	 	install_script nph-custom-auth.sh
> 	+	install_script nph-custom-auth-anon.sh
> 	 
> 	 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
> 	 
> 	diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
> 	index 2aac922376c..7a63a9169c3 100644
> 	--- a/t/lib-httpd/apache.conf
> 	+++ b/t/lib-httpd/apache.conf
> 	@@ -140,6 +140,7 @@ ScriptAlias /error_smart/ error-smart-http.sh/
> 	 ScriptAlias /error/ error.sh/
> 	 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
> 	 ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
> 	+ScriptAliasMatch /custom_auth_anon/(.*) nph-custom-auth-anon.sh/$1
> 	 <Directory ${GIT_EXEC_PATH}>
> 	 	Options FollowSymlinks
> 	 </Directory>
> 	diff --git a/t/lib-httpd/nph-custom-auth-anon.sh b/t/lib-httpd/nph-custom-auth-anon.sh
> 	new file mode 100755
> 	index 00000000000..3c7a24fed6b
> 	--- /dev/null
> 	+++ b/t/lib-httpd/nph-custom-auth-anon.sh
> 	@@ -0,0 +1,4 @@
> 	+#!/bin/sh
> 	+
> 	+echo 'HTTP/1.1 200 OK'
> 	+exec "$GIT_EXEC_PATH"/git-http-backend
> 	diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
> 	index 8f851aebac4..e3ee61c8c9e 100755
> 	--- a/t/lib-httpd/nph-custom-auth.sh
> 	+++ b/t/lib-httpd/nph-custom-auth.sh
> 	@@ -1,28 +1,15 @@
> 	 #!/bin/sh
> 	 
> 	+set -e
> 	+
> 	 VALID_CREDS_FILE=custom-auth.valid
> 	-CHALLENGE_FILE=custom-auth.challenge
> 	-ANONYMOUS_FILE=custom-auth.anonymous
> 	 
> 	-#
> 	-# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
> 	-#
> 	 # If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
> 	 # credential for the current request. Each line in the file is considered a
> 	 # valid HTTP Authorization header value. For example:
> 	 #
> 	 # Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
> 	-#
> 	-# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
> 	-# in a 401 response if no valid authentication credentials were included in the
> 	-# request. For example:
> 	-#
> 	-# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> 	-# WWW-Authenticate: Basic realm="example.com"
> 	-#
> 	-
> 	-if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
> 	-	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
> 	+if grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE"
> 	 then
> 	 	# Note that although git-http-backend returns a status line, it
> 	 	# does so using a CGI 'Status' header. Because this script is an
> 	@@ -31,12 +18,15 @@ then
> 	 	# This is only a test script, so we don't bother to check for
> 	 	# the actual status from git-http-backend and always return 200.
> 	 	echo 'HTTP/1.1 200 OK'
> 	-	exec "$GIT_EXEC_PATH"/git-http-backend
> 	+	exit 1
> 	 fi
> 	 
> 	+# Output of our challenge file as headers
> 	+# in a 401 response if no valid authentication credentials were included in the
> 	+# request. For example:
> 	+#
> 	+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> 	+# WWW-Authenticate: Basic realm="example.com"
> 	 echo 'HTTP/1.1 401 Authorization Required'
> 	-if test -f "$CHALLENGE_FILE"
> 	-then
> 	-	cat "$CHALLENGE_FILE"
> 	-fi
> 	+cat custom-auth.challenge
> 	 echo
> 	diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
> 	index a7b1e5bd1af..feb8149de8f 100755
> 	--- a/t/t5563-simple-http-auth.sh
> 	+++ b/t/t5563-simple-http-auth.sh
> 	@@ -47,8 +47,7 @@ test_expect_success 'setup repository' '
> 	 
> 	 test_expect_success 'access anonymous no challenge' '
> 	 	test_when_finished "per_test_cleanup" &&
> 	-	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
> 	-	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
> 	+	git ls-remote "$HTTPD_URL/custom_auth_anon/repo.git"
> 	 '
> 	 
> 	 test_expect_success 'access using basic auth' '
> 
> I think that's much better, now we just have a 2-line script to handle
> this "anon auth" case. Instead of creating a "custom-auth.anonymous"
> file to communicate how the remote end should behave, let's just
> communicate that by requesting a different URL, one that accepts
> anonymous authentication.

Actually, we don't really need to test the anonymous auth case at all
because all other tests that try accessing a remote repository over HTTP
are already exercising this. See t5551-http-fetch-smart for example..
here we're performing various requests without auth.
Should we be erronously issuing credential helper challenges in these
scenarios then the tests would fail with an askpass prompt.

I will drop the anon auth test and script support.

> I did insert a deliberate bug here, or:
> 
> 	-	exec "$GIT_EXEC_PATH"/git-http-backend
> 	+	exit 1
> 
> So aside from your "exec" comment it seems both of us missed that this
> "exec" does nothing useful, the test will fail if we emit different
> headers, but it doesn't matter that we execute the git-http-backend.
> 
> Or maybe it does, but the tests aren't good enough to spot the
> difference.
> 
> The above is a rough WIP, I'm leaving it here for Matthew to follow-up
> on. I think it might benefit from being further split-up, i.e. we know
> which URLs we expect to fail auth, so if we just had another URL for
> "the auth response fails here" we'd have 3x trivial scripts with no
> if/else; but maybe that sucks, I didn't try it.

[PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c      |   1 +
 credential.h      |  15 ++++++
 git-compat-util.h |  22 +++++++++
 http.c            | 120 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 158 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/git-compat-util.h b/git-compat-util.h
index a76d0526f79..f11c44517d7 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1266,6 +1266,28 @@ static inline int skip_iprefix(const char *str, const char *prefix,
 	return 0;
 }
 
+/*
+ * Like skip_prefix_mem, but compare case-insensitively. Note that the
+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
+ * characters or locale-specific conversions).
+ */
+static inline int skip_iprefix_mem(const char *buf, size_t len,
+				   const char *prefix,
+				   const char **out, size_t *outlen)
+{
+	size_t prefix_len = strlen(prefix);
+	if (len < prefix_len)
+		return 0;
+
+	if (!strncasecmp(buf, prefix, prefix_len)){
+		*out = buf + prefix_len;
+		*outlen = len - prefix_len;
+		return 1;
+	}
+
+	return 0;
+}
+
 static inline int strtoul_ui(char const *s, int base, unsigned int *result)
 {
 	unsigned long ul;
diff --git a/http.c b/http.c
index 8a5ba3f4776..7a56a3db5f7 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+/*
+ * A folded header continuation line starts with at least one single whitespace
+ * character. It is not a continuation line if the line is *just* a newline.
+ * The RFC for HTTP states that CRLF is the header field line ending, but some
+ * servers may use LF only; we accept both.
+ */
+static inline int is_hdr_continuation(const char *ptr, const size_t size)
+{
+	/* totally empty line or normal header */
+	if (!size || !isspace(*ptr))
+		return 0;
+
+	/* empty line with LF line ending */
+	if (size == 1 && ptr[0] == '\n')
+		return 0;
+
+	/* empty line with CRLF line ending */
+	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
+		return 0;
+
+	return 1;
+}
+
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	size_t val_len;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
+		strbuf_add(&buf, val, val_len);
+
+		/*
+		 * Strip the CRLF that should be present at the end of each
+		 * field as well as any trailing or leading whitespace from the
+		 * value.
+		 */
+		strbuf_trim(&buf);
+
+		strvec_push(values, buf.buf);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
+		/*
+		 * Trim the CRLF and any leading or trailing from this line.
+		 */
+		strbuf_add(&buf, ptr, size);
+		strbuf_trim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (!strncasecmp(ptr, "http/", 5))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1982,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>

> +static inline int skip_iprefix_mem(const char *buf, size_t len,
> +				   const char *prefix,
> +				   const char **out, size_t *outlen)
> +{
> +	size_t prefix_len = strlen(prefix);

Style nit: We typically use \n\n after the end of decls, so an extra
newline here before the code.

> +	if (len < prefix_len)
> +		return 0;
> +
> +	if (!strncasecmp(buf, prefix, prefix_len)){

Style: ")) {", not ")){".

Re: [PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> diff --git a/git-compat-util.h b/git-compat-util.h
> index a76d0526f79..f11c44517d7 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -1266,6 +1266,28 @@ static inline int skip_iprefix(const char *str, const char *prefix,
>  	return 0;
>  }
>  
> +/*
> + * Like skip_prefix_mem, but compare case-insensitively. Note that the
> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
> + * characters or locale-specific conversions).
> + */
> +static inline int skip_iprefix_mem(const char *buf, size_t len,
> +				   const char *prefix,
> +				   const char **out, size_t *outlen)
> +{
> +	size_t prefix_len = strlen(prefix);
> +	if (len < prefix_len)
> +		return 0;
> +
> +	if (!strncasecmp(buf, prefix, prefix_len)){
> +		*out = buf + prefix_len;
> +		*outlen = len - prefix_len;
> +		return 1;
> +	}
> +
> +	return 0;
> +}
> +
>  static inline int strtoul_ui(char const *s, int base, unsigned int *result)
>  {
>  	unsigned long ul;
> diff --git a/http.c b/http.c
> index 8a5ba3f4776..7a56a3db5f7 100644
> --- a/http.c
> +++ b/http.c
> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  	return nmemb;
>  }
>  
> +/*
> + * A folded header continuation line starts with at least one single whitespace
> + * character. It is not a continuation line if the line is *just* a newline.
> + * The RFC for HTTP states that CRLF is the header field line ending, but some
> + * servers may use LF only; we accept both.
> + */
> +static inline int is_hdr_continuation(const char *ptr, const size_t size)
> +{
> +	/* totally empty line or normal header */
> +	if (!size || !isspace(*ptr))
> +		return 0;
> +
> +	/* empty line with LF line ending */
> +	if (size == 1 && ptr[0] == '\n')
> +		return 0;
> +
> +	/* empty line with CRLF line ending */
> +	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
> +		return 0;
> +
> +	return 1;
> +}
> +
> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = eltsize * nmemb;
> +	struct strvec *values = &http_auth.wwwauth_headers;
> +	struct strbuf buf = STRBUF_INIT;
> +	const char *val;
> +	size_t val_len;
> +
> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
> +		strbuf_add(&buf, val, val_len);
> +
> +		/*
> +		 * Strip the CRLF that should be present at the end of each
> +		 * field as well as any trailing or leading whitespace from the
> +		 * value.
> +		 */
> +		strbuf_trim(&buf);
> +
> +		strvec_push(values, buf.buf);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;
> +	}
> +
> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 */
> +	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
> +		/*
> +		 * Trim the CRLF and any leading or trailing from this line.
> +		 */
> +		strbuf_add(&buf, ptr, size);
> +		strbuf_trim(&buf);
> +
> +		/*
> +		 * At this point we should always have at least one existing
> +		 * value, even if it is empty. Do not bother appending the new
> +		 * value if this continuation header is itself empty.
> +		 */
> +		if (!values->nr) {
> +			BUG("should have at least one existing header value");
> +		} else if (buf.len) {
> +			char *prev = xstrdup(values->v[values->nr - 1]);
> +
> +			/* Join two non-empty values with a single space. */
> +			const char *const sp = *prev ? " " : "";
> +
> +			strvec_pop(values);
> +			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
> +			free(prev);
> +		}
> +
> +		goto exit;
> +	}
> +
> +	/* This is the start of a new header we don't care about */
> +	http_auth.header_is_last_match = 0;
> +
> +	/*
> +	 * If this is a HTTP status line and not a header field, this signals
> +	 * a different HTTP response. libcurl writes all the output of all
> +	 * response headers of all responses, including redirects.
> +	 * We only care about the last HTTP request response's headers so clear
> +	 * the existing array.
> +	 */
> +	if (!strncasecmp(ptr, "http/", 5))
> +		strvec_clear(values);

I found this updated version of 'fwrite_wwwauth()' (using
'skip_iprefix_mem()', 'is_hdr_continuation()', and 'strncasecmp()') a bit
easier to read than previous iterations - possibly because all the
prefix-skipping is done before adding to 'buf', so 'buf' represents *only*
the line's header value (possibly with leading/trailing whitespace, which is
trimmed). Plus, avoiding unnecessary allocations is always nice. 

> +
> +exit:
> +	strbuf_release(&buf);
> +	return size;
> +}

[PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |  11 ++
 t/t5563-simple-http-auth.sh      | 236 +++++++++++++++++++++++++++++++
 3 files changed, 265 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..9f39ebc3c7e 100644
--- a/credential.c
+++ b/credential.c
@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
+static void credential_write_strvec(FILE *fp, const char *key,
+				    const struct strvec *vec)
+{
+	char *full_key = xstrfmt("%s[]", key);
+	for (size_t i = 0; i < vec->nr; i++) {
+		credential_write_item(fp, full_key, vec->v[i], 0);
+	}
+	free(full_key);
+}
+
 void credential_write(const struct credential *c, FILE *fp)
 {
 	credential_write_item(fp, "protocol", c->protocol, 1);
@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index 004eac5d1ed..a7b1e5bd1af 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -73,6 +73,242 @@ test_expect_success 'access using basic auth' '
 	expect_credential_query get <<-EOF &&
 	protocol=http
 	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=baduser
+	password=wrong-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=baduser
+	password=wrong-passwd
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
+test_expect_success 'access using basic auth with extra challenges' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth mixed-case wwwauth header name' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	www-authenticate: foobar param1="value1" param2="value2"
+	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=foobar param1="value1" param2="value2"
+	wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=baSiC realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1"
+	 param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	 p=1
+	 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header empty continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
+	printf " p=1\r\n" >>$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " q=0\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Basic realm="example.com"
 	EOF
 
 	expect_credential_query store <<-EOF
-- 
gitgitgadget

Re: [PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:

> From: Matthew John Cheetham <mjcheetham@outlook.com>

> @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>  	fprintf(fp, "%s=%s\n", key, value);
>  }
>  
> +static void credential_write_strvec(FILE *fp, const char *key,
> +				    const struct strvec *vec)
> +{
> +	char *full_key = xstrfmt("%s[]", key);

FWIW you could avoid this allocation if you just renamed the current
"credential_write_item()" to "credential_write_fmt()", and had it take a
format instead of its current hardcoded "%s=%s\n".

Then you could have two wrappers, credential_write_item() and
credential_write_items() (instead of "strvec"), the first passing
"%s=%s\n", the other "%s[]=%s\n".

Just a thought.

> +	for (size_t i = 0; i < vec->nr; i++) {
> +		credential_write_item(fp, full_key, vec->v[i], 0);

The {} here can be dropped in any case.

Re: [PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham]

On 2023-02-06 12:45, Ævar Arnfjörð Bjarmason wrote:

> 
> On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:
> 
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
> 
>> @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>>  	fprintf(fp, "%s=%s\n", key, value);
>>  }
>>  
>> +static void credential_write_strvec(FILE *fp, const char *key,
>> +				    const struct strvec *vec)
>> +{
>> +	char *full_key = xstrfmt("%s[]", key);
> 
> FWIW you could avoid this allocation if you just renamed the current
> "credential_write_item()" to "credential_write_fmt()", and had it take a
> format instead of its current hardcoded "%s=%s\n".
> 
> Then you could have two wrappers, credential_write_item() and
> credential_write_items() (instead of "strvec"), the first passing
> "%s=%s\n", the other "%s[]=%s\n".
> 
> Just a thought.

Given this is the only `items/strvec` that I'm writing out, just inlining
the for-loop and calling `credential_write_item` directly with the key as
"wwwauth[]" would avoid the allocation without needing to make the write_item
function more complicated.

-static void credential_write_strvec(FILE *fp, const char *key,
-                                   const struct strvec *vec)
-{
-       char *full_key = xstrfmt("%s[]", key);
-       for (size_t i = 0; i < vec->nr; i++) {
-               credential_write_item(fp, full_key, vec->v[i], 0);
-       }
-       free(full_key);
-}
-
 void credential_write(const struct credential *c, FILE *fp)
 {
        credential_write_item(fp, "protocol", c->protocol, 1);
@@ -280,7 +270,10 @@ void credential_write(const struct credential *c, FILE *fp)
        credential_write_item(fp, "path", c->path, 0);
        credential_write_item(fp, "username", c->username, 0);
        credential_write_item(fp, "password", c->password, 0);
-       credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
+       for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
+               credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i],
+                                     0);
 }


>> +	for (size_t i = 0; i < vec->nr; i++) {
>> +		credential_write_item(fp, full_key, vec->v[i], 0);
> 
> The {} here can be dropped in any case.
> 

Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:

> Updates in v8
> =============
>
>  * Drop custom HTTP test helper tool in favour of using a CGI shell script
>    and Apache; avoiding the need to implement an HTTP server.
>
>  * Avoid allocations in header reading callback unless we have a header we
>    care about; act on the char* from libcurl directly rather than create a
>    strbuf for each header.
>
>  * Drop st_mult overflow guarding function in curl callback functions; we're
>    not allocating memory based on the resulting value and just adds to
>    potential confusion in the future.

I just had some nit-y and other trivial comments spotted on a
read-through, but with the caveat that I'm not too familiar with the
credential infrastructure this looks good to me.

I'm rather neutral on the whole question of whether we eventually ship a
httpd in-tree, but I think the v7 to v8 clearly demonstrates that
whatever we do there, this topic is much improved by having that
question un-tangled from the credential improvements here.

Thanks!

Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers

[Victoria Dye]

Matthew John Cheetham via GitGitGadget wrote:
> Updates in v8
> =============
> 
>  * Drop custom HTTP test helper tool in favour of using a CGI shell script
>    and Apache; avoiding the need to implement an HTTP server.
> 
>  * Avoid allocations in header reading callback unless we have a header we
>    care about; act on the char* from libcurl directly rather than create a
>    strbuf for each header.
> 
>  * Drop st_mult overflow guarding function in curl callback functions; we're
>    not allocating memory based on the resulting value and just adds to
>    potential confusion in the future.
> 
The core functionality change in this series (passing through
'WWW-Authenticate' headers to credential requests) didn't change much
between v7 and v8, and what was updated was a net improvement [1].
Otherwise, the new test infrastructure is concise and made for an easy
re-review; I had one small suggestion for patch 1 [2], but it's not critical
enough on its own to hold this series up. I think this could probably be
merged to 'next' as-is, but a final re-roll with some of the minor fixups
suggested in other reviews wouldn't hurt either. 

Thanks again for the time & effort you've put into perfecting these patches!

[1] https://lore.kernel.org/git/983fc35b-55e8-50df-5035-191a10b4ddac@github.com/
[2] https://lore.kernel.org/git/87f79e79-1591-ca28-4975-3bca5b8f7266@github.com/

Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers

[Junio C Hamano]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> In this patch series I update the existing credential helper design in order
> to allow for some new scenarios, and future evolution of auth methods that
> Git hosts may wish to provide. I outline the background, summary of changes
> and some challenges below.
>
> Testing these new additions, I use a small CGI shell script that acts as a
> frontend to git-http-backend; simple authentication is configurable by
> files.

Thanks for an update, and thanks Victoria and Ævar for your prompt
reviews.

Queued on 'seen', anticipating a small finishing touches before
merging to 'next'.

[PATCH v9 0/3] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I use a small CGI shell script that acts as a
frontend to git-http-backend; simple authentication is configurable by
files.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.


Updates in v7
=============

 * Address several whitespace and arg/param list alignment issues.

 * Rethink the test-http-helper worker-mode error and result enum to be more
   simple and more informative to the nature of the error.

 * Use uintmax_t to store the Content-Length of a request in the helper
   test-http-server. Maintain a bit flag to store if we received such a
   header.

 * Return a "400 Bad Request" HTTP response if we fail to parse the request
   in the test-http-server.

 * Add test case to cover request message parsing in test-http-server.

 * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
   get_auth_module.

 * Correctly free the split strbufs created in the header parsing loop in
   test-http-server.

 * Avoid needless comparison > 0 for unsigned types.

 * Always set optional outputs to NULL if not present in test helper config
   value handling.

 * Remove an accidentally commented-out test cleanup line for one test case
   in t5556.


Updates in v8
=============

 * Drop custom HTTP test helper tool in favour of using a CGI shell script
   and Apache; avoiding the need to implement an HTTP server.

 * Avoid allocations in header reading callback unless we have a header we
   care about; act on the char* from libcurl directly rather than create a
   strbuf for each header.

 * Drop st_mult overflow guarding function in curl callback functions; we're
   not allocating memory based on the resulting value and just adds to
   potential confusion in the future.


Updates in v9
=============

 * Drop anoynmous auth tests as these cases are already covered by all other
   tests that perform HTTP interactions with a remote today.

 * In the custom auth CGI script, avoid the empty-substitution in favour of
   testing explicitly for an empty string. Also simplify some other
   conditional expressions.

 * Avoid an allocation on each wwwauth[] credential helper key-value pair
   write.

 * Various style fixups.

Matthew John Cheetham (3):
  t5563: add tests for basic and anoymous HTTP access
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt |  19 +-
 credential.c                     |   4 +
 credential.h                     |  15 ++
 git-compat-util.h                |  23 +++
 http.c                           | 120 ++++++++++++
 t/lib-httpd.sh                   |   1 +
 t/lib-httpd/apache.conf          |   6 +
 t/lib-httpd/nph-custom-auth.sh   |  39 ++++
 t/t5563-simple-http-auth.sh      | 317 +++++++++++++++++++++++++++++++
 9 files changed, 543 insertions(+), 1 deletion(-)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v9
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v9
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v8:

 1:  d362f7016d3 ! 1:  05449ec892b t5563: add tests for basic and anoymous HTTP access
     @@ t/lib-httpd/nph-custom-auth.sh (new)
      +
      +VALID_CREDS_FILE=custom-auth.valid
      +CHALLENGE_FILE=custom-auth.challenge
     -+ANONYMOUS_FILE=custom-auth.anonymous
      +
      +#
     -+# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
     -+#
      +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
      +# credential for the current request. Each line in the file is considered a
      +# valid HTTP Authorization header value. For example:
     @@ t/lib-httpd/nph-custom-auth.sh (new)
      +# WWW-Authenticate: Basic realm="example.com"
      +#
      +
     -+if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
     -+	grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
     ++if test -n "$HTTP_AUTHORIZATION" && \
     ++	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE"
      +then
      +	# Note that although git-http-backend returns a status line, it
      +	# does so using a CGI 'Status' header. Because this script is an
     @@ t/t5563-simple-http-auth.sh (new)
      +start_httpd
      +
      +test_expect_success 'setup_credential_helper' '
     -+	mkdir -p "$TRASH_DIRECTORY/bin" &&
     ++	mkdir "$TRASH_DIRECTORY/bin" &&
      +	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
      +	export PATH &&
      +
     @@ t/t5563-simple-http-auth.sh (new)
      +	cmd=$1
      +	teefile=$cmd-query.cred
      +	catfile=$cmd-reply.cred
     -+	sed -n -e "/^$/q" -e "p" >> $teefile
     -+	if test "$cmd" = "get"; then
     ++	sed -n -e "/^$/q" -e "p" >>$teefile
     ++	if test "$cmd" = "get"
     ++	then
      +		cat $catfile
      +	fi
      +	EOF
     @@ t/t5563-simple-http-auth.sh (new)
      +	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
      +'
      +
     -+test_expect_success 'access anonymous no challenge' '
     -+	test_when_finished "per_test_cleanup" &&
     -+	touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
     -+	git ls-remote "$HTTPD_URL/custom_auth/repo.git"
     -+'
     -+
      +test_expect_success 'access using basic auth' '
      +	test_when_finished "per_test_cleanup" &&
      +
 2:  cd9a02ba94e ! 2:  f3f28e508c1 http: read HTTP WWW-Authenticate response headers
     @@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *p
      +				   const char **out, size_t *outlen)
      +{
      +	size_t prefix_len = strlen(prefix);
     ++
      +	if (len < prefix_len)
      +		return 0;
      +
     -+	if (!strncasecmp(buf, prefix, prefix_len)){
     ++	if (!strncasecmp(buf, prefix, prefix_len)) {
      +		*out = buf + prefix_len;
      +		*outlen = len - prefix_len;
      +		return 1;
 3:  149aedf5501 ! 3:  eba58c0d08d credential: add WWW-Authenticate header to cred requests
     @@ Documentation/git-credential.txt: empty string.
       GIT
      
       ## credential.c ##
     -@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
     - 	fprintf(fp, "%s=%s\n", key, value);
     - }
     - 
     -+static void credential_write_strvec(FILE *fp, const char *key,
     -+				    const struct strvec *vec)
     -+{
     -+	char *full_key = xstrfmt("%s[]", key);
     -+	for (size_t i = 0; i < vec->nr; i++) {
     -+		credential_write_item(fp, full_key, vec->v[i], 0);
     -+	}
     -+	free(full_key);
     -+}
     -+
     - void credential_write(const struct credential *c, FILE *fp)
     - {
     - 	credential_write_item(fp, "protocol", c->protocol, 1);
      @@ credential.c: void credential_write(const struct credential *c, FILE *fp)
       	credential_write_item(fp, "path", c->path, 0);
       	credential_write_item(fp, "username", c->username, 0);
       	credential_write_item(fp, "password", c->password, 0);
     -+	credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
     ++	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
     ++		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i],
     ++				      0);
       }
       
       static int run_credential_helper(struct credential *c,

-- 
gitgitgadget

[PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a test showing simple anoymous HTTP access to an unprotected
repository, that results in no credential helper invocations.
Also add a test demonstrating simple basic authentication with
simple credential helper support.

Leverage a no-parsed headers (NPH) CGI script so that we can directly
control the HTTP responses to simulate a multitude of good, bad and ugly
remote server implementations around auth.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/lib-httpd.sh                 |  1 +
 t/lib-httpd/apache.conf        |  6 +++
 t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++
 t/t5563-simple-http-auth.sh    | 81 ++++++++++++++++++++++++++++++++++
 4 files changed, 127 insertions(+)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index 608949ea80b..2c49569f675 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -137,6 +137,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
+	install_script nph-custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 0294739a77a..76335cdb24d 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_HTTP_EXPORT_ALL
 	SetEnv GIT_PROTOCOL
 </LocationMatch>
+<LocationMatch /custom_auth/>
+	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
+	SetEnv GIT_HTTP_EXPORT_ALL
+	CGIPassAuth on
+</LocationMatch>
 ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
 ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
 ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
new file mode 100755
index 00000000000..2dd35d6fa39
--- /dev/null
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+VALID_CREDS_FILE=custom-auth.valid
+CHALLENGE_FILE=custom-auth.challenge
+
+#
+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
+# credential for the current request. Each line in the file is considered a
+# valid HTTP Authorization header value. For example:
+#
+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+#
+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
+# in a 401 response if no valid authentication credentials were included in the
+# request. For example:
+#
+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+# WWW-Authenticate: Basic realm="example.com"
+#
+
+if test -n "$HTTP_AUTHORIZATION" && \
+	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE"
+then
+	# Note that although git-http-backend returns a status line, it
+	# does so using a CGI 'Status' header. Because this script is an
+	# No Parsed Headers (NPH) script, we must return a real HTTP
+	# status line.
+	# This is only a test script, so we don't bother to check for
+	# the actual status from git-http-backend and always return 200.
+	echo 'HTTP/1.1 200 OK'
+	exec "$GIT_EXEC_PATH"/git-http-backend
+fi
+
+echo 'HTTP/1.1 401 Authorization Required'
+if test -f "$CHALLENGE_FILE"
+then
+	cat "$CHALLENGE_FILE"
+fi
+echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
new file mode 100755
index 00000000000..e0682039de7
--- /dev/null
+++ b/t/t5563-simple-http-auth.sh
@@ -0,0 +1,81 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup_credential_helper' '
+	mkdir "$TRASH_DIRECTORY/bin" &&
+	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
+	export PATH &&
+
+	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
+	write_script "$CREDENTIAL_HELPER" <<-\EOF
+	cmd=$1
+	teefile=$cmd-query.cred
+	catfile=$cmd-reply.cred
+	sed -n -e "/^$/q" -e "p" >>$teefile
+	if test "$cmd" = "get"
+	then
+		cat $catfile
+	fi
+	EOF
+'
+
+set_credential_reply() {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query() {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
+
+per_test_cleanup () {
+	rm -f *.cred &&
+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
+}
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+test_expect_success 'access using basic auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access

[Junio C Hamano]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> +if test -n "$HTTP_AUTHORIZATION" && \
> +	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE"

Do we require a regexp match (and worry about metacharacters in
HTTP_AUTHORIZATION variable), or would we want to use "grep -F -x"
here to force match with the entire line?

> +then
> +	# Note that although git-http-backend returns a status line, it
> +	# does so using a CGI 'Status' header. Because this script is an
> +	# No Parsed Headers (NPH) script, we must return a real HTTP
> +	# status line.
> +	# This is only a test script, so we don't bother to check for
> +	# the actual status from git-http-backend and always return 200.
> +	echo 'HTTP/1.1 200 OK'
> +	exec "$GIT_EXEC_PATH"/git-http-backend
> +fi

OK.  That's the successful auth case.  Otherwise ...

> +echo 'HTTP/1.1 401 Authorization Required'
> +if test -f "$CHALLENGE_FILE"
> +then
> +	cat "$CHALLENGE_FILE"
> +fi
> +echo

OK.  We'll just give a challenge.

> diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
> new file mode 100755
> index 00000000000..e0682039de7
> --- /dev/null
> +++ b/t/t5563-simple-http-auth.sh
> @@ -0,0 +1,81 @@
> +#!/bin/sh
> +
> +test_description='test http auth header and credential helper interop'
> +
> +. ./test-lib.sh
> +. "$TEST_DIRECTORY"/lib-httpd.sh
> +
> +start_httpd
> +
> +test_expect_success 'setup_credential_helper' '
> +	mkdir "$TRASH_DIRECTORY/bin" &&
> +	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
> +	export PATH &&
> +
> +	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
> +	write_script "$CREDENTIAL_HELPER" <<-\EOF
> +	cmd=$1
> +	teefile=$cmd-query.cred
> +	catfile=$cmd-reply.cred
> +	sed -n -e "/^$/q" -e "p" >>$teefile
> +	if test "$cmd" = "get"
> +	then
> +		cat $catfile
> +	fi
> +	EOF
> +'
> +
> +set_credential_reply() {

Style. Have SP before "()" as well as after.

> +	cat >"$TRASH_DIRECTORY/$1-reply.cred"
> +}
> +
> +expect_credential_query() {

Style. Have SP before "()" as well as after.

> +	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
> +	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
> +		 "$TRASH_DIRECTORY/$1-query.cred"
> +}
> +
> +per_test_cleanup () {
> +	rm -f *.cred &&
> +	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
> +}
> +
> +test_expect_success 'setup repository' '
> +	test_commit foo &&
> +	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
> +	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
> +'

OK.

> +test_expect_success 'access using basic auth' '
> +	test_when_finished "per_test_cleanup" &&
> +
> +	set_credential_reply get <<-EOF &&
> +	username=alice
> +	password=secret-passwd
> +	EOF
> +
> +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
> +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
> +	EOF

Perhaps we want to note that this matches the "alice:secret-passwd"
we prepared earlier?

> +	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
> +	WWW-Authenticate: Basic realm="example.com"
> +	EOF

OK.

> +	test_config_global credential.helper test-helper &&
> +	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
> +
> +	expect_credential_query get <<-EOF &&
> +	protocol=http
> +	host=$HTTPD_DEST
> +	EOF
> +
> +	expect_credential_query store <<-EOF
> +	protocol=http
> +	host=$HTTPD_DEST
> +	username=alice
> +	password=secret-passwd
> +	EOF
> +'

OK.

> +test_done

Re: [PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham]

On 2023-02-15 14:15, Junio C Hamano wrote:

> "Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
> 
>> +if test -n "$HTTP_AUTHORIZATION" && \
>> +	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE"
> 
> Do we require a regexp match (and worry about metacharacters in
> HTTP_AUTHORIZATION variable), or would we want to use "grep -F -x"
> here to force match with the entire line?

You're right. We don't need a regex match here. Will fix.

>> +then
>> +	# Note that although git-http-backend returns a status line, it
>> +	# does so using a CGI 'Status' header. Because this script is an
>> +	# No Parsed Headers (NPH) script, we must return a real HTTP
>> +	# status line.
>> +	# This is only a test script, so we don't bother to check for
>> +	# the actual status from git-http-backend and always return 200.
>> +	echo 'HTTP/1.1 200 OK'
>> +	exec "$GIT_EXEC_PATH"/git-http-backend
>> +fi
> 
> OK.  That's the successful auth case.  Otherwise ...
> 
>> +echo 'HTTP/1.1 401 Authorization Required'
>> +if test -f "$CHALLENGE_FILE"
>> +then
>> +	cat "$CHALLENGE_FILE"
>> +fi
>> +echo
> 
> OK.  We'll just give a challenge.
> 
>> diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
>> new file mode 100755
>> index 00000000000..e0682039de7
>> --- /dev/null
>> +++ b/t/t5563-simple-http-auth.sh
>> @@ -0,0 +1,81 @@
>> +#!/bin/sh
>> +
>> +test_description='test http auth header and credential helper interop'
>> +
>> +. ./test-lib.sh
>> +. "$TEST_DIRECTORY"/lib-httpd.sh
>> +
>> +start_httpd
>> +
>> +test_expect_success 'setup_credential_helper' '
>> +	mkdir "$TRASH_DIRECTORY/bin" &&
>> +	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
>> +	export PATH &&
>> +
>> +	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
>> +	write_script "$CREDENTIAL_HELPER" <<-\EOF
>> +	cmd=$1
>> +	teefile=$cmd-query.cred
>> +	catfile=$cmd-reply.cred
>> +	sed -n -e "/^$/q" -e "p" >>$teefile
>> +	if test "$cmd" = "get"
>> +	then
>> +		cat $catfile
>> +	fi
>> +	EOF
>> +'
>> +
>> +set_credential_reply() {
> 
> Style. Have SP before "()" as well as after.

Will fix.

>> +	cat >"$TRASH_DIRECTORY/$1-reply.cred"
>> +}
>> +
>> +expect_credential_query() {
> 
> Style. Have SP before "()" as well as after.

Ditto.

>> +	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
>> +	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
>> +		 "$TRASH_DIRECTORY/$1-query.cred"
>> +}
>> +
>> +per_test_cleanup () {
>> +	rm -f *.cred &&
>> +	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
>> +}
>> +
>> +test_expect_success 'setup repository' '
>> +	test_commit foo &&
>> +	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
>> +	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
>> +'
> 
> OK.
> 
>> +test_expect_success 'access using basic auth' '
>> +	test_when_finished "per_test_cleanup" &&
>> +
>> +	set_credential_reply get <<-EOF &&
>> +	username=alice
>> +	password=secret-passwd
>> +	EOF
>> +
>> +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
>> +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
>> +	EOF
> 
> Perhaps we want to note that this matches the "alice:secret-passwd"
> we prepared earlier?

Good point. It's helpful. Will re-introduce.

>> +	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
>> +	WWW-Authenticate: Basic realm="example.com"
>> +	EOF
> 
> OK.
> 
>> +	test_config_global credential.helper test-helper &&
>> +	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
>> +
>> +	expect_credential_query get <<-EOF &&
>> +	protocol=http
>> +	host=$HTTPD_DEST
>> +	EOF
>> +
>> +	expect_credential_query store <<-EOF
>> +	protocol=http
>> +	host=$HTTPD_DEST
>> +	username=alice
>> +	password=secret-passwd
>> +	EOF
>> +'
> 
> OK.
> 
>> +test_done

[PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c      |   1 +
 credential.h      |  15 ++++++
 git-compat-util.h |  23 +++++++++
 http.c            | 120 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 159 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Used to keep track of split header fields
+	 * in order to fold multiple lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +144,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/git-compat-util.h b/git-compat-util.h
index a76d0526f79..a59230564e8 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,
 	return 0;
 }
 
+/*
+ * Like skip_prefix_mem, but compare case-insensitively. Note that the
+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
+ * characters or locale-specific conversions).
+ */
+static inline int skip_iprefix_mem(const char *buf, size_t len,
+				   const char *prefix,
+				   const char **out, size_t *outlen)
+{
+	size_t prefix_len = strlen(prefix);
+
+	if (len < prefix_len)
+		return 0;
+
+	if (!strncasecmp(buf, prefix, prefix_len)) {
+		*out = buf + prefix_len;
+		*outlen = len - prefix_len;
+		return 1;
+	}
+
+	return 0;
+}
+
 static inline int strtoul_ui(char const *s, int base, unsigned int *result)
 {
 	unsigned long ul;
diff --git a/http.c b/http.c
index 8a5ba3f4776..7a56a3db5f7 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+/*
+ * A folded header continuation line starts with at least one single whitespace
+ * character. It is not a continuation line if the line is *just* a newline.
+ * The RFC for HTTP states that CRLF is the header field line ending, but some
+ * servers may use LF only; we accept both.
+ */
+static inline int is_hdr_continuation(const char *ptr, const size_t size)
+{
+	/* totally empty line or normal header */
+	if (!size || !isspace(*ptr))
+		return 0;
+
+	/* empty line with LF line ending */
+	if (size == 1 && ptr[0] == '\n')
+		return 0;
+
+	/* empty line with CRLF line ending */
+	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
+		return 0;
+
+	return 1;
+}
+
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	size_t val_len;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 2616 (even though this has since been
+	 * deprecated in RFC 7230). A continuation header field value is
+	 * identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 2616 is:
+	 *
+	 *   message-header = field-name ":" [ field-value ]
+	 *   field-name     = token
+	 *   field-value    = *( field-content | LWS )
+	 *   field-content  = <the OCTETs making up the field-value
+	 *                    and consisting of either *TEXT or combinations
+	 *                    of token, separators, and quoted-string>
+	 */
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
+		strbuf_add(&buf, val, val_len);
+
+		/*
+		 * Strip the CRLF that should be present at the end of each
+		 * field as well as any trailing or leading whitespace from the
+		 * value.
+		 */
+		strbuf_trim(&buf);
+
+		strvec_push(values, buf.buf);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
+		/*
+		 * Trim the CRLF and any leading or trailing from this line.
+		 */
+		strbuf_add(&buf, ptr, size);
+		strbuf_trim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* This is the start of a new header we don't care about */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (!strncasecmp(ptr, "http/", 5))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1982,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers

[Junio C Hamano]

"Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> According to RFC2616 Section 4.2 [1], header field names are not
> case-sensitive meaning when collecting multiple values for the same
> field name, we can just use the case of the first observed instance of
> each field name and no normalisation is required.

If the names are not case-sensitive, you can choose to first
downcase the names you see, and use that consistently, and the
result would still be valid.  IOW, "not case-sensitive" does not at
all mean you have to use the first observed instance without
normalization.  You are allowed to choose such an implementation,
but "not case-sensitive" is not a justification to choose such an
implementation among possible implementation that would be allowed
under the rule.

> The collection of all header values matching the WWW-Authenticate
> header is complicated by the fact that it is legal for header fields to
> be continued over multiple lines, but libcurl only gives us one line at
> a time.

Saying "one physical line" at a time may make it clear what you are
pointing out as a weak point in the interface libcURL gives us (I
think you are getting at "if they handled header folding for us and
fed us one logical line at a time, it would have been nicer").

> @@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
>  	free(c->username);
>  	free(c->password);
>  	string_list_clear(&c->helpers, 0);
> +	strvec_clear(&c->wwwauth_headers);
>  
>  	credential_init(c);
>  }
> diff --git a/credential.h b/credential.h
> index f430e77fea4..6f2e5bc610b 100644
> --- a/credential.h
> +++ b/credential.h
> @@ -2,6 +2,7 @@
>  #define CREDENTIAL_H
>  
>  #include "string-list.h"
> +#include "strvec.h"
>  
>  /**
>   * The credentials API provides an abstracted way of gathering username and
> @@ -115,6 +116,19 @@ struct credential {
>  	 */
>  	struct string_list helpers;
>  
> +	/**
> +	 * A `strvec` of WWW-Authenticate header values. Each string
> +	 * is the value of a WWW-Authenticate header in an HTTP response,
> +	 * in the order they were received in the response.
> +	 */
> +	struct strvec wwwauth_headers;
> +
> +	/**
> +	 * Internal use only. Used to keep track of split header fields

The technical term for what you call "split header" here seems to be
"line folding" (RFC 7230, which deprecates it).

> +	 * in order to fold multiple lines into one value.
> +	 */
> +	unsigned header_is_last_match:1;
> +
>  	unsigned approved:1,
>  		 configured:1,
>  		 quit:1,
> @@ -130,6 +144,7 @@ struct credential {
>  
>  #define CREDENTIAL_INIT { \
>  	.helpers = STRING_LIST_INIT_DUP, \
> +	.wwwauth_headers = STRVEC_INIT, \
>  }
>  
>  /* Initialize a credential structure, setting all fields to empty. */
> diff --git a/git-compat-util.h b/git-compat-util.h
> index a76d0526f79..a59230564e8 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,
>  	return 0;
>  }
>  
> +/*
> + * Like skip_prefix_mem, but compare case-insensitively. Note that the
> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
> + * characters or locale-specific conversions).
> + */
> +static inline int skip_iprefix_mem(const char *buf, size_t len,
> +				   const char *prefix,
> +				   const char **out, size_t *outlen)
> +{
> +	size_t prefix_len = strlen(prefix);
> +
> +	if (len < prefix_len)
> +		return 0;
> +
> +	if (!strncasecmp(buf, prefix, prefix_len)) {
> +		*out = buf + prefix_len;
> +		*outlen = len - prefix_len;
> +		return 1;
> +	}
> +
> +	return 0;
> +}

OK.

> diff --git a/http.c b/http.c
> index 8a5ba3f4776..7a56a3db5f7 100644
> --- a/http.c
> +++ b/http.c
> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>  	return nmemb;
>  }
>  
> +/*
> + * A folded header continuation line starts with at least one single whitespace
> + * character. It is not a continuation line if the line is *just* a newline.
> + * The RFC for HTTP states that CRLF is the header field line ending, but some
> + * servers may use LF only; we accept both.
> + */

Nice.

> +static inline int is_hdr_continuation(const char *ptr, const size_t size)
> +{
> +	/* totally empty line or normal header */
> +	if (!size || !isspace(*ptr))
> +		return 0;

obs-fold (RFC7230) begins the next line with SP or HTAB, but
isspace() allows not just SP and HT but also CR and LF.  So
this is a bit pessimistic but rejects what is not a folded
continuation line reliably.

> +	/* empty line with LF line ending */
> +	if (size == 1 && ptr[0] == '\n')
> +		return 0;

And this is a blank line after the headers, with LF (not conforming
but is OK).

> +	/* empty line with CRLF line ending */
> +	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
> +		return 0;

And this is another form of a blank line after the headers, with
CRLF.

> +	return 1;
> +}

After rejecting the above two "blank", it is a folded continuation
line.  OK.

I wonder if

	static inline int ... () {
	  	return (size && (*ptr == ' ' || *ptr == '\t'));
	}

sufficient and easier to grok, though.

> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> +{
> +	size_t size = eltsize * nmemb;
> +	struct strvec *values = &http_auth.wwwauth_headers;
> +	struct strbuf buf = STRBUF_INIT;
> +	const char *val;
> +	size_t val_len;
> +
> +	/*
> +	 * Header lines may not come NULL-terminated from libcurl so we must
> +	 * limit all scans to the maximum length of the header line, or leverage
> +	 * strbufs for all operations.
> +	 *
> +	 * In addition, it is possible that header values can be split over
> +	 * multiple lines as per RFC 2616 (even though this has since been
> +	 * deprecated in RFC 7230). A continuation header field value is
> +	 * identified as starting with a space or horizontal tab.
> +	 *
> +	 * The formal definition of a header field as given in RFC 2616 is:
> +	 *
> +	 *   message-header = field-name ":" [ field-value ]
> +	 *   field-name     = token
> +	 *   field-value    = *( field-content | LWS )
> +	 *   field-content  = <the OCTETs making up the field-value
> +	 *                    and consisting of either *TEXT or combinations
> +	 *                    of token, separators, and quoted-string>
> +	 */
> +
> +	/* Start of a new WWW-Authenticate header */
> +	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
> +		strbuf_add(&buf, val, val_len);
> +
> +		/*
> +		 * Strip the CRLF that should be present at the end of each
> +		 * field as well as any trailing or leading whitespace from the
> +		 * value.
> +		 */
> +		strbuf_trim(&buf);
> +
> +		strvec_push(values, buf.buf);
> +		http_auth.header_is_last_match = 1;
> +		goto exit;

OK.  We remember that we have seen the beginning of a header we are
interested in (so that we can append if it is a continuation we see
next).  Good.

> +	}
> +
> +	/*
> +	 * This line could be a continuation of the previously matched header
> +	 * field. If this is the case then we should append this value to the
> +	 * end of the previously consumed value.
> +	 */
> +	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
> +		/*
> +		 * Trim the CRLF and any leading or trailing from this line.
> +		 */
> +		strbuf_add(&buf, ptr, size);
> +		strbuf_trim(&buf);
> +
> +		/*
> +		 * At this point we should always have at least one existing
> +		 * value, even if it is empty. Do not bother appending the new
> +		 * value if this continuation header is itself empty.
> +		 */
> +		if (!values->nr) {
> +			BUG("should have at least one existing header value");

OK, we should set _is_last_match to true only after we recorded the
header that might see a continuation, so it would be a bug if we
didn't have anything there.  Good.

> +		} else if (buf.len) {
> +			char *prev = xstrdup(values->v[values->nr - 1]);
> +
> +			/* Join two non-empty values with a single space. */
> +			const char *const sp = *prev ? " " : "";
> +
> +			strvec_pop(values);
> +			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
> +			free(prev);
> +		}
> +
> +		goto exit;

Good that we are prepared to see a logical line split over more than
two lines (i.e. by not toggling _is_last_match off prematurely here).

> +	}
> +
> +	/* This is the start of a new header we don't care about */
> +	http_auth.header_is_last_match = 0;

Or what we just saw and ignored could be a continuation line of a
header we ignored.  The comment is slightly misleading.

Other than that, looking good.

Thanks.

Re: [PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2023-02-15 15:26, Junio C Hamano wrote:

> "Matthew John Cheetham via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
> 
>> According to RFC2616 Section 4.2 [1], header field names are not
>> case-sensitive meaning when collecting multiple values for the same
>> field name, we can just use the case of the first observed instance of
>> each field name and no normalisation is required.
> 
> If the names are not case-sensitive, you can choose to first
> downcase the names you see, and use that consistently, and the
> result would still be valid.  IOW, "not case-sensitive" does not at
> all mean you have to use the first observed instance without
> normalization.  You are allowed to choose such an implementation,
> but "not case-sensitive" is not a justification to choose such an
> implementation among possible implementation that would be allowed
> under the rule.

Re-reading this paragraph, it doens't really need to even be here. This was
an artefact of a time when I was storing all headers, including keys and
values. Since we're only interested now in the WWW-Authenticate header
_values_, there's no need to call out this out. Will drop this paragraph.

>> The collection of all header values matching the WWW-Authenticate
>> header is complicated by the fact that it is legal for header fields to
>> be continued over multiple lines, but libcurl only gives us one line at
>> a time.
> 
> Saying "one physical line" at a time may make it clear what you are
> pointing out as a weak point in the interface libcURL gives us (I
> think you are getting at "if they handled header folding for us and
> fed us one logical line at a time, it would have been nicer").

Logical header fields vs physical header lines is useful and clearer
terminology - I will update the commit message to reflect in the next
iteration. Thanks!

>> @@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
>>  	free(c->username);
>>  	free(c->password);
>>  	string_list_clear(&c->helpers, 0);
>> +	strvec_clear(&c->wwwauth_headers);
>>  
>>  	credential_init(c);
>>  }
>> diff --git a/credential.h b/credential.h
>> index f430e77fea4..6f2e5bc610b 100644
>> --- a/credential.h
>> +++ b/credential.h
>> @@ -2,6 +2,7 @@
>>  #define CREDENTIAL_H
>>  
>>  #include "string-list.h"
>> +#include "strvec.h"
>>  
>>  /**
>>   * The credentials API provides an abstracted way of gathering username and
>> @@ -115,6 +116,19 @@ struct credential {
>>  	 */
>>  	struct string_list helpers;
>>  
>> +	/**
>> +	 * A `strvec` of WWW-Authenticate header values. Each string
>> +	 * is the value of a WWW-Authenticate header in an HTTP response,
>> +	 * in the order they were received in the response.
>> +	 */
>> +	struct strvec wwwauth_headers;
>> +
>> +	/**
>> +	 * Internal use only. Used to keep track of split header fields
> 
> The technical term for what you call "split header" here seems to be
> "line folding" (RFC 7230, which deprecates it).
> 
>> +	 * in order to fold multiple lines into one value.
>> +	 */
>> +	unsigned header_is_last_match:1;
>> +
>>  	unsigned approved:1,
>>  		 configured:1,
>>  		 quit:1,
>> @@ -130,6 +144,7 @@ struct credential {
>>  
>>  #define CREDENTIAL_INIT { \
>>  	.helpers = STRING_LIST_INIT_DUP, \
>> +	.wwwauth_headers = STRVEC_INIT, \
>>  }
>>  
>>  /* Initialize a credential structure, setting all fields to empty. */
>> diff --git a/git-compat-util.h b/git-compat-util.h
>> index a76d0526f79..a59230564e8 100644
>> --- a/git-compat-util.h
>> +++ b/git-compat-util.h
>> @@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,
>>  	return 0;
>>  }
>>  
>> +/*
>> + * Like skip_prefix_mem, but compare case-insensitively. Note that the
>> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
>> + * characters or locale-specific conversions).
>> + */
>> +static inline int skip_iprefix_mem(const char *buf, size_t len,
>> +				   const char *prefix,
>> +				   const char **out, size_t *outlen)
>> +{
>> +	size_t prefix_len = strlen(prefix);
>> +
>> +	if (len < prefix_len)
>> +		return 0;
>> +
>> +	if (!strncasecmp(buf, prefix, prefix_len)) {
>> +		*out = buf + prefix_len;
>> +		*outlen = len - prefix_len;
>> +		return 1;
>> +	}
>> +
>> +	return 0;
>> +}
> 
> OK.
> 
>> diff --git a/http.c b/http.c
>> index 8a5ba3f4776..7a56a3db5f7 100644
>> --- a/http.c
>> +++ b/http.c
>> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
>>  	return nmemb;
>>  }
>>  
>> +/*
>> + * A folded header continuation line starts with at least one single whitespace
>> + * character. It is not a continuation line if the line is *just* a newline.
>> + * The RFC for HTTP states that CRLF is the header field line ending, but some
>> + * servers may use LF only; we accept both.
>> + */
> 
> Nice.
> 
>> +static inline int is_hdr_continuation(const char *ptr, const size_t size)
>> +{
>> +	/* totally empty line or normal header */
>> +	if (!size || !isspace(*ptr))
>> +		return 0;
> 
> obs-fold (RFC7230) begins the next line with SP or HTAB, but
> isspace() allows not just SP and HT but also CR and LF.  So
> this is a bit pessimistic but rejects what is not a folded
> continuation line reliably.
> 
>> +	/* empty line with LF line ending */
>> +	if (size == 1 && ptr[0] == '\n')
>> +		return 0;
> 
> And this is a blank line after the headers, with LF (not conforming
> but is OK).
> 
>> +	/* empty line with CRLF line ending */
>> +	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
>> +		return 0;
> 
> And this is another form of a blank line after the headers, with
> CRLF.
> 
>> +	return 1;
>> +}
> 
> After rejecting the above two "blank", it is a folded continuation
> line.  OK.
> 
> I wonder if
> 
> 	static inline int ... () {
> 	  	return (size && (*ptr == ' ' || *ptr == '\t'));
> 	}
> 
> sufficient and easier to grok, though.

You're correct. This implementation is 'more correct' and easier to grok.

>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> +{
>> +	size_t size = eltsize * nmemb;
>> +	struct strvec *values = &http_auth.wwwauth_headers;
>> +	struct strbuf buf = STRBUF_INIT;
>> +	const char *val;
>> +	size_t val_len;
>> +
>> +	/*
>> +	 * Header lines may not come NULL-terminated from libcurl so we must
>> +	 * limit all scans to the maximum length of the header line, or leverage
>> +	 * strbufs for all operations.
>> +	 *
>> +	 * In addition, it is possible that header values can be split over
>> +	 * multiple lines as per RFC 2616 (even though this has since been
>> +	 * deprecated in RFC 7230). A continuation header field value is
>> +	 * identified as starting with a space or horizontal tab.
>> +	 *
>> +	 * The formal definition of a header field as given in RFC 2616 is:
>> +	 *
>> +	 *   message-header = field-name ":" [ field-value ]
>> +	 *   field-name     = token
>> +	 *   field-value    = *( field-content | LWS )
>> +	 *   field-content  = <the OCTETs making up the field-value
>> +	 *                    and consisting of either *TEXT or combinations
>> +	 *                    of token, separators, and quoted-string>
>> +	 */
>> +
>> +	/* Start of a new WWW-Authenticate header */
>> +	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
>> +		strbuf_add(&buf, val, val_len);
>> +
>> +		/*
>> +		 * Strip the CRLF that should be present at the end of each
>> +		 * field as well as any trailing or leading whitespace from the
>> +		 * value.
>> +		 */
>> +		strbuf_trim(&buf);
>> +
>> +		strvec_push(values, buf.buf);
>> +		http_auth.header_is_last_match = 1;
>> +		goto exit;
> 
> OK.  We remember that we have seen the beginning of a header we are
> interested in (so that we can append if it is a continuation we see
> next).  Good.
> 
>> +	}
>> +
>> +	/*
>> +	 * This line could be a continuation of the previously matched header
>> +	 * field. If this is the case then we should append this value to the
>> +	 * end of the previously consumed value.
>> +	 */
>> +	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
>> +		/*
>> +		 * Trim the CRLF and any leading or trailing from this line.
>> +		 */
>> +		strbuf_add(&buf, ptr, size);
>> +		strbuf_trim(&buf);
>> +
>> +		/*
>> +		 * At this point we should always have at least one existing
>> +		 * value, even if it is empty. Do not bother appending the new
>> +		 * value if this continuation header is itself empty.
>> +		 */
>> +		if (!values->nr) {
>> +			BUG("should have at least one existing header value");
> 
> OK, we should set _is_last_match to true only after we recorded the
> header that might see a continuation, so it would be a bug if we
> didn't have anything there.  Good.
> 
>> +		} else if (buf.len) {
>> +			char *prev = xstrdup(values->v[values->nr - 1]);
>> +
>> +			/* Join two non-empty values with a single space. */
>> +			const char *const sp = *prev ? " " : "";
>> +
>> +			strvec_pop(values);
>> +			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
>> +			free(prev);
>> +		}
>> +
>> +		goto exit;
> 
> Good that we are prepared to see a logical line split over more than
> two lines (i.e. by not toggling _is_last_match off prematurely here).
> 
>> +	}
>> +
>> +	/* This is the start of a new header we don't care about */
>> +	http_auth.header_is_last_match = 0;
> 
> Or what we just saw and ignored could be a continuation line of a
> header we ignored.  The comment is slightly misleading.

I'll try and reword this to make it more accurate - we have determined
this line is not a continuation of the previous WWW-Authenticate header.

> Other than that, looking good.
> 
> Thanks.

[PATCH v9 3/3] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |   3 +
 t/t5563-simple-http-auth.sh      | 236 +++++++++++++++++++++++++++++++
 3 files changed, 257 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..f566c8ab195 100644
--- a/credential.c
+++ b/credential.c
@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
+		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i],
+				      0);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index e0682039de7..7b390a7bf95 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -68,6 +68,242 @@ test_expect_success 'access using basic auth' '
 	expect_credential_query get <<-EOF &&
 	protocol=http
 	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=baduser
+	password=wrong-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=baduser
+	password=wrong-passwd
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
+test_expect_success 'access using basic auth with extra challenges' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth mixed-case wwwauth header name' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	www-authenticate: foobar param1="value1" param2="value2"
+	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=foobar param1="value1" param2="value2"
+	wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=baSiC realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1"
+	 param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	 p=1
+	 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header empty continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
+	printf " p=1\r\n" >>$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " q=0\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Basic realm="example.com"
 	EOF
 
 	expect_credential_query store <<-EOF
-- 
gitgitgadget

[PATCH v10 0/3] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I use a small CGI shell script that acts as a
frontend to git-http-backend; simple authentication is configurable by
files.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.


Updates in v7
=============

 * Address several whitespace and arg/param list alignment issues.

 * Rethink the test-http-helper worker-mode error and result enum to be more
   simple and more informative to the nature of the error.

 * Use uintmax_t to store the Content-Length of a request in the helper
   test-http-server. Maintain a bit flag to store if we received such a
   header.

 * Return a "400 Bad Request" HTTP response if we fail to parse the request
   in the test-http-server.

 * Add test case to cover request message parsing in test-http-server.

 * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
   get_auth_module.

 * Correctly free the split strbufs created in the header parsing loop in
   test-http-server.

 * Avoid needless comparison > 0 for unsigned types.

 * Always set optional outputs to NULL if not present in test helper config
   value handling.

 * Remove an accidentally commented-out test cleanup line for one test case
   in t5556.


Updates in v8
=============

 * Drop custom HTTP test helper tool in favour of using a CGI shell script
   and Apache; avoiding the need to implement an HTTP server.

 * Avoid allocations in header reading callback unless we have a header we
   care about; act on the char* from libcurl directly rather than create a
   strbuf for each header.

 * Drop st_mult overflow guarding function in curl callback functions; we're
   not allocating memory based on the resulting value and just adds to
   potential confusion in the future.


Updates in v9
=============

 * Drop anoynmous auth tests as these cases are already covered by all other
   tests that perform HTTP interactions with a remote today.

 * In the custom auth CGI script, avoid the empty-substitution in favour of
   testing explicitly for an empty string. Also simplify some other
   conditional expressions.

 * Avoid an allocation on each wwwauth[] credential helper key-value pair
   write.

 * Various style fixups.


Updates in v10
==============

 * Style fixups.

 * Only consider space (SP ' ') and horizontal tab (HTAB '\t') when
   detecting a header continuation line, as per the latest RFC on the
   matter.

 * Update references to old HTTP specs and formal grammars of header fields
   in comments.

 * Rewording of commit messages to remove confusing comment about the case
   sensitivity of header field names - this is not relevant with the current
   iteration of the header parsing code. Also update the message around
   libcurl header support to clarify that physical header lines are
   returned, but not 'logical' header lines.

 * Reword struct credential member doc comment to clarify the purpose of
   header_is_last_match is for re-folding lines of the WWW-Authenticate
   header.

 * Reintroduce helpful comments in tests to show the origin of the 'magic'
   base64 basic auth value.

 * Use grep -F to ensure we don't do regex matching; avoid interpreting
   special characters. Remove erronous insensitive comparison flag.

Matthew John Cheetham (3):
  t5563: add tests for basic and anoymous HTTP access
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt |  19 +-
 credential.c                     |   4 +
 credential.h                     |  16 ++
 git-compat-util.h                |  23 +++
 http.c                           | 111 +++++++++++
 t/lib-httpd.sh                   |   1 +
 t/lib-httpd/apache.conf          |   6 +
 t/lib-httpd/nph-custom-auth.sh   |  39 ++++
 t/t5563-simple-http-auth.sh      | 324 +++++++++++++++++++++++++++++++
 9 files changed, 542 insertions(+), 1 deletion(-)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v10
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v10
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v9:

 1:  05449ec892b ! 1:  f3ccc53055a t5563: add tests for basic and anoymous HTTP access
     @@ t/lib-httpd/nph-custom-auth.sh (new)
      +#
      +
      +if test -n "$HTTP_AUTHORIZATION" && \
     -+	grep -qsi "^${HTTP_AUTHORIZATION}\$" "$VALID_CREDS_FILE"
     ++	grep -Fqsx "${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
      +then
      +	# Note that although git-http-backend returns a status line, it
      +	# does so using a CGI 'Status' header. Because this script is an
     @@ t/t5563-simple-http-auth.sh (new)
      +	EOF
      +'
      +
     -+set_credential_reply() {
     ++set_credential_reply () {
      +	cat >"$TRASH_DIRECTORY/$1-reply.cred"
      +}
      +
     -+expect_credential_query() {
     ++expect_credential_query () {
      +	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
      +	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
      +		 "$TRASH_DIRECTORY/$1-query.cred"
     @@ t/t5563-simple-http-auth.sh (new)
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
 2:  f3f28e508c1 ! 2:  703ac15222f http: read HTTP WWW-Authenticate response headers
     @@ Commit message
          information to credential helpers or others that would otherwise have
          been lost.
      
     -    According to RFC2616 Section 4.2 [1], header field names are not
     -    case-sensitive meaning when collecting multiple values for the same
     -    field name, we can just use the case of the first observed instance of
     -    each field name and no normalisation is required.
     -
          libcurl only provides us with the ability to read all headers recieved
          for a particular request, including any intermediate redirect requests
          or proxies. The lines returned by libcurl include HTTP status lines
     @@ Commit message
      
          The collection of all header values matching the WWW-Authenticate
          header is complicated by the fact that it is legal for header fields to
     -    be continued over multiple lines, but libcurl only gives us one line at
     -    a time.
     +    be continued over multiple lines, but libcurl only gives us each
     +    physical line a time, not each logical header. This line folding feature
     +    is deprecated in RFC 7230 [1] but older servers may still emit them, so
     +    we need to handle them.
      
          In the future [2] we may be able to leverage functions to read headers
          from libcurl itself, but as of today we must do this ourselves.
      
     -    [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
     +    [1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2
          [2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/
      
          Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
     @@ credential.h: struct credential {
      +	struct strvec wwwauth_headers;
      +
      +	/**
     -+	 * Internal use only. Used to keep track of split header fields
     -+	 * in order to fold multiple lines into one value.
     ++	 * Internal use only. Keeps track of if we previously matched against a
     ++	 * WWW-Authenticate header line in order to re-fold future continuation
     ++	 * lines into one value.
      +	 */
      +	unsigned header_is_last_match:1;
      +
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
       }
       
      +/*
     -+ * A folded header continuation line starts with at least one single whitespace
     -+ * character. It is not a continuation line if the line is *just* a newline.
     -+ * The RFC for HTTP states that CRLF is the header field line ending, but some
     -+ * servers may use LF only; we accept both.
     ++ * A folded header continuation line starts with any number of spaces or
     ++ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.
     ++ * It is not a continuation line if the line starts with any other character.
      + */
      +static inline int is_hdr_continuation(const char *ptr, const size_t size)
      +{
     -+	/* totally empty line or normal header */
     -+	if (!size || !isspace(*ptr))
     -+		return 0;
     -+
     -+	/* empty line with LF line ending */
     -+	if (size == 1 && ptr[0] == '\n')
     -+		return 0;
     -+
     -+	/* empty line with CRLF line ending */
     -+	if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
     -+		return 0;
     -+
     -+	return 1;
     ++	return size && (*ptr == ' ' || *ptr == '\t');
      +}
      +
      +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * strbufs for all operations.
      +	 *
      +	 * In addition, it is possible that header values can be split over
     -+	 * multiple lines as per RFC 2616 (even though this has since been
     -+	 * deprecated in RFC 7230). A continuation header field value is
     -+	 * identified as starting with a space or horizontal tab.
     ++	 * multiple lines as per RFC 7230. 'Line folding' has been deprecated
     ++	 * but older servers may still emit them. A continuation header field
     ++	 * value is identified as starting with a space or horizontal tab.
     ++	 *
     ++	 * The formal definition of a header field as given in RFC 7230 is:
     ++	 *
     ++	 * header-field   = field-name ":" OWS field-value OWS
      +	 *
     -+	 * The formal definition of a header field as given in RFC 2616 is:
     ++	 * field-name     = token
     ++	 * field-value    = *( field-content / obs-fold )
     ++	 * field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
     ++	 * field-vchar    = VCHAR / obs-text
      +	 *
     -+	 *   message-header = field-name ":" [ field-value ]
     -+	 *   field-name     = token
     -+	 *   field-value    = *( field-content | LWS )
     -+	 *   field-content  = <the OCTETs making up the field-value
     -+	 *                    and consisting of either *TEXT or combinations
     -+	 *                    of token, separators, and quoted-string>
     ++	 * obs-fold       = CRLF 1*( SP / HTAB )
     ++	 *                ; obsolete line folding
     ++	 *                ; see Section 3.2.4
      +	 */
      +
      +	/* Start of a new WWW-Authenticate header */
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +		goto exit;
      +	}
      +
     -+	/* This is the start of a new header we don't care about */
     ++	/* Not a continuation of a previously matched auth header line. */
      +	http_auth.header_is_last_match = 0;
      +
      +	/*
 3:  eba58c0d08d ! 3:  186da54fd3b credential: add WWW-Authenticate header to cred requests
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=wrong-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF
     @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
      +	password=secret-passwd
      +	EOF
      +
     ++	# Basic base64(alice:secret-passwd)
      +	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
      +	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
      +	EOF

-- 
gitgitgadget

[PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a test showing simple anoymous HTTP access to an unprotected
repository, that results in no credential helper invocations.
Also add a test demonstrating simple basic authentication with
simple credential helper support.

Leverage a no-parsed headers (NPH) CGI script so that we can directly
control the HTTP responses to simulate a multitude of good, bad and ugly
remote server implementations around auth.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/lib-httpd.sh                 |  1 +
 t/lib-httpd/apache.conf        |  6 +++
 t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++
 t/t5563-simple-http-auth.sh    | 82 ++++++++++++++++++++++++++++++++++
 4 files changed, 128 insertions(+)
 create mode 100755 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index 608949ea80b..2c49569f675 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -137,6 +137,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
+	install_script nph-custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 0294739a77a..76335cdb24d 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_HTTP_EXPORT_ALL
 	SetEnv GIT_PROTOCOL
 </LocationMatch>
+<LocationMatch /custom_auth/>
+	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
+	SetEnv GIT_HTTP_EXPORT_ALL
+	CGIPassAuth on
+</LocationMatch>
 ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
 ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
 ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
new file mode 100755
index 00000000000..f5345e775e4
--- /dev/null
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+VALID_CREDS_FILE=custom-auth.valid
+CHALLENGE_FILE=custom-auth.challenge
+
+#
+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
+# credential for the current request. Each line in the file is considered a
+# valid HTTP Authorization header value. For example:
+#
+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+#
+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
+# in a 401 response if no valid authentication credentials were included in the
+# request. For example:
+#
+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+# WWW-Authenticate: Basic realm="example.com"
+#
+
+if test -n "$HTTP_AUTHORIZATION" && \
+	grep -Fqsx "${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
+then
+	# Note that although git-http-backend returns a status line, it
+	# does so using a CGI 'Status' header. Because this script is an
+	# No Parsed Headers (NPH) script, we must return a real HTTP
+	# status line.
+	# This is only a test script, so we don't bother to check for
+	# the actual status from git-http-backend and always return 200.
+	echo 'HTTP/1.1 200 OK'
+	exec "$GIT_EXEC_PATH"/git-http-backend
+fi
+
+echo 'HTTP/1.1 401 Authorization Required'
+if test -f "$CHALLENGE_FILE"
+then
+	cat "$CHALLENGE_FILE"
+fi
+echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
new file mode 100755
index 00000000000..40f1b381d1b
--- /dev/null
+++ b/t/t5563-simple-http-auth.sh
@@ -0,0 +1,82 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup_credential_helper' '
+	mkdir "$TRASH_DIRECTORY/bin" &&
+	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
+	export PATH &&
+
+	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
+	write_script "$CREDENTIAL_HELPER" <<-\EOF
+	cmd=$1
+	teefile=$cmd-query.cred
+	catfile=$cmd-reply.cred
+	sed -n -e "/^$/q" -e "p" >>$teefile
+	if test "$cmd" = "get"
+	then
+		cat $catfile
+	fi
+	EOF
+'
+
+set_credential_reply () {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query () {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
+
+per_test_cleanup () {
+	rm -f *.cred &&
+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
+}
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+test_expect_success 'access using basic auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_done
-- 
gitgitgadget

Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access

[Jeff King]

On Thu, Feb 16, 2023 at 10:34:39PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> Leverage a no-parsed headers (NPH) CGI script so that we can directly
> control the HTTP responses to simulate a multitude of good, bad and ugly
> remote server implementations around auth.

Hmm, today I learned about NPH scripts.

Obviously it works here, but I have to wonder: is there a reason we need
this? AFAICT the only thing we do is set the HTTP response code, which
could also be done with a Status: header.

I.e., this passes your test:

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index ccd5f3cf82..1eadfa4bbc 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -140,7 +140,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
-	install_script nph-custom-auth.sh
+	install_script custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 2aac922376..0f9083dd6c 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -139,7 +139,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
-ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
+ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/custom-auth.sh
similarity index 94%
rename from t/lib-httpd/nph-custom-auth.sh
rename to t/lib-httpd/custom-auth.sh
index f5345e775e..8bf07e9398 100755
--- a/t/lib-httpd/nph-custom-auth.sh
+++ b/t/lib-httpd/custom-auth.sh
@@ -27,11 +27,10 @@ then
 	# status line.
 	# This is only a test script, so we don't bother to check for
 	# the actual status from git-http-backend and always return 200.
-	echo 'HTTP/1.1 200 OK'
 	exec "$GIT_EXEC_PATH"/git-http-backend
 fi
 
-echo 'HTTP/1.1 401 Authorization Required'
+echo 'Status: 401'
 if test -f "$CHALLENGE_FILE"
 then
 	cat "$CHALLENGE_FILE"


The other, more invisible thing happening behind the scenes is that
Apache isn't adding any of its usual headers. But I don't know of any
that would interfere with our goal of doing auth here. Is there some
feature you're planning where it would?

I think you could argue that it's mostly a matter of personal preference
and doesn't matter much either way. But all things being equal, I'd
usually go with the thing that is simpler and closer to the rest of the
system (e.g., I think you kill the ability of http-backend to return a
non-200 status, though I doubt it matters much in practice).

So I dunno. We are on v10 and this is arguably a nit. Mostly I'm just
curious what led you in this direction in the first place.

> ---
>  t/lib-httpd.sh                 |  1 +
>  t/lib-httpd/apache.conf        |  6 +++
>  t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++
>  t/t5563-simple-http-auth.sh    | 82 ++++++++++++++++++++++++++++++++++
>  4 files changed, 128 insertions(+)
>  create mode 100755 t/lib-httpd/nph-custom-auth.sh

Most of the other scripts here don't have an execute bit. They get one
when they're copied by instal_script in lib-httpd.sh. The exception is
error.sh, but I don't think there's any good reason for it. So probably
not a big deal either way, but another nit. :)

The rest of it all looks quite nice to me.

-Peff

Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access

[Jeff King]

On Thu, Feb 23, 2023 at 04:16:28AM -0500, Jeff King wrote:

> Hmm, today I learned about NPH scripts.
> 
> Obviously it works here, but I have to wonder: is there a reason we need
> this? AFAICT the only thing we do is set the HTTP response code, which
> could also be done with a Status: header.
> 
> I.e., this passes your test:

Having looked at patch 3 now, this also needs:

diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index 64d2acd032..afdf388677 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -37,7 +37,7 @@ expect_credential_query () {
 
 per_test_cleanup () {
 	rm -f *.cred &&
-	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.valid "$HTTPD_ROOT_PATH"/custom-auth.challenge
 }
 
 test_expect_success 'setup repository' '

or comedy ensues. But more importantly, realized why you want to use NPH
here. Apache will happily munge:

  WWW-Authenticate: foo
  WWW-Authenticate: bar

into:

  WWW-Authenticate: foo, bar

and you want to stress the parser with specific syntactic forms. So that
makes sense, and I agree NPH is the right solution here.

I think you did try to say this in the commit message as:

  Leverage a no-parsed headers (NPH) CGI script so that we can directly
  control the HTTP responses to simulate a multitude of good, bad and
  ugly remote server implementations around auth.

but I was too dense to realize quite what that meant. :)

-Peff

Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham]

On 2023-02-23 01:16, Jeff King wrote:

> On Thu, Feb 16, 2023 at 10:34:39PM +0000, Matthew John Cheetham via GitGitGadget wrote:
> 
>> Leverage a no-parsed headers (NPH) CGI script so that we can directly
>> control the HTTP responses to simulate a multitude of good, bad and ugly
>> remote server implementations around auth.
> 
> Hmm, today I learned about NPH scripts.
> 
> Obviously it works here, but I have to wonder: is there a reason we need
> this? AFAICT the only thing we do is set the HTTP response code, which
> could also be done with a Status: header.

Yep - I think you realised why in a later email. It's because Apache is
doing some CGI -> HTTP header normalisation, but we want to control the
exact byte output of WWW-Authenticate headers for exercising the new code :-)

> I.e., this passes your test:
> 
> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
> index ccd5f3cf82..1eadfa4bbc 100644
> --- a/t/lib-httpd.sh
> +++ b/t/lib-httpd.sh
> @@ -140,7 +140,7 @@ prepare_httpd() {
>  	install_script error-smart-http.sh
>  	install_script error.sh
>  	install_script apply-one-time-perl.sh
> -	install_script nph-custom-auth.sh
> +	install_script custom-auth.sh
>  
>  	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
>  
> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
> index 2aac922376..0f9083dd6c 100644
> --- a/t/lib-httpd/apache.conf
> +++ b/t/lib-httpd/apache.conf
> @@ -139,7 +139,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
>  ScriptAlias /error_smart/ error-smart-http.sh/
>  ScriptAlias /error/ error.sh/
>  ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
> -ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
> +ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1
>  <Directory ${GIT_EXEC_PATH}>
>  	Options FollowSymlinks
>  </Directory>
> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/custom-auth.sh
> similarity index 94%
> rename from t/lib-httpd/nph-custom-auth.sh
> rename to t/lib-httpd/custom-auth.sh
> index f5345e775e..8bf07e9398 100755
> --- a/t/lib-httpd/nph-custom-auth.sh
> +++ b/t/lib-httpd/custom-auth.sh
> @@ -27,11 +27,10 @@ then
>  	# status line.
>  	# This is only a test script, so we don't bother to check for
>  	# the actual status from git-http-backend and always return 200.
> -	echo 'HTTP/1.1 200 OK'
>  	exec "$GIT_EXEC_PATH"/git-http-backend
>  fi
>  
> -echo 'HTTP/1.1 401 Authorization Required'
> +echo 'Status: 401'
>  if test -f "$CHALLENGE_FILE"
>  then
>  	cat "$CHALLENGE_FILE"
> 
> 
> The other, more invisible thing happening behind the scenes is that
> Apache isn't adding any of its usual headers. But I don't know of any
> that would interfere with our goal of doing auth here. Is there some
> feature you're planning where it would?
> 
> I think you could argue that it's mostly a matter of personal preference
> and doesn't matter much either way. But all things being equal, I'd
> usually go with the thing that is simpler and closer to the rest of the
> system (e.g., I think you kill the ability of http-backend to return a
> non-200 status, though I doubt it matters much in practice).
> 
> So I dunno. We are on v10 and this is arguably a nit. Mostly I'm just
> curious what led you in this direction in the first place.
> 
>> ---
>>  t/lib-httpd.sh                 |  1 +
>>  t/lib-httpd/apache.conf        |  6 +++
>>  t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++
>>  t/t5563-simple-http-auth.sh    | 82 ++++++++++++++++++++++++++++++++++
>>  4 files changed, 128 insertions(+)
>>  create mode 100755 t/lib-httpd/nph-custom-auth.sh
> 
> Most of the other scripts here don't have an execute bit. They get one
> when they're copied by instal_script in lib-httpd.sh. The exception is
> error.sh, but I don't think there's any good reason for it. So probably
> not a big deal either way, but another nit. :)

Oh.. that's something I missed. I just added the executable bit by habit.
Will remove to match the others in lib-httpd/.

> The rest of it all looks quite nice to me.
> 
> -Peff

[PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us each
physical line a time, not each logical header. This line folding feature
is deprecated in RFC 7230 [1] but older servers may still emit them, so
we need to handle them.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c      |   1 +
 credential.h      |  16 +++++++
 git-compat-util.h |  23 ++++++++++
 http.c            | 111 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 151 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..3756a54c74d 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,20 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Keeps track of if we previously matched against a
+	 * WWW-Authenticate header line in order to re-fold future continuation
+	 * lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +145,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/git-compat-util.h b/git-compat-util.h
index a76d0526f79..a59230564e8 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,
 	return 0;
 }
 
+/*
+ * Like skip_prefix_mem, but compare case-insensitively. Note that the
+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
+ * characters or locale-specific conversions).
+ */
+static inline int skip_iprefix_mem(const char *buf, size_t len,
+				   const char *prefix,
+				   const char **out, size_t *outlen)
+{
+	size_t prefix_len = strlen(prefix);
+
+	if (len < prefix_len)
+		return 0;
+
+	if (!strncasecmp(buf, prefix, prefix_len)) {
+		*out = buf + prefix_len;
+		*outlen = len - prefix_len;
+		return 1;
+	}
+
+	return 0;
+}
+
 static inline int strtoul_ui(char const *s, int base, unsigned int *result)
 {
 	unsigned long ul;
diff --git a/http.c b/http.c
index 8a5ba3f4776..3ff570ee3a9 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,115 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+/*
+ * A folded header continuation line starts with any number of spaces or
+ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.
+ * It is not a continuation line if the line starts with any other character.
+ */
+static inline int is_hdr_continuation(const char *ptr, const size_t size)
+{
+	return size && (*ptr == ' ' || *ptr == '\t');
+}
+
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	size_t val_len;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 7230. 'Line folding' has been deprecated
+	 * but older servers may still emit them. A continuation header field
+	 * value is identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 7230 is:
+	 *
+	 * header-field   = field-name ":" OWS field-value OWS
+	 *
+	 * field-name     = token
+	 * field-value    = *( field-content / obs-fold )
+	 * field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
+	 * field-vchar    = VCHAR / obs-text
+	 *
+	 * obs-fold       = CRLF 1*( SP / HTAB )
+	 *                ; obsolete line folding
+	 *                ; see Section 3.2.4
+	 */
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
+		strbuf_add(&buf, val, val_len);
+
+		/*
+		 * Strip the CRLF that should be present at the end of each
+		 * field as well as any trailing or leading whitespace from the
+		 * value.
+		 */
+		strbuf_trim(&buf);
+
+		strvec_push(values, buf.buf);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
+		/*
+		 * Trim the CRLF and any leading or trailing from this line.
+		 */
+		strbuf_add(&buf, ptr, size);
+		strbuf_trim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* Not a continuation of a previously matched auth header line. */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (!strncasecmp(ptr, "http/", 5))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1973,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers

[Jeff King]

On Thu, Feb 16, 2023 at 10:34:40PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> +/*
> + * Like skip_prefix_mem, but compare case-insensitively. Note that the
> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
> + * characters or locale-specific conversions).
> + */
> +static inline int skip_iprefix_mem(const char *buf, size_t len,
> +				   const char *prefix,
> +				   const char **out, size_t *outlen)
> +{
> +	size_t prefix_len = strlen(prefix);
> +
> +	if (len < prefix_len)
> +		return 0;
> +
> +	if (!strncasecmp(buf, prefix, prefix_len)) {
> +		*out = buf + prefix_len;
> +		*outlen = len - prefix_len;
> +		return 1;
> +	}
> +
> +	return 0;
> +}

The comment at the top of the function seems out of date. It's using
strncasecmp(), so it probably would be locale-dependent. I think that's
probably OK, but we should probably fix the comment.

Alternatively, you could copy the tolower() loop from skip_iprefix().
Something like:
  
diff --git a/git-compat-util.h b/git-compat-util.h
index 28456241b6..f671a0ec3f 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,
 				   const char *prefix,
 				   const char **out, size_t *outlen)
 {
-	size_t prefix_len = strlen(prefix);
-
-	if (len < prefix_len)
-		return 0;
-
-	if (!strncasecmp(buf, prefix, prefix_len)) {
-		*out = buf + prefix_len;
-		*outlen = len - prefix_len;
-		return 1;
-	}
-
+	do {
+		if (!*prefix) {
+			*out = buf;
+			*outlen = len;
+			return 1;
+		}
+	} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));
 	return 0;
 }
 

looks right to me, though only lightly tested (via t5563). I'm happy
with either implementation.

> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
> [...]
> +	/*
> +	 * If this is a HTTP status line and not a header field, this signals
> +	 * a different HTTP response. libcurl writes all the output of all
> +	 * response headers of all responses, including redirects.
> +	 * We only care about the last HTTP request response's headers so clear
> +	 * the existing array.
> +	 */
> +	if (!strncasecmp(ptr, "http/", 5))
> +		strvec_clear(values);

Since "ptr" isn't NUL terminated, using strncasecmp() may walk off the
end. I think you'd need to check that there are five bytes. You could
even use skip_iprefix_mem(), though of course we'd throw away the output
values. (For strings there is also istarts_with(), but I don't think we
have a "mem" equivalent).

The rest of the parsing looks good to me.

-Peff

Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Alternatively, you could copy the tolower() loop from skip_iprefix().
> Something like:
>   
> diff --git a/git-compat-util.h b/git-compat-util.h
> index 28456241b6..f671a0ec3f 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,
>  				   const char *prefix,
>  				   const char **out, size_t *outlen)
>  {
> -	size_t prefix_len = strlen(prefix);
> -
> -	if (len < prefix_len)
> -		return 0;
> -
> -	if (!strncasecmp(buf, prefix, prefix_len)) {
> -		*out = buf + prefix_len;
> -		*outlen = len - prefix_len;
> -		return 1;
> -	}
> -
> +	do {
> +		if (!*prefix) {
> +			*out = buf;
> +			*outlen = len;
> +			return 1;
> +		}
> +	} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));
>  	return 0;
>  }
>  
>
> looks right to me, though only lightly tested (via t5563). I'm happy
> with either implementation.

Yeah, the alternative version looks clearer to me.

>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>> [...]
>> +	/*
>> +	 * If this is a HTTP status line and not a header field, this signals
>> +	 * a different HTTP response. libcurl writes all the output of all
>> +	 * response headers of all responses, including redirects.
>> +	 * We only care about the last HTTP request response's headers so clear
>> +	 * the existing array.
>> +	 */
>> +	if (!strncasecmp(ptr, "http/", 5))
>> +		strvec_clear(values);
>
> Since "ptr" isn't NUL terminated, using strncasecmp() may walk off the
> end. I think you'd need to check that there are five bytes. You could
> even use skip_iprefix_mem(), though of course we'd throw away the output
> values. (For strings there is also istarts_with(), but I don't think we
> have a "mem" equivalent).

Yuck, thank you very much for carefully reading.  I missed this one
when I queued it.

> The rest of the parsing looks good to me.

Thanks.

Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham]

On 2023-02-23 11:49, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
>> Alternatively, you could copy the tolower() loop from skip_iprefix().
>> Something like:
>>   
>> diff --git a/git-compat-util.h b/git-compat-util.h
>> index 28456241b6..f671a0ec3f 100644
>> --- a/git-compat-util.h
>> +++ b/git-compat-util.h
>> @@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,
>>  				   const char *prefix,
>>  				   const char **out, size_t *outlen)
>>  {
>> -	size_t prefix_len = strlen(prefix);
>> -
>> -	if (len < prefix_len)
>> -		return 0;
>> -
>> -	if (!strncasecmp(buf, prefix, prefix_len)) {
>> -		*out = buf + prefix_len;
>> -		*outlen = len - prefix_len;
>> -		return 1;
>> -	}
>> -
>> +	do {
>> +		if (!*prefix) {
>> +			*out = buf;
>> +			*outlen = len;
>> +			return 1;
>> +		}
>> +	} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));
>>  	return 0;
>>  }
>>  
>>
>> looks right to me, though only lightly tested (via t5563). I'm happy
>> with either implementation.
> 
> Yeah, the alternative version looks clearer to me.

Will update - thanks!

>>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
>>> [...]
>>> +	/*
>>> +	 * If this is a HTTP status line and not a header field, this signals
>>> +	 * a different HTTP response. libcurl writes all the output of all
>>> +	 * response headers of all responses, including redirects.
>>> +	 * We only care about the last HTTP request response's headers so clear
>>> +	 * the existing array.
>>> +	 */
>>> +	if (!strncasecmp(ptr, "http/", 5))
>>> +		strvec_clear(values);
>>
>> Since "ptr" isn't NUL terminated, using strncasecmp() may walk off the
>> end. I think you'd need to check that there are five bytes. You could
>> even use skip_iprefix_mem(), though of course we'd throw away the output
>> values. (For strings there is also istarts_with(), but I don't think we
>> have a "mem" equivalent).
> 
> Yuck, thank you very much for carefully reading.  I missed this one
> when I queued it.

Oops! Will update in a v11

>> The rest of the parsing looks good to me.
> 
> Thanks.

[PATCH v10 3/3] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |   3 +
 t/t5563-simple-http-auth.sh      | 242 +++++++++++++++++++++++++++++++
 3 files changed, 263 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..f566c8ab195 100644
--- a/credential.c
+++ b/credential.c
@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
+		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i],
+				      0);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index 40f1b381d1b..64d2acd0328 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -69,6 +69,248 @@ test_expect_success 'access using basic auth' '
 	expect_credential_query get <<-EOF &&
 	protocol=http
 	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=baduser
+	password=wrong-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=baduser
+	password=wrong-passwd
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
+test_expect_success 'access using basic auth with extra challenges' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth mixed-case wwwauth header name' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	www-authenticate: foobar param1="value1" param2="value2"
+	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=foobar param1="value1" param2="value2"
+	wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=baSiC realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1"
+	 param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	 p=1
+	 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header empty continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
+	printf " p=1\r\n" >>$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " q=0\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Basic realm="example.com"
 	EOF
 
 	expect_credential_query store <<-EOF
-- 
gitgitgadget

[PATCH v11 0/3] Enhance credential helper protocol to include auth headers

[Matthew John Cheetham via GitGitGadget]

Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.

In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.

Testing these new additions, I use a small CGI shell script that acts as a
frontend to git-http-backend; simple authentication is configurable by
files.


Background
==========

Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).

The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.

After a HTTP 401 response, Git would call a credential helper with the
following over standard input:

protocol=https
host=example.com


And then a credential helper would return over standard output:

protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>


Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)


Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.


Limitations
===========

Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:

 1. To generate valid credentials, additional information about the request
    (or indeed the requestee and their device) may be required. For example,
    OAuth is based around scopes. A scope, like "git.read", might be
    required to read data from the remote. However, the remote cannot tell
    the credential helper what scope is required for this request.

 2. This system is not fully extensible. Each time a new type of
    authentication (like OAuth Bearer) is invented, Git needs updates before
    credential helpers can take advantage of it (or leverage a new
    capability in libcurl).


Goals
=====

 * As a user with multiple federated cloud identities:
   
   * Reach out to a remote and have my credential helper automatically
     prompt me for the correct identity.
   * Allow credential helpers to differentiate between different authorities
     or authentication/authorization challenge types, even from the same DNS
     hostname (and without needing to use credential.useHttpPath).
   * Leverage existing authentication systems built-in to many operating
     systems and devices to boost security and reduce reliance on passwords.

 * As a Git host and/or cloud identity provider:
   
   * Enforce security policies (like requiring two-factor authentication)
     dynamically.
   * Allow integration with third party standard based identity providers in
     enterprises allowing customers to have a single plane of control for
     critical identities with access to source code.


Design Principles
=================

 * Use the existing infrastructure. Git credential helpers are an
   already-working model.
 * Follow widely-adopted time-proven open standards, avoid net new ideas in
   the authentication space.
 * Minimize knowledge of authentication in Git; maintain modularity and
   extensibility.


Proposed Changes
================

 1. Teach Git to read HTTP response headers, specifically the standard
    WWW-Authenticate (RFC 7235 Section 4.1) headers.

 2. Teach Git to include extra information about HTTP responses that require
    authentication when calling credential helpers. Specifically the
    WWW-Authenticate header information.
    
    Because the extra information forms an ordered list, and the existing
    credential helper I/O format only provides for simple key=value pairs,
    we introduce a new convention for transmitting an ordered list of
    values. Key names that are suffixed with a C-style array syntax should
    have values considered to form an order list, i.e. key[]=value, where
    the order of the key=value pairs in the stream specifies the order.
    
    For the WWW-Authenticate header values we opt to use the key wwwauth[].


Handling the WWW-Authenticate header in detail
==============================================

RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.

First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).

Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.

For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.

Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.

Git sends over standard input:

protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"


A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.

The credential helper would then return over standard output:

protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>


Note that WWW-Authenticate supports multiple challenges, either in one
header:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"


or in multiple headers:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"


These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.

An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:

...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"



Future work
===========

In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.

More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.

For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"


With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:

protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"


And then an enlightened credential helper could return over standard output:

protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF


Git would be expected to attach this authorization header to the next
request:

GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF



Why not SSH?
============

There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.


References
==========

 * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
   headers
   https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/

 * [1] [PATCH 0/3] Correct credential helper discrepancies handling input
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * [2] Git on the Server - The Protocols
   https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols

 * [3] HTTP Authentication: Basic and Digest Access Authentication
   https://datatracker.ietf.org/doc/html/rfc2617

 * [4] The Simple and Protected GSS-API Negotiation Mechanism
   https://datatracker.ietf.org/doc/html/rfc2478

 * [5] Git Credentials - Custom Helpers
   https://git-scm.com/docs/gitcredentials#_custom_helpers

 * [6] The OAuth 2.0 Authorization Framework
   https://datatracker.ietf.org/doc/html/rfc6749

 * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
   https://datatracker.ietf.org/doc/html/rfc7235

 * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
   https://datatracker.ietf.org/doc/html/rfc6750

 * [9] OpenID Connect Core 1.0
   https://openid.net/specs/openid-connect-core-1_0.html

 * [10] OpenID Connect Discovery 1.0
   https://openid.net/specs/openid-connect-discovery-1_0.html

 * [11] OpenID Connect Dynamic Client Registration 1.0
   https://openid.net/specs/openid-connect-registration-1_0.html

 * [12] Hypertext Transfer Protocol (HTTP/1.1)
   https://datatracker.ietf.org/doc/html/rfc2616


Updates from RFC
================

 * Submitted first three patches as separate submission:
   https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/

 * Various style fixes and updates to- and addition of comments.

 * Drop the explicit integer index in new 'array' style credential helper
   attrbiutes ("key[n]=value" becomes just "key[]=value").

 * Added test helper; a mini HTTP server, and several tests.


Updates in v3
=============

 * Split final patch that added the test-http-server in to several, easier
   to review patches.

 * Updated wording in git-credential.txt to clarify which side of the
   credential helper protocol is sending/receiving the new wwwauth and
   authtype attributes.


Updates in v4
=============

 * Drop authentication scheme selection authtype attribute patches to
   greatly simplify the series; auth scheme selection is punted to a future
   series. This series still allows credential helpers to generate
   credentials and intelligently select correct identities for a given auth
   challenge.


Updates in v5
=============

 * Libify parts of daemon.c and share implementation with test-http-server.

 * Clarify test-http-server Git request regex pattern and auth logic
   comments.

 * Use STD*_FILENO in place of 'magic' file descriptor numbers.

 * Use strbuf_* functions in continuation header parsing.

 * Use configuration file to configure auth for test-http-server rather than
   command-line arguments. Add ability to specify arbitrary extra headers
   that is useful for testing 'malformed' server responses.

 * Use st_mult over unchecked multiplication in http.c curl callback
   functions.

 * Fix some documentation line break issues.

 * Reorder some commits to bring in the tests and test-http-server helper
   first and, then the WWW-Authentication changes, alongside tests to cover.

 * Expose previously static strvec_push_nodup function.

 * Merge the two timeout args for test-http-server (--timeout and
   --init-timeout) that were a hang-over from the original daemon.c but are
   no longer required here.

 * Be more careful around continuation headers where they may be empty
   strings. Add more tests to cover these header types.

 * Include standard trace2 tracing calls at start of test-http-server
   helper.


Updates in v6
=============

 * Clarify the change to make logging optional in the check_dead_children()
   function during libification of daemon.c.

 * Fix missing pointer dereference bugs identified in libification of child
   process handling functions for daemon.c.

 * Add doc comments to child process handling function declarations in the
   daemon-utils.h header.

 * Align function parameter names with variable names at callsites for
   libified daemon functions.

 * Re-split out the test-http-server test helper commits in to smaller
   patches: error response handling, request parsing, http-backend
   pass-through, simple authentication, arbitrary header support.

 * Call out auth configuration file format for test-http-server test helper
   and supported options in commit messages, as well as a test to exercise
   and demonstrate these options.

 * Permit auth.token and auth.challenge to appear in any order; create the
   struct auth_module just-in-time as options for that scheme are read. This
   simplifies the configuration authoring of the test-http-server test
   helper.

 * Update tests to use auth.allowAnoymous in the patch that introduces the
   new test helper option.

 * Drop the strvec_push_nodup() commit and update the implementation of HTTP
   request header line folding to use xstrdup and strvec_pop and _pushf.

 * Use size_t instead of int in credential.c when iterating over the struct
   strvec credential members. Also drop the not required const and cast from
   the full_key definition and free.

 * Replace in-tree test-credential-helper-reply.sh test cred helper script
   with the lib-credential-helper.sh reusable 'lib' test script and shell
   functions to configure the helper behaviour.

 * Leverage sed over the while read $line loop in the test credential helper
   script.


Updates in v7
=============

 * Address several whitespace and arg/param list alignment issues.

 * Rethink the test-http-helper worker-mode error and result enum to be more
   simple and more informative to the nature of the error.

 * Use uintmax_t to store the Content-Length of a request in the helper
   test-http-server. Maintain a bit flag to store if we received such a
   header.

 * Return a "400 Bad Request" HTTP response if we fail to parse the request
   in the test-http-server.

 * Add test case to cover request message parsing in test-http-server.

 * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
   get_auth_module.

 * Correctly free the split strbufs created in the header parsing loop in
   test-http-server.

 * Avoid needless comparison > 0 for unsigned types.

 * Always set optional outputs to NULL if not present in test helper config
   value handling.

 * Remove an accidentally commented-out test cleanup line for one test case
   in t5556.


Updates in v8
=============

 * Drop custom HTTP test helper tool in favour of using a CGI shell script
   and Apache; avoiding the need to implement an HTTP server.

 * Avoid allocations in header reading callback unless we have a header we
   care about; act on the char* from libcurl directly rather than create a
   strbuf for each header.

 * Drop st_mult overflow guarding function in curl callback functions; we're
   not allocating memory based on the resulting value and just adds to
   potential confusion in the future.


Updates in v9
=============

 * Drop anoynmous auth tests as these cases are already covered by all other
   tests that perform HTTP interactions with a remote today.

 * In the custom auth CGI script, avoid the empty-substitution in favour of
   testing explicitly for an empty string. Also simplify some other
   conditional expressions.

 * Avoid an allocation on each wwwauth[] credential helper key-value pair
   write.

 * Various style fixups.


Updates in v10
==============

 * Style fixups.

 * Only consider space (SP ' ') and horizontal tab (HTAB '\t') when
   detecting a header continuation line, as per the latest RFC on the
   matter.

 * Update references to old HTTP specs and formal grammars of header fields
   in comments.

 * Rewording of commit messages to remove confusing comment about the case
   sensitivity of header field names - this is not relevant with the current
   iteration of the header parsing code. Also update the message around
   libcurl header support to clarify that physical header lines are
   returned, but not 'logical' header lines.

 * Reword struct credential member doc comment to clarify the purpose of
   header_is_last_match is for re-folding lines of the WWW-Authenticate
   header.

 * Reintroduce helpful comments in tests to show the origin of the 'magic'
   base64 basic auth value.

 * Use grep -F to ensure we don't do regex matching; avoid interpreting
   special characters. Remove erronous insensitive comparison flag.


Updates in v11
==============

 * Delete custom-auth.valid and .challenge explicitly in test cleanup.

 * Use tolower over strncasecmp in implementation of skip_iprefix_mem.

 * Use skip_iprefix_mem to match "HTTP/" header lines.

Matthew John Cheetham (3):
  t5563: add tests for basic and anoymous HTTP access
  http: read HTTP WWW-Authenticate response headers
  credential: add WWW-Authenticate header to cred requests

 Documentation/git-credential.txt |  19 +-
 credential.c                     |   4 +
 credential.h                     |  16 ++
 git-compat-util.h                |  19 ++
 http.c                           | 111 +++++++++++
 t/lib-httpd.sh                   |   1 +
 t/lib-httpd/apache.conf          |   6 +
 t/lib-httpd/nph-custom-auth.sh   |  39 ++++
 t/t5563-simple-http-auth.sh      | 325 +++++++++++++++++++++++++++++++
 9 files changed, 539 insertions(+), 1 deletion(-)
 create mode 100644 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh


base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v11
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v11
Pull-Request: https://github.com/gitgitgadget/git/pull/1352

Range-diff vs v10:

 1:  f3ccc53055a ! 1:  f7b234c4038 t5563: add tests for basic and anoymous HTTP access
     @@ t/t5563-simple-http-auth.sh (new)
      +
      +per_test_cleanup () {
      +	rm -f *.cred &&
     -+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
     ++	rm -f "$HTTPD_ROOT_PATH"/custom-auth.valid \
     ++	      "$HTTPD_ROOT_PATH"/custom-auth.challenge
      +}
      +
      +test_expect_success 'setup repository' '
 2:  703ac15222f ! 2:  3cca9ea0736 http: read HTTP WWW-Authenticate response headers
     @@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *p
      +				   const char *prefix,
      +				   const char **out, size_t *outlen)
      +{
     -+	size_t prefix_len = strlen(prefix);
     -+
     -+	if (len < prefix_len)
     -+		return 0;
     -+
     -+	if (!strncasecmp(buf, prefix, prefix_len)) {
     -+		*out = buf + prefix_len;
     -+		*outlen = len - prefix_len;
     -+		return 1;
     -+	}
     -+
     ++	do {
     ++		if (!*prefix) {
     ++			*out = buf;
     ++			*outlen = len;
     ++			return 1;
     ++		}
     ++	} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));
      +	return 0;
      +}
      +
     @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
      +	 * We only care about the last HTTP request response's headers so clear
      +	 * the existing array.
      +	 */
     -+	if (!strncasecmp(ptr, "http/", 5))
     ++	if (skip_iprefix_mem(ptr, size, "http/", &val, &val_len))
      +		strvec_clear(values);
      +
      +exit:
 3:  186da54fd3b = 3:  b774acf3896 credential: add WWW-Authenticate header to cred requests

-- 
gitgitgadget

[PATCH v11 1/3] t5563: add tests for basic and anoymous HTTP access

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add a test showing simple anoymous HTTP access to an unprotected
repository, that results in no credential helper invocations.
Also add a test demonstrating simple basic authentication with
simple credential helper support.

Leverage a no-parsed headers (NPH) CGI script so that we can directly
control the HTTP responses to simulate a multitude of good, bad and ugly
remote server implementations around auth.

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 t/lib-httpd.sh                 |  1 +
 t/lib-httpd/apache.conf        |  6 +++
 t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++
 t/t5563-simple-http-auth.sh    | 83 ++++++++++++++++++++++++++++++++++
 4 files changed, 129 insertions(+)
 create mode 100644 t/lib-httpd/nph-custom-auth.sh
 create mode 100755 t/t5563-simple-http-auth.sh

diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index 608949ea80b..2c49569f675 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -137,6 +137,7 @@ prepare_httpd() {
 	install_script error-smart-http.sh
 	install_script error.sh
 	install_script apply-one-time-perl.sh
+	install_script nph-custom-auth.sh
 
 	ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
 
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 0294739a77a..76335cdb24d 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/
 	SetEnv GIT_HTTP_EXPORT_ALL
 	SetEnv GIT_PROTOCOL
 </LocationMatch>
+<LocationMatch /custom_auth/>
+	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
+	SetEnv GIT_HTTP_EXPORT_ALL
+	CGIPassAuth on
+</LocationMatch>
 ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
 ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
 ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/
 ScriptAlias /error_smart/ error-smart-http.sh/
 ScriptAlias /error/ error.sh/
 ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
 <Directory ${GIT_EXEC_PATH}>
 	Options FollowSymlinks
 </Directory>
diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
new file mode 100644
index 00000000000..f5345e775e4
--- /dev/null
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+VALID_CREDS_FILE=custom-auth.valid
+CHALLENGE_FILE=custom-auth.challenge
+
+#
+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
+# credential for the current request. Each line in the file is considered a
+# valid HTTP Authorization header value. For example:
+#
+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+#
+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
+# in a 401 response if no valid authentication credentials were included in the
+# request. For example:
+#
+# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+# WWW-Authenticate: Basic realm="example.com"
+#
+
+if test -n "$HTTP_AUTHORIZATION" && \
+	grep -Fqsx "${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
+then
+	# Note that although git-http-backend returns a status line, it
+	# does so using a CGI 'Status' header. Because this script is an
+	# No Parsed Headers (NPH) script, we must return a real HTTP
+	# status line.
+	# This is only a test script, so we don't bother to check for
+	# the actual status from git-http-backend and always return 200.
+	echo 'HTTP/1.1 200 OK'
+	exec "$GIT_EXEC_PATH"/git-http-backend
+fi
+
+echo 'HTTP/1.1 401 Authorization Required'
+if test -f "$CHALLENGE_FILE"
+then
+	cat "$CHALLENGE_FILE"
+fi
+echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
new file mode 100755
index 00000000000..bc880bf80f9
--- /dev/null
+++ b/t/t5563-simple-http-auth.sh
@@ -0,0 +1,83 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+start_httpd
+
+test_expect_success 'setup_credential_helper' '
+	mkdir "$TRASH_DIRECTORY/bin" &&
+	PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
+	export PATH &&
+
+	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
+	write_script "$CREDENTIAL_HELPER" <<-\EOF
+	cmd=$1
+	teefile=$cmd-query.cred
+	catfile=$cmd-reply.cred
+	sed -n -e "/^$/q" -e "p" >>$teefile
+	if test "$cmd" = "get"
+	then
+		cat $catfile
+	fi
+	EOF
+'
+
+set_credential_reply () {
+	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+}
+
+expect_credential_query () {
+	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
+		 "$TRASH_DIRECTORY/$1-query.cred"
+}
+
+per_test_cleanup () {
+	rm -f *.cred &&
+	rm -f "$HTTPD_ROOT_PATH"/custom-auth.valid \
+	      "$HTTPD_ROOT_PATH"/custom-auth.challenge
+}
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+test_expect_success 'access using basic auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_done
-- 
gitgitgadget

[PATCH v11 2/3] http: read HTTP WWW-Authenticate response headers

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Read and store the HTTP WWW-Authenticate response headers made for
a particular request.

This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.

libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.

The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us each
physical line a time, not each logical header. This line folding feature
is deprecated in RFC 7230 [1] but older servers may still emit them, so
we need to handle them.

In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.

[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 credential.c      |   1 +
 credential.h      |  16 +++++++
 git-compat-util.h |  19 ++++++++
 http.c            | 111 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 147 insertions(+)

diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	string_list_clear(&c->helpers, 0);
+	strvec_clear(&c->wwwauth_headers);
 
 	credential_init(c);
 }
diff --git a/credential.h b/credential.h
index f430e77fea4..3756a54c74d 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
 #define CREDENTIAL_H
 
 #include "string-list.h"
+#include "strvec.h"
 
 /**
  * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,20 @@ struct credential {
 	 */
 	struct string_list helpers;
 
+	/**
+	 * A `strvec` of WWW-Authenticate header values. Each string
+	 * is the value of a WWW-Authenticate header in an HTTP response,
+	 * in the order they were received in the response.
+	 */
+	struct strvec wwwauth_headers;
+
+	/**
+	 * Internal use only. Keeps track of if we previously matched against a
+	 * WWW-Authenticate header line in order to re-fold future continuation
+	 * lines into one value.
+	 */
+	unsigned header_is_last_match:1;
+
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
@@ -130,6 +145,7 @@ struct credential {
 
 #define CREDENTIAL_INIT { \
 	.helpers = STRING_LIST_INIT_DUP, \
+	.wwwauth_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/git-compat-util.h b/git-compat-util.h
index a76d0526f79..62747bf6676 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1266,6 +1266,25 @@ static inline int skip_iprefix(const char *str, const char *prefix,
 	return 0;
 }
 
+/*
+ * Like skip_prefix_mem, but compare case-insensitively. Note that the
+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
+ * characters or locale-specific conversions).
+ */
+static inline int skip_iprefix_mem(const char *buf, size_t len,
+				   const char *prefix,
+				   const char **out, size_t *outlen)
+{
+	do {
+		if (!*prefix) {
+			*out = buf;
+			*outlen = len;
+			return 1;
+		}
+	} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));
+	return 0;
+}
+
 static inline int strtoul_ui(char const *s, int base, unsigned int *result)
 {
 	unsigned long ul;
diff --git a/http.c b/http.c
index 8a5ba3f4776..677266afff1 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,115 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
 	return nmemb;
 }
 
+/*
+ * A folded header continuation line starts with any number of spaces or
+ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.
+ * It is not a continuation line if the line starts with any other character.
+ */
+static inline int is_hdr_continuation(const char *ptr, const size_t size)
+{
+	return size && (*ptr == ' ' || *ptr == '\t');
+}
+
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+	size_t size = eltsize * nmemb;
+	struct strvec *values = &http_auth.wwwauth_headers;
+	struct strbuf buf = STRBUF_INIT;
+	const char *val;
+	size_t val_len;
+
+	/*
+	 * Header lines may not come NULL-terminated from libcurl so we must
+	 * limit all scans to the maximum length of the header line, or leverage
+	 * strbufs for all operations.
+	 *
+	 * In addition, it is possible that header values can be split over
+	 * multiple lines as per RFC 7230. 'Line folding' has been deprecated
+	 * but older servers may still emit them. A continuation header field
+	 * value is identified as starting with a space or horizontal tab.
+	 *
+	 * The formal definition of a header field as given in RFC 7230 is:
+	 *
+	 * header-field   = field-name ":" OWS field-value OWS
+	 *
+	 * field-name     = token
+	 * field-value    = *( field-content / obs-fold )
+	 * field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
+	 * field-vchar    = VCHAR / obs-text
+	 *
+	 * obs-fold       = CRLF 1*( SP / HTAB )
+	 *                ; obsolete line folding
+	 *                ; see Section 3.2.4
+	 */
+
+	/* Start of a new WWW-Authenticate header */
+	if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
+		strbuf_add(&buf, val, val_len);
+
+		/*
+		 * Strip the CRLF that should be present at the end of each
+		 * field as well as any trailing or leading whitespace from the
+		 * value.
+		 */
+		strbuf_trim(&buf);
+
+		strvec_push(values, buf.buf);
+		http_auth.header_is_last_match = 1;
+		goto exit;
+	}
+
+	/*
+	 * This line could be a continuation of the previously matched header
+	 * field. If this is the case then we should append this value to the
+	 * end of the previously consumed value.
+	 */
+	if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
+		/*
+		 * Trim the CRLF and any leading or trailing from this line.
+		 */
+		strbuf_add(&buf, ptr, size);
+		strbuf_trim(&buf);
+
+		/*
+		 * At this point we should always have at least one existing
+		 * value, even if it is empty. Do not bother appending the new
+		 * value if this continuation header is itself empty.
+		 */
+		if (!values->nr) {
+			BUG("should have at least one existing header value");
+		} else if (buf.len) {
+			char *prev = xstrdup(values->v[values->nr - 1]);
+
+			/* Join two non-empty values with a single space. */
+			const char *const sp = *prev ? " " : "";
+
+			strvec_pop(values);
+			strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+			free(prev);
+		}
+
+		goto exit;
+	}
+
+	/* Not a continuation of a previously matched auth header line. */
+	http_auth.header_is_last_match = 0;
+
+	/*
+	 * If this is a HTTP status line and not a header field, this signals
+	 * a different HTTP response. libcurl writes all the output of all
+	 * response headers of all responses, including redirects.
+	 * We only care about the last HTTP request response's headers so clear
+	 * the existing array.
+	 */
+	if (skip_iprefix_mem(ptr, size, "http/", &val, &val_len))
+		strvec_clear(values);
+
+exit:
+	strbuf_release(&buf);
+	return size;
+}
+
 size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
 {
 	return nmemb;
@@ -1864,6 +1973,8 @@ static int http_request(const char *url,
 					 fwrite_buffer);
 	}
 
+	curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
 	accept_language = http_get_accept_language_header();
 
 	if (accept_language)
-- 
gitgitgadget

[PATCH v11 3/3] credential: add WWW-Authenticate header to cred requests

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Add the value of the WWW-Authenticate response header to credential
requests. Credential helpers that understand and support HTTP
authentication and authorization can use this standard header (RFC 2616
Section 14.47 [1]) to generate valid credentials.

WWW-Authenticate headers can contain information pertaining to the
authority, authentication mechanism, or extra parameters/scopes that are
required.

The current I/O format for credential helpers only allows for unique
names for properties/attributes, so in order to transmit multiple header
values (with a specific order) we introduce a new convention whereby a
C-style array syntax is used in the property name to denote multiple
ordered values for the same property.

In this case we send multiple `wwwauth[]` properties where the order
that the repeated attributes appear in the conversation reflects the
order that the WWW-Authenticate headers appeared in the HTTP response.

Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.

[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47

Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
 Documentation/git-credential.txt |  19 ++-
 credential.c                     |   3 +
 t/t5563-simple-http-auth.sh      | 242 +++++++++++++++++++++++++++++++
 3 files changed, 263 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ac2818b9f66..50759153ef1 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.
 The key may contain any bytes except `=`, newline, or NUL. The value may
 contain any bytes except newline or NUL.
 
-In both cases, all bytes are treated as-is (i.e., there is no quoting,
+Attributes with keys that end with C-style array brackets `[]` can have
+multiple values. Each instance of a multi-valued attribute forms an
+ordered list of values - the order of the repeated attributes defines
+the order of the values. An empty multi-valued attribute (`key[]=\n`)
+acts to clear any previous entries and reset the list.
+
+In all cases, all bytes are treated as-is (i.e., there is no quoting,
 and one cannot transmit a value with newline or NUL in it). The list of
 attributes is terminated by a blank line or end-of-file.
 
@@ -160,6 +166,17 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`wwwauth[]`::
+
+	When an HTTP response is received by Git that includes one or more
+	'WWW-Authenticate' authentication headers, these will be passed by Git
+	to credential helpers.
++
+Each 'WWW-Authenticate' header value is passed as a multi-valued
+attribute 'wwwauth[]', where the order of the attributes is the same as
+they appear in the HTTP response. This attribute is 'one-way' from Git
+to pass additional information to credential helpers.
+
 Unrecognised attributes are silently discarded.
 
 GIT
diff --git a/credential.c b/credential.c
index 897b4679333..f566c8ab195 100644
--- a/credential.c
+++ b/credential.c
@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
+		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i],
+				      0);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index bc880bf80f9..ccf7e54b073 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -70,6 +70,248 @@ test_expect_success 'access using basic auth' '
 	expect_credential_query get <<-EOF &&
 	protocol=http
 	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=baduser
+	password=wrong-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=baduser
+	password=wrong-passwd
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
+test_expect_success 'access using basic auth with extra challenges' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth mixed-case wwwauth header name' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	www-authenticate: foobar param1="value1" param2="value2"
+	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=foobar param1="value1" param2="value2"
+	wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=baSiC realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1"
+	 param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	 p=1
+	 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header empty continuations' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
+	printf " p=1\r\n" >>$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf " q=0\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	protocol=http
+	host=$HTTPD_DEST
+	username=alice
+	password=secret-passwd
+	EOF
+'
+
+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	username=alice
+	password=secret-passwd
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	# Note that leading and trailing whitespace is important to correctly
+	# simulate a continuation/folded header.
+	printf "">$CHALLENGE &&
+	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
+	printf " \r\n" >>$CHALLENGE &&
+	printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
+	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Basic realm="example.com"
 	EOF
 
 	expect_credential_query store <<-EOF
-- 
gitgitgadget

Re: [PATCH v11 0/3] Enhance credential helper protocol to include auth headers

[Jeff King]

On Mon, Feb 27, 2023 at 05:20:17PM +0000, Matthew John Cheetham via GitGitGadget wrote:

> Updates in v11
> ==============
> 
>  * Delete custom-auth.valid and .challenge explicitly in test cleanup.
> 
>  * Use tolower over strncasecmp in implementation of skip_iprefix_mem.
> 
>  * Use skip_iprefix_mem to match "HTTP/" header lines.

Thanks, I looked over all three changes and the whole thing looks good
to me. The first one isn't strictly necessary if we're not renaming the
script, but I agree that it is probably worth being a bit more strict
when deleting in $HTTPD_ROOT_PATH.

-Peff