[PATCH] credential: new attribute oauth_refresh_token

[PATCH] credential: new attribute oauth_refresh_token

[M Hickford via GitGitGadget]

From: M Hickford <mirth.hickford@gmail.com>

Git authentication with OAuth access token is supported by every popular
Git host including GitHub, GitLab and BitBucket [1][2][3]. Credential
helpers Git Credential Manager (GCM) and git-credential-oauth generate
OAuth credentials [4][5]. Following RFC 6749, the application prints a
link for the user to authorize access in browser. A loopback redirect
communicates the response including access token to the application.

For security, RFC 6749 recommends that OAuth response also includes
expiry date and refresh token [6]. After expiry, applications can use
the refresh token to generate a new access token without user
reauthorization in browser. GitLab and BitBucket set the expiry at two
hours [2][3]. (GitHub doesn't populate expiry or refresh token.)

However the Git credential protocol has no attribute to store the OAuth
refresh token (unrecognised attributes are silently discarded). This
means that the user has to regularly reauthorize the helper in browser.
On a browserless system, this is particularly intrusive, requiring a
second device.

Introduce a new attribute oauth_refresh_token. This is especially
useful when a storage helper and a read-only OAuth helper are configured
together. Recall that `credential fill` calls each helper until it has a
non-expired password.

```
[credential]
	helper = storage  # eg. cache or osxkeychain
	helper = oauth
```

The OAuth helper can use the stored refresh token forwarded by
`credential fill` to generate a fresh access token without opening the
browser. See
https://github.com/hickford/git-credential-oauth/pull/3/files
for an implementation tested with this patch.

Add support for the new attribute to credential-cache. Eventually, I
hope to see support in other popular storage helpers.

Alternatives considered: ask helpers to store all unrecognised
attributes. This seems excessively complex for no obvious gain.
Helpers would also need extra information to distinguish between
confidential and non-confidential attributes.

Workarounds: GCM abuses the helper get/store/erase contract to store the
refresh token during credential *get* as the password for a fictitious
host [7] (I wrote this hack). This workaround is only feasible for a
monolithic helper with its own storage.

[1] https://github.blog/2012-09-21-easier-builds-and-deployments-using-git-over-https-and-oauth/
[2] https://docs.gitlab.com/ee/api/oauth2.html#access-git-over-https-with-access-token
[3] https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/#Cloning-a-repository-with-an-access-token
[4] https://github.com/GitCredentialManager/git-credential-manager
[5] https://github.com/hickford/git-credential-oauth
[6] https://datatracker.ietf.org/doc/html/rfc6749#section-5.1
[7] https://github.com/GitCredentialManager/git-credential-manager/blob/66b94e489ad8cc1982836355493e369770b30211/src/shared/GitLab/GitLabHostProvider.cs#L207

Signed-off-by: M Hickford <mirth.hickford@gmail.com>
---
    credential: new attribute oauth_refresh_token
    
    A minimal extension to the credential protocol to communicate OAuth
    refresh token. This is worthwhile because OAuth for Git authentication
    is ubiquitous. Current workaround is only feasible for a monolithic
    credential helper, discouraging interoperability between helpers.
    
    Details in commit message.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1394%2Fhickford%2Fcredential-refresh-token-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1394/hickford/credential-refresh-token-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1394

 Documentation/git-credential.txt   |  6 ++++++
 builtin/credential-cache--daemon.c |  3 +++
 credential.c                       |  6 ++++++
 credential.h                       |  1 +
 t/t0300-credentials.sh             | 18 ++++++++++++++++++
 5 files changed, 34 insertions(+)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 29d184ab824..3677d68ce61 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -150,6 +150,12 @@ Git understands the following attributes:
 	When reading credentials from helpers, `git credential fill` ignores expired
 	passwords. Represented as Unix time UTC, seconds since 1970.
 
+`oauth_refresh_token`::
+
+	An OAuth refresh token may accompany a password that is an OAuth access
+	token. Helpers must treat this attribute as confidential like the password
+	attribute. Git itself has no special behaviour for this attribute.
+
 `url`::
 
 	When this special attribute is read by `git credential`, the
diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
index 338058be7f9..f253a8278dc 100644
--- a/builtin/credential-cache--daemon.c
+++ b/builtin/credential-cache--daemon.c
@@ -130,6 +130,9 @@ static void serve_one_client(FILE *in, FILE *out)
 			if (e->item.password_expiry_utc != TIME_MAX)
 				fprintf(out, "password_expiry_utc=%"PRItime"\n",
 					e->item.password_expiry_utc);
+			if (e->item.oauth_refresh_token)
+				fprintf(out, "oauth_refresh_token=%s\n",
+					e->item.oauth_refresh_token);
 		}
 	}
 	else if (!strcmp(action.buf, "exit")) {
diff --git a/credential.c b/credential.c
index f32011343f9..3b5d3d058cf 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->oauth_refresh_token);
 	string_list_clear(&c->helpers, 0);
 
 	credential_init(c);
@@ -240,6 +241,9 @@ int credential_read(struct credential *c, FILE *fp)
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
 			if (c->password_expiry_utc == 0 || errno == ERANGE)
 				c->password_expiry_utc = TIME_MAX;
+		} else if (!strcmp(key, "oauth_refresh_token")) {
+			free(c->oauth_refresh_token);
+			c->oauth_refresh_token = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -275,6 +279,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
 		credential_write_item(fp, "password_expiry_utc", s, 0);
@@ -398,6 +403,7 @@ void credential_reject(struct credential *c)
 
 	FREE_AND_NULL(c->username);
 	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->oauth_refresh_token);
 	c->password_expiry_utc = TIME_MAX;
 	c->approved = 0;
 }
diff --git a/credential.h b/credential.h
index 935b28a70f1..b2eda372461 100644
--- a/credential.h
+++ b/credential.h
@@ -126,6 +126,7 @@ struct credential {
 	char *protocol;
 	char *host;
 	char *path;
+	char *oauth_refresh_token;
 	timestamp_t password_expiry_utc;
 };
 
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index c66d91e82d8..b49fc14a2bd 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -214,6 +214,24 @@ test_expect_success 'credential_approve stores password expiry' '
 	EOF
 '
 
+test_expect_success 'credential_approve stores oauth refresh token' '
+	check approve useless <<-\EOF
+	protocol=http
+	host=example.com
+	username=foo
+	password=bar
+	oauth_refresh_token=xyzzy
+	--
+	--
+	useless: store
+	useless: protocol=http
+	useless: host=example.com
+	useless: username=foo
+	useless: password=bar
+	useless: oauth_refresh_token=xyzzy
+	EOF
+'
+
 test_expect_success 'do not bother storing password-less credential' '
 	check approve useless <<-\EOF
 	protocol=http

base-commit: 73876f4861cd3d187a4682290ab75c9dccadbc56
-- 
gitgitgadget

[PATCH v2] credential: new attribute oauth_refresh_token

[M Hickford via GitGitGadget]

From: M Hickford <mirth.hickford@gmail.com>

Git authentication with OAuth access token is supported by every popular
Git host including GitHub, GitLab and BitBucket [1][2][3]. Credential
helpers Git Credential Manager (GCM) and git-credential-oauth generate
OAuth credentials [4][5]. Following RFC 6749, the application prints a
link for the user to authorize access in browser. A loopback redirect
communicates the response including access token to the application.

For security, RFC 6749 recommends that OAuth response also includes
expiry date and refresh token [6]. After expiry, applications can use
the refresh token to generate a new access token without user
reauthorization in browser. GitLab and BitBucket set the expiry at two
hours [2][3]. (GitHub doesn't populate expiry or refresh token.)

However the Git credential protocol has no attribute to store the OAuth
refresh token (unrecognised attributes are silently discarded). This
means that the user has to regularly reauthorize the helper in browser.
On a browserless system, this is particularly intrusive, requiring a
second device.

Introduce a new attribute oauth_refresh_token. This is especially
useful when a storage helper and a read-only OAuth helper are configured
together. Recall that `credential fill` calls each helper until it has a
non-expired password.

```
[credential]
	helper = storage  # eg. cache or osxkeychain
	helper = oauth
```

The OAuth helper can use the stored refresh token forwarded by
`credential fill` to generate a fresh access token without opening the
browser. See
https://github.com/hickford/git-credential-oauth/pull/3/files
for an implementation tested with this patch.

Add support for the new attribute to credential-cache. Eventually, I
hope to see support in other popular storage helpers.

Alternatives considered: ask helpers to store all unrecognised
attributes. This seems excessively complex for no obvious gain.
Helpers would also need extra information to distinguish between
confidential and non-confidential attributes.

Workarounds: GCM abuses the helper get/store/erase contract to store the
refresh token during credential *get* as the password for a fictitious
host [7] (I wrote this hack). This workaround is only feasible for a
monolithic helper with its own storage.

[1] https://github.blog/2012-09-21-easier-builds-and-deployments-using-git-over-https-and-oauth/
[2] https://docs.gitlab.com/ee/api/oauth2.html#access-git-over-https-with-access-token
[3] https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/#Cloning-a-repository-with-an-access-token
[4] https://github.com/GitCredentialManager/git-credential-manager
[5] https://github.com/hickford/git-credential-oauth
[6] https://datatracker.ietf.org/doc/html/rfc6749#section-5.1
[7] https://github.com/GitCredentialManager/git-credential-manager/blob/66b94e489ad8cc1982836355493e369770b30211/src/shared/GitLab/GitLabHostProvider.cs#L207

Signed-off-by: M Hickford <mirth.hickford@gmail.com>
---
    credential: new attribute oauth_refresh_token
    
    A minimal extension to the credential protocol to communicate OAuth
    refresh token. This is worthwhile because OAuth for Git authentication
    is ubiquitous. Current workaround is only feasible for a monolithic
    credential helper, discouraging interoperability between helpers.
    
    Patch v2 adds an additional test

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1394%2Fhickford%2Fcredential-refresh-token-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1394/hickford/credential-refresh-token-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1394

Range-diff vs v1:

 1:  299a26919f0 ! 1:  79e77a4426b credential: new attribute oauth_refresh_token
     @@ credential.c: void credential_clear(struct credential *c)
       	free(c->password);
      +	free(c->oauth_refresh_token);
       	string_list_clear(&c->helpers, 0);
     + 	strvec_clear(&c->wwwauth_headers);
       
     - 	credential_init(c);
      @@ credential.c: int credential_read(struct credential *c, FILE *fp)
       			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
       			if (c->password_expiry_utc == 0 || errno == ERANGE)
     @@ credential.h: struct credential {
       };
       
      
     + ## t/lib-credential.sh ##
     +@@ t/lib-credential.sh: helper_test_clean() {
     + 	reject $1 https example.com store-user
     + 	reject $1 https example.com user1
     + 	reject $1 https example.com user2
     ++	reject $1 https example.com user4
     + 	reject $1 http path.tld user
     + 	reject $1 https timeout.tld user
     + 	reject $1 https sso.tld
     +@@ t/lib-credential.sh: helper_test_timeout() {
     + 	'
     + }
     + 
     ++helper_test_oauth_refresh_token() {
     ++	HELPER=$1
     ++
     ++	test_expect_success "helper ($HELPER) stores oauth_refresh_token" '
     ++		check approve $HELPER <<-\EOF
     ++		protocol=https
     ++		host=example.com
     ++		username=user4
     ++		password=pass
     ++		oauth_refresh_token=xyzzy
     ++		EOF
     ++	'
     ++
     ++	test_expect_success "helper ($HELPER) gets oauth_refresh_token" '
     ++		check fill $HELPER <<-\EOF
     ++		protocol=https
     ++		host=example.com
     ++		username=user4
     ++		--
     ++		protocol=https
     ++		host=example.com
     ++		username=user4
     ++		password=pass
     ++		oauth_refresh_token=xyzzy
     ++		--
     ++		EOF
     ++	'
     ++}
     ++
     + write_script askpass <<\EOF
     + echo >&2 askpass: $*
     + what=$(echo $1 | cut -d" " -f1 | tr A-Z a-z | tr -cd a-z)
     +
       ## t/t0300-credentials.sh ##
      @@ t/t0300-credentials.sh: test_expect_success 'credential_approve stores password expiry' '
       	EOF
     @@ t/t0300-credentials.sh: test_expect_success 'credential_approve stores password
       test_expect_success 'do not bother storing password-less credential' '
       	check approve useless <<-\EOF
       	protocol=http
     +
     + ## t/t0301-credential-cache.sh ##
     +@@ t/t0301-credential-cache.sh: test_atexit 'git credential-cache exit'
     + 
     + # test that the daemon works with no special setup
     + helper_test cache
     ++helper_test_oauth_refresh_token cache
     + 
     + test_expect_success 'socket defaults to ~/.cache/git/credential/socket' '
     + 	test_when_finished "


 Documentation/git-credential.txt   |  6 ++++++
 builtin/credential-cache--daemon.c |  3 +++
 credential.c                       |  6 ++++++
 credential.h                       |  1 +
 t/lib-credential.sh                | 30 ++++++++++++++++++++++++++++++
 t/t0300-credentials.sh             | 18 ++++++++++++++++++
 t/t0301-credential-cache.sh        |  1 +
 7 files changed, 65 insertions(+)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 3394c036113..0e6d9e85ec7 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -156,6 +156,12 @@ Git understands the following attributes:
 	When reading credentials from helpers, `git credential fill` ignores expired
 	passwords. Represented as Unix time UTC, seconds since 1970.
 
+`oauth_refresh_token`::
+
+	An OAuth refresh token may accompany a password that is an OAuth access
+	token. Helpers must treat this attribute as confidential like the password
+	attribute. Git itself has no special behaviour for this attribute.
+
 `url`::
 
 	When this special attribute is read by `git credential`, the
diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
index 62c09a271d6..9db5f00184d 100644
--- a/builtin/credential-cache--daemon.c
+++ b/builtin/credential-cache--daemon.c
@@ -133,6 +133,9 @@ static void serve_one_client(FILE *in, FILE *out)
 			if (e->item.password_expiry_utc != TIME_MAX)
 				fprintf(out, "password_expiry_utc=%"PRItime"\n",
 					e->item.password_expiry_utc);
+			if (e->item.oauth_refresh_token)
+				fprintf(out, "oauth_refresh_token=%s\n",
+					e->item.oauth_refresh_token);
 		}
 	}
 	else if (!strcmp(action.buf, "exit")) {
diff --git a/credential.c b/credential.c
index e6417bf8804..84e32971a92 100644
--- a/credential.c
+++ b/credential.c
@@ -24,6 +24,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->oauth_refresh_token);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -243,6 +244,9 @@ int credential_read(struct credential *c, FILE *fp)
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
 			if (c->password_expiry_utc == 0 || errno == ERANGE)
 				c->password_expiry_utc = TIME_MAX;
+		} else if (!strcmp(key, "oauth_refresh_token")) {
+			free(c->oauth_refresh_token);
+			c->oauth_refresh_token = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -278,6 +282,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
 		credential_write_item(fp, "password_expiry_utc", s, 0);
@@ -403,6 +408,7 @@ void credential_reject(struct credential *c)
 
 	FREE_AND_NULL(c->username);
 	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->oauth_refresh_token);
 	c->password_expiry_utc = TIME_MAX;
 	c->approved = 0;
 }
diff --git a/credential.h b/credential.h
index 2b5958cd431..b8e2936d1dc 100644
--- a/credential.h
+++ b/credential.h
@@ -141,6 +141,7 @@ struct credential {
 	char *protocol;
 	char *host;
 	char *path;
+	char *oauth_refresh_token;
 	timestamp_t password_expiry_utc;
 };
 
diff --git a/t/lib-credential.sh b/t/lib-credential.sh
index 5ea8bc9f1dc..33666406d92 100644
--- a/t/lib-credential.sh
+++ b/t/lib-credential.sh
@@ -43,6 +43,7 @@ helper_test_clean() {
 	reject $1 https example.com store-user
 	reject $1 https example.com user1
 	reject $1 https example.com user2
+	reject $1 https example.com user4
 	reject $1 http path.tld user
 	reject $1 https timeout.tld user
 	reject $1 https sso.tld
@@ -298,6 +299,35 @@ helper_test_timeout() {
 	'
 }
 
+helper_test_oauth_refresh_token() {
+	HELPER=$1
+
+	test_expect_success "helper ($HELPER) stores oauth_refresh_token" '
+		check approve $HELPER <<-\EOF
+		protocol=https
+		host=example.com
+		username=user4
+		password=pass
+		oauth_refresh_token=xyzzy
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) gets oauth_refresh_token" '
+		check fill $HELPER <<-\EOF
+		protocol=https
+		host=example.com
+		username=user4
+		--
+		protocol=https
+		host=example.com
+		username=user4
+		password=pass
+		oauth_refresh_token=xyzzy
+		--
+		EOF
+	'
+}
+
 write_script askpass <<\EOF
 echo >&2 askpass: $*
 what=$(echo $1 | cut -d" " -f1 | tr A-Z a-z | tr -cd a-z)
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index c66d91e82d8..b49fc14a2bd 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -214,6 +214,24 @@ test_expect_success 'credential_approve stores password expiry' '
 	EOF
 '
 
+test_expect_success 'credential_approve stores oauth refresh token' '
+	check approve useless <<-\EOF
+	protocol=http
+	host=example.com
+	username=foo
+	password=bar
+	oauth_refresh_token=xyzzy
+	--
+	--
+	useless: store
+	useless: protocol=http
+	useless: host=example.com
+	useless: username=foo
+	useless: password=bar
+	useless: oauth_refresh_token=xyzzy
+	EOF
+'
+
 test_expect_success 'do not bother storing password-less credential' '
 	check approve useless <<-\EOF
 	protocol=http
diff --git a/t/t0301-credential-cache.sh b/t/t0301-credential-cache.sh
index 698b7159f03..c02a3b5969c 100755
--- a/t/t0301-credential-cache.sh
+++ b/t/t0301-credential-cache.sh
@@ -29,6 +29,7 @@ test_atexit 'git credential-cache exit'
 
 # test that the daemon works with no special setup
 helper_test cache
+helper_test_oauth_refresh_token cache
 
 test_expect_success 'socket defaults to ~/.cache/git/credential/socket' '
 	test_when_finished "

base-commit: 667fcf4e15379790f0b609d6a83d578e69f20301
-- 
gitgitgadget

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[Junio C Hamano]

"M Hickford via GitGitGadget" <gitgitgadget@gmail.com> writes:

>     Patch v2 adds an additional test

What we have below shows some differences outside the test, but the
range-diff I see locally does show that the change since v1 is
purely to the tests.

>  Documentation/git-credential.txt   |  6 ++++++
>  builtin/credential-cache--daemon.c |  3 +++
>  credential.c                       |  6 ++++++
>  credential.h                       |  1 +
>  t/lib-credential.sh                | 30 ++++++++++++++++++++++++++++++
>  t/t0300-credentials.sh             | 18 ++++++++++++++++++
>  t/t0301-credential-cache.sh        |  1 +
>  7 files changed, 65 insertions(+)

Will replace what I kept in 'seen'.

The original unfortunately did not see anybody interested enough in
the topic to review and comment.  Let's hope this time is different
;-)

THanks.

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[M Hickford]

On Fri, 21 Apr 2023 at 17:50, Junio C Hamano <gitster@pobox.com> wrote:
>
> "M Hickford via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> >     Patch v2 adds an additional test
>
> What we have below shows some differences outside the test, but the
> range-diff I see locally does show that the change since v1 is
> purely to the tests.

Thank Junio for your reply. Yes I also rebased.

>
> >  Documentation/git-credential.txt   |  6 ++++++
> >  builtin/credential-cache--daemon.c |  3 +++
> >  credential.c                       |  6 ++++++
> >  credential.h                       |  1 +
> >  t/lib-credential.sh                | 30 ++++++++++++++++++++++++++++++
> >  t/t0300-credentials.sh             | 18 ++++++++++++++++++
> >  t/t0301-credential-cache.sh        |  1 +
> >  7 files changed, 65 insertions(+)
>
> Will replace what I kept in 'seen'.
>
> The original unfortunately did not see anybody interested enough in
> the topic to review and comment.  Let's hope this time is different
> ;-)
>
> THanks.

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[Jeff King]

On Fri, Apr 21, 2023 at 09:47:59AM +0000, M Hickford via GitGitGadget wrote:

> Git authentication with OAuth access token is supported by every popular
> Git host including GitHub, GitLab and BitBucket [1][2][3]. Credential
> helpers Git Credential Manager (GCM) and git-credential-oauth generate
> OAuth credentials [4][5]. Following RFC 6749, the application prints a
> link for the user to authorize access in browser. A loopback redirect
> communicates the response including access token to the application.
> 
> For security, RFC 6749 recommends that OAuth response also includes
> expiry date and refresh token [6]. After expiry, applications can use
> the refresh token to generate a new access token without user
> reauthorization in browser. GitLab and BitBucket set the expiry at two
> hours [2][3]. (GitHub doesn't populate expiry or refresh token.)
> 
> However the Git credential protocol has no attribute to store the OAuth
> refresh token (unrecognised attributes are silently discarded). This
> means that the user has to regularly reauthorize the helper in browser.
> On a browserless system, this is particularly intrusive, requiring a
> second device.
> 
> Introduce a new attribute oauth_refresh_token. This is especially
> useful when a storage helper and a read-only OAuth helper are configured
> together. Recall that `credential fill` calls each helper until it has a
> non-expired password.
> 
> ```
> [credential]
> 	helper = storage  # eg. cache or osxkeychain
> 	helper = oauth
> ```

OK. I don't have much knowledge of OAuth, but taking the notion of "this
is a useful thing for oauth clients to store" as a given, the
implementation seems reasonable.

It does feel a bit funny, in that nothing _except_ credential-cache is
going to understand or store this thing. In general, I'd expect most
helpers doing anything as complex as oauth to just implement their own
storage layer. But I get that your goal here is trying to keep things as
composable as possible. I just wonder how successful it will be if there
is only one helper that you can actually use for storage.

> Add support for the new attribute to credential-cache. Eventually, I
> hope to see support in other popular storage helpers.

At least your optimism is documented. :)

> Alternatives considered: ask helpers to store all unrecognised
> attributes. This seems excessively complex for no obvious gain.
> Helpers would also need extra information to distinguish between
> confidential and non-confidential attributes.

I've written the "store arbitrary attributes" patch before, and it
actually is quite simple (if you neglect the "confidential" concept,
which is largely what the current protocol and helpers do). But I don't
know that it buys that much anyway, if helpers need to decide whether
and how to store these items anyway. We get support in credential-cache
for "free" because it happens to store the same in-memory struct that
Git itself does, but most other helpers would need special patches. And
because they're often interfacing with system storage, they can't
necessarily just store arbitrary items.

> diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
> index 62c09a271d6..9db5f00184d 100644
> --- a/builtin/credential-cache--daemon.c
> +++ b/builtin/credential-cache--daemon.c
> @@ -133,6 +133,9 @@ static void serve_one_client(FILE *in, FILE *out)
>  			if (e->item.password_expiry_utc != TIME_MAX)
>  				fprintf(out, "password_expiry_utc=%"PRItime"\n",
>  					e->item.password_expiry_utc);
> +			if (e->item.oauth_refresh_token)
> +				fprintf(out, "oauth_refresh_token=%s\n",
> +					e->item.oauth_refresh_token);
>  		}
>  	}
>  	else if (!strcmp(action.buf, "exit")) {

This hunk makes me wonder if the cache daemon should just be using
credential_write(), which would then support this automatically (along
with any other fields, like the expiry, or the new wwwauth stuff).

It would mean that it spits back parts of the query (like host, etc) for
a "get", rather than just username/password. That's not wrong, but is
maybe a little weird.

I can live with it as-is, certainly.

> +helper_test_oauth_refresh_token() {
> +	HELPER=$1
> +
> +	test_expect_success "helper ($HELPER) stores oauth_refresh_token" '
> +		check approve $HELPER <<-\EOF
> +		protocol=https
> +		host=example.com
> +		username=user4
> +		password=pass
> +		oauth_refresh_token=xyzzy
> +		EOF
> +	'
> +
> +	test_expect_success "helper ($HELPER) gets oauth_refresh_token" '
> +		check fill $HELPER <<-\EOF
> +		protocol=https
> +		host=example.com
> +		username=user4
> +		--
> +		protocol=https
> +		host=example.com
> +		username=user4
> +		password=pass
> +		oauth_refresh_token=xyzzy
> +		--
> +		EOF
> +	'
> +}

This is in a separate function, so normal t0303 tests won't run it (and
confuse users when they fail because they don't support this field).
Good.

Possibly t0303 should learn to optionally trigger this the way it does
with helper_test_timeout. But I'm also content to leave it until
somebody has a helper they want to test with it.

> diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> index c66d91e82d8..b49fc14a2bd 100755
> --- a/t/t0300-credentials.sh
> +++ b/t/t0300-credentials.sh
> @@ -214,6 +214,24 @@ test_expect_success 'credential_approve stores password expiry' '
>  	EOF
>  '
>  
> +test_expect_success 'credential_approve stores oauth refresh token' '
> +	check approve useless <<-\EOF
> +	protocol=http
> +	host=example.com
> +	username=foo
> +	password=bar
> +	oauth_refresh_token=xyzzy
> +	--
> +	--
> +	useless: store
> +	useless: protocol=http
> +	useless: host=example.com
> +	useless: username=foo
> +	useless: password=bar
> +	useless: oauth_refresh_token=xyzzy
> +	EOF
> +'

Makes sense. This file is just checking how Git passes the data around
between helpers.

> diff --git a/t/t0301-credential-cache.sh b/t/t0301-credential-cache.sh
> index 698b7159f03..c02a3b5969c 100755
> --- a/t/t0301-credential-cache.sh
> +++ b/t/t0301-credential-cache.sh
> @@ -29,6 +29,7 @@ test_atexit 'git credential-cache exit'
>  
>  # test that the daemon works with no special setup
>  helper_test cache
> +helper_test_oauth_refresh_token cache
>  
>  test_expect_success 'socket defaults to ~/.cache/git/credential/socket' '
>  	test_when_finished "

And this one is just testing credential-cache, which we know supports
the feature. Good.

So the patch looks like it correctly implements the intent. I don't feel
strongly about what the patch is trying to do, but it's not a problem
area I know a lot about or use myself. So if it's doing something useful
for you, that sounds reasonable to me. :)

-Peff

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[M Hickford]

On Tue, 25 Apr 2023 at 07:48, Jeff King <peff@peff.net> wrote:
>
> On Fri, Apr 21, 2023 at 09:47:59AM +0000, M Hickford via GitGitGadget wrote:
>
> > Git authentication with OAuth access token is supported by every popular
> > Git host including GitHub, GitLab and BitBucket [1][2][3]. Credential
> > helpers Git Credential Manager (GCM) and git-credential-oauth generate
> > OAuth credentials [4][5]. Following RFC 6749, the application prints a
> > link for the user to authorize access in browser. A loopback redirect
> > communicates the response including access token to the application.
> >
> > For security, RFC 6749 recommends that OAuth response also includes
> > expiry date and refresh token [6]. After expiry, applications can use
> > the refresh token to generate a new access token without user
> > reauthorization in browser. GitLab and BitBucket set the expiry at two
> > hours [2][3]. (GitHub doesn't populate expiry or refresh token.)
> >
> > However the Git credential protocol has no attribute to store the OAuth
> > refresh token (unrecognised attributes are silently discarded). This
> > means that the user has to regularly reauthorize the helper in browser.
> > On a browserless system, this is particularly intrusive, requiring a
> > second device.
> >
> > Introduce a new attribute oauth_refresh_token. This is especially
> > useful when a storage helper and a read-only OAuth helper are configured
> > together. Recall that `credential fill` calls each helper until it has a
> > non-expired password.
> >
> > ```
> > [credential]
> >       helper = storage  # eg. cache or osxkeychain
> >       helper = oauth
> > ```
>
> OK. I don't have much knowledge of OAuth, but taking the notion of "this
> is a useful thing for oauth clients to store" as a given, the
> implementation seems reasonable.
>
> It does feel a bit funny, in that nothing _except_ credential-cache is
> going to understand or store this thing. In general, I'd expect most
> helpers doing anything as complex as oauth to just implement their own
> storage layer. But I get that your goal here is trying to keep things as
> composable as possible. I just wonder how successful it will be if there
> is only one helper that you can actually use for storage.
>
> > Add support for the new attribute to credential-cache. Eventually, I
> > hope to see support in other popular storage helpers.
>
> At least your optimism is documented. :)

I have a draft patch for credential-libsecret
https://github.com/gitgitgadget/git/pull/1524 which I could add to
this patch series if you like. Helpers credential-wincred and
credential-osxkeychain would be easy to update following the same
approach.

>
> > Alternatives considered: ask helpers to store all unrecognised
> > attributes. This seems excessively complex for no obvious gain.
> > Helpers would also need extra information to distinguish between
> > confidential and non-confidential attributes.
>
> I've written the "store arbitrary attributes" patch before, and it
> actually is quite simple (if you neglect the "confidential" concept,
> which is largely what the current protocol and helpers do). But I don't
> know that it buys that much anyway, if helpers need to decide whether
> and how to store these items anyway. We get support in credential-cache
> for "free" because it happens to store the same in-memory struct that
> Git itself does, but most other helpers would need special patches. And
> because they're often interfacing with system storage, they can't
> necessarily just store arbitrary items.
>
> > diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
> > index 62c09a271d6..9db5f00184d 100644
> > --- a/builtin/credential-cache--daemon.c
> > +++ b/builtin/credential-cache--daemon.c
> > @@ -133,6 +133,9 @@ static void serve_one_client(FILE *in, FILE *out)
> >                       if (e->item.password_expiry_utc != TIME_MAX)
> >                               fprintf(out, "password_expiry_utc=%"PRItime"\n",
> >                                       e->item.password_expiry_utc);
> > +                     if (e->item.oauth_refresh_token)
> > +                             fprintf(out, "oauth_refresh_token=%s\n",
> > +                                     e->item.oauth_refresh_token);
> >               }
> >       }
> >       else if (!strcmp(action.buf, "exit")) {
>
> This hunk makes me wonder if the cache daemon should just be using
> credential_write(), which would then support this automatically (along
> with any other fields, like the expiry, or the new wwwauth stuff).
>
> It would mean that it spits back parts of the query (like host, etc) for
> a "get", rather than just username/password. That's not wrong, but is
> maybe a little weird.
>
> I can live with it as-is, certainly.
>
> > +helper_test_oauth_refresh_token() {
> > +     HELPER=$1
> > +
> > +     test_expect_success "helper ($HELPER) stores oauth_refresh_token" '
> > +             check approve $HELPER <<-\EOF
> > +             protocol=https
> > +             host=example.com
> > +             username=user4
> > +             password=pass
> > +             oauth_refresh_token=xyzzy
> > +             EOF
> > +     '
> > +
> > +     test_expect_success "helper ($HELPER) gets oauth_refresh_token" '
> > +             check fill $HELPER <<-\EOF
> > +             protocol=https
> > +             host=example.com
> > +             username=user4
> > +             --
> > +             protocol=https
> > +             host=example.com
> > +             username=user4
> > +             password=pass
> > +             oauth_refresh_token=xyzzy
> > +             --
> > +             EOF
> > +     '
> > +}
>
> This is in a separate function, so normal t0303 tests won't run it (and
> confuse users when they fail because they don't support this field).
> Good.
>
> Possibly t0303 should learn to optionally trigger this the way it does
> with helper_test_timeout. But I'm also content to leave it until
> somebody has a helper they want to test with it.
>
> > diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> > index c66d91e82d8..b49fc14a2bd 100755
> > --- a/t/t0300-credentials.sh
> > +++ b/t/t0300-credentials.sh
> > @@ -214,6 +214,24 @@ test_expect_success 'credential_approve stores password expiry' '
> >       EOF
> >  '
> >
> > +test_expect_success 'credential_approve stores oauth refresh token' '
> > +     check approve useless <<-\EOF
> > +     protocol=http
> > +     host=example.com
> > +     username=foo
> > +     password=bar
> > +     oauth_refresh_token=xyzzy
> > +     --
> > +     --
> > +     useless: store
> > +     useless: protocol=http
> > +     useless: host=example.com
> > +     useless: username=foo
> > +     useless: password=bar
> > +     useless: oauth_refresh_token=xyzzy
> > +     EOF
> > +'
>
> Makes sense. This file is just checking how Git passes the data around
> between helpers.
>
> > diff --git a/t/t0301-credential-cache.sh b/t/t0301-credential-cache.sh
> > index 698b7159f03..c02a3b5969c 100755
> > --- a/t/t0301-credential-cache.sh
> > +++ b/t/t0301-credential-cache.sh
> > @@ -29,6 +29,7 @@ test_atexit 'git credential-cache exit'
> >
> >  # test that the daemon works with no special setup
> >  helper_test cache
> > +helper_test_oauth_refresh_token cache
> >
> >  test_expect_success 'socket defaults to ~/.cache/git/credential/socket' '
> >       test_when_finished "
>
> And this one is just testing credential-cache, which we know supports
> the feature. Good.
>
> So the patch looks like it correctly implements the intent. I don't feel
> strongly about what the patch is trying to do, but it's not a problem
> area I know a lot about or use myself. So if it's doing something useful
> for you, that sounds reasonable to me. :)

Thanks Jeff for the review

>
> -Peff

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[Jeff King]

On Tue, Apr 25, 2023 at 08:19:40PM +0100, M Hickford wrote:

> > > Add support for the new attribute to credential-cache. Eventually, I
> > > hope to see support in other popular storage helpers.
> >
> > At least your optimism is documented. :)
> 
> I have a draft patch for credential-libsecret
> https://github.com/gitgitgadget/git/pull/1524 which I could add to
> this patch series if you like. Helpers credential-wincred and
> credential-osxkeychain would be easy to update following the same
> approach.

Good, thanks for doing that work. I don't think it needs to be part of
this series. Knowing that it's being worked on gives a sense that this
patch is going in a useful direction.

I do find the "shove extra data into the password field" strategy a
little gross, but that may be the best that can be done when interfacing
with existing stores. And at least the details are all contained inside
the helper itself (though I guess a user looking at it with other wallet
management tools would see the weird field).

-Peff

Re: [PATCH v2] credential: new attribute oauth_refresh_token

[Felipe Contreras]

Jeff King wrote:
> On Fri, Apr 21, 2023 at 09:47:59AM +0000, M Hickford via GitGitGadget wrote:
> 
> > Git authentication with OAuth access token is supported by every popular
> > Git host including GitHub, GitLab and BitBucket [1][2][3]. Credential
> > helpers Git Credential Manager (GCM) and git-credential-oauth generate
> > OAuth credentials [4][5]. Following RFC 6749, the application prints a
> > link for the user to authorize access in browser. A loopback redirect
> > communicates the response including access token to the application.
> > 
> > For security, RFC 6749 recommends that OAuth response also includes
> > expiry date and refresh token [6]. After expiry, applications can use
> > the refresh token to generate a new access token without user
> > reauthorization in browser. GitLab and BitBucket set the expiry at two
> > hours [2][3]. (GitHub doesn't populate expiry or refresh token.)
> > 
> > However the Git credential protocol has no attribute to store the OAuth
> > refresh token (unrecognised attributes are silently discarded). This
> > means that the user has to regularly reauthorize the helper in browser.
> > On a browserless system, this is particularly intrusive, requiring a
> > second device.
> > 
> > Introduce a new attribute oauth_refresh_token. This is especially
> > useful when a storage helper and a read-only OAuth helper are configured
> > together. Recall that `credential fill` calls each helper until it has a
> > non-expired password.
> > 
> > ```
> > [credential]
> > 	helper = storage  # eg. cache or osxkeychain
> > 	helper = oauth
> > ```
> 
> OK. I don't have much knowledge of OAuth, but taking the notion of "this
> is a useful thing for oauth clients to store" as a given, the
> implementation seems reasonable.

I don't think this is specific to OAuth, I've seen different authorization
methods use something like that.

In general you just need two variables: the refresh token, and the expiration
time of the refresh token. The logic is very simple: if the refresh token has
expired, you ask for a new one. This way you don't have to go through the
authorization process again.

-- 
Felipe Contreras