[PATCH] Improvement to only call Git Credential Helper once

[PATCH] Improvement to only call Git Credential Helper once

[Kyle Hubert]

When calling the Git Credential Helper that is set in the git config,
the get command can return a credential. Git immediately turns around
and calls the store command, even though that credential was just
retrieved by the Helper. This creates two side effects. First of all,
if the Helper requires a passphrase, the user has to type it in
twice. Secondly, if the user has a number of helpers, this retrieves
the credential from one service and writes it to all services.

This commit introduces a new field in the credential struct that
detects when the credential was retrieved using the Helper, and early
exits when called to store the credential.
---
 credential.c | 8 +++++++-
 credential.h | 3 ++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/credential.c b/credential.c
index 62be651b0..79bf62d49 100644
--- a/credential.c
+++ b/credential.c
@@ -280,8 +280,10 @@ void credential_fill(struct credential *c)

  for (i = 0; i < c->helpers.nr; i++) {
  credential_do(c, c->helpers.items[i].string, "get");
- if (c->username && c->password)
+ if (c->username && c->password) {
+ c->retrieved = 1;
  return;
+ }
  if (c->quit)
  die("credential helper '%s' told us to quit",
      c->helpers.items[i].string);
@@ -300,6 +302,10 @@ void credential_approve(struct credential *c)
  return;
  if (!c->username || !c->password)
  return;
+ if (c->retrieved) {
+ c->approved = 1;
+ return;
+ }

  credential_apply_config(c);

diff --git a/credential.h b/credential.h
index 6b0cd16be..d99df2f52 100644
--- a/credential.h
+++ b/credential.h
@@ -8,7 +8,8 @@ struct credential {
  unsigned approved:1,
  configured:1,
  quit:1,
- use_http_path:1;
+ use_http_path:1,
+ retrieved:1;

  char *username;
  char *password;

Re: [PATCH] Improvement to only call Git Credential Helper once

[Kyle Hubert]

Sorry, this patch is damaged. I'm moving to `git send-email` now.

-Kyle

On Fri, Sep 28, 2018 at 11:10 AM Kyle Hubert <khubert@gmail.com> wrote:
>
> When calling the Git Credential Helper that is set in the git config,
> the get command can return a credential. Git immediately turns around
> and calls the store command, even though that credential was just
> retrieved by the Helper. This creates two side effects. First of all,
> if the Helper requires a passphrase, the user has to type it in
> twice. Secondly, if the user has a number of helpers, this retrieves
> the credential from one service and writes it to all services.
>
> This commit introduces a new field in the credential struct that
> detects when the credential was retrieved using the Helper, and early
> exits when called to store the credential.
> ---
>  credential.c | 8 +++++++-
>  credential.h | 3 ++-
>  2 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/credential.c b/credential.c
> index 62be651b0..79bf62d49 100644
> --- a/credential.c
> +++ b/credential.c
> @@ -280,8 +280,10 @@ void credential_fill(struct credential *c)
>
>   for (i = 0; i < c->helpers.nr; i++) {
>   credential_do(c, c->helpers.items[i].string, "get");
> - if (c->username && c->password)
> + if (c->username && c->password) {
> + c->retrieved = 1;
>   return;
> + }
>   if (c->quit)
>   die("credential helper '%s' told us to quit",
>       c->helpers.items[i].string);
> @@ -300,6 +302,10 @@ void credential_approve(struct credential *c)
>   return;
>   if (!c->username || !c->password)
>   return;
> + if (c->retrieved) {
> + c->approved = 1;
> + return;
> + }
>
>   credential_apply_config(c);
>
> diff --git a/credential.h b/credential.h
> index 6b0cd16be..d99df2f52 100644
> --- a/credential.h
> +++ b/credential.h
> @@ -8,7 +8,8 @@ struct credential {
>   unsigned approved:1,
>   configured:1,
>   quit:1,
> - use_http_path:1;
> + use_http_path:1,
> + retrieved:1;
>
>   char *username;
>   char *password;

[PATCH] Improvement to only call Git Credential Helper once

[Kyle Hubert]

When calling the Git Credential Helper that is set in the git config,
the get command can return a credential. Git immediately turns around
and calls the store command, even though that credential was just
retrieved by the Helper. This creates two side effects. First of all,
if the Helper requires a passphrase, the user has to type it in
twice. Secondly, if the user has a number of helpers, this retrieves
the credential from one service and writes it to all services.

This commit introduces a new field in the credential struct that
detects when the credential was retrieved using the Helper, and early
exits when called to store the credential.
---
 credential.c | 8 +++++++-
 credential.h | 3 ++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/credential.c b/credential.c
index 62be651b0..79bf62d49 100644
--- a/credential.c
+++ b/credential.c
@@ -280,8 +280,10 @@ void credential_fill(struct credential *c)
 
 	for (i = 0; i < c->helpers.nr; i++) {
 		credential_do(c, c->helpers.items[i].string, "get");
-		if (c->username && c->password)
+		if (c->username && c->password) {
+			c->retrieved = 1;
 			return;
+		}
 		if (c->quit)
 			die("credential helper '%s' told us to quit",
 			    c->helpers.items[i].string);
@@ -300,6 +302,10 @@ void credential_approve(struct credential *c)
 		return;
 	if (!c->username || !c->password)
 		return;
+	if (c->retrieved) {
+		c->approved = 1;
+		return;
+	}
 
 	credential_apply_config(c);
 
diff --git a/credential.h b/credential.h
index 6b0cd16be..d99df2f52 100644
--- a/credential.h
+++ b/credential.h
@@ -8,7 +8,8 @@ struct credential {
 	unsigned approved:1,
 		 configured:1,
 		 quit:1,
-		 use_http_path:1;
+		 use_http_path:1,
+		 retrieved:1;
 
 	char *username;
 	char *password;
-- 
2.11.0

Re: [PATCH] Improvement to only call Git Credential Helper once

[Junio C Hamano]

Kyle Hubert <khubert@gmail.com> writes:

> Subject: Re: [PATCH] Improvement to only call Git Credential Helper once

Nobody will send in a patch to worsen things, so phrases like
"Improvement to" that convey no useful information has no place on
the title.

There probably are multiple ways that credential helpers are not
called just once and many of them probably are legit (e.g. "get" is
not the only request one helper can receive).  It is unclear why
"only call helper once" is an improvement unless the reader reads
more, which means the title could be a lot more improved.

	Side note: this matters because in "git shortlog --no-merges"
	the title is the only thing you can tell your readers what
	your contribution was about.

> When calling the Git Credential Helper that is set in the git config,
> the get command can return a credential. Git immediately turns around
> and calls the store command, even though that credential was just
> retrieved by the Helper. 

Good summary of the current behaviour.  A paragraph break here would
make the result easier to read.

> This creates two side effects. First of all,
> if the Helper requires a passphrase, the user has to type it in
> twice.

Hmph, because...?  If I am reading the flow correctly, an
application would

 - call credential_fill(), which returns when both username and
   password are obtained by a "get" request to one of the helpers.

 - use the credential to authenticate with a service and find that
   it was accepted.

 - call credential_approve(), which does "store" to all the helpers.

Where does the "twice" come from?

Ah, that is not between the application and the service, but between
the helper and the user you are required to "unlock" the helper?

OK, that wish makes sense.

It does not make much sense to ask helper A for credential and then
tell it to write it back the same thing.

HOWEVER.  Let me play a devil's advocate.

The "store" does not have to necessarily mean "write it back", no?

Imagine a helper that is connected to an OTP password device that
gives a different passcode every time button A is pressed, and there
are two other buttons B to tell the device that the password was
accepted and C to tell the device that the password was rejected
(i.e. we are out of sync).  "get" would press button A and read the
output, "store" would press button B and "erase" would press button
C, I would imagine.  With the current credential.c framework, you
can construct such a helper.  The proposed patch that stops calling
"store" unconditionally makes it impossible to build.

> Secondly, if the user has a number of helpers, this retrieves
> the credential from one service and writes it to all services.

It is unclear why you think it is a bad thing.  You need to
elaborate.

On the other hand, I can think of a case to illustrate why it is a
bad idea to unconditionally stop calling "store" to other helpers.
If one helper is a read-only "encrypted on disk" one, you may want
to require passphrase to "decrypt" to implement the "get" request to
the helper.  You would then overlay a "stay only in-core for a short
time" helper and give higher priority to it.

By doing so, on the first "get" request will ask the in-core one,
which says "I dunno", then the encrypted-on-disk one interacts with
the end-user and gives the credential.  The current code "store"s to
the in-core one as well as the encrypted-on-disk one, and second and
subseqhent "get" request can be served from that in-core helper.

	Side note: and the "store" to encrypted-on-disk one may not
	even need passphrase, even if "get" from it may need one.

"We got the credential from some helper, so we won't call store"
makes it impossible to build such an arrangement.

The above is a devil's advocate response in that I do not mean to
say that your proposed workaround does not solve *your* immediate
need, but to point out that you are closing many doors for needs
other people would have, or needs they already satisfy by taking
advantage of the current behaviour the proposed patch is breaking.

So, I dunno.  I certainly do not think it is a bad idea to stop
feeding _other_ helpers.  I also do not think it is a good idea to
unconditionally stop calling "store" to the same helper, but I can
see the benefit for having an option to skip "store" to the same
helper.  I am not sure if there should be an option to stop feeding
other helpers.

Thanks.

Re: [PATCH] Improvement to only call Git Credential Helper once

[Kyle Hubert]

Thank you for the review, commenting inline.

On Fri, Sep 28, 2018 at 3:29 PM Junio C Hamano <gitster@pobox.com> wrote:
> Kyle Hubert <khubert@gmail.com> writes:
>
> > Subject: Re: [PATCH] Improvement to only call Git Credential Helper once
>
> Nobody will send in a patch to worsen things, so phrases like
> "Improvement to" that convey no useful information has no place on
> the title.
>
> There probably are multiple ways that credential helpers are not
> called just once and many of them probably are legit (e.g. "get" is
> not the only request one helper can receive).  It is unclear why
> "only call helper once" is an improvement unless the reader reads
> more, which means the title could be a lot more improved.
>
>         Side note: this matters because in "git shortlog --no-merges"
>         the title is the only thing you can tell your readers what
>         your contribution was about.

Yes, thank you. I just joined the mailing list. I'm getting a sense
for the style here. If this patch moves forward, I will rewrite the
title.

> > When calling the Git Credential Helper that is set in the git config,
> > the get command can return a credential. Git immediately turns around
> > and calls the store command, even though that credential was just
> > retrieved by the Helper.
>
> Good summary of the current behaviour.  A paragraph break here would
> make the result easier to read.

Check.

> > This creates two side effects. First of all,
> > if the Helper requires a passphrase, the user has to type it in
> > twice.
>
> Hmph, because...?  If I am reading the flow correctly, an
> application would
>
>  - call credential_fill(), which returns when both username and
>    password are obtained by a "get" request to one of the helpers.
>
>  - use the credential to authenticate with a service and find that
>    it was accepted.
>
>  - call credential_approve(), which does "store" to all the helpers.
>
> Where does the "twice" come from?
>
> Ah, that is not between the application and the service, but between
> the helper and the user you are required to "unlock" the helper?
>
> OK, that wish makes sense.
>
> It does not make much sense to ask helper A for credential and then
> tell it to write it back the same thing.
>
> HOWEVER.  Let me play a devil's advocate.
>
> The "store" does not have to necessarily mean "write it back", no?
>
> Imagine a helper that is connected to an OTP password device that
> gives a different passcode every time button A is pressed, and there
> are two other buttons B to tell the device that the password was
> accepted and C to tell the device that the password was rejected
> (i.e. we are out of sync).  "get" would press button A and read the
> output, "store" would press button B and "erase" would press button
> C, I would imagine.  With the current credential.c framework, you
> can construct such a helper.  The proposed patch that stops calling
> "store" unconditionally makes it impossible to build.

This example helper would require knowing external state between
button pushes. What about aborting the operation? Regardless of the
example, I can understand your concern. If a helper depended on
receiving confirmation and rejection, this change would break that
behavior.

The patch is intended to address the common problem of a multi-user
system running on a server in headless mode, in other words, without
libsecret available via DBus. As such, this patch could eliminate
having to double type passwords for every git operation accessing the
stored credential.

> > Secondly, if the user has a number of helpers, this retrieves
> > the credential from one service and writes it to all services.
>
> It is unclear why you think it is a bad thing.  You need to
> elaborate.
>
> On the other hand, I can think of a case to illustrate why it is a
> bad idea to unconditionally stop calling "store" to other helpers.
> If one helper is a read-only "encrypted on disk" one, you may want
> to require passphrase to "decrypt" to implement the "get" request to
> the helper.  You would then overlay a "stay only in-core for a short
> time" helper and give higher priority to it.
>
> By doing so, on the first "get" request will ask the in-core one,
> which says "I dunno", then the encrypted-on-disk one interacts with
> the end-user and gives the credential.  The current code "store"s to
> the in-core one as well as the encrypted-on-disk one, and second and
> subseqhent "get" request can be served from that in-core helper.
>
>         Side note: and the "store" to encrypted-on-disk one may not
>         even need passphrase, even if "get" from it may need one.
>
> "We got the credential from some helper, so we won't call store"
> makes it impossible to build such an arrangement.
>
> The above is a devil's advocate response in that I do not mean to
> say that your proposed workaround does not solve *your* immediate
> need, but to point out that you are closing many doors for needs
> other people would have, or needs they already satisfy by taking
> advantage of the current behaviour the proposed patch is breaking.

This seems very valid, I like the configuration your propose. The
argument for not storing to all helpers is to prevent leaking
credentials across many key stores. In some cases, a developer can
short cut looking up the plain text credential, and instead copy the
encrypted credential to a new server. This behavior can lead to
copying to a local development machine which is not headless, and does
have an OS keychain available. If allowed to write the secret
credential to all available helpers, this will leak the credential to
the developers local machine.

> So, I dunno.  I certainly do not think it is a bad idea to stop
> feeding _other_ helpers.  I also do not think it is a good idea to
> unconditionally stop calling "store" to the same helper, but I can
> see the benefit for having an option to skip "store" to the same
> helper.  I am not sure if there should be an option to stop feeding
> other helpers.

I understand your thinking. Is there code you can point me to that
would allow referencing an option to specific helpers? Maybe it could
be reasoned that configuration could dictate not leaking credentials,
and retrieval doesn't need confirmation.

Thank you for your review,
-Kyle

Re: [PATCH] Improvement to only call Git Credential Helper once

[Jeff King]

On Fri, Sep 28, 2018 at 12:37:16PM -0400, Kyle Hubert wrote:

> When calling the Git Credential Helper that is set in the git config,
> the get command can return a credential. Git immediately turns around
> and calls the store command, even though that credential was just
> retrieved by the Helper. This creates two side effects. First of all,
> if the Helper requires a passphrase, the user has to type it in
> twice. Secondly, if the user has a number of helpers, this retrieves
> the credential from one service and writes it to all services.
> 
> This commit introduces a new field in the credential struct that
> detects when the credential was retrieved using the Helper, and early
> exits when called to store the credential.

Wow, what's old is new again. Here's more or less the same patch from
2012:

  https://public-inbox.org/git/20120407033417.GA13914@sigill.intra.peff.net/

Unfortunately, some people seem to rely on this multi-helper behavior. I
recommend reading the whole thread, as there are some interesting bits
in it (that I had always meant to return to, but somehow 6 years went
by).

I'm not entirely opposed to breaking the current behavior in the name of
better security (namely not unexpectedly propagating credentials), but
it would be nice if we provided better tools for people to let their
helpers interact (like the credential-wrap thing I showed in that
thread).

> ---
>  credential.c | 8 +++++++-
>  credential.h | 3 ++-
>  2 files changed, 9 insertions(+), 2 deletions(-)

I know your patch is right, because it's almost identical to mine. :)
(Mine didn't use the "retrieved" flag in the middle, but just set
"approved" directly).

If we do go this route, though, we might want to steal the test from
that earlier round.

-Peff

Re: [PATCH] Improvement to only call Git Credential Helper once

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Wow, what's old is new again. Here's more or less the same patch from
> 2012:
>
>   https://public-inbox.org/git/20120407033417.GA13914@sigill.intra.peff.net/
>
> Unfortunately, some people seem to rely on this multi-helper behavior. I
> recommend reading the whole thread, as there are some interesting bits
> in it (that I had always meant to return to, but somehow 6 years went
> by).

After reading that thread again, my version of summary is that

 - storing the credential obtained from a helper back to the same
   helper may be pointless at best and may incur end-user
   interaction (e.g. asking for symmetric encryption key before the
   data hits the disk), but it can be used as a "we used what you
   gave us successfully" ping-back signal.  We may not have designed
   approve() to do "store" back to the same helper by default and
   instead to do so only when asked by the helper, if we knew
   better.  But an unconditional change of behaviour will break
   existing users and helpers.

 - storing the credential obtained from a helper to a different
   helper may have security implications, and we may not have
   designed approve() to do "store" by default to all helpers if we
   knew better.  But an unconditional change of behaviour will break
   existing users and helpers.

Assuming that the above summary is accurate, I think the former is
easier to solve.  In the ideal end game state, we would also have
"accepted" in the protocol and call the helper that gave us the
accepted credential material with an earlier "get" (if the helper is
updated to understand "accepted").  The "accepted" may not even need
to receive the credential material as the payload.

The latter is trickier, as there are design considerations.

 - We may want to allow the helper that gives the credential back
   from the outside world upon "get" request to say "you can 'store'
   this to other helpers of this kind but not of that kind".  If we
   want to do so, we'd need to extend what is returned from the
   helper upon "get" (e.g. "get2" will give more than what "get"
   does back).

 - We may want to allow the helper that receives the credential
   material from others to behave differently depending on where it
   came from, and what other helpers done to it (e.g. even a new
   credential the end user just typed may not want to go to all
   helpers; an on-disk helper may feel "I'd take it if the only
   other helpers that responded to 'store' are the transient
   'in-core' kind, but otherwise I'd ignore").  I am not offhand
   sure what kind of flexibility and protocol extension is
   necessary.

 - We may want to be able to override the preference the helper
   makes in the above two.  The user may want to say "Only use this
   on-disk helper as a read-only source and never call 'store' on it
   to add new entries (or update an existing one)."

Re: [PATCH] Improvement to only call Git Credential Helper once

[Jeff King]

On Sat, Sep 29, 2018 at 12:06:38PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > Wow, what's old is new again. Here's more or less the same patch from
> > 2012:
> >
> >   https://public-inbox.org/git/20120407033417.GA13914@sigill.intra.peff.net/
> >
> > Unfortunately, some people seem to rely on this multi-helper behavior. I
> > recommend reading the whole thread, as there are some interesting bits
> > in it (that I had always meant to return to, but somehow 6 years went
> > by).
> 
> After reading that thread again, my version of summary is that
> 
>  - storing the credential obtained from a helper back to the same
>    helper may be pointless at best and may incur end-user
>    interaction (e.g. asking for symmetric encryption key before the
>    data hits the disk), but it can be used as a "we used what you
>    gave us successfully" ping-back signal.  We may not have designed
>    approve() to do "store" back to the same helper by default and
>    instead to do so only when asked by the helper, if we knew
>    better.  But an unconditional change of behaviour will break
>    existing users and helpers.
> 
>  - storing the credential obtained from a helper to a different
>    helper may have security implications, and we may not have
>    designed approve() to do "store" by default to all helpers if we
>    knew better.  But an unconditional change of behaviour will break
>    existing users and helpers.

Yeah, I agree with that summary, except I want to pick a nit with the
very last sentence.

You're not wrong that it will break those existing users, but there are
security implications. If you're expecting and relying on that behavior,
then all is well with the current code, and you'd be annoyed by a
change. But if you're not expecting it, the system is not just broken:
it may be leaking credentials from a higher-security first-choice helper
to a lower-security second-choice one.

And that's why I wonder if it might be the right thing to break
compatibility in this case.

> Assuming that the above summary is accurate, I think the former is
> easier to solve.  In the ideal end game state, we would also have
> "accepted" in the protocol and call the helper that gave us the
> accepted credential material with an earlier "get" (if the helper is
> updated to understand "accepted").  The "accepted" may not even need
> to receive the credential material as the payload.

I think you can really solve both by adding a "retrieved" key to the
store step of the protocol (which can be done in a backwards-compatible
way, since all sides are supposed to ignore keys they don't understand).

Then a helper which understands "retrieved" can say "Oh, this came from
another helper; I don't need to store it". And that includes things that
might have come from itself! Likewise, helpers might take options to
tell them how to behave. So the "do some https thing and cache it"
combination from that earlier thread would be:

  [credential]
  helper = magic-https-thing
  helper = cache

and the safe version of "try high-security thing, and then fall back to
caching" would be:

  [credential]
  helper = high-security-thing
  helper = cache --no-retrieved

(that keeps the default matching the current behavior, but obviously we
could flip it if we wanted to have a safer default but leave the
existing case with an escape hatch).

> The latter is trickier, as there are design considerations.
> 
>  - We may want to allow the helper that gives the credential back
>    from the outside world upon "get" request to say "you can 'store'
>    this to other helpers of this kind but not of that kind".  If we
>    want to do so, we'd need to extend what is returned from the
>    helper upon "get" (e.g. "get2" will give more than what "get"
>    does back).

I don't think you need a "get2". The helpers should be free to pass back
extra keys (but old versions of Git just won't do anything with them).

>  - We may want to allow the helper that receives the credential
>    material from others to behave differently depending on where it
>    came from, and what other helpers done to it (e.g. even a new
>    credential the end user just typed may not want to go to all
>    helpers; an on-disk helper may feel "I'd take it if the only
>    other helpers that responded to 'store' are the transient
>    'in-core' kind, but otherwise I'd ignore").  I am not offhand
>    sure what kind of flexibility and protocol extension is
>    necessary.

If we have a "retrieved" flag, it could be "retrieved=some-helper" to
show which helper it came from. One tricky thing, though, is that
helpers do not always have a simple name (they can be arbitrary shell
expressions).

>  - We may want to be able to override the preference the helper
>    makes in the above two.  The user may want to say "Only use this
>    on-disk helper as a read-only source and never call 'store' on it
>    to add new entries (or update an existing one)."

Yes. I think if we go this route that helpers would just annotate
credentials they pass back (e.g., a key that says "this credential has
property foo") and other credentials would take user direction to
respect them (e.g., credential-cache would grow an option to say "do not
cache things with property foo").

The final patch in that earlier thread:

  https://public-inbox.org/git/20120408071300.GB13662@sigill.intra.peff.net/

showed what it would look like to just pass the properties around
between the helpers. That would be enough so that an individual helper
"foo" could:

  - set a key like "from=foo"

  - treat "from=foo" as a no-op for the "store" step (thus avoiding
    the ask-for-passphrase-again problem that started this current thread)

And then other helpers like credential-cache could grow options like
"--ignore=from", which would not cache anything that has a non-empty
"from" property.

-Peff