[PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

[PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

[Erik E Brady]

credential.c, run_credential_helper(): now ignores SIGPIPE
when writing to credential helper.  Avoids problem with race
where cred helper exits very quickly and, after, git tries
to write to it, generating SIGPIPE and crashing git.  To
reproduce this the cred helper must not read from STDIN.

This was seen with a custom credential helper, written in
Go, which ignored the store command (STDIN not read) and
then did a quick exit.  Even with this fast helper the race
was pretty rare, ie: was only seen on some of our older VM's
running 2.6.18-416.el5 #1 SMP linux for whatever reason.  On
these VM's it occurred only once every few hundred git cmds.
---
 credential.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/credential.c b/credential.c
index 9747f47b1..62be651b0 100644
--- a/credential.c
+++ b/credential.c
@@ -5,6 +5,7 @@
 #include "run-command.h"
 #include "url.h"
 #include "prompt.h"
+#include "sigchain.h"
 
 void credential_init(struct credential *c)
 {
@@ -227,8 +228,10 @@ static int run_credential_helper(struct credential *c,
 		return -1;
 
 	fp = xfdopen(helper.in, "w");
+	sigchain_push(SIGPIPE, SIG_IGN);
 	credential_write(c, fp);
 	fclose(fp);
+	sigchain_pop(SIGPIPE);
 
 	if (want_output) {
 		int r;
-- 
2.16.3.dirty

Re: [PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

[Jeff King]

On Wed, Mar 28, 2018 at 03:20:51PM -0700, Erik E Brady wrote:

> Subject: Re: [PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

Thanks for sending this. The patch itself looks good to me, but I have a
few nits with your commit message.

We usually write commit messages in the imperative, with the subject
summarizing the change. So:

  Subject: credential: ignore SIGPIPE when writing to credential helpers

or similar.

> credential.c, run_credential_helper(): now ignores SIGPIPE
> when writing to credential helper.  Avoids problem with race
> where cred helper exits very quickly and, after, git tries
> to write to it, generating SIGPIPE and crashing git.  To
> reproduce this the cred helper must not read from STDIN.

We can stop being terse outside of the subject line. :) I'd probably
write something like:

  The credential subsystem can trigger SIGPIPE when writing to an
  external helper if that helper closes its stdin before reading the
  whole input. Normally this is rare, since helpers would need to read
  that input to make a decision about how to respond, but:

    1. It's reasonable to configure a helper which blindly a "get"
       answer, and trigger it only for certain hosts via config like:

         [credential "https://example.com"]
	 helper = "!get-example-password"

    2. A broken or misbehaving helper might exit immediately. That's an
       error, but it's not reasonable for it to take down the parent Git
       process with SIGPIPE.

  Even with such a helper, seeing this problem should be rare. Getting
  SIGPIPE requires the helper racily exiting before we've written the
  fairly small credential output.

Feel free to steal or adapt any of that as you see fit.

> This was seen with a custom credential helper, written in
> Go, which ignored the store command (STDIN not read) and
> then did a quick exit.  Even with this fast helper the race
> was pretty rare, ie: was only seen on some of our older VM's
> running 2.6.18-416.el5 #1 SMP linux for whatever reason.  On
> these VM's it occurred only once every few hundred git cmds.
> ---

Missing signoff. See Documentation/SubmittingPatches, especially the
'sign-off' and 'dco' sections.

>  credential.c | 3 +++
>  1 file changed, 3 insertions(+)

No test, but I think that's fine here. Any such test would be inherently
racy.

> @@ -227,8 +228,10 @@ static int run_credential_helper(struct credential *c,
>  		return -1;
>  
>  	fp = xfdopen(helper.in, "w");
> +	sigchain_push(SIGPIPE, SIG_IGN);
>  	credential_write(c, fp);
>  	fclose(fp);
> +	sigchain_pop(SIGPIPE);

This looks like the right place to put the push/pop (as you noted
before, we may not write until fclose flushes, so it definitely has to
go after that).

Thanks again for digging this up. It's pretty subtle. :)

-Peff

Re: [PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

[Erik Brady -X (brady - ROBERT HALF INTERNATIONAL INC at Cisco)]

Thanks Jeff.

OK, will retry on the comment.  I guess I misunderstood the guidelines a bit on the signoff as well (ie: non-optional), apologies.  Will resubmit via 'git send-email' after adjusting the comment and recommitting with the -s option.  First time for everything I suppose, doh.

As to your comment suggestion, appreciated, looks good.  I might reword the #1 item you have just a bit (I removed the host specific stuff since I think the race can occur regardless of host specific or not... but I might be missing something there?).  Anyhow, how about something like this:

--
Subject: credential: ignore SIGPIPE when writing to credential helpers

The credential subsystem can trigger SIGPIPE when writing to an
external helper if that helper closes its stdin before reading the
whole input. Normally this is rare, since helpers would need to read
that input to make a decision about how to respond, but:

1. It's reasonable to configure a helper which only handles "get"
    while ignoring "store".  Such a handler might not read stdin 
    for "store", thereby rapidly closing stdin upon helper exit.

2. A broken or misbehaving helper might exit immediately. That's an
     error, but it's not reasonable for it to take down the parent Git
     process with SIGPIPE.
    
Even with such a helper, seeing this problem should be rare. Getting
SIGPIPE requires the helper racily exiting before we've written the
fairly small credential output.
--

As to testing, yes, that was my thought as well.  Anyhow, I will try the above unless you see a problem or would like any further change (?).

Thanks,
Erik

On 3/29/18, 4:19 AM, "Jeff King" <peff@peff.net> wrote:

    On Wed, Mar 28, 2018 at 03:20:51PM -0700, Erik E Brady wrote:
    
    > Subject: Re: [PATCH] credential: cred helper fast exit can cause SIGPIPE, crash
    
    Thanks for sending this. The patch itself looks good to me, but I have a
    few nits with your commit message.
    
    We usually write commit messages in the imperative, with the subject
    summarizing the change. So:
    
      Subject: credential: ignore SIGPIPE when writing to credential helpers
    
    or similar.
    
    > credential.c, run_credential_helper(): now ignores SIGPIPE
    > when writing to credential helper.  Avoids problem with race
    > where cred helper exits very quickly and, after, git tries
    > to write to it, generating SIGPIPE and crashing git.  To
    > reproduce this the cred helper must not read from STDIN.
    
    We can stop being terse outside of the subject line. :) I'd probably
    write something like:
    
      The credential subsystem can trigger SIGPIPE when writing to an
      external helper if that helper closes its stdin before reading the
      whole input. Normally this is rare, since helpers would need to read
      that input to make a decision about how to respond, but:
    
        1. It's reasonable to configure a helper which blindly a "get"
           answer, and trigger it only for certain hosts via config like:
    
             [credential "https://example.com"]
    	 helper = "!get-example-password"
    
        2. A broken or misbehaving helper might exit immediately. That's an
           error, but it's not reasonable for it to take down the parent Git
           process with SIGPIPE.
    
      Even with such a helper, seeing this problem should be rare. Getting
      SIGPIPE requires the helper racily exiting before we've written the
      fairly small credential output.
    
    Feel free to steal or adapt any of that as you see fit.
    
    > This was seen with a custom credential helper, written in
    > Go, which ignored the store command (STDIN not read) and
    > then did a quick exit.  Even with this fast helper the race
    > was pretty rare, ie: was only seen on some of our older VM's
    > running 2.6.18-416.el5 #1 SMP linux for whatever reason.  On
    > these VM's it occurred only once every few hundred git cmds.
    > ---
    
    Missing signoff. See Documentation/SubmittingPatches, especially the
    'sign-off' and 'dco' sections.
    
    >  credential.c | 3 +++
    >  1 file changed, 3 insertions(+)
    
    No test, but I think that's fine here. Any such test would be inherently
    racy.
    
    > @@ -227,8 +228,10 @@ static int run_credential_helper(struct credential *c,
    >  		return -1;
    >  
    >  	fp = xfdopen(helper.in, "w");
    > +	sigchain_push(SIGPIPE, SIG_IGN);
    >  	credential_write(c, fp);
    >  	fclose(fp);
    > +	sigchain_pop(SIGPIPE);
    
    This looks like the right place to put the push/pop (as you noted
    before, we may not write until fclose flushes, so it definitely has to
    go after that).
    
    Thanks again for digging this up. It's pretty subtle. :)
    
    -Peff
    

Re: [PATCH] credential: cred helper fast exit can cause SIGPIPE, crash

[Jeff King]

On Thu, Mar 29, 2018 at 05:25:04PM +0000, Erik Brady -X (brady - ROBERT HALF INTERNATIONAL INC at Cisco) wrote:

> OK, will retry on the comment.  I guess I misunderstood the guidelines
> a bit on the signoff as well (ie: non-optional), apologies.  Will
> resubmit via 'git send-email' after adjusting the comment and
> recommitting with the -s option.  First time for everything I suppose,
> doh.

The signoff (for our project) is all about the "yes, this contribution
can go under the gpl". So that part is very non-optional. :)

> As to your comment suggestion, appreciated, looks good.  I might
> reword the #1 item you have just a bit (I removed the host specific
> stuff since I think the race can occur regardless of host specific or
> not... but I might be missing something there?).  Anyhow, how about
> something like this:

Yeah, it can definitely occur regardless. It was just the plausible
reason to have a handler which does not bother to look at the incoming
data (since otherwise you are spewing your password to any host you
connect to; maybe that's OK if you configure it only inside a specific
repo and only fetch from one server).

Your update looks fine to me.

Thanks.

-Peff

[PATCH] credential: ignore SIGPIPE when writing to credential helpers

[Erik E Brady]

The credential subsystem can trigger SIGPIPE when writing to an
external helper if that helper closes its stdin before reading the
whole input. Normally this is rare, since helpers would need to read
that input to make a decision about how to respond, but:

1. It's reasonable to configure a helper which only handles "get"
   while ignoring "store".  Such a handler might not read stdin
   for "store", thereby rapidly closing stdin upon helper exit.

2. A broken or misbehaving helper might exit immediately. That's an
   error, but it's not reasonable for it to take down the parent Git
   process with SIGPIPE.

Even with such a helper, seeing this problem should be rare. Getting
SIGPIPE requires the helper racily exiting before we've written the
fairly small credential output.

Signed-off-by: Erik E Brady <brady@cisco.com>
---
 credential.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/credential.c b/credential.c
index 9747f47b1..62be651b0 100644
--- a/credential.c
+++ b/credential.c
@@ -5,6 +5,7 @@
 #include "run-command.h"
 #include "url.h"
 #include "prompt.h"
+#include "sigchain.h"
 
 void credential_init(struct credential *c)
 {
@@ -227,8 +228,10 @@ static int run_credential_helper(struct credential *c,
 		return -1;
 
 	fp = xfdopen(helper.in, "w");
+	sigchain_push(SIGPIPE, SIG_IGN);
 	credential_write(c, fp);
 	fclose(fp);
+	sigchain_pop(SIGPIPE);
 
 	if (want_output) {
 		int r;
-- 
2.16.3.dirty

Re: [PATCH] credential: ignore SIGPIPE when writing to credential helpers

[Jeff King]

On Thu, Mar 29, 2018 at 11:00:56AM -0700, Erik E Brady wrote:

> The credential subsystem can trigger SIGPIPE when writing to an
> external helper if that helper closes its stdin before reading the
> whole input. Normally this is rare, since helpers would need to read
> that input to make a decision about how to respond, but:
> 
> 1. It's reasonable to configure a helper which only handles "get"
>    while ignoring "store".  Such a handler might not read stdin
>    for "store", thereby rapidly closing stdin upon helper exit.
> 
> 2. A broken or misbehaving helper might exit immediately. That's an
>    error, but it's not reasonable for it to take down the parent Git
>    process with SIGPIPE.
> 
> Even with such a helper, seeing this problem should be rare. Getting
> SIGPIPE requires the helper racily exiting before we've written the
> fairly small credential output.
> 
> Signed-off-by: Erik E Brady <brady@cisco.com>
> ---
>  credential.c | 3 +++
>  1 file changed, 3 insertions(+)

This version looks good to me. Thanks!

-Peff

Re: [PATCH] credential: ignore SIGPIPE when writing to credential helpers

[Erik Brady -X (brady - ROBERT HALF INTERNATIONAL INC at Cisco)]

Thanks Jeff.

I appreciate your time.  Quick Q... is there a way to track the patch through to release?  If not I can just scan release notes/etc so no worries.

Cheers,
Erik

On 3/29/18, 2:51 PM, "Jeff King" <peff@peff.net> wrote:

    On Thu, Mar 29, 2018 at 11:00:56AM -0700, Erik E Brady wrote:
    
    > The credential subsystem can trigger SIGPIPE when writing to an
    > external helper if that helper closes its stdin before reading the
    > whole input. Normally this is rare, since helpers would need to read
    > that input to make a decision about how to respond, but:
    > 
    > 1. It's reasonable to configure a helper which only handles "get"
    >    while ignoring "store".  Such a handler might not read stdin
    >    for "store", thereby rapidly closing stdin upon helper exit.
    > 
    > 2. A broken or misbehaving helper might exit immediately. That's an
    >    error, but it's not reasonable for it to take down the parent Git
    >    process with SIGPIPE.
    > 
    > Even with such a helper, seeing this problem should be rare. Getting
    > SIGPIPE requires the helper racily exiting before we've written the
    > fairly small credential output.
    > 
    > Signed-off-by: Erik E Brady <brady@cisco.com>
    > ---
    >  credential.c | 3 +++
    >  1 file changed, 3 insertions(+)
    
    This version looks good to me. Thanks!
    
    -Peff
    

Re: [PATCH] credential: ignore SIGPIPE when writing to credential helpers

[Jeff King]

On Thu, Mar 29, 2018 at 10:20:40PM +0000, Erik Brady -X (brady - ROBERT HALF INTERNATIONAL INC at Cisco) wrote:

> I appreciate your time.  Quick Q... is there a way to track the patch
> through to release?  If not I can just scan release notes/etc so no
> worries.

When the maintainer picks up the patch, he usually says something like
"thanks, will queue". Then you can track its progress either by:

 - fetching from https://github.com/gitster/git, which has all of the
   topic branches. Yours will be "eb/something", depending what Junio
   names it. And then you can periodically "git branch -a --contains
   eb/something" to see it progress through the various integration
   branches ('pu', 'next', 'master'). The branches are described in the
   "note from the maintainer" that's sent to the list periodically.
   E.g.:

     https://public-inbox.org/git/xmqqy3nt40pq.fsf@gitster.mtv.corp.google.com/

 - alternatively, you can read the "What's cooking in git.git" messages
   sent out by the maintainer a few times a week. E.g.:

     https://public-inbox.org/git/xmqqefkm6s06.fsf@gitster-ct.c.googlers.com/

   There your eb/something topic will be mentioned, along with the
   status (where it is in the graduation cycle, what's coming next, and
   if it's stalled, why).

We're in a release freeze right for v2.17 right now, so I'd expect your
patch to probably go to the 'maint' track and end up in v2.17.1.

Of course somebody else may see something wrong I didn't and ask you to
correct it, which would be in a reply to this thread. :)

-Peff

Re: [PATCH] credential: ignore SIGPIPE when writing to credential helpers

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Mar 29, 2018 at 11:00:56AM -0700, Erik E Brady wrote:
>
>> The credential subsystem can trigger SIGPIPE when writing to an
>> external helper if that helper closes its stdin before reading the
>> whole input. Normally this is rare, since helpers would need to read
>> that input to make a decision about how to respond, but:
>> 
>> 1. It's reasonable to configure a helper which only handles "get"
>>    while ignoring "store".  Such a handler might not read stdin
>>    for "store", thereby rapidly closing stdin upon helper exit.
>> 
>> 2. A broken or misbehaving helper might exit immediately. That's an
>>    error, but it's not reasonable for it to take down the parent Git
>>    process with SIGPIPE.
>> 
>> Even with such a helper, seeing this problem should be rare. Getting
>> SIGPIPE requires the helper racily exiting before we've written the
>> fairly small credential output.
>> 
>> Signed-off-by: Erik E Brady <brady@cisco.com>
>> ---
>>  credential.c | 3 +++
>>  1 file changed, 3 insertions(+)
>
> This version looks good to me. Thanks!

Yup, looks good.  Thanks, both.