[PATCH] http(s): automatically try NTLM authentication first

[PATCH] http(s): automatically try NTLM authentication first

[David Turner]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

It is common in corporate setups to have permissions managed via a
domain account. That means that the user does not really have to log in
when accessing a central repository via https://, but that the login
credentials are used to authenticate with that repository.

The common way to do that used to require empty credentials, i.e. hitting
Enter twice when being asked for user name and password, or by using the
very funny notation https://:@server/repository

A recent commit (5275c3081c (http: http.emptyauth should allow empty (not
just NULL) usernames, 2016-10-04)) broke that usage, though, all of a
sudden requiring users to set http.emptyAuth = true.

Which brings us to the bigger question why http.emptyAuth defaults to
false, to begin with.

It would be one thing if cURL would not let the user specify credentials
interactively after attempting NTLM authentication (i.e. login
credentials), but that is not the case.

It would be another thing if attempting NTLM authentication was not
usually what users need to do when trying to authenticate via https://.
But that is also not the case.

So let's just go ahead and change the default, and unbreak the NTLM
authentication. As a bonus, this also makes the "you need to hit Enter
twice" (which is hard to explain: why enter empty credentials when you
want to authenticate with your login credentials?) and the ":@" hack
(which is also pretty, pretty hard to explain to users) obsolete.

This fixes https://github.com/git-for-windows/git/issues/987

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: David Turner <dturner@twosigma.com>
---
This has been in git for Windows for a few months (without the
config.txt change).  We've also been using it internally.  So I think
it's time to merge back to upstream git.

 Documentation/config.txt | 3 ++-
 http.c                   | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index fc5a28a320..b0da64ed33 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1742,7 +1742,8 @@ http.emptyAuth::
 	Attempt authentication without seeking a username or password.  This
 	can be used to attempt GSS-Negotiate authentication without specifying
 	a username in the URL, as libcurl normally requires a username for
-	authentication.
+	authentication.  Default is true, since if this fails, git will fall
+	back to asking the user for their username/password.
 
 http.delegation::
 	Control GSSAPI credential delegation. The delegation is disabled
diff --git a/http.c b/http.c
index 90a1c0f113..943e630ea6 100644
--- a/http.c
+++ b/http.c
@@ -109,7 +109,7 @@ static int curl_save_cookies;
 struct credential http_auth = CREDENTIAL_INIT;
 static int http_proactive_auth;
 static const char *user_agent;
-static int curl_empty_auth;
+static int curl_empty_auth = 1;
 
 enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL;
 
-- 
2.11.GIT

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

David Turner <dturner@twosigma.com> writes:

> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> It is common in corporate setups to have permissions managed via a
> domain account. That means that the user does not really have to log in
> when accessing a central repository via https://, but that the login
> credentials are used to authenticate with that repository.
>
> The common way to do that used to require empty credentials, i.e. hitting
> Enter twice when being asked for user name and password, or by using the
> very funny notation https://:@server/repository
>
> A recent commit (5275c3081c (http: http.emptyauth should allow empty (not
> just NULL) usernames, 2016-10-04)) broke that usage, though, all of a
> sudden requiring users to set http.emptyAuth = true.
>
> Which brings us to the bigger question why http.emptyAuth defaults to
> false, to begin with.

This is a valid question, and and I do not see it explicitly asked
in the thread:

https://public-inbox.org/git/CAPig+cSphEu3iRJrkdBA+BRhi9HnopLJnKOHVuGhUqavtV1RXg@mail.gmail.com/#t

even though there is a hint of it already there.

> It would be one thing if cURL would not let the user specify credentials
> interactively after attempting NTLM authentication (i.e. login
> credentials), but that is not the case.
>
> It would be another thing if attempting NTLM authentication was not
> usually what users need to do when trying to authenticate via https://.
> But that is also not the case.

Some other possible worries we may have had I can think of are:

 - With this enabled unconditionally, would we leak some information?

 - With this enabled unconditionally, would we always incur an extra
   roundtrip for people who are not running NTLM at all?

I do not think the former is the case, but what would I know (adding a
few people involved in the original thread to CC: ;-)

>  Documentation/config.txt | 3 ++-
>  http.c                   | 2 +-
>  2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index fc5a28a320..b0da64ed33 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -1742,7 +1742,8 @@ http.emptyAuth::
>  	Attempt authentication without seeking a username or password.  This
>  	can be used to attempt GSS-Negotiate authentication without specifying
>  	a username in the URL, as libcurl normally requires a username for
> -	authentication.
> +	authentication.  Default is true, since if this fails, git will fall
> +	back to asking the user for their username/password.
>  
>  http.delegation::
>  	Control GSSAPI credential delegation. The delegation is disabled
> diff --git a/http.c b/http.c
> index 90a1c0f113..943e630ea6 100644
> --- a/http.c
> +++ b/http.c
> @@ -109,7 +109,7 @@ static int curl_save_cookies;
>  struct credential http_auth = CREDENTIAL_INIT;
>  static int http_proactive_auth;
>  static const char *user_agent;
> -static int curl_empty_auth;
> +static int curl_empty_auth = 1;
>  
>  enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL;

RE: [PATCH] http(s): automatically try NTLM authentication first

[David Turner]

> -----Original Message-----
> From: Junio C Hamano [mailto:jch2355@gmail.com] On Behalf Of Junio C
> Hamano
> Sent: Wednesday, February 22, 2017 3:20 PM
> To: David Turner <David.Turner@twosigma.com>
> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; Johannes Schindelin
> <johannes.schindelin@gmx.de>; Eric Sunshine
> <sunshine@sunshineco.com>; Jeff King <peff@peff.net>
> Subject: Re: [PATCH] http(s): automatically try NTLM authentication first
> 
> David Turner <dturner@twosigma.com> writes:
> 
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > It is common in corporate setups to have permissions managed via a
> > domain account. That means that the user does not really have to log
> > in when accessing a central repository via https://, but that the
> > login credentials are used to authenticate with that repository.
> >
> > The common way to do that used to require empty credentials, i.e.
> > hitting Enter twice when being asked for user name and password, or by
> > using the very funny notation https://:@server/repository
> >
> > A recent commit (5275c3081c (http: http.emptyauth should allow empty
> > (not just NULL) usernames, 2016-10-04)) broke that usage, though, all
> > of a sudden requiring users to set http.emptyAuth = true.
> >
> > Which brings us to the bigger question why http.emptyAuth defaults to
> > false, to begin with.
> 
> This is a valid question, and and I do not see it explicitly asked in the thread:
> 
> https://public-
> inbox.org/git/CAPig+cSphEu3iRJrkdBA+BRhi9HnopLJnKOHVuGhUqavtV1RXg
> @mail.gmail.com/#t
> 
> even though there is a hint of it already there.
> 
> > It would be one thing if cURL would not let the user specify
> > credentials interactively after attempting NTLM authentication (i.e.
> > login credentials), but that is not the case.
> >
> > It would be another thing if attempting NTLM authentication was not
> > usually what users need to do when trying to authenticate via https://.
> > But that is also not the case.
> 
> Some other possible worries we may have had I can think of are:
> 
>  - With this enabled unconditionally, would we leak some information?

I think "NTLM" is actually a misnomer here (I just copied Johannes's 
commit message). The mechanism is actually SPNEGO, if I understand this 
correctly. It seems to me that this is probably secure, since it is apparently
widely implemented in browsers.

>  - With this enabled unconditionally, would we always incur an extra
>    roundtrip for people who are not running NTLM at all?
>
> I do not think the former is the case, but what would I know (adding a few
> people involved in the original thread to CC: ;-)

Always, no.  For failed authentication (or authorization), apparently, yes.  
I tested this by  setting the variable to false and then true, and trying to 
Push to a github repository which I didn't have write access to, with 
both an empty username (https://@:github.com/...) and no username 
(http://github.com/...).   I ran this under GIT_CURL_VERBOSE=1 and
I saw two 401 responses in the "http.emptyauth=true" case and one
in the false case.  I also tried with a repo that I did have access to (first
configuring the necessary tokens for HTTPS push access), and saw two
401 responses in *both* cases.  

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

David Turner <David.Turner@twosigma.com> writes:

> Always, no.  For failed authentication (or authorization), apparently, yes.  
> I tested this by  setting the variable to false and then true, and trying to 
> Push to a github repository which I didn't have write access to, with 
> both an empty username (https://@:github.com/...) and no username 
> (http://github.com/...).   I ran this under GIT_CURL_VERBOSE=1 and
> I saw two 401 responses in the "http.emptyauth=true" case and one
> in the false case.  I also tried with a repo that I did have access to (first
> configuring the necessary tokens for HTTPS push access), and saw two
> 401 responses in *both* cases.  

Thanks; that matches my observation.  I do not think we care about
an extra roundtrip for the failure case, but as long as we do not
increase the number of roundtrip in the normal case, we can declare
that this is an improvement.  I am not quite sure where that extra
401 comes from in the normal case, and that might be an indication
that we already are doing something wrong, though.

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 01:16:33PM -0800, Junio C Hamano wrote:

> David Turner <David.Turner@twosigma.com> writes:
> 
> > Always, no.  For failed authentication (or authorization), apparently, yes.  
> > I tested this by  setting the variable to false and then true, and trying to 
> > Push to a github repository which I didn't have write access to, with 
> > both an empty username (https://@:github.com/...) and no username 
> > (http://github.com/...).   I ran this under GIT_CURL_VERBOSE=1 and
> > I saw two 401 responses in the "http.emptyauth=true" case and one
> > in the false case.  I also tried with a repo that I did have access to (first
> > configuring the necessary tokens for HTTPS push access), and saw two
> > 401 responses in *both* cases.  
> 
> Thanks; that matches my observation.  I do not think we care about
> an extra roundtrip for the failure case, but as long as we do not
> increase the number of roundtrip in the normal case, we can declare
> that this is an improvement.  I am not quite sure where that extra
> 401 comes from in the normal case, and that might be an indication
> that we already are doing something wrong, though.

This patch drops the useless probe request:

diff --git a/http.c b/http.c
index 943e630ea..7b4c2db86 100644
--- a/http.c
+++ b/http.c
@@ -1663,6 +1663,9 @@ static int http_request(const char *url,
 		curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
 				options->effective_url);
 
+	if (results.auth_avail == CURLAUTH_BASIC)
+		http_auth_methods = CURLAUTH_BASIC;
+
 	curl_slist_free_all(headers);
 	strbuf_release(&buf);
 

but setting http.emptyauth adds back in the useless request. I think
that could be fixed by skipping the empty-auth thing when
http_auth_methods does not have CURLAUTH_NEGOTIATE in it (or perhaps
other methods need it to, so maybe skip it if _just_ BASIC is set).

I suspect the patch above could probably be generalized as:

  /* cut out methods we know the server doesn't support */
  http_auth_methods &= results.auth_avail;

and let curl figure it out from there.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Johannes Schindelin]

Hi Peff,

On Wed, 22 Feb 2017, Jeff King wrote:

> On Wed, Feb 22, 2017 at 01:16:33PM -0800, Junio C Hamano wrote:
> 
> > David Turner <David.Turner@twosigma.com> writes:
> > 
> > > Always, no.  For failed authentication (or authorization),
> > > apparently, yes.  I tested this by  setting the variable to false
> > > and then true, and trying to Push to a github repository which I
> > > didn't have write access to, with both an empty username
> > > (https://@:github.com/...) and no username (http://github.com/...).
> > > I ran this under GIT_CURL_VERBOSE=1 and I saw two 401 responses in
> > > the "http.emptyauth=true" case and one in the false case.  I also
> > > tried with a repo that I did have access to (first configuring the
> > > necessary tokens for HTTPS push access), and saw two 401 responses
> > > in *both* cases.  
> > 
> > Thanks; that matches my observation.  I do not think we care about
> > an extra roundtrip for the failure case, but as long as we do not
> > increase the number of roundtrip in the normal case, we can declare
> > that this is an improvement.  I am not quite sure where that extra
> > 401 comes from in the normal case, and that might be an indication
> > that we already are doing something wrong, though.
> 
> This patch drops the useless probe request:
> 
> diff --git a/http.c b/http.c
> index 943e630ea..7b4c2db86 100644
> --- a/http.c
> +++ b/http.c
> @@ -1663,6 +1663,9 @@ static int http_request(const char *url,
>  		curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
>  				options->effective_url);
>  
> +	if (results.auth_avail == CURLAUTH_BASIC)
> +		http_auth_methods = CURLAUTH_BASIC;
> +
>  	curl_slist_free_all(headers);
>  	strbuf_release(&buf);
>  
> 
> but setting http.emptyauth adds back in the useless request. I think
> that could be fixed by skipping the empty-auth thing when
> http_auth_methods does not have CURLAUTH_NEGOTIATE in it (or perhaps
> other methods need it to, so maybe skip it if _just_ BASIC is set).
> 
> I suspect the patch above could probably be generalized as:
> 
>   /* cut out methods we know the server doesn't support */
>   http_auth_methods &= results.auth_avail;
> 
> and let curl figure it out from there.

Maybe this patch (or a variation thereof) would also be able to fix this
problem with the patch:

	https://github.com/git-for-windows/git/issues/1034

Short version: for certain servers (that do *not* advertise Negotiate),
setting emptyauth to true will result in a failed fetch, without letting
the user type in their credentials.

Ciao,
Johannes

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Wed, 22 Feb 2017, Jeff King wrote:
>> This patch drops the useless probe request:
> ...
>> but setting http.emptyauth adds back in the useless request. I think
>> that could be fixed by skipping the empty-auth thing when
>> http_auth_methods does not have CURLAUTH_NEGOTIATE in it (or perhaps
>> other methods need it to, so maybe skip it if _just_ BASIC is set).
>> 
>> I suspect the patch above could probably be generalized as:
>> 
>>   /* cut out methods we know the server doesn't support */
>>   http_auth_methods &= results.auth_avail;
>> 
>> and let curl figure it out from there.
>
> Maybe this patch (or a variation thereof) would also be able to fix this
> problem with the patch:
>
> 	https://github.com/git-for-windows/git/issues/1034
>
> Short version: for certain servers (that do *not* advertise Negotiate),
> setting emptyauth to true will result in a failed fetch, without letting
> the user type in their credentials.

The issue described in that page looks rather serious.

I believe that a "variation" has become the first part of a
two-patch series that appear in the downthread from here.  Perhaps
you can ask them to test it out (or even better if you have a setup
you can easily test against yourself)?

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Thu, Feb 23, 2017 at 06:08:49PM +0100, Johannes Schindelin wrote:

> > I suspect the patch above could probably be generalized as:
> > 
> >   /* cut out methods we know the server doesn't support */
> >   http_auth_methods &= results.auth_avail;
> > 
> > and let curl figure it out from there.
> 
> Maybe this patch (or a variation thereof) would also be able to fix this
> problem with the patch:
> 
> 	https://github.com/git-for-windows/git/issues/1034
> 
> Short version: for certain servers (that do *not* advertise Negotiate),
> setting emptyauth to true will result in a failed fetch, without letting
> the user type in their credentials.

I suspect it isn't enough to help without 2/2. This will tell curl that
the server does not do Negotiate, so it will skip the probe request. But
Git will still feed curl the bogus empty credential.

That's what 2/2 tries to fix: only kick in the emptyAuth hack when there
is something besides Basic[1] to try. The way it is written adds an
extra "auto" mode to emptyAuth, as I wanted to leave "emptyauth=true" as
a workaround in case the "auto" behavior does not work. And then I
turned on "auto" by default, since that was what the discussion was
shooting for.

But if we are worried about turning on emptyAuth everywhere, the auto
behavior could be tied to emptyauth=true (and have something like
"emptyauth=always" to _really_ force it). I don't have an opinion there.
It sounds like emptyauth has been enabled by default on Windows for a
while. It's not clear to me if that's a security problem or not.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I suspect it isn't enough to help without 2/2. This will tell curl that
> the server does not do Negotiate, so it will skip the probe request. But
> Git will still feed curl the bogus empty credential.
>
> That's what 2/2 tries to fix: only kick in the emptyAuth hack when there
> is something besides Basic[1] to try. The way it is written adds an

In your [1] you wanted to mention that Digest would have the same
property as Basic, or something like that?

> extra "auto" mode to emptyAuth, as I wanted to leave "emptyauth=true" as
> a workaround in case the "auto" behavior does not work. And then I
> turned on "auto" by default, since that was what the discussion was
> shooting for.
>
> But if we are worried about turning on emptyAuth everywhere, the auto
> behavior could be tied to emptyauth=true (and have something like
> "emptyauth=always" to _really_ force it). I don't have an opinion there.

I do not have a strong opinion, either, but it sounds like that even
the "disable emptyAuth hack if the server is Basic only" variant
would be much better than setting emptyAuth on by default.  At least
the user whose issue was reported in Dscho's message would be fixed
by such a variant, I would think (i.e. talking to a server with no
Negotiate and emptyAuth set to true results in no attempt to give
the user a chance to tell who s/he is --- your 2/2 will turn
emptyAuth off in that case).

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Thu, Feb 23, 2017 at 12:37:25PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > I suspect it isn't enough to help without 2/2. This will tell curl that
> > the server does not do Negotiate, so it will skip the probe request. But
> > Git will still feed curl the bogus empty credential.
> >
> > That's what 2/2 tries to fix: only kick in the emptyAuth hack when there
> > is something besides Basic[1] to try. The way it is written adds an
> 
> In your [1] you wanted to mention that Digest would have the same
> property as Basic, or something like that?

Oops, yeah. What I was going to say is that we may want a list of auth
types where we _do_ want the hack on, rather than ones where we know it
does not work. People are more likely to notice when the list is wrong,
then.

> > But if we are worried about turning on emptyAuth everywhere, the auto
> > behavior could be tied to emptyauth=true (and have something like
> > "emptyauth=always" to _really_ force it). I don't have an opinion there.
> 
> I do not have a strong opinion, either, but it sounds like that even
> the "disable emptyAuth hack if the server is Basic only" variant
> would be much better than setting emptyAuth on by default.  At least
> the user whose issue was reported in Dscho's message would be fixed
> by such a variant, I would think (i.e. talking to a server with no
> Negotiate and emptyAuth set to true results in no attempt to give
> the user a chance to tell who s/he is --- your 2/2 will turn
> emptyAuth off in that case).

Yes, I agree that the "auto" behavior is better than defaulting to
"true". I am speaking from the perspective of git.git, which is
currently defaulting to "false". It is not clear to me if "auto" is
better than "false" because of the security implications.

For Git for Windows, it seems like the auto behavior would be a strict
improvement over the "true" default they've been shipping.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Johannes Schindelin]

Hi Peff,

On Thu, 23 Feb 2017, Jeff King wrote:

> For Git for Windows, [PATCH 2/2] seems like the auto behavior would be a
> strict improvement over the "true" default they've been shipping.

Absolutely. Thank you for your tremendous help!

Ciao,
Dscho

Re: [PATCH] http(s): automatically try NTLM authentication first

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

On Wed, Feb 22, 2017 at 09:04:14PM +0000, David Turner wrote:
> > -----Original Message-----
> > From: Junio C Hamano [mailto:jch2355@gmail.com] On Behalf Of Junio C
> > Hamano
> > Sent: Wednesday, February 22, 2017 3:20 PM
> > To: David Turner <David.Turner@twosigma.com>
> > Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; Johannes Schindelin
> > <johannes.schindelin@gmx.de>; Eric Sunshine
> > <sunshine@sunshineco.com>; Jeff King <peff@peff.net>
> > Subject: Re: [PATCH] http(s): automatically try NTLM authentication first
> > 
> > 
> > Some other possible worries we may have had I can think of are:
> > 
> >  - With this enabled unconditionally, would we leak some information?
> 
> I think "NTLM" is actually a misnomer here (I just copied Johannes's 
> commit message). The mechanism is actually SPNEGO, if I understand this 
> correctly. It seems to me that this is probably secure, since it is apparently
> widely implemented in browsers.

This is SPNEGO.  It will work with NTLM as well as Kerberos.

Browsers usually disable this feature by default, as it basically will
attempt to authenticate to any site that sends a 401.  For Kerberos
against a malicious site, the user will either not have a valid ticket
for that domain, or the user's Kerberos server will refuse to provide a
ticket to pass to the server, so there's no security risk involved.

I'm unclear how SPNEGO works with NTLM, so I can't speak for the
security of it.  From what I understand of NTLM and from RFC 4559, it
consists of a shared secret.  I'm unsure what security measures are in
place to not send that to an untrusted server.

As far as Kerberos, this is a desirable feature to have enabled, with
little downside.  I just don't know about the security of the NTLM part,
and I don't think we should take this patch unless we're sure we know
the consequences of it.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 11:34:19PM +0000, brian m. carlson wrote:

> Browsers usually disable this feature by default, as it basically will
> attempt to authenticate to any site that sends a 401.  For Kerberos
> against a malicious site, the user will either not have a valid ticket
> for that domain, or the user's Kerberos server will refuse to provide a
> ticket to pass to the server, so there's no security risk involved.
> 
> I'm unclear how SPNEGO works with NTLM, so I can't speak for the
> security of it.  From what I understand of NTLM and from RFC 4559, it
> consists of a shared secret.  I'm unsure what security measures are in
> place to not send that to an untrusted server.
> 
> As far as Kerberos, this is a desirable feature to have enabled, with
> little downside.  I just don't know about the security of the NTLM part,
> and I don't think we should take this patch unless we're sure we know
> the consequences of it.

Hmm. That would be a problem with my proposed patch 2 then, too, if only
because it turns the feature on by default in more places.

If it _is_ dangerous to turn on all the time, I'd think we should
consider warning people in the http.emptyauth documentation.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Feb 22, 2017 at 11:34:19PM +0000, brian m. carlson wrote:
>
>> Browsers usually disable this feature by default, as it basically will
>> attempt to authenticate to any site that sends a 401.  For Kerberos
>> against a malicious site, the user will either not have a valid ticket
>> for that domain, or the user's Kerberos server will refuse to provide a
>> ticket to pass to the server, so there's no security risk involved.
>> 
>> I'm unclear how SPNEGO works with NTLM, so I can't speak for the
>> security of it.  From what I understand of NTLM and from RFC 4559, it
>> consists of a shared secret.  I'm unsure what security measures are in
>> place to not send that to an untrusted server.
>> 
>> As far as Kerberos, this is a desirable feature to have enabled, with
>> little downside.  I just don't know about the security of the NTLM part,
>> and I don't think we should take this patch unless we're sure we know
>> the consequences of it.
>
> Hmm. That would be a problem with my proposed patch 2 then, too, if only
> because it turns the feature on by default in more places.
>
> If it _is_ dangerous to turn on all the time, I'd think we should
> consider warning people in the http.emptyauth documentation.

Yeah, http.<url>.emptyAuth that knows where it is going may be a lot
safer but a blanket http.emptyAuth does sound bad.

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Feb 22, 2017 at 11:34:19PM +0000, brian m. carlson wrote:
>
>> Browsers usually disable this feature by default, as it basically will
>> attempt to authenticate to any site that sends a 401.  For Kerberos
>> against a malicious site, the user will either not have a valid ticket
>> for that domain, or the user's Kerberos server will refuse to provide a
>> ticket to pass to the server, so there's no security risk involved.
>> 
>> I'm unclear how SPNEGO works with NTLM, so I can't speak for the
>> security of it.  From what I understand of NTLM and from RFC 4559, it
>> consists of a shared secret.  I'm unsure what security measures are in
>> place to not send that to an untrusted server.
>> 
>> As far as Kerberos, this is a desirable feature to have enabled, with
>> little downside.  I just don't know about the security of the NTLM part,
>> and I don't think we should take this patch unless we're sure we know
>> the consequences of it.
>
> Hmm. That would be a problem with my proposed patch 2 then, too, if only
> because it turns the feature on by default in more places.
>
> If it _is_ dangerous to turn on all the time, I'd think we should
> consider warning people in the http.emptyauth documentation.

I presume that we have finished discussing the security
ramification, and if I am not mistaken the conclusion was that it
could leak information if we turned on emptyAuth unconditionally
when talking to a wrong server, and that the documentation needs an
update to recommend those who use emptyAuth because they want to
talk to Negotiate servers to use the http.<site>.emptyAuth form,
limited to such servers, not a more generic http.emptyAuth, to avoid
information leakage?

If that is the case, let's take your 1/2 in the near-by thread
without 2/2 (auto-enable emptyAuth) for now, as Dscho seems to have
a case that may be helped by it.

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Thu, Feb 23, 2017 at 11:11:05AM -0800, Junio C Hamano wrote:

> >> As far as Kerberos, this is a desirable feature to have enabled, with
> >> little downside.  I just don't know about the security of the NTLM part,
> >> and I don't think we should take this patch unless we're sure we know
> >> the consequences of it.
> >
> > Hmm. That would be a problem with my proposed patch 2 then, too, if only
> > because it turns the feature on by default in more places.
> >
> > If it _is_ dangerous to turn on all the time, I'd think we should
> > consider warning people in the http.emptyauth documentation.
> 
> I presume that we have finished discussing the security
> ramification, and if I am not mistaken the conclusion was that it
> could leak information if we turned on emptyAuth unconditionally
> when talking to a wrong server, and that the documentation needs an
> update to recommend those who use emptyAuth because they want to
> talk to Negotiate servers to use the http.<site>.emptyAuth form,
> limited to such servers, not a more generic http.emptyAuth, to avoid
> information leakage?

I don't know enough to evaluate the claims of emptyAuth being dangerous
or not (nor do I use it myself or admin a server whose users need it).
So I will let interested parties hash out whether it is a good idea or
not, and I'm happy to drop my 2/2 for now.

If we are to make it more widely available, I would prefer something
more like my 2/2 than always turning on http.emptyAuth, if only because
it reduces the cost to people not using the feature. I'm happy to work
more on the patch if we decide to go that route.

> If that is the case, let's take your 1/2 in the near-by thread
> without 2/2 (auto-enable emptyAuth) for now, as Dscho seems to have
> a case that may be helped by it.

Yes, I think 1/2 stands on its own. Whether it helps Dscho's case or
not, it eliminates an HTTP round-trip for Basic-only servers, which I
think is worth it.

-Peff

RE: [PATCH] http(s): automatically try NTLM authentication first

[David Turner]

> -----Original Message-----
> From: brian m. carlson [mailto:sandals@crustytoothpaste.net]
> 
> This is SPNEGO.  It will work with NTLM as well as Kerberos.
> 
> Browsers usually disable this feature by default, as it basically will attempt to
> authenticate to any site that sends a 401.  For Kerberos against a malicious
> site, the user will either not have a valid ticket for that domain, or the user's
> Kerberos server will refuse to provide a ticket to pass to the server, so
> there's no security risk involved.
> 
> I'm unclear how SPNEGO works with NTLM, so I can't speak for the security
> of it.  From what I understand of NTLM and from RFC 4559, it consists of a
> shared secret.  I'm unsure what security measures are in place to not send
> that to an untrusted server.
> 
> As far as Kerberos, this is a desirable feature to have enabled, with little
> downside.  I just don't know about the security of the NTLM part, and I don't
> think we should take this patch unless we're sure we know the
> consequences of it.

NTLM on its own is bad:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa378749(v=vs.85).aspx
says:

"
1. (Interactive authentication only) A user accesses a client computer and 
provides a domain name, user name, and password. The client computes a 
cryptographic hash of the password and discards the actual password.
2. The client sends the user name to the server (in plaintext).
3. The server generates a 16-byte random number, called a challenge or 
nonce, and sends it to the client.
4. The client encrypts this challenge with the hash of the user's password 
and returns the result to the server. This is called the response.
..."

Wait, what?  If I'm a malicious server, I can get access to an offline oracle
for whether I've correctly guessed the user's password?  That doesn't 
sound secure at all!  Skimming the SPNEGO RFCs, there appears to be no
mitigation for this.  

So, I guess, this patch might be considered a security risk. But on the 
other hand, even *without* this patch, and without http.allowempty at 
all, I think a config which simply uses a https://  url without the magic :@
would try SPNEGO.  As I understand it, the http.allowempty config just 
makes the traditional :@ urls work. 

Actually, though, I am not sure this is as bad as it seems, because gssapi
might protect us.  When I locally tried a fake server, git (libcurl) refused to 
send my Kerberos credentials because "Server not found in Kerberos 
database".  I don't have a machine set up with NTLM authentication 
(because, apparently, that would be insane), so I don't know how to 
confirm that gssapi would operate off of a whitelist for NTLM as well. 

Re: [PATCH] http(s): automatically try NTLM authentication first

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

On Thu, Feb 23, 2017 at 01:03:39AM +0000, David Turner wrote:
> So, I guess, this patch might be considered a security risk. But on the 
> other hand, even *without* this patch, and without http.allowempty at 
> all, I think a config which simply uses a https://  url without the magic :@
> would try SPNEGO.  As I understand it, the http.allowempty config just 
> makes the traditional :@ urls work. 

No, it's a bit different.  libcurl won't try to authenticate to a server
unless it has a username (and possibly password).  With the curl command
line client, you use a dummy value or -u: to force it to do auth anyway
(because you want, say, GSSAPI).  http.emptyAuth just sets that option
to “:” so libcurl will auth:

		if (curl_empty_auth)
			curl_easy_setopt(result, CURLOPT_USERPWD, ":");

I just use a dummy username for my URLs, but you can write :@ or any
other permutation to get it to work without emptyAuth.  As a
consequence, you have to opt-in to that on a per-URL (or per-domain)
basis, which is a bit more secure.

> Actually, though, I am not sure this is as bad as it seems, because gssapi
> might protect us.  When I locally tried a fake server, git (libcurl) refused to 
> send my Kerberos credentials because "Server not found in Kerberos 
> database".  I don't have a machine set up with NTLM authentication 
> (because, apparently, that would be insane), so I don't know how to 
> confirm that gssapi would operate off of a whitelist for NTLM as well. 

Yup.  That's pretty much what I thought would happen, since the Kerberos
server has no HTTP/malicious.evil.tld@YOURREALM.TLD service ticket.
Again, I don't know how NTLM does things, or if it's wrapped in a
suitable ticket format somehow.

Last I base64-decoded an NTLM SPNEGO response, it did not contain the
OID required by GSSAPI as a prefix; it instead contained an “NTLMSSP”
header, which isn't a valid OID.  I didn't delve much further, since I
was pretty sure I didn't want to know more.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH] http(s): automatically try NTLM authentication first

[Mantas Mikulėnas]

On 2017-02-23 03:03, David Turner wrote:
> Actually, though, I am not sure this is as bad as it seems, because gssapi
> might protect us.  When I locally tried a fake server, git (libcurl) refused to 
> send my Kerberos credentials because "Server not found in Kerberos 
> database".  I don't have a machine set up with NTLM authentication 
> (because, apparently, that would be insane), so I don't know how to 
> confirm that gssapi would operate off of a whitelist for NTLM as well. 

NTLM and Kerberos work very differently in that regard.

Kerberos is ticket-based so the client *first* has to obtain a ticket
from the domain's KDC, so a malicious server at minimum needs to know
what principal name to provide (i.e. which real server to try
impersonating). And even if it does that, the ticket doesn't contain
crackable hashes, just data encrypted with a key known only to the KDC
and the real server. So the whitelist is only for privacy and/or
performance reasons, I guess?

NTLM is challenge/response without any third party, and yes, it requires
the application to implement its own whitelisting to avoid the security
problems.

-- 
Mantas Mikulėnas <grawity@gmail.com>

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 12:19:56PM -0800, Junio C Hamano wrote:

> > It would be one thing if cURL would not let the user specify credentials
> > interactively after attempting NTLM authentication (i.e. login
> > credentials), but that is not the case.
> >
> > It would be another thing if attempting NTLM authentication was not
> > usually what users need to do when trying to authenticate via https://.
> > But that is also not the case.
> 
> Some other possible worries we may have had I can think of are:
> 
>  - With this enabled unconditionally, would we leak some information?
> 
>  - With this enabled unconditionally, would we always incur an extra
>    roundtrip for people who are not running NTLM at all?
> 
> I do not think the former is the case, but what would I know (adding a
> few people involved in the original thread to CC: ;-)

I don't think it incurs an extra round-trip now, because of the way
libcurl works. Though I think it _does_ make it harder for curl to later
optimize out that extra round-trip.

The easiest way to see the difference is to run something like:

  GIT_CURL_VERBOSE=1 \
  git ls-remote https://example.com/repo-which-needs-auth.git 2>trace
  egrep '^>|^< HTTP|Authorization' trace

Before this patch, I get (this is against github.com, which only does
Basic auth):

  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic <actual credentials>
  < HTTP/1.1 200 OK

And after I get:

  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic Og==
  < HTTP/1.1 401 Authorization Required
  > GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic <actual credentials>
  < HTTP/1.1 200 OK

In the current trace, you can see that libcurl insists on making a
second auth-less request after we've fed it credentials. I'm not sure
how to get rid of this useless extra round-trip, but it would be nice to
do so (IIRC, it is a probe request to find out the list of auth types
that the server supports, which are not remembered from the previous
request).

With http.emptyauth, the second round-trip _isn't_ useless. It's trying
to send the empty credential.

So while curl isn't currently optimizing out the second call, I think
http.emptyauth makes it harder to do the right thing. That's probably
fixable if the logic ends up more like:

  - curl reports a 401 to us; actually look at the list of auth methods.

  - if there was gss-negotiate, then kick in the empty-auth magic
    automatically.

  - if empty-auth failed (or if we decided not to try it), ask for a
    credential and retry the request. Either way, tell curl that we want
    to use "Basic" so it doesn't have to do the probe request (and
    obviously if the server did not support Basic, then fail
    immediately).

I think that would keep it to 2 round-trips for the normal "Basic" case,
as well as for the GSSNegotiate case. It would be 3 requests when the
server offers GSSNegotiate but you can't use it (but you could set
http.emptyauth=false to optimize that out).

That's all theoretical, though. I might not even be right about the
reason for the second request, and I certainly haven't written any code
(nor do I have a GSSNegotiate setup to test against).

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I don't think it incurs an extra round-trip now, because of the way
> libcurl works. Though I think it _does_ make it harder for curl to later
> optimize out that extra round-trip.
> ...
> In the current trace, you can see that libcurl insists on making a
> second auth-less request after we've fed it credentials. I'm not sure
> how to get rid of this useless extra round-trip, but it would be nice to
> do so (IIRC, it is a probe request to find out the list of auth types
> that the server supports, which are not remembered from the previous
> request).
> ...
> With http.emptyauth, the second round-trip _isn't_ useless. It's trying
> to send the empty credential.
> 
> So while curl isn't currently optimizing out the second call, I think
> http.emptyauth makes it harder to do the right thing.
> ...
> I think that would keep it to 2 round-trips for the normal "Basic" case,
> as well as for the GSSNegotiate case. It would be 3 requests when the
> server offers GSSNegotiate but you can't use it (but you could set
> http.emptyauth=false to optimize that out).

Thanks for your thoughts.  I'd think that we should take this change
and leave the optimization for later, then.  It's not like the
change of the default is making the normal situation any worse, it
seems.

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 01:25:11PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > I don't think it incurs an extra round-trip now, because of the way
> > libcurl works. Though I think it _does_ make it harder for curl to later
> > optimize out that extra round-trip.
> > ...
> > In the current trace, you can see that libcurl insists on making a
> > second auth-less request after we've fed it credentials. I'm not sure
> > how to get rid of this useless extra round-trip, but it would be nice to
> > do so (IIRC, it is a probe request to find out the list of auth types
> > that the server supports, which are not remembered from the previous
> > request).
> > ...
> > With http.emptyauth, the second round-trip _isn't_ useless. It's trying
> > to send the empty credential.
> > 
> > So while curl isn't currently optimizing out the second call, I think
> > http.emptyauth makes it harder to do the right thing.
> > ...
> > I think that would keep it to 2 round-trips for the normal "Basic" case,
> > as well as for the GSSNegotiate case. It would be 3 requests when the
> > server offers GSSNegotiate but you can't use it (but you could set
> > http.emptyauth=false to optimize that out).
> 
> Thanks for your thoughts.  I'd think that we should take this change
> and leave the optimization for later, then.  It's not like the
> change of the default is making the normal situation any worse, it
> seems.

I'm not excited that it will start making known bogus-username requests
by default to servers which do not even support Negotiate. I guess that
is really the server-operators problem, but it feels pretty hacky.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Feb 22, 2017 at 01:25:11PM -0800, Junio C Hamano wrote:
>> 
>> Thanks for your thoughts.  I'd think that we should take this change
>> and leave the optimization for later, then.  It's not like the
>> change of the default is making the normal situation any worse, it
>> seems.
>
> I'm not excited that it will start making known bogus-username requests
> by default to servers which do not even support Negotiate. I guess that
> is really the server-operators problem, but it feels pretty hacky.

I guess that's another valid concern.  The servers used to be able
to say "Ah, this repository needs auth and this request does not, so
reject it without asking the auth-db".  Now it must say "Ah, this
repository needs auth and this request does have one, but it is
empty so let's not even bother the auth-db" in order to reject a
useless "empty-auth" request with the same efficiency.

After the first request without auth (that fails), do we learn
anything useful from the server side (like "it knows Negotiate")
that we can use to flip the "empty-auth" bit to give a better
default to people from both worlds, I wonder...?

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 01:57:28PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > On Wed, Feb 22, 2017 at 01:25:11PM -0800, Junio C Hamano wrote:
> >> 
> >> Thanks for your thoughts.  I'd think that we should take this change
> >> and leave the optimization for later, then.  It's not like the
> >> change of the default is making the normal situation any worse, it
> >> seems.
> >
> > I'm not excited that it will start making known bogus-username requests
> > by default to servers which do not even support Negotiate. I guess that
> > is really the server-operators problem, but it feels pretty hacky.
> 
> I guess that's another valid concern.  The servers used to be able
> to say "Ah, this repository needs auth and this request does not, so
> reject it without asking the auth-db".  Now it must say "Ah, this
> repository needs auth and this request does have one, but it is
> empty so let's not even bother the auth-db" in order to reject a
> useless "empty-auth" request with the same efficiency.
> 
> After the first request without auth (that fails), do we learn
> anything useful from the server side (like "it knows Negotiate")
> that we can use to flip the "empty-auth" bit to give a better
> default to people from both worlds, I wonder...?

Yes, that's exactly what I was trying to say in my first message.

-Peff

Re: [PATCH] http(s): automatically try NTLM authentication first

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Feb 22, 2017 at 01:57:28PM -0800, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>> 
>> > On Wed, Feb 22, 2017 at 01:25:11PM -0800, Junio C Hamano wrote:
>> >> 
>> >> Thanks for your thoughts.  I'd think that we should take this change
>> >> and leave the optimization for later, then.  It's not like the
>> >> change of the default is making the normal situation any worse, it
>> >> seems.
>> >
>> > I'm not excited that it will start making known bogus-username requests
>> > by default to servers which do not even support Negotiate. I guess that
>> > is really the server-operators problem, but it feels pretty hacky.
>> 
>> I guess that's another valid concern.  The servers used to be able
>> to say "Ah, this repository needs auth and this request does not, so
>> reject it without asking the auth-db".  Now it must say "Ah, this
>> repository needs auth and this request does have one, but it is
>> empty so let's not even bother the auth-db" in order to reject a
>> useless "empty-auth" request with the same efficiency.
>> 
>> After the first request without auth (that fails), do we learn
>> anything useful from the server side (like "it knows Negotiate")
>> that we can use to flip the "empty-auth" bit to give a better
>> default to people from both worlds, I wonder...?
>
> Yes, that's exactly what I was trying to say in my first message.

I see.  I am still inclined to take this as-is for now to cook in
'next', though.  

A solution along your line would help Negotiate users OOB experience
without hurting the servers that do not offer Negotiate, but until
that materializes, users can set the lazier http.emptyAuth on
(without selectively setting http.<host>.emptyAuth off for sites
without Negotiate) and hurt the servers by throwing an empty auth
anyway regardless of the default, so the flipping of the default is
not fundamentally adding more harm in that sense.

Re: [PATCH] http(s): automatically try NTLM authentication first

[Jeff King]

On Wed, Feb 22, 2017 at 02:35:11PM -0800, Junio C Hamano wrote:

> A solution along your line would help Negotiate users OOB experience
> without hurting the servers that do not offer Negotiate, but until
> that materializes, users can set the lazier http.emptyAuth on
> (without selectively setting http.<host>.emptyAuth off for sites
> without Negotiate) and hurt the servers by throwing an empty auth
> anyway regardless of the default, so the flipping of the default is
> not fundamentally adding more harm in that sense.

I was hoping to materialize it today. :)

Here's what I came up with. I have a lot of questions about the second
patch which I'll outline there. But I think it may be a good start.

  [1/2]: http: restrict auth methods to what the server advertises
  [2/2]: http: add an "auto" mode for http.emptyauth

 http.c | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)

-Peff

[PATCH 1/2] http: restrict auth methods to what the server advertises

[Jeff King]

By default, we tell curl to use CURLAUTH_ANY, which does not
limit its set of auth methods. However, this results in an
extra round-trip to the server when authentication is
required. After we've fed the credential to curl, it wants
to probe the server to find its list of available methods
before sending an Authorization header.

We can shortcut this by limiting our http_auth_methods by
what the server told us it supports. In some cases (such as
when the server only supports Basic), that lets curl skip
the extra probe request.

The end result should look the same to the user, but you can
use GIT_TRACE_CURL to verify the sequence of requests:

  GIT_TRACE_CURL=1 \
  git ls-remote https://example.com/repo.git \
  2>&1 >/dev/null |
  egrep '(Send|Recv) header: (GET|HTTP|Auth)'

Before this patch, hitting a Basic-only server like
github.com results in:

  Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Recv header: HTTP/1.1 401 Authorization Required
  Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Recv header: HTTP/1.1 401 Authorization Required
  Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: Authorization: Basic <redacted>
  Recv header: HTTP/1.1 200 OK

And after:

  Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Recv header: HTTP/1.1 401 Authorization Required
  Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: Authorization: Basic <redacted>
  Recv header: HTTP/1.1 200 OK

The possible downsides are:

  - This only helps for a Basic-only server; for a server
    with multiple auth options, curl may still send a probe
    request to see which ones are available (IOW, there's no
    way to say "don't probe, I already know what the server
    will say").

  - The http_auth_methods variable is global, so this will
    apply to all further requests. That's acceptable for
    Git's usage of curl, though, which also treats the
    credentials as global. I.e., in any given program
    invocation we hit only one conceptual server (we may be
    redirected at the outset, but in that case that's whose
    auth_avail field we'd see).

Signed-off-by: Jeff King <peff@peff.net>
---
 http.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/http.c b/http.c
index 90a1c0f11..a05609766 100644
--- a/http.c
+++ b/http.c
@@ -1347,6 +1347,8 @@ static int handle_curl_result(struct slot_results *results)
 		} else {
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 			http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;
+			if (results->auth_avail)
+				http_auth_methods &= results->auth_avail;
 #endif
 			return HTTP_REAUTH;
 		}
-- 
2.12.0.rc2.597.g959f68882

[PATCH 2/2] http: add an "auto" mode for http.emptyauth

[Jeff King]

This variable needs to be specified to make some types of
non-basic authentication work, but ideally this would just
work out of the box for everyone.

However, simply setting it to "1" by default introduces an
extra round-trip for cases where it _isn't_ useful. We end
up sending a bogus empty credential that the server rejects.

Instead, let's introduce an automatic mode, that works like
this:

  1. We won't try to send the bogus credential on the first
     request. We'll wait to get an HTTP 401, as usual.

  2. After seeing an HTTP 401, the empty-auth hack will kick
     in only when we know there is an auth method beyond
     "Basic" to be tried.

That should make it work out of the box, without incurring
any extra round-trips for people hitting Basic-only servers.

This _does_ incur an extra round-trip if you really want to
use "Basic" but your server advertises other methods (the
emptyauth hack will kick in but fail, and then Git will
actually ask for a password).

The auto mode may incur an extra round-trip over setting
http.emptyauth=true, because part of the emptyauth hack is
to feed this blank password to curl even before we've made a
single request.

Signed-off-by: Jeff King <peff@peff.net>
---
My open questions are:

  - I don't have anything but a Basic server to test against. So it's
    entirely possible that this doesn't actually work in the NTLM case.

  - what does a request log look like for somebody actually using NTLM?
    It's possible if the initial request sends a restrict auth_avail,
    that curl could get away without the extra probe request, and we'd
    end up with the same number of requests for "auto" mode versus
    http.emptyauth=true.

  - the whole "don't use this on the initial request" flag feels really
    hacky. It's a side effect of how emptyauth tries to kick in even
    before we have sent any requests. Probably it should have been
    handled in the 401 code path originally, but I'm hesitant to change
    it now. I suspect it is eliminating a round-trip in practice when it
    is enabled.

  - I didn't test a server that advertises Basic and something else, but
    really only takes Basic. So I'm just assuming that it incurs the
    extra round-trip (actually probably two, one for curl's method
    probe).

  - When your curl is too old to do CURLAUTH_ANY, I just left the
    default to disable emptyauth. But it could easily be "1" if people
    care.

 http.c | 38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/http.c b/http.c
index a05609766..ea70ec1ee 100644
--- a/http.c
+++ b/http.c
@@ -109,7 +109,7 @@ static int curl_save_cookies;
 struct credential http_auth = CREDENTIAL_INIT;
 static int http_proactive_auth;
 static const char *user_agent;
-static int curl_empty_auth;
+static int curl_empty_auth = -1;
 
 enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL;
 
@@ -125,6 +125,7 @@ static struct credential cert_auth = CREDENTIAL_INIT;
 static int ssl_cert_password_required;
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 static unsigned long http_auth_methods = CURLAUTH_ANY;
+static int http_auth_methods_restricted;
 #endif
 
 static struct curl_slist *pragma_header;
@@ -382,10 +383,37 @@ static int http_options(const char *var, const char *value, void *cb)
 	return git_default_config(var, value, cb);
 }
 
+static int curl_empty_auth_enabled(void)
+{
+	if (curl_empty_auth < 0) {
+#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
+		/*
+		 * In the automatic case, kick in the empty-auth
+		 * hack as long as we would potentially try some
+		 * method more exotic than "Basic".
+		 *
+		 * But only do so when this is _not_ our initial
+		 * request, as we would not then yet know what
+		 * methods are available.
+		 */
+		return http_auth_methods_restricted &&
+		       http_auth_methods != CURLAUTH_BASIC;
+#else
+		/*
+		 * Our libcurl is too old to do AUTH_ANY in the first place;
+		 * just default to turning the feature off.
+		 */
+		return 0;
+#endif
+	}
+
+	return curl_empty_auth;
+}
+
 static void init_curl_http_auth(CURL *result)
 {
 	if (!http_auth.username || !*http_auth.username) {
-		if (curl_empty_auth)
+		if (curl_empty_auth_enabled())
 			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
 		return;
 	}
@@ -1079,7 +1107,7 @@ struct active_request_slot *get_active_slot(void)
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
 #endif
-	if (http_auth.password || curl_empty_auth)
+	if (http_auth.password || curl_empty_auth_enabled())
 		init_curl_http_auth(slot->curl);
 
 	return slot;
@@ -1347,8 +1375,10 @@ static int handle_curl_result(struct slot_results *results)
 		} else {
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 			http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;
-			if (results->auth_avail)
+			if (results->auth_avail) {
 				http_auth_methods &= results->auth_avail;
+				http_auth_methods_restricted = 1;
+			}
 #endif
 			return HTTP_REAUTH;
 		}
-- 
2.12.0.rc2.597.g959f68882

RE: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[David Turner]

I don't know enough about how libcurl handles authentication to know whether 
these patches are a good idea, but I have a minor comment anyway.

> -----Original Message-----
> From: Jeff King [mailto:peff@peff.net]
> +static int curl_empty_auth_enabled(void) {
> +	if (curl_empty_auth < 0) {
> +#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
> +		/*
> +		 * In the automatic case, kick in the empty-auth
> +		 * hack as long as we would potentially try some
> +		 * method more exotic than "Basic".
> +		 *
> +		 * But only do so when this is _not_ our initial
> +		 * request, as we would not then yet know what
> +		 * methods are available.
> +		 */

Eliminate double-negative:

"But only do this when this is our second or subsequent request, 
as by then we know what methods are available."

Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[Jeff King]

On Thu, Feb 23, 2017 at 01:16:33AM +0000, David Turner wrote:

> I don't know enough about how libcurl handles authentication to know whether 
> these patches are a good idea, but I have a minor comment anyway.

As somebody who is using non-Basic auth, can you apply these patches and
show us the output of:

   GIT_TRACE_CURL=1 \
   git ls-remote https://your-server 2>&1 >/dev/null |
   egrep '(Send|Recv) header: (GET|HTTP|Auth)'

(without http.emptyauth turned on, obviously).

> > +		 * But only do so when this is _not_ our initial
> > +		 * request, as we would not then yet know what
> > +		 * methods are available.
> > +		 */
> 
> Eliminate double-negative:
> 
> "But only do this when this is our second or subsequent request, 
> as by then we know what methods are available."

Yeah, that is clearer.

-Peff

RE: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[David Turner]

> -----Original Message-----
> From: Jeff King [mailto:peff@peff.net]
> Sent: Wednesday, February 22, 2017 8:38 PM
> To: David Turner <David.Turner@twosigma.com>
> Cc: Junio C Hamano <gitster@pobox.com>; git@vger.kernel.org;
> sandals@crustytoothpaste.net; Johannes Schindelin
> <johannes.schindelin@gmx.de>; Eric Sunshine <sunshine@sunshineco.com>
> Subject: Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth
> 
> On Thu, Feb 23, 2017 at 01:16:33AM +0000, David Turner wrote:
> 
> > I don't know enough about how libcurl handles authentication to know
> > whether these patches are a good idea, but I have a minor comment
> anyway.
> 
> As somebody who is using non-Basic auth, can you apply these patches and
> show us the output of:
> 
>    GIT_TRACE_CURL=1 \
>    git ls-remote https://your-server 2>&1 >/dev/null |
>    egrep '(Send|Recv) header: (GET|HTTP|Auth)'
> 
> (without http.emptyauth turned on, obviously).

The results appear to be identical with and without
the patch.  With http.emptyauth turned off,
16:27:28.208924 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
16:27:28.212872 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required
Username for 'http://git': [I just pressed enter]
Password for 'http://git': [ditto]
16:27:29.928872 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
16:27:29.929787 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required

(if someone else wants to replicate this, delete >/dev/null bit 
from Jeff's shell snippet)

Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[Jeff King]

On Thu, Feb 23, 2017 at 04:31:13PM +0000, David Turner wrote:

> > As somebody who is using non-Basic auth, can you apply these patches and
> > show us the output of:
> > 
> >    GIT_TRACE_CURL=1 \
> >    git ls-remote https://your-server 2>&1 >/dev/null |
> >    egrep '(Send|Recv) header: (GET|HTTP|Auth)'
> > 
> > (without http.emptyauth turned on, obviously).
> 
> The results appear to be identical with and without
> the patch.  With http.emptyauth turned off,
> 16:27:28.208924 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
> 16:27:28.212872 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required
> Username for 'http://git': [I just pressed enter]
> Password for 'http://git': [ditto]
> 16:27:29.928872 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
> 16:27:29.929787 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required

Just to be sure: did you remove http.emptyauth config completely from
your config files, or did you turn it to "false"? Because the new
behavior only kicks in when it isn't configured at all (probably we
should respect "auto" as a user-provided name).

> (if someone else wants to replicate this, delete >/dev/null bit 
> from Jeff's shell snippet)

Hrm, you shouldn't need to. The stderr redirection comes first, so it
should become the new stdout.

-Peff

RE: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[David Turner]

> -----Original Message-----
> From: Jeff King [mailto:peff@peff.net]
> Sent: Thursday, February 23, 2017 2:44 PM
> To: David Turner <David.Turner@twosigma.com>
> Cc: Junio C Hamano <gitster@pobox.com>; git@vger.kernel.org;
> sandals@crustytoothpaste.net; Johannes Schindelin
> <johannes.schindelin@gmx.de>; Eric Sunshine <sunshine@sunshineco.com>
> Subject: Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth
> 
> On Thu, Feb 23, 2017 at 04:31:13PM +0000, David Turner wrote:
> 
> > > As somebody who is using non-Basic auth, can you apply these patches
> > > and show us the output of:
> > >
> > >    GIT_TRACE_CURL=1 \
> > >    git ls-remote https://your-server 2>&1 >/dev/null |
> > >    egrep '(Send|Recv) header: (GET|HTTP|Auth)'
> > >
> > > (without http.emptyauth turned on, obviously).
> >
> > The results appear to be identical with and without the patch.  With
> > http.emptyauth turned off,
> > 16:27:28.208924 http.c:524              => Send header: GET
> /info/refs?service=git-upload-pack HTTP/1.1
> > 16:27:28.212872 http.c:524              <= Recv header: HTTP/1.1 401
> Authorization Required
> > Username for 'http://git': [I just pressed enter] Password for
> > 'http://git': [ditto]
> > 16:27:29.928872 http.c:524              => Send header: GET
> /info/refs?service=git-upload-pack HTTP/1.1
> > 16:27:29.929787 http.c:524              <= Recv header: HTTP/1.1 401
> Authorization Required
> 
> Just to be sure: did you remove http.emptyauth config completely from your
> config files, or did you turn it to "false"? Because the new behavior only kicks
> in when it isn't configured at all (probably we should respect "auto" as a user-
> provided name).

I turned it to false. With it completely removed, I get this, both times:

20:03:49.896797 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
20:03:49.900776 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required
20:03:49.900929 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
20:03:49.904754 http.c:524              <= Recv header: HTTP/1.1 401 Authorization Required
20:03:49.906649 http.c:524              => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
20:03:49.906654 http.c:524              => Send header: Authorization: Negotiate <redacted>
20:03:49.956753 http.c:524              <= Recv header: HTTP/1.1 200 OK - $gitservername

> > (if someone else wants to replicate this, delete >/dev/null bit from
> > Jeff's shell snippet)
> 
> Hrm, you shouldn't need to. The stderr redirection comes first, so it should
> become the new stdout.

Weird.  It didn't appear work earlier, but I must have screwed something up.
And I learned something about shell redirection.

Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[Johannes Schindelin]

Hi,

On Wed, 22 Feb 2017, Jeff King wrote:

> [two beautiful patches]

I applied them and verified that the reported issue is fixed. Thank you!

Hopefully you do not mind that I cherry-picked them in preparation for
Git for Windows v2.12.0?

I added a small fixup (https://github.com/dscho/git/commit/44ae0bcae5):

-- snip --
Subject: [PATCH] fixup! http: add an "auto" mode for http.emptyauth

Note: we keep a "black list" of authentication methods for which we do
not want to enable http.emptyAuth automatically. A white list would be
nicer, but less robust, as we want to support linking to several cURL
versions and the list of authentication methods (as well as their names)
changed over time.

[jes: actually added the "auto" handling, excluded Digest, too]

This fixes https://github.com/git-for-windows/git/issues/1034

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 http.c | 55 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 33 insertions(+), 22 deletions(-)

diff --git a/http.c b/http.c
index f8eb0f23d6c..fb94c444c80 100644
--- a/http.c
+++ b/http.c
@@ -334,7 +334,10 @@ static int http_options(const char *var, const char *value, void *cb)
 		return git_config_string(&user_agent, var, value);
 
 	if (!strcmp("http.emptyauth", var)) {
-		curl_empty_auth = git_config_bool(var, value);
+		if (value && !strcmp("auto", value))
+			curl_empty_auth = -1;
+		else
+			curl_empty_auth = git_config_bool(var, value);
 		return 0;
 	}
 
@@ -385,29 +388,37 @@ static int http_options(const char *var, const char *value, void *cb)
 
 static int curl_empty_auth_enabled(void)
 {
-	if (curl_empty_auth < 0) {
-#ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
-		/*
-		 * In the automatic case, kick in the empty-auth
-		 * hack as long as we would potentially try some
-		 * method more exotic than "Basic".
-		 *
-		 * But only do so when this is _not_ our initial
-		 * request, as we would not then yet know what
-		 * methods are available.
-		 */
-		return http_auth_methods_restricted &&
-		       http_auth_methods != CURLAUTH_BASIC;
+	if (curl_empty_auth >= 0)
+		return curl_empty_auth;
+
+#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY
+	/*
+	 * Our libcurl is too old to do AUTH_ANY in the first place;
+	 * just default to turning the feature off.
+	 */
 #else
-		/*
-		 * Our libcurl is too old to do AUTH_ANY in the first place;
-		 * just default to turning the feature off.
-		 */
-		return 0;
+	/*
+	 * In the automatic case, kick in the empty-auth
+	 * hack as long as we would potentially try some
+	 * method more exotic than "Basic".
+	 *
+	 * But only do this when this is our second or
+	 * subsequent * request, as by then we know what
+	 * methods are available.
+	 */
+	if (http_auth_methods_restricted)
+		switch (http_auth_methods) {
+		case CURLAUTH_BASIC:
+		case CURLAUTH_DIGEST:
+#ifdef CURLAUTH_DIGEST_IE
+		case CURLAUTH_DIGEST_IE:
 #endif
-	}
-
-	return curl_empty_auth;
+			return 0;
+		default:
+			return 1;
+		}
+#endif
+	return 0;
 }
 
 static void init_curl_http_auth(CURL *result)
-- snap --

As you can see, I actually implemented the handling for
http.emptyauth=auto, and I was more comfortable with handling the "easy"
cases first in the curl_empty_auth_enabled function.

I also took Dave's suggestion:

> On Thu, Feb 23, 2017 at 01:16:33AM +0000, David Turner wrote:
> 
> > > +		 * But only do so when this is _not_ our initial
> > > +		 * request, as we would not then yet know what
> > > +		 * methods are available.
> > > +		 */
> > 
> > Eliminate double-negative:
> > 
> > "But only do this when this is our second or subsequent request, 
> > as by then we know what methods are available."
> 
> Yeah, that is clearer.

Thank you all!

Now, how to get this into upstream Git, too? Jeff, do you want to submit a
v2? In that case, would you please consider the fixup! I mentioned above?
Otherwise I'd be happy to take it from here.

Ciao,
Dscho

Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth

[Jeff King]

On Sat, Feb 25, 2017 at 12:48:54PM +0100, Johannes Schindelin wrote:

> Hi,
> 
> On Wed, 22 Feb 2017, Jeff King wrote:
> 
> > [two beautiful patches]
> 
> I applied them and verified that the reported issue is fixed. Thank you!
> 
> Hopefully you do not mind that I cherry-picked them in preparation for
> Git for Windows v2.12.0?

No, I don't mind. I'm happy that more people with a non-Basic setup are
verifying that they work. :)

Of the changes:

> diff --git a/http.c b/http.c
> index f8eb0f23d6c..fb94c444c80 100644
> --- a/http.c
> +++ b/http.c
> @@ -334,7 +334,10 @@ static int http_options(const char *var, const char *value, void *cb)
>  		return git_config_string(&user_agent, var, value);
>  
>  	if (!strcmp("http.emptyauth", var)) {
> -		curl_empty_auth = git_config_bool(var, value);
> +		if (value && !strcmp("auto", value))
> +			curl_empty_auth = -1;
> +		else
> +			curl_empty_auth = git_config_bool(var, value);
>  		return 0;
>  	}

Obviously good, I should have included this in the original.

> +#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY
> +	/*
> +	 * Our libcurl is too old to do AUTH_ANY in the first place;
> +	 * just default to turning the feature off.
> +	 */
>  #else
> -		/*
> -		 * Our libcurl is too old to do AUTH_ANY in the first place;
> -		 * just default to turning the feature off.
> -		 */

The ifdef reordering here is good.

> +	/*
> +	 * In the automatic case, kick in the empty-auth
> +	 * hack as long as we would potentially try some
> +	 * method more exotic than "Basic".
> +	 *
> +	 * But only do this when this is our second or
> +	 * subsequent * request, as by then we know what
> +	 * methods are available.
> +	 */
> +	if (http_auth_methods_restricted)
> +		switch (http_auth_methods) {
> +		case CURLAUTH_BASIC:
> +		case CURLAUTH_DIGEST:
> +#ifdef CURLAUTH_DIGEST_IE
> +		case CURLAUTH_DIGEST_IE:
>  #endif
> [...]
> +			return 0;
> +		default:
> +			return 1;
> +		}

This is an improvement over my basic-only, but I think you actually want
to bitmask here. A server which advertises only BASIC|DIGEST should not
do empty-auth, but wouldn't match your switch statement.

Patch below.

> Now, how to get this into upstream Git, too? Jeff, do you want to submit a
> v2? In that case, would you please consider the fixup! I mentioned above?
> Otherwise I'd be happy to take it from here.

I don't mind doing a v2. I'm unsure of whether we want to default to
"auto" or not upstream. It seems from your releases that you think it is
safe enough to do in Windows. And I guess nobody outside of that is
really doing NTLM. So it's OK, I guess?

<shrug> I don't have enough information to make an intelligent opinion,
so I'm happy to defer.

I'll send my v2 in a minute. Here's the interdiff/fixup if you need to
apply it separately:

diff --git a/http.c b/http.c
index 523c43cf9..dd637d031 100644
--- a/http.c
+++ b/http.c
@@ -126,6 +126,13 @@ static int ssl_cert_password_required;
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 static unsigned long http_auth_methods = CURLAUTH_ANY;
 static int http_auth_methods_restricted;
+/* Modes for which empty_auth cannot actually help us. */
+static unsigned long empty_auth_useless =
+	CURLAUTH_BASIC
+#ifdef CURLAUTH_DIGEST_IE
+	| CURLAUTH_DIGEST_IE
+#endif
+	| CURLAUTH_DIGEST;
 #endif
 
 static struct curl_slist *pragma_header;
@@ -400,23 +407,15 @@ static int curl_empty_auth_enabled(void)
 	/*
 	 * In the automatic case, kick in the empty-auth
 	 * hack as long as we would potentially try some
-	 * method more exotic than "Basic".
+	 * method more exotic than "Basic" or "Digest".
 	 *
 	 * But only do this when this is our second or
 	 * subsequent * request, as by then we know what
 	 * methods are available.
 	 */
-	if (http_auth_methods_restricted)
-		switch (http_auth_methods) {
-		case CURLAUTH_BASIC:
-		case CURLAUTH_DIGEST:
-#ifdef CURLAUTH_DIGEST_IE
-		case CURLAUTH_DIGEST_IE:
-#endif
-			return 0;
-		default:
-			return 1;
-		}
+	if (http_auth_methods_restricted &&
+	    (http_auth_methods & ~empty_auth_useless))
+		return 1;
 #endif
 	return 0;
 }

[PATCH] http: add an "auto" mode for http.emptyauth

[Jeff King]

This variable needs to be specified to make some types of
non-basic authentication work, but ideally this would just
work out of the box for everyone.

However, simply setting it to "1" by default introduces an
extra round-trip for cases where it _isn't_ useful. We end
up sending a bogus empty credential that the server rejects.

Instead, let's introduce an automatic mode, that works like
this:

  1. We won't try to send the bogus credential on the first
     request. We'll wait to get an HTTP 401, as usual.

  2. After seeing an HTTP 401, the empty-auth hack will kick
     in only when we know there is an auth method available
     that might make use of it (i.e., something besides
     "Basic" or "Digest").

That should make it work out of the box, without incurring
any extra round-trips for people hitting Basic-only servers.

This _does_ incur an extra round-trip if you really want to
use "Basic" but your server advertises other methods (the
emptyauth hack will kick in but fail, and then Git will
actually ask for a password).

The auto mode may incur an extra round-trip over setting
http.emptyauth=true, because part of the emptyauth hack is
to feed this blank password to curl even before we've made a
single request.

Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Jeff King <peff@peff.net>
---
And here's the full patch. It is meant to go on top of the
already-queued 1/2, though I suspect it could apply separately.

Test reports welcome from people who actually have NTLM or Kerberos
servers. The changes from the previous are fairly minimal, but this kind
of bit-mangling is exactly the kind of thing where I tend to
accidentally invert the logic. ;)

 http.c | 50 +++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 45 insertions(+), 5 deletions(-)

diff --git a/http.c b/http.c
index a05609766..dd637d031 100644
--- a/http.c
+++ b/http.c
@@ -109,7 +109,7 @@ static int curl_save_cookies;
 struct credential http_auth = CREDENTIAL_INIT;
 static int http_proactive_auth;
 static const char *user_agent;
-static int curl_empty_auth;
+static int curl_empty_auth = -1;
 
 enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL;
 
@@ -125,6 +125,14 @@ static struct credential cert_auth = CREDENTIAL_INIT;
 static int ssl_cert_password_required;
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 static unsigned long http_auth_methods = CURLAUTH_ANY;
+static int http_auth_methods_restricted;
+/* Modes for which empty_auth cannot actually help us. */
+static unsigned long empty_auth_useless =
+	CURLAUTH_BASIC
+#ifdef CURLAUTH_DIGEST_IE
+	| CURLAUTH_DIGEST_IE
+#endif
+	| CURLAUTH_DIGEST;
 #endif
 
 static struct curl_slist *pragma_header;
@@ -333,7 +341,10 @@ static int http_options(const char *var, const char *value, void *cb)
 		return git_config_string(&user_agent, var, value);
 
 	if (!strcmp("http.emptyauth", var)) {
-		curl_empty_auth = git_config_bool(var, value);
+		if (value && !strcmp("auto", value))
+			curl_empty_auth = -1;
+		else
+			curl_empty_auth = git_config_bool(var, value);
 		return 0;
 	}
 
@@ -382,10 +393,37 @@ static int http_options(const char *var, const char *value, void *cb)
 	return git_default_config(var, value, cb);
 }
 
+static int curl_empty_auth_enabled(void)
+{
+	if (curl_empty_auth >= 0)
+		return curl_empty_auth;
+
+#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY
+	/*
+	 * Our libcurl is too old to do AUTH_ANY in the first place;
+	 * just default to turning the feature off.
+	 */
+#else
+	/*
+	 * In the automatic case, kick in the empty-auth
+	 * hack as long as we would potentially try some
+	 * method more exotic than "Basic" or "Digest".
+	 *
+	 * But only do this when this is our second or
+	 * subsequent * request, as by then we know what
+	 * methods are available.
+	 */
+	if (http_auth_methods_restricted &&
+	    (http_auth_methods & ~empty_auth_useless))
+		return 1;
+#endif
+	return 0;
+}
+
 static void init_curl_http_auth(CURL *result)
 {
 	if (!http_auth.username || !*http_auth.username) {
-		if (curl_empty_auth)
+		if (curl_empty_auth_enabled())
 			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
 		return;
 	}
@@ -1079,7 +1117,7 @@ struct active_request_slot *get_active_slot(void)
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
 #endif
-	if (http_auth.password || curl_empty_auth)
+	if (http_auth.password || curl_empty_auth_enabled())
 		init_curl_http_auth(slot->curl);
 
 	return slot;
@@ -1347,8 +1385,10 @@ static int handle_curl_result(struct slot_results *results)
 		} else {
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 			http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;
-			if (results->auth_avail)
+			if (results->auth_avail) {
 				http_auth_methods &= results->auth_avail;
+				http_auth_methods_restricted = 1;
+			}
 #endif
 			return HTTP_REAUTH;
 		}
-- 
2.12.0.616.g5f622f3b1

Re: [PATCH] http: add an "auto" mode for http.emptyauth

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> The auto mode may incur an extra round-trip over setting
> http.emptyauth=true, because part of the emptyauth hack is
> to feed this blank password to curl even before we've made a
> single request.

IOW, people who care about an extra round-trip have this workaround,
which is good.

This, along with the possible security implications, may want to be
added to the documentation but that is outside the topic of this
change, and I think we would want to see such an update come from
those who actually use NTLM (or Kerberos, but they know they have
minimum security implications).

> +#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY
> +	/*
> +	 * Our libcurl is too old to do AUTH_ANY in the first place;
> +	 * just default to turning the feature off.
> +	 */
> +#else
> +	/*
> +	 * In the automatic case, kick in the empty-auth
> +	 * hack as long as we would potentially try some
> +	 * method more exotic than "Basic" or "Digest".
> +	 *
> +	 * But only do this when this is our second or
> +	 * subsequent * request, as by then we know what

I'll drop the '*' that you left while line-wrapping ;-)

> +	 * methods are available.
> +	 */

Thanks.  This looks good.

Re: [PATCH] http: add an "auto" mode for http.emptyauth

[Johannes Schindelin]

Hi,

On Mon, 27 Feb 2017, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > The auto mode may incur an extra round-trip over setting
> > http.emptyauth=true, because part of the emptyauth hack is to feed
> > this blank password to curl even before we've made a single request.
> 
> IOW, people who care about an extra round-trip have this workaround,
> which is good.
> 
> This, along with the possible security implications, may want to be
> added to the documentation but that is outside the topic of this change,
> and I think we would want to see such an update come from those who
> actually use NTLM (or Kerberos, but they know they have minimum security
> implications).
> 
> > +#ifndef LIBCURL_CAN_HANDLE_AUTH_ANY +	/* +	 * Our libcurl is
> > too old to do AUTH_ANY in the first place; +	 * just default to
> > turning the feature off.  +	 */ +#else +	/* +	 * In the
> > automatic case, kick in the empty-auth +	 * hack as long as we
> > would potentially try some +	 * method more exotic than "Basic"
> > or "Digest".  +	 * +	 * But only do this when this is our
> > second or +	 * subsequent * request, as by then we know what
> 
> I'll drop the '*' that you left while line-wrapping ;-)
> 
> > +	 * methods are available.  +	 */
> 
> Thanks.  This looks good.

I replaced the previous version in Git for Windows' `master` branch with
the one in `pu`.

Thanks,
Johannes