FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

Need help to modify git to work with the http protocol _and_
a proxy using HTLM authentication.

--

The following changes were made to http.c (for NTLM authentication):

$ diff -c /mnt/nfs/git/git-1.5.4.2/http.c.orig \
    /mnt/nfs/git/git-1.5.4.2/http.c
--- 215,220 ----
--- 215,223 ----
        if (curl_http_proxy)
                curl_easy_setopt(result, CURLOPT_PROXY,
curl_http_proxy);

+       curl_easy_setopt(result, CURLOPT_PROXYAUTH,
(long)CURLAUTH_NTLM);
+       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
+               "<user-id>:<password>");
+
        return result;
  }
$

--

git fetch _not_ working via an NTML proxy:

$ http_proxy=<proxy-domain>:<proxy-port> GIT_CURL_VERBOSE=1 \
    git fetch http://www.kernel.org/pub/scm/git/git.git

[A few paragraphs of proxy authentication debug output that
BogoFilter apparently gives a high SPAM quotient to, so I've
given up trying to include it.]

fatal: Couldn't find remote ref HEAD
$ 

It seems that git fetch (via HTLM proxy) works until

> fatal: Couldn't find remote ref HEAD

git 1.5.4.2 dies on line 158 of builtin-fetch.c,
because the call to get_remote_ref() in remote.c fails.

Does anything need to be changed in remote.c?

Any other suggestions?

Thanks,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Daniel Stenberg]

On Tue, 26 Feb 2008, Ken.Fuchs@bench.com wrote:

> +       curl_easy_setopt(result, CURLOPT_PROXYAUTH, (long)CURLAUTH_NTLM);
> +       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
> +               "<user-id>:<password>");

First, you should rather allow any auth and not just the specific one you 
want.

Then, the userid and password is probably better passed in embedded in the 
proxy URL as that's given on the command line/environment already. Or as 
separate arguments.

> It seems that git fetch (via HTLM proxy) works until
>
>> fatal: Couldn't find remote ref HEAD

Well, the CURLOPT_PROXY is set in transport.c as well which your patch didn't 
address. If that's the case, I figure the verbose output should've shown some 
auth failures with the proxy?

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> Ken Fuchs wrote:
> 
> > +       curl_easy_setopt(result, CURLOPT_PROXYAUTH, 
> (long)CURLAUTH_NTLM);
> > +       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
> > +               "<user-id>:<password>");

Daniel Stenberg wrote:

> First, you should rather allow any auth and not just the 
> specific one you want.
> 
> Then, the userid and password is probably better passed in 
> embedded in the proxy URL as that's given on the command
> line/environment already. Or as separate arguments.

Agreed.  (I'd just like to get it working first.)

> > It seems that git fetch (via HTLM proxy) works until
> >
> >> fatal: Couldn't find remote ref HEAD
> 
> Well, the CURLOPT_PROXY is set in transport.c as well which 
> your patch didn't address.

Thanks, I just did a similar patch to transport.c:

$ diff -u ../git-1.5.4.2/transport.c.orig \
          ../git-1.5.4.2/transport.c
@@ -456,7 +456,8 @@
        if (transport->remote->http_proxy)
                curl_easy_setopt(slot->curl, CURLOPT_PROXY,
                                 transport->remote->http_proxy);
-
+       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
(long)CURLAUTH_NTLM);
+       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
"<user-id>:<password>");
        if (start_active_slot(slot)) {
                run_active_slot(slot);
                if (results.curl_result != CURLE_OK) {
$

> If that's the case, I figure the verbose output 
> should've shown some auth failures with the proxy?

No, actually the last 2 lines of debug output (unchanged) is:

* Connection #0 to host <proxy domain> left intact
fatal: Couldn't find remote ref HEAD

So, the proxy communication via NTLM authentication seems to be working.
The patch to transport.c did not change anything as far as I can see.

The fatal error is from remote.c.  Perhaps, it also requires some
changes.

--

Example of curl (sans git) working via an NTLM proxy:

$ curl --proxy-ntlm --proxy-user <user-id> \
    --proxy <proxy-domain>:<proxy-port> http://slashdot.org/
Enter proxy password for user '<user-id>': <non-echoed password>
<HTML of /. home page>

Thus, it seems that git could be modified to work via HTLM
authentication, but my simple changes to http.c and transport.c
above are probably not sufficient.

Suggestions are most welcome.

Thanks,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Tue, Feb 26, 2008 at 05:46:21PM -0600, Ken.Fuchs@bench.com wrote:
> > Well, the CURLOPT_PROXY is set in transport.c as well which 
> > your patch didn't address.
> 
> Thanks, I just did a similar patch to transport.c:
> 
> $ diff -u ../git-1.5.4.2/transport.c.orig \
>           ../git-1.5.4.2/transport.c
> @@ -456,7 +456,8 @@
>         if (transport->remote->http_proxy)
>                 curl_easy_setopt(slot->curl, CURLOPT_PROXY,
>                                  transport->remote->http_proxy);
> -
> +       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
> (long)CURLAUTH_NTLM);
> +       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
> "<user-id>:<password>");
>         if (start_active_slot(slot)) {
>                 run_active_slot(slot);
>                 if (results.curl_result != CURLE_OK) {
> $

Starting with curl 7.14.1, you're supposed to be able to use the
http://user:pass@proxy/ syntax, though I'm not sure it deals well with
NTLM domains. You can probably leave CURLOPT_PROXYUSERPWD out if you
set your proxy url correctly.

As for CURLOPT_PROXYAUTH, it would be better to set it from another
config.

Note that remote.<name>.proxy config doesn't work as expected, you
should use http.proxy which just work (and the change in transport.c is
useless, then). I have, as part of by http-refactoring topic, a patch
for remote.<name>.proxy to work better, though it doesn't support
changing the proxy authentication method.

> > If that's the case, I figure the verbose output 
> > should've shown some auth failures with the proxy?
> 
> No, actually the last 2 lines of debug output (unchanged) is:
> 
> * Connection #0 to host <proxy domain> left intact
> fatal: Couldn't find remote ref HEAD
> 
> So, the proxy communication via NTLM authentication seems to be working.
> The patch to transport.c did not change anything as far as I can see.
> 
> The fatal error is from remote.c.  Perhaps, it also requires some
> changes.

Does your remote have a HEAD ref ?

Mike

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Wed, Feb 27, 2008 at 08:20:12AM +0100, Mike Hommey wrote:
> On Tue, Feb 26, 2008 at 05:46:21PM -0600, Ken.Fuchs@bench.com wrote:
> > > Well, the CURLOPT_PROXY is set in transport.c as well which 
> > > your patch didn't address.
> > 
> > Thanks, I just did a similar patch to transport.c:
> > 
> > $ diff -u ../git-1.5.4.2/transport.c.orig \
> >           ../git-1.5.4.2/transport.c
> > @@ -456,7 +456,8 @@
> >         if (transport->remote->http_proxy)
> >                 curl_easy_setopt(slot->curl, CURLOPT_PROXY,
> >                                  transport->remote->http_proxy);
> > -
> > +       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
> > (long)CURLAUTH_NTLM);
> > +       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
> > "<user-id>:<password>");
> >         if (start_active_slot(slot)) {
> >                 run_active_slot(slot);
> >                 if (results.curl_result != CURLE_OK) {
> > $
> 
> Starting with curl 7.14.1, you're supposed to be able to use the
> http://user:pass@proxy/ syntax, though I'm not sure it deals well with
> NTLM domains. You can probably leave CURLOPT_PROXYUSERPWD out if you
> set your proxy url correctly.
> 
> As for CURLOPT_PROXYAUTH, it would be better to set it from another
> config.

Or we should set it to CURLOPT_AUTHANY by default.

Mike

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> > $ GIT_CURL_VERBOSE=1 git clone \
> >   http://www.kernel.org/pub/scm/git/git.git
> > Initialized empty Git repository in /mnt/s2u1/git/test-git/git/.git/
> > Cannot get remote repository information.
> > Perhaps git-update-server-info needs to be run there?
> > $
 
> git-clone is not a builtin yet. When it is (and a patch has come for
> that), it will correctly use proxys.

Is there a work-around for cloning a git repository via a proxy?

Worse case, it should be possible to clone a repository on a
machine that doesn't require a proxy.  Convert the repository
into a tarball, copy it to the machine that requires a proxy
and change the config to work with the proxy, right?  At this
point, the git fetch command should work with the proxy, since
the local repository now has the required refs, right?

Thanks,

Ken Fuchs

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Johannes Schindelin]

Hi,

On Wed, 27 Feb 2008, Ken.Fuchs@bench.com wrote:

[Ken did not say who said this:]

> > git-clone is not a builtin yet. When it is (and a patch has come for
> > that), it will correctly use proxys.
> 
> Is there a work-around for cloning a git repository via a proxy?

Yes.

Just export "http_proxy".  AFAICT this works... See

http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb2f371afa84dcf56

for a working example.

Hth,
Dscho

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

On Wed, 27 Feb 2008, Johannes Schindelin wrote:

> On Wed, 27 Feb 2008, Ken Fuchs wrote:
> 
> [Ken did not say who said this:]

Sorry, Mike Hommey wrote the following on 27 Feb 2008:

> > > git-clone is not a builtin yet. When it is (and a patch 
> > > has come for that), it will correctly use proxys.
> > 
> > Is there a work-around for cloning a git repository via a proxy?

> Yes.
> 
> Just export "http_proxy".  AFAICT this works... See
> 
>
http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb
2f371afa84dcf56

> for a working example.

Unfortunately, this will not work because "git clone" is
implemented by git-clone.sh and has no support for dealing
with proxies at all.  Note that the example uses "git fetch"
and not "get clone".

So it seems that git's http protocol via a proxy isn't completely
supported.  For example, "git fetch" works with a proxy, since
it is implemented via a builtin (executable) which includes
proxy support.

At least that is my understanding of Mike's comments above.

The only work-around for "git clone" that I'm aware of is a
"proxy agent" that intercepts and modifies all Internet bound
requests and responses so they go (transparently) through the
proxy server rather than directly to the Internet.  Thus, none
of the clients being used need to be proxy aware, including
"git clone" (or any git function for that matter).

However, I was looking for a different work-around for
using "git clone" via http protocol and proxy (NTLM auth.),
but there do not appear to be any.

Thanks,

Ken Fuchs

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Johannes Schindelin]

Hi,

On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:

> On Wed, 27 Feb 2008, Johannes Schindelin wrote:
> 
> > On Wed, 27 Feb 2008, Ken Fuchs wrote:
> > 
> > [Ken did not say who said this:]
> 
> Sorry, Mike Hommey wrote the following on 27 Feb 2008:
> 
> > > > git-clone is not a builtin yet. When it is (and a patch 
> > > > has come for that), it will correctly use proxys.
> > > 
> > > Is there a work-around for cloning a git repository via a proxy?
> 
> > Yes.
> > 
> > Just export "http_proxy".  AFAICT this works... See
> > 
> >
> http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb
> 2f371afa84dcf56
> 
> > for a working example.
> 
> Unfortunately, this will not work because "git clone" is
> implemented by git-clone.sh and has no support for dealing
> with proxies at all.

I fail to see why git-clone.sh, which uses git http-fetch directly, should 
behave differently from git fetch, which uses git http-fetch directly.

I strongly suspect that you did not even try what I suggested.

> So it seems that git's http protocol via a proxy isn't completely 
> supported.

If that should really be the case, well, Git is a programmers' tool, so 
you should be perfectly able to find out what is happening _exactly_.

If not, well, I heard subversion and Mercurial have nice Windows support.

Ciao,
Dscho

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:
> 
> > Unfortunately, this will not work because "git clone" is
> > implemented by git-clone.sh and has no support for dealing
> > with proxies at all.
> 
> I fail to see why git-clone.sh, which uses git http-fetch 
> directly, should behave differently from git fetch, which
> uses git http-fetch directly.

Sorry, I wasn't very clear.  When I said "git fetch" works, I meant
only the proxy is initiated properly.  "git fetch" itself still fails:

fatal: Couldn't find remote ref HEAD 

As I understand it, this is due to "git clone" failing earlier.

> I strongly suspect that you did not even try what I suggested.

I know you were trying to help.  I sincerely regret not thanking
you in my earlier response to your suggestion.  However, I had
already tried your suggestion over a week ago (plus numerous
user-id:password variants of the http_proxy environment variable).
I also tried "git config http.proxy <proxy.domain>:<proxy-port>".
I did not try "git config remote.<name>.proxy" which had an issue
that was recently fixed via a patch submission a few days ago.

> > So it seems that git's http protocol via a proxy isn't completely 
> > supported.
> 
> If that should really be the case, well, Git is a programmers' tool,
so 
> you should be perfectly able to find out what is happening _exactly_.

Well, all git http proxy stuff is handled by curl, so GIT_CURL_VERBOSE=1
should get us pretty close to "what is happening _exactly_":

$ GIT_CURL_VERBOSE=1 git clone http://www.kernel.org/pub/scm/git/git.git
Initialized empty Git repository in /mnt/nfs/git/test-git/git/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?
$

No verbose output from curl?  How could that happen?  Maybe
GIT_CURL_VERBOSE=1 isn't logging all curl (especially proxy) actions?

So, let's try tcpdump to see what's happening on the wire.  In another
shell (logged in as root), we start:
"tcpdump -i eth0 | grep <proxy.domain>"

We should at least see DNS lookup requests in the tcpdump for the
<proxy.domain> if curl is using the proxy.

We repeat the above "git clone" command and find that absolutely no
proxy related traffic has been sent via eth0.  (The above tcpdump
pipe to grep generated no output whatsoever.)

On the other hand, I know that "git fetch" does properly interact with
the proxy, so as a benchmark, we repeat the tcpdump experiment with
"git fetch".  Now "tcpdump -i eth0 | grep <proxy.domain>" generates
plenty of output, consisting of DNS requests/responses for
<proxy.domain> and many packets between the git client and the proxy.

Also, the command

$ GIT_CURL_VERBOSE=1 git fetch http://www.kernel.org/pub/scm/git/git.git

generates plenty of curl logging information (indicating successful
proxy actions).  Since Bogofilter thinks this type of output is SPAM,
I won't try to include it all, but just the last 2 lines:

* Connection #0 to host <proxy.domain> left intact
fatal: Couldn't find remote ref HEAD

Again, as I understand it, "git fetch" (which is proxy aware) is
failing not due to proxy issues, but due to missing git filesystem
structure that "git clone" (which is not proxy aware) needed to provide.
If the necessary git filesystem structure existed in the local git
repository, the "git fetch" command would probably succeed via http
proxy.

Before closing, I'll note that http.c was patched slightly to allow
NTLM authentication via an http proxy server.  The informal patches
can be found earlier in this thread; they are clearly a
work-in-progress.
For example, proper user interface mechanisms (via "git config") still
need to be developed/incorporated.

Sincerely,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Fri, Feb 29, 2008 at 02:06:49PM -0600, Ken.Fuchs@bench.com wrote:
> > On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:
> > 
> > > Unfortunately, this will not work because "git clone" is
> > > implemented by git-clone.sh and has no support for dealing
> > > with proxies at all.
> > 
> > I fail to see why git-clone.sh, which uses git http-fetch 
> > directly, should behave differently from git fetch, which
> > uses git http-fetch directly.
> 
> Sorry, I wasn't very clear.  When I said "git fetch" works, I meant
> only the proxy is initiated properly.  "git fetch" itself still fails:
> 
> fatal: Couldn't find remote ref HEAD 
> 
> As I understand it, this is due to "git clone" failing earlier.
> 
> > I strongly suspect that you did not even try what I suggested.
> 
> I know you were trying to help.  I sincerely regret not thanking
> you in my earlier response to your suggestion.  However, I had
> already tried your suggestion over a week ago (plus numerous
> user-id:password variants of the http_proxy environment variable).
> I also tried "git config http.proxy <proxy.domain>:<proxy-port>".
> I did not try "git config remote.<name>.proxy" which had an issue
> that was recently fixed via a patch submission a few days ago.
> 
> > > So it seems that git's http protocol via a proxy isn't completely 
> > > supported.
> > 
> > If that should really be the case, well, Git is a programmers' tool,
> so 
> > you should be perfectly able to find out what is happening _exactly_.
> 
> Well, all git http proxy stuff is handled by curl, so GIT_CURL_VERBOSE=1
> should get us pretty close to "what is happening _exactly_":
> 
> $ GIT_CURL_VERBOSE=1 git clone http://www.kernel.org/pub/scm/git/git.git
> Initialized empty Git repository in /mnt/nfs/git/test-git/git/.git/
> Cannot get remote repository information.
> Perhaps git-update-server-info needs to be run there?
> $

git clone is a shell script and doesn't put all the wrap-up around curl.
GIT_CURL_VERBOSE is of no effect, and git http proxy settings won't work.
Only http_proxy environment variable will work.

You can try modifying git-clone and add --proxy-ntlm or --proxy-anyauth
to curl calls, and set the http_proxy environment variable.

Alternatively, you can try again with (work in progress) builtin-clone.c.
http://thread.gmane.org/gmane.comp.version-control.git/75070/focus=75126

Mike