Git doesn't honor NO_PROXY environment variable while cloning

Git doesn't honor NO_PROXY environment variable while cloning

[Ondrej Pohorelsky]

Hi,

we've got a bug[0] reported in Red Hat Bugzilla. It seems like Git
doesn't honor NO_PROXY environment variable while cloning repositories

Reporter has provided these steps to reproduce using Git version 2.18.4:

1. Start a ubi8 container with podman:
```
$ podman run --rm -it registry.access.redhat.com/ubi8/ubi /bin/bash
```
2. Install git in the container - `# yum install -y git`
3. Add "bad" _PROXY environment variables:
```
# export HTTP_PROXY=http://user:pass@bad.proxy
# export HTTPS_PROXY=https://user:pass@bad.proxy
```
4. Add Github as an ignorable proxy domain:
```
# export NO_PROXY=github.com
```
5. Clone a github repo:
```
# mkdir -p /tmp/clone-test
# cd /tmp/clone-test
# git clone https://github.com/sclorg/nodejs-ex.git
# echo $?
```

Actual results:
```
# git clone https://github.com/sclorg/nodejs-ex.git
Cloning into 'nodejs-ex'...
# echo $?
128
```

Expected results:

Git clone succeeds, bypassing the proxy.



However I've found out that this possible issue is present even in
newer versions e.g. 2.28.0.

There is a workaround. You can rewrite proxy to be blank in .gitconfig
for specific websites.
```
[http "https://github.com/"]
        proxy = ""
```

Is this an issue or expected behaviour? And if it is expected
behaviour, then why?

Best regards,
Ondřej Pohořelský


[0]https://bugzilla.redhat.com/show_bug.cgi?id=1875639

Re: Git doesn't honor NO_PROXY environment variable while cloning

[Jeff King]

[tl;dr This looks to me like a problem in libcurl. I've included my
 whole line of reasoning below. Folks interested in libcurl might want
 to jump straight to the backtrace].

On Fri, Sep 11, 2020 at 02:15:59PM +0200, Ondrej Pohorelsky wrote:

> we've got a bug[0] reported in Red Hat Bugzilla. It seems like Git
> doesn't honor NO_PROXY environment variable while cloning repositories

It's not so much "doesn't honor NO_PROXY" as "NO_PROXY seems to trigger
another bug". Here's a slightly simplified recipe:

  [set up our bogus proxy]
  $ export HTTPS_PROXY=https://user:pass@bad.proxy

  [so far so good; we expect this to complain about the proxy]
  $ git clone https://github.com/git/git
  Cloning into 'git'...
  fatal: unable to access 'https://github.com/git/git/': Could not resolve proxy: bad.proxy

  [now let's set NO_PROXY]
  $ export NO_PROXY=github.com
  $ git clone https://github.com/git/git
  Cloning into 'git'...
  $ echo $?
  128

So we are looking at the variable, but then we exit with failure (and
with nothing to stderr!). That was with v2.28 above. But let's try it
with the current tip of master:

  $ git clone https://github.com/git/git
  Cloning into 'git'...
  error: git-remote-https died of signal 11

Ah, that's interesting. The switch in behavior is due to 675df192c5
(transport-helper: do not run git-remote-ext etc. in dashed form,
2020-08-26). Before that patch the transport code ran git-remote-https
directly. If it returned a non-zero exit code, we just assumed it had
printed its own error to stdout and exited. But after that patch we run
"git remote-https", and the git.c wrapper has code to complain about
segfaults.

So that's a little digression from our real bug. But it points us in the
right direction: remote-https is segfaulting. So here's a more direct
reproduction:

  export HTTPS_PROXY=https://user:pass@bad.proxy
  export NO_PROXY=github.com
  echo list | ./git-remote-https https://github.com/git/git

which results in a segfault. And we can run that process under gdb to
get a backtrace:

  echo list >input
  gdb -ex 'set args https://github.com/git/git <input' \
      -ex 'run' \
      -ex 'bt' \
      ./git-remote-https

which yields (after install debug symbols for system curl, etc):

  #0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
  #1  0x00007ffff7da6934 in __GI___inet_pton (af=af@entry=2, src=src@entry=0x0, dst=dst@entry=0x7fffffffd600)
      at inet_pton.c:69

Well, the issue is clear enough. Somebody is passing NULL to
inet_pton(). But who?

  #2  0x00007ffff7f88a08 in ossl_connect_step1 (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0)
      at vtls/openssl.c:3136
  #3  0x00007ffff7f8a62f in ossl_connect_common (conn=0x5555558275b0, sockindex=0, nonblocking=true, 
      done=0x555555827bea) at vtls/openssl.c:3985
  #4  0x00007ffff7f8b43b in Curl_ssl_connect_nonblocking (conn=conn@entry=0x5555558275b0, sockindex=sockindex@entry=0, 
      done=done@entry=0x555555827bea) at vtls/vtls.c:331
  #5  0x00007ffff7f552db in https_proxy_connect (sockindex=0, conn=0x5555558275b0) at http_proxy.c:57
  #6  Curl_proxy_connect (conn=0x5555558275b0, sockindex=0) at http_proxy.c:77
  #7  0x00007ffff7f4c148 in Curl_http_connect (conn=0x5555558275b0, done=0x7fffffffdc58) at http.c:1397
  #8  0x00007ffff7f5ffa6 in multi_runsingle (multi=0x555555811240, now=..., data=0x555555821c90) at multi.c:1802
  #9  0x00007ffff7f610e1 in curl_multi_perform (multi=0x555555811240, running_handles=0x7fffffffddc8) at multi.c:2420

Uh oh, it's deep within libcurl. It may be that we're passing it bogus
data, but I don't think so. The rest of our backtrace shows what Git is
doing:

  #10 0x0000555555566f41 in step_active_slots () at http.c:1458
  #11 0x0000555555566f9c in run_active_slot (slot=0x555555814370) at http.c:1479
  #12 0x000055555556762e in run_one_slot (slot=0x555555814370, results=0x7fffffffe050) at http.c:1680
  #13 0x000055555556813b in http_request (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      target=0, options=0x7fffffffe160) at http.c:1964
  #14 0x0000555555568371 in http_request_reauth (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      target=0, options=0x7fffffffe160) at http.c:2040
  #15 0x0000555555568514 in http_get_strbuf (
      url=0x555555812ae0 "https://github.com/git/git/info/refs?service=git-upload-pack", result=0x7fffffffe210, 
      options=0x7fffffffe160) at http.c:2088
  #16 0x00005555555605fb in discover_refs (service=0x5555556f73fb "git-upload-pack", for_push=0) at remote-curl.c:484
  [...]

So that's the very first http request we make of libcurl. And I don't
think we'll have regained control via any callbacks, etc. It's possible
we've screwed up passing the NO_PROXY variable somehow, but we don't
really do anything interesting with it:

  $ git grep -A2 NO_PROXY
  http.c:         var_override(&curl_no_proxy, getenv("NO_PROXY"));
  http.c-         var_override(&curl_no_proxy, getenv("no_proxy"));
  http.c-         curl_easy_setopt(result, CURLOPT_NOPROXY, curl_no_proxy);

And if I comment out that curl_easy_setopt() line, we still hit the
problem (because modern libcurl will read NO_PROXY itself).

So it seems at first glance that this is a curl bug. It's triggering via
ssl routines, so let's try plain http:

  $ export http_proxy=http://user:pass@proxy.example.com
  $ export NO_PROXY=example.com
  $ git clone https://example.com
  Cloning into 'example.com'...
  fatal: repository 'http://example.com/' not found

No segfault, and it seems to have actually contacted the server (and you
can verify that with GIT_CURL_VERBOSE=1 in the environment).

We can't do quite the same test with github.com because it will redirect
http to https. But its result is interesting, too:

  $ export NO_PROXY=github.com
  $ git clone http://github.com/git/git foo
  Cloning into 'foo'...
  warning: redirecting to https://github.com/git/git/
  remote: Enumerating objects: 292855, done.
  [...etc...]

So it _does_ work after the redirect. It only fails if the initial
request is https.

So I dunno. This seems like a libcurl bug, but it's possible we're feeding
it data wrong somehow.

-Peff

Re: Git doesn't honor NO_PROXY environment variable while cloning

[Jeff King]

On Fri, Sep 11, 2020 at 09:59:28AM -0400, Jeff King wrote:

> So I dunno. This seems like a libcurl bug, but it's possible we're feeding
> it data wrong somehow.

I forgot to mention: I'm using libcurl 7.72.0 (the current in debian
unstable). But it also reproduces on my debian stable machine with
libcurl 7.64.0.

-Peff

Re: Git doesn't honor NO_PROXY environment variable while cloning

[Daniel Stenberg]

On Fri, 11 Sep 2020, Jeff King wrote:

> So I dunno. This seems like a libcurl bug, but it's possible we're feeding 
> it data wrong somehow.

I didn't check very closely, but it certainly sounds like this one:

   https://github.com/curl/curl/pull/5902

Fixed in curl master in this commit (8 days ago):

   https://github.com/curl/curl/commit/3eff1c5092e542819ac7e6454a70c94b36ab

-- 

  / daniel.haxx.se

Re: Git doesn't honor NO_PROXY environment variable while cloning

[Jeff King]

On Fri, Sep 11, 2020 at 05:31:16PM +0200, Daniel Stenberg wrote:

> On Fri, 11 Sep 2020, Jeff King wrote:
> 
> > So I dunno. This seems like a libcurl bug, but it's possible we're
> > feeding it data wrong somehow.
> 
> I didn't check very closely, but it certainly sounds like this one:
> 
>   https://github.com/curl/curl/pull/5902

You know, I almost cc'd you, but I didn't want to bug you until I was
more sure it was a curl issue. But here you are anyway. :)

That indeed looks a lot like the same issue. And building the tip of
libcurl's master and linking git against it makes the problem go away.
So that is almost certainly it.

Thanks for the quick response!

-Peff

Re: Git doesn't honor NO_PROXY environment variable while cloning

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Fri, Sep 11, 2020 at 05:31:16PM +0200, Daniel Stenberg wrote:
>
>> On Fri, 11 Sep 2020, Jeff King wrote:
>> 
>> > So I dunno. This seems like a libcurl bug, but it's possible we're
>> > feeding it data wrong somehow.
>> 
>> I didn't check very closely, but it certainly sounds like this one:
>> 
>>   https://github.com/curl/curl/pull/5902
>
> You know, I almost cc'd you, but I didn't want to bug you until I was
> more sure it was a curl issue. But here you are anyway. :)
>
> That indeed looks a lot like the same issue. And building the tip of
> libcurl's master and linking git against it makes the problem go away.
> So that is almost certainly it.
>
> Thanks for the quick response!

Thanks, both.