Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Bezdeka, Florian]

Hi all,

I just updated from git 2.30.2 (from Debian 11) to 2.39.0 (from Debian
testing) and realized that I can no longer clone repositories using the
git:// protocol.

There is one specialty in my setup: I'm located behind a proxy, so
GIT_PROXY_COMMAND is set. I'm usiung the oe-git-proxy script [1] here.
My environment provides the http_proxy variable and privoxy [2] is
running on the server side. That information should be sufficient to
reproduce.

I tried the following two repositories for testing:
 - git clone git://git.code.sf.net/p/linuxptp/code linuxptp
 - git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

The result is:
Cloning into 'linuxptp'...
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

I was able to "git bisect" it to the following commit:
ae1a7eefffe6 ("fetch-pack: signal v2 server that we are done making requests")

Reverting this commit on top of the master branch fixes my issue.
All people involved in this commit should be in CC.

Looking at the TCP byte stream shows that the socket is closed after
the client received the first "part" of the packfile.

...
0032want ec3f28a0ac13df805278164f2c72e69676d13134
0032want 57caf5d94876e8329be65d2dc29d3c528b149724
0009done
0000000dpackfile

Let me know if you need further information. Hopefully this was the
correct way of submitting a bug to git...

Best regards,
Florian

[1] https://wiki.yoctoproject.org/wiki/Working_Behind_a_Network_Proxy
[2] https://www.privoxy.org/

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]

On 2023-01-31 at 10:52:47, Bezdeka, Florian wrote:
> Hi all,

Hey,

> I just updated from git 2.30.2 (from Debian 11) to 2.39.0 (from Debian
> testing) and realized that I can no longer clone repositories using the
> git:// protocol.
> 
> There is one specialty in my setup: I'm located behind a proxy, so
> GIT_PROXY_COMMAND is set. I'm usiung the oe-git-proxy script [1] here.
> My environment provides the http_proxy variable and privoxy [2] is
> running on the server side. That information should be sufficient to
> reproduce.
> 
> I tried the following two repositories for testing:
>  - git clone git://git.code.sf.net/p/linuxptp/code linuxptp
>  - git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> 
> The result is:
> Cloning into 'linuxptp'...
> fetch-pack: unexpected disconnect while reading sideband packet
> fatal: early EOF
> fatal: fetch-pack: invalid index-pack output
> 
> I was able to "git bisect" it to the following commit:
> ae1a7eefffe6 ("fetch-pack: signal v2 server that we are done making requests")
> 
> Reverting this commit on top of the master branch fixes my issue.
> All people involved in this commit should be in CC.
> 
> Looking at the TCP byte stream shows that the socket is closed after
> the client received the first "part" of the packfile.
> 
> ...
> 0032want ec3f28a0ac13df805278164f2c72e69676d13134
> 0032want 57caf5d94876e8329be65d2dc29d3c528b149724
> 0009done
> 0000000dpackfile
> 
> Let me know if you need further information. Hopefully this was the
> correct way of submitting a bug to git...

I think this may have come up before, and I think the rule is that you
need a proxy where closing standard input doesn't close standard output.
Since that script is using socat, I believe you need the -t option to
make this work, or some other approach where standard input and standard
output can be closed independently.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Florian Bezdeka]

On Tue, 2023-01-31 at 11:02 +0000, brian m. carlson wrote:
> On 2023-01-31 at 10:52:47, Bezdeka, Florian wrote:
> > Hi all,
> 
> Hey,
> 
> > I just updated from git 2.30.2 (from Debian 11) to 2.39.0 (from Debian
> > testing) and realized that I can no longer clone repositories using the
> > git:// protocol.
> > 
> > There is one specialty in my setup: I'm located behind a proxy, so
> > GIT_PROXY_COMMAND is set. I'm usiung the oe-git-proxy script [1] here.
> > My environment provides the http_proxy variable and privoxy [2] is
> > running on the server side. That information should be sufficient to
> > reproduce.
> > 
> > I tried the following two repositories for testing:
> >  - git clone git://git.code.sf.net/p/linuxptp/code linuxptp
> >  - git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> > 
> > The result is:
> > Cloning into 'linuxptp'...
> > fetch-pack: unexpected disconnect while reading sideband packet
> > fatal: early EOF
> > fatal: fetch-pack: invalid index-pack output
> > 
> > I was able to "git bisect" it to the following commit:
> > ae1a7eefffe6 ("fetch-pack: signal v2 server that we are done making requests")
> > 
> > Reverting this commit on top of the master branch fixes my issue.
> > All people involved in this commit should be in CC.
> > 
> > Looking at the TCP byte stream shows that the socket is closed after
> > the client received the first "part" of the packfile.
> > 
> > ...
> > 0032want ec3f28a0ac13df805278164f2c72e69676d13134
> > 0032want 57caf5d94876e8329be65d2dc29d3c528b149724
> > 0009done
> > 0000000dpackfile
> > 
> > Let me know if you need further information. Hopefully this was the
> > correct way of submitting a bug to git...
> 
> I think this may have come up before, and I think the rule is that you
> need a proxy where closing standard input doesn't close standard output.
> Since that script is using socat, I believe you need the -t option to
> make this work, or some other approach where standard input and standard
> output can be closed independently.

Thanks for the super fast response, highly appreciated!

I was able to get it running by switching to ncat using the --no-
shutdown option, but I failed to bring back socat support so far.

For me this is still a regression. We have to change our
infrastructure/environment because we have a new requirement
(independent handling of stdin/out) after updating git now. I would
expect some noise from the yocto/OE community in the future where oe-
git-proxy is heavily used.

I guess proxy support was forgotten when the referenced change was
made. Any chance we can avoid closing stdout when running "in proxy
mode" to restore backward compatibility?

Thanks a lot!

Florian

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Junio C Hamano]

Florian Bezdeka <florian.bezdeka@siemens.com> writes:

> I guess proxy support was forgotten when the referenced change was
> made. Any chance we can avoid closing stdout when running "in proxy
> mode" to restore backward compatibility?

See the last paragraph of
https://lore.kernel.org/git/YS1Bni+QuZBOgkUI@coredump.intra.peff.net/

Nobody brought anything new to the table since then (the original
discussion was from Aug 2021) to change the conclusion that socat
was not doing the right thing in its default form.

Thanks.

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Jacob Keller]

On Tue, Jan 31, 2023 at 4:17 AM Florian Bezdeka
<florian.bezdeka@siemens.com> wrote:
> Thanks for the super fast response, highly appreciated!
>
> I was able to get it running by switching to ncat using the --no-
> shutdown option, but I failed to bring back socat support so far.
>
> For me this is still a regression. We have to change our
> infrastructure/environment because we have a new requirement
> (independent handling of stdin/out) after updating git now. I would
> expect some noise from the yocto/OE community in the future where oe-
> git-proxy is heavily used.
>
> I guess proxy support was forgotten when the referenced change was
> made. Any chance we can avoid closing stdout when running "in proxy
> mode" to restore backward compatibility?
>
> Thanks a lot!

I had this issue in the past and i got it working with socat using "-t
10". You need a timeout that is larger than the keep alive value of 5
seconds which is sent by the git protocol.

Junio pointed out the excellent analysis from Peff regarding the
situation and the fact that socat is wrong here.

What value of -t did you try?

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Florian Bezdeka]

On Tue, 2023-01-31 at 12:31 -0800, Jacob Keller wrote:
> On Tue, Jan 31, 2023 at 4:17 AM Florian Bezdeka
> <florian.bezdeka@siemens.com> wrote:
> > Thanks for the super fast response, highly appreciated!
> > 
> > I was able to get it running by switching to ncat using the --no-
> > shutdown option, but I failed to bring back socat support so far.
> > 
> > For me this is still a regression. We have to change our
> > infrastructure/environment because we have a new requirement
> > (independent handling of stdin/out) after updating git now. I would
> > expect some noise from the yocto/OE community in the future where oe-
> > git-proxy is heavily used.
> > 
> > I guess proxy support was forgotten when the referenced change was
> > made. Any chance we can avoid closing stdout when running "in proxy
> > mode" to restore backward compatibility?
> > 
> > Thanks a lot!
> 
> I had this issue in the past and i got it working with socat using "-t
> 10". You need a timeout that is larger than the keep alive value of 5
> seconds which is sent by the git protocol.
> 
> Junio pointed out the excellent analysis from Peff regarding the
> situation and the fact that socat is wrong here.

Thanks for pointing me to the old discussion. I was quite sure that I'm
not the first one facing this problem but couldn't find something.

It might be that socat is doing something wrong. But git is the
component that triggers the problem. Did someone talk to the socat
maintainers yet?

Peff also mentioned that the half-duplex shutdown of the socket is
inconsistent between proxy and raw TCP git://. It seems still a valid
option to skip the half-shutdown for the git:// proxy scenario.

> 
> What value of -t did you try?

I was playing with -t 10 and -t 60 so far. Both does not work for
cloning a kernel stable tree. I guess it's hard to find a value that
works under all circumstances as timings might be different depending
on server/network speed.

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Jeff King]

On Wed, Feb 01, 2023 at 12:19:55AM +0100, Florian Bezdeka wrote:

> > Junio pointed out the excellent analysis from Peff regarding the
> > situation and the fact that socat is wrong here.
> 
> Thanks for pointing me to the old discussion. I was quite sure that I'm
> not the first one facing this problem but couldn't find something.
> 
> It might be that socat is doing something wrong. But git is the
> component that triggers the problem. Did someone talk to the socat
> maintainers yet?

I'm not sure I'd say that socat is wrong. It's a generic tool, and it
doesn't know what type of protocol the two sides are expecting, or how
they'll handle half-duplex shutdowns. The default behavior is to wait
0.5 seconds to see if the other side has anything to say, which is a
reasonable compromise. It's just not enough for use a Git proxy in this
case.

The ideal, of course, would be an option to send the half-duplex
shutdown to the server and then wait for the server to hang up. But I
don't think it has such an option (you can just simulate it with a
really large "-t"). Netcat does, FWIW ("-q -1").

> Peff also mentioned that the half-duplex shutdown of the socket is
> inconsistent between proxy and raw TCP git://. It seems still a valid
> option to skip the half-shutdown for the git:// proxy scenario.

It could be done, but that would reintroduce the "oops, socat died while
we were waiting" that ae1a7eefff was solving. The original motivation
was with ssh, but the same problem exists for proxies. It _doesn't_
exist for raw TCP, because nobody notices the connection died (we just
close() it), and there's no error to propagate.

The raw TCP version does still suffer from leaving the connection open
unnecessarily, so it would benefit from getting the same treatment. I
didn't care enough to implement it (and TBH, I kind of hoped that git://
was on the decline; especially with the v2 protocol, it's pretty much
worse in every way than git-over-http).

> > What value of -t did you try?
> 
> I was playing with -t 10 and -t 60 so far. Both does not work for
> cloning a kernel stable tree. I guess it's hard to find a value that
> works under all circumstances as timings might be different depending
> on server/network speed.

Anything over "5" should be sufficient, because the other side should be
sending keep-alive packets (at the Git protocol level) every 5 seconds.
It might be worth running socat under strace to see what it's seeing and
doing.

Another workaround is to set protocol.version to "0" in your Git config.

-Peff

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Florian Bezdeka]

On Wed, 2023-02-01 at 07:28 -0500, Jeff King wrote:
> On Wed, Feb 01, 2023 at 12:19:55AM +0100, Florian Bezdeka wrote:
> 
> > > Junio pointed out the excellent analysis from Peff regarding the
> > > situation and the fact that socat is wrong here.
> > 
> > Thanks for pointing me to the old discussion. I was quite sure that I'm
> > not the first one facing this problem but couldn't find something.
> > 
> > It might be that socat is doing something wrong. But git is the
> > component that triggers the problem. Did someone talk to the socat
> > maintainers yet?
> 
> I'm not sure I'd say that socat is wrong. It's a generic tool, and it
> doesn't know what type of protocol the two sides are expecting, or how
> they'll handle half-duplex shutdowns. The default behavior is to wait
> 0.5 seconds to see if the other side has anything to say, which is a
> reasonable compromise. It's just not enough for use a Git proxy in this
> case.
> 
> The ideal, of course, would be an option to send the half-duplex
> shutdown to the server and then wait for the server to hang up. But I
> don't think it has such an option (you can just simulate it with a
> really large "-t"). Netcat does, FWIW ("-q -1").

-t doesn't help here. With massive help from the socat maintainer
(thanks Gerhard!, now in CC) I was able to get the following log out of
socat:

2023/02/01 11:06:29.960194 socat[18916] D read(0, 0x56111c858000, 8192)
2023/02/01 11:06:29.960208 socat[18916] D read -> 0

stdin had EOF. Socat half closes the socket:

2023/02/01 11:06:29.960231 socat[18916] I shutdown(6, 1)

And then, within less than 0.2s, the peer (proxy?) closes the other
channel:

2023/02/01 11:06:30.118216 socat[18916] D read(6, 0x56111c858000, 8192)
2023/02/01 11:06:30.118238 socat[18916] D read -> 0

It's quite clear now that the remote peer (proxy or server) closes the
complete connection after receiving the partial shutdown. That's
nothing that is under my control.

With privoxy and the infrastructure at work (zscaler based) there are
at least two proxy implementations showing this behavior. 

Switching to ncat --no-shutdown qualifies as workaround for now, but so
far I didn't manage to get socat back into the game. Downgrading git is
the other possibility.

> 
> > Peff also mentioned that the half-duplex shutdown of the socket is
> > inconsistent between proxy and raw TCP git://. It seems still a valid
> > option to skip the half-shutdown for the git:// proxy scenario.
> 
> It could be done, but that would reintroduce the "oops, socat died while
> we were waiting" that ae1a7eefff was solving. The original motivation
> was with ssh, but the same problem exists for proxies. It _doesn't_
> exist for raw TCP, because nobody notices the connection died (we just
> close() it), and there's no error to propagate.
> 
> The raw TCP version does still suffer from leaving the connection open
> unnecessarily, so it would benefit from getting the same treatment. I
> didn't care enough to implement it (and TBH, I kind of hoped that git://
> was on the decline; especially with the v2 protocol, it's pretty much
> worse in every way than git-over-http).
> 
> > > What value of -t did you try?
> > 
> > I was playing with -t 10 and -t 60 so far. Both does not work for
> > cloning a kernel stable tree. I guess it's hard to find a value that
> > works under all circumstances as timings might be different depending
> > on server/network speed.
> 
> Anything over "5" should be sufficient, because the other side should be
> sending keep-alive packets (at the Git protocol level) every 5 seconds.
> It might be worth running socat under strace to see what it's seeing and
> doing.
> 
> Another workaround is to set protocol.version to "0" in your Git config.
> 
> -Peff

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Bezdeka, Florian]

On Wed, 2023-02-01 at 13:53 +0100, Florian Bezdeka wrote:
> On Wed, 2023-02-01 at 07:28 -0500, Jeff King wrote:
> > On Wed, Feb 01, 2023 at 12:19:55AM +0100, Florian Bezdeka wrote:
> > 
> > > > Junio pointed out the excellent analysis from Peff regarding the
> > > > situation and the fact that socat is wrong here.
> > > 
> > > Thanks for pointing me to the old discussion. I was quite sure that I'm
> > > not the first one facing this problem but couldn't find something.
> > > 
> > > It might be that socat is doing something wrong. But git is the
> > > component that triggers the problem. Did someone talk to the socat
> > > maintainers yet?
> > 
> > I'm not sure I'd say that socat is wrong. It's a generic tool, and it
> > doesn't know what type of protocol the two sides are expecting, or how
> > they'll handle half-duplex shutdowns. The default behavior is to wait
> > 0.5 seconds to see if the other side has anything to say, which is a
> > reasonable compromise. It's just not enough for use a Git proxy in this
> > case.
> > 
> > The ideal, of course, would be an option to send the half-duplex
> > shutdown to the server and then wait for the server to hang up. But I
> > don't think it has such an option (you can just simulate it with a
> > really large "-t"). Netcat does, FWIW ("-q -1").
> 
> -t doesn't help here. With massive help from the socat maintainer
> (thanks Gerhard!, now in CC) I was able to get the following log out of
> socat:
> 
> 2023/02/01 11:06:29.960194 socat[18916] D read(0, 0x56111c858000, 8192)
> 2023/02/01 11:06:29.960208 socat[18916] D read -> 0
> 
> stdin had EOF. Socat half closes the socket:
> 
> 2023/02/01 11:06:29.960231 socat[18916] I shutdown(6, 1)
> 
> And then, within less than 0.2s, the peer (proxy?) closes the other
> channel:
> 
> 2023/02/01 11:06:30.118216 socat[18916] D read(6, 0x56111c858000, 8192)
> 2023/02/01 11:06:30.118238 socat[18916] D read -> 0
> 
> It's quite clear now that the remote peer (proxy or server) closes the
> complete connection after receiving the partial shutdown. That's
> nothing that is under my control.
> 
> With privoxy and the infrastructure at work (zscaler based) there are
> at least two proxy implementations showing this behavior. 
> 
> Switching to ncat --no-shutdown qualifies as workaround for now, but so
> far I didn't manage to get socat back into the game. Downgrading git is
> the other possibility.

With another hint from Gerhard:

Using ignoreeof works!

My proxy script does now (for one of the test scenarios):

socat STDIO,ignoreeof PROXY:<proxy-dns-name>:git.code.sf.net:9418,proxyport=9400
           ^
           This is new
> 
> > 
> > > Peff also mentioned that the half-duplex shutdown of the socket is
> > > inconsistent between proxy and raw TCP git://. It seems still a valid
> > > option to skip the half-shutdown for the git:// proxy scenario.
> > 
> > It could be done, but that would reintroduce the "oops, socat died while
> > we were waiting" that ae1a7eefff was solving. The original motivation
> > was with ssh, but the same problem exists for proxies. It _doesn't_
> > exist for raw TCP, because nobody notices the connection died (we just
> > close() it), and there's no error to propagate.
> > 
> > The raw TCP version does still suffer from leaving the connection open
> > unnecessarily, so it would benefit from getting the same treatment. I
> > didn't care enough to implement it (and TBH, I kind of hoped that git://
> > was on the decline; especially with the v2 protocol, it's pretty much
> > worse in every way than git-over-http).
> > 
> > > > What value of -t did you try?
> > > 
> > > I was playing with -t 10 and -t 60 so far. Both does not work for
> > > cloning a kernel stable tree. I guess it's hard to find a value that
> > > works under all circumstances as timings might be different depending
> > > on server/network speed.
> > 
> > Anything over "5" should be sufficient, because the other side should be
> > sending keep-alive packets (at the Git protocol level) every 5 seconds.
> > It might be worth running socat under strace to see what it's seeing and
> > doing.
> > 
> > Another workaround is to set protocol.version to "0" in your Git config.
> > 
> > -Peff
> 

Re: Bug: Cloning git repositories behind a proxy using the git:// protocol broken since 2.32

[Jeff King]

On Wed, Feb 01, 2023 at 01:53:34PM +0100, Florian Bezdeka wrote:

> > The ideal, of course, would be an option to send the half-duplex
> > shutdown to the server and then wait for the server to hang up. But I
> > don't think it has such an option (you can just simulate it with a
> > really large "-t"). Netcat does, FWIW ("-q -1").
> 
> -t doesn't help here. With massive help from the socat maintainer
> (thanks Gerhard!, now in CC) I was able to get the following log out of
> socat:
> 
> 2023/02/01 11:06:29.960194 socat[18916] D read(0, 0x56111c858000, 8192)
> 2023/02/01 11:06:29.960208 socat[18916] D read -> 0
> 
> stdin had EOF. Socat half closes the socket:
> 
> 2023/02/01 11:06:29.960231 socat[18916] I shutdown(6, 1)
> 
> And then, within less than 0.2s, the peer (proxy?) closes the other
> channel:
> 
> 2023/02/01 11:06:30.118216 socat[18916] D read(6, 0x56111c858000, 8192)
> 2023/02/01 11:06:30.118238 socat[18916] D read -> 0
> 
> It's quite clear now that the remote peer (proxy or server) closes the
> complete connection after receiving the partial shutdown. That's
> nothing that is under my control.
>
> With privoxy and the infrastructure at work (zscaler based) there are
> at least two proxy implementations showing this behavior.

OK, so the problem is that socat is not terminating at git-daemon, but
rather at some other random infrastructure that also does not handle
half-duplex shutdowns in a reasonable way. <sigh>

So if we took socat out of the loop entirely, and if raw git-over-tcp
did the half-duplex shutdown for the v2 protocol (which it really ought
to be doing), then those proxies would presumably similarly break.

I dunno. I am sympathetic that this thing used to work, and now doesn't.
But really the issue is in the v2 protocol, which has no way to signal
"I'm done" short of closing the socket. Combining that with raw
git-over-tcp and over-zealous proxies is going to be a problem.

> Switching to ncat --no-shutdown qualifies as workaround for now, but so
> far I didn't manage to get socat back into the game. Downgrading git is
> the other possibility.

Did you try setting protocol.version to 0? I expect that would also
work.

IMHO the best option, if possible, is to use git-over-http. It's more
secure and generally more tested than git-over-tcp.

-Peff