Exit code 255 after large clone (32-bit PowerPC)

Exit code 255 after large clone (32-bit PowerPC)

[Evan Miller]

What did you do before the bug happened? (Steps to reproduce your issue)

$ git clone -v git@github.com:macports/macports-ports.git
Cloning into 'macports-ports'...
remote: Enumerating objects: 1223319, done.
remote: Counting objects: 100% (685/685), done.
remote: Compressing objects: 100% (341/341), done.
remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634
Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.
Connection to github.com closed by remote host.
Resolving deltas: 100% (702052/702052), done.

What did you expect to happen? (Expected behavior)

A successful clone.

What happened instead? (Actual behavior)

$ echo $?
255
$ ls -a macports-ports/
.    ..   .git

What's different between what you expected and what actually happened?

Exit value was 255 instead of 0; no regular files (only .git files) are visible in the cloned directory.

Anything else you want to add:

Other repositories have cloned just fine; however, this repo is considerably larger than the successful cases.

This is a 32-bit PowerPC machine.

[System Info]
git version:
git version 2.32.0
cpu: Power
no commit associated with this build
sizeof-long: 4
sizeof-size_t: 4
shell-path: /bin/sh
uname: Darwin 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh
compiler info: gnuc: 4.2
libc info: no libc information available

Re: Exit code 255 after large clone (32-bit PowerPC)

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On 2021-07-26 at 17:54:07, Evan Miller wrote:
> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> $ git clone -v git@github.com:macports/macports-ports.git
> Cloning into 'macports-ports'...
> remote: Enumerating objects: 1223319, done.
> remote: Counting objects: 100% (685/685), done.
> remote: Compressing objects: 100% (341/341), done.
> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634
> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.
> Connection to github.com closed by remote host.

This message is the relevant detail here.  This means that the
connection was reset, which could be due to the remote host (GitHub),
but is more likely due to a network issue of some sort.  You'll have to
do normal network troubleshooting to see why that might be.

It could very well be related to the fact that you're running a nearly
14-year old operating system, but I just can't say for certain.  It's
not a bug in Git, however.

Even if the data is otherwise transferred successfully, Git will exit
unsuccessfully if this happens, and that will result in your data not
being checked out.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: Exit code 255 after large clone (32-bit PowerPC)

[Evan Miller]

> On Jul 26, 2021, at 18:09, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> 
> On 2021-07-26 at 17:54:07, Evan Miller wrote:
>> What did you do before the bug happened? (Steps to reproduce your issue)
>> 
>> $ git clone -v git@github.com:macports/macports-ports.git
>> Cloning into 'macports-ports'...
>> remote: Enumerating objects: 1223319, done.
>> remote: Counting objects: 100% (685/685), done.
>> remote: Compressing objects: 100% (341/341), done.
>> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634
>> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.
>> Connection to github.com closed by remote host.
> 
> This message is the relevant detail here.  This means that the
> connection was reset, which could be due to the remote host (GitHub),
> but is more likely due to a network issue of some sort.  You'll have to
> do normal network troubleshooting to see why that might be.
> 
> It could very well be related to the fact that you're running a nearly
> 14-year old operating system, but I just can't say for certain.  It's
> not a bug in Git, however.
> 
> Even if the data is otherwise transferred successfully, Git will exit
> unsuccessfully if this happens, and that will result in your data not
> being checked out.

Thanks. The issue happens repeatedly on this machine, which is connected via Ethernet to residential fiber. I am doubtful of a generic network issue, but I will try an updated SSH client. (The issue does not happen over HTTPS.)

In any event, it would be good to have a more informative error message in this kind of situation. It is surprising that the data transfer is successful, and the Deltas are computed, but the overall checkout fails.

Evan


> -- 
> brian m. carlson (he/him or they/them)
> Toronto, Ontario, CA

Re: Exit code 255 after large clone (32-bit PowerPC)

[Ævar Arnfjörð Bjarmason]

On Mon, Jul 26 2021, brian m. carlson wrote:

> [[PGP Signed Part:Undecided]]
> On 2021-07-26 at 17:54:07, Evan Miller wrote:
>> What did you do before the bug happened? (Steps to reproduce your issue)
>> 
>> $ git clone -v git@github.com:macports/macports-ports.git
>> Cloning into 'macports-ports'...
>> remote: Enumerating objects: 1223319, done.
>> remote: Counting objects: 100% (685/685), done.
>> remote: Compressing objects: 100% (341/341), done.
>> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634
>> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.
>> Connection to github.com closed by remote host.
>
> This message is the relevant detail here.  This means that the
> connection was reset, which could be due to the remote host (GitHub),
> but is more likely due to a network issue of some sort.  You'll have to
> do normal network troubleshooting to see why that might be.
>
> It could very well be related to the fact that you're running a nearly
> 14-year old operating system, but I just can't say for certain.  It's
> not a bug in Git, however.

I'm not so sure it's not, I think the "Connection to github.com closed
by remote host" message is emitted by the C library, not Git itself (we
don't seem to have that exact wording anywhere, but maybe I missed
it).

I've seen other cases where I think OSX in particular is quite verbose
in this area, but maybe I'm misrecalling.

We've already received all objects as noted downthread, so having the
connection go away should be something we handle gracefully, and the
code in transport.c seems to try to do that.

It's also quite unusual for us to exit with code 255, I don't think we
do that intentionally anywhere (not from die, BUG etc.).

Evan: Can you run this with some of GIT_TRACE=1 /
GIT_TRACE_EVENT=/dev/stderr GIT_TRACE_PACKET=1 and see if some of those
show what's happening in/around that 255 exit?

Re: Exit code 255 after large clone (32-bit PowerPC)

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

On 2021-07-27 at 00:51:05, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Jul 26 2021, brian m. carlson wrote:
> 
> > [[PGP Signed Part:Undecided]]
> > On 2021-07-26 at 17:54:07, Evan Miller wrote:
> >> What did you do before the bug happened? (Steps to reproduce your issue)
> >> 
> >> $ git clone -v git@github.com:macports/macports-ports.git
> >> Cloning into 'macports-ports'...
> >> remote: Enumerating objects: 1223319, done.
> >> remote: Counting objects: 100% (685/685), done.
> >> remote: Compressing objects: 100% (341/341), done.
> >> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634
> >> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.
> >> Connection to github.com closed by remote host.
> >
> > This message is the relevant detail here.  This means that the
> > connection was reset, which could be due to the remote host (GitHub),
> > but is more likely due to a network issue of some sort.  You'll have to
> > do normal network troubleshooting to see why that might be.
> >
> > It could very well be related to the fact that you're running a nearly
> > 14-year old operating system, but I just can't say for certain.  It's
> > not a bug in Git, however.
> 
> I'm not so sure it's not, I think the "Connection to github.com closed
> by remote host" message is emitted by the C library, not Git itself (we
> don't seem to have that exact wording anywhere, but maybe I missed
> it).

That message comes from OpenSSH.  I've seen it quite frequently in
various other (non-Git) cases.  I think it's fair for us to exit
unsuccessfully if OpenSSH exits unsuccessfully in this case.  For
example, an attacker could try to tamper with the connection and send
additional data, which OpenSSH would detect and exit unsuccessfully for.
We also in general need to detect truncation attacks, which OpenSSH will
do for us here.

It's possible that if there's an older version of OpenSSH being used,
that the problem happens to be related to a bug of some sort.  There
were some versions which had various bugs that could be triggered by a
rekey, which, if the threshold is set low enough, could be the cause of
this particular problem.

I think the fact that it's not being seen with HTTPS is the ultimate
clue here.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: Exit code 255 after large clone (32-bit PowerPC)

[Evan Miller]

> On Jul 26, 2021, at 20:51, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> 
> Evan: Can you run this with some of GIT_TRACE=1 /
> GIT_TRACE_EVENT=/dev/stderr GIT_TRACE_PACKET=1 and see if some of those
> show what's happening in/around that 255 exit?

Running it again – the "Connection closed" message appears while the Deltas are being computed. Until that message appears, the ssh process is still alive.

There are no trace messages immediately surrounding the closed connection. The 255 exit does not occur until after the deltas are computed and the updates are resolved. (Which makes the error confusing from a user perspective.)

I would think that the SSH connection could/should be closed immediately after all the objects have been received. I am still compiling a more recent SSH, and will report back tomorrow if that changes anything. As of this writing:

$ ssh -v
OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006

Evan

Re: Exit code 255 after large clone (32-bit PowerPC)

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> That message comes from OpenSSH.  I've seen it quite frequently in
> various other (non-Git) cases.  I think it's fair for us to exit
> unsuccessfully if OpenSSH exits unsuccessfully in this case.  For
> example, an attacker could try to tamper with the connection and send
> additional data, which OpenSSH would detect and exit unsuccessfully for.
> We also in general need to detect truncation attacks, which OpenSSH will
> do for us here.
>
> It's possible that if there's an older version of OpenSSH being used,
> that the problem happens to be related to a bug of some sort.  There
> were some versions which had various bugs that could be triggered by a
> rekey, which, if the threshold is set low enough, could be the cause of
> this particular problem.
>
> I think the fact that it's not being seen with HTTPS is the ultimate
> clue here.

This suspiously sounds familiar.  Asking for "close ssh connection"
in the lore.kernel.org archive finds this:

  https://lore.kernel.org/git/YKTg8nYjSGpKbq8W@coredump.intra.peff.net/

Re: Exit code 255 after large clone (32-bit PowerPC)

[Evan Miller]

> On Jul 26, 2021, at 22:20, Junio C Hamano <gitster@pobox.com> wrote:
> 
> This suspiously sounds familiar.  Asking for "close ssh connection"
> in the lore.kernel.org archive finds this:
> 
>  https://lore.kernel.org/git/YKTg8nYjSGpKbq8W@coredump.intra.peff.net/
> 

The described symptoms match what I am seeing. It makes sense that ancient hardware would bring out the bug – triggering the server timeout with a long local computation.

Switching to protocol 0 successfully works around the issue. This concludes my interest in the error. I suggest reviving a discussion of Jeff King's patch here:

https://lore.kernel.org/git/YKU4mbq%2Fam%2FliAN4@coredump.intra.peff.net/

Evan

Re: Exit code 255 after large clone (32-bit PowerPC)

[Jeff King]

On Mon, Jul 26, 2021 at 11:05:27PM -0400, Evan Miller wrote:

> The described symptoms match what I am seeing. It makes sense that
> ancient hardware would bring out the bug – triggering the server
> timeout with a long local computation.
> 
> Switching to protocol 0 successfully works around the issue. This
> concludes my interest in the error. I suggest reviving a discussion of
> Jeff King's patch here:
> 
> https://lore.kernel.org/git/YKU4mbq%2Fam%2FliAN4@coredump.intra.peff.net/

Yeah, everything you've described is consistent with the problem being
fixed there. Good news, then: the patch is already in Git's "master"
branch, and will be part of the next release (which should be v2.33.0).

If you can reproduce the problem at will and building from master isn't
too hard, it would be nice to know if that indeed fixes it for you.

-Peff