Suspicious fetch-pack behaviour

Suspicious fetch-pack behaviour

[Guilhem Bonnefille]

Hi,

One of my users reported a strange problem: a simple HTTPS clone did
not work with Git 1.8.3.1 on RedHat 7.
I did many tests and I was not able to understand why his clone don't
work while I'm able to do it on other similar host.

Nevertheless, we did more investigations. One of them: a raw strace.
I discovered two strange behaviours:
- fetch-pack closes its standard input and standard output and then
tries to print the references on standard input and finaly dies.
- git-remote-https does not react to fetch-pack death and continue
polling an empty set of FD.

Reading fetch-pack code, the behaviour is explicit:
When "--stateless-rpc" is provided, fd is filled with standard input
and standard ouput which are then closed.
https://git.kernel.org/pub/scm/git/git.git/tree/builtin/fetch-pack.c?h=v1.8.3.1#n156

Reading this, I did not understand why it could work.
Any help appreciated.

Here is the strace's extract:

2801  getdents(3, /* 2 entries */, 32768) = 48
2801  getdents(3, /* 0 entries */, 32768) = 0
2801  close(3)                          = 0
2801  close(0)                          = 0
2801  close(1)                          = 0
2801  fstat(1, 0x7ffe10ab4730)          = -1 EBADF (Bad file descriptor)
2801  mmap(NULL, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f220c9e3000
2801  fstat(1, 0x7ffe10ab5040)          = -1 EBADF (Bad file descriptor)
2801  write(1, "4df1dbf3224064cc5dd2e4c095da2dda"..., 159) = -1 EBADF
(Bad file descriptor)
2801  exit_group(0)                     = ?
2801  +++ exited with 0 +++
2769  <... poll resumed> )              = ? ERESTART_RESTARTBLOCK
(Interrupted by signal)
2769  --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2801,
si_uid=7561, si_status=0, si_utime=0, si_stime=0} ---
2769  restart_syscall(<... resuming interrupted poll ...>) = 0
2769  poll(NULL, 0, 1000)               = 0 (Timeout)
2769  poll(NULL, 0, 1000)               = 0 (Timeout)
2769  poll(NULL, 0, 1000)               = 0 (Timeout)
2769  poll(NULL, 0, 1000)               = 0 (Timeout)
2769  poll(NULL, 0, 1000)               = 0 (Timeout)
2769  poll(NULL, 0, 1000)               = 0 (Timeout)

-- 
Guilhem BONNEFILLE
-=- JID: guyou@im.apinc.org MSN: guilhem_bonnefille@hotmail.com
-=- mailto:guilhem.bonnefille@gmail.com
-=- http://nathguil.free.fr/

Re: Suspicious fetch-pack behaviour

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

On Thu, Jan 03, 2019 at 10:52:48AM +0100, Guilhem Bonnefille wrote:
> Hi,
> 
> One of my users reported a strange problem: a simple HTTPS clone did
> not work with Git 1.8.3.1 on RedHat 7.
> I did many tests and I was not able to understand why his clone don't
> work while I'm able to do it on other similar host.
> 
> Nevertheless, we did more investigations. One of them: a raw strace.
> I discovered two strange behaviours:
> - fetch-pack closes its standard input and standard output and then
> tries to print the references on standard input and finaly dies.
> - git-remote-https does not react to fetch-pack death and continue
> polling an empty set of FD.
> 
> Reading fetch-pack code, the behaviour is explicit:
> When "--stateless-rpc" is provided, fd is filled with standard input
> and standard ouput which are then closed.
> https://git.kernel.org/pub/scm/git/git.git/tree/builtin/fetch-pack.c?h=v1.8.3.1#n156
> 
> Reading this, I did not understand why it could work.
> Any help appreciated.

When --stateless-rpc is passed, git fetch-pack usually has its standard
input and output wired up to the ends of a socket. Those file
descriptors are then passed to do_fetch_pack, which calls get_common to
negotiate refs with the remote side and get_pack to get the resulting
pack data. The negotiation should function regardless of the final ref
printing.

It's true that attempting to write to the standard output fails in that
case, but that's okay, since we wouldn't have wanted to write that data
to the socket anyway.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Suspicious fetch-pack behaviour

[Jeff King]

On Thu, Jan 03, 2019 at 10:52:48AM +0100, Guilhem Bonnefille wrote:

> One of my users reported a strange problem: a simple HTTPS clone did
> not work with Git 1.8.3.1 on RedHat 7.
> I did many tests and I was not able to understand why his clone don't
> work while I'm able to do it on other similar host.
> 
> Nevertheless, we did more investigations. One of them: a raw strace.
> I discovered two strange behaviours:
> - fetch-pack closes its standard input and standard output and then
> tries to print the references on standard input and finaly dies.
> - git-remote-https does not react to fetch-pack death and continue
> polling an empty set of FD.
>
> [...]
>
> 2769  poll(NULL, 0, 1000)               = 0 (Timeout)

We actually don't use poll() very much in Git. And poking around the
v1.8.3.1 source, I do not see any places where remote-https would call
poll(), and none outside of "git help" and "git credential-cache-daemon"
that would ever provide a timeout like "1000".

I wonder if this poll is actually being run by libcurl.  Is it possible
to get a backtrace of the looping process with gdb?

I'd also point out that v1.8.3.1 is over 5 years old, and there have
been quite a few http-related fixes over the years. There is a good
chance that if this is a Git bug, it has long since been fixed. Is it
possible to reproduce with a more modern version of Git?

-Peff

Re: Suspicious fetch-pack behaviour

[Guilhem Bonnefille]

Le mar. 8 janv. 2019 à 07:34, Jeff King <peff@peff.net> a écrit :
>
> On Thu, Jan 03, 2019 at 10:52:48AM +0100, Guilhem Bonnefille wrote:
>
> > One of my users reported a strange problem: a simple HTTPS clone did
> > not work with Git 1.8.3.1 on RedHat 7.
> > I did many tests and I was not able to understand why his clone don't
> > work while I'm able to do it on other similar host.
> >
> > Nevertheless, we did more investigations. One of them: a raw strace.
> > I discovered two strange behaviours:
> > - fetch-pack closes its standard input and standard output and then
> > tries to print the references on standard input and finaly dies.
> > - git-remote-https does not react to fetch-pack death and continue
> > polling an empty set of FD.
> >
> > [...]
> >
> > 2769  poll(NULL, 0, 1000)               = 0 (Timeout)
>
> We actually don't use poll() very much in Git. And poking around the
> v1.8.3.1 source, I do not see any places where remote-https would call
> poll(), and none outside of "git help" and "git credential-cache-daemon"
> that would ever provide a timeout like "1000".
>
> I wonder if this poll is actually being run by libcurl.  Is it possible
> to get a backtrace of the looping process with gdb?

Quite hard to debug as it occured on a computer provided by a
customer, far from me...

> I'd also point out that v1.8.3.1 is over 5 years old, and there have
> been quite a few http-related fixes over the years.

Yes, it is an old version, but it is the version provided with RedHat 7.

> There is a good
> chance that if this is a Git bug, it has long since been fixed. Is it
> possible to reproduce with a more modern version of Git?
>

What is surprisingly is that I was unable to reproduce with the same
version on an other computer.

During this time, my user discovered the IT team of the customer
prvide a much more recent version of Git (2.X). With this new version,
the problem was not reproduced.


Thanks for all your investigations.


-- 
Guilhem BONNEFILLE
-=- JID: guyou@im.apinc.org MSN: guilhem_bonnefille@hotmail.com
-=- mailto:guilhem.bonnefille@gmail.com
-=- http://nathguil.free.fr/