Re: Git 2.26 fetches many times more objects than it should, wasting gigabytes

[Jonathan Nieder <jrnieder@gmail.com>]

(+cc: Jonathan Tan)
Hi,

Jeff King wrote:

> Here's a recipe based on your fetches that shows the problem.
>
>   # start with an up-to-date regular clone of linus's tree; I had one
>   # lying around from https://github.com/torvalds/linux, but the source
>   # shouldn't matter
>   rm -rf repo.git
>   git clone --bare /path/to/linux repo.git
>   cd repo.git
>
>   git remote add next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next
>   git remote add xo git@github.com:hackerspace/olpc-xo175-linux
>   git fetch --all
>
> The "next" fetch grabs about 30MB of objects. But the xo one downloads
> 1.5GB from 7.4M objects. That's using v2.26.2, so protocol 2.

Thanks!  I'll give it a try.

[...]
> There are a few data points we've been wanting to collect:
>
>  - does setting fetch.negotiationAlgorithm=skipping help? Yes, but not
>    as much as the v0 protocol does. It sends 84k objects, 33MB.

That's pretty good.  Tightening it further would require changing the
protocol to allow the client to say "please don't send me a pack; I want
to continue with negotiation".

>  - does the same fetch over v0 stateless-http have similar problems? No,
>    swapping out the second "remote add" for:
>
>      git remote add xo https://github.com/hackerspace/olpc-xo175-linux
>
>    results in the same 48k, 32MB fetch. The v0 conversation involved 10
>    POST requests. The v2 conversation only took 6 (and generates the
>    same big response as the ssh session, unsurprisingly).
>
> So it really does seem like something in v2 is not trying as hard to
> negotiate as v0 did, even when using stateless-http.

Interesting!  So it sounds like some refs that are not being fetched
are important here to the negotiation.  And the default (non-skipping)
negotiation algorithm is doing a bad job of exploring that part of
history.

Will take a closer look.

I think this still suggests that we should go ahead and switch
negotiation algorithms, both because it avoids this MAX_IN_VAIN and
because it reduces the number of rounds needed to make progress.

I'd also be tempted to get rid of MAX_IN_VAIN.  If we're at the point
of giving up, shouldn't we error out instead of having the server send
a copy of the entirety of history?

Jonathan