Re: [PATCH 0/3] Filter alternate references

[Stefan Beller <sbeller@google.com>]

On Thu, Sep 20, 2018 at 11:04 AM Taylor Blau <ttaylorr@github.com> wrote:
>
> Hi,
>
> This is a series to customize Git's behavior when listing references
> from an alternate repository. It is motivated by the following example:
>
> Consider an upstream repository, a fork of it, and a local copy of that
> fork. Ideally, running "git pull upstream" from the local copy followed
> by a "git push fork" should be a lightweight operation, ideally because
> the fork already "knows" about the new objects introduced upstream.
>
> Today, we do this by means of the special ".have" references advertised
> by 'git receive-pack'. This special part of the advertisement is
> designed to tell the pusher about tips that it might want to know about,
> to avoid sending them again.
>
> This optimization is a good one and works well, particularly when the
> upstream repository has a relatively normal number of references. When
> the upstream has a pathologically _large_ number of references, the
> advertisement alone can be so time consuming, that it's faster to send
> redundant objects to the fork.

(tangent:)
The current fetch protocol consists of 2 parts:
negotiation + sending the packfile, and the negotiation only tries
to trim down the size of the packfile to send, without taking its own
cost (in terms of time and band width) into account, just to produce
a perfect pack to send to the client.

When talking about designing protocol v2 for push (which has not
landed yet[1]), we had some in-office discussions whether we
want to have a proper negotiation on push, as it would help
pushing to remotes that have non-ff pushes, but not necessarily
regular pushes, as they should be fine with just the refs advertisement.

[1] https://github.com/bmwill/git/commit/57a4e6e5d18a2d4d806fc8dec644b89affd50853
bmwill@ no longer works on it though.


>
> To make the reference advertisement manageable even with a large number
> of references, let's allow the fork to select which ones it thinks might
> be "interesting", and only advertise those. This makes the advertisement
> much smaller, and lets us take advantage of the ".have" references, even
> when the upstream contains more references than we're advertising.
>
> This series implements the above functionality by means of
> "core.alternateRefsCommand", and "core.alternateRefsPrefixes", either a
> command to run in place of "git for-each-ref", or arguments to be
> appended to "git for-each-ref".
>
> The order of precedence when listing references from an alternate is as
> follows:
>
>   1. If the fork configures "core.alternateRefsCommand", run that.
>
>   2. If the fork configures "core.alternateRefsPrefixes", run 'git
>      for-each-ref', limiting results to references that have any of the
>      given values as a prefix.
>
>   3. Otherwise, run 'git for-each-ref' in the alternate.
>
> In a previous version of this series, I taught the configuration
> property to the alternate, as in "these are the references that _I_
> think _you_ will find interesting," rather than the other way around. I
> ultimately decided on what is attached here so that the fork does not
> have to trust the upstream to run arbitrary shell commands.

Would it make sense to estimate the value of each .have before
advertising them and then advertise only the <n> most valuable
.haves ?
(e.g. if a .have is only one small commit ahead of origin/master,
it may not bring a lot of value as the potential savings are small,
but if that .have contains history between master..TIP that has lots
of big blobs or objects in general, this may be valuable to know)

Stefan