Is anyone working on a next-gen Git protocol?

Is anyone working on a next-gen Git protocol?

[Ævar Arnfjörð Bjarmason]

On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> I'm creating a system where a lot of remotes constantly fetch from a
>> central repository for deployment purposes, but I've noticed that even
>> with a remote.$name.fetch configuration to only get certain refs a
>> "git fetch" will still call git-upload pack which will provide a list
>> of all references.
>
> It has been observed that the sender has to advertise megabytes of
> refs because it has to speak first before knowing what the receiver
> wants, even when the receiver is interested in getting updates from
> only one of them, or worse yet, when the receiver is only trying to
> peek the ref it is interested has been updated.

Has anyone started working on a next-gen Git protocol as a result of
this discussion? If not I thought I'd give it a shot if/when I have
time.

The current protocol is basically (S = Server, C = Client)

 S: Spew out first ref
 S: Advertisement of capabilities
 S: Dump of all our refs
 C/S: Declare wanted refs, negotiate with server
 S: Send pack to client, if needed

And I thought I'd basically turn it into:

 C: Connect to server, declare what protocol we understand
 C: Advertisement of capabilities
 S: Advertisement of capabilities
 C/S: Negotiate what we want
 C/S: Same as v1, without the advertisement of capabilities, and maybe
don't dump refs at all

Basically future-proofing it by having the client say what it supports
to begin with along with what it can handle (like in HTTP).

Then in the negotiation phase the client & server would go back &
forth about what they want & how they want it. I'd planned to
implement something like:

    C: want_refs refs/heads/*
    S: OK to that
    C: want_refs refs/tags/*
    S: OK to that

Or:

    C: want_refs refs/heads/master
    S: OK to that
    C: want_refs refs/tags/v*
    S: OK to that

As a proof of concept (and also something that'll solve the issue I
had), but by adding an initial negotiation phase the protocol should
be open to any future extensions without making assumptions about the
client wanting to know about all of the server's refs, unlike the
current protocol.

Re: Is anyone working on a next-gen Git protocol?

[Ilari Liusvaara]

On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:
> 
> Has anyone started working on a next-gen Git protocol as a result of
> this discussion? If not I thought I'd give it a shot if/when I have
> time.

Unfortunately, client signaling the version is nasty to do in ways that
wouldn't cause current servers to hang up or do other undesirable things.

git://: Git-daemon will hang up[1] if it receives command it won't
understand (and one can't add arguments either).

ssh://: Commands are NAKed in non-standard ways (e.g. Gitolite vs. shell)
and one can't add arguments.

file://: That's easy.

CONNECT: The helper needs to be told that v2 is supported (helper doing
the rest).

Maybe with git://, one could hack the stuff in similar way as virtual
hosting was added. But that won't work with SSH (nor one can use environment
with SSH).

:-/

[1] And there is no guarantee that the server end of git:// is git-daemon.
There's at least one git:// server implemetation that responds to unknown
commands by ERR packet followed by hangup. 

-Ilari

Re: Is anyone working on a next-gen Git protocol?

[Jeff King]

On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:

> Has anyone started working on a next-gen Git protocol as a result of
> this discussion? If not I thought I'd give it a shot if/when I have
> time.

I haven't, and don't really plan on it soon (I have a few smaller things
I'm working on, then I'd like to look into the EWAH bitmap stuff from
Shawn next).

> The current protocol is basically (S = Server, C = Client)
> 
>  S: Spew out first ref
>  S: Advertisement of capabilities
>  S: Dump of all our refs
>  C/S: Declare wanted refs, negotiate with server
>  S: Send pack to client, if needed

In the "C" portion there, there is also "client acknowledges a
subset of capabilities shown by server" while it is declaring wanted
refs.

> And I thought I'd basically turn it into:
> 
>  C: Connect to server, declare what protocol we understand
>  C: Advertisement of capabilities
>  S: Advertisement of capabilities

The capability negotiation right now is that the server offers and the
client accepts. Are you swapping that so that the client offers and the
server accepts? Or are you thinking that they would be sent
simultaneously here? That could drop one round-trip (it's probably not
that important for git-over-tcp, but smart-http cares a lot about round
trips). But it also introduces a complexity with future additions (one
side may not know how to present its capabilities until understanding
what the other side can do).

>  C/S: Negotiate what we want

Refs we want, or capabilities we want?

>  C/S: Same as v1, without the advertisement of capabilities, and maybe
> don't dump refs at all
> 
> Basically future-proofing it by having the client say what it supports
> to begin with along with what it can handle (like in HTTP).

I feel like this "maybe..." bit needs more fleshed out before designing
the first part. I like the idea of future-proofing first and then adding
new features second, but what does the "don't advertise all refs"
protocol look like? Presumably the client is going to say "I'm
interested in refs/heads/* and refs/tags/*" or something. Does that come
with the capabilities? Or is it a new protocol phase?

I think we need to know what the second half of the two-step process
will look like to be sure the first half will accommodate it (and the
answer may be as simple as saying "they're not sending capabilities,
they're sending arbitrary key/value items, with the knowledge that the
other side may not understand particular keys, and we have to be
prepared to handle both cases).

> Then in the negotiation phase the client & server would go back &
> forth about what they want & how they want it. I'd planned to
> implement something like:
> 
>     C: want_refs refs/heads/*
>     S: OK to that
>     C: want_refs refs/tags/*
>     S: OK to that
> 
> Or:
> 
>     C: want_refs refs/heads/master
>     S: OK to that
>     C: want_refs refs/tags/v*
>     S: OK to that

That seems simple. But how will it work over smart-http? Are we adding a
round-trip to do want_refs negotiation?

-Peff

Re: Is anyone working on a next-gen Git protocol?

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>>
>>> I'm creating a system where a lot of remotes constantly fetch from a
>>> central repository for deployment purposes, but I've noticed that even
>>> with a remote.$name.fetch configuration to only get certain refs a
>>> "git fetch" will still call git-upload pack which will provide a list
>>> of all references.
>>
>> It has been observed that the sender has to advertise megabytes of
>> refs because it has to speak first before knowing what the receiver
>> wants, even when the receiver is interested in getting updates from
>> only one of them, or worse yet, when the receiver is only trying to
>> peek the ref it is interested has been updated.
>
> Has anyone started working on a next-gen Git protocol as a result of
> this discussion?

I and Shawn helped privately somebody from Gerrit circle, where the
initial ref advertisement is a huge problem (primarily because they
add tons of refs to one commit that eventually goes to their
integration branch), to coming up with a problem description and
proposal document to kick-start a discussion some time ago, but not
much has happened since.  Unless I hear from them soonish, I'll send
a cleaned-up version of the draft before I leave for my vacation.

The gist of it is that the current protocol cannot be upgraded in
place because "who speaks first" is not something you can update
with capability, so we would need upload-pack-v2 that lets the
fetching side speak first.

"What is spoken in the first message" is a separate issue, and one
of the things it can address is to allow the ends to reduce the
amount of ref advertisement that ends up not getting used in the
end, but once we allow the fetcher to speak first, we have much
wider possibilities.

Re: Is anyone working on a next-gen Git protocol?

[Junio C Hamano]

On Sun, Oct 7, 2012 at 3:31 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> I and Shawn helped privately somebody from Gerrit circle, where the
> initial ref advertisement is a huge problem (primarily because they
> add tons of refs to one commit that eventually goes to their
> integration branch), to coming up with a problem description and
> proposal document to kick-start a discussion some time ago, but not
> much has happened since.  Unless I hear from them soonish, I'll send
> a cleaned-up version of the draft before I leave for my vacation.

Which I forgot and never happened. Here is a link (not cleaned-up)

http://tinyurl.com/WhoSpeaksFirstInGit

Re: Is anyone working on a next-gen Git protocol?

[Andreas Ericsson]

On 10/07/2012 09:57 PM, Ævar Arnfjörð Bjarmason wrote:
> On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>>
>>> I'm creating a system where a lot of remotes constantly fetch from a
>>> central repository for deployment purposes, but I've noticed that even
>>> with a remote.$name.fetch configuration to only get certain refs a
>>> "git fetch" will still call git-upload pack which will provide a list
>>> of all references.
>>
>> It has been observed that the sender has to advertise megabytes of
>> refs because it has to speak first before knowing what the receiver
>> wants, even when the receiver is interested in getting updates from
>> only one of them, or worse yet, when the receiver is only trying to
>> peek the ref it is interested has been updated.
> 
> Has anyone started working on a next-gen Git protocol as a result of
> this discussion? If not I thought I'd give it a shot if/when I have
> time.
> 
> The current protocol is basically (S = Server, C = Client)
> 
>   S: Spew out first ref
>   S: Advertisement of capabilities
>   S: Dump of all our refs
>   C/S: Declare wanted refs, negotiate with server
>   S: Send pack to client, if needed
> 
> And I thought I'd basically turn it into:
> 
>   C: Connect to server, declare what protocol we understand
>   C: Advertisement of capabilities
>   S: Advertisement of capabilities
>   C/S: Negotiate what we want
>   C/S: Same as v1, without the advertisement of capabilities, and maybe
> don't dump refs at all
> 
> Basically future-proofing it by having the client say what it supports
> to begin with along with what it can handle (like in HTTP).
> 
> Then in the negotiation phase the client & server would go back &
> forth about what they want & how they want it. I'd planned to
> implement something like:
> 
>      C: want_refs refs/heads/*
>      S: OK to that
>      C: want_refs refs/tags/*
>      S: OK to that
> 
> Or:
> 
>      C: want_refs refs/heads/master
>      S: OK to that
>      C: want_refs refs/tags/v*
>      S: OK to that
> 

You'll want that to be a single "wants" message to avoid incurring
insane amounts of roundtrip latency with lots of refs. github and
other hosted services are quite popular, but with my 120ms ping
rtt I'd be spending half a minute just telling the other side what
I want when I fetch from a repo with 250 refs.

It's a flagday and a half to change the protocol though, so I expect
it'll have to wait for 2.0, unless the current client-side part of
it is dumb and ignores existing refs when requesting its "wants", in
which case the server can just stop advertising existing refs and
most of the speedup is already done.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Is anyone working on a next-gen Git protocol?

[Junio C Hamano]

Andreas Ericsson <ae@op5.se> writes:

> You'll want that to be a single "wants" message to avoid incurring
> insane amounts of roundtrip latency with lots of refs. github and
> other hosted services are quite popular, but with my 120ms ping
> rtt I'd be spending half a minute just telling the other side what
> I want when I fetch from a repo with 250 refs.

Peff's recent patch when applied on the server side would help
alleviate the load to produce these refs, but it obviously would not
cut the network cost.  In order to change this, we need to swap "who
speaks first".

Once we go into "want/have" phase, I do not think there is a need
for fundamental change in the protocol (by this, I am not counting a
change to send "have"s sparsely and possibly backtracking to bisect
history, etc. as "fundamental").

Re: Is anyone working on a next-gen Git protocol?

[Steffen Prohaska]

On Oct 8, 2012, at 6:27 PM, Junio C Hamano wrote:

> Once we go into "want/have" phase, I do not think there is a need
> for fundamental change in the protocol (by this, I am not counting a
> change to send "have"s sparsely and possibly backtracking to bisect
> history, etc. as "fundamental").

I've recently discovered that the current protocol can be amazingly
inefficient when it comes to transferring binary objects.  Assuming two
repositories that are in sync.  After a 'git checkout --orphan && git
commit', a subsequent transfers sends all the blobs attached to the new
commit, although the other side already has all the blobs.

This behavior is especially annoying when (mis)using git to store binary
files.  I was thinking for a while that it might be a reasonable idea to
store binary files in a submodule and frequently cut the history in
order to save space.  The history would have little value anyway, since
diff and merge don't make much sense with binary files.

Eventually, I abandoned the idea due to the current behavior of the
protocol.  I had expected that git would be smarter and behave more like
rsync, for example, by skipping big blobs as soon as it recognizes that
they are already available at both sides.

Maybe the new protocol could include an optimization for the described
case.  I don't know whether this would be a fundamental change.

    Steffen

Re: Is anyone working on a next-gen Git protocol?

[Junio C Hamano]

Steffen Prohaska <prohaska@zib.de> writes:

> I've recently discovered that the current protocol can be amazingly
> inefficient when it comes to transferring binary objects.  Assuming two
> repositories that are in sync.  After a 'git checkout --orphan && git
> commit', a subsequent transfers sends all the blobs attached to the new
> commit, although the other side already has all the blobs.

I do not think it has anything to do with binary, but what you
deserve from using orphan, where you declared that the history does
not have anything to do with the original.

If both of your repositories had the two paralle lines of these
histories as branches, the transfer would have went well with or
without binary objects.

Re: Is anyone working on a next-gen Git protocol?

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
> Steffen Prohaska <prohaska@zib.de> writes:
>
>> I've recently discovered that the current protocol can be amazingly
>> inefficient when it comes to transferring binary objects.  Assuming 
>> two
>> repositories that are in sync.  After a 'git checkout --orphan && git
>> commit', a subsequent transfers sends all the blobs attached to the 
>> new
>> commit, although the other side already has all the blobs.
>
> I do not think it has anything to do with binary, but what you
> deserve from using orphan, where you declared that the history does
> not have anything to do with the original.
>
> If both of your repositories had the two paralle lines of these
> histories as branches, the transfer would have went well with or
> without binary objects.
> --
Steffen,
An alternative could be a shallow clone for just those branches with the 
binary objects, so that the git objects are still identical. Or use a 
replace/graft to trim the line of development. It's still a fudge, but 
something you could look at. 

Re: Is anyone working on a next-gen Git protocol?

[Nguyen Thai Ngoc Duy]

On Thu, Oct 11, 2012 at 3:46 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Steffen Prohaska <prohaska@zib.de> writes:
>
>> I've recently discovered that the current protocol can be amazingly
>> inefficient when it comes to transferring binary objects.  Assuming two
>> repositories that are in sync.  After a 'git checkout --orphan && git
>> commit', a subsequent transfers sends all the blobs attached to the new
>> commit, although the other side already has all the blobs.
>
> I do not think it has anything to do with binary, but what you
> deserve from using orphan, where you declared that the history does
> not have anything to do with the original.
>
> If both of your repositories had the two paralle lines of these
> histories as branches, the transfer would have went well with or
> without binary objects.

On the same inefficient subject, git does not try to share common
objects for non-commit refs, for example tags pointing to trees. I
have such a peculiar repo and if a new tag shares 90% the tree with
existing tags, git-fetch to sends the whole tree of the new tag over
the wire. It does not seem easy to fix though and is probably rare
enough that does not justify proper support. As a work around, I
generate commits that link all these tags/trees together in a
predetermined order. Not nice but works ok.
-- 
Duy

Re: Is anyone working on a next-gen Git protocol?

[Shawn Pearce]

On Wed, Oct 10, 2012 at 6:44 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 3:46 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Steffen Prohaska <prohaska@zib.de> writes:
>>
>>> I've recently discovered that the current protocol can be amazingly
>>> inefficient when it comes to transferring binary objects.  Assuming two
>>> repositories that are in sync.  After a 'git checkout --orphan && git
>>> commit', a subsequent transfers sends all the blobs attached to the new
>>> commit, although the other side already has all the blobs.
>>
>> I do not think it has anything to do with binary, but what you
>> deserve from using orphan, where you declared that the history does
>> not have anything to do with the original.
>>
>> If both of your repositories had the two paralle lines of these
>> histories as branches, the transfer would have went well with or
>> without binary objects.
>
> On the same inefficient subject, git does not try to share common
> objects for non-commit refs, for example tags pointing to trees. I
> have such a peculiar repo and if a new tag shares 90% the tree with
> existing tags, git-fetch to sends the whole tree of the new tag over
> the wire. It does not seem easy to fix though and is probably rare
> enough that does not justify proper support. As a work around, I
> generate commits that link all these tags/trees together in a
> predetermined order. Not nice but works ok.

Aside from saving a huge amount of CPU during the "Counting objects"
phase, the compressed bitmap work we presented in JGit solves this by
working off the complete reachability graph, and not just some subset
related to a cut made across the commit graph. Unfortunately we took a
shortcut and didn't create bitmaps for non-commits, but this is a
trivial modification to the algorithm and the storage.