mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Derrick Stolee <>
To: Brandon Williams <>,
Subject: Re: [RFC] protocol version 2
Date: Wed, 25 Oct 2017 09:09:12 -0400	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On 10/20/2017 1:18 PM, Brandon Williams wrote:
>   Overview
> ==========
> This document presents a specification for a version 2 of Git's wire
> protocol.  Protocol v2 will improve upon v1 in the following ways:
>    * Instead of multiple service names, multiple commands will be
>      supported by a single service
>    * Easily extendable as capabilities are moved into their own section
>      of the protocol, no longer being hidden behind a NUL byte and
>      limited by the size of a pkt-line (as there will be a single
>      capability per pkt-line).
>    * Separate out other information hidden behind NUL bytes (e.g. agent
>      string as a capability and symrefs can be requested using 'ls-ref')
>    * Ref advertisement will be omitted unless explicitly requested
>    * ls-ref command to explicitly request some refs

Hi Brandon,

I'm very interested in your protocol as a former server-side dev for the 
VSTS Git server, and understand some of these headaches. We built 
limited refs specifically to target the problem you are solving with 
ls-ref, but it requires knowledge about the authenticated user in order 
to work. I believe your suggestion is a better solution for the Git 

The "easily extendable" part has specifically caught my interest, as we 
(Microsoft) would like to move most of the GVFS protocol into core Git, 
and this is a great way to do it. Even if not all features are accepted 
by upstream, we could use our GVFS-specific fork of Git to communicate 
to our servers without breaking normal users' interactions.

Please CC me in future versions of this proposal. Let me know if you 
want to chat directly about the "TODO" items below.

Speaking of TODOs, how much of this concept do you have working in a 
prototype? Do you have code that performs this version 2 handshake and 
communicates the ls-refs result?

>   Ls-refs
> ---------
> Ls-refs can be looked at as the equivalent of the current ls-remote as
> it is a way to query a remote for the references that it has.  Unlike
> the current ls-remote, the filtering of the output is done on the server
> side by passing a number of parameters to the server-side command
> instead of the filtering occurring on the client.
> Ls-ref takes in the following parameters:
>    --head, --tags: Limit to only refs/heads or refs/tags

Nit: It would be better to use "--heads" to match refs/heads and your 
use of "--tags" for refs/tags.

>    --refs: Do not show peeled tags or pseudorefs like HEAD

Assuming we are in the case where the server has a HEAD ref, why would 
that ever be advertised? Also, does this imply that without the --refs 
option we would peel annotated tags until we find non-tag OIDs? Neither 
of these functions seem useful as default behavior.

>    --symref: In addition to the object pointed by it, show the underlying
>              ref pointed by it when showing a symbolic ref
>    <refspec>: When specified, only references matching the given patterns
>               are displayed.

Can you be specific about the patterns? For instance, it is not a good 
idea to allow the client to submit a regex for the server to compute. 
Instead, can we limit this pattern-matching to a prefix-set, such as the 
following list of prefixes:

>   Fetch
> -------
> Fetch will need to be a modified version of the v1 fetch protocol.  Some
> potential areas for improvement are: Ref-in-want, CDN offloading,
> Fetch-options.
> Since we'll have an 'ls-ref' service we can eliminate the need of fetch
> to perform a ref-advertisement, instead a client can run the 'ls-refs'
> service first, in order to find out what refs the server has, and then
> request those refs directly using the fetch service.
> //TODO Flush out the design
>   Fetch-object
> --------------
> This service could be used by partial clones in order to request missing
> objects.
> //TODO Flush out the design

As you flesh our these "fetch" and "fetch-object" commands, keep in mind 
that partial clones could mean any of the following:

  * fetch all reachable objects except for blobs.

  * fetch all reachable objects except for blobs above a
    certain size.

  * fetch all commits, trees, (and blobs?) within a certain
    "cone" of the file system.

>   Push
> ------
> Push will need to be a modified version of the v1 push protocol.  Some
> potential areas for improvement are: Fix push-options, Negotiation for
> force push.

Negotiation is something to keep in mind for all pushes, especially in 
an ecosystem full of fork-based workflows. If you are working across 
forks and someone else syncs data between your remotes, you may re-push 
a large chunk of objects that are already present in a fork. Adding an 
ls-refs step before push would be a step in the right direction.
>   Other Considerations
> ======================
>    * Move away from pkt-line framing?
>    * Have responses structured in well known formats (e.g. JSON)
>    * Eliminate initial round-trip using 'GIT_PROTOCOL' side-channel
>    * Additional commands in a partial clone world (e.g. log, grep)


I too have thought about making calls like "log" and "blame" available 
for calling remotes. One reason GVFS sends a "prefetch pack" of _all_ 
commits and trees is because one "git log -- path/to/file" command would 
start downloading thousands of objects one at a time as the walk moves 
through the history. If the remote can compute the commands that require 
historical data, then our partial clones can be more "pure" (i.e. only 
contain objects required for the user's changes).

One major caveat: if someone runs "log" from HEAD, then they may be 
working over data that is not on the remote, which means they would need 
to start the history walk until reaching commits that are known to be on 
the remote. If there are merges in the local history, then this could 
include multiple independent commits.

Further complicating this area, the server may not want to allow certain 
types of commands (i.e. regexes, expensive history options like 

In conclusion, I think it is a great idea to have the protocol allow 
these extensions, especially in a way that is easy to extend without 
breaking client/server compat scenarios (after both have v2 enabled).

[End Tangent]


  parent reply	other threads:[~2017-10-25 13:09 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-20 17:18 [RFC] protocol version 2 Brandon Williams
2017-10-24  6:48 ` Junio C Hamano
2017-10-24 18:35   ` Brandon Williams
2017-10-25  1:22     ` Junio C Hamano
2017-10-26  0:59     ` Junio C Hamano
2017-10-25 13:09 ` Derrick Stolee [this message]
2017-10-25 18:10   ` Brandon Williams
2017-10-28 22:57 ` Philip Oakley
2017-10-31 18:42   ` Brandon Williams
2017-11-10 20:13 ` Jonathan Tan
2017-12-04 23:58 ` [WIP 00/15] " Brandon Williams
2017-12-04 23:58   ` [WIP 01/15] pkt-line: introduce packet_read_with_status Brandon Williams
2017-12-07 20:53     ` Stefan Beller
2017-12-08 18:03       ` Brandon Williams
2017-12-04 23:58   ` [WIP 02/15] pkt-line: introduce struct packet_reader Brandon Williams
2017-12-07 22:01     ` Stefan Beller
2017-12-08 18:11       ` Brandon Williams
2017-12-04 23:58   ` [WIP 03/15] pkt-line: add delim packet support Brandon Williams
2017-12-07 22:30     ` Stefan Beller
2017-12-08 20:08       ` Brandon Williams
2017-12-04 23:58   ` [WIP 04/15] upload-pack: convert to a builtin Brandon Williams
2017-12-06 21:59     ` Junio C Hamano
2017-12-07 16:14       ` Johannes Schindelin
2017-12-08 20:26         ` Junio C Hamano
2017-12-08 20:12       ` Brandon Williams
2017-12-04 23:58   ` [WIP 05/15] upload-pack: factor out processing lines Brandon Williams
2017-12-04 23:58   ` [WIP 06/15] transport: use get_refs_via_connect to get refs Brandon Williams
2017-12-06 22:10     ` Junio C Hamano
2017-12-07 18:40       ` Brandon Williams
2017-12-04 23:58   ` [WIP 07/15] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
2017-12-06 22:39     ` Junio C Hamano
2017-12-08 20:19       ` Brandon Williams
2017-12-04 23:58   ` [WIP 08/15] connect: discover protocol version outside of get_remote_heads Brandon Williams
2017-12-07 18:50     ` Junio C Hamano
2017-12-07 19:04       ` Brandon Williams
2017-12-07 19:30         ` Junio C Hamano
2017-12-08 20:11           ` Brandon Williams
2017-12-04 23:58   ` [WIP 09/15] transport: store protocol version Brandon Williams
2017-12-04 23:58   ` [WIP 10/15] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
2017-12-04 23:58   ` [WIP 11/15] serve: introduce git-serve Brandon Williams
2017-12-07 23:42     ` Junio C Hamano
2017-12-08 20:25       ` Brandon Williams
2017-12-04 23:58   ` [WIP 12/15] ls-refs: introduce ls-refs server command Brandon Williams
2017-12-13 16:30     ` Philip Oakley
2017-12-04 23:58   ` [WIP 13/15] connect: request remote refs using v2 Brandon Williams
2017-12-04 23:58   ` [WIP 14/15] upload_pack: introduce fetch server command Brandon Williams
2017-12-04 23:58   ` [WIP 15/15] fetch-pack: perform a fetch using v2 Brandon Williams

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).