mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Brandon Williams <>
Cc: Brandon Williams <>,,,,,,,
Subject: [RFC] protocol version 2
Date: Fri, 20 Oct 2017 10:18:39 -0700	[thread overview]
Message-ID: <> (raw)


Replace Git's current wire protocol with a simpler, less wasteful
protocol that can evolve over time.


Git's wire protocol is the language used to clone/fetch/push from/to a
remote git repository.  A detailed explanation of the current protocol
spec can be found
Some of the pain points with the current protocol spec are:

  * The server's initial response is the ref advertisement.  This
    advertisement cannot be omitted and can become an issue due to the
    sheer number of refs that can be sent with large repositories.  For
    example, when contacting the internal equivalent of
    ``, the server will send
    approximately 1 million refs totaling 71MB.  This is data that is
    sent during each and every fetch and is not scalable.

  * Capabilities were implemented as a hack and are hidden behind a NUL
    byte after the first ref sent from the server during the ref

	<SHA1> <Ref Name>\0<capabilities space separated> <symref> <agent>

    Since they are sent in the context of a pkt-line they are also subject
    to the same length limitations (1k bytes with old clients).  While we
    may not be close to hitting this limitation with capabilities alone, it
    has become a problem when trying to abuse capabilities for other
    purposes (e.g. [symrefs](

  * Various other technical debt (e.g. abusing capabilities to
    communicate agent and symref data, service name set using a query


This document presents a specification for a version 2 of Git's wire
protocol.  Protocol v2 will improve upon v1 in the following ways:

  * Instead of multiple service names, multiple commands will be
    supported by a single service
  * Easily extendable as capabilities are moved into their own section
    of the protocol, no longer being hidden behind a NUL byte and
    limited by the size of a pkt-line (as there will be a single
    capability per pkt-line).
  * Separate out other information hidden behind NUL bytes (e.g. agent
    string as a capability and symrefs can be requested using 'ls-ref')
  * Ref advertisement will be omitted unless explicitly requested
  * ls-ref command to explicitly request some refs

 Detailed Design

A client can request to speak protocol v2 by sending `version=2` in the
side-channel `GIT_PROTOCOL` in the initial request to the server.

In protocol v2 communication is command oriented.  When first contacting
a server a list of capabilities will advertised.  Some of these
capabilities will be commands which a client can request be executed.
Once a command has completed, a client can reuse the connection and
request that other commands be executed

 Special Packets

In protocol v2 these special packets will have the following semantics:

  * '0000' Flush Packet (flush-pkt) - indicates the end of a message
  * '0001' End-of-List delimiter (delim-pkt) - indicates the end of a list

 Capability Advertisement

A server which decides to communicate (based on a request from a client)
using protocol version 2, notifies the client by sending a version
string in its initial response followed by an advertisement of its
capabilities.  Each capability is a key with an optional value.  Clients
must ignore all unknown keys.  Semantics of unknown values are left to
the definition of each key.  Some capabilities will describe commands
which can be requested to be executed by the client.

    capability-advertisement = protocol-version

    protocol-version = PKT-LINE("version 2" LF)
    capability-list = *capability
    capability = PKT-LINE(key[=value] LF)

    key = 1*CHAR
    value = 1*CHAR
    CHAR = 1*(ALPHA / DIGIT / "-" / "_")

A client then responds to select the command it wants with any
particular capabilities or arguments.  There is then an optional section
where the client can provide any command specific parameters or queries.

    command-request = command
                      (command specific parameters)
    command = PKT-LINE("command=" key LF)

The server will then acknowledge the command and requested capabilities
by echoing them back to the client and then launch into the command.

    acknowledge-request = command
    execute-command = <defined by each command>

A particular command can last for as many rounds as are required to
complete the service (multiple for negotiation during fetch and push or
no additional trips in the case of ls-refs).

 Commands in v2

Services are the core actions that a client wants to perform (fetch,
push, etc).  Each service has its own set of capabilities and its own
language of commands (think 'want' lines in fetch).  Optionally a
service can take in initial parameters or data when a client sends it
service request.


Ls-refs can be looked at as the equivalent of the current ls-remote as
it is a way to query a remote for the references that it has.  Unlike
the current ls-remote, the filtering of the output is done on the server
side by passing a number of parameters to the server-side command
instead of the filtering occurring on the client.

Ls-ref takes in the following parameters:

  --head, --tags: Limit to only refs/heads or refs/tags
  --refs: Do not show peeled tags or pseudorefs like HEAD
  --symref: In addition to the object pointed by it, show the underlying
            ref pointed by it when showing a symbolic ref
  <refspec>: When specified, only references matching the given patterns
             are displayed.

The output of ls-refs is as follows:

    output = (no-refs / list-of-refs)

    no-refs = PKT-LINE(zero-id SP no-refs LF)
    list-of-refs = *ref
    ref = PKT-LINE((tip / peeled) LF)
    tip = obj-id SP refname
    peeled = obj-id SP refname "^{}"

    symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
    shallow = PKT-LINE("shallow" SP obj-id LF)


Fetch will need to be a modified version of the v1 fetch protocol.  Some
potential areas for improvement are: Ref-in-want, CDN offloading,

Since we'll have an 'ls-ref' service we can eliminate the need of fetch
to perform a ref-advertisement, instead a client can run the 'ls-refs'
service first, in order to find out what refs the server has, and then
request those refs directly using the fetch service.

//TODO Flush out the design


This service could be used by partial clones in order to request missing

//TODO Flush out the design


Push will need to be a modified version of the v1 push protocol.  Some
potential areas for improvement are: Fix push-options, Negotiation for
force push.

One change that will need to happen is to improve how `push-options` are
sent to the server (so that they aren't sent twice!!).  Also the
report-status needs to be better than it currently is in v1 so that
tools like gerrit can explain what it did with the ref-update the client
sent to it. Maybe have a push-rebase capability or command?

//TODO Flush out the design

 Other Considerations

  * Move away from pkt-line framing?
  * Have responses structured in well known formats (e.g. JSON)
  * Eliminate initial round-trip using 'GIT_PROTOCOL' side-channel
  * Additional commands in a partial clone world (e.g. log, grep)

             reply	other threads:[~2017-10-20 17:19 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-20 17:18 Brandon Williams [this message]
2017-10-24  6:48 ` [RFC] protocol version 2 Junio C Hamano
2017-10-24 18:35   ` Brandon Williams
2017-10-25  1:22     ` Junio C Hamano
2017-10-26  0:59     ` Junio C Hamano
2017-10-25 13:09 ` Derrick Stolee
2017-10-25 18:10   ` Brandon Williams
2017-10-28 22:57 ` Philip Oakley
2017-10-31 18:42   ` Brandon Williams
2017-11-10 20:13 ` Jonathan Tan
2017-12-04 23:58 ` [WIP 00/15] " Brandon Williams
2017-12-04 23:58   ` [WIP 01/15] pkt-line: introduce packet_read_with_status Brandon Williams
2017-12-07 20:53     ` Stefan Beller
2017-12-08 18:03       ` Brandon Williams
2017-12-04 23:58   ` [WIP 02/15] pkt-line: introduce struct packet_reader Brandon Williams
2017-12-07 22:01     ` Stefan Beller
2017-12-08 18:11       ` Brandon Williams
2017-12-04 23:58   ` [WIP 03/15] pkt-line: add delim packet support Brandon Williams
2017-12-07 22:30     ` Stefan Beller
2017-12-08 20:08       ` Brandon Williams
2017-12-04 23:58   ` [WIP 04/15] upload-pack: convert to a builtin Brandon Williams
2017-12-06 21:59     ` Junio C Hamano
2017-12-07 16:14       ` Johannes Schindelin
2017-12-08 20:26         ` Junio C Hamano
2017-12-08 20:12       ` Brandon Williams
2017-12-04 23:58   ` [WIP 05/15] upload-pack: factor out processing lines Brandon Williams
2017-12-04 23:58   ` [WIP 06/15] transport: use get_refs_via_connect to get refs Brandon Williams
2017-12-06 22:10     ` Junio C Hamano
2017-12-07 18:40       ` Brandon Williams
2017-12-04 23:58   ` [WIP 07/15] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
2017-12-06 22:39     ` Junio C Hamano
2017-12-08 20:19       ` Brandon Williams
2017-12-04 23:58   ` [WIP 08/15] connect: discover protocol version outside of get_remote_heads Brandon Williams
2017-12-07 18:50     ` Junio C Hamano
2017-12-07 19:04       ` Brandon Williams
2017-12-07 19:30         ` Junio C Hamano
2017-12-08 20:11           ` Brandon Williams
2017-12-04 23:58   ` [WIP 09/15] transport: store protocol version Brandon Williams
2017-12-04 23:58   ` [WIP 10/15] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
2017-12-04 23:58   ` [WIP 11/15] serve: introduce git-serve Brandon Williams
2017-12-07 23:42     ` Junio C Hamano
2017-12-08 20:25       ` Brandon Williams
2017-12-04 23:58   ` [WIP 12/15] ls-refs: introduce ls-refs server command Brandon Williams
2017-12-13 16:30     ` Philip Oakley
2017-12-04 23:58   ` [WIP 13/15] connect: request remote refs using v2 Brandon Williams
2017-12-04 23:58   ` [WIP 14/15] upload_pack: introduce fetch server command Brandon Williams
2017-12-04 23:58   ` [WIP 15/15] fetch-pack: perform a fetch using v2 Brandon Williams

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).