git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
From: Stefan Beller <sbeller@google.com>
To: git@vger.kernel.org
Cc: pclouds@gmail.com, peff@peff.net, gitster@pobox.com,
	Stefan Beller <sbeller@google.com>
Subject: [RFC/WIP PATCH 00/11] Protocol version 2, again!
Date: Tue, 26 May 2015 15:01:04 -0700
Message-ID: <1432677675-5118-1-git-send-email-sbeller@google.com> (raw)

    "Just give us something to play around with" - Peff at GitMerge 2015

This is another approach for updating the pack protocol of Git.
While in the previous attempts I tried to come up with the perfect
specification of the new protocol I realized that such an approach
doesn't lead anywhere. So this time actual working code is included!

The included code is not complete, but a minimal example of

    git config remote.origin.transportversion 2
    git fetch origin master # uses the new protocol!

works.

In a previous discussion[1] towards version 2 of the pack protocol I wanted
to come up with a protocol which even includes negotiating what is done
in the protocol exchange, such as having a command "push, fetch, ls-remote"
being part of the protocol. This is not a very good approach as it's too much
work to be done at the same time.

This patch series focusses on just the fetching side, so the first four patches
are teaching git-upload-pack about the new pack protocol. The next five patches
will teach git fetch and fetch-pack how to use the version 2 protocol. Then there
will be a small test and a documentation update.

The new protocol works just like the old protocol, except for
the capabilities negotiation being before any exchange of real data.

This solves the problem of having a first huge chunk of data (the refs
advertisement) sent over the wire and just realizing in between that we
don't need these data for that operation and no way to cancel.

By having a capabilites negotiation first the new protocol may be even
more future-proof than the current one, as the capabilites will hopefully
be kept small and all larger data transfers will get their own later stage
in the protocol.

To determine the protocol version we check the ending of the
invoked program for an appended version number to see which protocol
version we're using in an exchange. At first I thought we should append
a unique suffix instead of a version number to make a distinction easier
for the kind of protocol we want to talk (There may be the traditional
protocol with no suffix, or the "upload-pack-capabilities-first" protocol
which will transmit the capabilities first).

My preference for a string suffix instead of a sequential number is
motiviated by the discussion of sha1 vs sequential numbers to describe
a state of a repository. The main difference here is however how often
we expect changes. Version 1 of the protocol is current for 10 years by
now, so I do not changes to the protocol number often, which makes it
suitable for just having a counter appended to the binary.

The advantage of just a counting number is its small size,
and I think this advantage outweights the disadvantage
of having named protocol versions.

This series doesn't include an automatic upgrade of the protocol version
for later changes if the server supports it, so for now the use of the new
protocol needs to be activated manually via setting remote.origin.transportversion.

Any comments welcome!

Thanks,
Stefan

[1] http://permalink.gmane.org/gmane.comp.version-control.git/267572

Nguyễn Thái Ngọc Duy (2):
  upload-pack: make client capability parsing code a separate function
  upload-pack: only accept capabilities on the first "want" line

Stefan Beller (9):
  upload-pack: move capabilities out of send_ref
  upload-pack-2: Implement the version 2 of upload-pack
  transport: add infrastructure to support a protocol version number
  remote.h: add get_remote_capabilities, request_capabilities
  fetch-pack: use the configured transport protocol
  transport: connect_setup appends protocol version number
  transport: get_refs_via_connect exchanges capabilities before refs.
  t5544: add a test case for the new protocol
  Document protocol version 2

 .gitignore                                        |   1 +
 Documentation/technical/pack-protocol.txt         |  86 ++++++++++++--
 Documentation/technical/protocol-capabilities.txt |  15 ---
 Makefile                                          |   2 +
 builtin/fetch-pack.c                              |  17 ++-
 builtin/fetch.c                                   |   6 +
 connect.c                                         |  48 +++++++-
 fetch-pack.h                                      |   1 +
 remote.c                                          |   2 +
 remote.h                                          |   5 +
 t/t5544-fetch-2.sh                                |  40 +++++++
 transport-helper.c                                |   1 +
 transport.c                                       |  60 +++++++++-
 transport.h                                       |   4 +
 upload-pack-2.c                                   |   1 +
 upload-pack.c                                     | 138 +++++++++++++++++-----
 16 files changed, 366 insertions(+), 61 deletions(-)
 create mode 100755 t/t5544-fetch-2.sh
 create mode 120000 upload-pack-2.c

-- 
2.4.1.345.gab207b6.dirty

             reply index

Thread overview: 61+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-05-26 22:01 Stefan Beller [this message]
2015-05-26 22:01 ` [RFC/WIP PATCH 01/11] upload-pack: make client capability parsing code a separate function Stefan Beller
2015-05-26 22:01 ` [RFC/WIP PATCH 02/11] upload-pack: only accept capabilities on the first "want" line Stefan Beller
2015-05-26 22:17   ` Junio C Hamano
2015-05-26 22:20     ` Stefan Beller
2015-05-26 22:01 ` [RFC/WIP PATCH 03/11] upload-pack: move capabilities out of send_ref Stefan Beller
2015-05-26 22:01 ` [RFC/WIP PATCH 04/11] upload-pack-2: Implement the version 2 of upload-pack Stefan Beller
2015-05-27  2:30   ` Eric Sunshine
2015-05-27  6:35   ` Jeff King
2015-05-27 17:30     ` Eric Sunshine
2015-05-27 20:14       ` Jeff King
2015-05-27 17:40     ` Stefan Beller
2015-05-27 20:34       ` Jeff King
2015-05-27 20:45         ` Stefan Beller
2015-05-27 21:46           ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 05/11] transport: add infrastructure to support a protocol version number Stefan Beller
2015-05-27  6:39   ` Jeff King
2015-05-27 19:01     ` Stefan Beller
2015-05-27 20:17       ` Jeff King
2015-05-27 19:10     ` Junio C Hamano
2015-05-26 22:01 ` [RFC/WIP PATCH 06/11] remote.h: add get_remote_capabilities, request_capabilities Stefan Beller
2015-05-27  3:25   ` Eric Sunshine
2015-05-27  6:50     ` Jeff King
2015-05-27 17:19       ` Eric Sunshine
2015-05-27 20:09         ` Jeff King
2015-05-27  6:45   ` Jeff King
2015-05-29 19:39     ` Stefan Beller
2015-05-29 22:08       ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 07/11] fetch-pack: use the configured transport protocol Stefan Beller
2015-05-26 22:19   ` Junio C Hamano
2015-05-26 22:23     ` Stefan Beller
2015-05-27  6:53   ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 08/11] transport: connect_setup appends protocol version number Stefan Beller
2015-05-26 22:21   ` Junio C Hamano
2015-05-26 22:31     ` Stefan Beller
2015-05-27  5:09       ` Junio C Hamano
2015-05-27  6:56         ` Jeff King
2015-05-27  3:33   ` Eric Sunshine
2015-05-27  7:02   ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 09/11] transport: get_refs_via_connect exchanges capabilities before refs Stefan Beller
2015-05-27  5:37   ` Eric Sunshine
2015-05-27  7:06   ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 10/11] t5544: add a test case for the new protocol Stefan Beller
2015-05-27  5:34   ` Eric Sunshine
2015-05-27  7:12   ` Jeff King
2015-05-26 22:01 ` [RFC/WIP PATCH 11/11] Document protocol version 2 Stefan Beller
2015-05-29 20:35   ` Junio C Hamano
2015-05-29 21:36     ` Stefan Beller
2015-05-29 21:52       ` Junio C Hamano
2015-05-29 22:21         ` Jeff King
2015-06-01 23:14           ` Stefan Beller
2015-06-01 23:40             ` Stefan Beller
2015-06-04 13:18               ` Jeff King
2015-06-04 17:01                 ` Junio C Hamano
2015-06-02 17:06             ` Junio C Hamano
2015-05-27  6:18 ` [RFC/WIP PATCH 00/11] Protocol version 2, again! Jeff King
2015-05-27  7:08   ` Jeff King
2015-06-01 17:49     ` Stefan Beller
2015-06-02 10:10       ` Duy Nguyen
2015-06-04 13:09       ` Jeff King
2015-06-04 16:44         ` Stefan Beller

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1432677675-5118-1-git-send-email-sbeller@google.com \
    --to=sbeller@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox