list mirror (unofficial, one of many)
 help / color / Atom feed
From: Jonathan Tan <>
To: Junio C Hamano <>
Subject: Re: [RFC 02/14] upload-pack: allow ref name and glob requests
Date: Thu, 26 Jan 2017 16:35:31 -0800
Message-ID: <> (raw)
In-Reply-To: <>

On 01/26/2017 02:23 PM, Junio C Hamano wrote:
> Jonathan Tan <> writes:
>> Currently, while performing packfile negotiation [1], upload-pack allows
>> clients to specify their desired objects only as SHA-1s. This causes:
>> (a) vulnerability to failure when an object turns non-existent during
>>     negotiation, which may happen if, for example, upload-pack is
>>     provided by multiple Git servers in a load-balancing arrangement,
>>     and
>> (b) dependence on the server first publishing a list of refs with
>>     associated objects.
>> To eliminate (a) and take a step towards eliminating (b), teach
>> upload-pack to support requests in the form of ref names and globs (in
>> addition to the existing support for SHA-1s) through a new line of the
>> form "want-ref <ref>" where ref is the full name of a ref, a glob
>> pattern, or a SHA-1. At the conclusion of negotiation, the server will
>> write "wanted-ref <SHA-1> <name>" for all requests that have been
>> specified this way.
> I am not sure if this "at the conclusion of" is sensible.  It is OK
> to assume that what the client side has is fixed, and it is probably
> OK to desire that what the server side has can change, but at the
> same time, it feels quite fragile to move the goalpost in between.

Do you have any specific concerns as to this fragility? Peff mentioned 
some concerns with the client making some decisions based on the initial 
SHA-1 vs the SHA-1 reported by "wanted-ref", to which I replied [1].

> Stepping back a bit, in an environment that involves multiple server
> instances that have inconsistent set of refs, can the negotiation
> even be sensibly and safely implemented?  The first server the
> client contacts may, in response to a "have", say "I do have that
> commit so you do not have to send its ancestors to me.  We found one
> cut-off point.  Please do explore other lines of histories."  The
> next server that concludes the negotiation exchange may not have
> that commit and will be unable to produce a pack that excludes the
> objects reachable from that commit---wouldn't that become a problem?

It's true that this patch set wouldn't solve this problem. This problem 
only occurs when there is a commit that the client knows but only a few 
of the servers know (maybe because the client just pushed it to one of 
them). If, for example, the client does not know a commit and only a few 
of the servers know it (for example, because another user just pushed 
it), this patch set does help. The latter scenario seems like it would 
occur relatively commonly.

> One way to prevent such a problem from hurting clients may be for
> these multiple server instances to coordinate and make sure they
> have a shared perception of the common history among them.  Some
> pushes may have come to one instance but may not have propagated to
> other instances, and such a commit cannot be accepted as usable
> "have" if the servers anticipate that the final client request would
> go to any of the servers.  Otherwise the multiple server arrangement
> would not work safely, methinks.
> And if the servers are ensuring the safety using such a mechanism,
> they can use the same mechanism to restrain "faster" instances from
> sending too fresh state of refs that other instances haven't caught
> up to, which would mean they can present a consistent set of refs to
> the client in the first place, no?
> So I am not sure if the mechanism to request history by refname
> instead of the tip commit would help the multi-server environment as
> advertised.  It may help solving other problems, though (e.g. like
> "somebody pushed to update after the initial advertisement was sent
> out" which can happen even in a single server environment).

This patch set would solve the problem you describe (whether in a single 
server environment or the coordination between multiple servers that 
provides "strong consistency"). (Although it may not be an important 
problem to solve, since it is probably OK if the client got a "slow" 
version of the state of the refs.)

>> To be flexible with respect to client needs, the server does not
>> indicate an error if a "want-ref" line corresponds to no refs, but
>> instead relies on the client to ensure that what the user needs has been
>> fetched. For example, a client could reasonably expand an abbreviated
>> name "foo" to "want-ref foo", "want-ref refs/heads/foo", "want-ref
>> refs/tags/foo", among others, and ensure that at least one such ref has
>> been fetched.
> Cute.  This may be one way to implement the DWIM thing within the
> constraint of eventually wanting to go to "client speaks first, the
> server does not advertise things the client is not interested in"
> world.
> But at the same time it may end up bloating the set of refs the
> client asks instead.  Instead of receiving the advertisement and
> then sending one request after picking the matching one from it,
> the client needs to send "refs/{heads,tags,whatever}/foo".

That is true, although I think that the client will typically send only 
a few ref names (with or without globs), so the request packet is still 
not that large.

[1] <>

  reply index

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-01-25 22:02 [RFC 00/14] Allow fetch-pack to send ref names (globs allowed) Jonathan Tan
2017-01-25 22:02 ` [RFC 01/14] upload-pack: move parsing of "want" line Jonathan Tan
2017-01-25 22:02 ` [RFC 02/14] upload-pack: allow ref name and glob requests Jonathan Tan
2017-01-26 22:23   ` Junio C Hamano
2017-01-27  0:35     ` Jonathan Tan [this message]
2017-01-27  1:54       ` Junio C Hamano
2017-01-25 22:02 ` [RFC 03/14] upload-pack: test negotiation with changing repo Jonathan Tan
2017-01-26 22:33   ` Junio C Hamano
2017-01-27  0:44     ` Jonathan Tan
2017-02-22 23:36       ` Junio C Hamano
2017-02-23 18:43         ` [PATCH] upload-pack: report "not our ref" to client Jonathan Tan
2017-02-23 20:14           ` Junio C Hamano
2017-01-25 22:02 ` [RFC 04/14] fetch: refactor the population of hashes Jonathan Tan
2017-01-25 22:02 ` [RFC 05/14] fetch: refactor fetch_refs into two functions Jonathan Tan
2017-01-25 22:02 ` [RFC 06/14] fetch: refactor to make function args narrower Jonathan Tan
2017-01-25 22:03 ` [RFC 07/14] fetch-pack: put shallow info in out param Jonathan Tan
2017-01-25 22:03 ` [RFC 08/14] fetch-pack: check returned refs for matches Jonathan Tan
2017-01-25 22:03 ` [RFC 09/14] transport: put ref oid in out param Jonathan Tan
2017-01-25 22:03 ` [RFC 10/14] fetch-pack: support partial names and globs Jonathan Tan
2017-01-25 22:03 ` [RFC 11/14] fetch-pack: support want-ref Jonathan Tan
2017-01-25 22:03 ` [RFC 12/14] fetch-pack: do not printf after closing stdout Jonathan Tan
2017-01-26  0:50   ` Stefan Beller
2017-01-26 18:18     ` Jonathan Tan
2017-01-25 22:03 ` [RFC 13/14] fetch: send want-ref and receive fetched refs Jonathan Tan
2017-01-25 22:03 ` [RFC 14/14] DONT USE advertise_ref_in_want=1 Jonathan Tan
2017-01-26 22:15 ` [RFC 00/14] Allow fetch-pack to send ref names (globs allowed) Stefan Beller
2017-01-26 23:00 ` Jeff King
2017-01-27  0:26   ` Jonathan Tan
2017-02-07 23:53 ` Jonathan Tan
2017-02-09  0:26   ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:

 note: .onion URLs require Tor:

AGPL code for this site: git clone