Re: [PATCH v2 1/6] docs: document bundle URI standard

[Derrick Stolee <derrickstolee@github.com>]

On 7/21/2022 5:39 PM, Josh Steadmon wrote:
> On 2022.06.29 20:40, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>

>> +bundle.heuristic::
>> +	If this string-valued key exists, then the bundle list is designed to
>> +	work well with incremental `git fetch` commands. The heuristic signals
>> +	that there are additional keys available for each bundle that help
>> +	determine which subset of bundles the client should download.
> 
> To be clear, the values of `bundle.heuristic` are keys underneath the
> following `bundle.<id>` segments?

At the moment, the only planned heuristic is associated with a single
key (bundle.<id>.creationToken) but future heuristics could be more
complicated. The general idea is to say "these bundles were organized
with this heuristic in mind, so either take all of them (ignoring the
heuristic) or apply a specific algorithm to download a subset at a
time." The creationToken heuristic uses the total order on the bundles
to do a greedy algorithm for downloading the most-recent bundles until
the required commits are all satisfied.

>> +Related Work: Packfile URIs
>> +---------------------------
> 
> Thanks for including this section; my very first question before I even
> started reading the doc was "How does this compare to Packfile URIs?".
> 
> 
>> +
>> +The Git protocol already has a capability where the Git server can list
>> +a set of URLs along with the packfile response when serving a client
>> +request. The client is then expected to download the packfiles at those
>> +locations in order to have a complete understanding of the response.
>> +
>> +This mechanism is used by the Gerrit server (implemented with JGit) and
>> +has been effective at reducing CPU load and improving user performance for
>> +clones.
>> +
>> +A major downside to this mechanism is that the origin server needs to know
>> +_exactly_ what is in those packfiles, and the packfiles need to be available
>> +to the user for some time after the server has responded. This coupling
>> +between the origin and the packfile data is difficult to manage.
> 
> It was not immediately clear to me why bundle URIs would avoid these two
> downsides, but after thinking about it and discussing it, I believe this
> is the reasoning (please correct me if I'm wrong):
> 
> Bundle URIs are intended to be fully processed before negotiation
> happens, so the server can rely solely on the client's reported Haves /
> Wants to determine its response, as usual, and therefore doesn't need to
> know the bundle contents. Packfile URIs are not processed by the client
> before negotiation, so the server needs to be aware of the packfile
> contents in order to determine which additional objects to send, and
> can't rely solely on the Haves/Wants from the client.
> 
> If that's accurate, then it seems fine that when the bundle URI is
> provided by the user on the command-line, Git can download and process
> the bundle before even attempting to contact the server. But in future
> series, when we add the ability for the server to provide a URI via a
> capability, the client will have to pause after seeing the server's
> advertised URI, fetch and process the bundle, and then proceed with a
> fetch command. This seems fine assuming the bundles can be handled in a
> reasonable amount of time. And even if the connection breaks before the
> fetch command can be issued, presumably the client would not need to
> attempt to re-download the bundle a second time when the client makes a
> second connection to re-attempt the fetch.

After the server has advertised the bundle URIs, the client should
consider disconnecting while downloading bundles, then reconnect to
the server after that is complete.

For stateless connections (https) we need to reconnect every time, so
this isn't a problem.

I anticipate that in the long-term view of this feature, the server
advertised bundle URIs will usually point to an independent bundle
list at a static URI. In that situation, we can store the chosen
bundle list URI in Git config and check that bundle list for new data
before even connecting to the origin Git server. We would only
"rediscover" the URIs when either the bundle URI returns 404 and/or
the origin server lists new URIs.

Of course, there is the option that the origin Git server wants to
track and advertise the individual bundles directly, in which case the
client can't do this opportunistic bundle pre-check.

> For the point about the packfiles needing to be available for some time,
> this makes sense because the server's response is processed by the
> client before the referenced packfile is downloaded, but the response is
> incomplete without the packfile. But with bundle URIs, the server's
> response doesn't depend on whether or not the client actually processed
> the referenced bundle, only on what was negotiated. So the server's
> response is still valid and useful even if the client is unable to get
> the bundle.

You're pointing out the critical feature: the bundles are completely
optional. If the client fails to download the bundles due to something
like this race, then it will just negotiate for the origin Git server
to provide the data.

What's perhaps not entirely fair about this point is that the bundle
provider should keep this race in mind between serving a bundle list
and having the client request the associated bundles. Something like
a 24-hour window between updating the bundle list and deleting an
unreferenced bundle file should satisfy the vast majority of clients.

Thanks for the close look. While I removed them from the context, I've
made note of your smaller comments on things like typos and will fix
them in v3.

Thanks,
-Stolee