Re: [PATCH v3 2/2] bundle-uri: add example bundle organization

[Derrick Stolee <derrickstolee@github.com>]

On 8/4/2022 12:09 PM, Matthew John Cheetham wrote:
> On 2022-07-25 14:53, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> The previous change introduced the bundle URI design document. It
>> creates a flexible set of options that allow bundle providers many ways
>> to organize Git object data and speed up clones and fetches. It is
>> particularly important that we have flexibility so we can apply future
>> advancements as new ideas for efficiently organizing Git data are
>> discovered.
>>
>> However, the design document does not provide even an example of how
>> bundles could be organized, and that makes it difficult to envision how
>> the feature should work at the end of the implementation plan.
>>
>> Add a section that details how a bundle provider could work, including
>> using the Git server advertisement for multiple geo-distributed servers.
>> This organization is based on the GVFS Cache Servers which have
>> successfully used similar ideas to provide fast object access and
>> reduced server load for very large repositories.
> Thanks! This patch is helpful guidance for bundle server implementors.
>> +This example organization is a simplified model of what is used by the
>> +GVFS Cache Servers (see section near the end of this document) which have
>> +been beneficial in speeding up clones and fetches for very large
>> +repositories, although using extra software outside of Git.
> 
> Nit: might be a good idea to use "VFS for Git" rather than the old name
> "GVFS" [1].

The rename from "GVFS" to "VFS for Git" is made even more confusing
because "the GVFS Protocol" keeps the name since it is independent of
the virtual filesystem part (and has "gvfs" in the API routes). In
particular, "the GVFS Cache Servers" provide a repository mirror using
the GVFS protocol and can be used by things like Scalar (when using
the microsoft/git fork).
 
>> +The bundle provider deploys servers across multiple geographies. Each
>> +server manages its own bundle set. The server can track a number of Git
>> +repositories, but provides a bundle list for each based on a pattern. For
>> +example, when mirroring a repository at `https://<domain>/<org>/<repo>`
>> +the bundle server could have its bundle list available at
>> +`https://<server-url>/<domain>/<org>/<repo>`. The origin Git server can
>> +list all of these servers under the "any" mode:
>> +
>> +	[bundle]
>> +		version = 1
>> +		mode = any
>> +		
>> +	[bundle "eastus"]
>> +		uri = https://eastus.example.com/<domain>/<org>/<repo>
>> +		
>> +	[bundle "europe"]
>> +		uri = https://europe.example.com/<domain>/<org>/<repo>
>> +		
>> +	[bundle "apac"]
>> +		uri = https://apac.example.com/<domain>/<org>/<repo>
>> +
>> +This "list of lists" is static and only changes if a bundle server is
>> +added or removed.
>> +
>> +Each bundle server manages its own set of bundles. The initial bundle list
>> +contains only a single bundle, containing all of the objects received from
>> +cloning the repository from the origin server. The list uses the
>> +`creationToken` heuristic and a `creationToken` is made for the bundle
>> +based on the server's timestamp.
> 
> Just to confirm, in this example the origin server advertises a single
> URL (over v2 protocol) that points to this example "list of lists"?

No, here the origin server provides the list of lists using the 'bundle-uri'
protocol v2 command. Using the config file format was an unfortunate choice
on my part because that actually uses "key=value" lines.

This could be more clear by using that format:

  bundle.version=1
  bundle.mode=any
  bundle.eastus.uri=https://eastus.example.com/<domain>/<org>/<repo>
  bundle.europe.uri=https://europe.example.com/<domain>/<org>/<repo>
  bundle.apac.uri=https://apac.example.com/<domain>/<org>/<repo>

> Remote -> 1 URL -> List(any/split by geo) -> List(all/split by time)

Adding the final link to each individual bundle, it would actually be:

  Remote (any/split by geo) -> List(all/split by time) -> Bundle

I can definitely update this part of the doc to be more clear.

Thanks,
-Stolee