git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
blob 6ed850930fea75b0fd40e856ee2ae60ac48bede6 3601 bytes (raw)
name: Documentation/technical/packfile-uri.txt 	 # note: path name is non-authoritative(*)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 
Packfile URIs
=============

This feature allows servers to serve part of their packfile response as URIs.
This allows server designs that improve scalability in bandwidth and CPU usage
(for example, by serving some data through a CDN), and (in the future) provides
some measure of resumability to clients.

This feature is available only in protocol version 2.

Protocol
--------

The server advertises the `packfile-uris` capability.

If the client then communicates which protocols (HTTPS, etc.) it supports with
a `packfile-uris` argument, the server MAY send a `packfile-uris` section
directly before the `packfile` section (right after `wanted-refs` if it is
sent) containing URIs of any of the given protocols. The URIs point to
packfiles that use only features that the client has declared that it supports
(e.g. ofs-delta and thin-pack). See protocol-v2.txt for the documentation of
this section.

Clients should then download and index all the given URIs (in addition to
downloading and indexing the packfile given in the `packfile` section of the
response) before performing the connectivity check.

Server design
-------------

The server can be trivially made compatible with the proposed protocol by
having it advertise `packfile-uris`, tolerating the client sending
`packfile-uris`, and never sending any `packfile-uris` section. But we should
include some sort of non-trivial implementation in the Minimum Viable Product,
at least so that we can test the client.

This is the implementation: a feature, marked experimental, that allows the
server to be configured by one or more entries with the format:

    uploadpack.excludeobject=<object-hash> <recursively> <pack-hash> <uri>

Value <object-hash> is the key of entry, and the object type can be a blob
or commit. Value <recursively> works for commit object, if <recursively>
is configured as '1', then the commit and all the referenced objects by
the commit will be recursively excluded. Otherwise, only the commit itself
will be excluded. Whenever the list of objects to be sent is assembled, all
such objects are excluded, replaced with URIs.

Client design
-------------

The client has a config variable `fetch.uriprotocols` that determines which
protocols the end user is willing to use. By default, this is empty.

When the client downloads the given URIs, it should store them with "keep"
files, just like it does with the packfile in the `packfile` section. These
additional "keep" files can only be removed after the refs have been updated -
just like the "keep" file for the packfile in the `packfile` section.

The division of work (initial fetch + additional URIs) introduces convenient
points for resumption of an interrupted clone - such resumption can be done
after the Minimum Viable Product (see "Future work").

Future work
-----------

The protocol design allows some evolution of the server and client without any
need for protocol changes, so only a small-scoped design is included here to
form the MVP. For example, the following can be done:

 * On the client, resumption of clone. If a clone is interrupted, information
   could be recorded in the repository's config and a "clone-resume" command
   can resume the clone in progress. (Resumption of subsequent fetches is more
   difficult because that must deal with the user wanting to use the repository
   even after the fetch was interrupted.)

There are some possible features that will require a change in protocol:

 * Additional HTTP headers (e.g. authentication)
 * Byte range support
 * Different file formats referenced by URIs (e.g. raw object)

debug log:

solving 6ed850930f ...
found 6ed850930f in https://public-inbox.org/git/4abab98a76966a34a166fc758b2e9b7cca38c156.1621327467.git.dyroneteng@gmail.com/
found f7eabc6c76 in https://80x24.org/mirrors/git.git
preparing index
index prepared:
100644 f7eabc6c76838d6577d14058c3ad43b69f3cf4f4	Documentation/technical/packfile-uri.txt

applying [1/1] https://public-inbox.org/git/4abab98a76966a34a166fc758b2e9b7cca38c156.1621327467.git.dyroneteng@gmail.com/
diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt
index f7eabc6c76..6ed850930f 100644

Checking patch Documentation/technical/packfile-uri.txt...
Applied patch Documentation/technical/packfile-uri.txt cleanly.

index at:
100644 6ed850930fea75b0fd40e856ee2ae60ac48bede6	Documentation/technical/packfile-uri.txt

(*) Git path names are given by the tree(s) the blob belongs to.
    Blobs themselves have no identifier aside from the hash of its contents.^

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).