From: Christian Couder <email@example.com>
Cc: Junio C Hamano <firstname.lastname@example.org>, Jeff King <email@example.com>,
Ben Peart <Ben.Peart@microsoft.com>,
Jonathan Tan <firstname.lastname@example.org>,
Jonathan Nieder <email@example.com>,
Stefan Beller <firstname.lastname@example.org>,
Nguyen Thai Ngoc Duy <email@example.com>,
Mike Hommey <firstname.lastname@example.org>,
Lars Schneider <email@example.com>,
Eric Wong <firstname.lastname@example.org>,
Christian Couder <email@example.com>,
Jeff Hostetler <firstname.lastname@example.org>,
Eric Sunshine <email@example.com>,
Beat Bolli <firstname.lastname@example.org>
Subject: [PATCH v2 12/13] partial-clone: add multiple remotes in the doc
Date: Tue, 22 Jan 2019 15:42:11 +0100 [thread overview]
Message-ID: <email@example.com> (raw)
While at it, let's remove a reference to ODB effort as the ODB
effort has been replaced by directly enhancing partial clone
and promisor remote features.
Signed-off-by: Christian Couder <firstname.lastname@example.org>
Documentation/technical/partial-clone.txt | 83 ++++++++++++++++-------
1 file changed, 58 insertions(+), 25 deletions(-)
diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt
index 896c7b3878..58adcc5ce1 100644
@@ -100,18 +100,18 @@ or commits that reference missing trees.
Handling Missing Objects
-- An object may be missing due to a partial clone or fetch, or missing due
- to repository corruption. To differentiate these cases, the local
- repository specially indicates such filtered packfiles obtained from the
- promisor remote as "promisor packfiles".
+- An object may be missing due to a partial clone or fetch, or missing
+ due to repository corruption. To differentiate these cases, the
+ local repository specially indicates such filtered packfiles
+ obtained from promisor remotes as "promisor packfiles".
These promisor packfiles consist of a "<name>.promisor" file with
arbitrary contents (like the "<name>.keep" files), in addition to
their "<name>.pack" and "<name>.idx" files.
- The local repository considers a "promisor object" to be an object that
- it knows (to the best of its ability) that the promisor remote has promised
- that it has, either because the local repository has that object in one of
+ it knows (to the best of its ability) that promisor remotes have promised
+ that they have, either because the local repository has that object in one of
its promisor packfiles, or because another promisor object refers to it.
When Git encounters a missing object, Git can see if it is a promisor object
@@ -123,7 +123,7 @@ expensive-to-modify list of missing objects.[a]
- Since almost all Git code currently expects any referenced object to be
present locally and because we do not want to force every command to do
a dry-run first, a fallback mechanism is added to allow Git to attempt
- to dynamically fetch missing objects from the promisor remote.
+ to dynamically fetch missing objects from promisor remotes.
When the normal object lookup fails to find an object, Git invokes
fetch-object to try to get the object from the server and then retry
@@ -157,8 +157,7 @@ and prefetch those objects in bulk.
We are not happy with this global variable and would like to remove it,
but that requires significant refactoring of the object code to pass an
-additional flag. We hope that concurrent efforts to add an ODB API can
Fetching Missing Objects
@@ -182,21 +181,53 @@ has been updated to not use any object flags when the corresponding argument
though they are not necessary.
+Using many promisor remotes
+Many promisor remotes can be configured and used.
+This allows for example a user to have multiple geographically-close
+cache servers for fetching missing blobs while continuing to do
+filtered `git-fetch` commands from the central server.
+When fetching objects, promisor remotes are tried one after the other
+until all the objects have been fetched.
+Remotes that are considered "promisor" remotes are those specified by
+the following configuration variables:
+- `extensions.partialClone = <name>`
+- `remote.<name>.promisor = true`
+- `remote.<name>.partialCloneFilter = ...`
+Only one promisor remote can be configured using the
+`extensions.partialClone` config variable. This promisor remote will
+be the last one tried when fetching objects.
+The other promisor remotes will be tried in the order they appear in
+the config file.
-- The remote used for a partial clone (or the first partial fetch
- following a regular clone) is marked as the "promisor remote".
+- It is not possible to specify the order in which the promisor
+ remotes are tried in other ways than the order in which they appear
+ in the config file.
-We are currently limited to a single promisor remote and only that
-remote may be used for subsequent partial fetches.
+It is also not possible to specify an order to be used when fetching
+from one remote and a different order when fetching from another
+- It is not possible to push only specific objects to a promisor
-We accept this limitation because we believe initial users of this
-feature will be using it on repositories with a strong single central
+It is not possible to push at the same time to multiple promisor
+remote in a specific order.
-- Dynamic object fetching will only ask the promisor remote for missing
- objects. We assume that the promisor remote has a complete view of the
+- Dynamic object fetching will only ask promisor remotes for missing
+ objects. We assume that promisor remotes have a complete view of the
repository and can satisfy all such requests.
- Repack essentially treats promisor and non-promisor packfiles as 2
@@ -218,15 +249,17 @@ server.
-- Allow more than one promisor remote and define a strategy for fetching
- missing objects from specific promisor remotes or of iterating over the
- set of promisor remotes until a missing object is found.
+- Improve the way to specify the order in which promisor remotes are
-A user might want to have multiple geographically-close cache servers
-for fetching missing blobs while continuing to do filtered `git-fetch`
-commands from the central server, for example.
+For example this could allow to specify explicitly something like:
+"When fetching from this remote, I want to use these promisor remotes
+in this order, though, when pushing or fetching to that remote, I want
+to use those promisor remotes in that order."
+- Allow pushing to promisor remotes.
-Or the user might want to work in a triangular work flow with multiple
+The user might want to work in a triangular work flow with multiple
promisor remotes that each have an incomplete view of the repository.
- Allow repack to work on promisor packfiles (while keeping them distinct
next prev parent reply other threads:[~2019-01-22 14:42 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-22 14:41 [PATCH v2 00/13] Many promisor remotes Christian Couder
2019-01-22 14:42 ` [PATCH v2 01/13] fetch: fix extensions.partialclone name in error message Christian Couder
2019-01-22 14:42 ` [PATCH v2 02/13] partial-clone: add missing 'is' in doc Christian Couder
2019-01-22 14:42 ` [PATCH v2 03/13] fetch-object: make functions return an error code Christian Couder
2019-01-22 14:42 ` [PATCH v2 04/13] Add initial support for many promisor remotes Christian Couder
2019-01-22 14:42 ` [PATCH v2 05/13] promisor-remote: implement promisors_get_direct() Christian Couder
2019-01-22 14:42 ` [PATCH v2 06/13] promisor-remote: add promisor_remote_reinit() Christian Couder
2019-01-22 14:42 ` [PATCH v2 07/13] promisor-remote: use repository_format_partial_clone Christian Couder
2019-01-22 14:42 ` [PATCH v2 08/13] Use promisor_remote_get_direct() and has_promisor_remote() Christian Couder
2019-01-22 14:42 ` [PATCH v2 09/13] promisor-remote: parse remote.*.partialclonefilter Christian Couder
2019-01-22 14:42 ` [PATCH v2 10/13] builtin/fetch: remove unique promisor remote limitation Christian Couder
2019-01-22 14:42 ` [PATCH v2 11/13] t0410: test fetching from many promisor remotes Christian Couder
2019-01-22 14:42 ` Christian Couder [this message]
2019-01-22 14:42 ` [PATCH v2 13/13] remote: add promisor and partial clone config to the doc Christian Couder
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).