From: git@jeffhostetler.com
To: git@vger.kernel.org
Cc: jeffhost@microsoft.com, peff@peff.net, gitster@pobox.com,
markbt@efaref.net, benpeart@microsoft.com,
jonathantanmy@google.com, Jeff Hostetler <git@jeffhostetler.com>
Subject: [PATCH 04/10] upload-pack: add partial (sparse) fetch
Date: Wed, 8 Mar 2017 18:50:33 +0000 [thread overview]
Message-ID: <1488999039-37631-5-git-send-email-git@jeffhostetler.com> (raw)
In-Reply-To: <1488999039-37631-1-git-send-email-git@jeffhostetler.com>
From: Jeff Hostetler <git@jeffhostetler.com>
Teach upload-pack to advertise the "partial" capability
in the fetch-pack/upload-pack protocol header and to pass
the value of partial-by-size and partial-special on to
pack-objects.
Update protocol documentation.
This might be used in conjunction with a partial (sparse) clone
or fetch to omit various blobs from the generated packfile.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
Documentation/technical/pack-protocol.txt | 14 ++++++++++
Documentation/technical/protocol-capabilities.txt | 7 +++++
upload-pack.c | 32 ++++++++++++++++++++++-
3 files changed, 52 insertions(+), 1 deletion(-)
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index c59ac99..0032729 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.
upload-request = want-list
*shallow-line
*1depth-request
+ *partial
flush-pkt
want-list = first-want
@@ -223,10 +224,15 @@ out of what the server said it could do with the first 'want' line.
PKT-LINE("deepen-since" SP timestamp) /
PKT-LINE("deepen-not" SP ref)
+ partial = PKT-LINE("partial-by-size" SP magnitude) /
+ PKT-LINE("partial-special)
+
first-want = PKT-LINE("want" SP obj-id SP capability-list)
additional-want = PKT-LINE("want" SP obj-id)
depth = 1*DIGIT
+
+ magnitude = 1*DIGIT [ "k" | "m" | "g" ]
----
Clients MUST send all the obj-ids it wants from the reference
@@ -249,6 +255,14 @@ complete those commits. Commits whose parents are not received as a
result are defined as shallow and marked as such in the server. This
information is sent back to the client in the next step.
+The client can optionally request a partial packfile that omits
+various blobs. The value of "partial-by-size" is a non-negative
+integer with optional units and requests blobs smaller than this
+value. The "partial-special" command requests git-special files,
+such as ".gitignore". Using both requests the union of the two.
+These requests are only valid if the server advertises the "partial"
+capability.
+
Once all the 'want's and 'shallow's (and optional 'deepen') are
transferred, clients MUST send a flush-pkt, to tell the server side
that it is done sending the list.
diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt
index 26dcc6f..9aa2123 100644
--- a/Documentation/technical/protocol-capabilities.txt
+++ b/Documentation/technical/protocol-capabilities.txt
@@ -309,3 +309,10 @@ to accept a signed push certificate, and asks the <nonce> to be
included in the push certificate. A send-pack client MUST NOT
send a push-cert packet unless the receive-pack server advertises
this capability.
+
+partial
+-------
+
+If the upload-pack server advertises this capability, fetch-pack
+may send various "partial-*" commands to request a partial clone
+or fetch where the server omits certain blobs from the packfile.
diff --git a/upload-pack.c b/upload-pack.c
index 7597ba3..74f9dfa 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -63,6 +63,11 @@ static int advertise_refs;
static int stateless_rpc;
static const char *pack_objects_hook;
+static struct strbuf partial_by_size = STRBUF_INIT;
+static int client_requested_partial_capability;
+static int have_partial_by_size;
+static int have_partial_special;
+
static void reset_timeout(void)
{
alarm(timeout);
@@ -130,6 +135,10 @@ static void create_pack_file(void)
argv_array_push(&pack_objects.args, "--delta-base-offset");
if (use_include_tag)
argv_array_push(&pack_objects.args, "--include-tag");
+ if (have_partial_by_size)
+ argv_array_push(&pack_objects.args, partial_by_size.buf);
+ if (have_partial_special)
+ argv_array_push(&pack_objects.args, "--partial-special");
pack_objects.in = -1;
pack_objects.out = -1;
@@ -793,6 +802,23 @@ static void receive_needs(void)
deepen_rev_list = 1;
continue;
}
+ if (skip_prefix(line, "partial-by-size ", &arg)) {
+ unsigned long s;
+ if (!client_requested_partial_capability)
+ die("git upload-pack: 'partial-by-size' option requires 'partial' capability");
+ if (!git_parse_ulong(arg, &s))
+ die("git upload-pack: invalid partial-by-size value: %s", line);
+ strbuf_addstr(&partial_by_size, "--partial-by-size=");
+ strbuf_addstr(&partial_by_size, arg);
+ have_partial_by_size = 1;
+ continue;
+ }
+ if (skip_prefix(line, "partial-special", &arg)) {
+ if (!client_requested_partial_capability)
+ die("git upload-pack: 'partial-special' option requires 'partial' capability");
+ have_partial_special = 1;
+ continue;
+ }
if (!skip_prefix(line, "want ", &arg) ||
get_sha1_hex(arg, sha1_buf))
die("git upload-pack: protocol error, "
@@ -820,6 +846,8 @@ static void receive_needs(void)
no_progress = 1;
if (parse_feature_request(features, "include-tag"))
use_include_tag = 1;
+ if (parse_feature_request(features, "partial"))
+ client_requested_partial_capability = 1;
o = parse_object(sha1_buf);
if (!o)
@@ -924,7 +952,9 @@ static int send_ref(const char *refname, const struct object_id *oid,
{
static const char *capabilities = "multi_ack thin-pack side-band"
" side-band-64k ofs-delta shallow deepen-since deepen-not"
- " deepen-relative no-progress include-tag multi_ack_detailed";
+ " deepen-relative no-progress include-tag multi_ack_detailed"
+ " partial"
+ ;
const char *refname_nons = strip_namespace(refname);
struct object_id peeled;
--
2.7.4
next prev parent reply other threads:[~2017-03-08 20:48 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-08 18:50 [PATCH 00/10] RFC Partial Clone and Fetch git
2017-03-08 18:50 ` [PATCH 01/10] pack-objects: eat CR in addition to LF after fgets git
2017-03-08 18:50 ` [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special git
2017-03-08 18:50 ` [PATCH 03/10] pack-objects: test for --partial-by-size --partial-special git
2017-03-08 18:50 ` git [this message]
2017-03-08 18:50 ` [PATCH 05/10] fetch-pack: add partial-by-size and partial-special git
2017-03-08 18:50 ` [PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks git
2017-03-08 18:50 ` [PATCH 07/10] index-pack: add --allow-partial option to relax blob existence checks git
2017-03-08 18:50 ` [PATCH 08/10] fetch: add partial-by-size and partial-special arguments git
2017-03-08 18:50 ` [PATCH 09/10] clone: " git
2017-03-08 18:50 ` [PATCH 10/10] ls-partial: created command to list missing blobs git
2017-03-09 20:18 ` [PATCH 00/10] RFC Partial Clone and Fetch Jonathan Tan
2017-03-16 21:43 ` Jeff Hostetler
2017-03-17 14:13 ` Jeff Hostetler
2017-03-22 15:16 ` ankostis
2017-03-22 16:21 ` Johannes Schindelin
2017-03-22 17:51 ` Jeff Hostetler
2017-05-03 16:38 ` Jeff Hostetler
2017-05-03 18:27 ` Jonathan Nieder
2017-05-04 16:51 ` Jeff Hostetler
2017-05-04 18:41 ` Jonathan Nieder
2017-05-08 0:15 ` Junio C Hamano
2017-05-03 20:40 ` Jonathan Tan
2017-05-03 21:08 ` Jonathan Nieder
-- strict thread matches above, loose matches on Subject: below --
2017-03-08 17:37 Jeff Hostetler
2017-03-08 17:37 ` [PATCH 04/10] upload-pack: add partial (sparse) fetch Jeff Hostetler
2017-03-09 7:48 ` Jeff King
2017-03-09 18:34 ` Jeff Hostetler
2017-03-09 19:09 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1488999039-37631-5-git-send-email-git@jeffhostetler.com \
--to=git@jeffhostetler.com \
--cc=benpeart@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jeffhost@microsoft.com \
--cc=jonathantanmy@google.com \
--cc=markbt@efaref.net \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).