From: Jonathan Nieder <jrnieder@gmail.com>
To: Jonathan Tan <jonathantanmy@google.com>
Cc: git@vger.kernel.org, Greg Thelen <gthelen@google.com>,
Michel Lespinasse <walken@google.com>
Subject: Re: [PATCH] fetch-pack: support negotiation tip whitelist
Date: Tue, 26 Jun 2018 10:53:24 -0700 [thread overview]
Message-ID: <20180626175324.GA12488@aiede.svl.corp.google.com> (raw)
In-Reply-To: <20180625193752.255319-1-jonathantanmy@google.com>
Hi,
Jonathan Tan wrote:
> During negotiation, fetch-pack eventually reports as "have" lines all
> commits reachable from all refs. Allow the user to restrict the commits
> sent in this way by providing a whitelist of tips; only the tips
> themselves and their ancestors will be sent.
>
> This feature is only supported for protocols that support connect or
> stateless-connect (such as HTTP with protocol v2).
>
> This will speed up negotiation when the repository has multiple
> relatively independent branches (for example, when a repository
> interacts with multiple repositories, such as with linux-next [1] and
> torvalds/linux [2]), and the user knows which local branch is likely to
> have commits in common with the upstream branch they are fetching.
>
> [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/
> [2] https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/
>
> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
> ---
Very neat. Thanks to Greg Thelen and Michel Lespinasse (cc-ed) for
this suggestion.
> This is based on jt/fetch-pack-negotiator, but if that branch is
> undesirable for whatever reason, I can port this to master.
>
> builtin/fetch.c | 21 ++++++++++++++++++
> fetch-pack.c | 19 ++++++++++++++--
> fetch-pack.h | 7 ++++++
> t/t5510-fetch.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++++
> transport-helper.c | 3 +++
> transport.c | 1 +
> transport.h | 10 +++++++++
> 7 files changed, 114 insertions(+), 2 deletions(-)
Small nit: could this be documented in Documentation/fetch.txt as well?
Thanks,
Jonathan
Patch left unsnipped for reference.
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index ea5b9669a..12daec0f3 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -63,6 +63,7 @@ static int shown_url = 0;
> static struct refspec refmap = REFSPEC_INIT_FETCH;
> static struct list_objects_filter_options filter_options;
> static struct string_list server_options = STRING_LIST_INIT_DUP;
> +static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
>
> static int git_fetch_config(const char *k, const char *v, void *cb)
> {
> @@ -174,6 +175,8 @@ static struct option builtin_fetch_options[] = {
> TRANSPORT_FAMILY_IPV4),
> OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
> TRANSPORT_FAMILY_IPV6),
> + OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
> + N_("report that we have only objects reachable from this object")),
> OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
> OPT_END()
> };
> @@ -1075,6 +1078,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)
> filter_options.filter_spec);
> set_option(transport, TRANS_OPT_FROM_PROMISOR, "1");
> }
> + if (negotiation_tip.nr) {
> + struct oid_array *oids;
> + if (transport->smart_options) {
> + int i;
> + oids = xcalloc(1, sizeof(*oids));
> + for (i = 0; i < negotiation_tip.nr; i++) {
> + struct object_id oid;
> + if (get_oid(negotiation_tip.items[i].string,
> + &oid))
> + die("%s is not a valid object",
> + negotiation_tip.items[i].string);
> + oid_array_append(oids, &oid);
> + }
> + transport->smart_options->negotiation_tips = oids;
> + } else {
> + warning("Ignoring --negotiation-tip because the protocol does not support it.");
> + }
> + }
> return transport;
> }
>
> diff --git a/fetch-pack.c b/fetch-pack.c
> index ba12085c4..c66bd49bd 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -213,6 +213,21 @@ static int next_flush(int stateless_rpc, int count)
> return count;
> }
>
> +static void mark_tips(struct fetch_negotiator *negotiator,
> + const struct oid_array *negotiation_tips)
> +{
> + int i;
> + if (!negotiation_tips) {
> + for_each_ref(rev_list_insert_ref_oid, negotiator);
> + return;
> + }
> +
> + for (i = 0; i < negotiation_tips->nr; i++)
> + rev_list_insert_ref(negotiator, NULL,
> + &negotiation_tips->oid[i]);
> + return;
> +}
> +
> static int find_common(struct fetch_negotiator *negotiator,
> struct fetch_pack_args *args,
> int fd[2], struct object_id *result_oid,
> @@ -230,7 +245,7 @@ static int find_common(struct fetch_negotiator *negotiator,
> if (args->stateless_rpc && multi_ack == 1)
> die(_("--stateless-rpc requires multi_ack_detailed"));
>
> - for_each_ref(rev_list_insert_ref_oid, negotiator);
> + mark_tips(negotiator, args->negotiation_tips);
> for_each_cached_alternate(negotiator, insert_one_alternate_object);
>
> fetching = 0;
> @@ -1295,7 +1310,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
> else
> state = FETCH_SEND_REQUEST;
>
> - for_each_ref(rev_list_insert_ref_oid, &negotiator);
> + mark_tips(&negotiator, args->negotiation_tips);
> for_each_cached_alternate(&negotiator,
> insert_one_alternate_object);
> break;
> diff --git a/fetch-pack.h b/fetch-pack.h
> index bb45a366a..1859ee927 100644
> --- a/fetch-pack.h
> +++ b/fetch-pack.h
> @@ -16,6 +16,13 @@ struct fetch_pack_args {
> const struct string_list *deepen_not;
> struct list_objects_filter_options filter_options;
> const struct string_list *server_options;
> +
> + /*
> + * If not NULL, during packfile negotiation, fetch-pack will send "have"
> + * lines only with these tips and their ancestors.
> + */
> + const struct oid_array *negotiation_tips;
> +
> unsigned deepen_relative:1;
> unsigned quiet:1;
> unsigned keep_pack:1;
> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
> index e402aee6a..ea1b5e53c 100755
> --- a/t/t5510-fetch.sh
> +++ b/t/t5510-fetch.sh
> @@ -865,4 +865,59 @@ test_expect_success C_LOCALE_OUTPUT 'fetch compact output' '
> test_cmp expect actual
> '
>
> +negotiator_tip () {
> + SERVER="$1"
> + URL="$2"
> + USE_PROTOCOL_V2="$3"
> +
> + rm -rf "$SERVER" client &&
> + git init "$SERVER" &&
> + test_commit -C "$SERVER" alpha_1 &&
> + test_commit -C "$SERVER" alpha_2 &&
> + git -C "$SERVER" checkout --orphan beta &&
> + test_commit -C "$SERVER" beta_1 &&
> + test_commit -C "$SERVER" beta_2 &&
> +
> + git clone "$URL" client &&
> +
> + if [ "$USE_PROTOCOL_V2" -eq 1 ]
> + then
> + git -C "$SERVER" config protocol.version 2
> + git -C client config protocol.version 2
> + fi &&
> +
> + test_commit -C "$SERVER" beta_s &&
> + git -C "$SERVER" checkout master &&
> + test_commit -C "$SERVER" alpha_s &&
> + git -C "$SERVER" tag -d alpha_1 alpha_2 beta_1 beta_2 &&
> +
> + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
> + --negotiation-tip=alpha_1 --negotiation-tip=beta_1 \
> + origin alpha_s beta_s &&
> +
> + # Ensure that {alpha,beta}_1 are sent as "have", but not {alpha_beta}_2
> + ALPHA_1=$(git -C client rev-parse alpha_1) &&
> + grep "fetch> have $ALPHA_1" trace &&
> + BETA_1=$(git -C client rev-parse beta_1) &&
> + grep "fetch> have $BETA_1" trace &&
> + ALPHA_2=$(git -C client rev-parse alpha_2) &&
> + ! grep "fetch> have $ALPHA_2" trace &&
> + BETA_2=$(git -C client rev-parse beta_2) &&
> + ! grep "fetch> have $BETA_2" trace
> +}
> +
> +test_expect_success '--negotiator-tip limits "have" lines sent' '
> + negotiator_tip server server 0
> +'
> +
> +. "$TEST_DIRECTORY"/lib-httpd.sh
> +start_httpd
> +
> +test_expect_success 'fetch with --filter=blob:limit=0 and HTTP' '
> + negotiator_tip "$HTTPD_DOCUMENT_ROOT_PATH/server" \
> + "$HTTPD_URL/smart/server" 1
> +'
> +
> +stop_httpd
> +
> test_done
> diff --git a/transport-helper.c b/transport-helper.c
> index 1f8ff7e94..ad8f7c772 100644
> --- a/transport-helper.c
> +++ b/transport-helper.c
> @@ -684,6 +684,9 @@ static int fetch(struct transport *transport,
> transport, "filter",
> data->transport_options.filter_options.filter_spec);
>
> + if (data->transport_options.negotiation_tips)
> + warning("Ignoring --negotiation-tip because the protocol does not support it.");
> +
> if (data->fetch)
> return fetch_with_fetch(transport, nr_heads, to_fetch);
>
> diff --git a/transport.c b/transport.c
> index a32da30de..9f10f8ad9 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -318,6 +318,7 @@ static int fetch_refs_via_pack(struct transport *transport,
> args.filter_options = data->options.filter_options;
> args.stateless_rpc = transport->stateless_rpc;
> args.server_options = transport->server_options;
> + args.negotiation_tips = data->options.negotiation_tips;
>
> if (!data->got_remote_heads)
> refs_tmp = get_refs_via_connect(transport, 0, NULL);
> diff --git a/transport.h b/transport.h
> index 7792b0858..d31be5be6 100644
> --- a/transport.h
> +++ b/transport.h
> @@ -25,6 +25,16 @@ struct git_transport_options {
> const char *receivepack;
> struct push_cas_option *cas;
> struct list_objects_filter_options filter_options;
> +
> + /*
> + * This is only used during fetch. See the documentation of
> + * negotiation_tips in struct fetch_pack_args.
> + *
> + * This field is only supported by transports that support connect or
> + * stateless_connect. Set this field directly instead of using
> + * transport_set_option().
> + */
> + struct oid_array *negotiation_tips;
> };
>
> enum transport_family {
> --
> 2.18.0.rc2.347.g0da03f3a46.dirty
>
next prev parent reply other threads:[~2018-06-26 17:53 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-25 19:37 [PATCH] fetch-pack: support negotiation tip whitelist Jonathan Tan
2018-06-26 17:53 ` Jonathan Nieder [this message]
2018-06-26 18:59 ` Junio C Hamano
2018-06-27 18:28 ` [PATCH v2] " Jonathan Tan
2018-06-28 15:56 ` Brandon Williams
2018-06-28 16:12 ` Jonathan Tan
2018-06-28 16:16 ` Brandon Williams
2018-06-28 22:15 ` [PATCH v3] " Jonathan Tan
2018-06-28 22:20 ` Brandon Williams
2018-06-29 16:28 ` Junio C Hamano
2018-07-02 22:39 ` [PATCH v4] " Jonathan Tan
2018-07-22 9:09 ` Duy Nguyen
2019-06-18 13:36 ` Ævar Arnfjörð Bjarmason
2019-06-18 17:30 ` Jonathan Tan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180626175324.GA12488@aiede.svl.corp.google.com \
--to=jrnieder@gmail.com \
--cc=git@vger.kernel.org \
--cc=gthelen@google.com \
--cc=jonathantanmy@google.com \
--cc=walken@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).