* [PATCH 0/2] Skip ls-refs if possible for HTTP
@ 2019-08-21 22:20 Jonathan Tan
2019-08-21 22:20 ` [PATCH 1/2] transport-helper: skip ls-refs if unnecessary Jonathan Tan
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Jonathan Tan @ 2019-08-21 22:20 UTC (permalink / raw)
To: git; +Cc: Jonathan Tan
This was discovered by one of my colleagues when using a partial clone.
I thought I had resolved the problem with the commits mentioned in patch
1 (e70a3030e7 and ancestors), but apparently that is not the case (that
only worked for native protocols). So here is a fix for HTTP.
I'm not sure of the value of the test in patch 2, but that test does
fail if I don't update fetch_refs_from_bundle() to first call
get_refs_from_bundle() if it hasn't already been called.
Jonathan Tan (2):
transport-helper: skip ls-refs if unnecessary
transport: teach all vtables to allow fetch first
t/t5607-clone-bundle.sh | 11 +++++++++++
t/t5702-protocol-v2.sh | 13 +++++++++++++
transport-helper.c | 38 ++++++++++++++++++++++++++++++++------
transport-internal.h | 6 ------
transport.c | 18 ++++++------------
5 files changed, 62 insertions(+), 24 deletions(-)
--
2.23.0.187.g17f5b7556c-goog
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] transport-helper: skip ls-refs if unnecessary
2019-08-21 22:20 [PATCH 0/2] Skip ls-refs if possible for HTTP Jonathan Tan
@ 2019-08-21 22:20 ` Jonathan Tan
2019-08-21 22:20 ` [PATCH 2/2] transport: teach all vtables to allow fetch first Jonathan Tan
2019-08-22 16:27 ` [PATCH 0/2] Skip ls-refs if possible for HTTP Junio C Hamano
2 siblings, 0 replies; 5+ messages in thread
From: Jonathan Tan @ 2019-08-21 22:20 UTC (permalink / raw)
To: git; +Cc: Jonathan Tan
Commit e70a3030e7 ("fetch: do not list refs if fetching only hashes",
2018-10-07) and its ancestors taught Git, as an optimization, to skip
the ls-refs step when it is not necessary during a protocol v2 fetch
(for example, when lazy fetching a missing object in a partial clone, or
when running "git fetch --no-tags <remote> <SHA-1>"). But that was only
done for natively supported protocols; in particular, HTTP was not
supported.
Teach Git to skip ls-refs when using remote helpers that support connect
or stateless-connect. To do this, fetch() is made an acceptable entry
point. Because fetch() can now be the first function in the vtable
called, "get_helper(transport);" has to be added to the beginning of
that function to set the transport up (if not yet set up) before
process_connect() is invoked.
When fetch() is called, the transport could be taken over (this happens
if "connect" or "stateless-connect" is successfully run without any
"fallback" response), or not. If the transport is taken over, execution
continues like execution for natively supported protocols
(fetch_refs_via_pack() is executed, which will fetch refs using ls-refs
if needed). If not, the remote helper interface will invoke
get_refs_list() if it hasn't been invoked yet, preserving existing
behavior.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
t/t5702-protocol-v2.sh | 13 +++++++++++++
transport-helper.c | 39 +++++++++++++++++++++++++++++++++------
2 files changed, 46 insertions(+), 6 deletions(-)
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 011b81d4fc..61f5080de0 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -631,6 +631,19 @@ test_expect_success 'fetch with http:// using protocol v2' '
grep "git< version 2" log
'
+test_expect_success 'fetch with http:// by hash without tag following with protocol v2 does not list refs' '
+ test_when_finished "rm -f log" &&
+
+ test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two_a &&
+ git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" rev-parse two_a >two_a_hash &&
+
+ GIT_TRACE_PACKET="$(pwd)/log" git -C http_child -c protocol.version=2 \
+ fetch --no-tags origin $(cat two_a_hash) &&
+
+ grep "fetch< version 2" log &&
+ ! grep "fetch> command=ls-refs" log
+'
+
test_expect_success 'fetch from namespaced repo respects namespaces' '
test_when_finished "rm -f log" &&
diff --git a/transport-helper.c b/transport-helper.c
index 6b05a88faf..1fb31e1a6e 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -33,6 +33,16 @@ struct helper_data {
check_connectivity : 1,
no_disconnect_req : 1,
no_private_update : 1;
+
+ /*
+ * As an optimization, the transport code may invoke fetch before
+ * get_refs_list. If this happens, and if the transport helper doesn't
+ * support connect or stateless_connect, we need to invoke
+ * get_refs_list ourselves if we haven't already done so. Keep track of
+ * whether we have invoked get_refs_list.
+ */
+ unsigned get_refs_list_called : 1;
+
char *export_marks;
char *import_marks;
/* These go from remote name (as in "list") to private name */
@@ -652,17 +662,25 @@ static int connect_helper(struct transport *transport, const char *name,
return 0;
}
+static struct ref *get_refs_list_using_list(struct transport *transport,
+ int for_push);
+
static int fetch(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct helper_data *data = transport->data;
int i, count;
+ get_helper(transport);
+
if (process_connect(transport, 0)) {
do_take_over(transport);
return transport->vtable->fetch(transport, nr_heads, to_fetch);
}
+ if (!data->get_refs_list_called)
+ get_refs_list_using_list(transport, 0);
+
count = 0;
for (i = 0; i < nr_heads; i++)
if (!(to_fetch[i]->status & REF_STATUS_UPTODATE))
@@ -1058,6 +1076,19 @@ static int has_attribute(const char *attrs, const char *attr)
static struct ref *get_refs_list(struct transport *transport, int for_push,
const struct argv_array *ref_prefixes)
+{
+ get_helper(transport);
+
+ if (process_connect(transport, for_push)) {
+ do_take_over(transport);
+ return transport->vtable->get_refs_list(transport, for_push, ref_prefixes);
+ }
+
+ return get_refs_list_using_list(transport, for_push);
+}
+
+static struct ref *get_refs_list_using_list(struct transport *transport,
+ int for_push)
{
struct helper_data *data = transport->data;
struct child_process *helper;
@@ -1066,13 +1097,9 @@ static struct ref *get_refs_list(struct transport *transport, int for_push,
struct ref *posn;
struct strbuf buf = STRBUF_INIT;
+ data->get_refs_list_called = 1;
helper = get_helper(transport);
- if (process_connect(transport, for_push)) {
- do_take_over(transport);
- return transport->vtable->get_refs_list(transport, for_push, ref_prefixes);
- }
-
if (data->push && for_push)
write_str_in_full(helper->in, "list for-push\n");
else
@@ -1119,7 +1146,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push,
}
static struct transport_vtable vtable = {
- 0,
+ 1,
set_helper_option,
get_refs_list,
fetch,
--
2.23.0.187.g17f5b7556c-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] transport: teach all vtables to allow fetch first
2019-08-21 22:20 [PATCH 0/2] Skip ls-refs if possible for HTTP Jonathan Tan
2019-08-21 22:20 ` [PATCH 1/2] transport-helper: skip ls-refs if unnecessary Jonathan Tan
@ 2019-08-21 22:20 ` Jonathan Tan
2019-08-22 16:27 ` [PATCH 0/2] Skip ls-refs if possible for HTTP Junio C Hamano
2 siblings, 0 replies; 5+ messages in thread
From: Jonathan Tan @ 2019-08-21 22:20 UTC (permalink / raw)
To: git; +Cc: Jonathan Tan
The only transport that does not allow fetch() to be called before
get_refs_list() is the bundle transport. Clean up the code by teaching
the bundle transport the ability to do this, and removing support for
transports that don't support this order of invocation.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
t/t5607-clone-bundle.sh | 11 +++++++++++
transport-helper.c | 1 -
transport-internal.h | 6 ------
transport.c | 18 ++++++------------
4 files changed, 17 insertions(+), 19 deletions(-)
diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh
index 2a0fb15cf1..b7a3fdf02d 100755
--- a/t/t5607-clone-bundle.sh
+++ b/t/t5607-clone-bundle.sh
@@ -83,4 +83,15 @@ test_expect_success 'failed bundle creation does not leave cruft' '
test_path_is_missing fail.bundle.lock
'
+test_expect_success 'fetch SHA-1 from bundle' '
+ test_create_repo foo &&
+ test_commit -C foo x &&
+ git -C foo bundle create tip.bundle -1 master &&
+ git -C foo rev-parse HEAD >hash &&
+
+ # Exercise to ensure that fetching a SHA-1 from a bundle works with no
+ # errors
+ git fetch --no-tags foo/tip.bundle "$(cat hash)"
+'
+
test_done
diff --git a/transport-helper.c b/transport-helper.c
index 1fb31e1a6e..96955d4004 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -1146,7 +1146,6 @@ static struct ref *get_refs_list_using_list(struct transport *transport,
}
static struct transport_vtable vtable = {
- 1,
set_helper_option,
get_refs_list,
fetch,
diff --git a/transport-internal.h b/transport-internal.h
index 004bee5e36..1cde6258a7 100644
--- a/transport-internal.h
+++ b/transport-internal.h
@@ -6,12 +6,6 @@ struct transport;
struct argv_array;
struct transport_vtable {
- /**
- * This transport supports the fetch() function being called
- * without get_refs_list() first being called.
- */
- unsigned fetch_without_list : 1;
-
/**
* Returns 0 if successful, positive if the option is not
* recognized or is inapplicable, and negative if the option
diff --git a/transport.c b/transport.c
index 778c60bf57..662a2d9ae0 100644
--- a/transport.c
+++ b/transport.c
@@ -122,6 +122,7 @@ static void set_upstreams(struct transport *transport, struct ref *refs,
struct bundle_transport_data {
int fd;
struct bundle_header header;
+ unsigned get_refs_from_bundle_called : 1;
};
static struct ref *get_refs_from_bundle(struct transport *transport,
@@ -135,6 +136,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport,
if (for_push)
return NULL;
+ data->get_refs_from_bundle_called = 1;
+
if (data->fd > 0)
close(data->fd);
data->fd = read_bundle_header(transport->url, &data->header);
@@ -154,6 +157,9 @@ static int fetch_refs_from_bundle(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct bundle_transport_data *data = transport->data;
+
+ if (!data->get_refs_from_bundle_called)
+ get_refs_from_bundle(transport, 0, NULL);
return unbundle(the_repository, &data->header, data->fd,
transport->progress ? BUNDLE_VERBOSE : 0);
}
@@ -742,7 +748,6 @@ static int disconnect_git(struct transport *transport)
}
static struct transport_vtable taken_over_vtable = {
- 1,
NULL,
get_refs_via_connect,
fetch_refs_via_pack,
@@ -892,7 +897,6 @@ void transport_check_allowed(const char *type)
}
static struct transport_vtable bundle_vtable = {
- 0,
NULL,
get_refs_from_bundle,
fetch_refs_from_bundle,
@@ -902,7 +906,6 @@ static struct transport_vtable bundle_vtable = {
};
static struct transport_vtable builtin_smart_vtable = {
- 1,
NULL,
get_refs_via_connect,
fetch_refs_via_pack,
@@ -1285,15 +1288,6 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)
struct ref **heads = NULL;
struct ref *rm;
- if (!transport->vtable->fetch_without_list)
- /*
- * Some transports (e.g. the built-in bundle transport and the
- * transport helper interface) do not work when fetching is
- * done immediately after transport creation. List the remote
- * refs anyway (if not already listed) as a workaround.
- */
- transport_get_remote_refs(transport, NULL);
-
for (rm = refs; rm; rm = rm->next) {
nr_refs++;
if (rm->peer_ref &&
--
2.23.0.187.g17f5b7556c-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 0/2] Skip ls-refs if possible for HTTP
2019-08-21 22:20 [PATCH 0/2] Skip ls-refs if possible for HTTP Jonathan Tan
2019-08-21 22:20 ` [PATCH 1/2] transport-helper: skip ls-refs if unnecessary Jonathan Tan
2019-08-21 22:20 ` [PATCH 2/2] transport: teach all vtables to allow fetch first Jonathan Tan
@ 2019-08-22 16:27 ` Junio C Hamano
2019-08-22 17:23 ` Jonathan Tan
2 siblings, 1 reply; 5+ messages in thread
From: Junio C Hamano @ 2019-08-22 16:27 UTC (permalink / raw)
To: Jonathan Tan; +Cc: git
Jonathan Tan <jonathantanmy@google.com> writes:
> This was discovered by one of my colleagues when using a partial clone.
> I thought I had resolved the problem with the commits mentioned in patch
> 1 (e70a3030e7 and ancestors), but apparently that is not the case (that
> only worked for native protocols). So here is a fix for HTTP.
>
> I'm not sure of the value of the test in patch 2, but that test does
> fail if I don't update fetch_refs_from_bundle() to first call
> get_refs_from_bundle() if it hasn't already been called.
This probably is totally off-tangent, but do any of these "let's
advertise fewer" changes at the protocol level have to take into
account the use of --prune option on the client side?
> Jonathan Tan (2):
> transport-helper: skip ls-refs if unnecessary
> transport: teach all vtables to allow fetch first
>
> t/t5607-clone-bundle.sh | 11 +++++++++++
> t/t5702-protocol-v2.sh | 13 +++++++++++++
> transport-helper.c | 38 ++++++++++++++++++++++++++++++++------
> transport-internal.h | 6 ------
> transport.c | 18 ++++++------------
> 5 files changed, 62 insertions(+), 24 deletions(-)
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/2] Skip ls-refs if possible for HTTP
2019-08-22 16:27 ` [PATCH 0/2] Skip ls-refs if possible for HTTP Junio C Hamano
@ 2019-08-22 17:23 ` Jonathan Tan
0 siblings, 0 replies; 5+ messages in thread
From: Jonathan Tan @ 2019-08-22 17:23 UTC (permalink / raw)
To: gitster; +Cc: jonathantanmy, git
> This probably is totally off-tangent, but do any of these "let's
> advertise fewer" changes at the protocol level have to take into
> account the use of --prune option on the client side?
I don't think so. According to what I understand from the documentation,
the prune option prunes based on the RHS of the refspec, and it doesn't
affect anything non-matching. When we advertise fewer, the only refs
that are missing are those that are non-matching anyway.
Some experimentation seems to show that that is the case also:
$ git init one
[snip]
$ git -C one commit --allow-empty -m x
[master (root-commit) 67056ac] x
$ git -C one branch maste
$ git -C one branch other
$ git clone "$(pwd)/one" two
Cloning into 'two'...
done.
$ git -C one branch -d maste
Deleted branch maste (was 67056ac).
$ git -C two checkout --detach HEAD
HEAD is now at 67056ac x
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 -C two fetch --prune origin refs/heads/m*:refs/remotes/origin/m*
[snipped lots of stuff; basically, only refs/heads/master is sent]
10:15:46.308264 pkt-line.c:80 packet: fetch> ref-prefix refs/heads/m
10:15:46.308276 pkt-line.c:80 packet: fetch> ref-prefix refs/tags/
10:15:46.308523 pkt-line.c:80 packet: fetch< 67056ac6d07814334716df760054ac5bec05b66a refs/heads/master
From /usr/local/google/home/jonathantanmy/tmp/g/one
- [deleted] (none) -> origin/maste
$ git -C two for-each-ref
67056ac6d07814334716df760054ac5bec05b66a commit refs/heads/master
67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/HEAD
67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/master
67056ac6d07814334716df760054ac5bec05b66a commit refs/remotes/origin/other
(Notice that refs/remotes/origin/other is untouched.)
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-08-22 17:44 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-21 22:20 [PATCH 0/2] Skip ls-refs if possible for HTTP Jonathan Tan
2019-08-21 22:20 ` [PATCH 1/2] transport-helper: skip ls-refs if unnecessary Jonathan Tan
2019-08-21 22:20 ` [PATCH 2/2] transport: teach all vtables to allow fetch first Jonathan Tan
2019-08-22 16:27 ` [PATCH 0/2] Skip ls-refs if possible for HTTP Junio C Hamano
2019-08-22 17:23 ` Jonathan Tan
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).