[PATCH 0/2] Bugfix for partial clones and ref-in-want servers

[PATCH 0/2] Bugfix for partial clones and ref-in-want servers

[Jonathan Tan]

A coworker discovered a bug when having a partial clone against a server
that supports ref-in-want - the lazy fetches no longer work, because the
client attempts to fetch with "want-ref <hash>", which is not permitted
by the protocol.

The first patch deduplicates some code, so that the bugfix need only be
applied once, and the second patch contains the actual bugfix.

Jonathan Tan (2):
  fetch-object: provide only one fetching function
  fetch-object: set exact_oid when fetching

 fetch-object.c           | 17 ++++++-----------
 fetch-object.h           |  8 ++------
 sha1-file.c              |  2 +-
 t/t0410-partial-clone.sh | 12 ++++++++++++
 unpack-trees.c           |  2 +-
 5 files changed, 22 insertions(+), 19 deletions(-)

-- 
2.19.0.397.gdd90340f6a-goog

[PATCH 1/2] fetch-object: provide only one fetching function

[Jonathan Tan]

fetch-object.h currently provides two functions (fetch_object() and
fetch_objects()) that could be replaced by a single, more flexible
function. Replace those two functions with the more flexible function.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 fetch-object.c | 16 +++++-----------
 fetch-object.h |  8 ++------
 sha1-file.c    |  2 +-
 unpack-trees.c |  2 +-
 4 files changed, 9 insertions(+), 19 deletions(-)

diff --git a/fetch-object.c b/fetch-object.c
index 853624f81..1af1bf857 100644
--- a/fetch-object.c
+++ b/fetch-object.c
@@ -23,21 +23,15 @@ static void fetch_refs(const char *remote_name, struct ref *ref)
 	fetch_if_missing = original_fetch_if_missing;
 }
 
-void fetch_object(const char *remote_name, const unsigned char *sha1)
-{
-	struct ref *ref = alloc_ref(sha1_to_hex(sha1));
-	hashcpy(ref->old_oid.hash, sha1);
-	fetch_refs(remote_name, ref);
-}
-
-void fetch_objects(const char *remote_name, const struct oid_array *to_fetch)
+void fetch_objects(const char *remote_name, const struct object_id *oids,
+		   int oid_nr)
 {
 	struct ref *ref = NULL;
 	int i;
 
-	for (i = 0; i < to_fetch->nr; i++) {
-		struct ref *new_ref = alloc_ref(oid_to_hex(&to_fetch->oid[i]));
-		oidcpy(&new_ref->old_oid, &to_fetch->oid[i]);
+	for (i = 0; i < oid_nr; i++) {
+		struct ref *new_ref = alloc_ref(oid_to_hex(&oids[i]));
+		oidcpy(&new_ref->old_oid, &oids[i]);
 		new_ref->next = ref;
 		ref = new_ref;
 	}
diff --git a/fetch-object.h b/fetch-object.h
index 4b269d07e..d2f996d4e 100644
--- a/fetch-object.h
+++ b/fetch-object.h
@@ -1,11 +1,7 @@
 #ifndef FETCH_OBJECT_H
 #define FETCH_OBJECT_H
 
-#include "sha1-array.h"
-
-extern void fetch_object(const char *remote_name, const unsigned char *sha1);
-
-extern void fetch_objects(const char *remote_name,
-			  const struct oid_array *to_fetch);
+void fetch_objects(const char *remote_name, const struct object_id *oids,
+		   int oid_nr);
 
 #endif
diff --git a/sha1-file.c b/sha1-file.c
index 97b742384..2edf4564f 100644
--- a/sha1-file.c
+++ b/sha1-file.c
@@ -1317,7 +1317,7 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
 			 * TODO Pass a repository struct through fetch_object,
 			 * such that arbitrary repositories work.
 			 */
-			fetch_object(repository_format_partial_clone, real->hash);
+			fetch_objects(repository_format_partial_clone, real, 1);
 			already_retried = 1;
 			continue;
 		}
diff --git a/unpack-trees.c b/unpack-trees.c
index f25089b87..82a83dbc6 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -392,7 +392,7 @@ static int check_updates(struct unpack_trees_options *o)
 		}
 		if (to_fetch.nr)
 			fetch_objects(repository_format_partial_clone,
-				      &to_fetch);
+				      to_fetch.oid, to_fetch.nr);
 		fetch_if_missing = fetch_if_missing_store;
 		oid_array_clear(&to_fetch);
 	}
-- 
2.19.0.397.gdd90340f6a-goog

[PATCH 2/2] fetch-object: set exact_oid when fetching

[Jonathan Tan]

fetch_objects() currently does not set exact_oid in struct ref when
invoking transport_fetch_refs(). If the server supports ref-in-want,
fetch_pack() uses this field to determine whether a wanted ref should be
requested as a "want-ref" line or a "want" line; without the setting of
exact_oid, the wrong line will be sent.

Set exact_oid, so that the correct line is sent.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 fetch-object.c           |  1 +
 t/t0410-partial-clone.sh | 12 ++++++++++++
 2 files changed, 13 insertions(+)

diff --git a/fetch-object.c b/fetch-object.c
index 1af1bf857..426654880 100644
--- a/fetch-object.c
+++ b/fetch-object.c
@@ -32,6 +32,7 @@ void fetch_objects(const char *remote_name, const struct object_id *oids,
 	for (i = 0; i < oid_nr; i++) {
 		struct ref *new_ref = alloc_ref(oid_to_hex(&oids[i]));
 		oidcpy(&new_ref->old_oid, &oids[i]);
+		new_ref->exact_oid = 1;
 		new_ref->next = ref;
 		ref = new_ref;
 	}
diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh
index 128130066..0ab02c337 100755
--- a/t/t0410-partial-clone.sh
+++ b/t/t0410-partial-clone.sh
@@ -170,6 +170,18 @@ test_expect_success 'fetching of missing objects' '
 	git verify-pack --verbose "$IDX" | grep "$HASH"
 '
 
+test_expect_success 'fetching of missing objects works with ref-in-want enabled' '
+	# ref-in-want requires protocol version 2
+	git -C server config protocol.version 2 &&
+	git -C server config uploadpack.allowrefinwant 1 &&
+	git -C repo config protocol.version 2 &&
+
+	rm -rf repo/.git/objects/* &&
+	rm -f trace &&
+	GIT_TRACE_PACKET="$(pwd)/trace" git -C repo cat-file -p "$HASH" &&
+	grep "git< fetch=.*ref-in-want" trace
+'
+
 test_expect_success 'rev-list stops traversal at missing and promised commit' '
 	rm -rf repo &&
 	test_create_repo repo &&
-- 
2.19.0.397.gdd90340f6a-goog

Re: [PATCH 1/2] fetch-object: provide only one fetching function

[Junio C Hamano]

Jonathan Tan <jonathantanmy@google.com> writes:

> fetch-object.h currently provides two functions (fetch_object() and
> fetch_objects()) that could be replaced by a single, more flexible
> function. Replace those two functions with the more flexible function.

The latter half of the first sentence and the second sentence are
pretty-much repeating the same thing.  

I think you wanted to justify two changes:

 (1) There are two helpers.  fetch_object() fetches a single object
     from a given remote; fetch_objects() fetches a set of objects
     from a given remote.  It is not like the former is implemented
     as a specialization of the latter (i.e. passing length=1
     array), and it is not like the former is optimized specially
     for a single object fetch that cannot be made into such a thin
     wrapper.  It is not like the latter is implemented as a
     repeated call to the former in a loop, either.  There is no
     justification to keep two independent implementations, and
     using a single helper that can be used by all the callers of
     these two would make sense.

 (2) The variant that fetches multiple objects takes oid_array.  To
     be used by all the existing callers of these two helpers, it is
     better to use a different calling convention, namely, a array[]
     and its length passed as separate parameters.
     
Instead of explaining why the new convention is better to justify
(2), the above three lines handwave by saying "more flexible"
twice.  We should do better.

	fetch-object: unify fetch_object[s] functions

	There are fetch_object() and fetch_objects() helpers in
	fetch-object.h; as the latter takes "struct oid_array",
	the former cannot be made into a thin wrapper around the
	latter without an extra allocation and set-up cost.

	Update fetch_objects() to take an array of "struct
	object_id" and number of elements in it as separate
	parameters, remove fetch_object(), and adjust all existing
	callers of these functions to use the new fetch_objects().

perhaps?

> diff --git a/sha1-file.c b/sha1-file.c
> index 97b742384..2edf4564f 100644
> --- a/sha1-file.c
> +++ b/sha1-file.c
> @@ -1317,7 +1317,7 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
>  			 * TODO Pass a repository struct through fetch_object,
>  			 * such that arbitrary repositories work.
>  			 */
> -			fetch_object(repository_format_partial_clone, real->hash);
> +			fetch_objects(repository_format_partial_clone, real, 1);
>  			already_retried = 1;
>  			continue;
>  		}
> diff --git a/unpack-trees.c b/unpack-trees.c
> index f25089b87..82a83dbc6 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -392,7 +392,7 @@ static int check_updates(struct unpack_trees_options *o)
>  		}
>  		if (to_fetch.nr)
>  			fetch_objects(repository_format_partial_clone,
> -				      &to_fetch);
> +				      to_fetch.oid, to_fetch.nr);
>  		fetch_if_missing = fetch_if_missing_store;
>  		oid_array_clear(&to_fetch);
>  	}

Changes to these two callers do explain why <ptr, nr> is an
interface that is easier to use.  Those who already have an
oid_array can pass its fields as separate parameters without too
much trouble, and those who only have an oid (or an array of oid and
knows how many are in the array) do not have to wrap them into an
oid_array.

Thanks.

Re: [PATCH 1/2] fetch-object: provide only one fetching function

[Jonathan Tan]

> Instead of explaining why the new convention is better to justify
> (2), the above three lines handwave by saying "more flexible"
> twice.  We should do better.
> 
> 	fetch-object: unify fetch_object[s] functions
> 
> 	There are fetch_object() and fetch_objects() helpers in
> 	fetch-object.h; as the latter takes "struct oid_array",
> 	the former cannot be made into a thin wrapper around the
> 	latter without an extra allocation and set-up cost.
> 
> 	Update fetch_objects() to take an array of "struct
> 	object_id" and number of elements in it as separate
> 	parameters, remove fetch_object(), and adjust all existing
> 	callers of these functions to use the new fetch_objects().
> 
> perhaps?

Thanks - your explanation is much clearer than mine. Let me know if you
want a reroll (or if you can update the commit message yourself, that's
fine too).

Re: [PATCH 1/2] fetch-object: provide only one fetching function

[Junio C Hamano]

Jonathan Tan <jonathantanmy@google.com> writes:

>> Instead of explaining why the new convention is better to justify
>> (2), the above three lines handwave by saying "more flexible"
>> twice.  We should do better.
>> 
>> 	fetch-object: unify fetch_object[s] functions
>> 
>> 	There are fetch_object() and fetch_objects() helpers in
>> 	fetch-object.h; as the latter takes "struct oid_array",
>> 	the former cannot be made into a thin wrapper around the
>> 	latter without an extra allocation and set-up cost.
>> 
>> 	Update fetch_objects() to take an array of "struct
>> 	object_id" and number of elements in it as separate
>> 	parameters, remove fetch_object(), and adjust all existing
>> 	callers of these functions to use the new fetch_objects().
>> 
>> perhaps?
>
> Thanks - your explanation is much clearer than mine. Let me know if you
> want a reroll (or if you can update the commit message yourself, that's
> fine too).

If there is no other change needed for either of the patches, I do
not mind rewording the 1/2 myself to save a round-trip.

Thanks.