[PATCH] submodule: explain first attempt failure clearly

[PATCH] submodule: explain first attempt failure clearly

[Jonathan Tan]

When cloning with --recurse-submodules a superproject with at least one
submodule with HEAD pointing to an unborn branch, the clone goes
something like this:

	Cloning into 'test'...
	<messages about cloning of superproject>
	Submodule '<name>' (<uri>) registered for path '<submodule path>'
	Cloning into '<submodule path>'...
	fatal: Couldn't find remote ref HEAD
	Unable to fetch in submodule path '<submodule path>'
	<messages about fetching with SHA-1>
	From <uri>
	 * branch            <hash> -> FETCH_HEAD
	Submodule path '<submodule path>': checked out '<hash>'

In other words, first, a fetch is done with no hash arguments (that is,
a fetch of HEAD) resulting in a "Couldn't find remote ref HEAD" error;
then, a fetch is done given a hash, which succeeds.

The fetch given a hash was added in fb43e31f2b ("submodule: try harder
to fetch needed sha1 by direct fetching sha1", 2016-02-24), and the
"Unable to fetch..." message was downgraded from a fatal error to a
notice in e30d833671 ("git-submodule.sh: try harder to fetch a
submodule", 2018-05-16).

This commit improves the notice to be clearer that we are retrying the
fetch, and that the previous messages do not necessarily indicate that
the whole command fails. In other words:

 - If the HEAD-fetch succeeds and we then have the commit we want, no
   extra messages are printed.
 - If the HEAD-fetch succeeds and we do not have the commit we want, but
   the hash-fetch succeeds, no additional messages are printed.
 - If the HEAD-fetch succeeds and we do not have the commit we want, but
   the hash-fetch fails, this is a fatal error.
 - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
   succeeds, no additional messages are printed.
 - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
   fails, this is a fatal error.

It could be said that we should just eliminate the HEAD-fetch
altogether, but that changes some behavior (in particular, some refs
that were opportunistically updated would no longer be), so I have left
that alone for now.

There is an analogous situation with the fetching code in fetch_finish()
and surrounding functions. For now, I have added a NEEDSWORK.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 git-submodule.sh | 2 +-
 submodule.c      | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 514ede2596..2c0fb6d723 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -594,7 +594,7 @@ cmd_update()
 				# is not reachable from a ref.
 				is_tip_reachable "$sm_path" "$sha1" ||
 				fetch_in_submodule "$sm_path" $depth ||
-				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
+				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'; trying to directly fetch \$sha1:")"
 
 				# Now we tried the usual fetch, but $sha1 may
 				# not be reachable from any of the refs
diff --git a/submodule.c b/submodule.c
index 21cf50ca15..b16c0ecc95 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1548,6 +1548,13 @@ static int fetch_finish(int retvalue, struct strbuf *err,
 	struct oid_array *commits;
 
 	if (retvalue)
+		/*
+		 * NEEDSWORK: This indicates that the overall fetch
+		 * failed, even though there may be a subsequent fetch
+		 * by commit hash that might work. It may be a good
+		 * idea to not indicate failure in this case, and only
+		 * indicate failure if the subsequent fetch fails.
+		 */
 		spf->result = 1;
 
 	if (!task || !task->sub)
-- 
2.21.0.155.ge902e9bcae.dirty

Re: [PATCH] submodule: explain first attempt failure clearly

[Junio C Hamano]

Jonathan Tan <jonathantanmy@google.com> writes:

> When cloning with --recurse-submodules a superproject with at least one
> submodule with HEAD pointing to an unborn branch, the clone goes
> something like this:
>
> 	Cloning into 'test'...
> 	<messages about cloning of superproject>
> 	Submodule '<name>' (<uri>) registered for path '<submodule path>'
> 	Cloning into '<submodule path>'...
> 	fatal: Couldn't find remote ref HEAD
> 	Unable to fetch in submodule path '<submodule path>'
> 	<messages about fetching with SHA-1>
> 	From <uri>
> 	 * branch            <hash> -> FETCH_HEAD
> 	Submodule path '<submodule path>': checked out '<hash>'
>
> In other words, first, a fetch is done with no hash arguments (that is,
> a fetch of HEAD) resulting in a "Couldn't find remote ref HEAD" error;
> then, a fetch is done given a hash, which succeeds.
>
> The fetch given a hash was added in fb43e31f2b ("submodule: try harder
> to fetch needed sha1 by direct fetching sha1", 2016-02-24), and the
> "Unable to fetch..." message was downgraded from a fatal error to a
> notice in e30d833671 ("git-submodule.sh: try harder to fetch a
> submodule", 2018-05-16).
>
> This commit improves the notice to be clearer that we are retrying the
> fetch, and that the previous messages do not necessarily indicate that
> the whole command fails. In other words:

"additional" is relative to something.  Is it additional to status-quo?
>
>  - If the HEAD-fetch succeeds and we then have the commit we want, no
>    extra messages are printed.
>  - If the HEAD-fetch succeeds and we do not have the commit we want, but
>    the hash-fetch succeeds, no additional messages are printed.

For example, the user would be helped to be told that we succeeded
doing a pointless fetch and we now are going to make a more specific
"give me that commit" fetch to see if it helps.  These may already
be given with the current code, or may not be (I don't know
offhand), but "no additional messages" does not help me in knowing
if the code with this patch would give such a hint to the users.

>  - If the HEAD-fetch succeeds and we do not have the commit we want, but
>    the hash-fetch fails, this is a fatal error.
>  - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
>    succeeds, no additional messages are printed.
>  - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
>    fails, this is a fatal error.
>
> It could be said that we should just eliminate the HEAD-fetch
> altogether, but that changes some behavior (in particular, some refs
> that were opportunistically updated would no longer be), so I have left
> that alone for now.

OK, that sounds good decision for this patch.

A more fundamental question is why these submodule repositories do
not have HEAD to recommend from which branch to fetch by default,
and if we can do something about it so that HEAD-fetch fails less
often for the users.

> diff --git a/git-submodule.sh b/git-submodule.sh
> index 514ede2596..2c0fb6d723 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -594,7 +594,7 @@ cmd_update()
>  				# is not reachable from a ref.
>  				is_tip_reachable "$sm_path" "$sha1" ||
>  				fetch_in_submodule "$sm_path" $depth ||
> -				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
> +				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'; trying to directly fetch \$sha1:")"
>  
>  				# Now we tried the usual fetch, but $sha1 may
>  				# not be reachable from any of the refs

Obviously good ;-)

Thanks.

> diff --git a/submodule.c b/submodule.c
> index 21cf50ca15..b16c0ecc95 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1548,6 +1548,13 @@ static int fetch_finish(int retvalue, struct strbuf *err,
>  	struct oid_array *commits;
>  
>  	if (retvalue)
> +		/*
> +		 * NEEDSWORK: This indicates that the overall fetch
> +		 * failed, even though there may be a subsequent fetch
> +		 * by commit hash that might work. It may be a good
> +		 * idea to not indicate failure in this case, and only
> +		 * indicate failure if the subsequent fetch fails.
> +		 */
>  		spf->result = 1;
>  
>  	if (!task || !task->sub)

[PATCH v2] submodule: explain first attempt failure clearly

[Jonathan Tan]

When cloning with --recurse-submodules a superproject with at least one
submodule with HEAD pointing to an unborn branch, the clone goes
something like this:

	Cloning into 'test'...
	<messages about cloning of superproject>
	Submodule '<name>' (<uri>) registered for path '<submodule path>'
	Cloning into '<submodule path>'...
	fatal: Couldn't find remote ref HEAD
	Unable to fetch in submodule path '<submodule path>'
	<messages about fetching with SHA-1>
	From <uri>
	 * branch            <hash> -> FETCH_HEAD
	Submodule path '<submodule path>': checked out '<hash>'

In other words, first, a fetch is done with no hash arguments (that is,
a fetch of HEAD) resulting in a "Couldn't find remote ref HEAD" error;
then, a fetch is done given a hash, which succeeds.

The fetch given a hash was added in fb43e31f2b ("submodule: try harder
to fetch needed sha1 by direct fetching sha1", 2016-02-24), and the
"Unable to fetch..." message was downgraded from a fatal error to a
notice in e30d833671 ("git-submodule.sh: try harder to fetch a
submodule", 2018-05-16).

This commit improves the notice to be clearer that we are retrying the
fetch, and that the previous messages (in particular, the fatal errors
from fetch) do not necessarily indicate that the whole command fails. In
other words:

 - If the HEAD-fetch succeeds and we then have the commit we want,
   git-submodule prints no explanation.
 - If the HEAD-fetch succeeds and we do not have the commit we want, but
   the hash-fetch succeeds, git-submodule prints no explanation.
 - If the HEAD-fetch succeeds and we do not have the commit we want, but
   the hash-fetch fails, git-submodule prints a fatal error.
 - If the HEAD-fetch fails, fetch prints a fatal error, and
   git-submodule informs the user that it will retry by fetching
   specific commits by hash.
   - If the hash-fetch then succeeds, git-submodule prints no
     explanation (besides the ones already printed).
   - If the HEAD-fetch then fails, git-submodule prints a fatal error.

It could be said that we should just eliminate the HEAD-fetch
altogether, but that changes some behavior (in particular, some refs
that were opportunistically updated would no longer be), so I have left
that alone for now.

There is an analogous situation with the fetching code in fetch_finish()
and surrounding functions. For now, I have added a NEEDSWORK.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
Change from v1: No code change; commit message change can be seen from
the range-diff below.

> "additional" is relative to something.  Is it additional to status-quo?
> >
> >  - If the HEAD-fetch succeeds and we then have the commit we want, no
> >    extra messages are printed.
> >  - If the HEAD-fetch succeeds and we do not have the commit we want, but
> >    the hash-fetch succeeds, no additional messages are printed.
> 
> For example, the user would be helped to be told that we succeeded
> doing a pointless fetch and we now are going to make a more specific
> "give me that commit" fetch to see if it helps.  These may already
> be given with the current code, or may not be (I don't know
> offhand), but "no additional messages" does not help me in knowing
> if the code with this patch would give such a hint to the users.

It is additional to what fetch prints. To make it clearer, I have
removed all mentions of "additional" from the commit message. But right
now I'm not sure if that whole section is important (since what happens
can be deduced quite easily by reading the fewer than 10 lines of code).

(Also, to answer your question, we currently don't tell the user that we
succeeded in a pointless fetch and will make another one.)

> A more fundamental question is why these submodule repositories do
> not have HEAD to recommend from which branch to fetch by default,
> and if we can do something about it so that HEAD-fetch fails less
> often for the users.

If you mean a recommendation to the repository administrator, I guess we
can recommend that their HEAD points to something, even if it's just a
single orphan commit with a .txt file explaining what's going on.

Range-diff against v1:
1:  1e3fd22f33 ! 1:  d7399fc35e submodule: explain first attempt failure clearly
    @@ -28,19 +28,22 @@
         submodule", 2018-05-16).
     
         This commit improves the notice to be clearer that we are retrying the
    -    fetch, and that the previous messages do not necessarily indicate that
    -    the whole command fails. In other words:
    +    fetch, and that the previous messages (in particular, the fatal errors
    +    from fetch) do not necessarily indicate that the whole command fails. In
    +    other words:
     
    -     - If the HEAD-fetch succeeds and we then have the commit we want, no
    -       extra messages are printed.
    +     - If the HEAD-fetch succeeds and we then have the commit we want,
    +       git-submodule prints no explanation.
          - If the HEAD-fetch succeeds and we do not have the commit we want, but
    -       the hash-fetch succeeds, no additional messages are printed.
    +       the hash-fetch succeeds, git-submodule prints no explanation.
          - If the HEAD-fetch succeeds and we do not have the commit we want, but
    -       the hash-fetch fails, this is a fatal error.
    -     - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
    -       succeeds, no additional messages are printed.
    -     - If the HEAD-fetch fails, we print the notice, and if the hash-fetch
    -       fails, this is a fatal error.
    +       the hash-fetch fails, git-submodule prints a fatal error.
    +     - If the HEAD-fetch fails, fetch prints a fatal error, and
    +       git-submodule informs the user that it will retry by fetching
    +       specific commits by hash.
    +       - If the hash-fetch then succeeds, git-submodule prints no
    +         explanation (besides the ones already printed).
    +       - If the HEAD-fetch then fails, git-submodule prints a fatal error.
     
         It could be said that we should just eliminate the HEAD-fetch
         altogether, but that changes some behavior (in particular, some refs

 git-submodule.sh | 2 +-
 submodule.c      | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 514ede2596..2c0fb6d723 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -594,7 +594,7 @@ cmd_update()
 				# is not reachable from a ref.
 				is_tip_reachable "$sm_path" "$sha1" ||
 				fetch_in_submodule "$sm_path" $depth ||
-				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
+				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'; trying to directly fetch \$sha1:")"
 
 				# Now we tried the usual fetch, but $sha1 may
 				# not be reachable from any of the refs
diff --git a/submodule.c b/submodule.c
index 21cf50ca15..b16c0ecc95 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1548,6 +1548,13 @@ static int fetch_finish(int retvalue, struct strbuf *err,
 	struct oid_array *commits;
 
 	if (retvalue)
+		/*
+		 * NEEDSWORK: This indicates that the overall fetch
+		 * failed, even though there may be a subsequent fetch
+		 * by commit hash that might work. It may be a good
+		 * idea to not indicate failure in this case, and only
+		 * indicate failure if the subsequent fetch fails.
+		 */
 		spf->result = 1;
 
 	if (!task || !task->sub)
-- 
2.21.0.155.ge902e9bcae.dirty

Re: [PATCH v2] submodule: explain first attempt failure clearly

[Junio C Hamano]

Jonathan Tan <jonathantanmy@google.com> writes:

> It is additional to what fetch prints. To make it clearer, I have
> removed all mentions of "additional" from the commit message. But right
> now I'm not sure if that whole section is important (since what happens
> can be deduced quite easily by reading the fewer than 10 lines of code).

Hmph, perhaps.  It was lot easier to follow, at least to me, though ;-)

> If you mean a recommendation to the repository administrator, I guess we
> can recommend that their HEAD points to something, even if it's just a
> single orphan commit with a .txt file explaining what's going on.

Yeah, that might make sense, too.