[PATCH 0/4] recursive submodules: git-reset!

[PATCH 0/4] recursive submodules: git-reset!

[Stefan Beller]

Now that the BIG one has landed, e394fa01d6 (Merge branch
'sb/checkout-recurse-submodules', 2017-03-28), you would expect that
teaching to recurse into submodules is easy for all the remaining 
working tree manipulations?

It turns out it is. See the last patch how we teach git-reset to recurse
into submodules.

However when thinking more about what git-reset is expected to do,
I added tests and some fixes for them (patch 2+3).

patch 1 is a correctness thing, required for patch 3.

Thanks,
Stefan

Stefan Beller (4):
  entry.c: submodule recursing: respect force flag correctly
  submodule.c: uninitialized submodules are ignored in recursive
    commands
  submodule.c: harden submodule_move_head against broken submodules
  builtin/reset: add --recurse-submodules switch

 builtin/reset.c            | 30 ++++++++++++++++++++++++++++++
 entry.c                    |  8 ++++----
 submodule.c                | 20 ++++++++++++++++++++
 t/lib-submodule-update.sh  | 24 +++++++++++++++++++++---
 t/t7112-reset-submodule.sh |  8 ++++++++
 unpack-trees.c             |  7 ++++++-
 6 files changed, 89 insertions(+), 8 deletions(-)

-- 
2.12.2.603.g7b28dc31ba

[PATCH 1/4] entry.c: submodule recursing: respect force flag correctly

[Stefan Beller]

In case of a non-forced worktree update, the submodule movement is tested
in a dry run first, such that it doesn't matter if the actual update is
done via the force flag. However for correctness, we want to give the
flag is specified by the user. All callers have been inspected and updated
if needed.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 entry.c        | 8 ++++----
 unpack-trees.c | 7 ++++++-
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/entry.c b/entry.c
index d2b512da90..d6b263f78e 100644
--- a/entry.c
+++ b/entry.c
@@ -208,7 +208,8 @@ static int write_entry(struct cache_entry *ce,
 		sub = submodule_from_ce(ce);
 		if (sub)
 			return submodule_move_head(ce->name,
-				NULL, oid_to_hex(&ce->oid), SUBMODULE_MOVE_HEAD_FORCE);
+				NULL, oid_to_hex(&ce->oid),
+				state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
 		break;
 	default:
 		return error("unknown file mode for %s in index", path);
@@ -282,12 +283,11 @@ int checkout_entry(struct cache_entry *ce,
 					unlink_or_warn(ce->name);
 
 				return submodule_move_head(ce->name,
-					NULL, oid_to_hex(&ce->oid),
-					SUBMODULE_MOVE_HEAD_FORCE);
+					NULL, oid_to_hex(&ce->oid), 0);
 			} else
 				return submodule_move_head(ce->name,
 					"HEAD", oid_to_hex(&ce->oid),
-					SUBMODULE_MOVE_HEAD_FORCE);
+					state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
 		}
 
 		if (!changed)
diff --git a/unpack-trees.c b/unpack-trees.c
index 8333da2cc9..7316ca99c2 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -252,14 +252,18 @@ static int check_submodule_move_head(const struct cache_entry *ce,
 				     const char *new_id,
 				     struct unpack_trees_options *o)
 {
+	unsigned flags = SUBMODULE_MOVE_HEAD_DRY_RUN;
 	const struct submodule *sub = submodule_from_ce(ce);
 	if (!sub)
 		return 0;
 
+	if (o->reset)
+		flags |= SUBMODULE_MOVE_HEAD_FORCE;
+
 	switch (sub->update_strategy.type) {
 	case SM_UPDATE_UNSPECIFIED:
 	case SM_UPDATE_CHECKOUT:
-		if (submodule_move_head(ce->name, old_id, new_id, SUBMODULE_MOVE_HEAD_DRY_RUN))
+		if (submodule_move_head(ce->name, old_id, new_id, flags))
 			return o->gently ? -1 :
 				add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
 		return 0;
@@ -308,6 +312,7 @@ static void unlink_entry(const struct cache_entry *ce)
 		case SM_UPDATE_CHECKOUT:
 		case SM_UPDATE_REBASE:
 		case SM_UPDATE_MERGE:
+			/* state.force is set at the caller. */
 			submodule_move_head(ce->name, "HEAD", NULL,
 					    SUBMODULE_MOVE_HEAD_FORCE);
 			break;
-- 
2.12.2.603.g7b28dc31ba

[PATCH 2/4] submodule.c: uninitialized submodules are ignored in recursive commands

[Stefan Beller]

This was an oversight when working on the working tree modifying commands
recursing into submodules.

To test for uninitialized submodules, introduce another submodule, that is
uninitialized in the actual tests. By adding it to the branch "add_sub1",
which is the starting point of all other branches, we have wide coverage.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c               | 3 +++
 t/lib-submodule-update.sh | 1 +
 2 files changed, 4 insertions(+)

diff --git a/submodule.c b/submodule.c
index c52d6634c5..2fa42519a4 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1332,6 +1332,9 @@ int submodule_move_head(const char *path,
 	struct child_process cp = CHILD_PROCESS_INIT;
 	const struct submodule *sub;
 
+	if (!is_submodule_initialized(path))
+		return 0;
+
 	sub = submodule_from_path(null_sha1, path);
 
 	if (!sub)
diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
index fb4f7b014e..22dd9e060c 100755
--- a/t/lib-submodule-update.sh
+++ b/t/lib-submodule-update.sh
@@ -73,6 +73,7 @@ create_lib_submodule_repo () {
 
 		git checkout -b "add_sub1" &&
 		git submodule add ../submodule_update_sub1 sub1 &&
+		git submodule add ../submodule_update_sub1 uninitialized_sub &&
 		git config -f .gitmodules submodule.sub1.ignore all &&
 		git config submodule.sub1.ignore all &&
 		git add .gitmodules &&
-- 
2.12.2.603.g7b28dc31ba

[PATCH 3/4] submodule.c: harden submodule_move_head against broken submodules

[Stefan Beller]

Early on in submodule_move_head just after the check if the submodule is
initialized, we need to check if the submodule is populated correctly.

If the submodule is initialized but doesn't look like populated, this
is a red flag and can indicate multiple sorts of failures:
(1) The submodule may be recorded at an object name, that is missing.
(2) The submodule '.git' file link may be broken and it is not pointing
    at a repository.

In both cases we want to complain to the user in the non-forced mode,
and in the forced mode ignoring the old state and just moving the
submodule into its new state with a fixed '.git' file link.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c               | 17 +++++++++++++++++
 t/lib-submodule-update.sh | 23 ++++++++++++++++++++---
 2 files changed, 37 insertions(+), 3 deletions(-)

diff --git a/submodule.c b/submodule.c
index 2fa42519a4..dda1ead854 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1331,10 +1331,19 @@ int submodule_move_head(const char *path,
 	int ret = 0;
 	struct child_process cp = CHILD_PROCESS_INIT;
 	const struct submodule *sub;
+	int *errorcode, error_code;
 
 	if (!is_submodule_initialized(path))
 		return 0;
 
+	if (flags & SUBMODULE_MOVE_HEAD_FORCE)
+		errorcode = &error_code;
+	else
+		errorcode = NULL;
+
+	if (old && !is_submodule_populated_gently(path, errorcode))
+		return 0;
+
 	sub = submodule_from_path(null_sha1, path);
 
 	if (!sub)
@@ -1361,6 +1370,14 @@ int submodule_move_head(const char *path,
 			/* make sure the index is clean as well */
 			submodule_reset_index(path);
 		}
+
+		if (old && (flags & SUBMODULE_MOVE_HEAD_FORCE)) {
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addf(&sb, "%s/modules/%s",
+				    get_git_common_dir(), sub->name);
+			connect_work_tree_and_git_dir(path, sb.buf);
+			strbuf_release(&sb);
+		}
 	}
 
 	prepare_submodule_repo_env_no_git_dir(&cp.env_array);
diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
index 22dd9e060c..f0b1b18206 100755
--- a/t/lib-submodule-update.sh
+++ b/t/lib-submodule-update.sh
@@ -1213,14 +1213,31 @@ test_submodule_forced_switch_recursing () {
 		)
 	'
 	# Updating a submodule from an invalid sha1 updates
-	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
+	test_expect_success "$command: modified submodule does update submodule work tree from invalid commit" '
 		prolog &&
 		reset_work_tree_to_interested invalid_sub1 &&
 		(
 			cd submodule_update &&
 			git branch -t valid_sub1 origin/valid_sub1 &&
-			test_must_fail $command valid_sub1 &&
-			test_superproject_content origin/invalid_sub1
+			$command valid_sub1 &&
+			test_superproject_content origin/valid_sub1 &&
+			test_submodule_content sub1 origin/valid_sub1
+		)
+	'
+
+	# Old versions of Git were buggy writing the .git link file
+	# (e.g. before f8eaa0ba98b and then moving the superproject repo
+	# whose submodules contained absolute paths)
+	test_expect_success "$command: updating submodules fixes .git links" '
+		prolog &&
+		reset_work_tree_to_interested add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t modify_sub1 origin/modify_sub1 &&
+			echo "gitdir: bogus/path" >sub1/.git &&
+			$command modify_sub1 &&
+			test_superproject_content origin/modify_sub1 &&
+			test_submodule_content sub1 origin/modify_sub1
 		)
 	'
 }
-- 
2.12.2.603.g7b28dc31ba

[PATCH 4/4] builtin/reset: add --recurse-submodules switch

[Stefan Beller]

git-reset is yet another working tree manipulator, which should
be taught about submodules.

One use case of "git-reset" is to reset to a known good state,
and dropping commits that did not work as expected.
In that case one of the expected outcomes from a hard reset
would be to have broken submodules reset to a known good
state as well.  A test for this was added in a prior patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/reset.c            | 30 ++++++++++++++++++++++++++++++
 t/t7112-reset-submodule.sh |  8 ++++++++
 2 files changed, 38 insertions(+)

diff --git a/builtin/reset.c b/builtin/reset.c
index fc3b906c47..5ce27fcaed 100644
--- a/builtin/reset.c
+++ b/builtin/reset.c
@@ -21,6 +21,27 @@
 #include "parse-options.h"
 #include "unpack-trees.h"
 #include "cache-tree.h"
+#include "submodule.h"
+#include "submodule-config.h"
+
+static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
+
+static int option_parse_recurse_submodules(const struct option *opt,
+					   const char *arg, int unset)
+{
+	if (unset) {
+		recurse_submodules = RECURSE_SUBMODULES_OFF;
+		return 0;
+	}
+	if (arg)
+		recurse_submodules =
+			parse_update_recurse_submodules_arg(opt->long_name,
+							    arg);
+	else
+		recurse_submodules = RECURSE_SUBMODULES_ON;
+
+	return 0;
+}
 
 static const char * const git_reset_usage[] = {
 	N_("git reset [--mixed | --soft | --hard | --merge | --keep] [-q] [<commit>]"),
@@ -283,6 +304,9 @@ int cmd_reset(int argc, const char **argv, const char *prefix)
 				N_("reset HEAD, index and working tree"), MERGE),
 		OPT_SET_INT(0, "keep", &reset_type,
 				N_("reset HEAD but keep local changes"), KEEP),
+		{ OPTION_CALLBACK, 0, "recurse-submodules", &recurse_submodules,
+			    "reset", "control recursive updating of submodules",
+			    PARSE_OPT_OPTARG, option_parse_recurse_submodules },
 		OPT_BOOL('p', "patch", &patch_mode, N_("select hunks interactively")),
 		OPT_BOOL('N', "intent-to-add", &intent_to_add,
 				N_("record only the fact that removed paths will be added later")),
@@ -295,6 +319,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)
 						PARSE_OPT_KEEP_DASHDASH);
 	parse_args(&pathspec, argv, prefix, patch_mode, &rev);
 
+	if (recurse_submodules != RECURSE_SUBMODULES_DEFAULT) {
+		gitmodules_config();
+		git_config(submodule_config, NULL);
+		set_config_update_recurse_submodules(RECURSE_SUBMODULES_ON);
+	}
+
 	unborn = !strcmp(rev, "HEAD") && get_sha1("HEAD", oid.hash);
 	if (unborn) {
 		/* reset on unborn branch: treat as reset to empty tree */
diff --git a/t/t7112-reset-submodule.sh b/t/t7112-reset-submodule.sh
index 2eda6adeb1..f86ccdf215 100755
--- a/t/t7112-reset-submodule.sh
+++ b/t/t7112-reset-submodule.sh
@@ -5,6 +5,14 @@ test_description='reset can handle submodules'
 . ./test-lib.sh
 . "$TEST_DIRECTORY"/lib-submodule-update.sh
 
+KNOWN_FAILURE_SUBMODULE_RECURSIVE_NESTED=1
+KNOWN_FAILURE_DIRECTORY_SUBMODULE_CONFLICTS=1
+KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1
+
+test_submodule_switch_recursing "git reset --recurse-submodules --keep"
+
+test_submodule_forced_switch_recursing "git reset --hard --recurse-submodules"
+
 test_submodule_switch "git reset --keep"
 
 test_submodule_switch "git reset --merge"
-- 
2.12.2.603.g7b28dc31ba

Re: [PATCH 1/4] entry.c: submodule recursing: respect force flag correctly

[Philip Oakley]

From: "Stefan Beller" <sbeller@google.com>
> In case of a non-forced worktree update, the submodule movement is tested
> in a dry run first, such that it doesn't matter if the actual update is
> done via the force flag. However for correctness, we want to give the
> flag is specified by the user. All callers have been inspected and updated
s/flag is/flag as/

> if needed.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
--
Philip


> entry.c        | 8 ++++----
> unpack-trees.c | 7 ++++++-
> 2 files changed, 10 insertions(+), 5 deletions(-)
>
> diff --git a/entry.c b/entry.c
> index d2b512da90..d6b263f78e 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -208,7 +208,8 @@ static int write_entry(struct cache_entry *ce,
>  sub = submodule_from_ce(ce);
>  if (sub)
>  return submodule_move_head(ce->name,
> - NULL, oid_to_hex(&ce->oid), SUBMODULE_MOVE_HEAD_FORCE);
> + NULL, oid_to_hex(&ce->oid),
> + state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
>  break;
>  default:
>  return error("unknown file mode for %s in index", path);
> @@ -282,12 +283,11 @@ int checkout_entry(struct cache_entry *ce,
>  unlink_or_warn(ce->name);
>
>  return submodule_move_head(ce->name,
> - NULL, oid_to_hex(&ce->oid),
> - SUBMODULE_MOVE_HEAD_FORCE);
> + NULL, oid_to_hex(&ce->oid), 0);
>  } else
>  return submodule_move_head(ce->name,
>  "HEAD", oid_to_hex(&ce->oid),
> - SUBMODULE_MOVE_HEAD_FORCE);
> + state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
>  }
>
>  if (!changed)
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 8333da2cc9..7316ca99c2 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -252,14 +252,18 @@ static int check_submodule_move_head(const struct 
> cache_entry *ce,
>       const char *new_id,
>       struct unpack_trees_options *o)
> {
> + unsigned flags = SUBMODULE_MOVE_HEAD_DRY_RUN;
>  const struct submodule *sub = submodule_from_ce(ce);
>  if (!sub)
>  return 0;
>
> + if (o->reset)
> + flags |= SUBMODULE_MOVE_HEAD_FORCE;
> +
>  switch (sub->update_strategy.type) {
>  case SM_UPDATE_UNSPECIFIED:
>  case SM_UPDATE_CHECKOUT:
> - if (submodule_move_head(ce->name, old_id, new_id, 
> SUBMODULE_MOVE_HEAD_DRY_RUN))
> + if (submodule_move_head(ce->name, old_id, new_id, flags))
>  return o->gently ? -1 :
>  add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
>  return 0;
> @@ -308,6 +312,7 @@ static void unlink_entry(const struct cache_entry *ce)
>  case SM_UPDATE_CHECKOUT:
>  case SM_UPDATE_REBASE:
>  case SM_UPDATE_MERGE:
> + /* state.force is set at the caller. */
>  submodule_move_head(ce->name, "HEAD", NULL,
>      SUBMODULE_MOVE_HEAD_FORCE);
>  break;
> -- 
> 2.12.2.603.g7b28dc31ba
>
> 

Re: [PATCH 3/4] submodule.c: harden submodule_move_head against broken submodules

[Philip Oakley]

From: "Stefan Beller" <sbeller@google.com>
> Early on in submodule_move_head just after the check if the submodule is
> initialized, we need to check if the submodule is populated correctly.
>
> If the submodule is initialized but doesn't look like populated, this
s/like//   or     s/like/like it is/

> is a red flag and can indicate multiple sorts of failures:
> (1) The submodule may be recorded at an object name, that is missing.
> (2) The submodule '.git' file link may be broken and it is not pointing
>    at a repository.
>
> In both cases we want to complain to the user in the non-forced mode,
> and in the forced mode ignoring the old state and just moving the
> submodule into its new state with a fixed '.git' file link.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
--
Philip

> submodule.c               | 17 +++++++++++++++++
> t/lib-submodule-update.sh | 23 ++++++++++++++++++++---
> 2 files changed, 37 insertions(+), 3 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 2fa42519a4..dda1ead854 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1331,10 +1331,19 @@ int submodule_move_head(const char *path,
>  int ret = 0;
>  struct child_process cp = CHILD_PROCESS_INIT;
>  const struct submodule *sub;
> + int *errorcode, error_code;
>
>  if (!is_submodule_initialized(path))
>  return 0;
>
> + if (flags & SUBMODULE_MOVE_HEAD_FORCE)
> + errorcode = &error_code;
> + else
> + errorcode = NULL;
> +
> + if (old && !is_submodule_populated_gently(path, errorcode))
> + return 0;
> +
>  sub = submodule_from_path(null_sha1, path);
>
>  if (!sub)
> @@ -1361,6 +1370,14 @@ int submodule_move_head(const char *path,
>  /* make sure the index is clean as well */
>  submodule_reset_index(path);
>  }
> +
> + if (old && (flags & SUBMODULE_MOVE_HEAD_FORCE)) {
> + struct strbuf sb = STRBUF_INIT;
> + strbuf_addf(&sb, "%s/modules/%s",
> +     get_git_common_dir(), sub->name);
> + connect_work_tree_and_git_dir(path, sb.buf);
> + strbuf_release(&sb);
> + }
>  }
>
>  prepare_submodule_repo_env_no_git_dir(&cp.env_array);
> diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
> index 22dd9e060c..f0b1b18206 100755
> --- a/t/lib-submodule-update.sh
> +++ b/t/lib-submodule-update.sh
> @@ -1213,14 +1213,31 @@ test_submodule_forced_switch_recursing () {
>  )
>  '
>  # Updating a submodule from an invalid sha1 updates
> - test_expect_success "$command: modified submodule does not update 
> submodule work tree from invalid commit" '
> + test_expect_success "$command: modified submodule does update submodule 
> work tree from invalid commit" '
>  prolog &&
>  reset_work_tree_to_interested invalid_sub1 &&
>  (
>  cd submodule_update &&
>  git branch -t valid_sub1 origin/valid_sub1 &&
> - test_must_fail $command valid_sub1 &&
> - test_superproject_content origin/invalid_sub1
> + $command valid_sub1 &&
> + test_superproject_content origin/valid_sub1 &&
> + test_submodule_content sub1 origin/valid_sub1
> + )
> + '
> +
> + # Old versions of Git were buggy writing the .git link file
> + # (e.g. before f8eaa0ba98b and then moving the superproject repo
> + # whose submodules contained absolute paths)
> + test_expect_success "$command: updating submodules fixes .git links" '
> + prolog &&
> + reset_work_tree_to_interested add_sub1 &&
> + (
> + cd submodule_update &&
> + git branch -t modify_sub1 origin/modify_sub1 &&
> + echo "gitdir: bogus/path" >sub1/.git &&
> + $command modify_sub1 &&
> + test_superproject_content origin/modify_sub1 &&
> + test_submodule_content sub1 origin/modify_sub1
>  )
>  '
> }
> -- 
> 2.12.2.603.g7b28dc31ba
>
> 

Re: [PATCH 2/4] submodule.c: uninitialized submodules are ignored in recursive commands

[Brandon Williams]

On 04/11, Stefan Beller wrote:
> This was an oversight when working on the working tree modifying commands
> recursing into submodules.
> 
> To test for uninitialized submodules, introduce another submodule, that is
> uninitialized in the actual tests. By adding it to the branch "add_sub1",
> which is the starting point of all other branches, we have wide coverage.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  submodule.c               | 3 +++
>  t/lib-submodule-update.sh | 1 +
>  2 files changed, 4 insertions(+)
> 
> diff --git a/submodule.c b/submodule.c
> index c52d6634c5..2fa42519a4 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1332,6 +1332,9 @@ int submodule_move_head(const char *path,
>  	struct child_process cp = CHILD_PROCESS_INIT;
>  	const struct submodule *sub;
>  
> +	if (!is_submodule_initialized(path))
> +		return 0;
> +
>  	sub = submodule_from_path(null_sha1, path);
>  
>  	if (!sub)
> diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
> index fb4f7b014e..22dd9e060c 100755
> --- a/t/lib-submodule-update.sh
> +++ b/t/lib-submodule-update.sh
> @@ -73,6 +73,7 @@ create_lib_submodule_repo () {
>  
>  		git checkout -b "add_sub1" &&
>  		git submodule add ../submodule_update_sub1 sub1 &&
> +		git submodule add ../submodule_update_sub1 uninitialized_sub &&

The 'submodule add' command will make the submodule active, so you'll
need to add in a line to subsequently make the submodule inactive for
this to work, unless you do in at a later point in time.

>  		git config -f .gitmodules submodule.sub1.ignore all &&
>  		git config submodule.sub1.ignore all &&
>  		git add .gitmodules &&
> -- 
> 2.12.2.603.g7b28dc31ba
> 

-- 
Brandon Williams

Re: [PATCH 3/4] submodule.c: harden submodule_move_head against broken submodules

[Brandon Williams]

On 04/11, Stefan Beller wrote:
> Early on in submodule_move_head just after the check if the submodule is
> initialized, we need to check if the submodule is populated correctly.
> 
> If the submodule is initialized but doesn't look like populated, this
> is a red flag and can indicate multiple sorts of failures:
> (1) The submodule may be recorded at an object name, that is missing.
> (2) The submodule '.git' file link may be broken and it is not pointing
>     at a repository.
> 
> In both cases we want to complain to the user in the non-forced mode,
> and in the forced mode ignoring the old state and just moving the
> submodule into its new state with a fixed '.git' file link.

What about the case where you have marked a submodule as active but
don't have its respective .gitdir yet?  For now i think it would be
acceptable to complain and do nothing/ignore it, in the future we may
want to actually clone and then check it out.

-- 
Brandon Williams

Re: [PATCH 2/4] submodule.c: uninitialized submodules are ignored in recursive commands

[Stefan Beller]

On Thu, Apr 13, 2017 at 12:05 PM, Brandon Williams <bmwill@google.com> wrote:
> On 04/11, Stefan Beller wrote:
>> This was an oversight when working on the working tree modifying commands
>> recursing into submodules.
>>
>> To test for uninitialized submodules, introduce another submodule, that is
>> uninitialized in the actual tests. By adding it to the branch "add_sub1",
>> which is the starting point of all other branches, we have wide coverage.
>>

...

>
> The 'submodule add' command will make the submodule active, so you'll
> need to add in a line to subsequently make the submodule inactive for
> this to work, unless you do in at a later point in time.

Yes, it will make it active, but that doesn't matter here, because at this
point (in create_lib_submodule_repo) we prepare an upstream
in submodule_update_repo

Any later test follows the structure of

    prolog &&
    reset_work_tree_to no_submodule &&
    (
        cd submodule_update &&
        # do actual test here, in submodule_update
    )

Note that 'prolog' performs a clone of submodule_update_repo
to submodule_update, manually setting 'sub1' to active.

'uninitialized_sub' is not active.

I tried to explain it via
    To test for uninitialized submodules, introduce another submodule,
    that is uninitialized in the actual tests.
in the commit message, but that is too concise apparently.
So the resend will explain that a bit more.

Thanks,
Stefan

Re: [PATCH 2/4] submodule.c: uninitialized submodules are ignored in recursive commands

[Brandon Williams]

On 04/13, Stefan Beller wrote:
> On Thu, Apr 13, 2017 at 12:05 PM, Brandon Williams <bmwill@google.com> wrote:
> > On 04/11, Stefan Beller wrote:
> >> This was an oversight when working on the working tree modifying commands
> >> recursing into submodules.
> >>
> >> To test for uninitialized submodules, introduce another submodule, that is
> >> uninitialized in the actual tests. By adding it to the branch "add_sub1",
> >> which is the starting point of all other branches, we have wide coverage.
> >>
> 
> ...
> 
> >
> > The 'submodule add' command will make the submodule active, so you'll
> > need to add in a line to subsequently make the submodule inactive for
> > this to work, unless you do in at a later point in time.
> 
> Yes, it will make it active, but that doesn't matter here, because at this
> point (in create_lib_submodule_repo) we prepare an upstream
> in submodule_update_repo
> 
> Any later test follows the structure of
> 
>     prolog &&
>     reset_work_tree_to no_submodule &&
>     (
>         cd submodule_update &&
>         # do actual test here, in submodule_update
>     )
> 
> Note that 'prolog' performs a clone of submodule_update_repo
> to submodule_update, manually setting 'sub1' to active.
> 
> 'uninitialized_sub' is not active.
> 
> I tried to explain it via
>     To test for uninitialized submodules, introduce another submodule,
>     that is uninitialized in the actual tests.
> in the commit message, but that is too concise apparently.
> So the resend will explain that a bit more.

Thanks!  I just wanted to be sure as you're more familiar with these
tests.

-- 
Brandon Williams

Re: [PATCH 3/4] submodule.c: harden submodule_move_head against broken submodules

[Stefan Beller]

On Thu, Apr 13, 2017 at 12:08 PM, Brandon Williams <bmwill@google.com> wrote:
> On 04/11, Stefan Beller wrote:
>> Early on in submodule_move_head just after the check if the submodule is
>> initialized, we need to check if the submodule is populated correctly.
>>
>> If the submodule is initialized but doesn't look like populated, this
>> is a red flag and can indicate multiple sorts of failures:
>> (1) The submodule may be recorded at an object name, that is missing.
>> (2) The submodule '.git' file link may be broken and it is not pointing
>>     at a repository.
>>
>> In both cases we want to complain to the user in the non-forced mode,
>> and in the forced mode ignoring the old state and just moving the
>> submodule into its new state with a fixed '.git' file link.
>
> What about the case where you have marked a submodule as active but
> don't have its respective .gitdir yet?  For now i think it would be
> acceptable to complain and do nothing/ignore it, in the future we may
> want to actually clone and then check it out.

I agree. With this patch we'd complain in non-forced mode, and in
forced mode we'd also complain as we lack the object.

In both cases in the future we may want to fetch the contents instead.

Re: [PATCH 1/4] entry.c: submodule recursing: respect force flag correctly

[Jonathan Tan]

On 04/11/2017 04:49 PM, Stefan Beller wrote:
> In case of a non-forced worktree update, the submodule movement is tested
> in a dry run first, such that it doesn't matter if the actual update is
> done via the force flag. However for correctness, we want to give the
> flag is specified by the user. All callers have been inspected and updated
> if needed.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  entry.c        | 8 ++++----
>  unpack-trees.c | 7 ++++++-
>  2 files changed, 10 insertions(+), 5 deletions(-)
>
> diff --git a/entry.c b/entry.c
> index d2b512da90..d6b263f78e 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -208,7 +208,8 @@ static int write_entry(struct cache_entry *ce,
>  		sub = submodule_from_ce(ce);
>  		if (sub)
>  			return submodule_move_head(ce->name,
> -				NULL, oid_to_hex(&ce->oid), SUBMODULE_MOVE_HEAD_FORCE);
> +				NULL, oid_to_hex(&ce->oid),
> +				state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
>  		break;
>  	default:
>  		return error("unknown file mode for %s in index", path);
> @@ -282,12 +283,11 @@ int checkout_entry(struct cache_entry *ce,
>  					unlink_or_warn(ce->name);
>
>  				return submodule_move_head(ce->name,
> -					NULL, oid_to_hex(&ce->oid),
> -					SUBMODULE_MOVE_HEAD_FORCE);
> +					NULL, oid_to_hex(&ce->oid), 0);

Should we be consistent (with the "else" block below and with the 
existing code) to use "state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0" 
instead of 0? (I glanced briefly through the code and 
SUBMODULE_MOVE_HEAD_FORCE might have no effect anyway if "old" is NULL, 
but it's probably still better to be consistent.)

>  			} else
>  				return submodule_move_head(ce->name,
>  					"HEAD", oid_to_hex(&ce->oid),
> -					SUBMODULE_MOVE_HEAD_FORCE);
> +					state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0);
>  		}
>
>  		if (!changed)
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 8333da2cc9..7316ca99c2 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -252,14 +252,18 @@ static int check_submodule_move_head(const struct cache_entry *ce,
>  				     const char *new_id,
>  				     struct unpack_trees_options *o)
>  {
> +	unsigned flags = SUBMODULE_MOVE_HEAD_DRY_RUN;
>  	const struct submodule *sub = submodule_from_ce(ce);
>  	if (!sub)
>  		return 0;
>
> +	if (o->reset)
> +		flags |= SUBMODULE_MOVE_HEAD_FORCE;

It seems to me that this is independent of the entry.c change, and might 
be better in its own patch. (Or if it is not, maybe the subject should 
be "entry, unpack-trees: propagate force when submodule recursing" or 
something like that, containing the names of both modified components.)

Also, you mentioned in the parent message that this patch is required 
for patch 3. Is only the entry.c part required, or unpack-trees.c, or both?

> +
>  	switch (sub->update_strategy.type) {
>  	case SM_UPDATE_UNSPECIFIED:
>  	case SM_UPDATE_CHECKOUT:
> -		if (submodule_move_head(ce->name, old_id, new_id, SUBMODULE_MOVE_HEAD_DRY_RUN))
> +		if (submodule_move_head(ce->name, old_id, new_id, flags))
>  			return o->gently ? -1 :
>  				add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
>  		return 0;
> @@ -308,6 +312,7 @@ static void unlink_entry(const struct cache_entry *ce)
>  		case SM_UPDATE_CHECKOUT:
>  		case SM_UPDATE_REBASE:
>  		case SM_UPDATE_MERGE:
> +			/* state.force is set at the caller. */

I don't understand the relevance of this comment - it is indeed set 
there, but "state" is not used there until after the invocation to 
unlink_entry so it doesn't seem related.

>  			submodule_move_head(ce->name, "HEAD", NULL,
>  					    SUBMODULE_MOVE_HEAD_FORCE);
>  			break;
>

Re: [PATCH 1/4] entry.c: submodule recursing: respect force flag correctly

[Stefan Beller]

On Fri, Apr 14, 2017 at 11:28 AM, Jonathan Tan <jonathantanmy@google.com> wrote:

>> @@ -282,12 +283,11 @@ int checkout_entry(struct cache_entry *ce,
>>                                         unlink_or_warn(ce->name);
>>
>>                                 return submodule_move_head(ce->name,
>> -                                       NULL, oid_to_hex(&ce->oid),
>> -                                       SUBMODULE_MOVE_HEAD_FORCE);
>> +                                       NULL, oid_to_hex(&ce->oid), 0);
>
>
> Should we be consistent (with the "else" block below and with the existing
> code) to use "state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0" instead of 0? (I
> glanced briefly through the code and SUBMODULE_MOVE_HEAD_FORCE might have no
> effect anyway if "old" is NULL, but it's probably still better to be
> consistent.)

ok, will do.

>>
>> +       if (o->reset)
>> +               flags |= SUBMODULE_MOVE_HEAD_FORCE;
>
>
> It seems to me that this is independent of the entry.c change, and might be
> better in its own patch. (Or if it is not, maybe the subject should be
> "entry, unpack-trees: propagate force when submodule recursing" or something
> like that, containing the names of both modified components.)

eh. I realize the patch evolved after writing the commit message initially.
Maybe:

  fix all submodule_move_head force flags

  Audit all callers of  submodule_move_head and make sure the
  force flag is handled correctly.


>
> Also, you mentioned in the parent message that this patch is required for
> patch 3. Is only the entry.c part required, or unpack-trees.c, or both?
>
>> +
>>         switch (sub->update_strategy.type) {
>>         case SM_UPDATE_UNSPECIFIED:
>>         case SM_UPDATE_CHECKOUT:
>> -               if (submodule_move_head(ce->name, old_id, new_id,
>> SUBMODULE_MOVE_HEAD_DRY_RUN))
>> +               if (submodule_move_head(ce->name, old_id, new_id, flags))
>>                         return o->gently ? -1 :
>>                                 add_rejected_path(o,
>> ERROR_WOULD_LOSE_SUBMODULE, ce->name);
>>                 return 0;
>> @@ -308,6 +312,7 @@ static void unlink_entry(const struct cache_entry *ce)
>>                 case SM_UPDATE_CHECKOUT:
>>                 case SM_UPDATE_REBASE:
>>                 case SM_UPDATE_MERGE:
>> +                       /* state.force is set at the caller. */
>
>
> I don't understand the relevance of this comment - it is indeed set there,
> but "state" is not used there until after the invocation to unlink_entry so
> it doesn't seem related.

Well we would have wanted to put
  state->force ? SUBMODULE_MOVE_HEAD_FORCE : 0
here, but state is not passed into this function, so just make a comment
why we keep it at force all the time.

Thanks,
Stefan

Re: [PATCH 3/4] submodule.c: harden submodule_move_head against broken submodules

[Jonathan Tan]

I think "harden" is the wrong word to use in the subject - to me, 
"harden" implies that you're defending against an invalid use. But here, 
not only is the use valid, but this patch also takes steps to support it.

On 04/11/2017 04:49 PM, Stefan Beller wrote:
> Early on in submodule_move_head just after the check if the submodule is
> initialized, we need to check if the submodule is populated correctly.
>
> If the submodule is initialized but doesn't look like populated, this
> is a red flag and can indicate multiple sorts of failures:
> (1) The submodule may be recorded at an object name, that is missing.
> (2) The submodule '.git' file link may be broken and it is not pointing
>     at a repository.
>
> In both cases we want to complain to the user in the non-forced mode,
> and in the forced mode ignoring the old state and just moving the
> submodule into its new state with a fixed '.git' file link.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  submodule.c               | 17 +++++++++++++++++
>  t/lib-submodule-update.sh | 23 ++++++++++++++++++++---
>  2 files changed, 37 insertions(+), 3 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 2fa42519a4..dda1ead854 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1331,10 +1331,19 @@ int submodule_move_head(const char *path,
>  	int ret = 0;
>  	struct child_process cp = CHILD_PROCESS_INIT;
>  	const struct submodule *sub;
> +	int *errorcode, error_code;

Maybe call it error_code_ptr?

>
>  	if (!is_submodule_initialized(path))
>  		return 0;
>
> +	if (flags & SUBMODULE_MOVE_HEAD_FORCE)
> +		errorcode = &error_code;
> +	else
> +		errorcode = NULL;
> +
> +	if (old && !is_submodule_populated_gently(path, errorcode))

This behavior of the null-ness of the pointer controlling whether the 
function dies upon error or not is quite confusing to me, so I would add 
a comment (e.g. under the "errorcode = &error_code" statement, saying 
"Pass a non-null pointer to make is_submodule_populated_gently report 
errors to use instead of die()-ing").

> +		return 0;
> +
>  	sub = submodule_from_path(null_sha1, path);
>
>  	if (!sub)
> @@ -1361,6 +1370,14 @@ int submodule_move_head(const char *path,
>  			/* make sure the index is clean as well */
>  			submodule_reset_index(path);
>  		}
> +
> +		if (old && (flags & SUBMODULE_MOVE_HEAD_FORCE)) {
> +			struct strbuf sb = STRBUF_INIT;
> +			strbuf_addf(&sb, "%s/modules/%s",
> +				    get_git_common_dir(), sub->name);
> +			connect_work_tree_and_git_dir(path, sb.buf);
> +			strbuf_release(&sb);

xstrfmt might be simpler, but I don't have a strong preference either way.

> +		}
>  	}
>
>  	prepare_submodule_repo_env_no_git_dir(&cp.env_array);
> diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
> index 22dd9e060c..f0b1b18206 100755
> --- a/t/lib-submodule-update.sh
> +++ b/t/lib-submodule-update.sh
> @@ -1213,14 +1213,31 @@ test_submodule_forced_switch_recursing () {
>  		)
>  	'
>  	# Updating a submodule from an invalid sha1 updates
> -	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
> +	test_expect_success "$command: modified submodule does update submodule work tree from invalid commit" '
>  		prolog &&
>  		reset_work_tree_to_interested invalid_sub1 &&
>  		(
>  			cd submodule_update &&
>  			git branch -t valid_sub1 origin/valid_sub1 &&
> -			test_must_fail $command valid_sub1 &&
> -			test_superproject_content origin/invalid_sub1
> +			$command valid_sub1 &&
> +			test_superproject_content origin/valid_sub1 &&
> +			test_submodule_content sub1 origin/valid_sub1
> +		)
> +	'
> +
> +	# Old versions of Git were buggy writing the .git link file
> +	# (e.g. before f8eaa0ba98b and then moving the superproject repo
> +	# whose submodules contained absolute paths)
> +	test_expect_success "$command: updating submodules fixes .git links" '
> +		prolog &&
> +		reset_work_tree_to_interested add_sub1 &&
> +		(
> +			cd submodule_update &&
> +			git branch -t modify_sub1 origin/modify_sub1 &&
> +			echo "gitdir: bogus/path" >sub1/.git &&
> +			$command modify_sub1 &&
> +			test_superproject_content origin/modify_sub1 &&
> +			test_submodule_content sub1 origin/modify_sub1
>  		)
>  	'
>  }
>