[RFC PATCHv2 00/17] Checkout aware of Submodules!

[RFC PATCHv2 00/17] Checkout aware of Submodules!

[Stefan Beller]

v2:
* based on top of the series sent out an hour ago
  "[PATCHv4 0/5] submodule embedgitdirs"
* Try to embed a submodule if we need to remove it.
* Strictly do not change behavior if not giving the new flag.
* I think I missed some review comments from v1, but I'd like to get
  the current state out over the weekend, as a lot has changed so far.
  On Monday I'll go through the previous discussion with a comb to see
  if I missed something.
  
v1:
When working with submodules, nearly anytime after checking out
a different state of the projects, that has submodules changed
you'd run "git submodule update" with a current version of Git.

There are two problems with this approach:

* The "submodule update" command is dangerous as it
  doesn't check for work that may be lost in the submodule
  (e.g. a dangling commit).
* you may forget to run the command as checkout is supposed
  to do all the work for you.

Integrate updating the submodules into git checkout, with the same
safety promises that git-checkout has, i.e. not throw away data unless
asked to. This is done by first checking if the submodule is at the same
sha1 as it is recorded in the superproject. If there are changes we stop
proceeding the checkout just like it is when checking out a file that
has local changes.

The integration happens in the code that is also used in other commands
such that it will be easier in the future to make other commands aware
of submodule.

This also solves d/f conflicts in case you replace a file/directory
with a submodule or vice versa.

The patches are still a bit rough, but the overall series seems
promising enough to me that I want to put it out here.

Any review, specifically on the design level welcome!

Thanks,
Stefan


Stefan Beller (17):
  submodule.h: add extern keyword to functions
  submodule: modernize ok_to_remove_submodule to use argv_array
  update submodules: move up prepare_submodule_repo_env
  update submodules: add is_submodule_populated
  update submodules: add submodule config parsing
  update submodules: add a config option to determine if submodules are
    updated
  update submodules: introduce submodule_is_interesting
  update submodules: add depopulate_submodule
  update submodules: add scheduling to update submodules
  update submodules: is_submodule_checkout_safe
  unpack-trees: teach verify_clean_submodule to inspect submodules
  unpack-trees: remove submodule contents if interesting
  entry: write_entry to write populate submodules
  submodule: teach unpack_trees() to update submodules
  checkout: recurse into submodules if asked to
  completion: add '--recurse-submodules' to checkout
  checkout: add config option to recurse into submodules by default

 Documentation/config.txt               |   6 +
 Documentation/git-checkout.txt         |   8 +
 builtin/checkout.c                     |  31 +++-
 cache.h                                |   2 +
 contrib/completion/git-completion.bash |   2 +-
 entry.c                                |  14 +-
 submodule-config.c                     |  22 +++
 submodule-config.h                     |  17 +-
 submodule.c                            | 292 +++++++++++++++++++++++++++++---
 submodule.h                            |  74 ++++++---
 t/t2013-checkout-submodule.sh          | 293 ++++++++++++++++++++++++++++++++-
 t/t9902-completion.sh                  |   1 +
 unpack-trees.c                         |  92 ++++++++---
 unpack-trees.h                         |   1 +
 14 files changed, 768 insertions(+), 87 deletions(-)

-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 01/17] submodule.h: add extern keyword to functions

[Stefan Beller]

As the upcoming series will add a lot of functions to the submodule
header, let's first make the header consistent to the rest of the project
by adding the extern keyword to functions.

As per the CodingGuidelines we try to stay below 80 characters per line,
so adapt all those functions to stay below 80 characters that are already
using more than one line.  Those function using just one line are better
kept in one line than breaking them up into multiple lines just for the
goal of staying below the character limit as it makes grepping
for functions easier if they are one liners.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.h | 57 ++++++++++++++++++++++++++++++---------------------------
 1 file changed, 30 insertions(+), 27 deletions(-)

diff --git a/submodule.h b/submodule.h
index d9e197a948..5db0b53b3f 100644
--- a/submodule.h
+++ b/submodule.h
@@ -29,50 +29,53 @@ struct submodule_update_strategy {
 };
 #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}
 
-int is_staging_gitmodules_ok(void);
-int update_path_in_gitmodules(const char *oldpath, const char *newpath);
-int remove_path_from_gitmodules(const char *path);
-void stage_updated_gitmodules(void);
-void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,
+extern int is_staging_gitmodules_ok(void);
+extern int update_path_in_gitmodules(const char *oldpath, const char *newpath);
+extern int remove_path_from_gitmodules(const char *path);
+extern void stage_updated_gitmodules(void);
+extern void set_diffopt_flags_from_submodule_config(struct diff_options *,
 		const char *path);
-int submodule_config(const char *var, const char *value, void *cb);
-void gitmodules_config(void);
-int parse_submodule_update_strategy(const char *value,
+extern int submodule_config(const char *var, const char *value, void *cb);
+extern void gitmodules_config(void);
+extern int parse_submodule_update_strategy(const char *value,
 		struct submodule_update_strategy *dst);
-const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);
-void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);
-void show_submodule_summary(FILE *f, const char *path,
+extern const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);
+extern void handle_ignore_submodules_arg(struct diff_options *, const char *);
+extern void show_submodule_summary(FILE *f, const char *path,
 		const char *line_prefix,
 		struct object_id *one, struct object_id *two,
 		unsigned dirty_submodule, const char *meta,
 		const char *del, const char *add, const char *reset);
-void show_submodule_inline_diff(FILE *f, const char *path,
+extern void show_submodule_inline_diff(FILE *f, const char *path,
 		const char *line_prefix,
 		struct object_id *one, struct object_id *two,
 		unsigned dirty_submodule, const char *meta,
 		const char *del, const char *add, const char *reset,
 		const struct diff_options *opt);
-void set_config_fetch_recurse_submodules(int value);
-void check_for_new_submodule_commits(unsigned char new_sha1[20]);
-int fetch_populated_submodules(const struct argv_array *options,
+extern void set_config_fetch_recurse_submodules(int value);
+extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
+extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
 			       int quiet, int max_parallel_jobs);
-unsigned is_submodule_modified(const char *path, int ignore_untracked);
-int submodule_uses_gitfile(const char *path);
-int ok_to_remove_submodule(const char *path);
-int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],
-		    const unsigned char a[20], const unsigned char b[20], int search);
-int find_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name,
-		struct string_list *needs_pushing);
-int push_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name);
-void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);
-int parallel_submodules(void);
+extern unsigned is_submodule_modified(const char *path, int ignore_untracked);
+extern int submodule_uses_gitfile(const char *path);
+extern int ok_to_remove_submodule(const char *path);
+extern int merge_submodule(unsigned char result[20], const char *path,
+			   const unsigned char base[20],
+			   const unsigned char a[20],
+			   const unsigned char b[20], int search);
+extern int find_unpushed_submodules(unsigned char new_sha1[20],
+				    const char *remotes_name,
+				    struct string_list *needs_pushing);
+extern int push_unpushed_submodules(unsigned char new_sha1[20],
+				    const char *remotes_name);
+extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);
+extern int parallel_submodules(void);
 
 /*
  * Prepare the "env_array" parameter of a "struct child_process" for executing
  * a submodule by clearing any repo-specific envirionment variables, but
  * retaining any config in the environment.
  */
-void prepare_submodule_repo_env(struct argv_array *out);
-
+extern void prepare_submodule_repo_env(struct argv_array *out);
 #endif
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 02/17] submodule: modernize ok_to_remove_submodule to use argv_array

[Stefan Beller]

Instead of constructing the NULL terminated array ourselves, we
should make use of the argv_array infrastructure.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/submodule.c b/submodule.c
index 66c5ce5a24..eb80b0c5ad 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1022,13 +1022,6 @@ int ok_to_remove_submodule(const char *path)
 {
 	ssize_t len;
 	struct child_process cp = CHILD_PROCESS_INIT;
-	const char *argv[] = {
-		"status",
-		"--porcelain",
-		"-u",
-		"--ignore-submodules=none",
-		NULL,
-	};
 	struct strbuf buf = STRBUF_INIT;
 	int ok_to_remove = 1;
 
@@ -1038,14 +1031,15 @@ int ok_to_remove_submodule(const char *path)
 	if (!submodule_uses_gitfile(path))
 		return 0;
 
-	cp.argv = argv;
+	argv_array_pushl(&cp.args, "status", "--porcelain", "-u",
+				   "--ignore-submodules=none", NULL);
 	prepare_submodule_repo_env(&cp.env_array);
 	cp.git_cmd = 1;
 	cp.no_stdin = 1;
 	cp.out = -1;
 	cp.dir = path;
 	if (start_command(&cp))
-		die("Could not run 'git status --porcelain -uall --ignore-submodules=none' in submodule %s", path);
+		die("Could not run 'git status --porcelain -u --ignore-submodules=none' in submodule %s", path);
 
 	len = strbuf_read(&buf, cp.out, 1024);
 	if (len > 2)
@@ -1053,7 +1047,7 @@ int ok_to_remove_submodule(const char *path)
 	close(cp.out);
 
 	if (finish_command(&cp))
-		die("'git status --porcelain -uall --ignore-submodules=none' failed in submodule %s", path);
+		die("'git status --porcelain -u --ignore-submodules=none' failed in submodule %s", path);
 
 	strbuf_release(&buf);
 	return ok_to_remove;
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 03/17] update submodules: move up prepare_submodule_repo_env

[Stefan Beller]

In a later patch we need to prepare the submodule environment with
another git directory, so split up the function.

Also move it up in the file such that we do not need to declare the
function later before using it.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/submodule.c b/submodule.c
index eb80b0c5ad..78b69b5a55 100644
--- a/submodule.c
+++ b/submodule.c
@@ -307,6 +307,22 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,
 	strbuf_release(&sb);
 }
 
+static void prepare_submodule_repo_env_no_git_dir(struct argv_array *out)
+{
+	const char * const *var;
+
+	for (var = local_repo_env; *var; var++) {
+		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
+			argv_array_push(out, *var);
+	}
+}
+
+void prepare_submodule_repo_env(struct argv_array *out)
+{
+	prepare_submodule_repo_env_no_git_dir(out);
+	argv_array_push(out, "GIT_DIR=.git");
+}
+
 /* Helper function to display the submodule header line prior to the full
  * summary output. If it can locate the submodule objects directory it will
  * attempt to lookup both the left and right commits and put them into the
@@ -1246,14 +1262,3 @@ int parallel_submodules(void)
 {
 	return parallel_jobs;
 }
-
-void prepare_submodule_repo_env(struct argv_array *out)
-{
-	const char * const *var;
-
-	for (var = local_repo_env; *var; var++) {
-		if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))
-			argv_array_push(out, *var);
-	}
-	argv_array_push(out, "GIT_DIR=.git");
-}
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 04/17] update submodules: add is_submodule_populated

[Stefan Beller]

See if a submodule is populated by checking if there
is a .git file or directory at the given path.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 11 +++++++++++
 submodule.h |  1 +
 2 files changed, 12 insertions(+)

diff --git a/submodule.c b/submodule.c
index 78b69b5a55..c29153e9ff 100644
--- a/submodule.c
+++ b/submodule.c
@@ -930,6 +930,17 @@ int fetch_populated_submodules(const struct argv_array *options,
 	return spf.result;
 }
 
+int is_submodule_populated(const char *path)
+{
+	int retval = 0;
+	struct strbuf gitdir = STRBUF_INIT;
+	strbuf_addf(&gitdir, "%s/.git", path);
+	if (resolve_gitdir(gitdir.buf))
+		retval = 1;
+	strbuf_release(&gitdir);
+	return retval;
+}
+
 unsigned is_submodule_modified(const char *path, int ignore_untracked)
 {
 	ssize_t len;
diff --git a/submodule.h b/submodule.h
index 5db0b53b3f..a9eabcc3d0 100644
--- a/submodule.h
+++ b/submodule.h
@@ -58,6 +58,7 @@ extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
 			       int quiet, int max_parallel_jobs);
 extern unsigned is_submodule_modified(const char *path, int ignore_untracked);
+extern int is_submodule_populated(const char *path);
 extern int submodule_uses_gitfile(const char *path);
 extern int ok_to_remove_submodule(const char *path);
 extern int merge_submodule(unsigned char result[20], const char *path,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 05/17] update submodules: add submodule config parsing

[Stefan Beller]

Similar to b33a15b08 (push: add recurseSubmodules config option,
2015-11-17) and 027771fcb1 (submodule: allow erroneous values for the
fetchRecurseSubmodules option, 2015-08-17), we add submodule-config code
that is later used to parse whether we are interested in updating
submodules.

We need the `die_on_error` parameter to be able to call this parsing
function for the config file as well, which if incorrect lets Git die.

As we're just touching the header file, also mark all functions extern.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule-config.c | 22 ++++++++++++++++++++++
 submodule-config.h | 17 +++++++++--------
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/submodule-config.c b/submodule-config.c
index 098085be69..4b5297e31d 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -234,6 +234,28 @@ int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)
 	return parse_fetch_recurse(opt, arg, 1);
 }
 
+static int parse_update_recurse(const char *opt, const char *arg,
+				int die_on_error)
+{
+	switch (git_config_maybe_bool(opt, arg)) {
+	case 1:
+		return RECURSE_SUBMODULES_ON;
+	case 0:
+		return RECURSE_SUBMODULES_OFF;
+	default:
+		if (!strcmp(arg, "checkout"))
+			return RECURSE_SUBMODULES_ON;
+		if (die_on_error)
+			die("bad %s argument: %s", opt, arg);
+		return RECURSE_SUBMODULES_ERROR;
+	}
+}
+
+int parse_update_recurse_submodules_arg(const char *opt, const char *arg)
+{
+	return parse_update_recurse(opt, arg, 1);
+}
+
 static int parse_push_recurse(const char *opt, const char *arg,
 			       int die_on_error)
 {
diff --git a/submodule-config.h b/submodule-config.h
index d05c542d2c..992bfbebc0 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -22,13 +22,14 @@ struct submodule {
 	int recommend_shallow;
 };
 
-int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);
-int parse_push_recurse_submodules_arg(const char *opt, const char *arg);
-int parse_submodule_config_option(const char *var, const char *value);
-const struct submodule *submodule_from_name(const unsigned char *commit_sha1,
-		const char *name);
-const struct submodule *submodule_from_path(const unsigned char *commit_sha1,
-		const char *path);
-void submodule_free(void);
+extern int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);
+extern int parse_update_recurse_submodules_arg(const char *opt, const char *arg);
+extern int parse_push_recurse_submodules_arg(const char *opt, const char *arg);
+extern int parse_submodule_config_option(const char *var, const char *value);
+extern const struct submodule *submodule_from_name(
+		const unsigned char *commit_sha1, const char *name);
+extern const struct submodule *submodule_from_path(
+		const unsigned char *commit_sha1, const char *path);
+extern void submodule_free(void);
 
 #endif /* SUBMODULE_CONFIG_H */
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 06/17] update submodules: add a config option to determine if submodules are updated

[Stefan Beller]

In later patches we introduce the options and flag for commands
that modify the working directory, e.g. git-checkout.

Have a central place to store such settings whether we want to update
a submodule.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 6 ++++++
 submodule.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/submodule.c b/submodule.c
index c29153e9ff..1ba398ba3b 100644
--- a/submodule.c
+++ b/submodule.c
@@ -16,6 +16,7 @@
 #include "quote.h"
 
 static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;
+static int config_update_recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
 static int parallel_jobs = 1;
 static struct string_list changed_submodule_paths = STRING_LIST_INIT_NODUP;
 static int initialized_fetch_ref_tips;
@@ -510,6 +511,11 @@ void set_config_fetch_recurse_submodules(int value)
 	config_fetch_recurse_submodules = value;
 }
 
+void set_config_update_recurse_submodules(int value)
+{
+	config_update_recurse_submodules = value;
+}
+
 static int has_remote(const char *refname, const struct object_id *oid,
 		      int flags, void *cb_data)
 {
diff --git a/submodule.h b/submodule.h
index a9eabcc3d0..21236b095c 100644
--- a/submodule.h
+++ b/submodule.h
@@ -53,6 +53,7 @@ extern void show_submodule_inline_diff(FILE *f, const char *path,
 		const char *del, const char *add, const char *reset,
 		const struct diff_options *opt);
 extern void set_config_fetch_recurse_submodules(int value);
+extern void set_config_update_recurse_submodules(int value);
 extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 07/17] update submodules: introduce submodule_is_interesting

[Stefan Beller]

In later patches we introduce the --recurse-submodule flag for commands
that modify the working directory, e.g. git-checkout.

It is potentially expensive to check if a submodule needs an update,
because a common theme to interact with submodules is to spawn a child
process for each interaction.

So let's introduce a function that pre checks if a submodule needs
to be checked for an update.

I am not particular happy with the name `submodule_is_interesting`,
in internal iterations I had `submodule_requires_check_for_update`
and `submodule_needs_update`, but I was even less happy with those
names. Maybe `submodule_interesting_for_update`?

Generally this is to answer "Am I allowed to touch the submodule
at all?" or: "Does the user expect me to touch it?"
which includes all of creation/deletion/update.

This patch is based off a prior attempt by Jens Lehmann to add
submodules to checkout.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 23 +++++++++++++++++++++++
 submodule.h |  9 +++++++++
 2 files changed, 32 insertions(+)

diff --git a/submodule.c b/submodule.c
index 1ba398ba3b..62e9ef3872 100644
--- a/submodule.c
+++ b/submodule.c
@@ -516,6 +516,29 @@ void set_config_update_recurse_submodules(int value)
 	config_update_recurse_submodules = value;
 }
 
+int submodules_interesting_for_update(void)
+{
+	/*
+	 * Update can't be "none", "merge" or "rebase",
+	 * treat any value as OFF, except an explicit ON.
+	 */
+	return config_update_recurse_submodules == RECURSE_SUBMODULES_ON;
+}
+
+int submodule_is_interesting(const char *path)
+{
+	const struct submodule *sub;
+
+	if (!submodules_interesting_for_update())
+		return 0;
+
+	sub = submodule_from_path(null_sha1, path);
+	if (!sub)
+		return 0;
+
+	return sub->update_strategy.type != SM_UPDATE_NONE;
+}
+
 static int has_remote(const char *refname, const struct object_id *oid,
 		      int flags, void *cb_data)
 {
diff --git a/submodule.h b/submodule.h
index 21236b095c..7d890e0464 100644
--- a/submodule.h
+++ b/submodule.h
@@ -54,6 +54,15 @@ extern void show_submodule_inline_diff(FILE *f, const char *path,
 		const struct diff_options *opt);
 extern void set_config_fetch_recurse_submodules(int value);
 extern void set_config_update_recurse_submodules(int value);
+
+/**
+ * When updating the working tree, we need to check if the submodule needs
+ * updating. We do not require a check if we are already sure that the
+ * submodule doesn't need updating, e.g. when we are not interested in submodules
+ * or the submodule is marked uninteresting by being not initialized.
+ */
+extern int submodule_is_interesting(const char *path);
+extern int submodules_interesting_for_update(void);
 extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 08/17] update submodules: add depopulate_submodule

[Stefan Beller]

Implement the functionality needed to enable work tree manipulating
commands so that a deleted submodule should not only affect the index
(leaving all the files of the submodule in the work tree) but also to
remove the work tree of the superproject (including any untracked
files).

To do so, we need an equivalent of "rm -rf", which is already found in
entry.c, so expose that and for clarity add a suffix "_or_dir" to it.

That will only work properly when the submodule uses a gitfile instead of
a .git directory and no untracked files are present. Otherwise the removal
will fail with a warning (which is just what happened until now).

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h     |  2 ++
 entry.c     |  5 +++++
 submodule.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 submodule.h |  1 +
 4 files changed, 54 insertions(+)

diff --git a/cache.h b/cache.h
index a50a61a197..b645ca2f9a 100644
--- a/cache.h
+++ b/cache.h
@@ -2018,4 +2018,6 @@ void sleep_millisec(int millisec);
  */
 void safe_create_dir(const char *dir, int share);
 
+extern void remove_directory_or_die(struct strbuf *path);
+
 #endif /* CACHE_H */
diff --git a/entry.c b/entry.c
index c6eea240b6..02c4ac9f22 100644
--- a/entry.c
+++ b/entry.c
@@ -73,6 +73,11 @@ static void remove_subtree(struct strbuf *path)
 		die_errno("cannot rmdir '%s'", path->buf);
 }
 
+void remove_directory_or_die(struct strbuf *path)
+{
+	remove_subtree(path);
+}
+
 static int create_file(const char *path, unsigned int mode)
 {
 	mode = (mode & 0100) ? 0777 : 0666;
diff --git a/submodule.c b/submodule.c
index 62e9ef3872..7bb64d6c69 100644
--- a/submodule.c
+++ b/submodule.c
@@ -324,6 +324,52 @@ void prepare_submodule_repo_env(struct argv_array *out)
 	argv_array_push(out, "GIT_DIR=.git");
 }
 
+int depopulate_submodule(const char *path)
+{
+	int ret = 0;
+	struct strbuf pathbuf = STRBUF_INIT;
+	char *dot_git = xstrfmt("%s/.git", path);
+
+	/* Is it populated? */
+	if (!resolve_gitdir(dot_git))
+		goto out;
+
+	/* Does it have a .git directory? */
+	if (!submodule_uses_gitfile(path)) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+
+		prepare_submodule_repo_env(&cp.env_array);
+		argv_array_pushl(&cp.args, "submodule--helper",
+				 "embed-git-dirs", path, NULL);
+		cp.git_cmd = 1;
+		if (run_command(&cp)) {
+			warning(_("Cannot remove submodule '%s'\n"
+				  "because it (or one of its nested submodules) has a git \n"
+				  "directory in the working tree, which could not be embedded\n"
+				  "the superprojects git directory automatically."), path);
+			ret = -1;
+			goto out;
+		}
+
+		if (!submodule_uses_gitfile(path)) {
+			/*
+			 * We should be using a gitfile by now, let's double
+			 * check as loosing the git dir would be fatal.
+			 */
+			die("BUG: \"git submodule--helper embed git-dirs '%s'\" "
+			    "did not embed the git-dirs recursively for '%s'",
+			    path, path);
+		}
+	}
+
+	strbuf_addstr(&pathbuf, path);
+	remove_directory_or_die(&pathbuf);
+out:
+	strbuf_release(&pathbuf);
+	free(dot_git);
+	return ret;
+}
+
 /* Helper function to display the submodule header line prior to the full
  * summary output. If it can locate the submodule objects directory it will
  * attempt to lookup both the left and right commits and put them into the
diff --git a/submodule.h b/submodule.h
index 7d890e0464..d8bb1d4baf 100644
--- a/submodule.h
+++ b/submodule.h
@@ -63,6 +63,7 @@ extern void set_config_update_recurse_submodules(int value);
  */
 extern int submodule_is_interesting(const char *path);
 extern int submodules_interesting_for_update(void);
+extern int depopulate_submodule(const char *path);
 extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 09/17] update submodules: add scheduling to update submodules

[Stefan Beller]

The walker of a tree is only expected to call `schedule_submodule_for_update`
and once done, to run `update_submodules`. This avoids directory/file
conflicts and later we can parallelize all submodule actions if needed.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 submodule.h |   5 +++
 2 files changed, 137 insertions(+)

diff --git a/submodule.c b/submodule.c
index 7bb64d6c69..02c28ef56b 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1348,3 +1348,135 @@ int parallel_submodules(void)
 {
 	return parallel_jobs;
 }
+
+static struct scheduled_submodules_update_type {
+	const char *path;
+	const struct object_id *oid;
+	/*
+	 * Do we need to perform a complete checkout or just incremental
+	 * update?
+	 */
+	unsigned is_new:1;
+} *scheduled_submodules;
+#define SCHEDULED_SUBMODULES_INIT {NULL, NULL, 0}
+
+static int scheduled_submodules_nr, scheduled_submodules_alloc;
+
+void schedule_submodule_for_update(const struct cache_entry *ce, int is_new)
+{
+	struct scheduled_submodules_update_type *ssu;
+	ALLOC_GROW(scheduled_submodules,
+		   scheduled_submodules_nr + 1,
+		   scheduled_submodules_alloc);
+	ssu = &scheduled_submodules[scheduled_submodules_nr++];
+	ssu->path = ce->name;
+	ssu->oid = &ce->oid;
+	ssu->is_new = !!is_new;
+}
+
+static int update_submodule(const char *path, const struct object_id *oid,
+			    int force, int is_new)
+{
+	const char *git_dir;
+	struct child_process cp = CHILD_PROCESS_INIT;
+	const struct submodule *sub = submodule_from_path(null_sha1, path);
+
+	if (!sub || !sub->name)
+		return -1;
+
+	git_dir = resolve_gitdir(git_common_path("modules/%s", sub->name));
+
+	if (!git_dir)
+		return -1;
+
+	if (is_new)
+		connect_work_tree_and_git_dir(path, git_dir);
+
+	/* update index via `read-tree --reset sha1` */
+	argv_array_pushl(&cp.args, "read-tree",
+				   force ? "--reset" : "-m",
+				   "-u", sha1_to_hex(oid->hash), NULL);
+	prepare_submodule_repo_env(&cp.env_array);
+	cp.git_cmd = 1;
+	cp.no_stdin = 1;
+	cp.dir = path;
+	if (run_command(&cp)) {
+		warning(_("reading the index in submodule '%s' failed"), path);
+		child_process_clear(&cp);
+		return -1;
+	}
+
+	/* write index to working dir */
+	child_process_clear(&cp);
+	child_process_init(&cp);
+	argv_array_pushl(&cp.args, "checkout-index", "-a", NULL);
+	cp.git_cmd = 1;
+	cp.no_stdin = 1;
+	cp.dir = path;
+	if (force)
+		argv_array_push(&cp.args, "-f");
+
+	if (run_command(&cp)) {
+		warning(_("populating the working directory in submodule '%s' failed"), path);
+		child_process_clear(&cp);
+		return -1;
+	}
+
+	/* get the HEAD right */
+	child_process_clear(&cp);
+	child_process_init(&cp);
+	argv_array_pushl(&cp.args, "checkout", "--recurse-submodules", NULL);
+	cp.git_cmd = 1;
+	cp.no_stdin = 1;
+	cp.dir = path;
+	if (force)
+		argv_array_push(&cp.args, "-f");
+	argv_array_push(&cp.args, sha1_to_hex(oid->hash));
+
+	if (run_command(&cp)) {
+		warning(_("setting the HEAD in submodule '%s' failed"), path);
+		child_process_clear(&cp);
+		return -1;
+	}
+
+	child_process_clear(&cp);
+	return 0;
+}
+
+int update_submodules(int force)
+{
+	int i;
+	gitmodules_config();
+
+	/*
+	 * NEEDSWORK: As submodule updates can potentially take some
+	 * time each and they do not overlap (i.e. no d/f conflicts;
+	 * this can be parallelized using the run_commands.h API.
+	 */
+	for (i = 0; i < scheduled_submodules_nr; i++) {
+		struct submodule *sub;
+		struct scheduled_submodules_update_type *ssu =
+			&scheduled_submodules[i];
+
+		if (!submodule_is_interesting(ssu->path))
+			continue;
+
+		sub = submodule_from_path(null_sha1, ssu->path);
+
+		switch (sub->update_strategy) {
+		case SM_UPDATE_UNSPECIFIED: /* fall thru */
+		case SM_UPDATE_CHECKOUT:
+			update_submodule(ssu->path, ssu->oid,
+					 force, ssu->is_new);
+			break;
+		case SM_UPDATE_REBASE:
+		case SM_UPDATE_MERGE:
+		case SM_UPDATE_NONE:
+		case SM_UPDATE_COMMAND:
+			warning("update strategy for submodule '%s' not supported", ssu->path);
+		}
+	}
+
+	scheduled_submodules_nr = 0;
+	return 0;
+}
diff --git a/submodule.h b/submodule.h
index d8bb1d4baf..74df8b93d5 100644
--- a/submodule.h
+++ b/submodule.h
@@ -90,4 +90,9 @@ extern int parallel_submodules(void);
  * retaining any config in the environment.
  */
 extern void prepare_submodule_repo_env(struct argv_array *out);
+
+extern void schedule_submodule_for_update(const struct cache_entry *ce,
+					  int new);
+extern int update_submodules(int force);
+
 #endif
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 10/17] update submodules: is_submodule_checkout_safe

[Stefan Beller]

In later patches we introduce the options and flag for commands
that modify the working directory, e.g. git-checkout.

This piece of code will answer the question:
"Is it safe to change the submodule to this new state?"
e.g. is it overwriting untracked files or are there local
changes that would be overwritten?

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 37 +++++++++++++++++++++++++++++++++++++
 submodule.h |  2 ++
 2 files changed, 39 insertions(+)

diff --git a/submodule.c b/submodule.c
index 02c28ef56b..4253f7f044 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1155,6 +1155,43 @@ int ok_to_remove_submodule(const char *path)
 	return ok_to_remove;
 }
 
+/**
+ * Check if a submodule update to a given sha1 is safe.
+ * Return 1 if it is safe, 0 when it is not.
+ *
+ * If the submodule is not populated, we need to check
+ */
+int is_submodule_checkout_safe(const char *path,
+			       const struct object_id *new_hash)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+
+	argv_array_pushl(&cp.args, "read-tree", "-n", "-m", "HEAD",
+			sha1_to_hex(new_hash->hash), NULL);
+
+	if (!is_submodule_populated(path)) {
+		const struct submodule *sub;
+
+		/* See if we have the submodule configured already: */
+		sub = submodule_from_path(null_sha1, path);
+
+		prepare_submodule_repo_env_no_git_dir(&cp.env_array);
+		argv_array_pushf(&cp.env_array, "GIT_DIR=%s",
+				 sub ? sub->name : path);
+		argv_array_pushf(&cp.env_array, "GIT_WORK_TREE=%s", path);
+	} else {
+		prepare_submodule_repo_env(&cp.env_array);
+	}
+
+	cp.git_cmd = 1;
+	cp.no_stdin = 1;
+	cp.no_stdout = 1;
+	cp.no_stderr = 1;
+	cp.dir = path;
+
+	return run_command(&cp) == 0;
+}
+
 static int find_first_merges(struct object_array *result, const char *path,
 		struct commit *a, struct commit *b)
 {
diff --git a/submodule.h b/submodule.h
index 74df8b93d5..1dfcd6939b 100644
--- a/submodule.h
+++ b/submodule.h
@@ -72,6 +72,8 @@ extern unsigned is_submodule_modified(const char *path, int ignore_untracked);
 extern int is_submodule_populated(const char *path);
 extern int submodule_uses_gitfile(const char *path);
 extern int ok_to_remove_submodule(const char *path);
+extern int is_submodule_checkout_safe(const char *path,
+				      const struct object_id *new_hash);
 extern int merge_submodule(unsigned char result[20], const char *path,
 			   const unsigned char base[20],
 			   const unsigned char a[20],
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 11/17] unpack-trees: teach verify_clean_submodule to inspect submodules

[Stefan Beller]

As a later patch will modify submodules, if they are interesting
we need to see if the submodule is clean in case they are
interesting.

If they are not interesting, then we do not care about the submodule
keeping historic behavior.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 unpack-trees.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index ea6bdd20e0..22e32eca96 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -9,6 +9,7 @@
 #include "refs.h"
 #include "attr.h"
 #include "split-index.h"
+#include "submodule.h"
 #include "dir.h"
 
 /*
@@ -1361,15 +1362,15 @@ static void invalidate_ce_path(const struct cache_entry *ce,
 /*
  * Check that checking out ce->sha1 in subdir ce->name is not
  * going to overwrite any working files.
- *
- * Currently, git does not checkout subprojects during a superproject
- * checkout, so it is not going to overwrite anything.
  */
 static int verify_clean_submodule(const struct cache_entry *ce,
 				  enum unpack_trees_error_types error_type,
 				  struct unpack_trees_options *o)
 {
-	return 0;
+	if (!submodule_is_interesting(ce->name))
+		return 0;
+
+	return !is_submodule_modified(ce->name, 0);
 }
 
 static int verify_clean_subdirectory(const struct cache_entry *ce,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 12/17] unpack-trees: remove submodule contents if interesting

[Stefan Beller]

Currently when a unlink_entry() is called on a submodule, this fails
as remove_or_warn detects it needs to delete a directory via rmdir.
However rmdir only works on empty directories, such that the
"_or_warn" part kicks in, and we get a warning message.

In case the submodule is of no interest we're not going to delete the
submodule, so a warning like that is useful.

If the submodule is interesting then we need to depopulate it properly,
there is no need to react on its return code the depopulation method
handles proper warnings nor do we need to schedule the directory for
removal later, such that we can return early.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 unpack-trees.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 22e32eca96..db03293347 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -214,7 +214,12 @@ static void unlink_entry(const struct cache_entry *ce)
 {
 	if (!check_leading_path(ce->name, ce_namelen(ce)))
 		return;
-	if (remove_or_warn(ce->ce_mode, ce->name))
+
+	if (S_ISGITLINK(ce->ce_mode) &&
+	    submodule_is_interesting(ce->name)) {
+		depopulate_submodule(ce->name);
+		return;
+	} else if (remove_or_warn(ce->ce_mode, ce->name))
 		return;
 	schedule_dir_for_removal(ce->name, ce_namelen(ce));
 }
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 13/17] entry: write_entry to write populate submodules

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 entry.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/entry.c b/entry.c
index 02c4ac9f22..a668025b8e 100644
--- a/entry.c
+++ b/entry.c
@@ -2,6 +2,7 @@
 #include "blob.h"
 #include "dir.h"
 #include "streaming.h"
+#include "submodule.h"
 
 static void create_directories(const char *path, int path_len,
 			       const struct checkout *state)
@@ -208,6 +209,7 @@ static int write_entry(struct cache_entry *ce,
 			return error("cannot create temporary submodule %s", path);
 		if (mkdir(path, 0777) < 0)
 			return error("cannot create submodule directory %s", path);
+		schedule_submodule_for_update(ce, 1);
 		break;
 	default:
 		return error("unknown file mode for %s in index", path);
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 14/17] submodule: teach unpack_trees() to update submodules

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 entry.c        |  7 +++---
 unpack-trees.c | 76 +++++++++++++++++++++++++++++++++++++++++++++-------------
 unpack-trees.h |  1 +
 3 files changed, 65 insertions(+), 19 deletions(-)

diff --git a/entry.c b/entry.c
index a668025b8e..3ed885b886 100644
--- a/entry.c
+++ b/entry.c
@@ -267,7 +267,7 @@ int checkout_entry(struct cache_entry *ce,
 
 	if (!check_path(path.buf, path.len, &st, state->base_dir_len)) {
 		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
-		if (!changed)
+		if (!changed && (!S_ISDIR(st.st_mode) || !S_ISGITLINK(ce->ce_mode)))
 			return 0;
 		if (!state->force) {
 			if (!state->quiet)
@@ -284,9 +284,10 @@ int checkout_entry(struct cache_entry *ce,
 		 * just do the right thing)
 		 */
 		if (S_ISDIR(st.st_mode)) {
-			/* If it is a gitlink, leave it alone! */
-			if (S_ISGITLINK(ce->ce_mode))
+			if (S_ISGITLINK(ce->ce_mode)) {
+				schedule_submodule_for_update(ce, 1);
 				return 0;
+			}
 			if (!state->force)
 				return error("%s is a directory", path.buf);
 			remove_subtree(&path);
diff --git a/unpack-trees.c b/unpack-trees.c
index db03293347..8b0f6dfd1a 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -29,6 +29,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {
 	/* ERROR_NOT_UPTODATE_DIR */
 	"Updating '%s' would lose untracked files in it",
 
+	/* ERROR_NOT_UPTODATE_SUBMODULE */
+	"Updating submodule '%s' would lose modifications in it",
+
 	/* ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN */
 	"Untracked working tree file '%s' would be overwritten by merge.",
 
@@ -81,6 +84,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,
 	msgs[ERROR_NOT_UPTODATE_DIR] =
 		_("Updating the following directories would lose untracked files in it:\n%s");
 
+	msgs[ERROR_NOT_UPTODATE_SUBMODULE] =
+		_("Updating the following submodules would lose modifications in them:\n%s");
+
 	if (!strcmp(cmd, "checkout"))
 		msg = advice_commit_before_merge
 		      ? _("The following untracked working tree files would be removed by checkout:\n%%s"
@@ -1320,19 +1326,17 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
-		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;
-		unsigned changed = ie_match_stat(o->src_index, ce, &st, flags);
-		if (!changed)
-			return 0;
-		/*
-		 * NEEDSWORK: the current default policy is to allow
-		 * submodule to be out of sync wrt the superproject
-		 * index.  This needs to be tightened later for
-		 * submodules that are marked to be automatically
-		 * checked out.
-		 */
-		if (S_ISGITLINK(ce->ce_mode))
-			return 0;
+		if (S_ISGITLINK(ce->ce_mode)) {
+			if (!submodule_is_interesting(ce->name))
+				return 0;
+			if (ce_stage(ce) ? is_submodule_checkout_safe(ce->name, &ce->oid)
+			    : !is_submodule_modified(ce->name, 1))
+				return 0;
+		} else {
+			int flags = CE_MATCH_IGNORE_VALID | CE_MATCH_IGNORE_SKIP_WORKTREE;
+			if (!ie_match_stat(o->src_index, ce, &st, flags))
+				return 0;
+		}
 		errno = 0;
 	}
 	if (errno == ENOENT)
@@ -1355,6 +1359,38 @@ static int verify_uptodate_sparse(const struct cache_entry *ce,
 	return verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE);
 }
 
+/*
+ * When a submodule gets turned into an unmerged entry, we want it to be
+ * up-to-date regarding the merge changes.
+ */
+static int verify_uptodate_submodule(const struct cache_entry *old,
+				     const struct cache_entry *new,
+				     struct unpack_trees_options *o)
+{
+	struct stat st;
+
+	if (o->index_only ||
+	    (!((old->ce_flags & CE_VALID) || ce_skip_worktree(old)) &&
+	      (o->reset || ce_uptodate(old))))
+		return 0;
+
+	if (!submodule_is_interesting(new->name))
+		return 0;
+
+	if (lstat(old->name, &st)) {
+		if (errno == ENOENT)
+			return 0;
+		return o->gently ? -1 :
+			add_rejected_path(o, ERROR_NOT_UPTODATE_SUBMODULE,
+					  old->name);
+	}
+
+	if (S_ISGITLINK(new->ce_mode))
+		return !is_submodule_checkout_safe(new->name, &new->oid);
+	else
+		return !ok_to_remove_submodule(old->name);
+}
+
 static void invalidate_ce_path(const struct cache_entry *ce,
 			       struct unpack_trees_options *o)
 {
@@ -1616,9 +1652,17 @@ static int merged_entry(const struct cache_entry *ce,
 			copy_cache_entry(merge, old);
 			update = 0;
 		} else {
-			if (verify_uptodate(old, o)) {
-				free(merge);
-				return -1;
+			if (S_ISGITLINK(old->ce_mode) ||
+			    S_ISGITLINK(merge->ce_mode)) {
+				if (verify_uptodate_submodule(old, merge, o)) {
+					free(merge);
+					return -1;
+				}
+			} else {
+				if (verify_uptodate(old, o)) {
+					free(merge);
+					return -1;
+				}
 			}
 			/* Migrate old flags over */
 			update |= old->ce_flags & (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);
diff --git a/unpack-trees.h b/unpack-trees.h
index 36a73a6d00..bee874088a 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -15,6 +15,7 @@ enum unpack_trees_error_types {
 	ERROR_WOULD_OVERWRITE = 0,
 	ERROR_NOT_UPTODATE_FILE,
 	ERROR_NOT_UPTODATE_DIR,
+	ERROR_NOT_UPTODATE_SUBMODULE,
 	ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN,
 	ERROR_WOULD_LOSE_UNTRACKED_REMOVED,
 	ERROR_BIND_OVERLAP,
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 15/17] checkout: recurse into submodules if asked to

[Stefan Beller]

Allow checkout to recurse into submodules via
the command line option --[no-]recurse-submodules.

The flag for recurse-submodules in its current form
could be an OPT_BOOL, but eventually we may want to have
it as:

    git checkout --recurse-submodules=rebase|merge| \
			cherry-pick|checkout|none

which resembles the submodule.<name>.update options,
so naturally a value such as
"as-configured-or-X-as-fallback" would also come in handy.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-checkout.txt |   7 ++
 builtin/checkout.c             |  31 ++++-
 t/t2013-checkout-submodule.sh  | 277 +++++++++++++++++++++++++++++++++++++++--
 3 files changed, 306 insertions(+), 9 deletions(-)

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 8e2c0662dd..a0ea2c5651 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -256,6 +256,13 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
 	out anyway. In other words, the ref can be held by more than one
 	worktree.
 
+--[no-]recurse-submodules::
+	Using --recurse-submodules will update the content of all initialized
+	submodules according to the commit recorded in the superproject. If
+	local modifications in a submodule would be overwritten the checkout
+	will fail until `-f` is used. If nothing (or --no-recurse-submodules)
+	is used, the work trees of submodules will not be updated.
+
 <branch>::
 	Branch to checkout; if it refers to a branch (i.e., a name that,
 	when prepended with "refs/heads/", is a valid ref), then that
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 512492aad9..5db0d933d1 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -21,12 +21,31 @@
 #include "submodule-config.h"
 #include "submodule.h"
 
+static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
+
 static const char * const checkout_usage[] = {
 	N_("git checkout [<options>] <branch>"),
 	N_("git checkout [<options>] [<branch>] -- <file>..."),
 	NULL,
 };
 
+int option_parse_recurse_submodules(const struct option *opt,
+				    const char *arg, int unset)
+{
+	if (unset) {
+		recurse_submodules = RECURSE_SUBMODULES_OFF;
+		return 0;
+	}
+	if (arg)
+		recurse_submodules =
+			parse_update_recurse_submodules_arg(opt->long_name,
+							    arg);
+	else
+		recurse_submodules = RECURSE_SUBMODULES_ON;
+
+	return 0;
+}
+
 struct checkout_opts {
 	int patch_mode;
 	int quiet;
@@ -826,7 +845,8 @@ static int switch_branches(const struct checkout_opts *opts,
 		parse_commit_or_die(new->commit);
 	}
 
-	ret = merge_working_tree(opts, &old, new, &writeout_error);
+	ret = merge_working_tree(opts, &old, new, &writeout_error) ||
+	      update_submodules(opts->force);
 	if (ret) {
 		free(path_to_free);
 		return ret;
@@ -1160,6 +1180,9 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 				N_("second guess 'git checkout <no-such-branch>'")),
 		OPT_BOOL(0, "ignore-other-worktrees", &opts.ignore_other_worktrees,
 			 N_("do not check if another worktree is holding the given ref")),
+		{ OPTION_CALLBACK, 0, "recurse-submodules", &recurse_submodules,
+			    "checkout", "control recursive updating of submodules",
+			    PARSE_OPT_OPTARG, option_parse_recurse_submodules },
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
 		OPT_END(),
 	};
@@ -1190,6 +1213,12 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 		git_xmerge_config("merge.conflictstyle", conflict_style, NULL);
 	}
 
+	if (recurse_submodules != RECURSE_SUBMODULES_OFF) {
+		git_config(submodule_config, NULL);
+		if (recurse_submodules != RECURSE_SUBMODULES_DEFAULT)
+			set_config_update_recurse_submodules(recurse_submodules);
+	}
+
 	if ((!!opts.new_branch + !!opts.new_branch_force + !!opts.new_orphan_branch) > 1)
 		die(_("-b, -B and --orphan are mutually exclusive"));
 
diff --git a/t/t2013-checkout-submodule.sh b/t/t2013-checkout-submodule.sh
index 6847f75822..33fb2f5de3 100755
--- a/t/t2013-checkout-submodule.sh
+++ b/t/t2013-checkout-submodule.sh
@@ -7,15 +7,21 @@ test_description='checkout can handle submodules'
 
 test_expect_success 'setup' '
 	mkdir submodule &&
-	(cd submodule &&
-	 git init &&
-	 test_commit first) &&
-	git add submodule &&
+	(
+		cd submodule &&
+		git init &&
+		test_commit first
+	) &&
+	echo first >file &&
+	git add file submodule &&
 	test_tick &&
 	git commit -m superproject &&
-	(cd submodule &&
-	 test_commit second) &&
-	git add submodule &&
+	(
+		cd submodule &&
+		test_commit second
+	) &&
+	echo second > file &&
+	git add file submodule &&
 	test_tick &&
 	git commit -m updated.superproject
 '
@@ -37,7 +43,8 @@ test_expect_success '"checkout <submodule>" updates the index only' '
 	git checkout HEAD^ submodule &&
 	test_must_fail git diff-files --quiet &&
 	git checkout HEAD submodule &&
-	git diff-files --quiet
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD
 '
 
 test_expect_success '"checkout <submodule>" honors diff.ignoreSubmodules' '
@@ -63,6 +70,260 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
 	! test -s actual
 '
 
+submodule_creation_must_succeed() {
+	# checkout base ($1)
+	git checkout -f --recurse-submodules $1 &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached $1 &&
+
+	# checkout target ($2)
+	if test -d submodule; then
+		echo change >>submodule/first.t &&
+		test_must_fail git checkout --recurse-submodules $2 &&
+		git checkout -f --recurse-submodules $2
+	else
+		git checkout --recurse-submodules $2
+	fi &&
+	test -e submodule/.git &&
+	test -f submodule/first.t &&
+	test -f submodule/second.t &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached $2
+}
+
+test_expect_success 'setup the submodule config' '
+	git config -f .gitmodules submodule.submodule.path submodule &&
+	git config -f .gitmodules submodule.submodule.url ./submodule.bare &&
+	git -C submodule clone --bare . ../submodule.bare &&
+	echo submodule.bare >>.gitignore &&
+	git add .gitignore .gitmodules submodule &&
+	git commit -m "submodule registered with a gitmodules file" &&
+	git config submodule.submodule.url ./submodule.bare
+'
+
+test_expect_success '"checkout --recurse-submodules" migrates submodule git dir before deleting' '
+	git checkout -b base &&
+	git checkout -b delete_submodule &&
+	git update-index --force-remove submodule &&
+	git config -f .gitmodules --unset submodule.submodule.path &&
+	git config -f .gitmodules --unset submodule.submodule.url &&
+	git add .gitmodules &&
+	git commit -m "submodule deleted" &&
+	git checkout base &&
+	git checkout --recurse-submodules delete_submodule 2>output.err 1>output.out &&
+	test_i18ngrep "Migrating git directory" output.out &&
+	! test -d submodule
+'
+
+test_expect_success '"check --recurse-submodules" removes deleted submodule' '
+	# Make sure we have the submodule here and ready.
+	git checkout base &&
+	git submodule embedgitdirs &&
+	git submodule update -f . &&
+	test -e submodule/.git &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached base &&
+
+	# Check if the checkout deletes the submodule.
+	echo change >>submodule/first.t &&
+	test_must_fail git checkout --recurse-submodules delete_submodule &&
+	git checkout -f --recurse-submodules delete_submodule &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached delete_submodule &&
+	! test -d submodule
+'
+
+test_expect_success '"checkout --recurse-submodules" repopulates submodule' '
+	submodule_creation_must_succeed delete_submodule base
+'
+
+test_expect_success '"checkout --recurse-submodules" repopulates submodule in existing directory' '
+	git checkout --recurse-submodules delete_submodule &&
+	mkdir submodule &&
+	submodule_creation_must_succeed delete_submodule base
+'
+
+test_expect_success '"checkout --recurse-submodules" replaces submodule with files' '
+	git checkout -f base &&
+	git checkout -b replace_submodule_with_file &&
+	git rm -f submodule &&
+	echo "file instead" >submodule &&
+	git add submodule &&
+	git commit -m "submodule replaced" &&
+	git checkout -f base &&
+	git submodule update -f . &&
+	git checkout --recurse-submodules replace_submodule_with_file &&
+	test -f submodule
+'
+
+test_expect_success '"checkout --recurse-submodules" removes files and repopulates submodule' '
+	submodule_creation_must_succeed replace_submodule_with_file base
+'
+
+test_expect_success '"checkout --recurse-submodules" replaces submodule with a directory' '
+	git checkout -f base &&
+	git checkout -b replace_submodule_with_dir &&
+	git rm -f submodule &&
+	mkdir -p submodule/dir &&
+	echo content >submodule/dir/file &&
+	git add submodule &&
+	git commit -m "submodule replaced with a directory (file inside)" &&
+	git checkout -f base &&
+	git submodule update -f . &&
+	git checkout --recurse-submodules replace_submodule_with_dir &&
+	test -d submodule &&
+	! test -e submodule/.git &&
+	! test -f submodule/first.t &&
+	! test -f submodule/second.t &&
+	test -d submodule/dir
+'
+
+test_expect_success '"checkout --recurse-submodules" removes the directory and repopulates submodule' '
+	submodule_creation_must_succeed replace_submodule_with_dir base
+'
+
+test_expect_success SYMLINKS '"checkout --recurse-submodules" replaces submodule with a link' '
+	git checkout -f base &&
+	git checkout -b replace_submodule_with_link &&
+	git rm -f submodule &&
+	ln -s submodule &&
+	git add submodule &&
+	git commit -m "submodule replaced with a link" &&
+	git checkout -f base &&
+	git submodule update -f . &&
+	git checkout --recurse-submodules replace_submodule_with_link &&
+	test -L submodule
+'
+
+test_expect_success SYMLINKS '"checkout --recurse-submodules" removes the link and repopulates submodule' '
+	submodule_creation_must_succeed replace_submodule_with_link base
+'
+
+test_expect_success '"checkout --recurse-submodules" updates the submodule' '
+	git checkout --recurse-submodules base &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD &&
+	git checkout -b updated_submodule &&
+	(
+		cd submodule &&
+		echo x >>first.t &&
+		git add first.t &&
+		test_commit third
+	) &&
+	git add submodule &&
+	test_tick &&
+	git commit -m updated.superproject &&
+	git checkout --recurse-submodules base &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD
+'
+
+# In 293ab15eea34 we considered untracked ignored files in submodules
+# expendable, we may want to revisit this decision by adding user as
+# well as command specific configuration for it.
+# When building in-tree the untracked ignored files are probably ok to remove
+# in a case as tested here, but e.g. when git.git is a submodule, then a user
+# may not want to lose a well crafted (but ignored by default) "config.mak"
+# Commands like "git rm" may care less about untracked files in a submodule
+# as the checkout command that removes a submodule as well.
+test_expect_failure 'untracked file is not deleted' '
+	git checkout --recurse-submodules base &&
+	echo important >submodule/untracked &&
+	test_must_fail git checkout --recurse-submodules delete_submodule &&
+	git checkout -f --recurse-submodules delete_submodule
+'
+
+test_expect_success 'ignored file works just fine' '
+	git checkout --recurse-submodules base &&
+	echo important >submodule/ignored &&
+	echo ignored >.git/modules/submodule/info/exclude &&
+	git checkout --recurse-submodules delete_submodule
+'
+
+test_expect_success 'dirty file file is not deleted' '
+	git checkout --recurse-submodules base &&
+	echo important >submodule/first.t &&
+	test_must_fail git checkout --recurse-submodules delete_submodule &&
+	git checkout -f --recurse-submodules delete_submodule
+'
+
+test_expect_success 'added to index is not deleted' '
+	git checkout --recurse-submodules base &&
+	echo important >submodule/to_index &&
+	git -C submodule add to_index &&
+	test_must_fail git checkout --recurse-submodules delete_submodule &&
+	git checkout -f --recurse-submodules delete_submodule
+'
+
+# This is ok in theory, we just need to make sure
+# the garbage collection doesn't eat the commit.
+test_expect_success 'different commit prevents from deleting' '
+	git checkout --recurse-submodules base &&
+	echo important >submodule/to_index &&
+	git -C submodule add to_index &&
+	test_must_fail git checkout --recurse-submodules delete_submodule &&
+	git checkout -f --recurse-submodules delete_submodule
+'
+
+test_expect_failure '"checkout --recurse-submodules" needs -f to update a modifed submodule commit' '
+	git -C submodule checkout --recurse-submodules HEAD^ &&
+	test_must_fail git checkout --recurse-submodules master &&
+	test_must_fail git diff-files --quiet submodule &&
+	git diff-files --quiet file &&
+	git checkout --recurse-submodules -f master &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD
+'
+
+test_expect_failure '"checkout --recurse-submodules" needs -f to update modifed submodule content' '
+	echo modified >submodule/second.t &&
+	test_must_fail git checkout --recurse-submodules HEAD^ &&
+	test_must_fail git diff-files --quiet submodule &&
+	git diff-files --quiet file &&
+	git checkout --recurse-submodules -f HEAD^ &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD &&
+	git checkout --recurse-submodules -f master &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD
+'
+
+test_expect_failure '"checkout --recurse-submodules" ignores modified submodule content that would not be changed' '
+	echo modified >expected &&
+	cp expected submodule/first.t &&
+	git checkout --recurse-submodules HEAD^ &&
+	test_cmp expected submodule/first.t &&
+	test_must_fail git diff-files --quiet submodule &&
+	git diff-index --quiet --cached HEAD &&
+	git checkout --recurse-submodules -f master &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached HEAD
+'
+
+test_expect_failure '"checkout --recurse-submodules" does not care about untracked submodule content' '
+	echo untracked >submodule/untracked &&
+	git checkout --recurse-submodules master &&
+	git diff-files --quiet --ignore-submodules=untracked &&
+	git diff-index --quiet --cached HEAD &&
+	rm submodule/untracked
+'
+
+test_expect_failure '"checkout --recurse-submodules" needs -f when submodule commit is not present (but does fail anyway)' '
+	git checkout --recurse-submodules -b bogus_commit master &&
+	git update-index --cacheinfo 160000 0123456789012345678901234567890123456789 submodule &&
+	BOGUS_TREE=$(git write-tree) &&
+	BOGUS_COMMIT=$(echo "bogus submodule commit" | git commit-tree $BOGUS_TREE) &&
+	git commit -m "bogus submodule commit" &&
+	git checkout --recurse-submodules -f master &&
+	test_must_fail git checkout --recurse-submodules bogus_commit &&
+	git diff-files --quiet &&
+	test_must_fail git checkout --recurse-submodules -f bogus_commit &&
+	test_must_fail git diff-files --quiet submodule &&
+	git diff-files --quiet file &&
+	git diff-index --quiet --cached HEAD &&
+	git checkout --recurse-submodules -f master
+'
+
 test_submodule_switch "git checkout"
 
 test_submodule_forced_switch "git checkout -f"
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 16/17] completion: add '--recurse-submodules' to checkout

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 contrib/completion/git-completion.bash | 2 +-
 t/t9902-completion.sh                  | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index 21016bf8df..28acfdb377 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1068,7 +1068,7 @@ _git_checkout ()
 	--*)
 		__gitcomp "
 			--quiet --ours --theirs --track --no-track --merge
-			--conflict= --orphan --patch
+			--conflict= --orphan --patch --recurse-submodules
 			"
 		;;
 	*)
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index 2ba62fbc17..d2d1102a3d 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -447,6 +447,7 @@ test_expect_success 'double dash "git checkout"' '
 	--conflict=
 	--orphan Z
 	--patch Z
+	--recurse-submodules Z
 	EOF
 '
 
-- 
2.11.0.rc2.28.g2673dad

[RFC PATCHv2 17/17] checkout: add config option to recurse into submodules by default

[Stefan Beller]

To make it easier for the user, who doesn't want to give the
`--recurse-submodules` option whenever they run checkout, have an
option for to set the default behavior for checkout to recurse into
submodules.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/config.txt       |  6 ++++++
 Documentation/git-checkout.txt |  5 +++--
 submodule.c                    |  6 +++---
 t/t2013-checkout-submodule.sh  | 16 ++++++++++++++++
 4 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index a0ab66aae7..67e0714358 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -949,6 +949,12 @@ clean.requireForce::
 	A boolean to make git-clean do nothing unless given -f,
 	-i or -n.   Defaults to true.
 
+checkout.recurseSubmodules::
+	This option can be set to a boolean value.
+	When set to true checkout will recurse into submodules and
+	update them. When set to false, which is the default, checkout
+	will leave submodules untouched.
+
 color.branch::
 	A boolean to enable/disable color in the output of
 	linkgit:git-branch[1]. May be set to `always`,
diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index a0ea2c5651..819c430b6e 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -260,8 +260,9 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
 	Using --recurse-submodules will update the content of all initialized
 	submodules according to the commit recorded in the superproject. If
 	local modifications in a submodule would be overwritten the checkout
-	will fail until `-f` is used. If nothing (or --no-recurse-submodules)
-	is used, the work trees of submodules will not be updated.
+	will fail until `-f` is used. If `--no-recurse-submodules` is used,
+	the work trees of submodules will not be updated. If no command line
+	argument is given, `checkout.recurseSubmodules` is used as a default.
 
 <branch>::
 	Branch to checkout; if it refers to a branch (i.e., a name that,
diff --git a/submodule.c b/submodule.c
index 4253f7f044..e4e326bce4 100644
--- a/submodule.c
+++ b/submodule.c
@@ -160,10 +160,10 @@ int submodule_config(const char *var, const char *value, void *cb)
 		return 0;
 	} else if (starts_with(var, "submodule."))
 		return parse_submodule_config_option(var, value);
-	else if (!strcmp(var, "fetch.recursesubmodules")) {
+	else if (!strcmp(var, "fetch.recursesubmodules"))
 		config_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);
-		return 0;
-	}
+	else if (!strcmp(var, "checkout.recursesubmodules"))
+		config_update_recurse_submodules = parse_update_recurse_submodules_arg(var, value);
 	return 0;
 }
 
diff --git a/t/t2013-checkout-submodule.sh b/t/t2013-checkout-submodule.sh
index 33fb2f5de3..673021fb80 100755
--- a/t/t2013-checkout-submodule.sh
+++ b/t/t2013-checkout-submodule.sh
@@ -137,6 +137,22 @@ test_expect_success '"checkout --recurse-submodules" repopulates submodule' '
 	submodule_creation_must_succeed delete_submodule base
 '
 
+test_expect_success 'option checkout.recurseSubmodules updates submodule' '
+	test_config checkout.recurseSubmodules 1 &&
+	git checkout base &&
+	git checkout -b advanced-base &&
+	git -C submodule commit --allow-empty -m "empty commit" &&
+	git add submodule &&
+	git commit -m "advance submodule" &&
+	git checkout base &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached base &&
+	git checkout advanced-base &&
+	git diff-files --quiet &&
+	git diff-index --quiet --cached advanced-base &&
+	git checkout --recurse-submodules base
+'
+
 test_expect_success '"checkout --recurse-submodules" repopulates submodule in existing directory' '
 	git checkout --recurse-submodules delete_submodule &&
 	mkdir submodule &&
-- 
2.11.0.rc2.28.g2673dad

Re: [RFC PATCHv2 15/17] checkout: recurse into submodules if asked to

[Brandon Williams]

On 12/02, Stefan Beller wrote:
>  
>  test_expect_success '"checkout <submodule>" honors diff.ignoreSubmodules' '
> @@ -63,6 +70,260 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
>  	! test -s actual
>  '

Should you use test_must_fail and not '!'?

-- 
Brandon Williams

Re: [RFC PATCHv2 17/17] checkout: add config option to recurse into submodules by default

[Brandon Williams]

On 12/02, Stefan Beller wrote:
> +test_expect_success 'option checkout.recurseSubmodules updates submodule' '
> +	test_config checkout.recurseSubmodules 1 &&
> +	git checkout base &&
> +	git checkout -b advanced-base &&
> +	git -C submodule commit --allow-empty -m "empty commit" &&
> +	git add submodule &&
> +	git commit -m "advance submodule" &&
> +	git checkout base &&
> +	git diff-files --quiet &&
> +	git diff-index --quiet --cached base &&
> +	git checkout advanced-base &&
> +	git diff-files --quiet &&
> +	git diff-index --quiet --cached advanced-base &&
> +	git checkout --recurse-submodules base
> +'
> +

This test doesn't look like it looks into the submodule to see if the
submodule has indeed changed.  Unless diff-index and diff-files recurse
into the submodules?
-- 
Brandon Williams

Re: [RFC PATCHv2 15/17] checkout: recurse into submodules if asked to

[Stefan Beller]

On Mon, Dec 5, 2016 at 11:25 AM, Brandon Williams <bmwill@google.com> wrote:
> On 12/02, Stefan Beller wrote:
>>
>>  test_expect_success '"checkout <submodule>" honors diff.ignoreSubmodules' '
>> @@ -63,6 +70,260 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
>>       ! test -s actual
>>  '
>
> Should you use test_must_fail and not '!'?

We use test_must_fail for git and '!' for non git thigns (test, grep etc),
as the test suite is about testing git.

The test_must_fail expects the command to be run to
* not reurn 0 (success)
* not segfault
* not return some other arbitrary return codes
  indicating abnormal failure (125 IIRC)

So in a way test_must_fail translates to:
"I want to run this git command and it should fail
gracefully because at this state, it is the best git can do"

The '!' however is just inverting the boolean expression.
We assume test, grep, et al. to be flawless here. ;)

Stefan

Re: [RFC PATCHv2 15/17] checkout: recurse into submodules if asked to

[Brandon Williams]

On 12/05, Stefan Beller wrote:
> On Mon, Dec 5, 2016 at 11:25 AM, Brandon Williams <bmwill@google.com> wrote:
> > On 12/02, Stefan Beller wrote:
> >>
> >>  test_expect_success '"checkout <submodule>" honors diff.ignoreSubmodules' '
> >> @@ -63,6 +70,260 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
> >>       ! test -s actual
> >>  '
> >
> > Should you use test_must_fail and not '!'?
> 
> We use test_must_fail for git and '!' for non git thigns (test, grep etc),
> as the test suite is about testing git.
> 
> The test_must_fail expects the command to be run to
> * not reurn 0 (success)
> * not segfault
> * not return some other arbitrary return codes
>   indicating abnormal failure (125 IIRC)
> 
> So in a way test_must_fail translates to:
> "I want to run this git command and it should fail
> gracefully because at this state, it is the best git can do"
> 
> The '!' however is just inverting the boolean expression.
> We assume test, grep, et al. to be flawless here. ;)

Ah, alright.  Thanks for the info :)

-- 
Brandon Williams

Re: [RFC PATCHv2 17/17] checkout: add config option to recurse into submodules by default

[Stefan Beller]

On Mon, Dec 5, 2016 at 11:29 AM, Brandon Williams <bmwill@google.com> wrote:
> On 12/02, Stefan Beller wrote:
>> +test_expect_success 'option checkout.recurseSubmodules updates submodule' '
>> +     test_config checkout.recurseSubmodules 1 &&
>> +     git checkout base &&
>> +     git checkout -b advanced-base &&
>> +     git -C submodule commit --allow-empty -m "empty commit" &&
>> +     git add submodule &&
>> +     git commit -m "advance submodule" &&
>> +     git checkout base &&
>> +     git diff-files --quiet &&
>> +     git diff-index --quiet --cached base &&
>> +     git checkout advanced-base &&
>> +     git diff-files --quiet &&
>> +     git diff-index --quiet --cached advanced-base &&
>> +     git checkout --recurse-submodules base
>> +'
>> +
>
> This test doesn't look like it looks into the submodule to see if the
> submodule has indeed changed.  Unless diff-index and diff-files recurse
> into the submodules?

I took the code from Jens once upon a time. Rereading the code, I agree it is
not obvious how this checks the submodule state.

However `git diff-files --quiet` is perfectly fine, as
we have submodule support by default via:

  Omitting the --submodule option or specifying --submodule=short,
  uses the short format.  This format just shows the names of the commits
  at the beginning and end of the range.

and then we turn it into an exit code via

       --quiet
           Disable all output of the program. Implies --exit-code.

       --exit-code
           Make the program exit with codes similar to diff(1).
  That is, it exits with 1 if there were differences and 0 means no differences.

Same for diff-index.

The main purpose of this specific test is to have checkout.recurseSubmodules
"to just make it work" without having to give --recurse-submodules manually.
All the other tests with the manual --recurse-submodules should test for
correctness of the behavior within the submodule.

So maybe I'll need to rewrite submodule_creation_must_succeed() in the previous
patch to be more obvious. (Well that already has some tests for
files/directories
in there, so it is a little more.)

But to be sure we can also add tests here that look more into the submodule.
I am thinking of "{new,old}_sub_sha1=$(git -C submodule rev-parse HEAD)" and
comparing them?

Re: [RFC PATCHv2 17/17] checkout: add config option to recurse into submodules by default

[Brandon Williams]

On 12/05, Stefan Beller wrote:
> On Mon, Dec 5, 2016 at 11:29 AM, Brandon Williams <bmwill@google.com> wrote:
> > On 12/02, Stefan Beller wrote:
> >> +test_expect_success 'option checkout.recurseSubmodules updates submodule' '
> >> +     test_config checkout.recurseSubmodules 1 &&
> >> +     git checkout base &&
> >> +     git checkout -b advanced-base &&
> >> +     git -C submodule commit --allow-empty -m "empty commit" &&
> >> +     git add submodule &&
> >> +     git commit -m "advance submodule" &&
> >> +     git checkout base &&
> >> +     git diff-files --quiet &&
> >> +     git diff-index --quiet --cached base &&
> >> +     git checkout advanced-base &&
> >> +     git diff-files --quiet &&
> >> +     git diff-index --quiet --cached advanced-base &&
> >> +     git checkout --recurse-submodules base
> >> +'
> >> +
> >
> > This test doesn't look like it looks into the submodule to see if the
> > submodule has indeed changed.  Unless diff-index and diff-files recurse
> > into the submodules?
> 
> I took the code from Jens once upon a time. Rereading the code, I agree it is
> not obvious how this checks the submodule state.
> 
> However `git diff-files --quiet` is perfectly fine, as
> we have submodule support by default via:
> 
>   Omitting the --submodule option or specifying --submodule=short,
>   uses the short format.  This format just shows the names of the commits
>   at the beginning and end of the range.
> 
> and then we turn it into an exit code via
> 
>        --quiet
>            Disable all output of the program. Implies --exit-code.
> 
>        --exit-code
>            Make the program exit with codes similar to diff(1).
>   That is, it exits with 1 if there were differences and 0 means no differences.
> 
> Same for diff-index.
> 
> The main purpose of this specific test is to have checkout.recurseSubmodules
> "to just make it work" without having to give --recurse-submodules manually.
> All the other tests with the manual --recurse-submodules should test for
> correctness of the behavior within the submodule.
> 
> So maybe I'll need to rewrite submodule_creation_must_succeed() in the previous
> patch to be more obvious. (Well that already has some tests for
> files/directories
> in there, so it is a little more.)
> 
> But to be sure we can also add tests here that look more into the submodule.
> I am thinking of "{new,old}_sub_sha1=$(git -C submodule rev-parse HEAD)" and
> comparing them?


Ah ok, that makes sense now.  Its kind of like if I run git status it
would show if a submodule is at a different sha1 than the superproject
has recorded.

-- 
Brandon Williams

RE: [RFC PATCHv2 15/17] checkout: recurse into submodules if asked to

[David Turner]

> -----Original Message-----
> From: Stefan Beller [mailto:sbeller@google.com]
> Sent: Friday, December 02, 2016 7:30 PM
> To: bmwill@google.com; David Turner
> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
> gitster@pobox.com; Stefan Beller
> Subject: [RFC PATCHv2 15/17] checkout: recurse into submodules if asked to
> 
> Allow checkout to recurse into submodules via the command line option --
> [no-]recurse-submodules.

This option probably needs to precede 9/17 "update submodules: add scheduling to update submodules", since that patch uses --recurse-submodules. 

RE: [RFC PATCHv2 09/17] update submodules: add scheduling to update submodules

[David Turner]

This patch confuses me -- see below.

> -----Original Message-----
> From: Stefan Beller [mailto:sbeller@google.com]
> Sent: Friday, December 02, 2016 7:30 PM
> To: bmwill@google.com; David Turner
> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
> gitster@pobox.com; Stefan Beller
> Subject: [RFC PATCHv2 09/17] update submodules: add scheduling to update
> submodules
[snip] 
> +static int update_submodule(const char *path, const struct object_id
> *oid,
> +			    int force, int is_new)
> +{
> +	const char *git_dir;
> +	struct child_process cp = CHILD_PROCESS_INIT;
> +	const struct submodule *sub = submodule_from_path(null_sha1, path);
> +
> +	if (!sub || !sub->name)
> +		return -1;
> +
> +	git_dir = resolve_gitdir(git_common_path("modules/%s", sub->name));
> +
> +	if (!git_dir)
> +		return -1;
> +
> +	if (is_new)
> +		connect_work_tree_and_git_dir(path, git_dir);
> +
> +	/* update index via `read-tree --reset sha1` */
> +	argv_array_pushl(&cp.args, "read-tree",
> +				   force ? "--reset" : "-m",
> +				   "-u", sha1_to_hex(oid->hash), NULL);
> +	prepare_submodule_repo_env(&cp.env_array);
> +	cp.git_cmd = 1;
> +	cp.no_stdin = 1;
> +	cp.dir = path;
> +	if (run_command(&cp)) {
> +		warning(_("reading the index in submodule '%s' failed"),
> path);

The error is not (usually) in "reading the index" -- it's "updating the index" (or the working tree)

> +		child_process_clear(&cp);
> +		return -1;
> +	}
> +
> +	/* write index to working dir */
> +	child_process_clear(&cp);
> +	child_process_init(&cp);
> +	argv_array_pushl(&cp.args, "checkout-index", "-a", NULL);

I'm confused -- doesn't read-tree -u already do this?  And if not, shouldn't we back out the result of the read-tree, to leave the submodule as it was?

> +	cp.git_cmd = 1;
> +	cp.no_stdin = 1;
> +	cp.dir = path;
> +	if (force)
> +		argv_array_push(&cp.args, "-f");
> +
> +	if (run_command(&cp)) {
> +		warning(_("populating the working directory in submodule '%s'
> failed"), path);
> +		child_process_clear(&cp);
> +		return -1;
> +	}
> +
> +	/* get the HEAD right */
> +	child_process_clear(&cp);
> +	child_process_init(&cp);
> +	argv_array_pushl(&cp.args, "checkout", "--recurse-submodules",
> NULL);


Why are we running checkout on the submodule when we've already done most of the checkout? The only thing left is to set HEAD and recurse, right?  I must be missing something.

RE: [RFC PATCHv2 14/17] submodule: teach unpack_trees() to update submodules

[David Turner]

> -----Original Message-----
> From: Stefan Beller [mailto:sbeller@google.com]
> Sent: Friday, December 02, 2016 7:30 PM
> To: bmwill@google.com; David Turner
> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
> gitster@pobox.com; Stefan Beller
> Subject: [RFC PATCHv2 14/17] submodule: teach unpack_trees() to update
> submodules
[snip]
>  	if (!lstat(ce->name, &st)) {
> -		int flags =
> CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;
> -		unsigned changed = ie_match_stat(o->src_index, ce, &st,
> flags);
> -		if (!changed)
> -			return 0;
> -		/*
> -		 * NEEDSWORK: the current default policy is to allow
> -		 * submodule to be out of sync wrt the superproject
> -		 * index.  This needs to be tightened later for
> -		 * submodules that are marked to be automatically
> -		 * checked out.
> -		 */
> -		if (S_ISGITLINK(ce->ce_mode))
> -			return 0;
> +		if (S_ISGITLINK(ce->ce_mode)) {
> +			if (!submodule_is_interesting(ce->name))
> +				return 0;
> +			if (ce_stage(ce) ? is_submodule_checkout_safe(ce->name,
> &ce->oid)
> +			    : !is_submodule_modified(ce->name, 1))

This logic is too convoluted.  Do a nested if or something.

> +				return 0;
> +		} else {
> +			int flags = CE_MATCH_IGNORE_VALID |
> CE_MATCH_IGNORE_SKIP_WORKTREE;
> +			if (!ie_match_stat(o->src_index, ce, &st, flags))
> +				return 0;

Nit: I liked the old temp var "changed" better -- it made it clear what
ie_match_stat is checking.

> +		}
>  		errno = 0;
>  	}
>  	if (errno == ENOENT)
> @@ -1355,6 +1359,38 @@ static int verify_uptodate_sparse(const struct
> cache_entry *ce,
>  	return verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE);  }
> 
> +/*
> + * When a submodule gets turned into an unmerged entry, we want it to
> +be
> + * up-to-date regarding the merge changes.
> + */
> +static int verify_uptodate_submodule(const struct cache_entry *old,
> +				     const struct cache_entry *new,
> +				     struct unpack_trees_options *o) {
> +	struct stat st;
> +
> +	if (o->index_only ||
> +	    (!((old->ce_flags & CE_VALID) || ce_skip_worktree(old)) &&
> +	      (o->reset || ce_uptodate(old))))
> +		return 0;
> +
> +	if (!submodule_is_interesting(new->name))
> +		return 0;
> +
> +	if (lstat(old->name, &st)) {

We never actually use st here.  Should we?  If not, is this really the right error message?  And should we use access() instead of lstat?

> +		if (errno == ENOENT)
> +			return 0;
> +		return o->gently ? -1 :
> +			add_rejected_path(o, ERROR_NOT_UPTODATE_SUBMODULE,
> +					  old->name);
> +	}
> +

RE: [RFC PATCHv2 08/17] update submodules: add depopulate_submodule

[David Turner]

> -----Original Message-----
> From: Stefan Beller [mailto:sbeller@google.com]
> Sent: Friday, December 02, 2016 7:30 PM
> To: bmwill@google.com; David Turner
> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
> gitster@pobox.com; Stefan Beller
> Subject: [RFC PATCHv2 08/17] update submodules: add depopulate_submodule
> 
> Implement the functionality needed to enable work tree manipulating
> commands so that a deleted submodule should not only affect the index
> (leaving all the files of the submodule in the work tree) but also to
> remove the work tree of the superproject (including any untracked files).

"including any untracked files" bothers me, I think.  Checkout is not usually willing to overwrite untracked files; it seems odd to me that it would be willing to do so in the submodule case.  I would be OK if they were both untracked and gitignored, I think.

> To do so, we need an equivalent of "rm -rf", which is already found in
> entry.c, so expose that and for clarity add a suffix "_or_dir" to it.
> 
> That will only work properly when the submodule uses a gitfile instead of
> a .git directory and no untracked files are present. Otherwise the removal
> will fail with a warning (which is just what happened until now).
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  cache.h     |  2 ++
>  entry.c     |  5 +++++
>  submodule.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  submodule.h |  1 +
>  4 files changed, 54 insertions(+)
> 
> diff --git a/cache.h b/cache.h
> index a50a61a197..b645ca2f9a 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -2018,4 +2018,6 @@ void sleep_millisec(int millisec);
>   */
>  void safe_create_dir(const char *dir, int share);
> 
> +extern void remove_directory_or_die(struct strbuf *path);
> +
>  #endif /* CACHE_H */
> diff --git a/entry.c b/entry.c
> index c6eea240b6..02c4ac9f22 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -73,6 +73,11 @@ static void remove_subtree(struct strbuf *path)
>  		die_errno("cannot rmdir '%s'", path->buf);  }
> 
> +void remove_directory_or_die(struct strbuf *path) {
> +	remove_subtree(path);
> +}
> +
>  static int create_file(const char *path, unsigned int mode)  {
>  	mode = (mode & 0100) ? 0777 : 0666;
> diff --git a/submodule.c b/submodule.c
> index 62e9ef3872..7bb64d6c69 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -324,6 +324,52 @@ void prepare_submodule_repo_env(struct argv_array
> *out)
>  	argv_array_push(out, "GIT_DIR=.git");
>  }
> 
> +int depopulate_submodule(const char *path) {
> +	int ret = 0;
> +	struct strbuf pathbuf = STRBUF_INIT;
> +	char *dot_git = xstrfmt("%s/.git", path);
> +
> +	/* Is it populated? */
> +	if (!resolve_gitdir(dot_git))
> +		goto out;
> +
> +	/* Does it have a .git directory? */
> +	if (!submodule_uses_gitfile(path)) {
> +		struct child_process cp = CHILD_PROCESS_INIT;
> +
> +		prepare_submodule_repo_env(&cp.env_array);
> +		argv_array_pushl(&cp.args, "submodule--helper",
> +				 "embed-git-dirs", path, NULL);
> +		cp.git_cmd = 1;
> +		if (run_command(&cp)) {
> +			warning(_("Cannot remove submodule '%s'\n"
> +				  "because it (or one of its nested submodules)
> has a git \n"
> +				  "directory in the working tree, which could not
> be embedded\n"
> +				  "the superprojects git directory
> automatically."), path);

What if instead it couldn't run the command because you're out of file descriptors or pids or memory or something?

I think this message should be in submodule--helper --embed-git-dirs instead, and we should just pass it through here.  Or, perhaps, instead of shelling out here, we should just call the functions directly?

> +			ret = -1;
> +			goto out;
> +		}
> +
> +		if (!submodule_uses_gitfile(path)) {
> +			/*
> +			 * We should be using a gitfile by now, let's double

Comma splice.  

> +			 * check as loosing the git dir would be fatal.

s/loosing/losing/

> +			 */
> +			die("BUG: \"git submodule--helper embed git-dirs '%s'\"
> "
> +			    "did not embed the git-dirs recursively for '%s'",
> +			    path, path);
> +		}
> +	}
> +
> +	strbuf_addstr(&pathbuf, path);
> +	remove_directory_or_die(&pathbuf);
> +out:
> +	strbuf_release(&pathbuf);
> +	free(dot_git);
> +	return ret;
> +}
> +
>  /* Helper function to display the submodule header line prior to the full
>   * summary output. If it can locate the submodule objects directory it
> will
>   * attempt to lookup both the left and right commits and put them into
> the diff --git a/submodule.h b/submodule.h index 7d890e0464..d8bb1d4baf
> 100644
> --- a/submodule.h
> +++ b/submodule.h
> @@ -63,6 +63,7 @@ extern void set_config_update_recurse_submodules(int
> value);
>   */
>  extern int submodule_is_interesting(const char *path);  extern int
> submodules_interesting_for_update(void);
> +extern int depopulate_submodule(const char *path);
>  extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
> extern int fetch_populated_submodules(const struct argv_array *options,
>  			       const char *prefix, int command_line_option,
> --
> 2.11.0.rc2.28.g2673dad

Re: [RFC PATCHv2 09/17] update submodules: add scheduling to update submodules

[Stefan Beller]

On Mon, Dec 5, 2016 at 3:37 PM, David Turner <David.Turner@twosigma.com> wrote:
> This patch confuses me -- see below.
>
>> -----Original Message-----
>> From: Stefan Beller [mailto:sbeller@google.com]
>> Sent: Friday, December 02, 2016 7:30 PM
>> To: bmwill@google.com; David Turner
>> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
>> gitster@pobox.com; Stefan Beller
>> Subject: [RFC PATCHv2 09/17] update submodules: add scheduling to update
>> submodules
> [snip]
>> +static int update_submodule(const char *path, const struct object_id
>> *oid,
>> +                         int force, int is_new)
>> +{
>> +     const char *git_dir;
>> +     struct child_process cp = CHILD_PROCESS_INIT;
>> +     const struct submodule *sub = submodule_from_path(null_sha1, path);
>> +
>> +     if (!sub || !sub->name)
>> +             return -1;
>> +
>> +     git_dir = resolve_gitdir(git_common_path("modules/%s", sub->name));
>> +
>> +     if (!git_dir)
>> +             return -1;
>> +
>> +     if (is_new)
>> +             connect_work_tree_and_git_dir(path, git_dir);
>> +
>> +     /* update index via `read-tree --reset sha1` */
>> +     argv_array_pushl(&cp.args, "read-tree",
>> +                                force ? "--reset" : "-m",
>> +                                "-u", sha1_to_hex(oid->hash), NULL);
>> +     prepare_submodule_repo_env(&cp.env_array);
>> +     cp.git_cmd = 1;
>> +     cp.no_stdin = 1;
>> +     cp.dir = path;
>> +     if (run_command(&cp)) {
>> +             warning(_("reading the index in submodule '%s' failed"),
>> path);
>
> The error is not (usually) in "reading the index" -- it's "updating the index" (or the working tree)
>
>> +             child_process_clear(&cp);
>> +             return -1;
>> +     }
>> +
>> +     /* write index to working dir */
>> +     child_process_clear(&cp);
>> +     child_process_init(&cp);
>> +     argv_array_pushl(&cp.args, "checkout-index", "-a", NULL);
>
> I'm confused -- doesn't read-tree -u already do this?  And if not, shouldn't we back out the result of the read-tree, to leave the submodule as it was?
>
>> +     cp.git_cmd = 1;
>> +     cp.no_stdin = 1;
>> +     cp.dir = path;
>> +     if (force)
>> +             argv_array_push(&cp.args, "-f");
>> +
>> +     if (run_command(&cp)) {
>> +             warning(_("populating the working directory in submodule '%s'
>> failed"), path);
>> +             child_process_clear(&cp);
>> +             return -1;
>> +     }
>> +
>> +     /* get the HEAD right */
>> +     child_process_clear(&cp);
>> +     child_process_init(&cp);
>> +     argv_array_pushl(&cp.args, "checkout", "--recurse-submodules",
>> NULL);
>
>
> Why are we running checkout on the submodule when we've already done most of the checkout? The only thing left is to set HEAD and recurse, right?  I must be missing something.
>

Yes this is only used to set the HEAD correctly and then recurse down.
I tried to remove the first 2 calls to ch8ild processes at one point in time,
which did not work out.  I should have written in the commit message why
that was a problem. So I'll redo that just to see the problem and improve
the commit message.

Re: [RFC PATCHv2 08/17] update submodules: add depopulate_submodule

[Stefan Beller]

On Mon, Dec 5, 2016 at 3:37 PM, David Turner <David.Turner@twosigma.com> wrote:
>> -----Original Message-----
>> From: Stefan Beller [mailto:sbeller@google.com]
>> Sent: Friday, December 02, 2016 7:30 PM
>> To: bmwill@google.com; David Turner
>> Cc: git@vger.kernel.org; sandals@crustytoothpaste.net; hvoigt@hvoigt.net;
>> gitster@pobox.com; Stefan Beller
>> Subject: [RFC PATCHv2 08/17] update submodules: add depopulate_submodule
>>
>> Implement the functionality needed to enable work tree manipulating
>> commands so that a deleted submodule should not only affect the index
>> (leaving all the files of the submodule in the work tree) but also to
>> remove the work tree of the superproject (including any untracked files).
>
> "including any untracked files" bothers me, I think.  Checkout is not usually willing to overwrite untracked files; it seems odd to me that it would be willing to do so in the submodule case.  I would be OK if they were both untracked and gitignored, I think.

I agree on being bothered, this is one of the things I thought how to solve.
See the test in "checkout: recurse into submodules if asked to", which
tests for untracked files and is still marked as a failure.

I think to address that issue, I'll add a flag to ok_to_remove_submodule
which let's you specify which files you care about and which you don't.

>> +                     warning(_("Cannot remove submodule '%s'\n"
>> +                               "because it (or one of its nested submodules)
>> has a git \n"
>> +                               "directory in the working tree, which could not
>> be embedded\n"
>> +                               "the superprojects git directory
>> automatically."), path);
>
> What if instead it couldn't run the command because you're out of file descriptors or pids or memory or something?
>
> I think this message should be in submodule--helper --embed-git-dirs instead, and we should just pass it through here.  Or, perhaps, instead of shelling out here, we should just call the functions directly?

heh, good point. Will call the function directly.

>