[RFC PATCH 0/5] Submodule Groups

[RFC PATCH 0/5] Submodule Groups

[Stefan Beller]

This is also available at https://github.com/stefanbeller/git/tree/submodule-groups
It applies on top of the submodule-parallel-patch series I sent a few minutes ago.

Consider having a real large software project in Git with each component
in a submodule (such as an operating system, Android, Debian, Fedora,
no toy OS such as https://github.com/gittup/gittup as that doesn't quite
demonstrate the scale of the problem).

If you have lots of submodules, you probably don't need all of them at once,
but you have functional units. Some submodules are absolutely required,
some are optional and only for very specific purposes.

This patch series adds meaning to a "groups" field in the .gitmodules file.

So you could have a .gitmodules file such as:

[submodule "gcc"]
        path = gcc
        url = git://...
        groups = default,devel
[submodule "linux"]
        path = linux
        url = git://...
        groups = default
[submodule "nethack"]
        path = nethack
        url = git://...
        groups = optional,games

and by this series you can work on an arbitrary subgroup of these submodules such
using these commands:

    git clone --group default --group devel git://...
    # will clone the superproject and recursively
    # checkout any submodule being in at least one of the groups.

    git submodule add --group default --group devel git://... ..
    # will add a submodule, adding 2 submodule
    # groups to its entry in .gitmodule
    
    # as support for clone we want to have:
    git config submodule.groups default
    git submodule init --groups
    # will init all submodules from the default group
    
    # as support for clone we want to have:
    git config submodule.groups default
    git submodule update --groups
    # will update all submodules from the default group

Any feedback welcome, specially on the design level!
(Do we want to have it stored in the .gitmodules file? Do we want to have
the groups configured in .git/config as "submodule.groups", any other way
to make it future proof and extend the groups syntax?)

Thanks,
Stefan

Stefan Beller (5):
  submodule-config: keep submodule groups around
  git submodule add can add a submodule with groups
  git submodule init to pass on groups
  submodule--helper: module_list and update-clone have --groups option
  builtin/clone: support submodule groups

 Documentation/git-clone.txt     |  11 ++++
 Documentation/git-submodule.txt |   8 ++-
 builtin/clone.c                 |  33 ++++++++++-
 builtin/submodule--helper.c     |  68 ++++++++++++++++++++++-
 git-submodule.sh                |  20 ++++++-
 submodule-config.c              |  14 +++++
 submodule-config.h              |   2 +
 t/t7400-submodule-basic.sh      | 118 ++++++++++++++++++++++++++++++++++++++++
 t/t7406-submodule-update.sh     |  32 +++++++++++
 9 files changed, 299 insertions(+), 7 deletions(-)

-- 
2.6.1.261.g0d9c4c1

[PATCH 1/5] submodule-config: keep submodule groups around

[Stefan Beller]

We need to query the groups in a later patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule-config.c | 14 ++++++++++++++
 submodule-config.h |  2 ++
 2 files changed, 16 insertions(+)

diff --git a/submodule-config.c b/submodule-config.c
index a32259e..f44ce20 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -60,6 +60,7 @@ static void free_one_config(struct submodule_entry *entry)
 {
 	free((void *) entry->config->path);
 	free((void *) entry->config->name);
+	free((void *) entry->config->groups);
 	free(entry->config);
 }
 
@@ -182,6 +183,8 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,
 	submodule->path = NULL;
 	submodule->url = NULL;
 	submodule->update = NULL;
+	submodule->groups = xmalloc(sizeof(*submodule->groups));
+	string_list_init(submodule->groups, 1);
 	submodule->fetch_recurse = RECURSE_SUBMODULES_NONE;
 	submodule->ignore = NULL;
 
@@ -324,6 +327,17 @@ static int parse_specific_submodule_config(const char *subsection, int subsectio
 			free((void *) submodule->update);
 			submodule->update = xstrdup(value);
 		}
+	} else if (!strcmp(key, "groups")) {
+		if (!value)
+			ret = config_error_nonbool(var);
+		else if (!me->overwrite && submodule->groups)
+			warn_multiple_config(me->commit_sha1, submodule->name,
+					     "groups");
+		else {
+			string_list_clear(submodule->groups, 0);
+			string_list_split(submodule->groups, value, ',', -1);
+			string_list_sort(submodule->groups);
+		}
 	}
 
 	return ret;
diff --git a/submodule-config.h b/submodule-config.h
index d9bbf9a..7fc21e1 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -3,6 +3,7 @@
 
 #include "hashmap.h"
 #include "strbuf.h"
+#include "string-list.h"
 
 /*
  * Submodule entry containing the information about a certain submodule
@@ -17,6 +18,7 @@ struct submodule {
 	const char *update;
 	/* the sha1 blob id of the responsible .gitmodules file */
 	unsigned char gitmodules_sha1[20];
+	struct string_list *groups;
 };
 
 int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);
-- 
2.6.1.261.g0d9c4c1

[PATCH 2/5] git submodule add can add a submodule with groups

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-submodule.txt |  8 +++++++-
 git-submodule.sh                |  9 +++++++++
 t/t7400-submodule-basic.sh      | 28 ++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index a87ff72..b434d8d 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -9,7 +9,7 @@ git-submodule - Initialize, update or inspect submodules
 SYNOPSIS
 --------
 [verse]
-'git submodule' [--quiet] add [-b <branch>] [-f|--force] [--name <name>]
+'git submodule' [--quiet] add [-b <branch>] [-f|--force] [-g <group>][--name <name>]
 	      [--reference <repository>] [--depth <depth>] [--] <repository> [<path>]
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
@@ -59,6 +59,9 @@ instead of treating the other project as a submodule. Directories
 that come from both projects can be cloned and checked out as a whole
 if you choose to go that route.
 
+If you manage a large set of submodules, but do not require all of them
+to be checked out, you should look into the submodule groups feature.
+
 COMMANDS
 --------
 add::
@@ -101,6 +104,9 @@ is the superproject and submodule repositories will be kept
 together in the same relative location, and only the
 superproject's URL needs to be provided: git-submodule will correctly
 locate the submodule using the relative URL in .gitmodules.
++
+If at least one group argument was given, all groups are recorded in the
+.gitmodules file in the groups field.
 
 status::
 	Show the status of the submodules. This will print the SHA-1 of the
diff --git a/git-submodule.sh b/git-submodule.sh
index 10c5af9..bbdcf78 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -203,6 +203,7 @@ cmd_add()
 {
 	# parse $args after "submodule ... add".
 	reference_path=
+	submodule_groups=
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -238,6 +239,10 @@ cmd_add()
 		--depth=*)
 			depth=$1
 			;;
+		-g|--group)
+			submodule_groups=${submodule_groups:+${submodule_groups},}"$2"
+			shift
+			;;
 		--)
 			shift
 			break
@@ -365,6 +370,10 @@ Use -f if you really want to add it." >&2
 
 	git config -f .gitmodules submodule."$sm_name".path "$sm_path" &&
 	git config -f .gitmodules submodule."$sm_name".url "$repo" &&
+	if test -n "$submodule_groups"
+	then
+		git config -f .gitmodules submodule."$sm_name".groups "${submodule_groups}"
+	fi &&
 	if test -n "$branch"
 	then
 		git config -f .gitmodules submodule."$sm_name".branch "$branch"
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 5991e3c..a422df3 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -986,6 +986,7 @@ test_expect_success 'submodule with UTF-8 name' '
 '
 
 test_expect_success 'submodule add clone shallow submodule' '
+	test_when_finished "rm -rf super" &&
 	mkdir super &&
 	pwd=$(pwd) &&
 	(
@@ -999,5 +1000,32 @@ test_expect_success 'submodule add clone shallow submodule' '
 	)
 '
 
+test_expect_success 'submodule add records a group' '
+	test_when_finished "rm -rf super" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git config -f .gitmodules submodule."submodule".groups >actual &&
+		echo groupA >expected &&
+		test_cmp expected actual
+	)
+'
+
+test_expect_success 'submodule add records groups' '
+	test_when_finished "rm -rf super" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA -g groupB file://"$pwd"/example2 submodule &&
+		git config -f .gitmodules submodule."submodule".groups >actual &&
+		echo groupA,groupB >expected &&
+		test_cmp expected actual
+	)
+'
 
 test_done
-- 
2.6.1.261.g0d9c4c1

[PATCH 3/5] git submodule init to pass on groups

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh           |  6 +++++-
 t/t7400-submodule-basic.sh | 21 +++++++++++++++++++++
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index bbdcf78..4092a48 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -455,6 +455,7 @@ cmd_foreach()
 #
 cmd_init()
 {
+	submodule_groups=
 	# parse $args after "submodule ... init".
 	while test $# -ne 0
 	do
@@ -462,6 +463,9 @@ cmd_init()
 		-q|--quiet)
 			GIT_QUIET=1
 			;;
+		-g|--groups)
+			submodule_groups=1
+			;;
 		--)
 			shift
 			break
@@ -476,7 +480,7 @@ cmd_init()
 		shift
 	done
 
-	git submodule--helper list --prefix "$wt_prefix" "$@" |
+	git submodule--helper list ${submodule_groups:+--groups} --prefix "$wt_prefix" "$@" |
 	while read mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode"
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index a422df3..caed4be 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1028,4 +1028,25 @@ test_expect_success 'submodule add records groups' '
 	)
 '
 
+test_expect_success 'submodule init --group works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.groups groupA &&
+		git submodule init --groups &&
+		git config submodule.submodule.url &&
+		test_must_fail git config submodule.submodule1.url
+	)
+'
+
 test_done
-- 
2.6.1.261.g0d9c4c1

[PATCH 4/5] submodule--helper: module_list and update-clone have --groups option

[Stefan Beller]

This will be useful in a later patch.
when passing in the --groups option, only the configured groups are
considered instead of all groups.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c | 68 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 254824a..6a208ac 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -67,16 +67,33 @@ static int module_list_compute(int argc, const char **argv,
 	return result;
 }
 
+static int load_submodule_groups(struct string_list **groups)
+{
+	const char *g = NULL;
+	if (git_config_get_string_const("submodule.groups", &g) < 0)
+		return -1;
+	if (!g)
+		return 1;
+	*groups = xmalloc(sizeof(**groups));
+	string_list_init(*groups, 1);
+	string_list_split(*groups, g, ',', -1);
+	string_list_sort(*groups);
+	return 0;
+}
+
 static int module_list(int argc, const char **argv, const char *prefix)
 {
-	int i;
+	int i, groups = 0;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
+	struct string_list *submodule_groups;
 
 	struct option module_list_options[] = {
 		OPT_STRING(0, "prefix", &prefix,
 			   N_("path"),
 			   N_("alternative anchor for relative paths")),
+		OPT_BOOL(0, "groups", &groups,
+			 N_("Only initialize configured submodule groups")),
 		OPT_END()
 	};
 
@@ -93,9 +110,33 @@ static int module_list(int argc, const char **argv, const char *prefix)
 		return 1;
 	}
 
+	if (groups) {
+		gitmodules_config();
+		if (load_submodule_groups(&submodule_groups))
+			die(_("No groups configured?"));
+	}
 	for (i = 0; i < list.nr; i++) {
 		const struct cache_entry *ce = list.entries[i];
 
+		if (groups) {
+			int found = 0;
+			struct string_list_item *item;
+			const struct submodule *sub = submodule_from_path(null_sha1, ce->name);
+			if (!sub)
+				die("BUG: Could not find submodule %s in cache, "
+				    "despite having found it earlier", ce->name);
+			else {
+				for_each_string_list_item(item, sub->groups) {
+					if (string_list_lookup(submodule_groups, item->string)) {
+						found = 1;
+						break;
+					}
+				}
+			if (!found)
+				continue;
+			}
+		}
+
 		if (ce_stage(ce))
 			printf("%06o %s U\t", ce->ce_mode, sha1_to_hex(null_sha1));
 		else
@@ -262,6 +303,7 @@ static int git_submodule_config(const char *var, const char *value, void *cb)
 
 struct submodule_update_clone {
 	/* states */
+	struct string_list *submodule_groups;
 	int count;
 	int print_unmatched;
 	/* configuration */
@@ -275,7 +317,7 @@ struct submodule_update_clone {
 	struct string_list projectlines;
 	struct pathspec pathspec;
 };
-#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}
+#define SUBMODULE_UPDATE_CLONE_INIT {NULL, 0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}
 
 static void fill_clone_command(struct child_process *cp, int quiet,
 			       const char *prefix, const char *path,
@@ -318,6 +360,7 @@ static int update_clone_get_next_task(void **pp_task_cb,
 		const char *update_module = NULL;
 		char *url = NULL;
 		int needs_cloning = 0;
+		int in_submodule_groups = 0;
 
 		if (ce_stage(ce)) {
 			if (pp->recursive_prefix)
@@ -372,6 +415,20 @@ static int update_clone_get_next_task(void **pp_task_cb,
 			continue;
 		}
 
+		if (pp->submodule_groups) {
+			struct string_list_item *item;
+			for_each_string_list_item(item, sub->groups) {
+				if (string_list_lookup(
+				    pp->submodule_groups, item->string)) {
+					in_submodule_groups = 1;
+					break;
+				}
+			}
+		}
+
+		if (pp->submodule_groups && !in_submodule_groups)
+			continue;
+
 		strbuf_reset(&sb);
 		strbuf_addf(&sb, "%s/.git", ce->name);
 		needs_cloning = !file_exists(sb.buf);
@@ -427,6 +484,7 @@ static int update_clone_task_finished(int result,
 static int update_clone(int argc, const char **argv, const char *prefix)
 {
 	int max_jobs = -1;
+	int submodule_groups = 0;
 	struct string_list_item *item;
 	struct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;
 
@@ -449,6 +507,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 			      "specified number of revisions")),
 		OPT_INTEGER('j', "jobs", &max_jobs,
 			    N_("parallel jobs")),
+		OPT_BOOL(0, "groups", &submodule_groups,
+			 N_("operate only on configured groups")),
 		OPT__QUIET(&pp.quiet, N_("do't print cloning progress")),
 		OPT_END()
 	};
@@ -467,6 +527,9 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 		return 1;
 	}
 
+	if (submodule_groups)
+		load_submodule_groups(&pp.submodule_groups);
+
 	gitmodules_config();
 	/* Overlay the parsed .gitmodules file with .git/config */
 	git_config(git_submodule_config, NULL);
@@ -490,6 +553,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 	for_each_string_list_item(item, &pp.projectlines)
 		utf8_fprintf(stdout, "%s", item->string);
 
+	free(pp.submodule_groups);
 	return 0;
 }
 
-- 
2.6.1.261.g0d9c4c1

[PATCH 5/5] builtin/clone: support submodule groups

[Stefan Beller]

This passes each group to the `submodule update` invocation and
additionally configures the groups to be automatically updated.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-clone.txt | 11 ++++++++
 builtin/clone.c             | 33 ++++++++++++++++++++--
 git-submodule.sh            |  5 ++++
 t/t7400-submodule-basic.sh  | 69 +++++++++++++++++++++++++++++++++++++++++++++
 t/t7406-submodule-update.sh | 32 +++++++++++++++++++++
 5 files changed, 147 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 59d8c67..fbf68ab 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -209,6 +209,17 @@ objects from the source repository into a pack in the cloned repository.
 	repository does not have a worktree/checkout (i.e. if any of
 	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
 
+--group::
+	After the clone is created, all submodules which are part of the
+	group are cloned. This option can be given multiple times to specify
+	different groups. This option will imply automatic submodule
+	updates for the groups by setting `submodule.update=groups`.
+	The group selection will be passed on recursively, i.e. if a submodule
+	is cloned because of group membership, its submodules will
+	be cloned according to group membership, too. If a submodule is
+	not cloned however, its submodules are not evaluated for group
+	membership.
+
 --separate-git-dir=<git dir>::
 	Instead of placing the cloned repository where it is supposed
 	to be, place the cloned repository at the specified directory,
diff --git a/builtin/clone.c b/builtin/clone.c
index ce578d2..17e9f54 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -51,6 +51,7 @@ static struct string_list option_config;
 static struct string_list option_reference;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list submodule_groups;
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {
 		   N_("separate git dir from working tree")),
 	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
 			N_("set config inside the new repository")),
+	OPT_STRING_LIST('g', "group", &submodule_groups, N_("group"),
+			N_("clone specific submodule groups")),
 	OPT_END()
 };
 
@@ -723,9 +726,18 @@ static int checkout(void)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (err)
+		goto out;
+
+	if (option_recursive || submodule_groups.nr > 0) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", "--init", NULL);
+
+		if (option_recursive)
+			argv_array_pushf(&args, "--recursive");
+
+		if (submodule_groups.nr > 0)
+			argv_array_pushf(&args, "--groups");
 
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
@@ -733,7 +745,7 @@ static int checkout(void)
 		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
 		argv_array_clear(&args);
 	}
-
+out:
 	return err;
 }
 
@@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		option_no_checkout = 1;
 	}
 
+	if (option_recursive && submodule_groups.nr > 0)
+		die(_("submodule groups and recursive flag are incompatible"));
+	if (submodule_groups.nr > 0) {
+		int first_item = 1;
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		strbuf_addstr(&sb, "submodule.groups=");
+		for_each_string_list_item(item, &submodule_groups) {
+			strbuf_addf(&sb, "%s%s", first_item ? "" : ",", item->string);
+			first_item = 0;
+		}
+		if (submodule_groups.nr > 0)
+			string_list_append(&option_config, strbuf_detach(&sb, 0));
+	}
+
 	if (!option_origin)
 		option_origin = "origin";
 
diff --git a/git-submodule.sh b/git-submodule.sh
index 4092a48..e3d1667 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -611,6 +611,7 @@ cmd_deinit()
 #
 cmd_update()
 {
+	groups=
 	# parse $args after "submodule ... update".
 	while test $# -ne 0
 	do
@@ -650,6 +651,9 @@ cmd_update()
 		--checkout)
 			update="checkout"
 			;;
+		--groups)
+			groups=1
+			;;
 		--depth)
 			case "$2" in '') usage ;; esac
 			depth="--depth=$2"
@@ -691,6 +695,7 @@ cmd_update()
 		${update:+--update "$update"} \
 		${reference:+--reference "$reference"} \
 		${depth:+--depth "$depth"} \
+		${groups:+--groups} \
 		${jobs:+$jobs} \
 		"$@" | {
 	err=
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index caed4be..e8654d7 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '
 	)
 '
 
+cat <<EOF > expected
+submodule
+-submodule1
+EOF
+
+test_expect_success 'submodule update --groups works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.groups groupA &&
+		git submodule init  &&
+		git submodule update --groups &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+test_expect_success 'submodule update --init --groups works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.groups groupA &&
+		git submodule update --init --groups &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone --group works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone --group groupA super super_clone &&
+	(
+		cd super_clone &&
+		test_pause
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+
 test_done
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 090891e..7e59846 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'
 	rm -rf super4
 '
 
+cat >expect <<-EOF &&
+-deeper/submodule
+-merging
+-moved/sub module
+-none
+-rebasing
+-submodule
+-submodule1
+EOF
+
+# none, merging rebasing, submodule1, submodule
+test_expect_success 'git clone works with submodule groups.' '
+	test_when_finished "rm -rf super5" &&
+	(
+		cd super &&
+		git config -f .gitmodules  submodule.submodule.groups default &&
+		git config -f .gitmodules  submodule.submodule1.groups "default,testing" &&
+		git config -f .gitmodules  submodule.none.groups testing &&
+		git commit -a -m "assigning groups to submodules"
+	) &&
+	git clone --group default --group testing super super5 &&
+	(
+		cd super5 &&
+		git submodule status |cut -c1,43- >../actual
+	) &&
+	test_cmp actual expect
+'
+
+test_expect_success 'git submodule update --groups' '
+	true
+'
+
 test_done
-- 
2.6.1.261.g0d9c4c1

Re: [RFC PATCH 0/5] Submodule Groups

[Jens Lehmann]

Am 25.11.2015 um 02:32 schrieb Stefan Beller:
> This is also available at https://github.com/stefanbeller/git/tree/submodule-groups
> It applies on top of the submodule-parallel-patch series I sent a few minutes ago.
>
> Consider having a real large software project in Git with each component
> in a submodule (such as an operating system, Android, Debian, Fedora,
> no toy OS such as https://github.com/gittup/gittup as that doesn't quite
> demonstrate the scale of the problem).
>
> If you have lots of submodules, you probably don't need all of them at once,
> but you have functional units. Some submodules are absolutely required,
> some are optional and only for very specific purposes.
>
> This patch series adds meaning to a "groups" field in the .gitmodules file.
>
> So you could have a .gitmodules file such as:
>
> [submodule "gcc"]
>          path = gcc
>          url = git://...
>          groups = default,devel
> [submodule "linux"]
>          path = linux
>          url = git://...
>          groups = default
> [submodule "nethack"]
>          path = nethack
>          url = git://...
>          groups = optional,games

Yup. Do you want the user to select only a single group or do you
plan to support selecting multiple groups at the same time too?

> and by this series you can work on an arbitrary subgroup of these submodules such
> using these commands:
>
>      git clone --group default --group devel git://...
>      # will clone the superproject and recursively
>      # checkout any submodule being in at least one of the groups.

Does this automatically configure the given group in .git/config, so
that all future submodule related commands know about this choice?
Me thinks that would make sense ...

>      git submodule add --group default --group devel git://... ..
>      # will add a submodule, adding 2 submodule
>      # groups to its entry in .gitmodule

Maybe '--groups default,devel' is easier to grok? Dunno.

>      # as support for clone we want to have:
>      git config submodule.groups default
>      git submodule init --groups

Hmm, I doubt it makes much sense to add the --group option to "git
submodule init". I'd rather init all submodules and do the group
handling only in the "git submodule update" command. That way
upstream can change grouping later without having the user to
fiddle with her configuration to make that work.

>      # will init all submodules from the default group
>
>      # as support for clone we want to have:
>      git config submodule.groups default
>      git submodule update --groups
>
>      # will update all submodules from the default group
>
> Any feedback welcome, specially on the design level!
> (Do we want to have it stored in the .gitmodules file? Do we want to have
> the groups configured in .git/config as "submodule.groups", any other way
> to make it future proof and extend the groups syntax?)

Not sure what exactly you mean by "it" here ;-)

Talking about what groups a submodule belongs to, an entry in the
.gitmodules file makes the most sense to me. That way upstream can
change submodule grouping or add new submodules with group assignments
from commit to commit, and "git submodule update" will do the right
thing for the superproject commit checked out.

And I believe that the choice which group(s?) the user is interested
should be recorded in .git/config, as that is his personal setting
that shouldn't be influenced by upstream changes.

Re: [RFC PATCH 0/5] Submodule Groups

[Jens Lehmann]

Am 25.11.2015 um 02:32 schrieb Stefan Beller:
> This is also available at https://github.com/stefanbeller/git/tree/submodule-groups
> It applies on top of the submodule-parallel-patch series I sent a few minutes ago.
>
> Consider having a real large software project in Git with each component
> in a submodule (such as an operating system, Android, Debian, Fedora,
> no toy OS such as https://github.com/gittup/gittup as that doesn't quite
> demonstrate the scale of the problem).
>
> If you have lots of submodules, you probably don't need all of them at once,
> but you have functional units. Some submodules are absolutely required,
> some are optional and only for very specific purposes.
>
> This patch series adds meaning to a "groups" field in the .gitmodules file.
>
> So you could have a .gitmodules file such as:
>
> [submodule "gcc"]
>          path = gcc
>          url = git://...
>          groups = default,devel
> [submodule "linux"]
>          path = linux
>          url = git://...
>          groups = default
> [submodule "nethack"]
>          path = nethack
>          url = git://...
>          groups = optional,games

Yup. Do you want the user to select only a single group or do you
plan to support selecting multiple groups at the same time too?

> and by this series you can work on an arbitrary subgroup of these submodules such
> using these commands:
>
>      git clone --group default --group devel git://...
>      # will clone the superproject and recursively
>      # checkout any submodule being in at least one of the groups.

Does this automatically configure the given group in .git/config, so
that all future submodule related commands know about this choice?
Me thinks that would make sense ...

>      git submodule add --group default --group devel git://... ..
>      # will add a submodule, adding 2 submodule
>      # groups to its entry in .gitmodule

Maybe '--groups default,devel' is easier to grok? Dunno.

>      # as support for clone we want to have:
>      git config submodule.groups default
>      git submodule init --groups

Hmm, I doubt it makes much sense to add the --group option to "git
submodule init". I'd rather init all submodules and do the group
handling only in the "git submodule update" command. That way
upstream can change grouping later without having the user to
fiddle with her configuration to make that work.

>      # will init all submodules from the default group
>
>      # as support for clone we want to have:
>      git config submodule.groups default
>      git submodule update --groups
>
>      # will update all submodules from the default group
>
> Any feedback welcome, specially on the design level!
> (Do we want to have it stored in the .gitmodules file? Do we want to have
> the groups configured in .git/config as "submodule.groups", any other way
> to make it future proof and extend the groups syntax?)

Not sure what exactly you mean by "it" here ;-)

Talking about what groups a submodule belongs to, an entry in the
.gitmodules file makes the most sense to me. That way upstream can
change submodule grouping or add new submodules with group assignments
from commit to commit, and "git submodule update" will do the right
thing for the superproject commit checked out.

And I believe that the choice which group(s?) the user is interested
should be recorded in .git/config, as that is his personal setting
that shouldn't be influenced by upstream changes.

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Jens Lehmann]

Am 25.11.2015 um 02:32 schrieb Stefan Beller:
> This passes each group to the `submodule update` invocation and
> additionally configures the groups to be automatically updated.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>   Documentation/git-clone.txt | 11 ++++++++
>   builtin/clone.c             | 33 ++++++++++++++++++++--
>   git-submodule.sh            |  5 ++++
>   t/t7400-submodule-basic.sh  | 69 +++++++++++++++++++++++++++++++++++++++++++++
>   t/t7406-submodule-update.sh | 32 +++++++++++++++++++++
>   5 files changed, 147 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index 59d8c67..fbf68ab 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -209,6 +209,17 @@ objects from the source repository into a pack in the cloned repository.
>   	repository does not have a worktree/checkout (i.e. if any of
>   	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
>
> +--group::
> +	After the clone is created, all submodules which are part of the
> +	group are cloned. This option can be given multiple times to specify
> +	different groups.

Ah, that answers my question in my response to the cover letter ;-)

> This option will imply automatic submodule
> +	updates for the groups by setting `submodule.update=groups`.

Please don't. The per-submodule update setting configures how a
submodule has to be updated, adding a global one with a completely
different meaning (what submodules should be updated?) is confusing.
Why not "submodule.groups=<groups>"?

> +	The group selection will be passed on recursively, i.e. if a submodule
> +	is cloned because of group membership, its submodules will
> +	be cloned according to group membership, too. If a submodule is
> +	not cloned however, its submodules are not evaluated for group
> +	membership.

What do you mean by the last sentence? Did the clone fail? Then you
cannot update the submodule anyway ...

>   --separate-git-dir=<git dir>::
>   	Instead of placing the cloned repository where it is supposed
>   	to be, place the cloned repository at the specified directory,
> diff --git a/builtin/clone.c b/builtin/clone.c
> index ce578d2..17e9f54 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -51,6 +51,7 @@ static struct string_list option_config;
>   static struct string_list option_reference;
>   static int option_dissociate;
>   static int max_jobs = -1;
> +static struct string_list submodule_groups;
>
>   static struct option builtin_clone_options[] = {
>   	OPT__VERBOSITY(&option_verbosity),
> @@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {
>   		   N_("separate git dir from working tree")),
>   	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
>   			N_("set config inside the new repository")),
> +	OPT_STRING_LIST('g', "group", &submodule_groups, N_("group"),
> +			N_("clone specific submodule groups")),
>   	OPT_END()
>   };
>
> @@ -723,9 +726,18 @@ static int checkout(void)
>   	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
>   			   sha1_to_hex(sha1), "1", NULL);
>
> -	if (!err && option_recursive) {
> +	if (err)
> +		goto out;
> +
> +	if (option_recursive || submodule_groups.nr > 0) {
>   		struct argv_array args = ARGV_ARRAY_INIT;
> -		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
> +		argv_array_pushl(&args, "submodule", "update", "--init", NULL);
> +
> +		if (option_recursive)
> +			argv_array_pushf(&args, "--recursive");
> +
> +		if (submodule_groups.nr > 0)
> +			argv_array_pushf(&args, "--groups");
>
>   		if (max_jobs != -1)
>   			argv_array_pushf(&args, "--jobs=%d", max_jobs);
> @@ -733,7 +745,7 @@ static int checkout(void)
>   		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
>   		argv_array_clear(&args);
>   	}
> -
> +out:
>   	return err;
>   }
>
> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>   		option_no_checkout = 1;
>   	}
>
> +	if (option_recursive && submodule_groups.nr > 0)
> +		die(_("submodule groups and recursive flag are incompatible"));

Me thinks this contradicts your description of the --group option
in the man page. I don't see why such a restriction would make
sense, what incompatibility are you trying to avoid here? Maybe
we need another submodule-specific setting to tell update what
groups to use inside that submodule?

> +	if (submodule_groups.nr > 0) {
> +		int first_item = 1;
> +		struct string_list_item *item;
> +		struct strbuf sb = STRBUF_INIT;
> +		strbuf_addstr(&sb, "submodule.groups=");
> +		for_each_string_list_item(item, &submodule_groups) {
> +			strbuf_addf(&sb, "%s%s", first_item ? "" : ",", item->string);
> +			first_item = 0;
> +		}
> +		if (submodule_groups.nr > 0)
> +			string_list_append(&option_config, strbuf_detach(&sb, 0));
> +	}
> +
>   	if (!option_origin)
>   		option_origin = "origin";
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 4092a48..e3d1667 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -611,6 +611,7 @@ cmd_deinit()
>   #
>   cmd_update()
>   {
> +	groups=
>   	# parse $args after "submodule ... update".
>   	while test $# -ne 0
>   	do
> @@ -650,6 +651,9 @@ cmd_update()
>   		--checkout)
>   			update="checkout"
>   			;;
> +		--groups)
> +			groups=1
> +			;;
>   		--depth)
>   			case "$2" in '') usage ;; esac
>   			depth="--depth=$2"
> @@ -691,6 +695,7 @@ cmd_update()
>   		${update:+--update "$update"} \
>   		${reference:+--reference "$reference"} \
>   		${depth:+--depth "$depth"} \
> +		${groups:+--groups} \
>   		${jobs:+$jobs} \
>   		"$@" | {
>   	err=
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index caed4be..e8654d7 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '
>   	)
>   '
>
> +cat <<EOF > expected
> +submodule
> +-submodule1
> +EOF
> +
> +test_expect_success 'submodule update --groups works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone super super_clone &&
> +	(
> +		cd super_clone &&
> +		git config submodule.groups groupA &&
> +		git submodule init  &&
> +		git submodule update --groups &&
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +test_expect_success 'submodule update --init --groups works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone super super_clone &&
> +	(
> +		cd super_clone &&
> +		git config submodule.groups groupA &&
> +		git submodule update --init --groups &&
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +test_expect_success 'clone --group works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone --group groupA super super_clone &&
> +	(
> +		cd super_clone &&
> +		test_pause
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +
>   test_done
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 090891e..7e59846 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'
>   	rm -rf super4
>   '
>
> +cat >expect <<-EOF &&
> +-deeper/submodule
> +-merging
> +-moved/sub module
> +-none
> +-rebasing
> +-submodule
> +-submodule1
> +EOF
> +
> +# none, merging rebasing, submodule1, submodule
> +test_expect_success 'git clone works with submodule groups.' '
> +	test_when_finished "rm -rf super5" &&
> +	(
> +		cd super &&
> +		git config -f .gitmodules  submodule.submodule.groups default &&
> +		git config -f .gitmodules  submodule.submodule1.groups "default,testing" &&
> +		git config -f .gitmodules  submodule.none.groups testing &&
> +		git commit -a -m "assigning groups to submodules"
> +	) &&
> +	git clone --group default --group testing super super5 &&
> +	(
> +		cd super5 &&
> +		git submodule status |cut -c1,43- >../actual
> +	) &&
> +	test_cmp actual expect
> +'
> +
> +test_expect_success 'git submodule update --groups' '
> +	true
> +'
> +
>   test_done
>

Re: [RFC PATCH 0/5] Submodule Groups

[Stefan Beller]

--cc Johannes Sixt

On Wed, Nov 25, 2015 at 9:35 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> [submodule "gcc"]
>>          path = gcc
>>          url = git://...
>>          groups = default,devel
>> [submodule "linux"]
>>          path = linux
>>          url = git://...
>>          groups = default
>> [submodule "nethack"]
>>          path = nethack
>>          url = git://...
>>          groups = optional,games
>
>
> Yup. Do you want the user to select only a single group or do you
> plan to support selecting multiple groups at the same time too?

Yes you should be able to select multiple groups, such as
default+devel or alternatively default+games.

The logical OR is supported in this patch series (all submodules which are
in at least one of the specified groups,i.e. A OR B OR C ...)

>
>> and by this series you can work on an arbitrary subgroup of these
>> submodules such
>> using these commands:
>>
>>      git clone --group default --group devel git://...
>>      # will clone the superproject and recursively
>>      # checkout any submodule being in at least one of the groups.
>
>
> Does this automatically configure the given group in .git/config, so
> that all future submodule related commands know about this choice?
> Me thinks that would make sense ...

It does. Internally it does

    git config submodule.groups A,B
    git submodule update --init --groups

whereas submodule update checks if the submodule.groups
value is set and if so operates on the groups only.

>
>>      git submodule add --group default --group devel git://... ..
>>      # will add a submodule, adding 2 submodule
>>      # groups to its entry in .gitmodule
>
>
> Maybe '--groups default,devel' is easier to grok? Dunno.

I guess that makes sense.

>
>>      # as support for clone we want to have:
>>      git config submodule.groups default
>>      git submodule init --groups
>
>
> Hmm, I doubt it makes much sense to add the --group option to "git
> submodule init". I'd rather init all submodules and do the group
> handling only in the "git submodule update" command. That way
> upstream can change grouping later without having the user to
> fiddle with her configuration to make that work.

Well if upstream changes grouping later, you could just run

    git submodule update --init --groups

and get what you want?

>
>>      # will init all submodules from the default group
>>
>>      # as support for clone we want to have:
>>      git config submodule.groups default
>>      git submodule update --groups
>>
>>      # will update all submodules from the default group
>>
>> Any feedback welcome, specially on the design level!
>> (Do we want to have it stored in the .gitmodules file? Do we want to have
>> the groups configured in .git/config as "submodule.groups", any other way
>> to make it future proof and extend the groups syntax?)
>
>
> Not sure what exactly you mean by "it" here ;-)
>
> Talking about what groups a submodule belongs to, an entry in the
> .gitmodules file makes the most sense to me. That way upstream can
> change submodule grouping or add new submodules with group assignments
> from commit to commit, and "git submodule update" will do the right
> thing for the superproject commit checked out.
>
> And I believe that the choice which group(s?) the user is interested
> should be recorded in .git/config, as that is his personal setting
> that shouldn't be influenced by upstream changes.

Right. I once discussed with Jonathan Nieder, who dreamed of a more
logical approach to the groups/sets of submodules. So more like set theory,
i.e. have a more complicated grammar: Get all submodules which are
in either A or B or (D AND E), but which are never in F.
So I'd imagine the groups are more like bit tags, and you can describe
a patterns you want.

I guess we want some more powerful eventually, so I asked this open ended
question there.

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Stefan Beller]

On Wed, Nov 25, 2015 at 9:52 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> +--group::
>> +       After the clone is created, all submodules which are part of the
>> +       group are cloned. This option can be given multiple times to
>> specify
>> +       different groups.
>
>
> Ah, that answers my question in my response to the cover letter ;-)
>
>> This option will imply automatic submodule
>> +       updates for the groups by setting `submodule.update=groups`.
>
>
> Please don't. The per-submodule update setting configures how a
> submodule has to be updated, adding a global one with a completely
> different meaning (what submodules should be updated?) is confusing.
> Why not "submodule.groups=<groups>"?

The documentation is out of date :/ as I was churning through lots of ideas,
so we do have a config submodule.groups=<groups> by now, but the
documentation is wrong.

>
>> +       The group selection will be passed on recursively, i.e. if a
>> submodule
>> +       is cloned because of group membership, its submodules will
>> +       be cloned according to group membership, too. If a submodule is
>> +       not cloned however, its submodules are not evaluated for group
>> +       membership.
>
>
> What do you mean by the last sentence? Did the clone fail? Then you
> cannot update the submodule anyway ...

Consider nested submodules:

    A: superproject containing
        B: which contains
            C.

If you clone A with group <C-but-not-B> you won't get C as we do not traverse
the submodules of B, as we don't clone B. Maybe it's obvious?

>> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char
>> *prefix)
>>                 option_no_checkout = 1;
>>         }
>>
>> +       if (option_recursive && submodule_groups.nr > 0)
>> +               die(_("submodule groups and recursive flag are
>> incompatible"));
>
>
> Me thinks this contradicts your description of the --group option
> in the man page. I don't see why such a restriction would make
> sense, what incompatibility are you trying to avoid here? Maybe
> we need another submodule-specific setting to tell update what
> groups to use inside that submodule?

So you want something like
    "In the top level respect the groups, but recursively get all of them"?

My thinking is that groups are implying recursive, whereas recursive implies
"all groups", so a git clone --group <half-the-submodules> --recursive
makes not much sense to me as it begs the question, what does --recursive
mean? Probably recurse into all submodules which are implied by the group
<half-the-submodules>. And then get all the nested submodules. But in case
you use the grouping feature, you could just mark the nested submodules with
groups, too?

Re: [RFC PATCH 0/5] Submodule Groups

[Jens Lehmann]

(Sorry for the resend of my last mail, but I received bounce messages
from my email provider)

Am 25.11.2015 um 19:00 schrieb Stefan Beller:
> --cc Johannes Sixt
>
> On Wed, Nov 25, 2015 at 9:35 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> [submodule "gcc"]
>>>           path = gcc
>>>           url = git://...
>>>           groups = default,devel
>>> [submodule "linux"]
>>>           path = linux
>>>           url = git://...
>>>           groups = default
>>> [submodule "nethack"]
>>>           path = nethack
>>>           url = git://...
>>>           groups = optional,games
>>
>>
>> Yup. Do you want the user to select only a single group or do you
>> plan to support selecting multiple groups at the same time too?
>
> Yes you should be able to select multiple groups, such as
> default+devel or alternatively default+games.
>
> The logical OR is supported in this patch series (all submodules which are
> in at least one of the specified groups,i.e. A OR B OR C ...)

Good, this is more flexible than restricting that to just a
single group.

>>> and by this series you can work on an arbitrary subgroup of these
>>> submodules such
>>> using these commands:
>>>
>>>       git clone --group default --group devel git://...
>>>       # will clone the superproject and recursively
>>>       # checkout any submodule being in at least one of the groups.
>>
>>
>> Does this automatically configure the given group in .git/config, so
>> that all future submodule related commands know about this choice?
>> Me thinks that would make sense ...
>
> It does. Internally it does
>
>      git config submodule.groups A,B
>      git submodule update --init --groups
>
> whereas submodule update checks if the submodule.groups
> value is set and if so operates on the groups only.

Makes sense (except for the "--groups" argument, see below ;-).

>>
>>>       # as support for clone we want to have:
>>>       git config submodule.groups default
>>>       git submodule init --groups
>>
>>
>> Hmm, I doubt it makes much sense to add the --group option to "git
>> submodule init". I'd rather init all submodules and do the group
>> handling only in the "git submodule update" command. That way
>> upstream can change grouping later without having the user to
>> fiddle with her configuration to make that work.
>
> Well if upstream changes grouping later, you could just run
>
>      git submodule update --init --groups
>
> and get what you want?

And make life harder than necessary for our users without having
a reason for that? Except for the URL copying submodule settings
on init is wrong, as it sets in stone what happened to be in the
.gitmodules file when you ran init and doesn't allow upstream to
easily change defaults later. We still do that with the update
setting for historical reasons, but I avoided making the same
mistake with all the options I added later. You can override
these settings if you want or need to, but that shouldn't be
necessary by default to make life easier for our users.

>>>       # will init all submodules from the default group
>>>
>>>       # as support for clone we want to have:
>>>       git config submodule.groups default
>>>       git submodule update --groups
>>>
>>>       # will update all submodules from the default group
>>>
>>> Any feedback welcome, specially on the design level!
>>> (Do we want to have it stored in the .gitmodules file? Do we want to have
>>> the groups configured in .git/config as "submodule.groups", any other way
>>> to make it future proof and extend the groups syntax?)
>>
>>
>> Not sure what exactly you mean by "it" here ;-)
>>
>> Talking about what groups a submodule belongs to, an entry in the
>> .gitmodules file makes the most sense to me. That way upstream can
>> change submodule grouping or add new submodules with group assignments
>> from commit to commit, and "git submodule update" will do the right
>> thing for the superproject commit checked out.
>>
>> And I believe that the choice which group(s?) the user is interested
>> should be recorded in .git/config, as that is his personal setting
>> that shouldn't be influenced by upstream changes.
>
> Right. I once discussed with Jonathan Nieder, who dreamed of a more
> logical approach to the groups/sets of submodules. So more like set theory,
> i.e. have a more complicated grammar: Get all submodules which are
> in either A or B or (D AND E), but which are never in F.
> So I'd imagine the groups are more like bit tags, and you can describe
> a patterns you want.

Ok, we can start with union and add intersection later when needed.

> I guess we want some more powerful eventually, so I asked this open ended
> question there.

And I don't think we need to implement everything right now, but we
should have thought things through as far as we can currently see,
to avoid running into problems later on ;-)

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Jens Lehmann]

Am 25.11.2015 um 19:08 schrieb Stefan Beller:
> On Wed, Nov 25, 2015 at 9:52 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> +--group::
>>> +       After the clone is created, all submodules which are part of the
>>> +       group are cloned. This option can be given multiple times to
>>> specify
>>> +       different groups.
>>
>>
>> Ah, that answers my question in my response to the cover letter ;-)
>>
>>> This option will imply automatic submodule
>>> +       updates for the groups by setting `submodule.update=groups`.
>>
>>
>> Please don't. The per-submodule update setting configures how a
>> submodule has to be updated, adding a global one with a completely
>> different meaning (what submodules should be updated?) is confusing.
>> Why not "submodule.groups=<groups>"?
>
> The documentation is out of date :/ as I was churning through lots of ideas,
> so we do have a config submodule.groups=<groups> by now, but the
> documentation is wrong.

Thanks for explaining, I did not look at the code very closely so
far so I missed that.

>>
>>> +       The group selection will be passed on recursively, i.e. if a
>>> submodule
>>> +       is cloned because of group membership, its submodules will
>>> +       be cloned according to group membership, too. If a submodule is
>>> +       not cloned however, its submodules are not evaluated for group
>>> +       membership.
>>
>>
>> What do you mean by the last sentence? Did the clone fail? Then you
>> cannot update the submodule anyway ...
>
> Consider nested submodules:
>
>      A: superproject containing
>          B: which contains
>              C.
>
> If you clone A with group <C-but-not-B> you won't get C as we do not traverse
> the submodules of B, as we don't clone B. Maybe it's obvious?

Maybe yes. Everything about submodule C is configured in B's
.gitmodules file, not in A's. So you cannot find submodule C
in A's .gitmodules (and it thus cannot be in one of A's submodule
groups either). And if cloning B fails, you have no .gitmodules
file to get the URL of C to clone it from in the first place. So
I think the concept 'group <C-but-not-B>' doesn't make any sense
when C is a submodule of B.

>>> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char
>>> *prefix)
>>>                  option_no_checkout = 1;
>>>          }
>>>
>>> +       if (option_recursive && submodule_groups.nr > 0)
>>> +               die(_("submodule groups and recursive flag are
>>> incompatible"));
>>
>>
>> Me thinks this contradicts your description of the --group option
>> in the man page. I don't see why such a restriction would make
>> sense, what incompatibility are you trying to avoid here? Maybe
>> we need another submodule-specific setting to tell update what
>> groups to use inside that submodule?
>
> So you want something like
>      "In the top level respect the groups, but recursively get all of them"?

Nope, only those that are chosen by the groups.

> My thinking is that groups are implying recursive, whereas recursive implies
> "all groups", so a git clone --group <half-the-submodules> --recursive
> makes not much sense to me as it begs the question, what does --recursive
> mean?

Groups are only about what submodules to update and have nothing to
do with recursion. It might make sense to imply recursion, but that's
just because that should have been the default for submodules from day
one. Recursion and groups are orthogonal, the first is about what to
do inside the submodules (carry on or not?) and the latter is about
what to do in the superproject (shall I update this submodule?).

 > Probably recurse into all submodules which are implied by the group
> <half-the-submodules>.

Yep. We also do not recurse into those submodules having set their
update setting to "none", so we do not do that for submodules not
in any chosen group either.

 > And then get all the nested submodules. But in case
> you use the grouping feature, you could just mark the nested submodules with
> groups, too?

Not in the top superproject. In a submodule you can specify new groups
for its sub-submodules, but these will in most cases be different from
those of the superproject.

Imagine I have this really cool Metaproject which contains the Android
superproject as a submodule. Those two will define different groups,
and when recursing into the android submodule I need to choose from the
Android specific groups. So my Metaproject's .gitmodules could look like
this:

[submodule "android"]
         path = android
         url = git://...
         groups = default,mobile
         subgroups = devel

"groups" tells git what superproject groups the android submodule
belongs to, and "subgroups" tells git what android submodules are
to be checked out when running recursively into it. If you do not
configure "subgroups", the whole android submodule is updated when
one of the groups "default" or "mobile" is chosen in the superproject.

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Stefan Beller]

On Wed, Nov 25, 2015 at 11:50 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>
>> My thinking is that groups are implying recursive, whereas recursive
>> implies
>> "all groups", so a git clone --group <half-the-submodules> --recursive
>> makes not much sense to me as it begs the question, what does --recursive
>> mean?
>
>
> Groups are only about what submodules to update and have nothing to
> do with recursion. It might make sense to imply recursion, but that's
> just because that should have been the default for submodules from day
> one. Recursion and groups are orthogonal, the first is about what to
> do inside the submodules (carry on or not?) and the latter is about
> what to do in the superproject (shall I update this submodule?).

I see. So we would not want to mutually exclude recurse and groups,
but rather have groups implies --recurse, but you are allowed to give
--no-recurse if you explicitely do not want to recurse into the subsubmodules.

>
>> Probably recurse into all submodules which are implied by the group
>>
>> <half-the-submodules>.
>
>
> Yep. We also do not recurse into those submodules having set their
> update setting to "none", so we do not do that for submodules not
> in any chosen group either.
>
>> And then get all the nested submodules. But in case
>>
>> you use the grouping feature, you could just mark the nested submodules
>> with
>> groups, too?
>
>
> Not in the top superproject. In a submodule you can specify new groups
> for its sub-submodules, but these will in most cases be different from
> those of the superproject.
>
> Imagine I have this really cool Metaproject which contains the Android
> superproject as a submodule. Those two will define different groups,
> and when recursing into the android submodule I need to choose from the
> Android specific groups. So my Metaproject's .gitmodules could look like
> this:
>
> [submodule "android"]
>         path = android
>         url = git://...
>         groups = default,mobile
>         subgroups = devel
>
> "groups" tells git what superproject groups the android submodule
> belongs to, and "subgroups" tells git what android submodules are
> to be checked out when running recursively into it. If you do not
> configure "subgroups", the whole android submodule is updated when
> one of the groups "default" or "mobile" is chosen in the superproject.

I like the concept of subgroups as it allows to have some control over
subsubmodules you may want to aggregate from a third party via the
middleman submodule.

I'd prefer to delay that feature though by not giving a high priority.
Also would you go with subsubgroups, too? When does the recursion
end? In case we have more than the union of groups, but also prohibitive
terms available, could subgroups clash with the submodules groups spec?

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Jens Lehmann]

Am 25.11.2015 um 21:03 schrieb Stefan Beller:
> On Wed, Nov 25, 2015 at 11:50 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>
>>> My thinking is that groups are implying recursive, whereas recursive
>>> implies
>>> "all groups", so a git clone --group <half-the-submodules> --recursive
>>> makes not much sense to me as it begs the question, what does --recursive
>>> mean?
>>
>>
>> Groups are only about what submodules to update and have nothing to
>> do with recursion. It might make sense to imply recursion, but that's
>> just because that should have been the default for submodules from day
>> one. Recursion and groups are orthogonal, the first is about what to
>> do inside the submodules (carry on or not?) and the latter is about
>> what to do in the superproject (shall I update this submodule?).
>
> I see. So we would not want to mutually exclude recurse and groups,
> but rather have groups implies --recurse, but you are allowed to give
> --no-recurse if you explicitely do not want to recurse into the subsubmodules.

Exactly.

>>> And then get all the nested submodules. But in case
>>>
>>> you use the grouping feature, you could just mark the nested submodules
>>> with
>>> groups, too?
>>
>>
>> Not in the top superproject. In a submodule you can specify new groups
>> for its sub-submodules, but these will in most cases be different from
>> those of the superproject.
>>
>> Imagine I have this really cool Metaproject which contains the Android
>> superproject as a submodule. Those two will define different groups,
>> and when recursing into the android submodule I need to choose from the
>> Android specific groups. So my Metaproject's .gitmodules could look like
>> this:
>>
>> [submodule "android"]
>>          path = android
>>          url = git://...
>>          groups = default,mobile
>>          subgroups = devel
>>
>> "groups" tells git what superproject groups the android submodule
>> belongs to, and "subgroups" tells git what android submodules are
>> to be checked out when running recursively into it. If you do not
>> configure "subgroups", the whole android submodule is updated when
>> one of the groups "default" or "mobile" is chosen in the superproject.
>
> I like the concept of subgroups as it allows to have some control over
> subsubmodules you may want to aggregate from a third party via the
> middleman submodule.

That's the point (though maybe someone might come up with a better
name than "subgroups" ;-). And each repo configures its own submodule
groups.

> I'd prefer to delay that feature though by not giving a high priority.

No problem, we can start with "check out all subsubmodules" for now.
But I suspect we'll need subgroups rather sooner than later.

> Also would you go with subsubgroups, too? When does the recursion
> end?

Subsubgroups do not make sense in the superproject, that can only
configure its direct submodules. I think you are talking about the
groups of the subsubmodules, and these have to be chosen inside the
first level submodules via the subgroups of its submodules (which
are the second level submodules of the superproject). Still with
me? ;-) So the recursion can go on forever even as soon as we
implement the subgroup configuration.

 > In case we have more than the union of groups, but also prohibitive
> terms available, could subgroups clash with the submodules groups spec?

Not that I'm aware of. Groups decide which submodules to update and
only for those submodules subgroups tell git what group to use inside
that submodule. And so on.

Re: [PATCH 5/5] builtin/clone: support submodule groups

[Stefan Beller]

On Wed, Nov 25, 2015 at 2:30 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>
>>
>> I like the concept of subgroups as it allows to have some control over
>> subsubmodules you may want to aggregate from a third party via the
>> middleman submodule.
>
>
> That's the point (though maybe someone might come up with a better
> name than "subgroups" ;-). And each repo configures its own submodule
> groups.
>
>> I'd prefer to delay that feature though by not giving a high priority.
>
>
> No problem, we can start with "check out all subsubmodules" for now.
> But I suspect we'll need subgroups rather sooner than later.

Oh!
I thought we'd recursively propagate the groups, so the subsubmodules
are checked to be either in groups or subgroups, and the subgroups are
just a way to enhance the union of groups.

>
>> Also would you go with subsubgroups, too? When does the recursion
>> end?
>
>
> Subsubgroups do not make sense in the superproject, that can only
> configure its direct submodules.

> I think you are talking about the
> groups of the subsubmodules, and these have to be chosen inside the
> first level submodules via the subgroups of its submodules (which
> are the second level submodules of the superproject). Still with
> me? ;-)

I believe so.

> So the recursion can go on forever even as soon as we
> implement the subgroup configuration.

So lets say you have your meta collection repository,
which looks like that:

operating systems:
    ubuntu
        linux
        nonfree-game
        ...
    gentoo
        ...
    fedora
        ...
    android
        linux
        linux-build-configs
            vendor-phones
            nexus-family

In the "operating systems" repo I have the submodules
ubuntu and android marked via a group: "work-related".

Now I want to specify to have the linux, linux-build-configs,
and in there the nexus-family in the android repository.

One way would be to have the "operating systems" repo to
have subsubgroups specifying the groups in the
submodules of linux-build-configs (3rd level of submodules).
You seem to oppose that.

The other way to do that, would be to have a fork of the android
repo and put in the right subgroups to select for the right submodules
in linux-build-configs. So forking and fixing the groups config
would be the way to make changes from upstream.

I personally would find it easier to have all the spec in the one
superproject repository as then I don't need to update the
forks. (the .gitmodules file would get some conflicts, in case of my
fork there, so it's not easy to maintain long term)

Is there yet another way to handle such a case of deeply nested
submodules properly?

>
>> In case we have more than the union of groups, but also prohibitive
>>
>> terms available, could subgroups clash with the submodules groups spec?
>
>
> Not that I'm aware of. Groups decide which submodules to update and
> only for those submodules subgroups tell git what group to use inside
> that submodule. And so on.

[PATCHv2] builtin/clone: support submodule groups

[Stefan Beller]

This passes each group to the `submodule update` invocation and
additionally configures the groups to be automatically updated.

Signed-off-by: Stefan Beller <sbeller@google.com>
---

This is a resend of the patch "[PATCH 5/5] builtin/clone: support submodule groups"
as that's where Jens and I discussed.

* reworded the documentation to match reality of the patch
* --recurse is now implied and can be turned off.

Thanks for the fast feedback,
Stefan

Interdiff to previous version [PATCH 5/5] builtin/clone: support submodule groups
        --- a/Documentation/git-clone.txt
        +++ b/Documentation/git-clone.txt
        @@ -211,14 +211,16 @@ objects from the source repository into a pack in the cloned repository.
         
         --group::
                After the clone is created, all submodules which are part of the
        -       group are cloned. This option can be given multiple times to specify
        -       different groups. This option will imply automatic submodule
        -       updates for the groups by setting `submodule.update=groups`.
        -       The group selection will be passed on recursively, i.e. if a submodule
        -       is cloned because of group membership, its submodules will
        -       be cloned according to group membership, too. If a submodule is
        -       not cloned however, its submodules are not evaluated for group
        -       membership.
        +       given groups are cloned. To specify multiple groups, you can either
        +       give the group argument multiple times or comma separate the groups.
        +       This option will be recorded in the `submodule.groups` config,
        +       which will affect the behavior of other submodule related commands,
        +       such as `git submodule update`.
        +       This option implies recursive submodule checkout. If you don't
        +       want to recurse into nested submodules, you need to specify
        +       `--no-recursive`. The group selection will be passed on recursively,
        +       i.e. if a submodule is cloned because of group membership, its
        +       submodules will be cloned according to group membership, too.
         
         --separate-git-dir=<git dir>::
                Instead of placing the cloned repository where it is supposed
        diff --git a/builtin/clone.c b/builtin/clone.c
        index 17e9f54..377c031 100644
        --- a/builtin/clone.c
        +++ b/builtin/clone.c
        @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
         };
         
         static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
        -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
        +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;
         static char *option_template, *option_depth;
         static char *option_origin = NULL;
         static char *option_branch = NULL;
        @@ -875,9 +875,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
                                die(_("--bare and --separate-git-dir are incompatible."));
                        option_no_checkout = 1;
                }
        -
        -       if (option_recursive && submodule_groups.nr > 0)
        -               die(_("submodule groups and recursive flag are incompatible"));
        +       if (option_recursive == -1) {
        +               if (submodule_groups.nr > 0)
        +                       option_recursive = 1; /* submodule groups implies recursive */
        +               else
        +                       option_recursive = 0; /* preserve historical default */
        +       }
                if (submodule_groups.nr > 0) {
                        int first_item = 1;
                        struct string_list_item *item;

Here comes the actual patch:

 Documentation/git-clone.txt | 13 +++++++++
 builtin/clone.c             | 38 ++++++++++++++++++++++---
 git-submodule.sh            |  5 ++++
 t/t7400-submodule-basic.sh  | 69 +++++++++++++++++++++++++++++++++++++++++++++
 t/t7406-submodule-update.sh | 32 +++++++++++++++++++++
 5 files changed, 153 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 59d8c67..2539fea 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -209,6 +209,19 @@ objects from the source repository into a pack in the cloned repository.
 	repository does not have a worktree/checkout (i.e. if any of
 	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
 
+--group::
+	After the clone is created, all submodules which are part of the
+	given groups are cloned. To specify multiple groups, you can either
+	give the group argument multiple times or comma separate the groups.
+	This option will be recorded in the `submodule.groups` config,
+	which will affect the behavior of other submodule related commands,
+	such as `git submodule update`.
+	This option implies recursive submodule checkout. If you don't
+	want to recurse into nested submodules, you need to specify
+	`--no-recursive`. The group selection will be passed on recursively,
+	i.e. if a submodule is cloned because of group membership, its
+	submodules will be cloned according to group membership, too.
+
 --separate-git-dir=<git dir>::
 	Instead of placing the cloned repository where it is supposed
 	to be, place the cloned repository at the specified directory,
diff --git a/builtin/clone.c b/builtin/clone.c
index ce578d2..377c031 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
-static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
+static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;
 static char *option_template, *option_depth;
 static char *option_origin = NULL;
 static char *option_branch = NULL;
@@ -51,6 +51,7 @@ static struct string_list option_config;
 static struct string_list option_reference;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list submodule_groups;
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {
 		   N_("separate git dir from working tree")),
 	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
 			N_("set config inside the new repository")),
+	OPT_STRING_LIST('g', "group", &submodule_groups, N_("group"),
+			N_("clone specific submodule groups")),
 	OPT_END()
 };
 
@@ -723,9 +726,18 @@ static int checkout(void)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (err)
+		goto out;
+
+	if (option_recursive || submodule_groups.nr > 0) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", "--init", NULL);
+
+		if (option_recursive)
+			argv_array_pushf(&args, "--recursive");
+
+		if (submodule_groups.nr > 0)
+			argv_array_pushf(&args, "--groups");
 
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
@@ -733,7 +745,7 @@ static int checkout(void)
 		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
 		argv_array_clear(&args);
 	}
-
+out:
 	return err;
 }
 
@@ -863,6 +875,24 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 			die(_("--bare and --separate-git-dir are incompatible."));
 		option_no_checkout = 1;
 	}
+	if (option_recursive == -1) {
+		if (submodule_groups.nr > 0)
+			option_recursive = 1; /* submodule groups implies recursive */
+		else
+			option_recursive = 0; /* preserve historical default */
+	}
+	if (submodule_groups.nr > 0) {
+		int first_item = 1;
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		strbuf_addstr(&sb, "submodule.groups=");
+		for_each_string_list_item(item, &submodule_groups) {
+			strbuf_addf(&sb, "%s%s", first_item ? "" : ",", item->string);
+			first_item = 0;
+		}
+		if (submodule_groups.nr > 0)
+			string_list_append(&option_config, strbuf_detach(&sb, 0));
+	}
 
 	if (!option_origin)
 		option_origin = "origin";
diff --git a/git-submodule.sh b/git-submodule.sh
index 4092a48..e3d1667 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -611,6 +611,7 @@ cmd_deinit()
 #
 cmd_update()
 {
+	groups=
 	# parse $args after "submodule ... update".
 	while test $# -ne 0
 	do
@@ -650,6 +651,9 @@ cmd_update()
 		--checkout)
 			update="checkout"
 			;;
+		--groups)
+			groups=1
+			;;
 		--depth)
 			case "$2" in '') usage ;; esac
 			depth="--depth=$2"
@@ -691,6 +695,7 @@ cmd_update()
 		${update:+--update "$update"} \
 		${reference:+--reference "$reference"} \
 		${depth:+--depth "$depth"} \
+		${groups:+--groups} \
 		${jobs:+$jobs} \
 		"$@" | {
 	err=
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index caed4be..e8654d7 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '
 	)
 '
 
+cat <<EOF > expected
+submodule
+-submodule1
+EOF
+
+test_expect_success 'submodule update --groups works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.groups groupA &&
+		git submodule init  &&
+		git submodule update --groups &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+test_expect_success 'submodule update --init --groups works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.groups groupA &&
+		git submodule update --init --groups &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone --group works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --group groupA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone --group groupA super super_clone &&
+	(
+		cd super_clone &&
+		test_pause
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+
 test_done
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 090891e..7e59846 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'
 	rm -rf super4
 '
 
+cat >expect <<-EOF &&
+-deeper/submodule
+-merging
+-moved/sub module
+-none
+-rebasing
+-submodule
+-submodule1
+EOF
+
+# none, merging rebasing, submodule1, submodule
+test_expect_success 'git clone works with submodule groups.' '
+	test_when_finished "rm -rf super5" &&
+	(
+		cd super &&
+		git config -f .gitmodules  submodule.submodule.groups default &&
+		git config -f .gitmodules  submodule.submodule1.groups "default,testing" &&
+		git config -f .gitmodules  submodule.none.groups testing &&
+		git commit -a -m "assigning groups to submodules"
+	) &&
+	git clone --group default --group testing super super5 &&
+	(
+		cd super5 &&
+		git submodule status |cut -c1,43- >../actual
+	) &&
+	test_cmp actual expect
+'
+
+test_expect_success 'git submodule update --groups' '
+	true
+'
+
 test_done
-- 
2.6.1.261.g0d9c4c1

Re: [PATCHv2] builtin/clone: support submodule groups

[Stefan Beller]

On Wed, Nov 25, 2015 at 4:31 PM, Stefan Beller <sbeller@google.com> wrote:
> This passes each group to the `submodule update` invocation and
> additionally configures the groups to be automatically updated.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>
> This is a resend of the patch "[PATCH 5/5] builtin/clone: support submodule groups"
> as that's where Jens and I discussed.
>
> * reworded the documentation to match reality of the patch

Oh wait, I just wrote down wishful thinking (at least partially)
You cannot quite specify the comma separated list yet in one --group parameter.

Re: [PATCHv2] builtin/clone: support submodule groups

[Trevor Saunders]

On Wed, Nov 25, 2015 at 04:31:24PM -0800, Stefan Beller wrote:
> This passes each group to the `submodule update` invocation and
> additionally configures the groups to be automatically updated.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> 
> This is a resend of the patch "[PATCH 5/5] builtin/clone: support submodule groups"
> as that's where Jens and I discussed.

Seeing the recent sparse checkout discussion I realized it might be
useful to have a similar sort of feature for sparse checkouts.  So say I
had a mobile and desktop client in the same repo and wanted to be able to
checkout the commoncode and one client without having to explicitly list
all the paths I care about. It seems like UI wise you might want to use
--group there too, or at least explaining the difference to users might
be interesting, but maybe that's worrying way too much abouta possible
future feature.

Trev

> 
> * reworded the documentation to match reality of the patch
> * --recurse is now implied and can be turned off.
> 
> Thanks for the fast feedback,
> Stefan
> 
> Interdiff to previous version [PATCH 5/5] builtin/clone: support submodule groups
>         --- a/Documentation/git-clone.txt
>         +++ b/Documentation/git-clone.txt
>         @@ -211,14 +211,16 @@ objects from the source repository into a pack in the cloned repository.
>          
>          --group::
>                 After the clone is created, all submodules which are part of the
>         -       group are cloned. This option can be given multiple times to specify
>         -       different groups. This option will imply automatic submodule
>         -       updates for the groups by setting `submodule.update=groups`.
>         -       The group selection will be passed on recursively, i.e. if a submodule
>         -       is cloned because of group membership, its submodules will
>         -       be cloned according to group membership, too. If a submodule is
>         -       not cloned however, its submodules are not evaluated for group
>         -       membership.
>         +       given groups are cloned. To specify multiple groups, you can either
>         +       give the group argument multiple times or comma separate the groups.
>         +       This option will be recorded in the `submodule.groups` config,
>         +       which will affect the behavior of other submodule related commands,
>         +       such as `git submodule update`.
>         +       This option implies recursive submodule checkout. If you don't
>         +       want to recurse into nested submodules, you need to specify
>         +       `--no-recursive`. The group selection will be passed on recursively,
>         +       i.e. if a submodule is cloned because of group membership, its
>         +       submodules will be cloned according to group membership, too.
>          
>          --separate-git-dir=<git dir>::
>                 Instead of placing the cloned repository where it is supposed
>         diff --git a/builtin/clone.c b/builtin/clone.c
>         index 17e9f54..377c031 100644
>         --- a/builtin/clone.c
>         +++ b/builtin/clone.c
>         @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
>          };
>          
>          static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
>         -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
>         +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;
>          static char *option_template, *option_depth;
>          static char *option_origin = NULL;
>          static char *option_branch = NULL;
>         @@ -875,9 +875,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>                                 die(_("--bare and --separate-git-dir are incompatible."));
>                         option_no_checkout = 1;
>                 }
>         -
>         -       if (option_recursive && submodule_groups.nr > 0)
>         -               die(_("submodule groups and recursive flag are incompatible"));
>         +       if (option_recursive == -1) {
>         +               if (submodule_groups.nr > 0)
>         +                       option_recursive = 1; /* submodule groups implies recursive */
>         +               else
>         +                       option_recursive = 0; /* preserve historical default */
>         +       }
>                 if (submodule_groups.nr > 0) {
>                         int first_item = 1;
>                         struct string_list_item *item;
> 
> Here comes the actual patch:
> 
>  Documentation/git-clone.txt | 13 +++++++++
>  builtin/clone.c             | 38 ++++++++++++++++++++++---
>  git-submodule.sh            |  5 ++++
>  t/t7400-submodule-basic.sh  | 69 +++++++++++++++++++++++++++++++++++++++++++++
>  t/t7406-submodule-update.sh | 32 +++++++++++++++++++++
>  5 files changed, 153 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index 59d8c67..2539fea 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -209,6 +209,19 @@ objects from the source repository into a pack in the cloned repository.
>  	repository does not have a worktree/checkout (i.e. if any of
>  	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
>  
> +--group::
> +	After the clone is created, all submodules which are part of the
> +	given groups are cloned. To specify multiple groups, you can either
> +	give the group argument multiple times or comma separate the groups.
> +	This option will be recorded in the `submodule.groups` config,
> +	which will affect the behavior of other submodule related commands,
> +	such as `git submodule update`.
> +	This option implies recursive submodule checkout. If you don't
> +	want to recurse into nested submodules, you need to specify
> +	`--no-recursive`. The group selection will be passed on recursively,
> +	i.e. if a submodule is cloned because of group membership, its
> +	submodules will be cloned according to group membership, too.
> +
>  --separate-git-dir=<git dir>::
>  	Instead of placing the cloned repository where it is supposed
>  	to be, place the cloned repository at the specified directory,
> diff --git a/builtin/clone.c b/builtin/clone.c
> index ce578d2..377c031 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
>  };
>  
>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
> -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
> +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;
>  static char *option_template, *option_depth;
>  static char *option_origin = NULL;
>  static char *option_branch = NULL;
> @@ -51,6 +51,7 @@ static struct string_list option_config;
>  static struct string_list option_reference;
>  static int option_dissociate;
>  static int max_jobs = -1;
> +static struct string_list submodule_groups;
>  
>  static struct option builtin_clone_options[] = {
>  	OPT__VERBOSITY(&option_verbosity),
> @@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {
>  		   N_("separate git dir from working tree")),
>  	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
>  			N_("set config inside the new repository")),
> +	OPT_STRING_LIST('g', "group", &submodule_groups, N_("group"),
> +			N_("clone specific submodule groups")),
>  	OPT_END()
>  };
>  
> @@ -723,9 +726,18 @@ static int checkout(void)
>  	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
>  			   sha1_to_hex(sha1), "1", NULL);
>  
> -	if (!err && option_recursive) {
> +	if (err)
> +		goto out;
> +
> +	if (option_recursive || submodule_groups.nr > 0) {
>  		struct argv_array args = ARGV_ARRAY_INIT;
> -		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
> +		argv_array_pushl(&args, "submodule", "update", "--init", NULL);
> +
> +		if (option_recursive)
> +			argv_array_pushf(&args, "--recursive");
> +
> +		if (submodule_groups.nr > 0)
> +			argv_array_pushf(&args, "--groups");
>  
>  		if (max_jobs != -1)
>  			argv_array_pushf(&args, "--jobs=%d", max_jobs);
> @@ -733,7 +745,7 @@ static int checkout(void)
>  		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
>  		argv_array_clear(&args);
>  	}
> -
> +out:
>  	return err;
>  }
>  
> @@ -863,6 +875,24 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>  			die(_("--bare and --separate-git-dir are incompatible."));
>  		option_no_checkout = 1;
>  	}
> +	if (option_recursive == -1) {
> +		if (submodule_groups.nr > 0)
> +			option_recursive = 1; /* submodule groups implies recursive */
> +		else
> +			option_recursive = 0; /* preserve historical default */
> +	}
> +	if (submodule_groups.nr > 0) {
> +		int first_item = 1;
> +		struct string_list_item *item;
> +		struct strbuf sb = STRBUF_INIT;
> +		strbuf_addstr(&sb, "submodule.groups=");
> +		for_each_string_list_item(item, &submodule_groups) {
> +			strbuf_addf(&sb, "%s%s", first_item ? "" : ",", item->string);
> +			first_item = 0;
> +		}
> +		if (submodule_groups.nr > 0)
> +			string_list_append(&option_config, strbuf_detach(&sb, 0));
> +	}
>  
>  	if (!option_origin)
>  		option_origin = "origin";
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 4092a48..e3d1667 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -611,6 +611,7 @@ cmd_deinit()
>  #
>  cmd_update()
>  {
> +	groups=
>  	# parse $args after "submodule ... update".
>  	while test $# -ne 0
>  	do
> @@ -650,6 +651,9 @@ cmd_update()
>  		--checkout)
>  			update="checkout"
>  			;;
> +		--groups)
> +			groups=1
> +			;;
>  		--depth)
>  			case "$2" in '') usage ;; esac
>  			depth="--depth=$2"
> @@ -691,6 +695,7 @@ cmd_update()
>  		${update:+--update "$update"} \
>  		${reference:+--reference "$reference"} \
>  		${depth:+--depth "$depth"} \
> +		${groups:+--groups} \
>  		${jobs:+$jobs} \
>  		"$@" | {
>  	err=
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index caed4be..e8654d7 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '
>  	)
>  '
>  
> +cat <<EOF > expected
> +submodule
> +-submodule1
> +EOF
> +
> +test_expect_success 'submodule update --groups works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone super super_clone &&
> +	(
> +		cd super_clone &&
> +		git config submodule.groups groupA &&
> +		git submodule init  &&
> +		git submodule update --groups &&
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +test_expect_success 'submodule update --init --groups works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone super super_clone &&
> +	(
> +		cd super_clone &&
> +		git config submodule.groups groupA &&
> +		git submodule update --init --groups &&
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +test_expect_success 'clone --group works' '
> +	test_when_finished "rm -rf super super_clone" &&
> +	mkdir super &&
> +	pwd=$(pwd) &&
> +	(
> +		cd super &&
> +		git init &&
> +		git submodule add --group groupA file://"$pwd"/example2 submodule &&
> +		git submodule add file://"$pwd"/example2 submodule1 &&
> +		git commit -a -m "create repository with 2 submodules, one is in a group"
> +	) &&
> +	git clone --group groupA super super_clone &&
> +	(
> +		cd super_clone &&
> +		test_pause
> +		git submodule status |cut -c1,42-52 | tr -d " " >../actual
> +	) &&
> +	test_cmp actual expected
> +'
> +
> +
>  test_done
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 090891e..7e59846 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'
>  	rm -rf super4
>  '
>  
> +cat >expect <<-EOF &&
> +-deeper/submodule
> +-merging
> +-moved/sub module
> +-none
> +-rebasing
> +-submodule
> +-submodule1
> +EOF
> +
> +# none, merging rebasing, submodule1, submodule
> +test_expect_success 'git clone works with submodule groups.' '
> +	test_when_finished "rm -rf super5" &&
> +	(
> +		cd super &&
> +		git config -f .gitmodules  submodule.submodule.groups default &&
> +		git config -f .gitmodules  submodule.submodule1.groups "default,testing" &&
> +		git config -f .gitmodules  submodule.none.groups testing &&
> +		git commit -a -m "assigning groups to submodules"
> +	) &&
> +	git clone --group default --group testing super super5 &&
> +	(
> +		cd super5 &&
> +		git submodule status |cut -c1,43- >../actual
> +	) &&
> +	test_cmp actual expect
> +'
> +
> +test_expect_success 'git submodule update --groups' '
> +	true
> +'
> +
>  test_done
> -- 
> 2.6.1.261.g0d9c4c1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCHv2] builtin/clone: support submodule groups

[Stefan Beller]

+ cc Duy, Michael, who discussed the sparse checkout recently

On Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders <tbsaunde@tbsaunde.org> wrote:
> Seeing the recent sparse checkout discussion I realized it might be
> useful to have a similar sort of feature for sparse checkouts.  So say I
> had a mobile and desktop client in the same repo and wanted to be able to
> checkout the commoncode and one client without having to explicitly list
> all the paths I care about. It seems like UI wise you might want to use
> --group there too, or at least explaining the difference to users might
> be interesting, but maybe that's worrying way too much abouta possible
> future feature.

For reference what I was proposing (coverletter [RFC PATCH 0/5]
Submodule Groups):
-->8--
This is also available at
https://github.com/stefanbeller/git/tree/submodule-groups
It applies on top of the submodule-parallel-patch series I sent a few
minutes ago.

Consider having a real large software project in Git with each component
in a submodule (such as an operating system, Android, Debian, Fedora,
no toy OS such as https://github.com/gittup/gittup as that doesn't quite
demonstrate the scale of the problem).

If you have lots of submodules, you probably don't need all of them at once,
but you have functional units. Some submodules are absolutely required,
some are optional and only for very specific purposes.

This patch series adds meaning to a "groups" field in the .gitmodules file.

So you could have a .gitmodules file such as:

[submodule "gcc"]
        path = gcc
        url = git://...
        groups = default,devel
[submodule "linux"]
        path = linux
        url = git://...
        groups = default
[submodule "nethack"]
        path = nethack
        url = git://...
        groups = optional,games

and by this series you can work on an arbitrary subgroup of these
submodules such
using these commands:

    git clone --group default --group devel git://...
    # will clone the superproject and recursively
    # checkout any submodule being in at least one of the groups.

    git submodule add --group default --group devel git://... ..
    # will add a submodule, adding 2 submodule
    # groups to its entry in .gitmodule

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule init --groups
    # will init all submodules from the default group

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule update --groups
    # will update all submodules from the default group

Any feedback welcome, specially on the design level!
(Do we want to have it stored in the .gitmodules file? Do we want to have
the groups configured in .git/config as "submodule.groups", any other way
to make it future proof and extend the groups syntax?)
-->8--

I think the biggest advantage with the groups is to have it not depending on the
path. Consider your one repository containing both mobile and desktop code,
where you have a sparse checkout for mobile.

Now what happens if files are renamed, i.e. the leading directory?
As this change comes in from your dear coworker, who has no idea how
your .git/info/sparse-checkout looks like, it may happen that more files appear
in your worktree as your patterns did not cover the renamed case.
Or some file contents go missing as they are in one of the ignored paths.

This groups feature would solve that as the groups are not dependent on
the paths or the data itself. However the groups in this proposal are only
meant to be applied on a submodule level, not in a repository itself.

How would you do that?
I could imagine a file like .gitgroups (It should be part of the repository,
such that everybody talks about the same groups), with a similar syntax like
.gitattributes where some file patterns are assigned one or more groups.
And in either .git/config (as it is for the submodules here) or in a file
.git/info/groups (as the sparse checkouts do it in that directory)
you'd configure the groups you are interested in.

It would be cool to have the same mechanism for both sparse-group-checkout
and submodules, so I'd propose to use a config option in .git/config,
such as checkout-group which covers both submodules as well as the
pattern groups as specified by .gitgroups.

---
Beware, this is just a first shot spinning around some ideas.

Thanks,
Stefan

Re: [RFC PATCH 0/5] Submodule Groups

[Stefan Beller]

On Wed, Nov 25, 2015 at 11:18 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>>
>>> Hmm, I doubt it makes much sense to add the --group option to "git
>>> submodule init". I'd rather init all submodules and do the group
>>> handling only in the "git submodule update" command. That way
>>> upstream can change grouping later without having the user to
>>> fiddle with her configuration to make that work.
>>

Mind to elaborate a bit more here?
The way I understand you now is to pass not --groups to init,
but init initializes all submodules. But that is worse IMHO
(In the naive way of dealing with groups in the first patch series)
as then we open up two possibilities:
 * a submodule which happened to be part of the repository
   when cloning is added to a new group, which a user has
   configured, on pulling, this is no problem, we just checkout
   the desired version of the submodule.
 * a submodule which was not part of the repository at the time
   of cloning, is added to the superproject with a group the user
   is subscribed to. This would not be checked out as it is uninitialized
   on disk.

So when a change of the set of submodules as defined by groups
occurs, that is the point in time, when we want to init/fetch/checkout
these submodules, no?

>>
>> Well if upstream changes grouping later, you could just run
>>
>>      git submodule update --init --groups
>>
>> and get what you want?
>
>
> And make life harder than necessary for our users without having
> a reason for that?

So if upstream changes groups, ideally we want to follow without much
hassle for the user. So a plain git pull should /just work/. (I am repeating
myself here I'd guess), we would need to react to that. if we drop the
--groups call to init, we'd still tell the user to run

     git submodule update

We do not need --groups any more in a later patch as instead of
passing in --groups we can detect for `git config submodule.groups`
to be available or not.

--init should not be needed as when the groups are there we automatically
init new submodules in the group set?

> Except for the URL copying submodule settings
> on init is wrong, as it sets in stone what happened to be in the
> .gitmodules file when you ran init and doesn't allow upstream to
> easily change defaults later. We still do that with the update
> setting for historical reasons, but I avoided making the same
> mistake with all the options I added later. You can override
> these settings if you want or need to, but that shouldn't be
> necessary by default to make life easier for our users.
>

Re: [PATCHv2] builtin/clone: support submodule groups

[Michael J Gruber]

Stefan Beller venit, vidit, dixit 30.11.2015 20:31:
> + cc Duy, Michael, who discussed the sparse checkout recently
> 
> On Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders <tbsaunde@tbsaunde.org> wrote:
>> Seeing the recent sparse checkout discussion I realized it might be
>> useful to have a similar sort of feature for sparse checkouts.  So say I
>> had a mobile and desktop client in the same repo and wanted to be able to
>> checkout the commoncode and one client without having to explicitly list
>> all the paths I care about. It seems like UI wise you might want to use
>> --group there too, or at least explaining the difference to users might
>> be interesting, but maybe that's worrying way too much abouta possible
>> future feature.
> 
> For reference what I was proposing (coverletter [RFC PATCH 0/5]
> Submodule Groups):
> -->8--
> This is also available at
> https://github.com/stefanbeller/git/tree/submodule-groups
> It applies on top of the submodule-parallel-patch series I sent a few
> minutes ago.
> 
> Consider having a real large software project in Git with each component
> in a submodule (such as an operating system, Android, Debian, Fedora,
> no toy OS such as https://github.com/gittup/gittup as that doesn't quite
> demonstrate the scale of the problem).
> 
> If you have lots of submodules, you probably don't need all of them at once,
> but you have functional units. Some submodules are absolutely required,
> some are optional and only for very specific purposes.
> 
> This patch series adds meaning to a "groups" field in the .gitmodules file.
> 
> So you could have a .gitmodules file such as:
> 
> [submodule "gcc"]
>         path = gcc
>         url = git://...
>         groups = default,devel
> [submodule "linux"]
>         path = linux
>         url = git://...
>         groups = default
> [submodule "nethack"]
>         path = nethack
>         url = git://...
>         groups = optional,games
> 
> and by this series you can work on an arbitrary subgroup of these
> submodules such
> using these commands:
> 
>     git clone --group default --group devel git://...
>     # will clone the superproject and recursively
>     # checkout any submodule being in at least one of the groups.
> 
>     git submodule add --group default --group devel git://... ..
>     # will add a submodule, adding 2 submodule
>     # groups to its entry in .gitmodule
> 
>     # as support for clone we want to have:
>     git config submodule.groups default
>     git submodule init --groups
>     # will init all submodules from the default group
> 
>     # as support for clone we want to have:
>     git config submodule.groups default
>     git submodule update --groups
>     # will update all submodules from the default group
> 
> Any feedback welcome, specially on the design level!
> (Do we want to have it stored in the .gitmodules file? Do we want to have
> the groups configured in .git/config as "submodule.groups", any other way
> to make it future proof and extend the groups syntax?)
> -->8--
> 
> I think the biggest advantage with the groups is to have it not depending on the
> path. Consider your one repository containing both mobile and desktop code,
> where you have a sparse checkout for mobile.
> 
> Now what happens if files are renamed, i.e. the leading directory?
> As this change comes in from your dear coworker, who has no idea how
> your .git/info/sparse-checkout looks like, it may happen that more files appear
> in your worktree as your patterns did not cover the renamed case.
> Or some file contents go missing as they are in one of the ignored paths.
> 
> This groups feature would solve that as the groups are not dependent on
> the paths or the data itself. However the groups in this proposal are only
> meant to be applied on a submodule level, not in a repository itself.
> 
> How would you do that?
> I could imagine a file like .gitgroups (It should be part of the repository,
> such that everybody talks about the same groups), with a similar syntax like
> .gitattributes where some file patterns are assigned one or more groups.
> And in either .git/config (as it is for the submodules here) or in a file
> .git/info/groups (as the sparse checkouts do it in that directory)
> you'd configure the groups you are interested in.
> 
> It would be cool to have the same mechanism for both sparse-group-checkout
> and submodules, so I'd propose to use a config option in .git/config,
> such as checkout-group which covers both submodules as well as the
> pattern groups as specified by .gitgroups.
> 
> ---
> Beware, this is just a first shot spinning around some ideas.
> 
> Thanks,
> Stefan

I think we have to solve more basic issues for sparse checkouts first.
I'm using them with extra worktrees now and everything seems to be
working fine. But we need to get the UI right for the simple case (no
submodules, maybe not even extra worktrees) first: setting up patterns
before checkout etc. Having submodules in mind doesn't hurt, tough.

I still consider sparse checkouts a local "cludge" (not technically
cludgy) in the sense that it helps you cater to some specific local
needs; not something whose config you'd want to transport as part of the
object store.

Minor implementation detail: Do we have any precedence of comma
separated values for config values? I'd say we rather use multiple
entries, don't we?

Michael

Re: [PATCHv2] builtin/clone: support submodule groups

[Stefan Beller]

On Mon, Nov 30, 2015 at 10:53 PM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
> I think we have to solve more basic issues for sparse checkouts first.
> I'm using them with extra worktrees now and everything seems to be
> working fine. But we need to get the UI right for the simple case (no
> submodules, maybe not even extra worktrees) first: setting up patterns
> before checkout etc. Having submodules in mind doesn't hurt, tough.

Well my thinking comes from the other side: "I want to improve submodule
handling, but do I need to pay any attention to sparse checkout?", as Trevor
pointed out, this may or may not be similar enough from a users perspective,
that we want to have a similar/same UI there.

>
> I still consider sparse checkouts a local "cludge" (not technically
> cludgy) in the sense that it helps you cater to some specific local
> needs; not something whose config you'd want to transport as part of the
> object store.

Right, the submodule groups would be in the same boat. Each user would decide
locally what groups they think is worth having. Unlike the sparse checkout
the repository contains the groups however. As fair as I understand the sparse
checkout you would specify to checkout /foo/* but not checkout /bar/*

Now it is likely that some people will have very similar preferences for their
sparse checkout, so it may make sense to add an abstraction layer in there,
which can be done by groups. These groups could be defined using similar
patterns as in .gitattributes or .gitignore in another .gitgroups file. Maybe
the .gitattributes file could be reused.

The definition of the groups would be in the repository, such that it is kept
maintained and the individual user only needs to specify a few groups they're
interested in.

Currently you can already checkout submodules in a sparse fashion by just
initializing and checking out those submodules you want. But I think this
is not feasible if you have a huge amount of submodules, because you cannot
apply file patterns like you could with a .git{attributes, ignore, groups}
file. Because of the missing pattern, I'd want to add the groups.

>
> Minor implementation detail: Do we have any precedence of comma
> separated values for config values? I'd say we rather use multiple
> entries, don't we?

Ok, I'll fix that.

>
> Michael

Re: [RFC PATCH 0/5] Submodule Groups

[Jens Lehmann]

Am 01.12.2015 um 00:54 schrieb Stefan Beller:
> On Wed, Nov 25, 2015 at 11:18 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>>>
>>>> Hmm, I doubt it makes much sense to add the --group option to "git
>>>> submodule init". I'd rather init all submodules and do the group
>>>> handling only in the "git submodule update" command. That way
>>>> upstream can change grouping later without having the user to
>>>> fiddle with her configuration to make that work.
>>>
>
> Mind to elaborate a bit more here?
> The way I understand you now is to pass not --groups to init,
> but init initializes all submodules. But that is worse IMHO

Hmm, I did not mean to imply that "git init" should initialize
all submodules. Me thinks that "git clone --groups" should do
that but then only fetch and checkout those submodules the
chosen groups select. I expect "git submodule init" to be
obsolete when submodule groups (or recursive update) are used,
and that's why IMO it doesn't need a --groups option. (If the
user wants to change the groups later we might need to teach
"git submodule sync" the --groups option though)

> (In the naive way of dealing with groups in the first patch series)
> as then we open up two possibilities:
>   * a submodule which happened to be part of the repository
>     when cloning is added to a new group, which a user has
>     configured, on pulling, this is no problem, we just checkout
>     the desired version of the submodule.

That'll only work automatically when we follow my proposal to
init all those submodules present on clone, because otherwise
it won't be initialized.

>   * a submodule which was not part of the repository at the time
>     of cloning, is added to the superproject with a group the user
>     is subscribed to. This would not be checked out as it is uninitialized
>     on disk.

That's why I propose a mechanism to "auto-init" new submodules
on fetching their gitlink in the superproject. Then both your
groups proposal and my recursive update could make them appear
in the work tree on the next update/checkout. And as fetch is
part of clone, I'd expect clone to "auto-init" all submodules
referenced in gitlinks too.

> So when a change of the set of submodules as defined by groups
> occurs, that is the point in time, when we want to init/fetch/checkout
> these submodules, no?
>
>>>
>>> Well if upstream changes grouping later, you could just run
>>>
>>>       git submodule update --init --groups
>>>
>>> and get what you want?
>>
>>
>> And make life harder than necessary for our users without having
>> a reason for that?
>
> So if upstream changes groups, ideally we want to follow without much
> hassle for the user. So a plain git pull should /just work/. (I am repeating
> myself here I'd guess), we would need to react to that. if we drop the
> --groups call to init, we'd still tell the user to run
>
>       git submodule update

Sure, that's still needed until we have recursive update.

> We do not need --groups any more in a later patch as instead of
> passing in --groups we can detect for `git config submodule.groups`
> to be available or not.

Yes.

> --init should not be needed as when the groups are there we automatically
> init new submodules in the group set?

Right.

>> Except for the URL copying submodule settings
>> on init is wrong, as it sets in stone what happened to be in the
>> .gitmodules file when you ran init and doesn't allow upstream to
>> easily change defaults later. We still do that with the update
>> setting for historical reasons, but I avoided making the same
>> mistake with all the options I added later. You can override
>> these settings if you want or need to, but that shouldn't be
>> necessary by default to make life easier for our users.
>>
>