[PATCH v3 0/2] Add git-for-each-repo

[PATCH v3 0/2] Add git-for-each-repo

[Lars Hjemli]

Lars Hjemli (2):
  for-each-repo: new command used for multi-repo operations
  git: rewrite `git -a` to become a git-for-each-repo command

 .gitignore                          |   1 +
 Documentation/git-for-each-repo.txt |  62 +++++++++++++++++++
 Makefile                            |   1 +
 builtin.h                           |   1 +
 builtin/for-each-repo.c             | 119 ++++++++++++++++++++++++++++++++++++
 git.c                               |  37 +++++++++++
 t/t6400-for-each-repo.sh            |  54 ++++++++++++++++
 7 files changed, 275 insertions(+)
 create mode 100644 Documentation/git-for-each-repo.txt
 create mode 100644 builtin/for-each-repo.c
 create mode 100755 t/t6400-for-each-repo.sh

-- 
1.8.1.1.350.g3346805

[PATCH v3 1/2] for-each-repo: new command used for multi-repo operations

[Lars Hjemli]

When working with multiple, unrelated (or loosly related) git repos,
there is often a need to locate all repos with uncommitted work and
perform some action on them (say, commit and push). Before this patch,
such tasks would require manually visiting all repositories, running
`git status` within each one and then decide what to do next.

This mundane task can now be automated by e.g. `git for-each-repo --dirty
status`, which will find all git repositories below the current directory
(even nested ones), check if they are dirty (as defined by `git diff --quiet
&& git diff --cached --quiet`), and for each dirty repo print the path to
the repo and then execute `git status` within the repo.

The command also honours the option '--clean' which restricts the set of
repos to those which '--dirty' would skip.

Finally, the command to execute within each repo is optional. If none is
given, git-for-each-repo will just print the path to each repo found.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 .gitignore                          |   1 +
 Documentation/git-for-each-repo.txt |  62 +++++++++++++++++++
 Makefile                            |   1 +
 builtin.h                           |   1 +
 builtin/for-each-repo.c             | 119 ++++++++++++++++++++++++++++++++++++
 git.c                               |   1 +
 t/t6400-for-each-repo.sh            |  48 +++++++++++++++
 7 files changed, 233 insertions(+)
 create mode 100644 Documentation/git-for-each-repo.txt
 create mode 100644 builtin/for-each-repo.c
 create mode 100755 t/t6400-for-each-repo.sh

diff --git a/.gitignore b/.gitignore
index 63d4904..5036b84 100644
--- a/.gitignore
+++ b/.gitignore
@@ -56,6 +56,7 @@
 /git-filter-branch
 /git-fmt-merge-msg
 /git-for-each-ref
+/git-for-each-repo
 /git-format-patch
 /git-fsck
 /git-fsck-objects
diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt
new file mode 100644
index 0000000..be49e96
--- /dev/null
+++ b/Documentation/git-for-each-repo.txt
@@ -0,0 +1,62 @@
+git-for-each-repo(1)
+====================
+
+NAME
+----
+git-for-each-repo - Execute a git command in multiple repositories
+
+SYNOPSIS
+--------
+[verse]
+'git for-each-repo' [--all|--clean|--dirty] [command]
+
+DESCRIPTION
+-----------
+The git-for-each-repo command is used to locate all git repositoris
+within the current directory tree, and optionally execute a git command
+in each of the found repos.
+
+OPTIONS
+-------
+-a::
+--all::
+	Include both clean and dirty repositories (this is the default
+	behaviour of `git-for-each-repo`).
+
+-c::
+--clean::
+	Only include repositories with a clean worktree.
+
+-d::
+--dirty::
+	Only include repositories with a dirty worktree.
+
+EXAMPLES
+--------
+
+Various ways to exploit this command::
++
+------------
+$ git for-each-repo            <1>
+$ git for-each-repo fetch      <2>
+$ git for-each-repo -d gui     <3>
+$ git for-each-repo -c push    <4>
+------------
++
+<1> Print the path to all repos found below the current directory.
+
+<2> Fetch updates from default remote in all repos.
+
+<3> Start linkgit:git-gui[1] in each repo containing uncommitted changes.
+
+<4> Push the current branch in each repo with no uncommited changes.
+
+NOTES
+-----
+
+For the purpose of `git-for-each-repo`, a dirty worktree is defined as a
+worktree with uncommitted changes.
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/Makefile b/Makefile
index a786d4c..8c42c17 100644
--- a/Makefile
+++ b/Makefile
@@ -870,6 +870,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o
 BUILTIN_OBJS += builtin/fetch.o
 BUILTIN_OBJS += builtin/fmt-merge-msg.o
 BUILTIN_OBJS += builtin/for-each-ref.o
+BUILTIN_OBJS += builtin/for-each-repo.o
 BUILTIN_OBJS += builtin/fsck.o
 BUILTIN_OBJS += builtin/gc.o
 BUILTIN_OBJS += builtin/grep.o
diff --git a/builtin.h b/builtin.h
index 7e7bbd6..02fc712 100644
--- a/builtin.h
+++ b/builtin.h
@@ -73,6 +73,7 @@ extern int cmd_fetch(int argc, const char **argv, const char *prefix);
 extern int cmd_fetch_pack(int argc, const char **argv, const char *prefix);
 extern int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);
 extern int cmd_for_each_ref(int argc, const char **argv, const char *prefix);
+extern int cmd_for_each_repo(int argc, const char **argv, const char *prefix);
 extern int cmd_format_patch(int argc, const char **argv, const char *prefix);
 extern int cmd_fsck(int argc, const char **argv, const char *prefix);
 extern int cmd_gc(int argc, const char **argv, const char *prefix);
diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c
new file mode 100644
index 0000000..9bdeb4a
--- /dev/null
+++ b/builtin/for-each-repo.c
@@ -0,0 +1,119 @@
+/*
+ * "git for-each-repo" builtin command.
+ *
+ * Copyright (c) 2013 Lars Hjemli <hjemli@gmail.com>
+ */
+#include "cache.h"
+#include "color.h"
+#include "builtin.h"
+#include "run-command.h"
+#include "parse-options.h"
+
+#define ALL 0
+#define DIRTY 1
+#define CLEAN 2
+
+static int match;
+
+static const char * const builtin_foreachrepo_usage[] = {
+	N_("git for-each-repo [--all|--clean|--dirty] [cmd]"),
+	NULL
+};
+
+static struct option builtin_foreachrepo_options[] = {
+	OPT_SET_INT('a', "all", &match, N_("match both clean and dirty repositories"), ALL),
+	OPT_SET_INT('c', "clean", &match, N_("only show clean repositories"), CLEAN),
+	OPT_SET_INT('d', "dirty", &match, N_("only show dirty repositories"), DIRTY),
+	OPT_END(),
+};
+
+static int get_repo_state()
+{
+	const char *diffidx[] = {"diff", "--quiet", "--cached", NULL};
+	const char *diffwd[] = {"diff", "--quiet", NULL};
+
+	if (run_command_v_opt(diffidx, RUN_GIT_CMD) != 0)
+		return DIRTY;
+	if (run_command_v_opt(diffwd, RUN_GIT_CMD) != 0)
+		return DIRTY;
+	return CLEAN;
+}
+
+static void handle_repo(char *path, const char **argv)
+{
+	if (path[0] == '.' && path[1] == '/')
+		path += 2;
+	if (match != ALL && match != get_repo_state())
+		return;
+	if (*argv) {
+		color_fprintf_ln(stdout, GIT_COLOR_YELLOW, "[%s]", path);
+		run_command_v_opt(argv, RUN_GIT_CMD);
+	} else
+		printf("%s\n", path);
+}
+
+static int walk(struct strbuf *path, int argc, const char **argv)
+{
+	DIR *dir;
+	struct dirent *ent;
+	struct stat st;
+	size_t len;
+	const char *gitdir;
+
+	dir = opendir(path->buf);
+	if (!dir)
+		return errno;
+	strbuf_addstr(path, "/");
+	len = path->len;
+	while ((ent = readdir(dir))) {
+		if (!strcmp(ent->d_name, ".") || !strcmp(ent->d_name, ".."))
+			continue;
+		if (!strcmp(ent->d_name, ".git")) {
+			strbuf_addstr(path, ent->d_name);
+			gitdir = resolve_gitdir(path->buf);
+			if (!gitdir) {
+				strbuf_setlen(path, len - 1);
+				strbuf_addstr(path, "/");
+				continue;
+			}
+			setenv(GIT_DIR_ENVIRONMENT, gitdir, 1);
+			strbuf_setlen(path, len - 1);
+			setenv(GIT_WORK_TREE_ENVIRONMENT, path->buf, 1);
+			handle_repo(path->buf, argv);
+			strbuf_addstr(path, "/");
+			continue;
+		}
+		strbuf_setlen(path, len);
+		strbuf_addstr(path, ent->d_name);
+		switch (DTYPE(ent)) {
+		case DT_UNKNOWN:
+		case DT_LNK:
+			/* Use stat() to figure out if this path leads
+			 * to a directory - it's  not important if it's
+			 * a symlink which gets us there.
+			 */
+			if (stat(path->buf, &st) || !S_ISDIR(st.st_mode))
+				break;
+			/* fallthrough */
+		case DT_DIR:
+			walk(path, argc, argv);
+			break;
+		}
+		strbuf_setlen(path, len);
+	}
+	closedir(dir);
+	return 0;
+}
+
+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, prefix,
+			     builtin_foreachrepo_options,
+			     builtin_foreachrepo_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
+
+	strbuf_addstr(&path, ".");
+	return walk(&path, argc, argv);
+}
diff --git a/git.c b/git.c
index ed66c66..6b53169 100644
--- a/git.c
+++ b/git.c
@@ -337,6 +337,7 @@ static void handle_internal_command(int argc, const char **argv)
 		{ "fetch-pack", cmd_fetch_pack, RUN_SETUP },
 		{ "fmt-merge-msg", cmd_fmt_merge_msg, RUN_SETUP },
 		{ "for-each-ref", cmd_for_each_ref, RUN_SETUP },
+		{ "for-each-repo", cmd_for_each_repo },
 		{ "format-patch", cmd_format_patch, RUN_SETUP },
 		{ "fsck", cmd_fsck, RUN_SETUP },
 		{ "fsck-objects", cmd_fsck, RUN_SETUP },
diff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh
new file mode 100755
index 0000000..4797629
--- /dev/null
+++ b/t/t6400-for-each-repo.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Copyright (c) 2013 Lars Hjemli
+#
+
+test_description='Test the git-for-each-repo command'
+
+. ./test-lib.sh
+
+test_expect_success "setup" '
+	test_create_repo clean &&
+	(cd clean && test_commit foo) &&
+	git init --separate-git-dir=.cleansub clean/gitfile &&
+	(cd clean/gitfile && test_commit foo && echo bar >>foo.t) &&
+	test_create_repo dirty-wt &&
+	(cd dirty-wt && mv .git .linkedgit && ln -s .linkedgit .git &&
+	  test_commit foo && rm foo.t) &&
+	test_create_repo dirty-idx &&
+	(cd dirty-idx && test_commit foo && git rm foo.t) &&
+	mkdir fakedir && mkdir fakedir/.git
+'
+
+test_expect_success "without flags, all repos are included" '
+	echo "." >expect &&
+	echo "clean" >>expect &&
+	echo "clean/gitfile" >>expect &&
+	echo "dirty-idx" >>expect &&
+	echo "dirty-wt" >>expect &&
+	git for-each-repo | sort >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success "--dirty only includes dirty repos" '
+	echo "clean/gitfile" >expect &&
+	echo "dirty-idx" >>expect &&
+	echo "dirty-wt" >>expect &&
+	git for-each-repo --dirty | sort >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success "--clean only includes clean repos" '
+	echo "." >expect &&
+	echo "clean" >>expect &&
+	git for-each-repo --clean | sort >actual &&
+	test_cmp expect actual
+'
+
+test_done
-- 
1.8.1.1.350.g3346805

[PATCH v3 2/2] git: rewrite `git -a` to become a git-for-each-repo command

[Lars Hjemli]

With this rewriting, it is now possible to run e.g. `git -ad gui` to
start up git-gui in each repo within the current directory which
contains uncommited work.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 git.c                    | 36 ++++++++++++++++++++++++++++++++++++
 t/t6400-for-each-repo.sh |  6 ++++++
 2 files changed, 42 insertions(+)

diff --git a/git.c b/git.c
index 6b53169..f933b5d 100644
--- a/git.c
+++ b/git.c
@@ -31,8 +31,42 @@ static void commit_pager_choice(void) {
 	}
 }
 
+/*
+ * Rewrite 'git -ad status' to 'git for-each-repo -d status'
+ */
+static int rewrite_foreach_repo(const char ***orig_argv,
+				const char **curr_argv,
+				int *curr_argc)
+{
+	const char **new_argv;
+	char *tmp;
+	int new_argc, curr_pos, i, j;
+
+	curr_pos = curr_argv - *orig_argv;
+	if (strlen(curr_argv[0]) == 2) {
+		curr_argv[0] = "for-each-repo";
+		return curr_pos - 1;
+	}
+
+	new_argc = curr_pos + *curr_argc + 1;
+	new_argv = xmalloc(new_argc * sizeof(void *));
+	for (i = j = 0; j < new_argc; i++, j++) {
+		if (i == curr_pos) {
+			asprintf(&tmp, "-%s", (*orig_argv)[i] + 2);
+			new_argv[j] = "for-each-repo";
+			new_argv[++j] = tmp;
+		} else {
+			new_argv[j] = (*orig_argv)[i];
+		}
+	}
+	*orig_argv = new_argv;
+	(*curr_argc)++;
+	return curr_pos;
+}
+
 static int handle_options(const char ***argv, int *argc, int *envchanged)
 {
+	const char ***pargv = argv;
 	const char **orig_argv = *argv;
 
 	while (*argc > 0) {
@@ -143,6 +177,8 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)
 			setenv(GIT_LITERAL_PATHSPECS_ENVIRONMENT, "0", 1);
 			if (envchanged)
 				*envchanged = 1;
+		} else if (!strncmp(cmd, "-a", 2)) {
+			return rewrite_foreach_repo(pargv, *argv, argc);
 		} else {
 			fprintf(stderr, "Unknown option: %s\n", cmd);
 			usage(git_usage_string);
diff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh
index 4797629..b501605 100755
--- a/t/t6400-for-each-repo.sh
+++ b/t/t6400-for-each-repo.sh
@@ -27,6 +27,8 @@ test_expect_success "without flags, all repos are included" '
 	echo "dirty-idx" >>expect &&
 	echo "dirty-wt" >>expect &&
 	git for-each-repo | sort >actual &&
+	test_cmp expect actual &&
+	git -a | sort >actual &&
 	test_cmp expect actual
 '
 
@@ -35,6 +37,8 @@ test_expect_success "--dirty only includes dirty repos" '
 	echo "dirty-idx" >>expect &&
 	echo "dirty-wt" >>expect &&
 	git for-each-repo --dirty | sort >actual &&
+	test_cmp expect actual &&
+	git -ad | sort >actual &&
 	test_cmp expect actual
 '
 
@@ -42,6 +46,8 @@ test_expect_success "--clean only includes clean repos" '
 	echo "." >expect &&
 	echo "clean" >>expect &&
 	git for-each-repo --clean | sort >actual &&
+	test_cmp expect actual &&
+	git -ac | sort >actual &&
 	test_cmp expect actual
 '
 
-- 
1.8.1.1.350.g3346805

Re: [PATCH v3 1/2] for-each-repo: new command used for multi-repo operations

[Junio C Hamano]

Lars Hjemli <hjemli@gmail.com> writes:

> diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt
> new file mode 100644
> index 0000000..be49e96
> --- /dev/null
> +++ b/Documentation/git-for-each-repo.txt
> @@ -0,0 +1,62 @@
> +git-for-each-repo(1)
> +====================
> +
> +NAME
> +----
> +git-for-each-repo - Execute a git command in multiple repositories

"multiple non-bare repositories", I think.

> +
> +SYNOPSIS
> +--------
> +[verse]
> +'git for-each-repo' [--all|--clean|--dirty] [command]
> +
> +DESCRIPTION
> +-----------
> +The git-for-each-repo command is used to locate all git repositoris

Likewise; "all non-bare Git repositories".

> diff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh
> new file mode 100755
> index 0000000..4797629
> --- /dev/null
> +++ b/t/t6400-for-each-repo.sh
> @@ -0,0 +1,48 @@
> +#!/bin/sh
> +#
> +# Copyright (c) 2013 Lars Hjemli
> +#
> +
> +test_description='Test the git-for-each-repo command'
> +
> +. ./test-lib.sh
> +
> +test_expect_success "setup" '
> +	test_create_repo clean &&
> +	(cd clean && test_commit foo) &&
> +	git init --separate-git-dir=.cleansub clean/gitfile &&
> +	(cd clean/gitfile && test_commit foo && echo bar >>foo.t) &&
> +	test_create_repo dirty-wt &&
> +	(cd dirty-wt && mv .git .linkedgit && ln -s .linkedgit .git &&
> +	  test_commit foo && rm foo.t) &&
> +	test_create_repo dirty-idx &&
> +	(cd dirty-idx && test_commit foo && git rm foo.t) &&
> +	mkdir fakedir && mkdir fakedir/.git
> +'
> +
> +test_expect_success "without flags, all repos are included" '
> +	echo "." >expect &&
> +	echo "clean" >>expect &&
> +	echo "clean/gitfile" >>expect &&
> +	echo "dirty-idx" >>expect &&
> +	echo "dirty-wt" >>expect &&
> +	git for-each-repo | sort >actual &&
> +	test_cmp expect actual
> +'
> +
> +test_expect_success "--dirty only includes dirty repos" '
> +	echo "clean/gitfile" >expect &&
> +	echo "dirty-idx" >>expect &&
> +	echo "dirty-wt" >>expect &&
> +	git for-each-repo --dirty | sort >actual &&
> +	test_cmp expect actual
> +'
> +
> +test_expect_success "--clean only includes clean repos" '
> +	echo "." >expect &&
> +	echo "clean" >>expect &&
> +	git for-each-repo --clean | sort >actual &&
> +	test_cmp expect actual
> +'

Please add tests to show some command executions (e.g. test output
from "git ls-files", or something).

> +static void handle_repo(char *path, const char **argv)
> +{
> +	if (path[0] == '.' && path[1] == '/')
> +		path += 2;
> +	if (match != ALL && match != get_repo_state())
> +		return;
> +	if (*argv) {
> +		color_fprintf_ln(stdout, GIT_COLOR_YELLOW, "[%s]", path);
> +		run_command_v_opt(argv, RUN_GIT_CMD);

This seems to allow people to run only a single Git subcommand,
which is probably not what most people want to see.  Don't we want
to support something as simple as this?

	git for-each-repository sh -c "ls *.c"

> +	} else
> +		printf("%s\n", path);

Assuming that the non *argv case is for consumption by programs and
scripts (similar to the way "ls-files" output is piped to downstream),
we prefer to (1) support "-z" so that "xargs -0" can read paths with
funny characters, and (2) use quote_c_style() from quote.c when "-z"
is not in effect.

> +}
> + ...
> +			setenv(GIT_DIR_ENVIRONMENT, gitdir, 1);
> +			strbuf_setlen(path, len - 1);
> +			setenv(GIT_WORK_TREE_ENVIRONMENT, path->buf, 1);
> +			handle_repo(path->buf, argv);

When you are only showing the path to a repository, I do not think
you want setenv() or chdir() at all. Shouldn't these be done inside
handle_repo() function?  As you are only dealing with non-bare
repositories (and that is what you print in "listing only" mode
anyway), handle_repo() can borrow path (not path->buf) and append
and strip "/.git" as needed.

Also, while it is a good idea to protect this program from stray
GIT_DIR/GIT_WORK_TREE the user may have in the environment when this
program is started, I think this is not enough, if you allow the
*argv commands to run worktree related operations in each repository
you discover.  You would need to chdir() to the top of the working
tree.

The run-command API lets you specify custom environment only for the
child process without affecting yourself by setting .env member of
the child_process structure, so we may want to use that instead of
doing setenv() on ourselves (and letting it inherited by the child).