[PATCH 0/3] git-describe deals gracefully with broken submodules

[PATCH 0/3] git-describe deals gracefully with broken submodules

[Stefan Beller]

Our own version generation in GIT-VERSION-GEN is somewhat sane by testing
if we have a .git dir, and use that as a signal whether the obtained
copy of git was obtained using git (clone/fetch) or if it is just a
downloaded tar ball.

Other scripts to generate a version are not as cautious and just run
"git describe". An error from git-describe is treated as a sufficient
signal to assume it is not a git repository.

When submodules come into play, this is not true, as a submodule
may be damaged instead, such that we're still in a git repository
but error out for the sake of reporting a severly broken submodule.

Add a flag to git-describe that instructs it to treat severe submodule
errors as "dirty" instead of erroring out.

Thanks,
Stefan

Stefan Beller (3):
  submodule.c: port is_submodule_modified to use porcelain 2
  revision machinery: gentle submodule errors
  builtin/describe: introduce --submodule-error-as-dirty flag

 builtin/describe.c                           | 17 ++---
 diff-lib.c                                   | 10 ++-
 diff.h                                       |  1 +
 diffcore.h                                   |  3 +-
 revision.c                                   |  2 +
 submodule.c                                  | 94 +++++++++++++++++-----------
 submodule.h                                  | 10 ++-
 t/t4060-diff-submodule-option-diff-format.sh | 22 +++++++
 t/t6120-describe.sh                          | 17 +++++
 9 files changed, 128 insertions(+), 48 deletions(-)

-- 
2.12.0.402.g4b3201c2d6.dirty

[PATCH 1/3] submodule.c: port is_submodule_modified to use porcelain 2

[Stefan Beller]

Migrate 'is_submodule_modified' to the new porcelain format of
git-status.

As the old porcelain only reported ' M' for submodules, no
matter what happened inside the submodule (untracked files,
changes to tracked files or move of HEAD), the new API
properly reports the different scenarios.

In a followup patch we will make use of these finer grained
reporting for git-status.

While porting this to the new API, add another extension
point that will get used in the future: When a submodule
is broken (e.g. the .git file pointing to a wrong directory,
not containing a git dir, as fallout of e.g. f8eaa0ba98
(submodule--helper, module_clone: always operate on absolute
paths, 2016-03-31)), we can chose to not die and report it
differently.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 85 +++++++++++++++++++++++++++++++++++--------------------------
 1 file changed, 49 insertions(+), 36 deletions(-)

diff --git a/submodule.c b/submodule.c
index 3200b7bb2b..81d44cb7e9 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1041,67 +1041,80 @@ int fetch_populated_submodules(const struct argv_array *options,
 
 unsigned is_submodule_modified(const char *path, int ignore_untracked)
 {
-	ssize_t len;
 	struct child_process cp = CHILD_PROCESS_INIT;
-	const char *argv[] = {
-		"status",
-		"--porcelain",
-		NULL,
-		NULL,
-	};
 	struct strbuf buf = STRBUF_INIT;
 	unsigned dirty_submodule = 0;
-	const char *line, *next_line;
 	const char *git_dir;
+	int error_code = 0;
 
 	strbuf_addf(&buf, "%s/.git", path);
-	git_dir = read_gitfile(buf.buf);
-	if (!git_dir)
-		git_dir = buf.buf;
-	if (!is_directory(git_dir)) {
-		strbuf_release(&buf);
-		/* The submodule is not checked out, so it is not modified */
-		return 0;
+	git_dir = resolve_gitdir_gently(buf.buf, &error_code);
 
+	if (!git_dir) {
+		switch (error_code) {
+		case READ_GITFILE_ERR_STAT_FAILED:
+		case READ_GITFILE_ERR_NOT_A_FILE:
+			/* We may have an uninitialized repo here */
+			return 0;
+		default:
+		case READ_GITFILE_ERR_OPEN_FAILED:
+		case READ_GITFILE_ERR_READ_FAILED:
+		case READ_GITFILE_ERR_INVALID_FORMAT:
+		case READ_GITFILE_ERR_NO_PATH:
+		case READ_GITFILE_ERR_NOT_A_REPO:
+		case READ_GITFILE_ERR_TOO_LARGE:
+			/*
+			 * All these other error codes are indicating
+			 * a broken submodule. We do not know what is
+			 * right here. Resolve again triggering die()
+			 * inside of the parsing.
+			 */
+			read_gitfile_gently(buf.buf, NULL);
+			die("BUG: read_gitfile_gently should have died.");
+		}
 	}
+
 	strbuf_reset(&buf);
 
+	argv_array_pushl(&cp.args, "status", "--porcelain=2", NULL);
 	if (ignore_untracked)
-		argv[2] = "-uno";
+		argv_array_push(&cp.args, "-uno");
 
-	cp.argv = argv;
 	prepare_submodule_repo_env(&cp.env_array);
 	cp.git_cmd = 1;
 	cp.no_stdin = 1;
 	cp.out = -1;
 	cp.dir = path;
 	if (start_command(&cp))
-		die("Could not run 'git status --porcelain' in submodule %s", path);
+		die("Could not run 'git status --porcelain=2' in submodule %s", path);
 
-	len = strbuf_read(&buf, cp.out, 1024);
-	line = buf.buf;
-	while (len > 2) {
-		if ((line[0] == '?') && (line[1] == '?')) {
+	while (strbuf_getwholeline_fd(&buf, cp.out, '\n') != EOF) {
+		/* regular untracked files */
+		if (buf.buf[0] == '?')
 			dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;
-			if (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)
-				break;
-		} else {
-			dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
-			if (ignore_untracked ||
-			    (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED))
-				break;
+
+		/* regular unmerged and renamed files */
+		if (buf.buf[0] == 'u' ||
+		    buf.buf[0] == '1' ||
+		    buf.buf[0] == '2') {
+			if (buf.buf[5] == 'S') {
+				/* nested submodule handling */
+				if (buf.buf[6] == 'C' || buf.buf[7] == 'M')
+					dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
+				if (buf.buf[8] == 'U')
+					dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;
+			} else
+				dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
 		}
-		next_line = strchr(line, '\n');
-		if (!next_line)
-			break;
-		next_line++;
-		len -= (next_line - line);
-		line = next_line;
+
+		if (dirty_submodule & DIRTY_SUBMODULE_MODIFIED &&
+		    dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)
+				break;
 	}
 	close(cp.out);
 
 	if (finish_command(&cp))
-		die("'git status --porcelain' failed in submodule %s", path);
+		die("'git status --porcelain=2' failed in submodule %s", path);
 
 	strbuf_release(&buf);
 	return dirty_submodule;
-- 
2.12.0.402.g4b3201c2d6.dirty

[PATCH 2/3] revision machinery: gentle submodule errors

[Stefan Beller]

When a submodule is broken (e.g. its git directory is deleted or is not
pointed to via the .git linking file), commands that deal with submodules
error out reporting the submodule error.

Sometimes users rather want to see the rest of the command succeed,
ignoring the submodule that is severely broken.

Introduce a flag for the revision machinery '--gentle-submodule-errors'
that will ask 'is_submodule_modified' to return the broken flag instead
of dying().

By switching the argument of is_submodule_modified to an unsigned bit
vector, we can signal more flags, such as the gentle bit introduced
in this patch. When this flag is given, avoid dying inside
'is_submodule_modified' for severe errors and return
DIRTY_SUBMODULE_BROKEN instead.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 diff-lib.c                                   | 10 ++++++++--
 diff.h                                       |  1 +
 diffcore.h                                   |  3 ++-
 revision.c                                   |  2 ++
 submodule.c                                  |  9 ++++++++-
 submodule.h                                  | 10 +++++++++-
 t/t4060-diff-submodule-option-diff-format.sh | 22 ++++++++++++++++++++++
 7 files changed, 52 insertions(+), 5 deletions(-)

diff --git a/diff-lib.c b/diff-lib.c
index 52447466b5..61d33b6d26 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -77,8 +77,14 @@ static int match_stat_with_submodule(struct diff_options *diffopt,
 		if (DIFF_OPT_TST(diffopt, IGNORE_SUBMODULES))
 			changed = 0;
 		else if (!DIFF_OPT_TST(diffopt, IGNORE_DIRTY_SUBMODULES)
-		    && (!changed || DIFF_OPT_TST(diffopt, DIRTY_SUBMODULES)))
-			*dirty_submodule = is_submodule_modified(ce->name, DIFF_OPT_TST(diffopt, IGNORE_UNTRACKED_IN_SUBMODULES));
+		    && (!changed || DIFF_OPT_TST(diffopt, DIRTY_SUBMODULES))) {
+			unsigned flags = 0;
+			if (DIFF_OPT_TST(diffopt, IGNORE_UNTRACKED_IN_SUBMODULES))
+				flags |= IS_SUBMODULE_MODIFIED_IGNORE_UNTRACKED;
+			if (diffopt->gentle_submodule_errors)
+				flags |= IS_SUBMODULE_MODIFIED_GENTLY;
+			*dirty_submodule = is_submodule_modified(ce->name, flags);
+		}
 		diffopt->flags = orig_flags;
 	}
 	return changed;
diff --git a/diff.h b/diff.h
index e9ccb38c26..d1e48de43c 100644
--- a/diff.h
+++ b/diff.h
@@ -164,6 +164,7 @@ struct diff_options {
 	const char *word_regex;
 	enum diff_words_type word_diff;
 	enum diff_submodule_format submodule_format;
+	int gentle_submodule_errors;
 
 	/* this is set by diffcore for DIFF_FORMAT_PATCH */
 	int found_changes;
diff --git a/diffcore.h b/diffcore.h
index 6230241354..ceef783570 100644
--- a/diffcore.h
+++ b/diffcore.h
@@ -40,9 +40,10 @@ struct diff_filespec {
 #define DIFF_FILE_VALID(spec) (((spec)->mode) != 0)
 	unsigned should_free : 1; /* data should be free()'ed */
 	unsigned should_munmap : 1; /* data should be munmap()'ed */
-	unsigned dirty_submodule : 2;  /* For submodules: its work tree is dirty */
+	unsigned dirty_submodule : 3;  /* For submodules: its work tree is dirty */
 #define DIRTY_SUBMODULE_UNTRACKED 1
 #define DIRTY_SUBMODULE_MODIFIED  2
+#define DIRTY_SUBMODULE_BROKEN 4
 	unsigned is_stdin : 1;
 	unsigned has_more_entries : 1; /* only appear in combined diff */
 	/* data should be considered "binary"; -1 means "don't know yet" */
diff --git a/revision.c b/revision.c
index 7ff61ff5f7..8ae1179b8d 100644
--- a/revision.c
+++ b/revision.c
@@ -2014,6 +2014,8 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 		revs->limited = 1;
 	} else if (!strcmp(arg, "--ignore-missing")) {
 		revs->ignore_missing = 1;
+	} else if (!strcmp(arg, "--gentle-submodule-errors")) {
+		revs->diffopt.gentle_submodule_errors = 1;
 	} else {
 		int opts = diff_opt_parse(&revs->diffopt, argv, argc, revs->prefix);
 		if (!opts)
diff --git a/submodule.c b/submodule.c
index 81d44cb7e9..d477297fab 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1039,13 +1039,18 @@ int fetch_populated_submodules(const struct argv_array *options,
 	return spf.result;
 }
 
-unsigned is_submodule_modified(const char *path, int ignore_untracked)
+/*
+ * Inspects a submodule and returns its state using flags
+ */
+unsigned is_submodule_modified(const char *path, unsigned flags)
 {
 	struct child_process cp = CHILD_PROCESS_INIT;
 	struct strbuf buf = STRBUF_INIT;
 	unsigned dirty_submodule = 0;
 	const char *git_dir;
 	int error_code = 0;
+	int ignore_untracked = flags & IS_SUBMODULE_MODIFIED_IGNORE_UNTRACKED;
+	int gently = flags & IS_SUBMODULE_MODIFIED_GENTLY;
 
 	strbuf_addf(&buf, "%s/.git", path);
 	git_dir = resolve_gitdir_gently(buf.buf, &error_code);
@@ -1069,6 +1074,8 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)
 			 * right here. Resolve again triggering die()
 			 * inside of the parsing.
 			 */
+			if (gently)
+				return DIRTY_SUBMODULE_BROKEN;
 			read_gitfile_gently(buf.buf, NULL);
 			die("BUG: read_gitfile_gently should have died.");
 		}
diff --git a/submodule.h b/submodule.h
index c8a0c9cb29..b7afac5e87 100644
--- a/submodule.h
+++ b/submodule.h
@@ -62,7 +62,15 @@ extern void check_for_new_submodule_commits(unsigned char new_sha1[20]);
 extern int fetch_populated_submodules(const struct argv_array *options,
 			       const char *prefix, int command_line_option,
 			       int quiet, int max_parallel_jobs);
-extern unsigned is_submodule_modified(const char *path, int ignore_untracked);
+
+/* Ignore untracked files */
+#define IS_SUBMODULE_MODIFIED_IGNORE_UNTRACKED 1
+/*
+ * Report severe errors in the submodule (e.g. missing git dir) via
+ * DIRTY_SUBMODULE_BROKEN instead of dying.
+ */
+#define IS_SUBMODULE_MODIFIED_GENTLY 2
+extern unsigned is_submodule_modified(const char *path, unsigned flags);
 extern int submodule_uses_gitfile(const char *path);
 
 #define SUBMODULE_REMOVAL_DIE_ON_ERROR (1<<0)
diff --git a/t/t4060-diff-submodule-option-diff-format.sh b/t/t4060-diff-submodule-option-diff-format.sh
index 7e23b55ea4..e5e11a8378 100755
--- a/t/t4060-diff-submodule-option-diff-format.sh
+++ b/t/t4060-diff-submodule-option-diff-format.sh
@@ -746,4 +746,26 @@ test_expect_success 'diff --submodule=diff with .git file' '
 	test_cmp expected actual
 '
 
+test_expect_success 'setup -- break submodule' '
+	# the gitlink in sm2 points at a missing git dir now
+	mv .real .moved_real
+'
+
+test_expect_success 'diff with severely broken submodule' '
+	test_must_fail git diff &&
+
+	sm_hash=$(git rev-parse HEAD:sm2) &&
+	cat >expect <<-EOF &&
+	diff --git a/sm2 b/sm2
+	--- a/sm2
+	+++ b/sm2
+	@@ -1 +1 @@
+	-Subproject commit $sm_hash
+	+Subproject commit $sm_hash-dirty
+	EOF
+
+	git diff --gentle-submodule-errors -- sm2 >out &&
+	test_cmp expect out
+'
+
 test_done
-- 
2.12.0.402.g4b3201c2d6.dirty

[PATCH 3/3] builtin/describe: introduce --submodule-error-as-dirty flag

[Stefan Beller]

git-describe tells you the version number you're at, or errors out, e.g.
when you run it outside of a repository, which may happen when downloading
a tar ball instead of using git to obtain the source code.

To keep this property of only erroring out, when not in a repository,
severe submodule errors must be downgraded to reporting them gently
instead of having git-describe error out completely.

This patch helps to fix the root cause in [1], which tries to work around
this situation.

[1] ("Work around git describe bug for build.")
https://gerrit-review.googlesource.com/#/c/99851/

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/describe.c  | 17 ++++++++++-------
 t/t6120-describe.sh | 17 +++++++++++++++++
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/builtin/describe.c b/builtin/describe.c
index 76c18059bf..569fef9ecf 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -31,13 +31,9 @@ static int have_util;
 static struct string_list patterns = STRING_LIST_INIT_NODUP;
 static struct string_list exclude_patterns = STRING_LIST_INIT_NODUP;
 static int always;
+static int gentle_submodule_errors;
 static const char *dirty;
 
-/* diff-index command arguments to check if working tree is dirty. */
-static const char *diff_index_args[] = {
-	"diff-index", "--quiet", "HEAD", "--", NULL
-};
-
 struct commit_name {
 	struct hashmap_entry entry;
 	struct object_id peeled;
@@ -442,6 +438,8 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 			   N_("do not consider tags matching <pattern>")),
 		OPT_BOOL(0, "always",        &always,
 			N_("show abbreviated commit object as fallback")),
+		OPT_BOOL(0, "submodule-error-as-dirty", &gentle_submodule_errors,
+			N_("show abbreviated commit object as fallback")),
 		{OPTION_STRING, 0, "dirty",  &dirty, N_("mark"),
 			N_("append <mark> on dirty working tree (default: \"-dirty\")"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "-dirty"},
@@ -496,6 +494,12 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 		if (dirty) {
 			static struct lock_file index_lock;
 			int fd;
+			struct argv_array args = ARGV_ARRAY_INIT;
+
+			argv_array_push(&args, "diff-index");
+			if (gentle_submodule_errors)
+				argv_array_push(&args, "--gentle-submodule-errors");
+			argv_array_pushl(&args, "--quiet", "HEAD", "--", NULL);
 
 			read_cache_preload(NULL);
 			refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED,
@@ -504,8 +508,7 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 			if (0 <= fd)
 				update_index_if_able(&the_index, &index_lock);
 
-			if (!cmd_diff_index(ARRAY_SIZE(diff_index_args) - 1,
-					    diff_index_args, prefix))
+			if (!cmd_diff_index(args.argc, args.argv, prefix))
 				dirty = NULL;
 		}
 		describe("HEAD", 1);
diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
index 167491fd5b..99e5ba44b7 100755
--- a/t/t6120-describe.sh
+++ b/t/t6120-describe.sh
@@ -233,4 +233,21 @@ test_expect_success 'describe --contains and --no-match' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup and absorb a submodule' '
+	test_create_repo sub1 &&
+	test_commit -C sub1 initial &&
+	git submodule add ./sub1 &&
+	git submodule absorbgitdirs &&
+	git commit -a -m "add submodule"
+'
+
+test_expect_success 'describe chokes on severly broken submodules' '
+	mv .git/modules/sub1/ .git/modules/sub_moved &&
+	test_must_fail git describe --dirty
+'
+test_expect_success 'describe ignoring a borken submodule' '
+	git describe --dirty --submodule-error-as-dirty >out &&
+	grep dirty out
+'
+
 test_done
-- 
2.12.0.402.g4b3201c2d6.dirty

Re: [PATCH 0/3] git-describe deals gracefully with broken submodules

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Our own version generation in GIT-VERSION-GEN is somewhat sane by testing
> if we have a .git dir, and use that as a signal whether the obtained
> copy of git was obtained using git (clone/fetch) or if it is just a
> downloaded tar ball.
>
> Other scripts to generate a version are not as cautious and just run
> "git describe". An error from git-describe is treated as a sufficient
> signal to assume it is not a git repository.
>
> When submodules come into play, this is not true, as a submodule
> may be damaged instead, such that we're still in a git repository
> but error out for the sake of reporting a severly broken submodule.
>
> Add a flag to git-describe that instructs it to treat severe submodule
> errors as "dirty" instead of erroring out.

I do not have a strong preference for or against the "treat a broken
repository as if nothing is wrong with the revision, but just mark
it as dirty" idea.  I would be more receptive if it substituted the
"-dirty" marker with something else, e.g. "-broken", though.

My knee-jerk reaction to the code change is that treating submodule
as something very special is probably not a good idea.  Even if you
do not use submodules, if some of the objects referenced from your
index and/or HEAD are damaged or otherwise causes some error while
accessing, the diff machinery would die, wouldn't it?

I saw that some new symbolic constants in the code to tell the
machinery to "gracefully die" (or "hide the breakage under the rug")
are named with SUBMODULE in them, which is probably a bad sign that
the design is being too centric to submodules.  The implementation
that covers only breakages in submodule as its first step may be OK
(you have to start somewhere, after all), but I think the aspiration
should be to cover all kinds of breakages in the end and turn them
to be "graceful", and if you had that goal in mind, you wouldn't be
naming these constants with SUBMODULE in them.

If "treat a broken repository as just being 'dirty'" were a good
idea, I'd suspect that we would want to see all breakages, not just
ones related to submodules, to be treated the same way.  

But it is possible there may be a reason why submodules are special.
I do not think the third paragraph quoted above is a good
justification.  A repository with broken submodule is just as broken
and untrustworthy as a broken repository without a submodule, and if
you want to allow such a checkout with broken submodule to call
itself v2.0-dirty, you would also want to allow a broken checkout
without any submodule to do so, too.

Re: [PATCH 0/3] git-describe deals gracefully with broken submodules

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I do not have a strong preference for or against the "treat a broken
> repository as if nothing is wrong with the revision, but just mark
> it as dirty" idea.  I would be more receptive if it substituted the
> "-dirty" marker with something else, e.g. "-broken", though.
> ...
> But it is possible there may be a reason why submodules are special.
> I do not think the third paragraph quoted above is a good
> justification.  A repository with broken submodule is just as broken
> and untrustworthy as a broken repository without a submodule, and if
> you want to allow such a checkout with broken submodule to call
> itself v2.0-dirty, you would also want to allow a broken checkout
> without any submodule to do so, too.

Having said all that, if I thought it were a good idea to optionally
allow people to treat "repository is corrupt in some ways to make it
impossible for us if the checkout is dirty or not, even though we
are fairly certain what commit .git/HEAD points at is" as a state
that is describable, I probably would

 - introduce a new "git describe --possibly-broken" option;

 - instead of running "diff-index" internally to decide the "-dirty"
   state, spawn "diff-index" as a separate process;

 - observe the exit status from "diff-index" and add "-dirty" suffix
   when it is _known_ to be dirty, add "-broken" suffix when it
   died, and leave out the suffix when we know that the checkout is
   clean.

That way, we wouldn't have to contaminate the generic codepath with
a "treat broken and modified states as if they are the same" logic,
only to support such a niche feature that we wouldn't want to use
anywhere else.

[PATCH] builtin/describe: introduce --broken flag

[Stefan Beller]

git-describe tells you the version number you're at, or errors out, e.g.
when you run it outside of a repository, which may happen when downloading
a tar ball instead of using git to obtain the source code.

To keep this property of only erroring out, when not in a repository,
severe (submodule) errors must be downgraded to reporting them gently
instead of having git-describe error out completely.

To achieve that a flag '--broken' is introduced, which is in the same
vein as '--dirty' but uses an actual child process to check for dirtiness.
When that child dies unexpectedly, we'll append '-broken' instead of
'-dirty'.

This patch helps to fix the root cause in [1], which tries to work around
this situation.

[1] ("Work around git describe bug for build.")
https://gerrit-review.googlesource.com/#/c/99851/

Signed-off-by: Stefan Beller <sbeller@google.com>
---

> I probably would
> 
> - introduce a new "git describe --possibly-broken" option;
> 
> - instead of running "diff-index" internally to decide the "-dirty"
>    state, spawn "diff-index" as a separate process;
> 
>  - observe the exit status from "diff-index" and add "-dirty" suffix

This is what this patch does. It doesn't need any preceeding refactor
patches, but comes on its own; developed on origin/master.

Thanks,
Stefan

 Documentation/git-describe.txt |  7 +++++
 builtin/describe.c             | 59 ++++++++++++++++++++++++++++++------------
 t/t6120-describe.sh            | 20 ++++++++++++++
 3 files changed, 70 insertions(+), 16 deletions(-)

diff --git a/Documentation/git-describe.txt b/Documentation/git-describe.txt
index 8755f3af7b..b71fa7a4ad 100644
--- a/Documentation/git-describe.txt
+++ b/Documentation/git-describe.txt
@@ -34,6 +34,13 @@ OPTIONS
 	It means describe HEAD and appends <mark> (`-dirty` by
 	default) if the working tree is dirty.
 
+--broken[=<mark>]::
+	Describe the working tree.
+	It means describe HEAD and appends <mark> (`-broken` by
+	default) if the working tree cannot be examined for dirtiness.
+	This implies `--dirty`, which is the fallback behavior when
+	the working tree examination works.
+
 --all::
 	Instead of using only the annotated tags, use any ref
 	found in `refs/` namespace.  This option enables matching
diff --git a/builtin/describe.c b/builtin/describe.c
index 76c18059bf..37a83520c9 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -9,6 +9,7 @@
 #include "diff.h"
 #include "hashmap.h"
 #include "argv-array.h"
+#include "run-command.h"
 
 #define SEEN		(1u << 0)
 #define MAX_TAGS	(FLAG_BITS - 1)
@@ -31,12 +32,7 @@ static int have_util;
 static struct string_list patterns = STRING_LIST_INIT_NODUP;
 static struct string_list exclude_patterns = STRING_LIST_INIT_NODUP;
 static int always;
-static const char *dirty;
-
-/* diff-index command arguments to check if working tree is dirty. */
-static const char *diff_index_args[] = {
-	"diff-index", "--quiet", "HEAD", "--", NULL
-};
+static const char *append, *dirty, *broken;
 
 struct commit_name {
 	struct hashmap_entry entry;
@@ -292,8 +288,8 @@ static void describe(const char *arg, int last_one)
 		display_name(n);
 		if (longformat)
 			show_suffix(0, n->tag ? &n->tag->tagged->oid : &oid);
-		if (dirty)
-			printf("%s", dirty);
+		if (append)
+			printf("%s", append);
 		printf("\n");
 		return;
 	}
@@ -369,8 +365,8 @@ static void describe(const char *arg, int last_one)
 		struct object_id *oid = &cmit->object.oid;
 		if (always) {
 			printf("%s", find_unique_abbrev(oid->hash, abbrev));
-			if (dirty)
-				printf("%s", dirty);
+			if (append)
+				printf("%s", append);
 			printf("\n");
 			return;
 		}
@@ -413,8 +409,8 @@ static void describe(const char *arg, int last_one)
 	display_name(all_matches[0].name);
 	if (abbrev)
 		show_suffix(all_matches[0].depth, &cmit->object.oid);
-	if (dirty)
-		printf("%s", dirty);
+	if (append)
+		printf("%s", append);
 	printf("\n");
 
 	if (!last_one)
@@ -445,6 +441,9 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 		{OPTION_STRING, 0, "dirty",  &dirty, N_("mark"),
 			N_("append <mark> on dirty working tree (default: \"-dirty\")"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "-dirty"},
+		{OPTION_STRING, 0, "broken",  &broken, N_("mark"),
+			N_("append <mark> on broken working tree (default: \"-broken\")"),
+			PARSE_OPT_OPTARG, NULL, (intptr_t) "-broken"},
 		OPT_END(),
 	};
 
@@ -493,10 +492,35 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 		die(_("No names found, cannot describe anything."));
 
 	if (argc == 0) {
-		if (dirty) {
+		if (broken) {
+			struct child_process cp = CHILD_PROCESS_INIT;
+			argv_array_pushl(&cp.args, "diff-index", "--quiet", "HEAD", "--", NULL);
+			cp.git_cmd = 1;
+			cp.no_stdin = 1;
+			cp.no_stdout = 1;
+
+			if (!dirty)
+				dirty = "-dirty";
+
+			switch (run_command(&cp)) {
+			case 0:
+				append = NULL;
+				break;
+			case 1:
+				/* keep dirty as is */
+				append = dirty;
+				break;
+			default:
+				/* diff-index aborted abnormally */
+				append = broken;
+			}
+		} else if (dirty) {
+			struct argv_array args = ARGV_ARRAY_INIT;
 			static struct lock_file index_lock;
 			int fd;
 
+			argv_array_pushl(&args, "diff-index", "--quiet", "HEAD", "--", NULL);
+
 			read_cache_preload(NULL);
 			refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED,
 				      NULL, NULL, NULL);
@@ -504,13 +528,16 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 			if (0 <= fd)
 				update_index_if_able(&the_index, &index_lock);
 
-			if (!cmd_diff_index(ARRAY_SIZE(diff_index_args) - 1,
-					    diff_index_args, prefix))
-				dirty = NULL;
+			if (cmd_diff_index(args.argc, args.argv, prefix))
+				append = dirty;
+			else
+				append = NULL;
 		}
 		describe("HEAD", 1);
 	} else if (dirty) {
 		die(_("--dirty is incompatible with commit-ishes"));
+	} else if (broken) {
+		die(_("--broken is incompatible with commit-ishes"));
 	} else {
 		while (argc-- > 0)
 			describe(*argv++, argc == 0);
diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
index 167491fd5b..16952e44fc 100755
--- a/t/t6120-describe.sh
+++ b/t/t6120-describe.sh
@@ -233,4 +233,24 @@ test_expect_success 'describe --contains and --no-match' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup and absorb a submodule' '
+	test_create_repo sub1 &&
+	test_commit -C sub1 initial &&
+	git submodule add ./sub1 &&
+	git submodule absorbgitdirs &&
+	git commit -a -m "add submodule" &&
+	git describe --dirty >expect &&
+	git describe --broken >out &&
+	test_cmp expect out
+'
+
+test_expect_success 'describe chokes on severly broken submodules' '
+	mv .git/modules/sub1/ .git/modules/sub_moved &&
+	test_must_fail git describe --dirty
+'
+test_expect_success 'describe ignoring a borken submodule' '
+	git describe --broken >out &&
+	grep broken out
+'
+
 test_done
-- 
2.12.0.402.g0501f7a28e.dirty

Re: [PATCH] builtin/describe: introduce --broken flag

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> This patch helps to fix the root cause in [1], which tries to work around
> this situation.

I do not necessarily think it is reasonable to give $version-dirty
and proceed when a repository corruption is detected; if there is a
breakage in the repository, "git describe" is correct to die when a
populated submodule is broken.  IOW, I do not agree that [1] below
is working with a sensible expectation.

This is a tangent, but how does the Gerrit repository get corrupted
in the way described in [1] in the first place?  That might be what
needs to be corrected, perhaps?

>  Documentation/git-describe.txt |  7 +++++
>  builtin/describe.c             | 59 ++++++++++++++++++++++++++++++------------
>  t/t6120-describe.sh            | 20 ++++++++++++++
>  3 files changed, 70 insertions(+), 16 deletions(-)
>
> diff --git a/Documentation/git-describe.txt b/Documentation/git-describe.txt
> index 8755f3af7b..b71fa7a4ad 100644
> --- a/Documentation/git-describe.txt
> +++ b/Documentation/git-describe.txt
> @@ -34,6 +34,13 @@ OPTIONS
>  	It means describe HEAD and appends <mark> (`-dirty` by
>  	default) if the working tree is dirty.
>  
> +--broken[=<mark>]::
> +	Describe the working tree.
> +	It means describe HEAD and appends <mark> (`-broken` by
> +	default) if the working tree cannot be examined for dirtiness.
> +	This implies `--dirty`, which is the fallback behavior when
> +	the working tree examination works.

The wording for the "--dirty" is already awkward, but this one is
even more so ("Describe the working tree. It means" conveys no
useful information).  I however cannot come up with something much
better.  This is the best I could come up with:

	Describe the state of the working tree.  When the working
	tree matches HEAD, the output is the same as "git describe
	HEAD" and "-dirty" is appended to it if the working tree has
	local modification.  When a repository is corrupt and Git
	cannot determine if there is local modification, instead of
	dying, append "-broken" instead.

The last sentence can be tweaked to replace the description for
"--dirty".

> diff --git a/builtin/describe.c b/builtin/describe.c
> index 76c18059bf..37a83520c9 100644
> --- a/builtin/describe.c
> +++ b/builtin/describe.c
> @@ -9,6 +9,7 @@
>  #include "diff.h"
>  #include "hashmap.h"
>  #include "argv-array.h"
> +#include "run-command.h"
>  
>  #define SEEN		(1u << 0)
>  #define MAX_TAGS	(FLAG_BITS - 1)
> @@ -31,12 +32,7 @@ static int have_util;
>  static struct string_list patterns = STRING_LIST_INIT_NODUP;
>  static struct string_list exclude_patterns = STRING_LIST_INIT_NODUP;
>  static int always;
> -static const char *dirty;
> -
> -/* diff-index command arguments to check if working tree is dirty. */
> -static const char *diff_index_args[] = {
> -	"diff-index", "--quiet", "HEAD", "--", NULL
> -};
> +static const char *append, *dirty, *broken;

Perhaps call it "suffix" or something?

> +		if (broken) {
> +			struct child_process cp = CHILD_PROCESS_INIT;
> +			argv_array_pushl(&cp.args, "diff-index", "--quiet", "HEAD", "--", NULL);
> ...
> +			}
> +		} else if (dirty) {
> +			struct argv_array args = ARGV_ARRAY_INIT;
>  			static struct lock_file index_lock;
>  			int fd;
>  
> +			argv_array_pushl(&args, "diff-index", "--quiet", "HEAD", "--", NULL);


Somehow it looks like the patch is making the code a lot worse by
losing diff_index_args[] and duplicating the same command line twice
in the code.  Wouldn't argv_array_pushv() into these two different args
array from the same template work better?

Re: [PATCH] builtin/describe: introduce --broken flag

[Stefan Beller]

On Tue, Mar 21, 2017 at 2:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> This patch helps to fix the root cause in [1], which tries to work around
>> this situation.
>
> I do not necessarily think it is reasonable to give $version-dirty
> and proceed when a repository corruption is detected; if there is a
> breakage in the repository, "git describe" is correct to die when a
> populated submodule is broken.  IOW, I do not agree that [1] below
> is working with a sensible expectation.

ok, so I won't quote it in the commit message

>
> This is a tangent, but how does the Gerrit repository get corrupted
> in the way described in [1] in the first place?  That might be what
> needs to be corrected, perhaps?

AFAICT, someone is (was?) using a version of Git that doesn't contain
f8eaa0ba98 (submodule--helper, module_clone: always operate
on absolute paths, 2016-03-31). So then the submodule paths were
made absolute paths on creation of the Gerrit repo.

And then someone moved the repo and the absolute paths broke.

Even after an upgrade of Git to its latest and greatest version, the underlying
issue of having broken submodule paths remains in that case.

So there are a couple of ways forward
0) as an immediate fix, manually fix the absolute path or make them relative

1A) have more error resilient tools in Git
1B) have a tool in git (e.g. "git submodule fsck-setup") that rewrites
    the .git file link and the core.worktree setting to be relative and correct.

I think we should do both A and B, I decided to go with A first, specifically
"git-describe" as that was reported to not work well in this situation with the
given broken data

> The wording for the "--dirty" is already awkward, but this one is
> even more so ("Describe the working tree. It means" conveys no
> useful information).  I however cannot come up with something much
> better.  This is the best I could come up with:
>
>         Describe the state of the working tree.  When the working
>         tree matches HEAD, the output is the same as "git describe
>         HEAD" and "-dirty" is appended to it if the working tree has
>         local modification.  When a repository is corrupt and Git
>         cannot determine if there is local modification, instead of
>         dying, append "-broken" instead.

ok, I'll reuse parts of it.

>> +static const char *append, *dirty, *broken;
>
> Perhaps call it "suffix" or something?

done


>> +                     argv_array_pushl(&args, "diff-index", "--quiet", "HEAD", "--", NULL);
..
>  Wouldn't argv_array_pushv() into these two different args
> array from the same template work better?

yes, looks much saner. Thanks

Re: [PATCH] builtin/describe: introduce --broken flag

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> AFAICT, someone is (was?) using a version of Git that doesn't contain
> f8eaa0ba98 (submodule--helper, module_clone: always operate
> on absolute paths, 2016-03-31). So then the submodule paths were
> made absolute paths on creation of the Gerrit repo.
>
> And then someone moved the repo and the absolute paths broke.
>
> Even after an upgrade of Git to its latest and greatest version, the underlying
> issue of having broken submodule paths remains in that case.
>
> So there are a couple of ways forward
> 0) as an immediate fix, manually fix the absolute path or make them relative
>
> 1A) have more error resilient tools in Git
> 1B) have a tool in git (e.g. "git submodule fsck-setup") that rewrites
>     the .git file link and the core.worktree setting to be relative and correct.
>
> I think we should do both A and B, I decided to go with A first, specifically
> "git-describe" as that was reported to not work well in this situation with the
> given broken data

Sounds sensible.  I would say that we should do less 1A (i.e. hiding
problems under the rug) and more 1B.  Of course, making the problem
less likely to happen in the first place would be more important ;-)

Thanks.

Re: [PATCH] builtin/describe: introduce --broken flag

[Stefan Beller]

On Tue, Mar 21, 2017 at 3:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> AFAICT, someone is (was?) using a version of Git that doesn't contain
>> f8eaa0ba98 (submodule--helper, module_clone: always operate
>> on absolute paths, 2016-03-31). So then the submodule paths were
>> made absolute paths on creation of the Gerrit repo.
>>
>> And then someone moved the repo and the absolute paths broke.
>>
>> Even after an upgrade of Git to its latest and greatest version, the underlying
>> issue of having broken submodule paths remains in that case.
>>
>> So there are a couple of ways forward
>> 0) as an immediate fix, manually fix the absolute path or make them relative
>>
>> 1A) have more error resilient tools in Git
>> 1B) have a tool in git (e.g. "git submodule fsck-setup") that rewrites
>>     the .git file link and the core.worktree setting to be relative and correct.
>>
>> I think we should do both A and B, I decided to go with A first, specifically
>> "git-describe" as that was reported to not work well in this situation with the
>> given broken data
>
> Sounds sensible.  I would say that we should do less 1A (i.e. hiding
> problems under the rug) and more 1B.  Of course, making the problem
> less likely to happen in the first place would be more important ;-)

Agreed on not introducing bugs as often. For (B), I wonder what the
right place is.

Maybe "reset --hard", in combination with "--recurse-submodules" would also
fix these submodule path issues. The made up "git submodule fsck-setup"
sounds like it would only detect (and not fix) the problems. But these absolute
paths issues are known how to fix, so a "fsck"-y tool may convey a wrong
impression.

[PATCH v2] builtin/describe: introduce --broken flag

[Stefan Beller]

git-describe tells you the version number you're at, or errors out, e.g.
when you run it outside of a repository, which may happen when downloading
a tar ball instead of using git to obtain the source code.

To keep this property of only erroring out, when not in a repository,
severe (submodule) errors must be downgraded to reporting them gently
instead of having git-describe error out completely.

To achieve that a flag '--broken' is introduced, which is in the same
vein as '--dirty' but uses an actual child process to check for dirtiness.
When that child dies unexpectedly, we'll append '-broken' instead of
'-dirty'.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-describe.txt | 11 +++++++---
 builtin/describe.c             | 47 ++++++++++++++++++++++++++++++++++--------
 t/t6120-describe.sh            | 20 ++++++++++++++++++
 3 files changed, 66 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-describe.txt b/Documentation/git-describe.txt
index 8755f3af7b..26f19d3b07 100644
--- a/Documentation/git-describe.txt
+++ b/Documentation/git-describe.txt
@@ -30,9 +30,14 @@ OPTIONS
 	Commit-ish object names to describe.  Defaults to HEAD if omitted.
 
 --dirty[=<mark>]::
-	Describe the working tree.
-	It means describe HEAD and appends <mark> (`-dirty` by
-	default) if the working tree is dirty.
+--broken[=<mark>]::
+	Describe the state of the working tree.  When the working
+	tree matches HEAD, the output is the same as "git describe
+	HEAD".  If the working tree has local modification "-dirty"
+	is appended to it.  If a repository is corrupt and Git
+	cannot determine if there is local modification, Git will
+	error out, unless `--broken' is given, which appends
+	the suffix "-broken" instead.
 
 --all::
 	Instead of using only the annotated tags, use any ref
diff --git a/builtin/describe.c b/builtin/describe.c
index 76c18059bf..45adbf67d5 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -9,6 +9,7 @@
 #include "diff.h"
 #include "hashmap.h"
 #include "argv-array.h"
+#include "run-command.h"
 
 #define SEEN		(1u << 0)
 #define MAX_TAGS	(FLAG_BITS - 1)
@@ -31,7 +32,7 @@ static int have_util;
 static struct string_list patterns = STRING_LIST_INIT_NODUP;
 static struct string_list exclude_patterns = STRING_LIST_INIT_NODUP;
 static int always;
-static const char *dirty;
+static const char *suffix, *dirty, *broken;
 
 /* diff-index command arguments to check if working tree is dirty. */
 static const char *diff_index_args[] = {
@@ -292,8 +293,8 @@ static void describe(const char *arg, int last_one)
 		display_name(n);
 		if (longformat)
 			show_suffix(0, n->tag ? &n->tag->tagged->oid : &oid);
-		if (dirty)
-			printf("%s", dirty);
+		if (suffix)
+			printf("%s", suffix);
 		printf("\n");
 		return;
 	}
@@ -369,8 +370,8 @@ static void describe(const char *arg, int last_one)
 		struct object_id *oid = &cmit->object.oid;
 		if (always) {
 			printf("%s", find_unique_abbrev(oid->hash, abbrev));
-			if (dirty)
-				printf("%s", dirty);
+			if (suffix)
+				printf("%s", suffix);
 			printf("\n");
 			return;
 		}
@@ -413,8 +414,8 @@ static void describe(const char *arg, int last_one)
 	display_name(all_matches[0].name);
 	if (abbrev)
 		show_suffix(all_matches[0].depth, &cmit->object.oid);
-	if (dirty)
-		printf("%s", dirty);
+	if (suffix)
+		printf("%s", suffix);
 	printf("\n");
 
 	if (!last_one)
@@ -445,6 +446,9 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 		{OPTION_STRING, 0, "dirty",  &dirty, N_("mark"),
 			N_("append <mark> on dirty working tree (default: \"-dirty\")"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "-dirty"},
+		{OPTION_STRING, 0, "broken",  &broken, N_("mark"),
+			N_("append <mark> on broken working tree (default: \"-broken\")"),
+			PARSE_OPT_OPTARG, NULL, (intptr_t) "-broken"},
 		OPT_END(),
 	};
 
@@ -493,7 +497,28 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 		die(_("No names found, cannot describe anything."));
 
 	if (argc == 0) {
-		if (dirty) {
+		if (broken) {
+			struct child_process cp = CHILD_PROCESS_INIT;
+			argv_array_pushv(&cp.args, diff_index_args);
+			cp.git_cmd = 1;
+			cp.no_stdin = 1;
+			cp.no_stdout = 1;
+
+			if (!dirty)
+				dirty = "-dirty";
+
+			switch (run_command(&cp)) {
+			case 0:
+				suffix = NULL;
+				break;
+			case 1:
+				suffix = dirty;
+				break;
+			default:
+				/* diff-index aborted abnormally */
+				suffix = broken;
+			}
+		} else if (dirty) {
 			static struct lock_file index_lock;
 			int fd;
 
@@ -506,11 +531,15 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
 
 			if (!cmd_diff_index(ARRAY_SIZE(diff_index_args) - 1,
 					    diff_index_args, prefix))
-				dirty = NULL;
+				suffix = NULL;
+			else
+				suffix = dirty;
 		}
 		describe("HEAD", 1);
 	} else if (dirty) {
 		die(_("--dirty is incompatible with commit-ishes"));
+	} else if (broken) {
+		die(_("--broken is incompatible with commit-ishes"));
 	} else {
 		while (argc-- > 0)
 			describe(*argv++, argc == 0);
diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
index 167491fd5b..16952e44fc 100755
--- a/t/t6120-describe.sh
+++ b/t/t6120-describe.sh
@@ -233,4 +233,24 @@ test_expect_success 'describe --contains and --no-match' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup and absorb a submodule' '
+	test_create_repo sub1 &&
+	test_commit -C sub1 initial &&
+	git submodule add ./sub1 &&
+	git submodule absorbgitdirs &&
+	git commit -a -m "add submodule" &&
+	git describe --dirty >expect &&
+	git describe --broken >out &&
+	test_cmp expect out
+'
+
+test_expect_success 'describe chokes on severly broken submodules' '
+	mv .git/modules/sub1/ .git/modules/sub_moved &&
+	test_must_fail git describe --dirty
+'
+test_expect_success 'describe ignoring a borken submodule' '
+	git describe --broken >out &&
+	grep broken out
+'
+
 test_done
-- 
2.12.1.385.gd4d53dfa89.dirty

Re: [PATCH v2] builtin/describe: introduce --broken flag

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> git-describe tells you the version number you're at, or errors out, e.g.
> when you run it outside of a repository, which may happen when downloading
> a tar ball instead of using git to obtain the source code.
>
> To keep this property of only erroring out, when not in a repository,
> severe (submodule) errors must be downgraded to reporting them gently
> instead of having git-describe error out completely.
>
> To achieve that a flag '--broken' is introduced, which is in the same
> vein as '--dirty' but uses an actual child process to check for dirtiness.
> When that child dies unexpectedly, we'll append '-broken' instead of
> '-dirty'.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  Documentation/git-describe.txt | 11 +++++++---
>  builtin/describe.c             | 47 ++++++++++++++++++++++++++++++++++--------
>  t/t6120-describe.sh            | 20 ++++++++++++++++++
>  3 files changed, 66 insertions(+), 12 deletions(-)

I admit that my suggestion to use pushv() was done without trying it
out myself, but I do agree that the result looks better, especially
because the "dirty" (not "broken") side does not even have to use a
separate argv_array to prepare the command line in the first place
(which I failed to spot exactly because I didn't try it myself ;-).

We probably _could_ use cp.argv to point at the diff_index_args()
without doing pushv(&cp.args), and that would save a bit of
allocation, but it would not matter in practice as this is not a
performance critical codepath.

Thanks.
> diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
> index 167491fd5b..16952e44fc 100755
> --- a/t/t6120-describe.sh
> +++ b/t/t6120-describe.sh
> @@ -233,4 +233,24 @@ test_expect_success 'describe --contains and --no-match' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'setup and absorb a submodule' '
> +	test_create_repo sub1 &&
> +	test_commit -C sub1 initial &&
> +	git submodule add ./sub1 &&
> +	git submodule absorbgitdirs &&
> +	git commit -a -m "add submodule" &&
> +	git describe --dirty >expect &&
> +	git describe --broken >out &&
> +	test_cmp expect out
> +'
> +
> +test_expect_success 'describe chokes on severly broken submodules' '
> +	mv .git/modules/sub1/ .git/modules/sub_moved &&
> +	test_must_fail git describe --dirty
> +'
> +test_expect_success 'describe ignoring a borken submodule' '
> +	git describe --broken >out &&
> +	grep broken out
> +'
> +
>  test_done

Re: [PATCH v2] builtin/describe: introduce --broken flag

[Jakub Narębski]

W dniu 21.03.2017 o 23:57, Stefan Beller pisze:

> --- a/Documentation/git-describe.txt
> +++ b/Documentation/git-describe.txt
> @@ -30,9 +30,14 @@ OPTIONS
>  	Commit-ish object names to describe.  Defaults to HEAD if omitted.
>  
>  --dirty[=<mark>]::
> -	Describe the working tree.
> -	It means describe HEAD and appends <mark> (`-dirty` by
> -	default) if the working tree is dirty.
> +--broken[=<mark>]::
> +	Describe the state of the working tree.  When the working
> +	tree matches HEAD, the output is the same as "git describe
> +	HEAD".  If the working tree has local modification "-dirty"
> +	is appended to it.  If a repository is corrupt and Git
> +	cannot determine if there is local modification, Git will
> +	error out, unless `--broken' is given, which appends
> +	the suffix "-broken" instead.

The common description reads better... but unfortunately it lost
information about the optional parameter, namely <mark>.  The
'-dirty' is just the default for <dirty-mark>, and '-broken' is
the default for <broken-mark>.

Maybe /the suffix "-broken"/<broken-mark> suffix ('-broken' by default)/
and similarly for "-dirty"?

Best,
-- 
Jakub Narębski

Re: [PATCH 0/3] git-describe deals gracefully with broken submodules

[Stefan Beller]

On Mon, Mar 20, 2017 at 11:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> Our own version generation in GIT-VERSION-GEN is somewhat sane by testing
>> if we have a .git dir, and use that as a signal whether the obtained
>> copy of git was obtained using git (clone/fetch) or if it is just a
>> downloaded tar ball.
>>
>> Other scripts to generate a version are not as cautious and just run
>> "git describe". An error from git-describe is treated as a sufficient
>> signal to assume it is not a git repository.
>>
>> When submodules come into play, this is not true, as a submodule
>> may be damaged instead, such that we're still in a git repository
>> but error out for the sake of reporting a severly broken submodule.
>>
>> Add a flag to git-describe that instructs it to treat severe submodule
>> errors as "dirty" instead of erroring out.
>
> I do not have a strong preference for or against the "treat a broken
> repository as if nothing is wrong with the revision, but just mark
> it as dirty" idea.  I would be more receptive if it substituted the
> "-dirty" marker with something else, e.g. "-broken", though.

ok. That was my initial plan. I plan on also incorporating this into
git-status.

>
> My knee-jerk reaction to the code change is that treating submodule
> as something very special is probably not a good idea.  Even if you
> do not use submodules, if some of the objects referenced from your
> index and/or HEAD are damaged or otherwise causes some error while
> accessing, the diff machinery would die, wouldn't it?

Yes; ideally we would want to have a generic "error with working tree"
for all these kinds of errors. Note that the probability of submodule errors
is way higher. (This has a couple of reasons: (1) a submodule has more
complexity than a file, (2) that opens up new areas in which errors can occur
and (3) submodules are less popular so we have fewer
bug reports and people working on it, hence less mature code.

As an example for (2), we had relative/absolute path issues in Git 2.7
for submodules, which is what Martin was complaining about.

> I saw that some new symbolic constants in the code to tell the
> machinery to "gracefully die" (or "hide the breakage under the rug")
> are named with SUBMODULE in them, which is probably a bad sign that
> the design is being too centric to submodules.  The implementation
> that covers only breakages in submodule as its first step may be OK
> (you have to start somewhere, after all), but I think the aspiration
> should be to cover all kinds of breakages in the end and turn them
> to be "graceful", and if you had that goal in mind, you wouldn't be
> naming these constants with SUBMODULE in them.

ok

>
> If "treat a broken repository as just being 'dirty'" were a good
> idea, I'd suspect that we would want to see all breakages, not just
> ones related to submodules, to be treated the same way.
>
> But it is possible there may be a reason why submodules are special.
> I do not think the third paragraph quoted above is a good
> justification.  A repository with broken submodule is just as broken
> and untrustworthy as a broken repository without a submodule, and if
> you want to allow such a checkout with broken submodule to call
> itself v2.0-dirty, you would also want to allow a broken checkout
> without any submodule to do so, too.

agreed.

> I probably would
>
> - introduce a new "git describe --possibly-broken" option;
>
> - instead of running "diff-index" internally to decide the "-dirty"
>   state, spawn "diff-index" as a separate process;
>
> - observe the exit status from "diff-index" and add "-dirty" suffix
>   when it is _known_ to be dirty, add "-broken" suffix when it
>   died, and leave out the suffix when we know that the checkout is
>   clean.
>
> That way, we wouldn't have to contaminate the generic codepath with
> a "treat broken and modified states as if they are the same" logic,
> only to support such a niche feature that we wouldn't want to use
> anywhere else.

I planned on a follow-up that adds similar behavior to git-status,
such that git-status can be used to get the big picture before
diagnosing the individual breakages of a submodule.

That is the reason why I started with the first patch porting
'is_submodule_modified' to porcelain2 of git-status, as that
allows way better reporting on submodules. Sorry for intermixing
these two things.

Thanks,
Stefan

[PATCHv3 0/3] short status: improve reporting for submodule changes

[Stefan Beller]

This comes as a series; first I'd like to refactor is_submodule_modified
to take advantage of the new porcelain=2 plumbing switch to check for changes
in the submodule.

On top of the refactoring comes the actual change, which moved the
rewriting of the submodule change indicator letter to the collection part.

Thanks,
Stefan


Stefan Beller (3):
  submodule.c: port is_submodule_modified to use porcelain 2
  submodule.c, is_submodule_modified: stricter checking for submodules
  short status: improve reporting for submodule changes

 Documentation/git-status.txt |  9 +++++++
 submodule.c                  | 58 +++++++++++++++++++++-----------------------
 t/t3600-rm.sh                | 18 ++++++++++----
 t/t7506-status-submodule.sh  | 24 ++++++++++++++++++
 wt-status.c                  | 13 ++++++++--
 5 files changed, 84 insertions(+), 38 deletions(-)

-- 
2.12.1.432.gfe308fe33c.dirty

[PATCH 1/3] submodule.c: port is_submodule_modified to use porcelain 2

[Stefan Beller]

Migrate 'is_submodule_modified' to the new porcelain format of
git-status.

As the old porcelain only reported ' M' for submodules, no
matter what happened inside the submodule (untracked files,
changes to tracked files or move of HEAD), the new API
properly reports the different scenarios.

In a followup patch we will make use of these finer grained
reporting for git-status.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c | 53 ++++++++++++++++++++++++-----------------------------
 1 file changed, 24 insertions(+), 29 deletions(-)

diff --git a/submodule.c b/submodule.c
index 3200b7bb2b..d355ddb46b 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1041,17 +1041,9 @@ int fetch_populated_submodules(const struct argv_array *options,
 
 unsigned is_submodule_modified(const char *path, int ignore_untracked)
 {
-	ssize_t len;
 	struct child_process cp = CHILD_PROCESS_INIT;
-	const char *argv[] = {
-		"status",
-		"--porcelain",
-		NULL,
-		NULL,
-	};
 	struct strbuf buf = STRBUF_INIT;
 	unsigned dirty_submodule = 0;
-	const char *line, *next_line;
 	const char *git_dir;
 
 	strbuf_addf(&buf, "%s/.git", path);
@@ -1066,42 +1058,45 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)
 	}
 	strbuf_reset(&buf);
 
+	argv_array_pushl(&cp.args, "status", "--porcelain=2", NULL);
 	if (ignore_untracked)
-		argv[2] = "-uno";
+		argv_array_push(&cp.args, "-uno");
 
-	cp.argv = argv;
 	prepare_submodule_repo_env(&cp.env_array);
 	cp.git_cmd = 1;
 	cp.no_stdin = 1;
 	cp.out = -1;
 	cp.dir = path;
 	if (start_command(&cp))
-		die("Could not run 'git status --porcelain' in submodule %s", path);
+		die("Could not run 'git status --porcelain=2' in submodule %s", path);
 
-	len = strbuf_read(&buf, cp.out, 1024);
-	line = buf.buf;
-	while (len > 2) {
-		if ((line[0] == '?') && (line[1] == '?')) {
+	while (strbuf_getwholeline_fd(&buf, cp.out, '\n') != EOF) {
+		/* regular untracked files */
+		if (buf.buf[0] == '?')
 			dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;
-			if (dirty_submodule & DIRTY_SUBMODULE_MODIFIED)
-				break;
-		} else {
-			dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
-			if (ignore_untracked ||
-			    (dirty_submodule & DIRTY_SUBMODULE_UNTRACKED))
-				break;
+
+		/* regular unmerged and renamed files */
+		if (buf.buf[0] == 'u' ||
+		    buf.buf[0] == '1' ||
+		    buf.buf[0] == '2') {
+			if (buf.buf[5] == 'S') {
+				/* nested submodule handling */
+				if (buf.buf[6] == 'C' || buf.buf[7] == 'M')
+					dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
+				if (buf.buf[8] == 'U')
+					dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;
+			} else
+				dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;
 		}
-		next_line = strchr(line, '\n');
-		if (!next_line)
-			break;
-		next_line++;
-		len -= (next_line - line);
-		line = next_line;
+
+		if (dirty_submodule & DIRTY_SUBMODULE_MODIFIED &&
+		    dirty_submodule & DIRTY_SUBMODULE_UNTRACKED)
+				break;
 	}
 	close(cp.out);
 
 	if (finish_command(&cp))
-		die("'git status --porcelain' failed in submodule %s", path);
+		die("'git status --porcelain=2' failed in submodule %s", path);
 
 	strbuf_release(&buf);
 	return dirty_submodule;
-- 
2.12.1.432.gfe308fe33c.dirty

Re: [PATCH 1/3] submodule.c: port is_submodule_modified to use porcelain 2

[Jonathan Nieder]

Stefan Beller wrote:

> Migrate 'is_submodule_modified' to the new porcelain format of
> git-status.
>
> As the old porcelain only reported ' M' for submodules, no
> matter what happened inside the submodule (untracked files,
> changes to tracked files or move of HEAD), the new API
> properly reports the different scenarios.
[...]
>  submodule.c | 53 ++++++++++++++++++++++++-----------------------------
>  1 file changed, 24 insertions(+), 29 deletions(-)

Neat.  Is this something that could be covered in tests, or should I
be patient and rely on patch 3/3 for that?

I think this would be easier to understand if it were two patches: one
that switched to --porcelain=2 with no change in behavior, and another
that took advantage of --porcelain=2 to return richer information.  As
is, I had trouble verifying that this isn't going to break anything
--- there's not enough local information here and in submodule.h to
tell what callers may rely on and I didn't audit callers.

Thanks and hope that helps,
Jonathan

Re: [PATCH 1/3] submodule.c: port is_submodule_modified to use porcelain 2

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Stefan Beller wrote:
>
>> Migrate 'is_submodule_modified' to the new porcelain format of
>> git-status.
>>
>> As the old porcelain only reported ' M' for submodules, no
>> matter what happened inside the submodule (untracked files,
>> changes to tracked files or move of HEAD), the new API
>> properly reports the different scenarios.
> [...]
>>  submodule.c | 53 ++++++++++++++++++++++++-----------------------------
>>  1 file changed, 24 insertions(+), 29 deletions(-)
>
> Neat.  Is this something that could be covered in tests, or should I
> be patient and rely on patch 3/3 for that?
>
> I think this would be easier to understand if it were two patches: one
> that switched to --porcelain=2 with no change in behavior, and another
> that took advantage of --porcelain=2 to return richer information.  

That sounds like a sensible organization.

> As is, I had trouble verifying that this isn't going to break
> anything --- there's not enough local information here and in
> submodule.h to tell what callers may rely on and I didn't audit
> callers.

Re: [PATCH 1/3] submodule.c: port is_submodule_modified to use porcelain 2

[Stefan Beller]

On Wed, Mar 22, 2017 at 11:09 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> Stefan Beller wrote:
>>
>>> Migrate 'is_submodule_modified' to the new porcelain format of
>>> git-status.
>>>
>>> As the old porcelain only reported ' M' for submodules, no
>>> matter what happened inside the submodule (untracked files,
>>> changes to tracked files or move of HEAD), the new API
>>> properly reports the different scenarios.
>> [...]
>>>  submodule.c | 53 ++++++++++++++++++++++++-----------------------------
>>>  1 file changed, 24 insertions(+), 29 deletions(-)
>>
>> Neat.  Is this something that could be covered in tests, or should I
>> be patient and rely on patch 3/3 for that?

I am not sure how to cover it in tests properly, as we do not expose this
function to the outside directly.

This function is used in only one place (diff-lib.c, in
match_stat_with_submodule,
which itself is used in run_diff_files and get_stat_data), which is deep down
in the diff library.

>> I think this would be easier to understand if it were two patches: one
>> that switched to --porcelain=2 with no change in behavior,

I don't think so as it would double the code to review.
I'll see if I can present this conversion in an easier way.

> That sounds like a sensible organization.

ok.

Thanks,
Stefan