[PATCH 0/4] dir: Treat a repository without commits as a repository

[PATCH 0/4] dir: Treat a repository without commits as a repository

[Kyle Meyer]

When the working tree contains a repository with no commits, it's
treated as an empty directory, not a repository:

    $ git init
    $ git init no-commit && touch no-commit/untracked
    $ git init with-commit && touch with-commit/untracked
    $ git -C with-commit commit --allow-empty -mmsg
    $ git ls-files -o
    no-commit/untracked
    with-commit/

That's admittedly a weird case that is unlikely to happen in the wild,
and indeed I didn't observe it in the wild.  In DataLad, we rely on
`ls-files -o` to stop at the boundary of a submodule (like it does
with "with-commit" above), and we noticed the "repository with no
commit" exception on a minimal snippet that was meant for debugging
another issue [*].

This series makes the "is repository?" check in treat_directory() use
is_nonbare_repository_dir() instead of resolve_gitlink_ref() so that a
repository without any commits is treated like a repository rather
than an empty directory.  This makes the reporting from commands like
ls-files consistent for the case shown above, and it also avoids the
case where 'git add no-commit' (no trailing slash) adds the untracked
files of the no-commit/ repository to the index of the current
repository.

[*]: https://github.com/datalad/datalad/issues/3139#issuecomment-460542647

Kyle Meyer (4):
  submodule: refuse to add repository with no commits
  t3000: move non-submodule repo test to separate file
  t3009: test that ls-files -o traverses bogus repo
  dir: do not traverse repositories with no commits

 dir.c                                   |  6 ++-
 git-submodule.sh                        | 12 +++++-
 t/t3000-ls-files-others.sh              |  7 ----
 t/t3009-ls-files-others-nonsubmodule.sh | 56 +++++++++++++++++++++++++
 t/t3700-add.sh                          |  1 +
 t/t7400-submodule-basic.sh              | 11 ++++-
 6 files changed, 81 insertions(+), 12 deletions(-)
 create mode 100755 t/t3009-ls-files-others-nonsubmodule.sh

-- 
2.21.0

[PATCH 1/4] submodule: refuse to add repository with no commits

[Kyle Meyer]

When the path given to 'git submodule add' is an existing repository
that is not in the index, the repository is passed to 'git add'.  If
this repository doesn't have any commits, we don't get a useful
result: there is no subproject OID to track, and any untracked files
in the sub-repository are added to the current repository.

Detect if the path is a repository with no commits and abort to avoid
getting into this state unless --force is used.  Reacting to --force
isn't very useful, especially because an upcoming commit will make
'git add' fail in this situation, but it allows us to use the same
'git add --dry-run' condition as the ignored path case.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 git-submodule.sh           | 12 ++++++++++--
 t/t7400-submodule-basic.sh | 11 ++++++++++-
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 514ede2596..6c74656027 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -234,10 +234,18 @@ cmd_add()
 	if test -z "$force" &&
 		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
 	then
-		eval_gettextln "The following path is ignored by one of your .gitignore files:
+		if test -d "$sm_path" &&
+			test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null) &&
+			! git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null
+		then
+			die "$(eval_gettext "'\$sm_path' does not have any commits")"
+		else
+			eval_gettextln "\
+The following path is ignored by one of your .gitignore files:
 \$sm_path
 Use -f if you really want to add it." >&2
-		exit 1
+			exit 1
+		fi
 	fi
 
 	if test -n "$custom_name"
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index aba2d4d6ee..6adf640143 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -46,6 +46,15 @@ test_expect_success 'submodule update aborts on missing gitmodules url' '
 	test_must_fail git submodule init
 '
 
+test_expect_success 'add aborts on repository with no commits' '
+	cat >expected <<-\EOF &&
+	'"'repo-no-commits'"' does not have any commits
+	EOF
+	git init repo-no-commits &&
+	test_must_fail git submodule add ../a ./repo-no-commits 2>actual &&
+	test_i18ncmp expected actual
+'
+
 test_expect_success 'setup - repository in init subdirectory' '
 	mkdir init &&
 	(
@@ -809,7 +818,7 @@ test_expect_success '../bar/a/b/c works with relative local path - ../foo/bar.gi
 		cp pristine-.git-config .git/config &&
 		cp pristine-.gitmodules .gitmodules &&
 		mkdir -p a/b/c &&
-		(cd a/b/c && git init) &&
+		(cd a/b/c && git init && test_commit msg) &&
 		git config remote.origin.url ../foo/bar.git &&
 		git submodule add ../bar/a/b/c ./a/b/c &&
 		git submodule init &&
-- 
2.21.0

[PATCH 2/4] t3000: move non-submodule repo test to separate file

[Kyle Meyer]

a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
2016-01-22) added a test to t3000-ls-files-others.sh to check that
'ls-files -o' does not die() when given a subdirectory that looks like
a repository but is actually a subdirectory containing a bogus .git
file.

Move this test to a separate file in preparation for testing scenarios
with non-submodule repositories that are not bogus.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 t/t3000-ls-files-others.sh              |  7 -------
 t/t3009-ls-files-others-nonsubmodule.sh | 21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 7 deletions(-)
 create mode 100755 t/t3009-ls-files-others-nonsubmodule.sh

diff --git a/t/t3000-ls-files-others.sh b/t/t3000-ls-files-others.sh
index afd4756134..b4f9fc4580 100755
--- a/t/t3000-ls-files-others.sh
+++ b/t/t3000-ls-files-others.sh
@@ -65,13 +65,6 @@ test_expect_success '--no-empty-directory hides empty directory' '
 	test_cmp expected3 output
 '
 
-test_expect_success 'ls-files --others handles non-submodule .git' '
-	mkdir not-a-submodule &&
-	echo foo >not-a-submodule/.git &&
-	git ls-files -o >output &&
-	test_cmp expected1 output
-'
-
 test_expect_success SYMLINKS 'ls-files --others with symlinked submodule' '
 	git init super &&
 	git init sub &&
diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
new file mode 100755
index 0000000000..cc66a4a14d
--- /dev/null
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+test_description='test git ls-files --others with non-submodule repositories'
+
+. ./test-lib.sh
+
+test_expect_success 'setup: expected output' '
+	cat >expected <<-EOF
+	expected
+	output
+	EOF
+'
+
+test_expect_success 'ls-files --others handles non-submodule .git' '
+	mkdir not-a-submodule &&
+	echo foo >not-a-submodule/.git &&
+	git ls-files -o >output &&
+	test_cmp expected output
+'
+
+test_done
-- 
2.21.0

[PATCH 3/4] t3009: test that ls-files -o traverses bogus repo

[Kyle Meyer]

When a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
2016-01-22) added this test, the purpose was to check the 'ls-files
-o' didn't die() when processing the bogus repository.  The expected
output didn't even need to be adjusted for the addition because the
bogus repository is treated as an empty directory.

Let's add another scenario to test that 'ls-files -o' lists an
untracked file within this directory.  Doing so isn't useful for the
original purpose of the test, but it will be helpful for highlighting
that this traversal does _not_ happen in the non-bogus scenarios that
will be added next.  Rename the original subdirectory to highlight the
difference with the new one.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 t/t3009-ls-files-others-nonsubmodule.sh | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
index cc66a4a14d..9ed75928aa 100755
--- a/t/t3009-ls-files-others-nonsubmodule.sh
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -1,6 +1,14 @@
 #!/bin/sh
 
-test_description='test git ls-files --others with non-submodule repositories'
+test_description='test git ls-files --others with non-submodule repositories
+
+This test runs git ls-files --others with the following working tree:
+
+    repo-bogus-no-files/
+      directory with no files aside from a bogus .git file
+    repo-bogus-untracked-file/
+      directory with a bogus .git file and another untracked file
+'
 
 . ./test-lib.sh
 
@@ -8,12 +16,19 @@ test_expect_success 'setup: expected output' '
 	cat >expected <<-EOF
 	expected
 	output
+	repo-bogus-untracked-file/untracked
 	EOF
 '
 
+test_expect_success 'setup: directories' '
+	mkdir repo-bogus-no-files &&
+	echo foo >repo-bogus-no-files/.git &&
+	mkdir repo-bogus-untracked-file &&
+	echo foo >repo-bogus-untracked-file/.git &&
+	: >repo-bogus-untracked-file/untracked
+'
+
 test_expect_success 'ls-files --others handles non-submodule .git' '
-	mkdir not-a-submodule &&
-	echo foo >not-a-submodule/.git &&
 	git ls-files -o >output &&
 	test_cmp expected output
 '
-- 
2.21.0

[PATCH 4/4] dir: do not traverse repositories with no commits

[Kyle Meyer]

When treat_directory() encounters a directory that is not in the index
and DIR_NO_GITLINKS is unset, it calls resolve_gitlink_ref() to decide
if a directory looks like a repository, in which case the directory
won't be traversed.  As a result, 'status -uall' and 'ls-files -o'
will show only the directory, even when there are untracked files
within the directory.

For the unusual case where a repository doesn't have any commits,
resolve_gitlink_ref() returns -1 because HEAD cannot be resolved, and
the directory is treated as a normal directory (i.e. traversal does
not stop at the repository boundary).  The status and ls-files
commands above list untracked files within the repository rather than
showing only the top-level directory.

The above case is a corner case in an already unusual situation of the
working tree containing a repository that is not a tracked submodule,
but we might as well treat anything that looks like a repository
consistently.  Loosen the "looks like a repository" criteria in
treat_directory() by replacing resolve_gitlink_ref() with
is_nonbare_repository_dir(), one of the checks that is performed
downstream when resolve_gitlink_ref() is called with an empty
repository.

As the required update to t3700-add shows, being looser with the check
means that we're stricter when adding empty repositories to the index:

  % git add repo
  warning: adding embedded git repository: repo
  hint: You've added another git repository inside your current repository.
  hint: [...]
  error: unable to index file 'repo/'
  fatal: adding files failed

That error message isn't particularly helpful in this situation, but
it seems preferable to the old behavior of adding the repository's
untracked files.  And if the caller really wants the previous
behavior, they can get it by adding a trailing slash.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 dir.c                                   |  6 ++++--
 t/t3009-ls-files-others-nonsubmodule.sh | 22 +++++++++++++++++++++-
 t/t3700-add.sh                          |  1 +
 3 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/dir.c b/dir.c
index b2cabadf25..a4e59eb351 100644
--- a/dir.c
+++ b/dir.c
@@ -1467,9 +1467,11 @@ static enum path_treatment treat_directory(struct dir_struct *dir,
 			return path_none;
 		}
 		if (!(dir->flags & DIR_NO_GITLINKS)) {
-			struct object_id oid;
-			if (resolve_gitlink_ref(dirname, "HEAD", &oid) == 0)
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addstr(&sb, dirname);
+			if (is_nonbare_repository_dir(&sb))
 				return exclude ? path_excluded : path_untracked;
+			strbuf_release(&sb);
 		}
 		return path_recurse;
 	}
diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
index 9ed75928aa..be4e7e26bc 100755
--- a/t/t3009-ls-files-others-nonsubmodule.sh
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -8,6 +8,14 @@ This test runs git ls-files --others with the following working tree:
       directory with no files aside from a bogus .git file
     repo-bogus-untracked-file/
       directory with a bogus .git file and another untracked file
+    repo-no-commit-no-files/
+      git repository without a commit or a file
+    repo-no-commit-untracked-file/
+      git repository without a commit but with an untracked file
+    repo-with-commit-no-files/
+      git repository with a commit and no untracked files
+    repo-with-commit-untracked-file/
+      git repository with a commit and an untracked file
 '
 
 . ./test-lib.sh
@@ -17,6 +25,10 @@ test_expect_success 'setup: expected output' '
 	expected
 	output
 	repo-bogus-untracked-file/untracked
+	repo-no-commit-no-files/
+	repo-no-commit-untracked-file/
+	repo-with-commit-no-files/
+	repo-with-commit-untracked-file/
 	EOF
 '
 
@@ -25,7 +37,15 @@ test_expect_success 'setup: directories' '
 	echo foo >repo-bogus-no-files/.git &&
 	mkdir repo-bogus-untracked-file &&
 	echo foo >repo-bogus-untracked-file/.git &&
-	: >repo-bogus-untracked-file/untracked
+	: >repo-bogus-untracked-file/untracked &&
+	git init repo-no-commit-no-files &&
+	git init repo-no-commit-untracked-file &&
+	: >repo-no-commit-untracked-file/untracked &&
+	git init repo-with-commit-no-files &&
+	git -C repo-with-commit-no-files commit --allow-empty -mmsg &&
+	git init repo-with-commit-untracked-file &&
+	test_commit -C repo-with-commit-untracked-file msg &&
+	: >repo-with-commit-untracked-file/untracked
 '
 
 test_expect_success 'ls-files --others handles non-submodule .git' '
diff --git a/t/t3700-add.sh b/t/t3700-add.sh
index be582a513b..5a8425962b 100755
--- a/t/t3700-add.sh
+++ b/t/t3700-add.sh
@@ -396,6 +396,7 @@ test_expect_success 'no file status change if no pathspec is given in subdir' '
 '
 
 test_expect_success 'all statuses changed in folder if . is given' '
+	rm -fr empty &&
 	git add --chmod=+x . &&
 	test $(git ls-files --stage | grep ^100644 | wc -l) -eq 0 &&
 	git add --chmod=-x . &&
-- 
2.21.0

Re: [PATCH 1/4] submodule: refuse to add repository with no commits

[Kyle Meyer]

Kyle Meyer <kyle@kyleam.com> writes:

[...]

> diff --git a/git-submodule.sh b/git-submodule.sh
> index 514ede2596..6c74656027 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -234,10 +234,18 @@ cmd_add()
>  	if test -z "$force" &&
>  		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
>  	then
> -		eval_gettextln "The following path is ignored by one of your .gitignore files:
> +		if test -d "$sm_path" &&
> +			test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null) &&
> +			! git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null
> +		then
> +			die "$(eval_gettext "'\$sm_path' does not have any commits")"
> +		else
> +			eval_gettextln "\
> +The following path is ignored by one of your .gitignore files:
>  \$sm_path
>  Use -f if you really want to add it." >&2
> -		exit 1
> +			exit 1
> +		fi

I didn't think through this check, which would have been obvious had I
ran the added test without the rest of the patches in this series.  It
assumes that the 'git add --dry-run' call fails, but that failure
depends on the last patch of this series.  So I'd need to move this
patch to the end or find a new place for this "no commits" check.

[PATCH v2 0/4] dir: Treat a repository without commits as a repository

[Kyle Meyer]

> Kyle Meyer <kyle@kyleam.com> writes:
>
> [...]
>
>> diff --git a/git-submodule.sh b/git-submodule.sh
>> index 514ede2596..6c74656027 100755
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -234,10 +234,18 @@ cmd_add()
>>  	if test -z "$force" &&
>>  		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
>>  	then
>> -		eval_gettextln "The following path is ignored by one of your .gitignore files:
>> +		if test -d "$sm_path" &&
>> +			test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null) &&
>> +			! git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null
>> +		then
>> +			die "$(eval_gettext "'\$sm_path' does not have any commits")"
>> +		else
>> +			eval_gettextln "\
>> +The following path is ignored by one of your .gitignore files:
>>  \$sm_path
>>  Use -f if you really want to add it." >&2
>> -		exit 1
>> +			exit 1
>> +		fi
>
> I didn't think through this check, which would have been obvious had I
> ran the added test without the rest of the patches in this series.  It
> assumes that the 'git add --dry-run' call fails, but that failure
> depends on the last patch of this series.  So I'd need to move this
> patch to the end or find a new place for this "no commits" check.

v2 moves the "no commits" check outside of the 'git add --dry-run'
failure condition so that the first patch doesn't depend on the final
patch in this series.


Kyle Meyer (4):
  submodule: refuse to add repository with no commits
  t3000: move non-submodule repo test to separate file
  t3009: test that ls-files -o traverses bogus repo
  dir: do not traverse repositories with no commits

 dir.c                                   |  6 ++-
 git-submodule.sh                        |  7 ++++
 t/t3000-ls-files-others.sh              |  7 ----
 t/t3009-ls-files-others-nonsubmodule.sh | 56 +++++++++++++++++++++++++
 t/t3700-add.sh                          |  1 +
 t/t7400-submodule-basic.sh              | 11 ++++-
 6 files changed, 78 insertions(+), 10 deletions(-)
 create mode 100755 t/t3009-ls-files-others-nonsubmodule.sh

Range-diff against v1:
1:  e0db7e3c3c ! 1:  b080e2c557 submodule: refuse to add repository with no commits
    @@ -9,10 +9,9 @@
         in the sub-repository are added to the current repository.
     
         Detect if the path is a repository with no commits and abort to avoid
    -    getting into this state unless --force is used.  Reacting to --force
    -    isn't very useful, especially because an upcoming commit will make
    -    'git add' fail in this situation, but it allows us to use the same
    -    'git add --dry-run' condition as the ignored path case.
    +    getting into this state.  Note that this check must come before the
    +    'git add --dry-run' check because an upcoming commit will make 'git
    +    add' fail in this situation.
     
         Signed-off-by: Kyle Meyer <kyle@kyleam.com>
     
    @@ -20,26 +19,19 @@
      --- a/git-submodule.sh
      +++ b/git-submodule.sh
     @@
    + 		die "$(eval_gettext "'\$sm_path' already exists in the index and is not a submodule")"
    + 	fi
    + 
    ++	if test -d "$sm_path" &&
    ++		test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null)
    ++	then
    ++	    git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null ||
    ++	    die "$(eval_gettext "'\$sm_path' does not have any commits")"
    ++	fi
    ++
      	if test -z "$force" &&
      		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
      	then
    --		eval_gettextln "The following path is ignored by one of your .gitignore files:
    -+		if test -d "$sm_path" &&
    -+			test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null) &&
    -+			! git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null
    -+		then
    -+			die "$(eval_gettext "'\$sm_path' does not have any commits")"
    -+		else
    -+			eval_gettextln "\
    -+The following path is ignored by one of your .gitignore files:
    - \$sm_path
    - Use -f if you really want to add it." >&2
    --		exit 1
    -+			exit 1
    -+		fi
    - 	fi
    - 
    - 	if test -n "$custom_name"
     
      diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
      --- a/t/t7400-submodule-basic.sh
2:  6eed1f5daf = 2:  c027701842 t3000: move non-submodule repo test to separate file
3:  7ba3209762 = 3:  97f53e30c0 t3009: test that ls-files -o traverses bogus repo
4:  2901375dc1 = 4:  a926b87102 dir: do not traverse repositories with no commits
-- 
2.21.0

[PATCH v2 1/4] submodule: refuse to add repository with no commits

[Kyle Meyer]

When the path given to 'git submodule add' is an existing repository
that is not in the index, the repository is passed to 'git add'.  If
this repository doesn't have any commits, we don't get a useful
result: there is no subproject OID to track, and any untracked files
in the sub-repository are added to the current repository.

Detect if the path is a repository with no commits and abort to avoid
getting into this state.  Note that this check must come before the
'git add --dry-run' check because an upcoming commit will make 'git
add' fail in this situation.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 git-submodule.sh           |  7 +++++++
 t/t7400-submodule-basic.sh | 11 ++++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 514ede2596..5a9dc9bf7e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -231,6 +231,13 @@ cmd_add()
 		die "$(eval_gettext "'\$sm_path' already exists in the index and is not a submodule")"
 	fi
 
+	if test -d "$sm_path" &&
+		test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null)
+	then
+	    git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null ||
+	    die "$(eval_gettext "'\$sm_path' does not have any commits")"
+	fi
+
 	if test -z "$force" &&
 		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
 	then
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index aba2d4d6ee..6adf640143 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -46,6 +46,15 @@ test_expect_success 'submodule update aborts on missing gitmodules url' '
 	test_must_fail git submodule init
 '
 
+test_expect_success 'add aborts on repository with no commits' '
+	cat >expected <<-\EOF &&
+	'"'repo-no-commits'"' does not have any commits
+	EOF
+	git init repo-no-commits &&
+	test_must_fail git submodule add ../a ./repo-no-commits 2>actual &&
+	test_i18ncmp expected actual
+'
+
 test_expect_success 'setup - repository in init subdirectory' '
 	mkdir init &&
 	(
@@ -809,7 +818,7 @@ test_expect_success '../bar/a/b/c works with relative local path - ../foo/bar.gi
 		cp pristine-.git-config .git/config &&
 		cp pristine-.gitmodules .gitmodules &&
 		mkdir -p a/b/c &&
-		(cd a/b/c && git init) &&
+		(cd a/b/c && git init && test_commit msg) &&
 		git config remote.origin.url ../foo/bar.git &&
 		git submodule add ../bar/a/b/c ./a/b/c &&
 		git submodule init &&
-- 
2.21.0

[PATCH v2 2/4] t3000: move non-submodule repo test to separate file

[Kyle Meyer]

a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
2016-01-22) added a test to t3000-ls-files-others.sh to check that
'ls-files -o' does not die() when given a subdirectory that looks like
a repository but is actually a subdirectory containing a bogus .git
file.

Move this test to a separate file in preparation for testing scenarios
with non-submodule repositories that are not bogus.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 t/t3000-ls-files-others.sh              |  7 -------
 t/t3009-ls-files-others-nonsubmodule.sh | 21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 7 deletions(-)
 create mode 100755 t/t3009-ls-files-others-nonsubmodule.sh

diff --git a/t/t3000-ls-files-others.sh b/t/t3000-ls-files-others.sh
index afd4756134..b4f9fc4580 100755
--- a/t/t3000-ls-files-others.sh
+++ b/t/t3000-ls-files-others.sh
@@ -65,13 +65,6 @@ test_expect_success '--no-empty-directory hides empty directory' '
 	test_cmp expected3 output
 '
 
-test_expect_success 'ls-files --others handles non-submodule .git' '
-	mkdir not-a-submodule &&
-	echo foo >not-a-submodule/.git &&
-	git ls-files -o >output &&
-	test_cmp expected1 output
-'
-
 test_expect_success SYMLINKS 'ls-files --others with symlinked submodule' '
 	git init super &&
 	git init sub &&
diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
new file mode 100755
index 0000000000..cc66a4a14d
--- /dev/null
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+test_description='test git ls-files --others with non-submodule repositories'
+
+. ./test-lib.sh
+
+test_expect_success 'setup: expected output' '
+	cat >expected <<-EOF
+	expected
+	output
+	EOF
+'
+
+test_expect_success 'ls-files --others handles non-submodule .git' '
+	mkdir not-a-submodule &&
+	echo foo >not-a-submodule/.git &&
+	git ls-files -o >output &&
+	test_cmp expected output
+'
+
+test_done
-- 
2.21.0

[PATCH v2 3/4] t3009: test that ls-files -o traverses bogus repo

[Kyle Meyer]

When a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
2016-01-22) added this test, the purpose was to check the 'ls-files
-o' didn't die() when processing the bogus repository.  The expected
output didn't even need to be adjusted for the addition because the
bogus repository is treated as an empty directory.

Let's add another scenario to test that 'ls-files -o' lists an
untracked file within this directory.  Doing so isn't useful for the
original purpose of the test, but it will be helpful for highlighting
that this traversal does _not_ happen in the non-bogus scenarios that
will be added next.  Rename the original subdirectory to highlight the
difference with the new one.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 t/t3009-ls-files-others-nonsubmodule.sh | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
index cc66a4a14d..9ed75928aa 100755
--- a/t/t3009-ls-files-others-nonsubmodule.sh
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -1,6 +1,14 @@
 #!/bin/sh
 
-test_description='test git ls-files --others with non-submodule repositories'
+test_description='test git ls-files --others with non-submodule repositories
+
+This test runs git ls-files --others with the following working tree:
+
+    repo-bogus-no-files/
+      directory with no files aside from a bogus .git file
+    repo-bogus-untracked-file/
+      directory with a bogus .git file and another untracked file
+'
 
 . ./test-lib.sh
 
@@ -8,12 +16,19 @@ test_expect_success 'setup: expected output' '
 	cat >expected <<-EOF
 	expected
 	output
+	repo-bogus-untracked-file/untracked
 	EOF
 '
 
+test_expect_success 'setup: directories' '
+	mkdir repo-bogus-no-files &&
+	echo foo >repo-bogus-no-files/.git &&
+	mkdir repo-bogus-untracked-file &&
+	echo foo >repo-bogus-untracked-file/.git &&
+	: >repo-bogus-untracked-file/untracked
+'
+
 test_expect_success 'ls-files --others handles non-submodule .git' '
-	mkdir not-a-submodule &&
-	echo foo >not-a-submodule/.git &&
 	git ls-files -o >output &&
 	test_cmp expected output
 '
-- 
2.21.0

[PATCH v2 4/4] dir: do not traverse repositories with no commits

[Kyle Meyer]

When treat_directory() encounters a directory that is not in the index
and DIR_NO_GITLINKS is unset, it calls resolve_gitlink_ref() to decide
if a directory looks like a repository, in which case the directory
won't be traversed.  As a result, 'status -uall' and 'ls-files -o'
will show only the directory, even when there are untracked files
within the directory.

For the unusual case where a repository doesn't have any commits,
resolve_gitlink_ref() returns -1 because HEAD cannot be resolved, and
the directory is treated as a normal directory (i.e. traversal does
not stop at the repository boundary).  The status and ls-files
commands above list untracked files within the repository rather than
showing only the top-level directory.

The above case is a corner case in an already unusual situation of the
working tree containing a repository that is not a tracked submodule,
but we might as well treat anything that looks like a repository
consistently.  Loosen the "looks like a repository" criteria in
treat_directory() by replacing resolve_gitlink_ref() with
is_nonbare_repository_dir(), one of the checks that is performed
downstream when resolve_gitlink_ref() is called with an empty
repository.

As the required update to t3700-add shows, being looser with the check
means that we're stricter when adding empty repositories to the index:

  % git add repo
  warning: adding embedded git repository: repo
  hint: You've added another git repository inside your current repository.
  hint: [...]
  error: unable to index file 'repo/'
  fatal: adding files failed

That error message isn't particularly helpful in this situation, but
it seems preferable to the old behavior of adding the repository's
untracked files.  And if the caller really wants the previous
behavior, they can get it by adding a trailing slash.

Signed-off-by: Kyle Meyer <kyle@kyleam.com>
---
 dir.c                                   |  6 ++++--
 t/t3009-ls-files-others-nonsubmodule.sh | 22 +++++++++++++++++++++-
 t/t3700-add.sh                          |  1 +
 3 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/dir.c b/dir.c
index b2cabadf25..a4e59eb351 100644
--- a/dir.c
+++ b/dir.c
@@ -1467,9 +1467,11 @@ static enum path_treatment treat_directory(struct dir_struct *dir,
 			return path_none;
 		}
 		if (!(dir->flags & DIR_NO_GITLINKS)) {
-			struct object_id oid;
-			if (resolve_gitlink_ref(dirname, "HEAD", &oid) == 0)
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addstr(&sb, dirname);
+			if (is_nonbare_repository_dir(&sb))
 				return exclude ? path_excluded : path_untracked;
+			strbuf_release(&sb);
 		}
 		return path_recurse;
 	}
diff --git a/t/t3009-ls-files-others-nonsubmodule.sh b/t/t3009-ls-files-others-nonsubmodule.sh
index 9ed75928aa..be4e7e26bc 100755
--- a/t/t3009-ls-files-others-nonsubmodule.sh
+++ b/t/t3009-ls-files-others-nonsubmodule.sh
@@ -8,6 +8,14 @@ This test runs git ls-files --others with the following working tree:
       directory with no files aside from a bogus .git file
     repo-bogus-untracked-file/
       directory with a bogus .git file and another untracked file
+    repo-no-commit-no-files/
+      git repository without a commit or a file
+    repo-no-commit-untracked-file/
+      git repository without a commit but with an untracked file
+    repo-with-commit-no-files/
+      git repository with a commit and no untracked files
+    repo-with-commit-untracked-file/
+      git repository with a commit and an untracked file
 '
 
 . ./test-lib.sh
@@ -17,6 +25,10 @@ test_expect_success 'setup: expected output' '
 	expected
 	output
 	repo-bogus-untracked-file/untracked
+	repo-no-commit-no-files/
+	repo-no-commit-untracked-file/
+	repo-with-commit-no-files/
+	repo-with-commit-untracked-file/
 	EOF
 '
 
@@ -25,7 +37,15 @@ test_expect_success 'setup: directories' '
 	echo foo >repo-bogus-no-files/.git &&
 	mkdir repo-bogus-untracked-file &&
 	echo foo >repo-bogus-untracked-file/.git &&
-	: >repo-bogus-untracked-file/untracked
+	: >repo-bogus-untracked-file/untracked &&
+	git init repo-no-commit-no-files &&
+	git init repo-no-commit-untracked-file &&
+	: >repo-no-commit-untracked-file/untracked &&
+	git init repo-with-commit-no-files &&
+	git -C repo-with-commit-no-files commit --allow-empty -mmsg &&
+	git init repo-with-commit-untracked-file &&
+	test_commit -C repo-with-commit-untracked-file msg &&
+	: >repo-with-commit-untracked-file/untracked
 '
 
 test_expect_success 'ls-files --others handles non-submodule .git' '
diff --git a/t/t3700-add.sh b/t/t3700-add.sh
index be582a513b..5a8425962b 100755
--- a/t/t3700-add.sh
+++ b/t/t3700-add.sh
@@ -396,6 +396,7 @@ test_expect_success 'no file status change if no pathspec is given in subdir' '
 '
 
 test_expect_success 'all statuses changed in folder if . is given' '
+	rm -fr empty &&
 	git add --chmod=+x . &&
 	test $(git ls-files --stage | grep ^100644 | wc -l) -eq 0 &&
 	git add --chmod=-x . &&
-- 
2.21.0

Re: [PATCH v2 2/4] t3000: move non-submodule repo test to separate file

[Junio C Hamano]

Kyle Meyer <kyle@kyleam.com> writes:

> a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
> 2016-01-22) added a test to t3000-ls-files-others.sh to check that
> 'ls-files -o' does not die() when given a subdirectory that looks like
> a repository but is actually a subdirectory containing a bogus .git
> file.
>
> Move this test to a separate file in preparation for testing scenarios
> with non-submodule repositories that are not bogus.

It is unclear to me why this is needed.

> +++ b/t/t3009-ls-files-others-nonsubmodule.sh
> @@ -0,0 +1,21 @@
> +#!/bin/sh
> +
> +test_description='test git ls-files --others with non-submodule repositories'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'setup: expected output' '
> +	cat >expected <<-EOF
> +	expected
> +	output
> +	EOF
> +'

I think this is overkill.  Usually we have one expectation for a
single test, so having the above inside the actual test below makes
more sense.

Or are you planning to add more tests before the test_done we see
below, all of which expect the above output?  It would make perfect
sense if it were the case, but I do not think that is what is
happenning here...

> +test_expect_success 'ls-files --others handles non-submodule .git' '
> +	mkdir not-a-submodule &&
> +	echo foo >not-a-submodule/.git &&
> +	git ls-files -o >output &&
> +	test_cmp expected output
> +'
> +
> +test_done

Re: [PATCH v2 4/4] dir: do not traverse repositories with no commits

[Junio C Hamano]

Kyle Meyer <kyle@kyleam.com> writes:

> The above case is a corner case in an already unusual situation of the
> working tree containing a repository that is not a tracked submodule,
> but we might as well treat anything that looks like a repository
> consistently.

Makes sense.

> As the required update to t3700-add shows, being looser with the check
> means that we're stricter when adding empty repositories to the index:
>
>   % git add repo
>   warning: adding embedded git repository: repo
>   hint: You've added another git repository inside your current repository.
>   hint: [...]
>   error: unable to index file 'repo/'
>   fatal: adding files failed

Hmph, why should we have the initial "warning" with hint?  Shouldn't
the above result in an outright error?  Something like:

    $ git add repo ;# or git add repo/
    error: repo does not have a commit checked out
    fatal: adding files failed

because the entire purpose of "git add repo" (or "git add repo/") when
"repo" is a repository is to register the commit that is checked out
in that sub-repository to the index of the top-level repository, as
part of updating (or starting) a submodule.

I very much like the direction these four patches want to take us.
I just have small troubles here and there in their execution, though
;-)

Thanks.

Re: [PATCH v2 2/4] t3000: move non-submodule repo test to separate file

[Kyle Meyer]

Junio C Hamano <gitster@pobox.com> writes:

> Kyle Meyer <kyle@kyleam.com> writes:
>
>> a2d5156c2b (resolve_gitlink_ref: ignore non-repository paths,
>> 2016-01-22) added a test to t3000-ls-files-others.sh to check that
>> 'ls-files -o' does not die() when given a subdirectory that looks like
>> a repository but is actually a subdirectory containing a bogus .git
>> file.
>>
>> Move this test to a separate file in preparation for testing scenarios
>> with non-submodule repositories that are not bogus.
>
> It is unclear to me why this is needed.

It's not needed.  My thinking, which I didn't do a good job of spelling
out above, is

    We're going to be adding a test that checks how 'ls-files -o'
    handles a few different scenarios involving untracked repositories.
    This new test should go into a separate file rather than
    t3000-ls-files-others.sh because it substantially changes the shared
    directory layout that the t3000 tests work on.  Like the upcoming
    test, the "non-submodule .git" test from t3000 deals with a (bogus)
    untracked repository, so let's split it off into a separate test
    file that will be extended with the other scenarios.

Perhaps that's not a good reason to touch t3000, though.  I could drop
this patch, as well as the next one, and just add the new test file in
the final patch.

>> +++ b/t/t3009-ls-files-others-nonsubmodule.sh
>> @@ -0,0 +1,21 @@
>> +#!/bin/sh
>> +
>> +test_description='test git ls-files --others with non-submodule repositories'
>> +
>> +. ./test-lib.sh
>> +
>> +test_expect_success 'setup: expected output' '
>> +	cat >expected <<-EOF
>> +	expected
>> +	output
>> +	EOF
>> +'
>
> I think this is overkill.  Usually we have one expectation for a
> single test, so having the above inside the actual test below makes
> more sense.

OK, I'll move this into the test.

Re: [PATCH v2 4/4] dir: do not traverse repositories with no commits

[Kyle Meyer]

Junio C Hamano <gitster@pobox.com> writes:

> Kyle Meyer <kyle@kyleam.com> writes:

>> As the required update to t3700-add shows, being looser with the check
>> means that we're stricter when adding empty repositories to the index:
>>
>>   % git add repo
>>   warning: adding embedded git repository: repo
>>   hint: You've added another git repository inside your current repository.
>>   hint: [...]
>>   error: unable to index file 'repo/'
>>   fatal: adding files failed
>
> Hmph, why should we have the initial "warning" with hint?  Shouldn't
> the above result in an outright error?  Something like:
>
>     $ git add repo ;# or git add repo/
>     error: repo does not have a commit checked out
>     fatal: adding files failed
>
> because the entire purpose of "git add repo" (or "git add repo/") when
> "repo" is a repository is to register the commit that is checked out
> in that sub-repository to the index of the top-level repository, as
> part of updating (or starting) a submodule.

Right, the hint makes no sense here.  I'll look into adjusting the
output to be appropriate.

As a minor note: I believe your "git add repo/" comments above describe
what should be and not what currently is.  The trailing slash results in
adding the content from the specified repository to the current
repository rather than registering the repository as a submodule:

  $ git --version
  git version 2.21.0
  $ git init
  $ git init a
  $ git -C a commit --allow-empty -mmsg
  $ touch a/f
  $ git add a/
  $ git status -s
  A  a/f

This behavior (and fixing it) was discussed at
https://public-inbox.org/git/20180618111919.GA10085@book.hvoigt.net

Re: [PATCH v2 1/4] submodule: refuse to add repository with no commits

[Junio C Hamano]

Kyle Meyer <kyle@kyleam.com> writes:

> When the path given to 'git submodule add' is an existing repository
> that is not in the index, the repository is passed to 'git add'.  If
> this repository doesn't have any commits, we don't get a useful
> result: there is no subproject OID to track, and any untracked files
> in the sub-repository are added to the current repository.
>
> Detect if the path is a repository with no commits and abort to avoid
> getting into this state.  Note that this check must come before the
> 'git add --dry-run' check because an upcoming commit will make 'git
> add' fail in this situation.

OK, in short, in an already-populated repository,

	$ git init junk ;# there was no 'junk' subdirectory
	$ git add junk

would have been a successful no-op, and after the above two
commands,

	$ >junk/one
	$ git add junk

would have added junk/one as a normal blob to the top-level project,
ignoring the fact that the user wanted to eventually populate junk
as a repository and add it as a submodule.  You'll be teaching "git
add" not to do so, and pay attention to the fact that junk/.git exists,
even though junk/.git/HEAD is on an unborn branch.

Which makes sense to me.

> Signed-off-by: Kyle Meyer <kyle@kyleam.com>
> ---
>  git-submodule.sh           |  7 +++++++
>  t/t7400-submodule-basic.sh | 11 ++++++++++-
>  2 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 514ede2596..5a9dc9bf7e 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -231,6 +231,13 @@ cmd_add()
>  		die "$(eval_gettext "'\$sm_path' already exists in the index and is not a submodule")"
>  	fi
>  
> +	if test -d "$sm_path" &&
> +		test -z $(git -C "$sm_path" rev-parse --show-cdup 2>/dev/null)
> +	then
> +	    git -C "$sm_path" rev-parse --verify -q HEAD >/dev/null ||
> +	    die "$(eval_gettext "'\$sm_path' does not have any commits")"
> +	fi
> +
>  	if test -z "$force" &&
>  		! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
>  	then
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index aba2d4d6ee..6adf640143 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -46,6 +46,15 @@ test_expect_success 'submodule update aborts on missing gitmodules url' '
>  	test_must_fail git submodule init
>  '
>  
> +test_expect_success 'add aborts on repository with no commits' '
> +	cat >expected <<-\EOF &&
> +	'"'repo-no-commits'"' does not have any commits
> +	EOF
> +	git init repo-no-commits &&
> +	test_must_fail git submodule add ../a ./repo-no-commits 2>actual &&
> +	test_i18ncmp expected actual
> +'
> +
>  test_expect_success 'setup - repository in init subdirectory' '
>  	mkdir init &&
>  	(
> @@ -809,7 +818,7 @@ test_expect_success '../bar/a/b/c works with relative local path - ../foo/bar.gi
>  		cp pristine-.git-config .git/config &&
>  		cp pristine-.gitmodules .gitmodules &&
>  		mkdir -p a/b/c &&
> -		(cd a/b/c && git init) &&
> +		(cd a/b/c && git init && test_commit msg) &&
>  		git config remote.origin.url ../foo/bar.git &&
>  		git submodule add ../bar/a/b/c ./a/b/c &&
>  		git submodule init &&