[PATCH 0/2] Sparse index: delete ignored files outside sparse cone

[PATCH 0/2] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee via GitGitGadget]

My commits are technically based on ds/add-with-sparse-index, but could
apply earlier, I believe.

We launched an experimental release [1] of the sparse-index feature to our
internal users. We immediately discovered a problem due to the isolated way
in which we tested the sparse index: we were never building the project and
changing our sparse-checkout definition.

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Users who ran a build in one project and then moved to another still had
build artifacts in their worktree that lived inside the old directories.
Since the files are marked by the .gitignore patterns, these files were not
removed by the 'git sparse-checkout set' command. However, they make the
sparse-index unusable because every 'git status' command needs to expand the
sparse-directory entries in order to see if the files are tracked or not.
This made the first experimental release actually slower for all users
because of this cost.

The solution we shipped to these customers was to change the way our fork
handles these ignored files. Specifically, instead of Git completely
ignoring the files, we changed Git to understand that with cone-mode
sparse-checkout patterns, the users is asking for entire directories to be
removed from the worktree. The link [1] included earlier has this change.

I believe that this is a reasonable expectation, though I recognize that it
might look like breaking the expectations of how .gitignore files work.

To limit the scope of this change, the focus is limited to behave this way
only when the sparse-index is enabled, because otherwise the sparse index
does not behave as intended. With that restriction in mind, the
implementation can use the sparse directory entries to assist.

I imagine that when the sparse index is mature enough, we will enable it by
default when cone mode sparse-checkout is enabled, in which case this
behavior would expand to all users with cone mode sparse-checkout.

I'm interested in the community's thoughts about this change, as it seems
like one that we should make carefully and intentionally.

While the rewrite of the t7519 test seems unrelated, it is required to avoid
a test failure with this change that deletes files outside of the cone. By
moving the test into repositories not at $TRASH_DIRECTORY, we gain more
control over the repository structure.

Thanks, -Stolee

Derrick Stolee (2):
  t7519: rewrite sparse index test
  sparse-checkout: clear tracked sparse dirs

 Documentation/git-sparse-checkout.txt |  6 +++
 builtin/sparse-checkout.c             | 73 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 42 +++++++++++++++
 t/t7519-status-fsmonitor.sh           | 38 +++++++-------
 4 files changed, 142 insertions(+), 17 deletions(-)


base-commit: adf5b15ac3d44d92e0438451ef36631ed3ee2a63
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1009
-- 
gitgitgadget

[PATCH 1/2] t7519: rewrite sparse index test

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The sparse index is tested with the FS Monitor hook and extension since
f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
was very fragile because it shared an index across sparse and non-sparse
behavior. Since that expansion and contraction could cause the index to
lose its FS Monitor bitmap and token, behavior is fragile to changes in
'git sparse-checkout set'.

Rewrite the test to use two clones of the original repo: full and
sparse. This allows us to also keep the test files (actual, expect,
trace2.txt) out of the repos we are testing with 'git status'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index deea88d4431..6f2cf306f66 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
 # If "!" is supplied, then we verify that we do not call ensure_full_index
 # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
 check_sparse_index_behavior () {
-	git status --porcelain=v2 >expect &&
-	git sparse-checkout init --cone --sparse-index &&
-	git sparse-checkout set dir1 dir2 &&
+	git -C full status --porcelain=v2 >expect &&
 	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-		git status --porcelain=v2 >actual &&
+		git -C sparse status --porcelain=v2 >actual &&
 	test_region $1 index ensure_full_index trace2.txt &&
 	test_region fsm_hook query trace2.txt &&
 	test_cmp expect actual &&
-	rm trace2.txt &&
-	git sparse-checkout disable
+	rm trace2.txt
 }
 
 test_expect_success 'status succeeds with sparse index' '
-	git reset --hard &&
+	git clone . full &&
+	git clone . sparse &&
+	git -C sparse sparse-checkout init --cone --sparse-index &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
-	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
-	check_sparse_index_behavior ! &&
-
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 	EOF
-	git config core.fsmonitor .git/hooks/fsmonitor-test &&
+	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
+	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
 	check_sparse_index_behavior ! &&
 
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1/modified\0"
 	EOF
 	check_sparse_index_behavior ! &&
 
-	cp -r dir1 dir1a &&
-	git add dir1a &&
-	git commit -m "add dir1a" &&
+	git -C sparse sparse-checkout add dir1a &&
+
+	for repo in full sparse
+	do
+		cp -r $repo/dir1 $repo/dir1a &&
+		git -C $repo add dir1a &&
+		git -C $repo commit -m "add dir1a" || return 1
+	done &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
 	# This one modifies outside the sparse-checkout definition
 	# and hence we expect to expand the sparse-index.
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1a/modified\0"
 	EOF
-- 
gitgitgadget

[PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When changing the scope of a sparse-checkout using cone mode, we might
have some tracked directories go out of scope. The current logic removes
the tracked files from within those directories, but leaves the ignored
files within those directories. This is a bit unexpected to users who
have given input to Git saying they don't need those directories
anymore.

This is something that is new to the cone mode pattern type: the user
has explicitly said "I want these directories and _not_ those
directories." The typical sparse-checkout patterns more generally apply
to "I want files with with these patterns" so it is natural to leave
ignored files as they are. This focus on directories in cone mode
provides us an opportunity to change the behavior.

Leaving these ignored files in the sparse directories makes it
impossible to gain performance benefits in the sparse index. When we
track into these directories, we need to know if the files are ignored
or not, which might depend on the _tracked_ .gitignore file(s) within
the sparse directory. This depends on the indexed version of the file,
so the sparse directory must be expanded.

By deleting the sparse directories when changing scope (or running 'git
sparse-checkout reapply') we regain these performance benefits as if the
repository was in a clean state.

Since these ignored files are frequently build output or helper files
from IDEs, the users should not need the files now that the tracked
files are removed. If the tracked files reappear, then they will have
newer timestamps than the build artifacts, so the artifacts will need to
be regenerated anyway.

If users depend on ignored files within the sparse directories, then
they have created a bad shape in their repository. Regardless, such
shapes would create risk that changing the behavior for all cone mode
users might be too risky to take on at the moment. Since this data shape
makes it impossible to get performance benefits using the sparse index,
we limit the change to only be enabled when the sparse index is enabled.
Users can opt out of this behavior by disabline the sparse index.

Depending on user feedback or real-world use, we might want to consider
expanding the behavior change to all of cone mode. Since we are
currently restricting to the sparse index case, we can use the existence
of sparse directory entries in the index as indicators of which
directories should be removed.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt |  6 +++
 builtin/sparse-checkout.c             | 73 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 42 +++++++++++++++
 3 files changed, 121 insertions(+)

diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index fdcf43f87cb..c443189f418 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -210,6 +210,12 @@ case-insensitive check. This corrects for case mismatched filenames in the
 'git sparse-checkout set' command to reflect the expected cone in the working
 directory.
 
+When the sparse index is enabled through the `index.sparse` config option,
+the cone mode sparse-checkout patterns will also remove ignored files that
+are not within the sparse-checkout definition. This is important behavior
+to preserve the performance of the sparse index, but also matches that
+cone mode patterns care about directories, not files.
+
 
 SUBMODULES
 ----------
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index a4bdd7c4940..5c03636b6ec 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -15,6 +15,7 @@
 #include "wt-status.h"
 #include "quote.h"
 #include "sparse-index.h"
+#include "run-command.h"
 
 static const char *empty_base = "";
 
@@ -100,6 +101,71 @@ static int sparse_checkout_list(int argc, const char **argv)
 	return 0;
 }
 
+static void clean_tracked_sparse_directories(struct repository *r)
+{
+	int i;
+	struct strbuf path = STRBUF_INIT;
+	size_t pathlen;
+
+	/*
+	 * If we are not using cone mode patterns, then we cannot
+	 * delete directories outside of the sparse cone.
+	 */
+	if (!r || !r->index || !r->index->sparse_checkout_patterns ||
+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
+		return;
+	/*
+	 * NEEDSWORK: For now, only use this behavior when index.sparse
+	 * is enabled. We may want this behavior enabled whenever using
+	 * cone mode patterns.
+	 */
+	prepare_repo_settings(r);
+	if (!r->worktree || !r->settings.sparse_index)
+		return;
+
+	/*
+	 * Since we now depend on the sparse index to enable this
+	 * behavior, use it to our advantage. This process is more
+	 * complicated without it.
+	 */
+	convert_to_sparse(r->index);
+
+	strbuf_addstr(&path, r->worktree);
+	strbuf_complete(&path, '/');
+	pathlen = path.len;
+
+	for (i = 0; i < r->index->cache_nr; i++) {
+		struct cache_entry *ce = r->index->cache[i];
+
+		/*
+		 * Is this a sparse directory? If so, then definitely
+		 * include it. All contained content is outside of the
+		 * patterns.
+		 */
+		if (S_ISSPARSEDIR(ce->ce_mode) &&
+		    repo_file_exists(r, ce->name)) {
+			strbuf_setlen(&path, pathlen);
+			strbuf_addstr(&path, ce->name);
+
+			/*
+			 * Removal is "best effort". If something blocks
+			 * the deletion, then continue with a warning.
+			 */
+			if (remove_dir_recursively(&path, 0))
+				warning(_("failed to remove directory '%s'"), path.buf);
+		}
+	}
+
+	strbuf_release(&path);
+
+	/*
+	 * This is temporary: the sparse-checkout builtin is not
+	 * integrated with the sparse-index yet, so we need to keep
+	 * it full during the process.
+	 */
+	ensure_full_index(r->index);
+}
+
 static int update_working_directory(struct pattern_list *pl)
 {
 	enum update_sparsity_result result;
@@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)
 	else
 		rollback_lock_file(&lock_file);
 
+	clean_tracked_sparse_directories(r);
+
 	r->index->sparse_checkout_patterns = NULL;
 	return result;
 }
@@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
 {
 	int result;
 	int changed_config = 0;
+	struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));
 	struct pattern_list *pl = xcalloc(1, sizeof(*pl));
 
+	get_sparse_checkout_patterns(old_pl);
+
 	switch (m) {
 	case ADD:
 		if (core_sparse_checkout_cone)
@@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
 		set_config(MODE_NO_PATTERNS);
 
 	clear_pattern_list(pl);
+	clear_pattern_list(old_pl);
 	free(pl);
+	free(old_pl);
 	return result;
 }
 
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..43eb314c94a 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -642,4 +642,46 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
 	check_files repo/deep a deeper1
 '
 
+test_expect_success 'cone mode clears ignored subdirectories' '
+	rm repo/.git/info/sparse-checkout &&
+
+	# NEEDSWORK: --sparse-index is required for now
+	git -C repo sparse-checkout init --cone --sparse-index &&
+	git -C repo sparse-checkout set deep/deeper1 &&
+
+	cat >repo/.gitignore <<-\EOF &&
+	obj/
+	*.o
+	EOF
+
+	git -C repo add .gitignore &&
+	git -C repo commit -m ".gitignore" &&
+
+	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
+	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
+		    deep/deeper2/obj/a deep/deeper2/file.o file.o
+	do
+		echo ignored >repo/$file || return 1
+	done &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout reapply &&
+	test_path_is_missing repo/folder1 &&
+	test_path_is_missing repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout set deep/deeper2 &&
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Elijah Newren]

On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> When changing the scope of a sparse-checkout using cone mode, we might
> have some tracked directories go out of scope. The current logic removes
> the tracked files from within those directories, but leaves the ignored
> files within those directories. This is a bit unexpected to users who
> have given input to Git saying they don't need those directories
> anymore.
>
> This is something that is new to the cone mode pattern type: the user
> has explicitly said "I want these directories and _not_ those
> directories." The typical sparse-checkout patterns more generally apply
> to "I want files with with these patterns" so it is natural to leave
> ignored files as they are. This focus on directories in cone mode
> provides us an opportunity to change the behavior.
>
> Leaving these ignored files in the sparse directories makes it
> impossible to gain performance benefits in the sparse index. When we
> track into these directories, we need to know if the files are ignored
> or not, which might depend on the _tracked_ .gitignore file(s) within
> the sparse directory. This depends on the indexed version of the file,
> so the sparse directory must be expanded.

Is this the issue I highlighted at
https://lore.kernel.org/git/CABPp-BHsiLTtv6AuycRrQ5K6q0=ZTJe0kd7uTUL+UT=nxj66zA@mail.gmail.com/,
the paragraph "...I thought the point of add_patterns()..." or is
there more or other things involved here?

> By deleting the sparse directories when changing scope (or running 'git
> sparse-checkout reapply') we regain these performance benefits as if the
> repository was in a clean state.

:-)

However, note that our repository is probably ~25x smaller than yours
in terms of number of files, and 'git status' performance really isn't
that big of an issue for us.  Users complained to us about these
directories sticking around simply because they were confused by the
directories remaining.  (Did sparsify not work?  What's all that
leftover stuff?  Why do they have to deal with it?  etc.)

> Since these ignored files are frequently build output or helper files
> from IDEs, the users should not need the files now that the tracked
> files are removed. If the tracked files reappear, then they will have
> newer timestamps than the build artifacts, so the artifacts will need to
> be regenerated anyway.

This is such a good explanation of why automatically removing these
directories was perfectly fine for us to do.

> If users depend on ignored files within the sparse directories, then
> they have created a bad shape in their repository. Regardless, such
> shapes would create risk that changing the behavior for all cone mode
> users might be too risky to take on at the moment. Since this data shape
> makes it impossible to get performance benefits using the sparse index,
> we limit the change to only be enabled when the sparse index is enabled.
> Users can opt out of this behavior by disabline the sparse index.

s/disabline/disabling/, otherwise, I fully agree with this paragraph.

> Depending on user feedback or real-world use, we might want to consider
> expanding the behavior change to all of cone mode. Since we are
> currently restricting to the sparse index case, we can use the existence
> of sparse directory entries in the index as indicators of which
> directories should be removed.

Let's just expand it to all of cone mode.

> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/git-sparse-checkout.txt |  6 +++
>  builtin/sparse-checkout.c             | 73 +++++++++++++++++++++++++++
>  t/t1091-sparse-checkout-builtin.sh    | 42 +++++++++++++++
>  3 files changed, 121 insertions(+)
>
> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
> index fdcf43f87cb..c443189f418 100644
> --- a/Documentation/git-sparse-checkout.txt
> +++ b/Documentation/git-sparse-checkout.txt
> @@ -210,6 +210,12 @@ case-insensitive check. This corrects for case mismatched filenames in the
>  'git sparse-checkout set' command to reflect the expected cone in the working
>  directory.
>
> +When the sparse index is enabled through the `index.sparse` config option,
> +the cone mode sparse-checkout patterns will also remove ignored files that
> +are not within the sparse-checkout definition. This is important behavior
> +to preserve the performance of the sparse index, but also matches that
> +cone mode patterns care about directories, not files.
> +
>
>  SUBMODULES
>  ----------
> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
> index a4bdd7c4940..5c03636b6ec 100644
> --- a/builtin/sparse-checkout.c
> +++ b/builtin/sparse-checkout.c
> @@ -15,6 +15,7 @@
>  #include "wt-status.h"
>  #include "quote.h"
>  #include "sparse-index.h"
> +#include "run-command.h"
>
>  static const char *empty_base = "";
>
> @@ -100,6 +101,71 @@ static int sparse_checkout_list(int argc, const char **argv)
>         return 0;
>  }
>
> +static void clean_tracked_sparse_directories(struct repository *r)
> +{
> +       int i;
> +       struct strbuf path = STRBUF_INIT;
> +       size_t pathlen;
> +
> +       /*
> +        * If we are not using cone mode patterns, then we cannot
> +        * delete directories outside of the sparse cone.
> +        */
> +       if (!r || !r->index || !r->index->sparse_checkout_patterns ||
> +           !r->index->sparse_checkout_patterns->use_cone_patterns)
> +               return;
> +       /*
> +        * NEEDSWORK: For now, only use this behavior when index.sparse
> +        * is enabled. We may want this behavior enabled whenever using
> +        * cone mode patterns.
> +        */
> +       prepare_repo_settings(r);
> +       if (!r->worktree || !r->settings.sparse_index)
> +               return;

s/may/do/  :-)

> +
> +       /*
> +        * Since we now depend on the sparse index to enable this
> +        * behavior, use it to our advantage. This process is more
> +        * complicated without it.
> +        */

Ah, the real reason why you limited this change to sparse-index comes out.  :-)

> +       convert_to_sparse(r->index);
> +
> +       strbuf_addstr(&path, r->worktree);
> +       strbuf_complete(&path, '/');
> +       pathlen = path.len;
> +
> +       for (i = 0; i < r->index->cache_nr; i++) {
> +               struct cache_entry *ce = r->index->cache[i];
> +
> +               /*
> +                * Is this a sparse directory? If so, then definitely
> +                * include it. All contained content is outside of the
> +                * patterns.

"include"?  I would have used the word "remove", but it gets confusing
because the question is if we want to include it in our list of things
to remove or remove it from the working directory.

> +                */
> +               if (S_ISSPARSEDIR(ce->ce_mode) &&
> +                   repo_file_exists(r, ce->name)) {
> +                       strbuf_setlen(&path, pathlen);
> +                       strbuf_addstr(&path, ce->name);
> +
> +                       /*
> +                        * Removal is "best effort". If something blocks
> +                        * the deletion, then continue with a warning.
> +                        */
> +                       if (remove_dir_recursively(&path, 0))
> +                               warning(_("failed to remove directory '%s'"), path.buf);

Um, doesn't this delete untracked files that are not ignored as well
as the ignored files?  If so, was that intentional?  I'm fully on
board with removing the gitignore'd files, but I'm worried removing
other untracked files is dangerous.

My implementation of this concept (in an external tool) was more along
the lines of

  * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`
output and finding common fully-sparse directories
  * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES


> +               }
> +       }
> +
> +       strbuf_release(&path);
> +
> +       /*
> +        * This is temporary: the sparse-checkout builtin is not
> +        * integrated with the sparse-index yet, so we need to keep
> +        * it full during the process.
> +        */
> +       ensure_full_index(r->index);
> +}
> +
>  static int update_working_directory(struct pattern_list *pl)
>  {
>         enum update_sparsity_result result;
> @@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)
>         else
>                 rollback_lock_file(&lock_file);
>
> +       clean_tracked_sparse_directories(r);
> +
>         r->index->sparse_checkout_patterns = NULL;
>         return result;
>  }

(Adding a comment here just to separate the two blocks below.)

> @@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
>  {
>         int result;
>         int changed_config = 0;
> +       struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));
>         struct pattern_list *pl = xcalloc(1, sizeof(*pl));
>
> +       get_sparse_checkout_patterns(old_pl);
> +
>         switch (m) {
>         case ADD:
>                 if (core_sparse_checkout_cone)
> @@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
>                 set_config(MODE_NO_PATTERNS);
>
>         clear_pattern_list(pl);
> +       clear_pattern_list(old_pl);
>         free(pl);
> +       free(old_pl);
>         return result;
>  }

You create an old_pl, populate it with get_sparse_checkout_patterns(),
do nothing with it, then clear and free it?  I'm confused by the above
two blocks.

>
> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index 38fc8340f5c..43eb314c94a 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -642,4 +642,46 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
>         check_files repo/deep a deeper1
>  '
>
> +test_expect_success 'cone mode clears ignored subdirectories' '
> +       rm repo/.git/info/sparse-checkout &&
> +
> +       # NEEDSWORK: --sparse-index is required for now
> +       git -C repo sparse-checkout init --cone --sparse-index &&
> +       git -C repo sparse-checkout set deep/deeper1 &&
> +
> +       cat >repo/.gitignore <<-\EOF &&
> +       obj/
> +       *.o
> +       EOF
> +
> +       git -C repo add .gitignore &&
> +       git -C repo commit -m ".gitignore" &&
> +
> +       mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
> +       for file in folder1/obj/a obj/a folder1/file.o folder1.o \
> +                   deep/deeper2/obj/a deep/deeper2/file.o file.o
> +       do
> +               echo ignored >repo/$file || return 1
> +       done &&
> +
> +       git -C repo status --porcelain=v2 >out &&
> +       test_must_be_empty out &&
> +
> +       git -C repo sparse-checkout reapply &&
> +       test_path_is_missing repo/folder1 &&
> +       test_path_is_missing repo/deep/deeper2 &&
> +       test_path_is_dir repo/obj &&
> +       test_path_is_file repo/file.o &&
> +
> +       git -C repo status --porcelain=v2 >out &&
> +       test_must_be_empty out &&
> +
> +       git -C repo sparse-checkout set deep/deeper2 &&
> +       test_path_is_missing repo/deep/deeper1 &&
> +       test_path_is_dir repo/deep/deeper2 &&

What's the expectation for repo/obj?

> +
> +       git -C repo status --porcelain=v2 >out &&
> +       test_must_be_empty out
> +'
> +
>  test_done
> --

Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee]

On 7/30/2021 9:52 AM, Elijah Newren wrote:
> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
...
>> Leaving these ignored files in the sparse directories makes it
>> impossible to gain performance benefits in the sparse index. When we
>> track into these directories, we need to know if the files are ignored
>> or not, which might depend on the _tracked_ .gitignore file(s) within
>> the sparse directory. This depends on the indexed version of the file,
>> so the sparse directory must be expanded.
> 
> Is this the issue I highlighted at
> https://lore.kernel.org/git/CABPp-BHsiLTtv6AuycRrQ5K6q0=ZTJe0kd7uTUL+UT=nxj66zA@mail.gmail.com/,
> the paragraph "...I thought the point of add_patterns()..." or is
> there more or other things involved here?

This is exactly that point. I'm not sure why I didn't pick up on your
earlier concerns as something to do _immediately_, but for some reason
I thought I could delay it until later.

>> If users depend on ignored files within the sparse directories, then
>> they have created a bad shape in their repository. Regardless, such
>> shapes would create risk that changing the behavior for all cone mode
>> users might be too risky to take on at the moment. Since this data shape
>> makes it impossible to get performance benefits using the sparse index,
>> we limit the change to only be enabled when the sparse index is enabled.
>> Users can opt out of this behavior by disabline the sparse index.
> 
> s/disabline/disabling/, otherwise, I fully agree with this paragraph.

Will fix. Thanks.
 
>> Depending on user feedback or real-world use, we might want to consider
>> expanding the behavior change to all of cone mode. Since we are
>> currently restricting to the sparse index case, we can use the existence
>> of sparse directory entries in the index as indicators of which
>> directories should be removed.
> 
> Let's just expand it to all of cone mode.

I'm open to that. I'll leave this version open for feedback a bit longer
before doing so.

>> +       /*
>> +        * NEEDSWORK: For now, only use this behavior when index.sparse
>> +        * is enabled. We may want this behavior enabled whenever using
>> +        * cone mode patterns.
>> +        */
>> +       prepare_repo_settings(r);
>> +       if (!r->worktree || !r->settings.sparse_index)
>> +               return;
> 
> s/may/do/  :-)

Maybe!
 
>> +
>> +       /*
>> +        * Since we now depend on the sparse index to enable this
>> +        * behavior, use it to our advantage. This process is more
>> +        * complicated without it.
>> +        */
> 
> Ah, the real reason why you limited this change to sparse-index comes out.  :-)
> 
>> +       convert_to_sparse(r->index);
>> +
>> +       strbuf_addstr(&path, r->worktree);
>> +       strbuf_complete(&path, '/');
>> +       pathlen = path.len;
>> +
>> +       for (i = 0; i < r->index->cache_nr; i++) {
>> +               struct cache_entry *ce = r->index->cache[i];
>> +
>> +               /*
>> +                * Is this a sparse directory? If so, then definitely
>> +                * include it. All contained content is outside of the
>> +                * patterns.
> 
> "include"?  I would have used the word "remove", but it gets confusing
> because the question is if we want to include it in our list of things
> to remove or remove it from the working directory.

This comment is a bit stale, sorry. Earlier I was populating a list of
the directories, but in this version I'm just deleting them immediately.

>> +                */
>> +               if (S_ISSPARSEDIR(ce->ce_mode) &&
>> +                   repo_file_exists(r, ce->name)) {
>> +                       strbuf_setlen(&path, pathlen);
>> +                       strbuf_addstr(&path, ce->name);
>> +
>> +                       /*
>> +                        * Removal is "best effort". If something blocks
>> +                        * the deletion, then continue with a warning.
>> +                        */
>> +                       if (remove_dir_recursively(&path, 0))
>> +                               warning(_("failed to remove directory '%s'"), path.buf);
> 
> Um, doesn't this delete untracked files that are not ignored as well
> as the ignored files?  If so, was that intentional?  I'm fully on
> board with removing the gitignore'd files, but I'm worried removing
> other untracked files is dangerous.

I believe that 'git sparse-checkout (set|add|reapply)' will fail before
reaching this method if there are untracked files that could potentially
be removed. I will double-check to ensure this is the case. It is
definitely my intention to protect any untracked, non-ignored files in
these directories by failing the sparse-checkout modification.

> My implementation of this concept (in an external tool) was more along
> the lines of
> 
>   * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`
> output and finding common fully-sparse directories
>   * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES

I initially was running 'git clean -dfx -- <dir> ...' but that also
requires parsing and expanding the index (or being very careful with
the sparse index).

> 
>> +               }
>> +       }
>> +
>> +       strbuf_release(&path);
>> +
>> +       /*
>> +        * This is temporary: the sparse-checkout builtin is not
>> +        * integrated with the sparse-index yet, so we need to keep
>> +        * it full during the process.
>> +        */
>> +       ensure_full_index(r->index);
>> +}
>> +
>>  static int update_working_directory(struct pattern_list *pl)
>>  {
>>         enum update_sparsity_result result;
>> @@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)
>>         else
>>                 rollback_lock_file(&lock_file);
>>
>> +       clean_tracked_sparse_directories(r);
>> +
>>         r->index->sparse_checkout_patterns = NULL;
>>         return result;
>>  }
> 
> (Adding a comment here just to separate the two blocks below.)
> 
>> @@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
>>  {
>>         int result;
>>         int changed_config = 0;
>> +       struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));
>>         struct pattern_list *pl = xcalloc(1, sizeof(*pl));
>>
>> +       get_sparse_checkout_patterns(old_pl);
>> +
>>         switch (m) {
>>         case ADD:
>>                 if (core_sparse_checkout_cone)
>> @@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
>>                 set_config(MODE_NO_PATTERNS);
>>
>>         clear_pattern_list(pl);
>> +       clear_pattern_list(old_pl);
>>         free(pl);
>> +       free(old_pl);
>>         return result;
>>  }
> 
> You create an old_pl, populate it with get_sparse_checkout_patterns(),
> do nothing with it, then clear and free it?  I'm confused by the above
> two blocks.

Sorry that this is also stale. I should read my own patches more carefully.

I was originally going to focus only on the directories that were "leaving"
the sparse-checkout definition, but that did not catch the 'reapply' case
or cases where the user created the directories by other means.

Will delete these bits.
>> +       git -C repo sparse-checkout reapply &&
>> +       test_path_is_missing repo/folder1 &&
>> +       test_path_is_missing repo/deep/deeper2 &&
>> +       test_path_is_dir repo/obj &&
>> +       test_path_is_file repo/file.o &&
>> +
>> +       git -C repo status --porcelain=v2 >out &&
>> +       test_must_be_empty out &&
>> +
>> +       git -C repo sparse-checkout set deep/deeper2 &&
>> +       test_path_is_missing repo/deep/deeper1 &&
>> +       test_path_is_dir repo/deep/deeper2 &&
> 
> What's the expectation for repo/obj?

I will add

	test_path_is_dir repo/obj

because this ignored directory should not be removed. This is
simulating the build artifacts that might appear in ignored
directories whose parents are in the sparse-checkout definition.

Thanks,
-Stolee

Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Elijah Newren]

On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 7/30/2021 9:52 AM, Elijah Newren wrote:
> > On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
> > <gitgitgadget@gmail.com> wrote:
...
> >> +                */
> >> +               if (S_ISSPARSEDIR(ce->ce_mode) &&
> >> +                   repo_file_exists(r, ce->name)) {
> >> +                       strbuf_setlen(&path, pathlen);
> >> +                       strbuf_addstr(&path, ce->name);
> >> +
> >> +                       /*
> >> +                        * Removal is "best effort". If something blocks
> >> +                        * the deletion, then continue with a warning.
> >> +                        */
> >> +                       if (remove_dir_recursively(&path, 0))
> >> +                               warning(_("failed to remove directory '%s'"), path.buf);
> >
> > Um, doesn't this delete untracked files that are not ignored as well
> > as the ignored files?  If so, was that intentional?  I'm fully on
> > board with removing the gitignore'd files, but I'm worried removing
> > other untracked files is dangerous.
>
> I believe that 'git sparse-checkout (set|add|reapply)' will fail before
> reaching this method if there are untracked files that could potentially
> be removed. I will double-check to ensure this is the case. It is
> definitely my intention to protect any untracked, non-ignored files in
> these directories by failing the sparse-checkout modification.
>
> > My implementation of this concept (in an external tool) was more along
> > the lines of
> >
> >   * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`
> > output and finding common fully-sparse directories
> >   * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES
>
> I initially was running 'git clean -dfx -- <dir> ...' but that also
> requires parsing and expanding the index (or being very careful with
> the sparse index).

`git clean -dfx -- <dir> ...` could also be very dangerous because
it'd delete untracked non-ignored files.  You want -X rather than -x.
One of those cases where capitalization is critical.

Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee]

On 8/2/2021 12:17 PM, Elijah Newren wrote:
> On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee <stolee@gmail.com> wrote:
>>
>> On 7/30/2021 9:52 AM, Elijah Newren wrote:
>>> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
>>> <gitgitgadget@gmail.com> wrote:
> ...
>>>> +                */
>>>> +               if (S_ISSPARSEDIR(ce->ce_mode) &&
>>>> +                   repo_file_exists(r, ce->name)) {
>>>> +                       strbuf_setlen(&path, pathlen);
>>>> +                       strbuf_addstr(&path, ce->name);
>>>> +
>>>> +                       /*
>>>> +                        * Removal is "best effort". If something blocks
>>>> +                        * the deletion, then continue with a warning.
>>>> +                        */
>>>> +                       if (remove_dir_recursively(&path, 0))
>>>> +                               warning(_("failed to remove directory '%s'"), path.buf);
>>>
>>> Um, doesn't this delete untracked files that are not ignored as well
>>> as the ignored files?  If so, was that intentional?  I'm fully on
>>> board with removing the gitignore'd files, but I'm worried removing
>>> other untracked files is dangerous.
>>
>> I believe that 'git sparse-checkout (set|add|reapply)' will fail before
>> reaching this method if there are untracked files that could potentially
>> be removed. I will double-check to ensure this is the case. It is
>> definitely my intention to protect any untracked, non-ignored files in
>> these directories by failing the sparse-checkout modification.

This is _not_ true, and I can document it with a test.

Having untracked files outside of the sparse cone is just as bad as
ignored files, so I want to ensure that these get cleaned up, too.

The correct thing would be to prevent the 'git sparse-checkout
(set|add|reapply)' command from making any changes to the sparse-checkout
cone or the worktree if there are untracked files that would be deleted.
(Right? Or is there another solution that I'm missing here?)

>>> My implementation of this concept (in an external tool) was more along
>>> the lines of
>>>
>>>   * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`
>>> output and finding common fully-sparse directories
>>>   * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES
>>
>> I initially was running 'git clean -dfx -- <dir> ...' but that also
>> requires parsing and expanding the index (or being very careful with
>> the sparse index).
> 
> `git clean -dfx -- <dir> ...` could also be very dangerous because
> it'd delete untracked non-ignored files.  You want -X rather than -x.
> One of those cases where capitalization is critical.

Good point. I'd like to avoid using `git clean` as a subcommand, if
possible, that way we have one fewer thing to do before integrating
the `git sparse-checkout` builtin with the sparse index.

Thanks,
-Stolee

Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs

[Elijah Newren]

On Wed, Aug 4, 2021 at 7:55 PM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/2/2021 12:17 PM, Elijah Newren wrote:
> > On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee <stolee@gmail.com> wrote:
> >>
> >> On 7/30/2021 9:52 AM, Elijah Newren wrote:
> >>> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
> >>> <gitgitgadget@gmail.com> wrote:
> > ...
> >>>> +                */
> >>>> +               if (S_ISSPARSEDIR(ce->ce_mode) &&
> >>>> +                   repo_file_exists(r, ce->name)) {
> >>>> +                       strbuf_setlen(&path, pathlen);
> >>>> +                       strbuf_addstr(&path, ce->name);
> >>>> +
> >>>> +                       /*
> >>>> +                        * Removal is "best effort". If something blocks
> >>>> +                        * the deletion, then continue with a warning.
> >>>> +                        */
> >>>> +                       if (remove_dir_recursively(&path, 0))
> >>>> +                               warning(_("failed to remove directory '%s'"), path.buf);
> >>>
> >>> Um, doesn't this delete untracked files that are not ignored as well
> >>> as the ignored files?  If so, was that intentional?  I'm fully on
> >>> board with removing the gitignore'd files, but I'm worried removing
> >>> other untracked files is dangerous.
> >>
> >> I believe that 'git sparse-checkout (set|add|reapply)' will fail before
> >> reaching this method if there are untracked files that could potentially
> >> be removed. I will double-check to ensure this is the case. It is
> >> definitely my intention to protect any untracked, non-ignored files in
> >> these directories by failing the sparse-checkout modification.
>
> This is _not_ true, and I can document it with a test.
>
> Having untracked files outside of the sparse cone is just as bad as
> ignored files, so I want to ensure that these get cleaned up, too.
>
> The correct thing would be to prevent the 'git sparse-checkout
> (set|add|reapply)' command from making any changes to the sparse-checkout
> cone or the worktree if there are untracked files that would be deleted.
> (Right? Or is there another solution that I'm missing here?)

We could sparsify as much as possible and print warnings, much like we
do with tracked files that are modified but not staged.  In fact, it
might feel inconsistent if we sparsify as much as possible for one
type of file, and abort if we cannot completely sparsify for a
different type of file.  We could consider changing how we treat
tracked files that are modified but not staged and have them abort the
sparse-checkout commands as well, but I worry that might cause
problems during conflict resolution in the middle of
merges/rebases/cherry-picks/reverts.  I don't want users caught where
they need to update their sparsity paths to gain new files/directories
that will help them resolve some conflicts, but be unable to update
their sparsity paths because they have conflicts.

That said, the basic idea of aborting sparse-checkout in cone mode
when there are untracked unignored files in the way of removing
directories sounds reasonable, if there's some clever way to avoid or
ameliorate the inconsistency issues mentioned above.  Implementing it
might require walking all untracked (and tracked?) files twice,
though, because if there are untracked unignored files in the way, we
probably don't want to abort after first deleting lots of ignored
files.  (And there's a small race window in the double walk...)
However, I don't expect people to run sparse-checkout commands all
that often, so the double walk is probably a perfectly reasonable
performance cost.  I just wanted to note it.

> >>> My implementation of this concept (in an external tool) was more along
> >>> the lines of
> >>>
> >>>   * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`
> >>> output and finding common fully-sparse directories
> >>>   * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES
> >>
> >> I initially was running 'git clean -dfx -- <dir> ...' but that also
> >> requires parsing and expanding the index (or being very careful with
> >> the sparse index).
> >
> > `git clean -dfx -- <dir> ...` could also be very dangerous because
> > it'd delete untracked non-ignored files.  You want -X rather than -x.
> > One of those cases where capitalization is critical.
>
> Good point. I'd like to avoid using `git clean` as a subcommand, if
> possible, that way we have one fewer thing to do before integrating
> the `git sparse-checkout` builtin with the sparse index.

Oh, I didn't want to invoke a subcommand, I was just pointing out
where similar code might be found in case we wanted to call the same
functions from elsewhere (or maybe even turn some of it into library
functions we could call).  But that might be a moot point if we end up
making sparse-checkout fail if there are untracked unignored files
hanging around in the relevant directories.

Re: [PATCH 0/2] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> My commits are technically based on ds/add-with-sparse-index, but could
> apply earlier, I believe.
>
> We launched an experimental release [1] of the sparse-index feature to our
> internal users. We immediately discovered a problem due to the isolated way
> in which we tested the sparse index: we were never building the project and
> changing our sparse-checkout definition.
>
> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp
>
> Users who ran a build in one project and then moved to another still had
> build artifacts in their worktree that lived inside the old directories.
> Since the files are marked by the .gitignore patterns, these files were not
> removed by the 'git sparse-checkout set' command. However, they make the
> sparse-index unusable because every 'git status' command needs to expand the
> sparse-directory entries in order to see if the files are tracked or not.
> This made the first experimental release actually slower for all users
> because of this cost.

As highlighted at
https://lore.kernel.org/git/CABPp-BHwRHhYE3BPxCG+fRFWf2dCZ60rnO=ykztquKoOp_x_QQ@mail.gmail.com/
?

"""So, although 'git status'
shouldn't have to check the tracked files anymore, it is still going
to have to look at each of the *untracked* files and compare to
.gitignore files in order to correctly classify each file as ignored
or just plain untracked."""

As I noted there,

"""about a month ago my
colleagues made the tool just auto-invoke [--remove-old-ignores] from other
sparsify invocations.  To my knowledge, there have been no complaints
about that being automatically turned on; but there were
complaints/confusion before about the directories being left around.
(Of course, non-ignored files are still left around by that option.)"""

I think your changes here shouldn't be limited to a sparse-index but
should apply to cone-mode sparse checkouts generally.

> The solution we shipped to these customers was to change the way our fork
> handles these ignored files. Specifically, instead of Git completely
> ignoring the files, we changed Git to understand that with cone-mode
> sparse-checkout patterns, the users is asking for entire directories to be
> removed from the worktree. The link [1] included earlier has this change.

Could you clarify what you are removing?
  1) Tracked but now SKIP_WOKRTREE files
  2) .gitignore'd files
  3) other untracked files

(1) has always been removed by sparse-checkout.  I think you're adding
(2) but it's not clear if you are also removing (3).

> I believe that this is a reasonable expectation, though I recognize that it
> might look like breaking the expectations of how .gitignore files work.
>
> To limit the scope of this change, the focus is limited to behave this way
> only when the sparse-index is enabled, because otherwise the sparse index
> does not behave as intended. With that restriction in mind, the
> implementation can use the sparse directory entries to assist.
>
> I imagine that when the sparse index is mature enough, we will enable it by
> default when cone mode sparse-checkout is enabled, in which case this
> behavior would expand to all users with cone mode sparse-checkout.

I'm all in favor of just doing it for all cone mode users, at least
for .gitignore'd files.

> I'm interested in the community's thoughts about this change, as it seems
> like one that we should make carefully and intentionally.

Yeah, I approached it pretty cautiously, just adding a new option in
our tool (`sparsify --remove-old-ignores`) because I wasn't sure if it
should be done by default but wanted to provide an easy way for users
to access it.  We got numerous complaints and after some months, added
a config option to run it automatically.  We still got complaints and
eventually my colleagues just made it automatically run and provided
no way to turn it off.  No user complaints since.

[PATCH v2 0/8] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee via GitGitGadget]

UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
and v2.33.0-rc1 because of some changes to dir.h that became important!

We launched an experimental release [1] of the sparse-index feature to our
internal users. We immediately discovered a problem due to the isolated way
in which we tested the sparse index: we were never building the project and
changing our sparse-checkout definition.

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Users who ran a build in one project and then moved to another still had
build artifacts in their worktree that lived inside the old directories.
Since the files are marked by the .gitignore patterns, these files were not
removed by the 'git sparse-checkout set' command. However, they make the
sparse-index unusable because every 'git status' command needs to expand the
sparse-directory entries in order to see if the files are tracked or not.
This made the first experimental release actually slower for all users
because of this cost.

The solution we shipped to these customers was to change the way our fork
handles these ignored files. Specifically, instead of Git completely
ignoring the files, we changed Git to understand that with cone-mode
sparse-checkout patterns, the users is asking for entire directories to be
removed from the worktree. The link [1] included earlier has this change.

I believe that this is a reasonable expectation, though I recognize that it
might look like breaking the expectations of how .gitignore files work.

Since feedback demonstrated that this is a desired behavior, v2 includes
this behavior for all "cone mode" repositories.

I'm interested in the community's thoughts about this change, as it seems
like one that we should make carefully and intentionally.

While the rewrite of the t7519 test seems unrelated, it is required to avoid
a test failure with this change that deletes files outside of the cone. By
moving the test into repositories not at $TRASH_DIRECTORY, we gain more
control over the repository structure.


Updates in V2
=============

 * This version correctly leaves untracked files alone. If untracked files
   are found, then the directory is left as-is, in case those ignored files
   are important to the user's work resolving those untracked files.

 * This behavior is now enabled by core.sparseCheckoutCone=true.

 * To use a sparse index as an in-memory data structure even when
   index.sparse is disabled, a new patch is included to modify the prototype
   of convert_to_sparse() to include a flags parameter.

 * A few cleanup patches that I was collecting based on feedback from the
   experimental release and intending for my next series were necessary for
   this implementation.

 * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
   implementation that used run_subcommand().

Thanks, -Stolee

Derrick Stolee (8):
  t7519: rewrite sparse index test
  sparse-index: silently return when not using cone-mode patterns
  sparse-index: silently return when cache tree fails
  unpack-trees: fix nested sparse-dir search
  sparse-checkout: create helper methods
  attr: be careful about sparse directories
  sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
  sparse-checkout: clear tracked sparse dirs

 Documentation/git-sparse-checkout.txt |  8 +++
 attr.c                                | 14 ++++
 builtin/add.c                         |  8 +--
 builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
 dir.c                                 | 29 ++++++++
 dir.h                                 |  6 ++
 read-cache.c                          |  4 +-
 sparse-index.c                        | 70 +++++++++++---------
 sparse-index.h                        |  3 +-
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 t/t7519-status-fsmonitor.sh           | 38 ++++++-----
 unpack-trees.c                        | 18 +++--
 12 files changed, 290 insertions(+), 62 deletions(-)


base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1009

Range-diff vs v1:

 1:  29f1633f6d5 = 1:  e66106f7a99 t7519: rewrite sparse index test
 -:  ----------- > 2:  fb3ff9108bf sparse-index: silently return when not using cone-mode patterns
 -:  ----------- > 3:  37198535268 sparse-index: silently return when cache tree fails
 -:  ----------- > 4:  10bcadb284e unpack-trees: fix nested sparse-dir search
 -:  ----------- > 5:  e9ed5cd2830 sparse-checkout: create helper methods
 -:  ----------- > 6:  5680df62e1c attr: be careful about sparse directories
 -:  ----------- > 7:  1dd73b36eb4 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
 2:  9212bbf4e3c ! 8:  f74015a2be9 sparse-checkout: clear tracked sparse dirs
     @@ Commit message
          newer timestamps than the build artifacts, so the artifacts will need to
          be regenerated anyway.
      
     -    If users depend on ignored files within the sparse directories, then
     -    they have created a bad shape in their repository. Regardless, such
     -    shapes would create risk that changing the behavior for all cone mode
     -    users might be too risky to take on at the moment. Since this data shape
     -    makes it impossible to get performance benefits using the sparse index,
     -    we limit the change to only be enabled when the sparse index is enabled.
     -    Users can opt out of this behavior by disabline the sparse index.
     -
     -    Depending on user feedback or real-world use, we might want to consider
     -    expanding the behavior change to all of cone mode. Since we are
     -    currently restricting to the sparse index case, we can use the existence
     -    of sparse directory entries in the index as indicators of which
     -    directories should be removed.
     +    Use the sparse-index as a data structure in order to find the sparse
     +    directories that can be safely deleted. Re-expand the index to a full
     +    one if it was full before.
      
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
     @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for
       'git sparse-checkout set' command to reflect the expected cone in the working
       directory.
       
     -+When the sparse index is enabled through the `index.sparse` config option,
     -+the cone mode sparse-checkout patterns will also remove ignored files that
     ++The cone mode sparse-checkout patterns will also remove ignored files that
      +are not within the sparse-checkout definition. This is important behavior
      +to preserve the performance of the sparse index, but also matches that
     -+cone mode patterns care about directories, not files.
     ++cone mode patterns care about directories, not files. If there exist files
     ++that are untracked and not ignored, then Git will not delete files within
     ++that directory other than the tracked files that are now out of scope.
     ++These files should be removed manually to ensure Git can behave optimally.
      +
       
       SUBMODULES
       ----------
      
       ## builtin/sparse-checkout.c ##
     -@@
     - #include "wt-status.h"
     - #include "quote.h"
     - #include "sparse-index.h"
     -+#include "run-command.h"
     - 
     - static const char *empty_base = "";
     - 
      @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char **argv)
       	return 0;
       }
       
      +static void clean_tracked_sparse_directories(struct repository *r)
      +{
     -+	int i;
     ++	int i, was_full = 0;
      +	struct strbuf path = STRBUF_INIT;
      +	size_t pathlen;
     ++	struct string_list_item *item;
     ++	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
      +
      +	/*
      +	 * If we are not using cone mode patterns, then we cannot
      +	 * delete directories outside of the sparse cone.
      +	 */
     -+	if (!r || !r->index || !r->index->sparse_checkout_patterns ||
     -+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
     ++	if (!r || !r->index || !r->worktree)
      +		return;
     -+	/*
     -+	 * NEEDSWORK: For now, only use this behavior when index.sparse
     -+	 * is enabled. We may want this behavior enabled whenever using
     -+	 * cone mode patterns.
     -+	 */
     -+	prepare_repo_settings(r);
     -+	if (!r->worktree || !r->settings.sparse_index)
     ++	init_sparse_checkout_patterns(r->index);
     ++	if (!r->index->sparse_checkout_patterns ||
     ++	    !r->index->sparse_checkout_patterns->use_cone_patterns)
      +		return;
      +
      +	/*
     -+	 * Since we now depend on the sparse index to enable this
     -+	 * behavior, use it to our advantage. This process is more
     -+	 * complicated without it.
     ++	 * Use the sparse index as a data structure to assist finding
     ++	 * directories that are safe to delete. This conversion to a
     ++	 * sparse index will not delete directories that contain
     ++	 * conflicted entries or submodules.
      +	 */
     -+	convert_to_sparse(r->index);
     ++	if (!r->index->sparse_index) {
     ++		/*
     ++		 * If something, such as a merge conflict or other concern,
     ++		 * prevents us from converting to a sparse index, then do
     ++		 * not try deleting files.
     ++		 */
     ++		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
     ++			return;
     ++		was_full = 1;
     ++	}
      +
      +	strbuf_addstr(&path, r->worktree);
      +	strbuf_complete(&path, '/');
      +	pathlen = path.len;
      +
     ++	/*
     ++	 * Collect directories that have gone out of scope but also
     ++	 * exist on disk, so there is some work to be done. We need to
     ++	 * store the entries in a list before exploring, since that might
     ++	 * expand the sparse-index again.
     ++	 */
      +	for (i = 0; i < r->index->cache_nr; i++) {
      +		struct cache_entry *ce = r->index->cache[i];
      +
     -+		/*
     -+		 * Is this a sparse directory? If so, then definitely
     -+		 * include it. All contained content is outside of the
     -+		 * patterns.
     -+		 */
      +		if (S_ISSPARSEDIR(ce->ce_mode) &&
     -+		    repo_file_exists(r, ce->name)) {
     -+			strbuf_setlen(&path, pathlen);
     -+			strbuf_addstr(&path, ce->name);
     ++		    repo_file_exists(r, ce->name))
     ++			string_list_append(&sparse_dirs, ce->name);
     ++	}
     ++
     ++	for_each_string_list_item(item, &sparse_dirs) {
     ++		struct dir_struct dir = DIR_INIT;
     ++		struct pathspec p = { 0 };
     ++		struct strvec s = STRVEC_INIT;
     ++
     ++		strbuf_setlen(&path, pathlen);
     ++		strbuf_addstr(&path, item->string);
     ++
     ++		dir.flags |= DIR_SHOW_IGNORED_TOO;
     ++
     ++		setup_standard_excludes(&dir);
     ++		strvec_push(&s, path.buf);
      +
     ++		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
     ++		fill_directory(&dir, r->index, &p);
     ++
     ++		if (dir.nr) {
     ++			warning(_("directory '%s' contains untracked files,"
     ++				  " but is not in the sparse-checkout cone"),
     ++				item->string);
     ++		} else if (remove_dir_recursively(&path, 0)) {
      +			/*
      +			 * Removal is "best effort". If something blocks
      +			 * the deletion, then continue with a warning.
      +			 */
     -+			if (remove_dir_recursively(&path, 0))
     -+				warning(_("failed to remove directory '%s'"), path.buf);
     ++			warning(_("failed to remove directory '%s'"),
     ++				item->string);
      +		}
     ++
     ++		dir_clear(&dir);
      +	}
      +
     ++	string_list_clear(&sparse_dirs, 0);
      +	strbuf_release(&path);
      +
     -+	/*
     -+	 * This is temporary: the sparse-checkout builtin is not
     -+	 * integrated with the sparse-index yet, so we need to keep
     -+	 * it full during the process.
     -+	 */
     -+	ensure_full_index(r->index);
     ++	if (was_full)
     ++		ensure_full_index(r->index);
      +}
      +
       static int update_working_directory(struct pattern_list *pl)
     @@ builtin/sparse-checkout.c: static int update_working_directory(struct pattern_li
       	r->index->sparse_checkout_patterns = NULL;
       	return result;
       }
     -@@ builtin/sparse-checkout.c: static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
     - {
     - 	int result;
     - 	int changed_config = 0;
     -+	struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));
     - 	struct pattern_list *pl = xcalloc(1, sizeof(*pl));
     - 
     -+	get_sparse_checkout_patterns(old_pl);
     -+
     - 	switch (m) {
     - 	case ADD:
     - 		if (core_sparse_checkout_cone)
     -@@ builtin/sparse-checkout.c: static int modify_pattern_list(int argc, const char **argv, enum modify_type m)
     - 		set_config(MODE_NO_PATTERNS);
     - 
     - 	clear_pattern_list(pl);
     -+	clear_pattern_list(old_pl);
     - 	free(pl);
     -+	free(old_pl);
     - 	return result;
     - }
     - 
      
       ## t/t1091-sparse-checkout-builtin.sh ##
      @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
     @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replace
      +test_expect_success 'cone mode clears ignored subdirectories' '
      +	rm repo/.git/info/sparse-checkout &&
      +
     -+	# NEEDSWORK: --sparse-index is required for now
     -+	git -C repo sparse-checkout init --cone --sparse-index &&
     ++	git -C repo sparse-checkout init --cone &&
      +	git -C repo sparse-checkout set deep/deeper1 &&
      +
      +	cat >repo/.gitignore <<-\EOF &&
     @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replace
      +	git -C repo sparse-checkout set deep/deeper2 &&
      +	test_path_is_missing repo/deep/deeper1 &&
      +	test_path_is_dir repo/deep/deeper2 &&
     ++	test_path_is_dir repo/obj &&
     ++	test_path_is_file repo/file.o &&
     ++
     ++	>repo/deep/deeper2/ignored.o &&
     ++	>repo/deep/deeper2/untracked &&
     ++
     ++	# When an untracked file is in the way, all untracked files
     ++	# (even ignored files) are preserved.
     ++	git -C repo sparse-checkout set folder1 2>err &&
     ++	grep "contains untracked files" err &&
     ++	test_path_is_file repo/deep/deeper2/ignored.o &&
     ++	test_path_is_file repo/deep/deeper2/untracked &&
     ++
     ++	# The rest of the cone matches expectation
     ++	test_path_is_missing repo/deep/deeper1 &&
     ++	test_path_is_dir repo/obj &&
     ++	test_path_is_file repo/file.o &&
      +
      +	git -C repo status --porcelain=v2 >out &&
     -+	test_must_be_empty out
     ++	echo "? deep/deeper2/untracked" >expect &&
     ++	test_cmp expect out
      +'
      +
       test_done

-- 
gitgitgadget

[PATCH v2 1/8] t7519: rewrite sparse index test

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The sparse index is tested with the FS Monitor hook and extension since
f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
was very fragile because it shared an index across sparse and non-sparse
behavior. Since that expansion and contraction could cause the index to
lose its FS Monitor bitmap and token, behavior is fragile to changes in
'git sparse-checkout set'.

Rewrite the test to use two clones of the original repo: full and
sparse. This allows us to also keep the test files (actual, expect,
trace2.txt) out of the repos we are testing with 'git status'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index deea88d4431..6f2cf306f66 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
 # If "!" is supplied, then we verify that we do not call ensure_full_index
 # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
 check_sparse_index_behavior () {
-	git status --porcelain=v2 >expect &&
-	git sparse-checkout init --cone --sparse-index &&
-	git sparse-checkout set dir1 dir2 &&
+	git -C full status --porcelain=v2 >expect &&
 	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-		git status --porcelain=v2 >actual &&
+		git -C sparse status --porcelain=v2 >actual &&
 	test_region $1 index ensure_full_index trace2.txt &&
 	test_region fsm_hook query trace2.txt &&
 	test_cmp expect actual &&
-	rm trace2.txt &&
-	git sparse-checkout disable
+	rm trace2.txt
 }
 
 test_expect_success 'status succeeds with sparse index' '
-	git reset --hard &&
+	git clone . full &&
+	git clone . sparse &&
+	git -C sparse sparse-checkout init --cone --sparse-index &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
-	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
-	check_sparse_index_behavior ! &&
-
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 	EOF
-	git config core.fsmonitor .git/hooks/fsmonitor-test &&
+	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
+	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
 	check_sparse_index_behavior ! &&
 
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1/modified\0"
 	EOF
 	check_sparse_index_behavior ! &&
 
-	cp -r dir1 dir1a &&
-	git add dir1a &&
-	git commit -m "add dir1a" &&
+	git -C sparse sparse-checkout add dir1a &&
+
+	for repo in full sparse
+	do
+		cp -r $repo/dir1 $repo/dir1a &&
+		git -C $repo add dir1a &&
+		git -C $repo commit -m "add dir1a" || return 1
+	done &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
 	# This one modifies outside the sparse-checkout definition
 	# and hence we expect to expand the sparse-index.
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1a/modified\0"
 	EOF
-- 
gitgitgadget

[PATCH v2 2/8] sparse-index: silently return when not using cone-mode patterns

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

While the sparse-index is only enabled when core.sparseCheckoutCone is
also enabled, it is possible for the user to modify the sparse-checkout
file manually in a way that does not match cone-mode patterns. In this
case, we should refuse to convert an index into a sparse index, since
the sparse_checkout_patterns will not be initialized with recursive and
parent path hashsets.

Also silently return if there are no cache entries, which is a simple
case: there are no paths to make sparse!

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index c6b4feec413..bc5900eae35 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -130,7 +130,8 @@ static int index_has_unmerged_entries(struct index_state *istate)
 int convert_to_sparse(struct index_state *istate)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index ||
+
+	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
@@ -158,10 +159,16 @@ int convert_to_sparse(struct index_state *istate)
 			return 0;
 	}
 
-	if (!istate->sparse_checkout_patterns->use_cone_patterns) {
-		warning(_("attempting to use sparse-index without cone mode"));
-		return -1;
-	}
+	/*
+	 * We need cone-mode patterns to use sparse-index. If a user edits
+	 * their sparse-checkout file manually, then we can detect during
+	 * parsing that they are not actually using cone-mode patterns and
+	 * hence we need to abort this conversion _without error_. Warnings
+	 * already exist in the pattern parsing to inform the user of their
+	 * bad patterns.
+	 */
+	if (!istate->sparse_checkout_patterns->use_cone_patterns)
+		return 0;
 
 	/*
 	 * NEEDSWORK: If we have unmerged entries, then stay full.
-- 
gitgitgadget

[PATCH v2 3/8] sparse-index: silently return when cache tree fails

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

If cache_tree_update() returns a non-zero value, then it could not
create the cache tree. This is likely due to a path having a merge
conflict. Since we are already returning early, let's return silently to
avoid making it seem like we failed to write the index at all.

If we remove our dependence on the cache tree within
convert_to_sparse(), then we could still recover from this scenario and
have a sparse index.

When constructing the cache-tree extension in convert_to_sparse(), it is
possible that we construct a tree object that is new to the object
database. Without the WRITE_TREE_MISSING_OK flag, this results in an
error that halts our conversion to a sparse index. Add this flag to
remove this limitation.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index bc5900eae35..b6e90417556 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)
 
 	/* Clear and recompute the cache-tree */
 	cache_tree_free(&istate->cache_tree);
-	if (cache_tree_update(istate, 0)) {
-		warning(_("unable to update cache-tree, staying full"));
-		return -1;
-	}
+	/*
+	 * Silently return if there is a problem with the cache tree update,
+	 * which might just be due to a conflict state in some entry.
+	 *
+	 * This might create new tree objects, so be sure to use
+	 * WRITE_TREE_MISSING_OK.
+	 */
+	if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
+		return 0;
 
 	remove_fsmonitor(istate);
 
-- 
gitgitgadget

Re: [PATCH v2 3/8] sparse-index: silently return when cache tree fails

[Elijah Newren]

On Tue, Aug 10, 2021 at 12:50 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> If cache_tree_update() returns a non-zero value, then it could not
> create the cache tree. This is likely due to a path having a merge
> conflict. Since we are already returning early, let's return silently to
> avoid making it seem like we failed to write the index at all.
>
> If we remove our dependence on the cache tree within
> convert_to_sparse(), then we could still recover from this scenario and
> have a sparse index.
>
> When constructing the cache-tree extension in convert_to_sparse(), it is
> possible that we construct a tree object that is new to the object
> database. Without the WRITE_TREE_MISSING_OK flag, this results in an
> error that halts our conversion to a sparse index. Add this flag to
> remove this limitation.

Would this only happen when the user has staged but uncommitted
changes outside the sparsity paths, and tries to sparsify while in
that state?  (Notably, this is a much different condition than the
above mentioned merge conflict case that would case
cache_tree_update() to just fail.)

I think it might be nicer to split this commit in two, just to make it
easier to understand for future readers.  This seems like two logical
changes and trying to understand them and why would I think be easier
if the two were split.  I'd be tempted to put the
WRITE_TREE_MISSING_OK first.

> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  sparse-index.c | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/sparse-index.c b/sparse-index.c
> index bc5900eae35..b6e90417556 100644
> --- a/sparse-index.c
> +++ b/sparse-index.c
> @@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)
>
>         /* Clear and recompute the cache-tree */
>         cache_tree_free(&istate->cache_tree);
> -       if (cache_tree_update(istate, 0)) {
> -               warning(_("unable to update cache-tree, staying full"));
> -               return -1;
> -       }
> +       /*
> +        * Silently return if there is a problem with the cache tree update,
> +        * which might just be due to a conflict state in some entry.
> +        *
> +        * This might create new tree objects, so be sure to use
> +        * WRITE_TREE_MISSING_OK.
> +        */
> +       if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
> +               return 0;
>
>         remove_fsmonitor(istate);
>
> --
> gitgitgadget

These feel like cases where it would be nice to have a testcase
demonstrating the change in behavior.  Perhaps just splitting the
commit would be enough, but it took a bit of time to try to understand
what would change and why, despite the simple changes.

Re: [PATCH v2 3/8] sparse-index: silently return when cache tree fails

[Derrick Stolee]

On 8/19/2021 2:24 PM, Elijah Newren wrote:
> On Tue, Aug 10, 2021 at 12:50 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> If cache_tree_update() returns a non-zero value, then it could not
>> create the cache tree. This is likely due to a path having a merge
>> conflict. Since we are already returning early, let's return silently to
>> avoid making it seem like we failed to write the index at all.
>>
>> If we remove our dependence on the cache tree within
>> convert_to_sparse(), then we could still recover from this scenario and
>> have a sparse index.
>>
>> When constructing the cache-tree extension in convert_to_sparse(), it is
>> possible that we construct a tree object that is new to the object
>> database. Without the WRITE_TREE_MISSING_OK flag, this results in an
>> error that halts our conversion to a sparse index. Add this flag to
>> remove this limitation.
> 
> Would this only happen when the user has staged but uncommitted
> changes outside the sparsity paths, and tries to sparsify while in
> that state?  (Notably, this is a much different condition than the
> above mentioned merge conflict case that would case
> cache_tree_update() to just fail.)
> 
> I think it might be nicer to split this commit in two, just to make it
> easier to understand for future readers.  This seems like two logical
> changes and trying to understand them and why would I think be easier
> if the two were split.  I'd be tempted to put the
> WRITE_TREE_MISSING_OK first.

Ironically, I _had_ this as two commits because I discovered the
problems independently. It wasn't until I was organizing things that
I realized I was editing the same 'if' twice and thought it better
to merge patches. But I also don't feel strongly about that, so I
can split them.

>> +       /*
>> +        * Silently return if there is a problem with the cache tree update,
>> +        * which might just be due to a conflict state in some entry.
>> +        *
>> +        * This might create new tree objects, so be sure to use
>> +        * WRITE_TREE_MISSING_OK.
>> +        */
>> +       if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
>> +               return 0;
...
> These feel like cases where it would be nice to have a testcase
> demonstrating the change in behavior.  Perhaps just splitting the
> commit would be enough, but it took a bit of time to try to understand
> what would change and why, despite the simple changes.

I found these were required in the Scalar functional tests, so I bet
that if I remove this change I can create a test case from that. Thanks.

-Stolee

[PATCH v2 4/8] unpack-trees: fix nested sparse-dir search

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The iterated search in find_cache_entry() was recently modified to
include a loop that searches backwards for a sparse directory entry that
matches the given traverse_info and name_entry. However, the string
comparison failed to actually concatenate those two strings, so this
failed to find a sparse directory when it was not a top-level directory.

This caused some errors in rare cases where a 'git checkout' spanned a
diff that modified files within the sparse directory entry, but we could
not correctly find the entry.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 unpack-trees.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 5786645f315..df1f4437723 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
 static struct cache_entry *find_cache_entry(struct traverse_info *info,
 					    const struct name_entry *p)
 {
-	struct cache_entry *ce;
+	struct cache_entry *ce = NULL;
 	int pos = find_cache_pos(info, p->path, p->pathlen);
 	struct unpack_trees_options *o = info->data;
+	struct strbuf full_path = STRBUF_INIT;
 
 	if (0 <= pos)
 		return o->src_index->cache[pos];
@@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	if (pos < 0 || pos >= o->src_index->cache_nr)
 		return NULL;
 
+	strbuf_addstr(&full_path, info->traverse_path);
+	strbuf_add(&full_path, p->path, p->pathlen);
+	strbuf_addch(&full_path, '/');
+
 	/*
 	 * Due to lexicographic sorting and sparse directory
 	 * entries ending with a trailing slash, our path as a
@@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	while (pos >= 0) {
 		ce = o->src_index->cache[pos];
 
-		if (strncmp(ce->name, p->path, p->pathlen))
-			return NULL;
+		if (strncmp(ce->name, full_path.buf, full_path.len)) {
+			ce = NULL;
+			break;
+		}
 
 		if (S_ISSPARSEDIR(ce->ce_mode) &&
 		    sparse_dir_matches_path(ce, info, p))
-			return ce;
+			break;
 
 		pos--;
 	}
 
-	return NULL;
+	strbuf_release(&full_path);
+	return ce;
 }
 
 static void debug_path(struct traverse_info *info)
-- 
gitgitgadget

[PATCH v2 5/8] sparse-checkout: create helper methods

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

As we integrate the sparse index into more builtins, we occasionally
need to check the sparse-checkout patterns to see if a path is within
the sparse-checkout cone. Create some helper methods that help
initialize the patterns and check for pattern matching to make this
easier.

The existing callers of commands like get_sparse_checkout_patterns() use
a custom 'struct pattern_list' that is not necessarily the one in the
'struct index_state', so there are not many previous uses that could
adopt these helpers. There are just two in builtin/add.c and
sparse-index.c that can use path_in_sparse_checkout().

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 builtin/add.c  |  8 ++------
 dir.c          | 29 +++++++++++++++++++++++++++++
 dir.h          |  6 ++++++
 sparse-index.c | 12 +++---------
 4 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 17528e8f922..f675bdeae4a 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 	int flags = REFRESH_IGNORE_SKIP_WORKTREE |
 		    (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);
-	struct pattern_list pl = { 0 };
-	int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);
 
 	seen = xcalloc(pathspec->nr, 1);
 	refresh_index(&the_index, flags, pathspec, seen,
@@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	for (i = 0; i < pathspec->nr; i++) {
 		if (!seen[i]) {
 			const char *path = pathspec->items[i].original;
-			int dtype = DT_REG;
 
 			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
-			    (sparse_checkout_enabled &&
-			     !path_matches_pattern_list(path, strlen(path), NULL,
-							&dtype, &pl, &the_index))) {
+			    (core_apply_sparse_checkout &&
+			     path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec->items[i].original);
 			} else {
diff --git a/dir.c b/dir.c
index 03c4d212672..7027bdfa068 100644
--- a/dir.c
+++ b/dir.c
@@ -1439,6 +1439,35 @@ done:
 	return result;
 }
 
+int init_sparse_checkout_patterns(struct index_state *istate)
+{
+	if (istate->sparse_checkout_patterns)
+		return 0;
+
+	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
+
+	if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {
+		FREE_AND_NULL(istate->sparse_checkout_patterns);
+		return -1;
+	}
+
+	return 0;
+}
+
+enum pattern_match_result path_in_sparse_checkout(const char *path,
+						  struct index_state *istate)
+{
+	int dtype = DT_REG;
+	init_sparse_checkout_patterns(istate);
+
+	if (!istate->sparse_checkout_patterns)
+		return MATCHED;
+
+	return path_matches_pattern_list(path, strlen(path), NULL, &dtype,
+					 istate->sparse_checkout_patterns,
+					 istate);
+}
+
 static struct path_pattern *last_matching_pattern_from_lists(
 		struct dir_struct *dir, struct index_state *istate,
 		const char *pathname, int pathlen,
diff --git a/dir.h b/dir.h
index b3e1a54a971..9af2e8c4ba4 100644
--- a/dir.h
+++ b/dir.h
@@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,
 				const char *basename, int *dtype,
 				struct pattern_list *pl,
 				struct index_state *istate);
+
+int init_sparse_checkout_patterns(struct index_state *state);
+
+enum pattern_match_result path_in_sparse_checkout(const char *path,
+						  struct index_state *istate);
+
 struct dir_entry *dir_add_ignored(struct dir_struct *dir,
 				  struct index_state *istate,
 				  const char *pathname, int len);
diff --git a/sparse-index.c b/sparse-index.c
index b6e90417556..2efc9fd4910 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,
 	int i, can_convert = 1;
 	int start_converted = num_converted;
 	enum pattern_match_result match;
-	int dtype = DT_UNKNOWN;
 	struct strbuf child_path = STRBUF_INIT;
-	struct pattern_list *pl = istate->sparse_checkout_patterns;
 
 	/*
 	 * Is the current path outside of the sparse cone?
 	 * Then check if the region can be replaced by a sparse
 	 * directory entry (everything is sparse and merged).
 	 */
-	match = path_matches_pattern_list(ct_path, ct_pathlen,
-					  NULL, &dtype, pl, istate);
+	match = path_in_sparse_checkout(ct_path, istate);
 	if (match != NOT_MATCHED)
 		can_convert = 0;
 
@@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo->settings.sparse_index)
 		return 0;
 
-	if (!istate->sparse_checkout_patterns) {
-		istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
-		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
-			return 0;
-	}
+	if (init_sparse_checkout_patterns(istate) < 0)
+		return 0;
 
 	/*
 	 * We need cone-mode patterns to use sparse-index. If a user edits
-- 
gitgitgadget

Re: [PATCH v2 5/8] sparse-checkout: create helper methods

[Derrick Stolee]

On 8/10/2021 3:50 PM, Derrick Stolee via GitGitGadget wrote:
...
> +enum pattern_match_result path_in_sparse_checkout(const char *path,
> +						  struct index_state *istate)
> +{
> +	int dtype = DT_REG;
> +	init_sparse_checkout_patterns(istate);
> +
> +	if (!istate->sparse_checkout_patterns)
> +		return MATCHED;
> +
> +	return path_matches_pattern_list(path, strlen(path), NULL, &dtype,
> +					 istate->sparse_checkout_patterns,
> +					 istate);

While expanding on this work to fix behavior in 'git (add|rm|mv)' around
sparse entries, I noticed a problem with this method, specifically with
non-cone-mode patterns:

1. The NULL here should be the "basename" of the path, not NULL. This doesn't
   matter for cone mode, but _does_ matter for more general patterns.

2. The return type here can be UNDECIDED with general patterns, which really
   means "not matched" but is distinct from NOT_MATCHED because of the recursive
   assumptions when a directory is returned with NOT_MATCHED. Since the usage
   pattern for path_in_sparse_checkout() is to get a boolean result, the
   return type should switch to 'int' and we should return
   "path_matches_pattern_list(...) > 0".

I'm still doing some more testing to ensure I've got the necessary tweaks in
place to work with the other changes I'm going for. Plan on me sending a v3
with the appropriate changes here.

>  	/*
>  	 * Is the current path outside of the sparse cone?
>  	 * Then check if the region can be replaced by a sparse
>  	 * directory entry (everything is sparse and merged).
>  	 */
> -	match = path_matches_pattern_list(ct_path, ct_pathlen,
> -					  NULL, &dtype, pl, istate);
> +	match = path_in_sparse_checkout(ct_path, istate);
>  	if (match != NOT_MATCHED)
>  		can_convert = 0;

We could remove the use of "match" here and use the boolean result of
path_in_sparse_checkout() instead.

Thanks,
-Stolee

[PATCH v2 6/8] attr: be careful about sparse directories

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 attr.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/attr.c b/attr.c
index d029e681f28..a1009f78029 100644
--- a/attr.c
+++ b/attr.c
@@ -14,6 +14,7 @@
 #include "utf8.h"
 #include "quote.h"
 #include "thread-utils.h"
+#include "dir.h"
 
 const char git_attr__true[] = "(builtin)true";
 const char git_attr__false[] = "\0(builtin)false";
@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
 	if (!istate)
 		return NULL;
 
+	/*
+	 * In the case of cone-mode sparse-checkout, getting the
+	 * .gitattributes file from a directory is meaningless: all
+	 * contained paths will be sparse if the .gitattributes is also
+	 * sparse. In the case of a sparse index, it is critical that we
+	 * don't go looking for one as it will expand the index.
+	 */
+	init_sparse_checkout_patterns(istate);
+	if (istate->sparse_checkout_patterns &&
+	    istate->sparse_checkout_patterns->use_cone_patterns &&
+	    path_in_sparse_checkout(path, istate) == NOT_MATCHED)
+		return NULL;
+
 	buf = read_blob_data_from_index(istate, path, NULL);
 	if (!buf)
 		return NULL;
-- 
gitgitgadget

[PATCH v2 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
environment variable or the "index.sparse" config setting before
converting the index to a sparse one. This is for ease of use since all
current consumers are preparing to compress the index before writing it
to disk. If these settings are not enabled, then convert_to_sparse()
silently returns without doing anything.

We will add a consumer in the next change that wants to use the sparse
index as an in-memory data structure, regardless of whether the on-disk
format should be sparse.

To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip
these config checks when enabled. All current consumers are modified to
pass '0' in the new 'flags' parameter.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 read-cache.c   |  4 ++--
 sparse-index.c | 30 ++++++++++++++++--------------
 sparse-index.h |  3 ++-
 3 files changed, 20 insertions(+), 17 deletions(-)

diff --git a/read-cache.c b/read-cache.c
index 9048ef9e905..f5d4385c408 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
 	int ret;
 	int was_full = !istate->sparse_index;
 
-	ret = convert_to_sparse(istate);
+	ret = convert_to_sparse(istate, 0);
 
 	if (ret) {
 		warning(_("failed to convert to a sparse-index"));
@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,
 	int ret, was_full = !istate->sparse_index;
 
 	move_cache_to_base_index(istate);
-	convert_to_sparse(istate);
+	convert_to_sparse(istate, 0);
 
 	trace2_region_enter_printf("index", "shared/do_write_index",
 				   the_repository, "%s", get_tempfile_path(*temp));
diff --git a/sparse-index.c b/sparse-index.c
index 2efc9fd4910..532fd11787e 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -124,7 +124,7 @@ static int index_has_unmerged_entries(struct index_state *istate)
 	return 0;
 }
 
-int convert_to_sparse(struct index_state *istate)
+int convert_to_sparse(struct index_state *istate, int flags)
 {
 	int test_env;
 
@@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo)
 		istate->repo = the_repository;
 
-	/*
-	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
-	 * index.sparse config variable to be on.
-	 */
-	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
-	if (test_env >= 0)
-		set_sparse_index_config(istate->repo, test_env);
+	if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
+		/*
+		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
+		 * index.sparse config variable to be on.
+		 */
+		test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
+		if (test_env >= 0)
+			set_sparse_index_config(istate->repo, test_env);
 
-	/*
-	 * Only convert to sparse if index.sparse is set.
-	 */
-	prepare_repo_settings(istate->repo);
-	if (!istate->repo->settings.sparse_index)
-		return 0;
+		/*
+		 * Only convert to sparse if index.sparse is set.
+		 */
+		prepare_repo_settings(istate->repo);
+		if (!istate->repo->settings.sparse_index)
+			return 0;
+	}
 
 	if (init_sparse_checkout_patterns(istate) < 0)
 		return 0;
diff --git a/sparse-index.h b/sparse-index.h
index 1115a0d7dd9..475f4f0f8da 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -2,7 +2,8 @@
 #define SPARSE_INDEX_H__
 
 struct index_state;
-int convert_to_sparse(struct index_state *istate);
+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)
+int convert_to_sparse(struct index_state *istate, int flags);
 
 /*
  * Some places in the codebase expect to search for a specific path.
-- 
gitgitgadget

[PATCH v2 8/8] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When changing the scope of a sparse-checkout using cone mode, we might
have some tracked directories go out of scope. The current logic removes
the tracked files from within those directories, but leaves the ignored
files within those directories. This is a bit unexpected to users who
have given input to Git saying they don't need those directories
anymore.

This is something that is new to the cone mode pattern type: the user
has explicitly said "I want these directories and _not_ those
directories." The typical sparse-checkout patterns more generally apply
to "I want files with with these patterns" so it is natural to leave
ignored files as they are. This focus on directories in cone mode
provides us an opportunity to change the behavior.

Leaving these ignored files in the sparse directories makes it
impossible to gain performance benefits in the sparse index. When we
track into these directories, we need to know if the files are ignored
or not, which might depend on the _tracked_ .gitignore file(s) within
the sparse directory. This depends on the indexed version of the file,
so the sparse directory must be expanded.

By deleting the sparse directories when changing scope (or running 'git
sparse-checkout reapply') we regain these performance benefits as if the
repository was in a clean state.

Since these ignored files are frequently build output or helper files
from IDEs, the users should not need the files now that the tracked
files are removed. If the tracked files reappear, then they will have
newer timestamps than the build artifacts, so the artifacts will need to
be regenerated anyway.

Use the sparse-index as a data structure in order to find the sparse
directories that can be safely deleted. Re-expand the index to a full
one if it was full before.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt |  8 +++
 builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 3 files changed, 162 insertions(+)

diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index fdcf43f87cb..f9022b9d555 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the
 'git sparse-checkout set' command to reflect the expected cone in the working
 directory.
 
+The cone mode sparse-checkout patterns will also remove ignored files that
+are not within the sparse-checkout definition. This is important behavior
+to preserve the performance of the sparse index, but also matches that
+cone mode patterns care about directories, not files. If there exist files
+that are untracked and not ignored, then Git will not delete files within
+that directory other than the tracked files that are now out of scope.
+These files should be removed manually to ensure Git can behave optimally.
+
 
 SUBMODULES
 ----------
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 8ba9f13787b..b06c8f885ac 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)
 	return 0;
 }
 
+static void clean_tracked_sparse_directories(struct repository *r)
+{
+	int i, was_full = 0;
+	struct strbuf path = STRBUF_INIT;
+	size_t pathlen;
+	struct string_list_item *item;
+	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
+
+	/*
+	 * If we are not using cone mode patterns, then we cannot
+	 * delete directories outside of the sparse cone.
+	 */
+	if (!r || !r->index || !r->worktree)
+		return;
+	init_sparse_checkout_patterns(r->index);
+	if (!r->index->sparse_checkout_patterns ||
+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
+		return;
+
+	/*
+	 * Use the sparse index as a data structure to assist finding
+	 * directories that are safe to delete. This conversion to a
+	 * sparse index will not delete directories that contain
+	 * conflicted entries or submodules.
+	 */
+	if (!r->index->sparse_index) {
+		/*
+		 * If something, such as a merge conflict or other concern,
+		 * prevents us from converting to a sparse index, then do
+		 * not try deleting files.
+		 */
+		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
+			return;
+		was_full = 1;
+	}
+
+	strbuf_addstr(&path, r->worktree);
+	strbuf_complete(&path, '/');
+	pathlen = path.len;
+
+	/*
+	 * Collect directories that have gone out of scope but also
+	 * exist on disk, so there is some work to be done. We need to
+	 * store the entries in a list before exploring, since that might
+	 * expand the sparse-index again.
+	 */
+	for (i = 0; i < r->index->cache_nr; i++) {
+		struct cache_entry *ce = r->index->cache[i];
+
+		if (S_ISSPARSEDIR(ce->ce_mode) &&
+		    repo_file_exists(r, ce->name))
+			string_list_append(&sparse_dirs, ce->name);
+	}
+
+	for_each_string_list_item(item, &sparse_dirs) {
+		struct dir_struct dir = DIR_INIT;
+		struct pathspec p = { 0 };
+		struct strvec s = STRVEC_INIT;
+
+		strbuf_setlen(&path, pathlen);
+		strbuf_addstr(&path, item->string);
+
+		dir.flags |= DIR_SHOW_IGNORED_TOO;
+
+		setup_standard_excludes(&dir);
+		strvec_push(&s, path.buf);
+
+		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
+		fill_directory(&dir, r->index, &p);
+
+		if (dir.nr) {
+			warning(_("directory '%s' contains untracked files,"
+				  " but is not in the sparse-checkout cone"),
+				item->string);
+		} else if (remove_dir_recursively(&path, 0)) {
+			/*
+			 * Removal is "best effort". If something blocks
+			 * the deletion, then continue with a warning.
+			 */
+			warning(_("failed to remove directory '%s'"),
+				item->string);
+		}
+
+		dir_clear(&dir);
+	}
+
+	string_list_clear(&sparse_dirs, 0);
+	strbuf_release(&path);
+
+	if (was_full)
+		ensure_full_index(r->index);
+}
+
 static int update_working_directory(struct pattern_list *pl)
 {
 	enum update_sparsity_result result;
@@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)
 	else
 		rollback_lock_file(&lock_file);
 
+	clean_tracked_sparse_directories(r);
+
 	r->index->sparse_checkout_patterns = NULL;
 	return result;
 }
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..71236981e64 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
 	check_files repo/deep a deeper1
 '
 
+test_expect_success 'cone mode clears ignored subdirectories' '
+	rm repo/.git/info/sparse-checkout &&
+
+	git -C repo sparse-checkout init --cone &&
+	git -C repo sparse-checkout set deep/deeper1 &&
+
+	cat >repo/.gitignore <<-\EOF &&
+	obj/
+	*.o
+	EOF
+
+	git -C repo add .gitignore &&
+	git -C repo commit -m ".gitignore" &&
+
+	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
+	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
+		    deep/deeper2/obj/a deep/deeper2/file.o file.o
+	do
+		echo ignored >repo/$file || return 1
+	done &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout reapply &&
+	test_path_is_missing repo/folder1 &&
+	test_path_is_missing repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout set deep/deeper2 &&
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	>repo/deep/deeper2/ignored.o &&
+	>repo/deep/deeper2/untracked &&
+
+	# When an untracked file is in the way, all untracked files
+	# (even ignored files) are preserved.
+	git -C repo sparse-checkout set folder1 2>err &&
+	grep "contains untracked files" err &&
+	test_path_is_file repo/deep/deeper2/ignored.o &&
+	test_path_is_file repo/deep/deeper2/untracked &&
+
+	# The rest of the cone matches expectation
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	echo "? deep/deeper2/untracked" >expect &&
+	test_cmp expect out
+'
+
 test_done
-- 
gitgitgadget

[PATCH v3 0/8] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee via GitGitGadget]

UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
and v2.33.0-rc1 because of some changes to dir.h that became important!

We launched an experimental release [1] of the sparse-index feature to our
internal users. We immediately discovered a problem due to the isolated way
in which we tested the sparse index: we were never building the project and
changing our sparse-checkout definition.

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Users who ran a build in one project and then moved to another still had
build artifacts in their worktree that lived inside the old directories.
Since the files are marked by the .gitignore patterns, these files were not
removed by the 'git sparse-checkout set' command. However, they make the
sparse-index unusable because every 'git status' command needs to expand the
sparse-directory entries in order to see if the files are tracked or not.
This made the first experimental release actually slower for all users
because of this cost.

The solution we shipped to these customers was to change the way our fork
handles these ignored files. Specifically, instead of Git completely
ignoring the files, we changed Git to understand that with cone-mode
sparse-checkout patterns, the users is asking for entire directories to be
removed from the worktree. The link [1] included earlier has this change.

I believe that this is a reasonable expectation, though I recognize that it
might look like breaking the expectations of how .gitignore files work.

Since feedback demonstrated that this is a desired behavior, v2 includes
this behavior for all "cone mode" repositories.

I'm interested in the community's thoughts about this change, as it seems
like one that we should make carefully and intentionally.

While the rewrite of the t7519 test seems unrelated, it is required to avoid
a test failure with this change that deletes files outside of the cone. By
moving the test into repositories not at $TRASH_DIRECTORY, we gain more
control over the repository structure.


Update in V3
============

 * As promised [2], the helper methods are fixed to work with non-cone-mode
   patterns. A later series will use them to their fullest potential
   (changing git add, git rm, and git mv when interacting with sparse
   entries).

[2]
https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/


Updates in V2
=============

 * This version correctly leaves untracked files alone. If untracked files
   are found, then the directory is left as-is, in case those ignored files
   are important to the user's work resolving those untracked files.

 * This behavior is now enabled by core.sparseCheckoutCone=true.

 * To use a sparse index as an in-memory data structure even when
   index.sparse is disabled, a new patch is included to modify the prototype
   of convert_to_sparse() to include a flags parameter.

 * A few cleanup patches that I was collecting based on feedback from the
   experimental release and intending for my next series were necessary for
   this implementation.

 * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
   implementation that used run_subcommand().

Thanks, -Stolee

Derrick Stolee (8):
  t7519: rewrite sparse index test
  sparse-index: silently return when not using cone-mode patterns
  sparse-index: silently return when cache tree fails
  unpack-trees: fix nested sparse-dir search
  sparse-checkout: create helper methods
  attr: be careful about sparse directories
  sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
  sparse-checkout: clear tracked sparse dirs

 Documentation/git-sparse-checkout.txt |  8 +++
 attr.c                                | 14 ++++
 builtin/add.c                         |  8 +--
 builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
 dir.c                                 | 33 ++++++++++
 dir.h                                 |  6 ++
 read-cache.c                          |  4 +-
 sparse-index.c                        | 70 +++++++++++---------
 sparse-index.h                        |  3 +-
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 t/t7519-status-fsmonitor.sh           | 38 ++++++-----
 unpack-trees.c                        | 18 +++--
 12 files changed, 294 insertions(+), 62 deletions(-)


base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1009

Range-diff vs v2:

 1:  e66106f7a99 = 1:  e66106f7a99 t7519: rewrite sparse index test
 2:  fb3ff9108bf = 2:  fb3ff9108bf sparse-index: silently return when not using cone-mode patterns
 3:  37198535268 = 3:  37198535268 sparse-index: silently return when cache tree fails
 4:  10bcadb284e = 4:  10bcadb284e unpack-trees: fix nested sparse-dir search
 5:  e9ed5cd2830 ! 5:  5d28570c82a sparse-checkout: create helper methods
     @@ dir.c: done:
       
      +int init_sparse_checkout_patterns(struct index_state *istate)
      +{
     -+	if (istate->sparse_checkout_patterns)
     ++	if (!core_apply_sparse_checkout ||
     ++	    istate->sparse_checkout_patterns)
      +		return 0;
      +
      +	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
     @@ dir.c: done:
      +	return 0;
      +}
      +
     -+enum pattern_match_result path_in_sparse_checkout(const char *path,
     -+						  struct index_state *istate)
     ++int path_in_sparse_checkout(const char *path,
     ++			    struct index_state *istate)
      +{
     ++	const char *base;
      +	int dtype = DT_REG;
      +	init_sparse_checkout_patterns(istate);
      +
      +	if (!istate->sparse_checkout_patterns)
      +		return MATCHED;
      +
     -+	return path_matches_pattern_list(path, strlen(path), NULL, &dtype,
     ++	base = strrchr(path, '/');
     ++	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
     ++					 &dtype,
      +					 istate->sparse_checkout_patterns,
     -+					 istate);
     ++					 istate) > 0;
      +}
      +
       static struct path_pattern *last_matching_pattern_from_lists(
     @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
      +
      +int init_sparse_checkout_patterns(struct index_state *state);
      +
     -+enum pattern_match_result path_in_sparse_checkout(const char *path,
     -+						  struct index_state *istate);
     ++int path_in_sparse_checkout(const char *path,
     ++			    struct index_state *istate);
      +
       struct dir_entry *dir_add_ignored(struct dir_struct *dir,
       				  struct index_state *istate,
 6:  5680df62e1c = 6:  c9e100e68f8 attr: be careful about sparse directories
 7:  1dd73b36eb4 = 7:  b0ece4b7dcc sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
 8:  f74015a2be9 = 8:  febef675f05 sparse-checkout: clear tracked sparse dirs

-- 
gitgitgadget

[PATCH v3 1/8] t7519: rewrite sparse index test

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The sparse index is tested with the FS Monitor hook and extension since
f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
was very fragile because it shared an index across sparse and non-sparse
behavior. Since that expansion and contraction could cause the index to
lose its FS Monitor bitmap and token, behavior is fragile to changes in
'git sparse-checkout set'.

Rewrite the test to use two clones of the original repo: full and
sparse. This allows us to also keep the test files (actual, expect,
trace2.txt) out of the repos we are testing with 'git status'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index deea88d4431..6f2cf306f66 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
 # If "!" is supplied, then we verify that we do not call ensure_full_index
 # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
 check_sparse_index_behavior () {
-	git status --porcelain=v2 >expect &&
-	git sparse-checkout init --cone --sparse-index &&
-	git sparse-checkout set dir1 dir2 &&
+	git -C full status --porcelain=v2 >expect &&
 	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-		git status --porcelain=v2 >actual &&
+		git -C sparse status --porcelain=v2 >actual &&
 	test_region $1 index ensure_full_index trace2.txt &&
 	test_region fsm_hook query trace2.txt &&
 	test_cmp expect actual &&
-	rm trace2.txt &&
-	git sparse-checkout disable
+	rm trace2.txt
 }
 
 test_expect_success 'status succeeds with sparse index' '
-	git reset --hard &&
+	git clone . full &&
+	git clone . sparse &&
+	git -C sparse sparse-checkout init --cone --sparse-index &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
-	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
-	check_sparse_index_behavior ! &&
-
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 	EOF
-	git config core.fsmonitor .git/hooks/fsmonitor-test &&
+	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
+	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
 	check_sparse_index_behavior ! &&
 
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1/modified\0"
 	EOF
 	check_sparse_index_behavior ! &&
 
-	cp -r dir1 dir1a &&
-	git add dir1a &&
-	git commit -m "add dir1a" &&
+	git -C sparse sparse-checkout add dir1a &&
+
+	for repo in full sparse
+	do
+		cp -r $repo/dir1 $repo/dir1a &&
+		git -C $repo add dir1a &&
+		git -C $repo commit -m "add dir1a" || return 1
+	done &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
 	# This one modifies outside the sparse-checkout definition
 	# and hence we expect to expand the sparse-index.
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1a/modified\0"
 	EOF
-- 
gitgitgadget

Re: [PATCH v3 1/8] t7519: rewrite sparse index test

[Johannes Schindelin]

Hi Stolee,

On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> The sparse index is tested with the FS Monitor hook and extension since
> f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
> was very fragile because it shared an index across sparse and non-sparse
> behavior. Since that expansion and contraction could cause the index to
> lose its FS Monitor bitmap and token, behavior is fragile to changes in
> 'git sparse-checkout set'.
>
> Rewrite the test to use two clones of the original repo: full and
> sparse. This allows us to also keep the test files (actual, expect,
> trace2.txt) out of the repos we are testing with 'git status'.

Makes sense.

It also should accelerate the test, as it does not have to convert between
sparse and full index all the time.

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
>  1 file changed, 21 insertions(+), 17 deletions(-)
>
> diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
> index deea88d4431..6f2cf306f66 100755
> --- a/t/t7519-status-fsmonitor.sh
> +++ b/t/t7519-status-fsmonitor.sh
> @@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
>  # If "!" is supplied, then we verify that we do not call ensure_full_index
>  # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
>  check_sparse_index_behavior () {
> -	git status --porcelain=v2 >expect &&
> -	git sparse-checkout init --cone --sparse-index &&
> -	git sparse-checkout set dir1 dir2 &&
> +	git -C full status --porcelain=v2 >expect &&
>  	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
> -		git status --porcelain=v2 >actual &&
> +		git -C sparse status --porcelain=v2 >actual &&
>  	test_region $1 index ensure_full_index trace2.txt &&
>  	test_region fsm_hook query trace2.txt &&
>  	test_cmp expect actual &&
> -	rm trace2.txt &&
> -	git sparse-checkout disable
> +	rm trace2.txt
>  }
>
>  test_expect_success 'status succeeds with sparse index' '
> -	git reset --hard &&
> +	git clone . full &&
> +	git clone . sparse &&
> +	git -C sparse sparse-checkout init --cone --sparse-index &&

Would it make sense to call `git clone --sparse . sparse`? I see that
there is no support for `--sparse=cone`, which makes me wonder whether we
want that at some stage. In any case, cloning with `--sparse` and then
setting up the cone mode should result in a little less work, right?

> +	git -C sparse sparse-checkout set dir1 dir2 &&
>
> -	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
> -	check_sparse_index_behavior ! &&
> -
> -	write_script .git/hooks/fsmonitor-test<<-\EOF &&
> +	write_script .git/hooks/fsmonitor-test <<-\EOF &&
>  		printf "last_update_token\0"

Technically, the backslash needs to be escaped because it is within double
quotes and we do not want the shell to interpolate the `\0`, but `printf`.
Practically, all the shells I tried handle this as expected.

Also, I have a slight preference for:

		printf "%s\\0" last_update_token

and later

		printf "%s\\0" last_update_token dir1/modified

What do your taste buds say about this?

These are only minor nits, of course. I do like the overall patch and
would be fine with it as-is.

Ciao,
Dscho

>  	EOF
> -	git config core.fsmonitor .git/hooks/fsmonitor-test &&
> +	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
> +	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
>  	check_sparse_index_behavior ! &&
>
> -	write_script .git/hooks/fsmonitor-test<<-\EOF &&
> +	write_script .git/hooks/fsmonitor-test <<-\EOF &&
>  		printf "last_update_token\0"
>  		printf "dir1/modified\0"
>  	EOF
>  	check_sparse_index_behavior ! &&
>
> -	cp -r dir1 dir1a &&
> -	git add dir1a &&
> -	git commit -m "add dir1a" &&
> +	git -C sparse sparse-checkout add dir1a &&
> +
> +	for repo in full sparse
> +	do
> +		cp -r $repo/dir1 $repo/dir1a &&
> +		git -C $repo add dir1a &&
> +		git -C $repo commit -m "add dir1a" || return 1
> +	done &&
> +	git -C sparse sparse-checkout set dir1 dir2 &&
>
>  	# This one modifies outside the sparse-checkout definition
>  	# and hence we expect to expand the sparse-index.
> -	write_script .git/hooks/fsmonitor-test<<-\EOF &&
> +	write_script .git/hooks/fsmonitor-test <<-\EOF &&
>  		printf "last_update_token\0"
>  		printf "dir1a/modified\0"
>  	EOF
> --
> gitgitgadget
>
>

Re: [PATCH v3 1/8] t7519: rewrite sparse index test

[Derrick Stolee]

On 8/19/2021 3:45 AM, Johannes Schindelin wrote:> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
>>  test_expect_success 'status succeeds with sparse index' '
>> -	git reset --hard &&
>> +	git clone . full &&
>> +	git clone . sparse &&
>> +	git -C sparse sparse-checkout init --cone --sparse-index &&
> 
> Would it make sense to call `git clone --sparse . sparse`? I see that
> there is no support for `--sparse=cone`, which makes me wonder whether we
> want that at some stage. In any case, cloning with `--sparse` and then
> setting up the cone mode should result in a little less work, right?

The amount of work saved is miniscule, but I can understand wanting to
shave what we can from the cost of tests.

>> +	git -C sparse sparse-checkout set dir1 dir2 &&
>>
>> -	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
>> -	check_sparse_index_behavior ! &&
>> -
>> -	write_script .git/hooks/fsmonitor-test<<-\EOF &&
>> +	write_script .git/hooks/fsmonitor-test <<-\EOF &&
>>  		printf "last_update_token\0"
> 
> Technically, the backslash needs to be escaped because it is within double
> quotes and we do not want the shell to interpolate the `\0`, but `printf`.
> Practically, all the shells I tried handle this as expected.
> 
> Also, I have a slight preference for:
> 
> 		printf "%s\\0" last_update_token
> 
> and later
> 
> 		printf "%s\\0" last_update_token dir1/modified
> 
> What do your taste buds say about this?

I have no opinions on this one way or another. I will just point out that
the pattern used in this test is also used throughout the test script, so
any change to that format should be applied universally. The test has been
operating without complaint since it was introduced in 5c8cdcf (fsmonitor:
add test cases for fsmonitor extension, 2017-09-22), so compatibility is
likely not a problem.

Thanks,
-Stolee

Re: [PATCH v3 1/8] t7519: rewrite sparse index test

[Eric Sunshine]

On Fri, Aug 20, 2021 at 11:09 AM Derrick Stolee <stolee@gmail.com> wrote:
> On 8/19/2021 3:45 AM, Johannes Schindelin wrote:> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
> >> +    write_script .git/hooks/fsmonitor-test <<-\EOF &&
> >>              printf "last_update_token\0"
> >
> > Technically, the backslash needs to be escaped because it is within double
> > quotes and we do not want the shell to interpolate the `\0`, but `printf`.
> > Practically, all the shells I tried handle this as expected.

Technically, for a POSIX-conforming shell, `\0` is correct, and the
backslash does not need to be escaped. Of course, it doesn't hurt to
write `\\0` since it resolves to the same thing, but it isn't
necessary. See POSIX section "2.2.3 Double-Quotes":
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_03

> I have no opinions on this one way or another. I will just point out that
> the pattern used in this test is also used throughout the test script, so
> any change to that format should be applied universally. The test has been
> operating without complaint since it was introduced in 5c8cdcf (fsmonitor:
> add test cases for fsmonitor extension, 2017-09-22), so compatibility is
> likely not a problem.

Since POSIX says that `\0` is correct and since we haven't encountered
any shells which handle this incorrectly, there seems little reason to
change it.

[PATCH v3 2/8] sparse-index: silently return when not using cone-mode patterns

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

While the sparse-index is only enabled when core.sparseCheckoutCone is
also enabled, it is possible for the user to modify the sparse-checkout
file manually in a way that does not match cone-mode patterns. In this
case, we should refuse to convert an index into a sparse index, since
the sparse_checkout_patterns will not be initialized with recursive and
parent path hashsets.

Also silently return if there are no cache entries, which is a simple
case: there are no paths to make sparse!

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index c6b4feec413..bc5900eae35 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -130,7 +130,8 @@ static int index_has_unmerged_entries(struct index_state *istate)
 int convert_to_sparse(struct index_state *istate)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index ||
+
+	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
@@ -158,10 +159,16 @@ int convert_to_sparse(struct index_state *istate)
 			return 0;
 	}
 
-	if (!istate->sparse_checkout_patterns->use_cone_patterns) {
-		warning(_("attempting to use sparse-index without cone mode"));
-		return -1;
-	}
+	/*
+	 * We need cone-mode patterns to use sparse-index. If a user edits
+	 * their sparse-checkout file manually, then we can detect during
+	 * parsing that they are not actually using cone-mode patterns and
+	 * hence we need to abort this conversion _without error_. Warnings
+	 * already exist in the pattern parsing to inform the user of their
+	 * bad patterns.
+	 */
+	if (!istate->sparse_checkout_patterns->use_cone_patterns)
+		return 0;
 
 	/*
 	 * NEEDSWORK: If we have unmerged entries, then stay full.
-- 
gitgitgadget

[PATCH v3 3/8] sparse-index: silently return when cache tree fails

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

If cache_tree_update() returns a non-zero value, then it could not
create the cache tree. This is likely due to a path having a merge
conflict. Since we are already returning early, let's return silently to
avoid making it seem like we failed to write the index at all.

If we remove our dependence on the cache tree within
convert_to_sparse(), then we could still recover from this scenario and
have a sparse index.

When constructing the cache-tree extension in convert_to_sparse(), it is
possible that we construct a tree object that is new to the object
database. Without the WRITE_TREE_MISSING_OK flag, this results in an
error that halts our conversion to a sparse index. Add this flag to
remove this limitation.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index bc5900eae35..b6e90417556 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)
 
 	/* Clear and recompute the cache-tree */
 	cache_tree_free(&istate->cache_tree);
-	if (cache_tree_update(istate, 0)) {
-		warning(_("unable to update cache-tree, staying full"));
-		return -1;
-	}
+	/*
+	 * Silently return if there is a problem with the cache tree update,
+	 * which might just be due to a conflict state in some entry.
+	 *
+	 * This might create new tree objects, so be sure to use
+	 * WRITE_TREE_MISSING_OK.
+	 */
+	if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
+		return 0;
 
 	remove_fsmonitor(istate);
 
-- 
gitgitgadget

[PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The iterated search in find_cache_entry() was recently modified to
include a loop that searches backwards for a sparse directory entry that
matches the given traverse_info and name_entry. However, the string
comparison failed to actually concatenate those two strings, so this
failed to find a sparse directory when it was not a top-level directory.

This caused some errors in rare cases where a 'git checkout' spanned a
diff that modified files within the sparse directory entry, but we could
not correctly find the entry.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 unpack-trees.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 5786645f315..df1f4437723 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
 static struct cache_entry *find_cache_entry(struct traverse_info *info,
 					    const struct name_entry *p)
 {
-	struct cache_entry *ce;
+	struct cache_entry *ce = NULL;
 	int pos = find_cache_pos(info, p->path, p->pathlen);
 	struct unpack_trees_options *o = info->data;
+	struct strbuf full_path = STRBUF_INIT;
 
 	if (0 <= pos)
 		return o->src_index->cache[pos];
@@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	if (pos < 0 || pos >= o->src_index->cache_nr)
 		return NULL;
 
+	strbuf_addstr(&full_path, info->traverse_path);
+	strbuf_add(&full_path, p->path, p->pathlen);
+	strbuf_addch(&full_path, '/');
+
 	/*
 	 * Due to lexicographic sorting and sparse directory
 	 * entries ending with a trailing slash, our path as a
@@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	while (pos >= 0) {
 		ce = o->src_index->cache[pos];
 
-		if (strncmp(ce->name, p->path, p->pathlen))
-			return NULL;
+		if (strncmp(ce->name, full_path.buf, full_path.len)) {
+			ce = NULL;
+			break;
+		}
 
 		if (S_ISSPARSEDIR(ce->ce_mode) &&
 		    sparse_dir_matches_path(ce, info, p))
-			return ce;
+			break;
 
 		pos--;
 	}
 
-	return NULL;
+	strbuf_release(&full_path);
+	return ce;
 }
 
 static void debug_path(struct traverse_info *info)
-- 
gitgitgadget

Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[Johannes Schindelin]

Hi Stolee,

On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> The iterated search in find_cache_entry() was recently modified to
> include a loop that searches backwards for a sparse directory entry that
> matches the given traverse_info and name_entry. However, the string
> comparison failed to actually concatenate those two strings, so this
> failed to find a sparse directory when it was not a top-level directory.
>
> This caused some errors in rare cases where a 'git checkout' spanned a
> diff that modified files within the sparse directory entry, but we could
> not correctly find the entry.

Good explanation.

I wonder a bit about the performance impact. How "hot" is this function?
I.e. how often is it called, on average?

I ask because I see opportunities to optimize in both directions: it could
be written more concisely (if speed does not matter as much), and it could
be made faster (if speed matters a lot). See below for more.

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  unpack-trees.c | 18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 5786645f315..df1f4437723 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
>  static struct cache_entry *find_cache_entry(struct traverse_info *info,
>  					    const struct name_entry *p)
>  {
> -	struct cache_entry *ce;
> +	struct cache_entry *ce = NULL;

Makes sense: since you need to release the allocated memory, you can no
longer `return NULL` early, but have to break out of the loop and return
`ce`.

>  	int pos = find_cache_pos(info, p->path, p->pathlen);
>  	struct unpack_trees_options *o = info->data;
> +	struct strbuf full_path = STRBUF_INIT;
>
>  	if (0 <= pos)
>  		return o->src_index->cache[pos];
> @@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>  	if (pos < 0 || pos >= o->src_index->cache_nr)
>  		return NULL;
>
> +	strbuf_addstr(&full_path, info->traverse_path);
> +	strbuf_add(&full_path, p->path, p->pathlen);
> +	strbuf_addch(&full_path, '/');

This could be reduced to:

	strbuf_addf(&full_path, "%s%.*s/",
		    info->traverse_path, (int)p->pathlen, p->path);

But if speed matters, we probably need something more like this:

	size_t full_path_len;
	const char *full_path;
	char *full_path_1 = NULL;

	if (!*info->traverse_path) {
		full_path = p->path;
		full_path_len = p->pathlen;
	} else {
		size_t len = strlen(info->traverse_path);

		full_path_len = len + p->pathlen + 1;
		full_path = full_path_1 = xmalloc(full_path_len + 1);
		memcpy(full_path_1, info->traverse_path, len);
		memcpy(full_path_1 + len, p->path, p->pathlen);
		full_path_1[full_path_len - 1] = '/';
		full_path_1[full_path_len] = '\0';
	}

	[...]

	free(full_path_1);

It would obviously be much nicer if we did not have to go for that ugly
long version...

> +
>  	/*
>  	 * Due to lexicographic sorting and sparse directory
>  	 * entries ending with a trailing slash, our path as a
> @@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>  	while (pos >= 0) {
>  		ce = o->src_index->cache[pos];
>
> -		if (strncmp(ce->name, p->path, p->pathlen))
> -			return NULL;
> +		if (strncmp(ce->name, full_path.buf, full_path.len)) {
> +			ce = NULL;
> +			break;
> +		}
>
>  		if (S_ISSPARSEDIR(ce->ce_mode) &&
>  		    sparse_dir_matches_path(ce, info, p))
> -			return ce;
> +			break;
>
>  		pos--;
>  	}
>
> -	return NULL;
> +	strbuf_release(&full_path);
> +	return ce;
>  }
>
>  static void debug_path(struct traverse_info *info)
> --
> gitgitgadget
>
>

Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[Derrick Stolee]

On 8/19/2021 4:01 AM, Johannes Schindelin wrote:
> Hi Stolee,
> 
> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
> 
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> The iterated search in find_cache_entry() was recently modified to
>> include a loop that searches backwards for a sparse directory entry that
>> matches the given traverse_info and name_entry. However, the string
>> comparison failed to actually concatenate those two strings, so this
>> failed to find a sparse directory when it was not a top-level directory.
>>
>> This caused some errors in rare cases where a 'git checkout' spanned a
>> diff that modified files within the sparse directory entry, but we could
>> not correctly find the entry.
> 
> Good explanation.
> 
> I wonder a bit about the performance impact. How "hot" is this function?
> I.e. how often is it called, on average?
> 
> I ask because I see opportunities to optimize in both directions: it could
> be written more concisely (if speed does not matter as much), and it could
> be made faster (if speed matters a lot). See below for more.

I would definitely optimize for speed here. This can be a very hot path,
I believe.

>> +	strbuf_addstr(&full_path, info->traverse_path);
>> +	strbuf_add(&full_path, p->path, p->pathlen);
>> +	strbuf_addch(&full_path, '/');
> 
> This could be reduced to:
> 
> 	strbuf_addf(&full_path, "%s%.*s/",
> 		    info->traverse_path, (int)p->pathlen, p->path);

We should definitely avoid formatted strings here, if possible.

> But if speed matters, we probably need something more like this:
> 
> 	size_t full_path_len;
> 	const char *full_path;
> 	char *full_path_1 = NULL;
> 
> 	if (!*info->traverse_path) {
> 		full_path = p->path;
> 		full_path_len = p->pathlen;
> 	} else {
> 		size_t len = strlen(info->traverse_path);
> 
> 		full_path_len = len + p->pathlen + 1;
> 		full_path = full_path_1 = xmalloc(full_path_len + 1);
> 		memcpy(full_path_1, info->traverse_path, len);
> 		memcpy(full_path_1 + len, p->path, p->pathlen);
> 		full_path_1[full_path_len - 1] = '/';
> 		full_path_1[full_path_len] = '\0';
> 	}

The critical benefit here is that we do not need to allocate a
buffer if the traverse_path does not exist. That might be a
worthwhile investment. That leads to justifying the use of
bare 'char *'s instead of 'struct strbuf'.

If the traverse_path is usually non-null, then we could continue using
strbufs as a helper and get the planned performance gains by using
strbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()
(instead of strbuf_addstr()). That would make this code a bit less
ugly with the only real overhead being the extra insertions of '\0'
characters as we add the strings to the strbuf().

I will need to investigate so see which one is the best.

Thanks,
-Stolee

Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[René Scharfe]

Am 20.08.21 um 17:18 schrieb Derrick Stolee:
> On 8/19/2021 4:01 AM, Johannes Schindelin wrote:
>> Hi Stolee,
>>
>> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
>>
>>> From: Derrick Stolee <dstolee@microsoft.com>
>>>
>>> The iterated search in find_cache_entry() was recently modified to
>>> include a loop that searches backwards for a sparse directory entry that
>>> matches the given traverse_info and name_entry. However, the string
>>> comparison failed to actually concatenate those two strings, so this
>>> failed to find a sparse directory when it was not a top-level directory.
>>>
>>> This caused some errors in rare cases where a 'git checkout' spanned a
>>> diff that modified files within the sparse directory entry, but we could
>>> not correctly find the entry.
>>
>> Good explanation.
>>
>> I wonder a bit about the performance impact. How "hot" is this function?
>> I.e. how often is it called, on average?
>>
>> I ask because I see opportunities to optimize in both directions: it could
>> be written more concisely (if speed does not matter as much), and it could
>> be made faster (if speed matters a lot). See below for more.
>
> I would definitely optimize for speed here. This can be a very hot path,
> I believe.
>
>>> +	strbuf_addstr(&full_path, info->traverse_path);
>>> +	strbuf_add(&full_path, p->path, p->pathlen);
>>> +	strbuf_addch(&full_path, '/');
>>
>> This could be reduced to:
>>
>> 	strbuf_addf(&full_path, "%s%.*s/",
>> 		    info->traverse_path, (int)p->pathlen, p->path);
>
> We should definitely avoid formatted strings here, if possible.
>
>> But if speed matters, we probably need something more like this:
>>
>> 	size_t full_path_len;
>> 	const char *full_path;
>> 	char *full_path_1 = NULL;
>>
>> 	if (!*info->traverse_path) {
>> 		full_path = p->path;
>> 		full_path_len = p->pathlen;
>> 	} else {
>> 		size_t len = strlen(info->traverse_path);
>>
>> 		full_path_len = len + p->pathlen + 1;
>> 		full_path = full_path_1 = xmalloc(full_path_len + 1);
>> 		memcpy(full_path_1, info->traverse_path, len);
>> 		memcpy(full_path_1 + len, p->path, p->pathlen);
>> 		full_path_1[full_path_len - 1] = '/';
>> 		full_path_1[full_path_len] = '\0';
>> 	}
>
> The critical benefit here is that we do not need to allocate a
> buffer if the traverse_path does not exist. That might be a
> worthwhile investment. That leads to justifying the use of
> bare 'char *'s instead of 'struct strbuf'.
>
> If the traverse_path is usually non-null, then we could continue using
> strbufs as a helper and get the planned performance gains by using
> strbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()
> (instead of strbuf_addstr()). That would make this code a bit less
> ugly with the only real overhead being the extra insertions of '\0'
> characters as we add the strings to the strbuf().

You create full_path only to compare it to another string.  You can
compare the pieces directly, without allocating and copying:

	const char *path;

	if (!skip_prefix(ce->name, info->traverse_path, &path) ||
	    strncmp(path, p->path, p->pathlen) ||
	    strcmp(path + p->pathlen, "/"))
		return NULL;

A test would be nice to demonstrate the fixed issue.

René

Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[René Scharfe]

Am 20.08.21 um 21:35 schrieb René Scharfe:
> Am 20.08.21 um 17:18 schrieb Derrick Stolee:
>> On 8/19/2021 4:01 AM, Johannes Schindelin wrote:
>>> Hi Stolee,
>>>
>>> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
>>>
>>>> From: Derrick Stolee <dstolee@microsoft.com>
>>>>
>>>> The iterated search in find_cache_entry() was recently modified to
>>>> include a loop that searches backwards for a sparse directory entry that
>>>> matches the given traverse_info and name_entry. However, the string
>>>> comparison failed to actually concatenate those two strings, so this
>>>> failed to find a sparse directory when it was not a top-level directory.
>>>>
>>>> This caused some errors in rare cases where a 'git checkout' spanned a
>>>> diff that modified files within the sparse directory entry, but we could
>>>> not correctly find the entry.
>>>
>>> Good explanation.
>>>
>>> I wonder a bit about the performance impact. How "hot" is this function?
>>> I.e. how often is it called, on average?
>>>
>>> I ask because I see opportunities to optimize in both directions: it could
>>> be written more concisely (if speed does not matter as much), and it could
>>> be made faster (if speed matters a lot). See below for more.
>>
>> I would definitely optimize for speed here. This can be a very hot path,
>> I believe.
>>
>>>> +	strbuf_addstr(&full_path, info->traverse_path);
>>>> +	strbuf_add(&full_path, p->path, p->pathlen);
>>>> +	strbuf_addch(&full_path, '/');
>>>
>>> This could be reduced to:
>>>
>>> 	strbuf_addf(&full_path, "%s%.*s/",
>>> 		    info->traverse_path, (int)p->pathlen, p->path);
>>
>> We should definitely avoid formatted strings here, if possible.
>>
>>> But if speed matters, we probably need something more like this:
>>>
>>> 	size_t full_path_len;
>>> 	const char *full_path;
>>> 	char *full_path_1 = NULL;
>>>
>>> 	if (!*info->traverse_path) {
>>> 		full_path = p->path;
>>> 		full_path_len = p->pathlen;
>>> 	} else {
>>> 		size_t len = strlen(info->traverse_path);
>>>
>>> 		full_path_len = len + p->pathlen + 1;
>>> 		full_path = full_path_1 = xmalloc(full_path_len + 1);
>>> 		memcpy(full_path_1, info->traverse_path, len);
>>> 		memcpy(full_path_1 + len, p->path, p->pathlen);
>>> 		full_path_1[full_path_len - 1] = '/';
>>> 		full_path_1[full_path_len] = '\0';
>>> 	}
>>
>> The critical benefit here is that we do not need to allocate a
>> buffer if the traverse_path does not exist. That might be a
>> worthwhile investment. That leads to justifying the use of
>> bare 'char *'s instead of 'struct strbuf'.
>>
>> If the traverse_path is usually non-null, then we could continue using
>> strbufs as a helper and get the planned performance gains by using
>> strbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()
>> (instead of strbuf_addstr()). That would make this code a bit less
>> ugly with the only real overhead being the extra insertions of '\0'
>> characters as we add the strings to the strbuf().
>
> You create full_path only to compare it to another string.  You can
> compare the pieces directly, without allocating and copying:
>
> 	const char *path;
>
> 	if (!skip_prefix(ce->name, info->traverse_path, &path) ||
> 	    strncmp(path, p->path, p->pathlen) ||
> 	    strcmp(path + p->pathlen, "/"))

The strcmp line is wrong (should be path[p->pathlen] != '/'), but
you get the idea..

> 		return NULL;
>
> A test would be nice to demonstrate the fixed issue.
>
> René
>

Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search

[Elijah Newren]

On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> The iterated search in find_cache_entry() was recently modified to
> include a loop that searches backwards for a sparse directory entry that
> matches the given traverse_info and name_entry. However, the string
> comparison failed to actually concatenate those two strings, so this
> failed to find a sparse directory when it was not a top-level directory.
>
> This caused some errors in rare cases where a 'git checkout' spanned a
> diff that modified files within the sparse directory entry, but we could
> not correctly find the entry.

Doh, sorry for missing that in my review of the earlier series.

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  unpack-trees.c | 18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 5786645f315..df1f4437723 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
>  static struct cache_entry *find_cache_entry(struct traverse_info *info,
>                                             const struct name_entry *p)
>  {
> -       struct cache_entry *ce;
> +       struct cache_entry *ce = NULL;
>         int pos = find_cache_pos(info, p->path, p->pathlen);
>         struct unpack_trees_options *o = info->data;
> +       struct strbuf full_path = STRBUF_INIT;
>
>         if (0 <= pos)
>                 return o->src_index->cache[pos];
> @@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>         if (pos < 0 || pos >= o->src_index->cache_nr)
>                 return NULL;
>
> +       strbuf_addstr(&full_path, info->traverse_path);
> +       strbuf_add(&full_path, p->path, p->pathlen);
> +       strbuf_addch(&full_path, '/');
> +
>         /*
>          * Due to lexicographic sorting and sparse directory
>          * entries ending with a trailing slash, our path as a
> @@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>         while (pos >= 0) {
>                 ce = o->src_index->cache[pos];
>
> -               if (strncmp(ce->name, p->path, p->pathlen))
> -                       return NULL;
> +               if (strncmp(ce->name, full_path.buf, full_path.len)) {
> +                       ce = NULL;
> +                       break;
> +               }
>
>                 if (S_ISSPARSEDIR(ce->ce_mode) &&
>                     sparse_dir_matches_path(ce, info, p))
> -                       return ce;
> +                       break;
>
>                 pos--;
>         }
>
> -       return NULL;
> +       strbuf_release(&full_path);
> +       return ce;
>  }
>
>  static void debug_path(struct traverse_info *info)
> --
> gitgitgadget

Makes sense.

[PATCH v3 5/8] sparse-checkout: create helper methods

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

As we integrate the sparse index into more builtins, we occasionally
need to check the sparse-checkout patterns to see if a path is within
the sparse-checkout cone. Create some helper methods that help
initialize the patterns and check for pattern matching to make this
easier.

The existing callers of commands like get_sparse_checkout_patterns() use
a custom 'struct pattern_list' that is not necessarily the one in the
'struct index_state', so there are not many previous uses that could
adopt these helpers. There are just two in builtin/add.c and
sparse-index.c that can use path_in_sparse_checkout().

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 builtin/add.c  |  8 ++------
 dir.c          | 33 +++++++++++++++++++++++++++++++++
 dir.h          |  6 ++++++
 sparse-index.c | 12 +++---------
 4 files changed, 44 insertions(+), 15 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 17528e8f922..f675bdeae4a 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 	int flags = REFRESH_IGNORE_SKIP_WORKTREE |
 		    (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);
-	struct pattern_list pl = { 0 };
-	int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);
 
 	seen = xcalloc(pathspec->nr, 1);
 	refresh_index(&the_index, flags, pathspec, seen,
@@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	for (i = 0; i < pathspec->nr; i++) {
 		if (!seen[i]) {
 			const char *path = pathspec->items[i].original;
-			int dtype = DT_REG;
 
 			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
-			    (sparse_checkout_enabled &&
-			     !path_matches_pattern_list(path, strlen(path), NULL,
-							&dtype, &pl, &the_index))) {
+			    (core_apply_sparse_checkout &&
+			     path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec->items[i].original);
 			} else {
diff --git a/dir.c b/dir.c
index 03c4d212672..6fd4f2e0f27 100644
--- a/dir.c
+++ b/dir.c
@@ -1439,6 +1439,39 @@ done:
 	return result;
 }
 
+int init_sparse_checkout_patterns(struct index_state *istate)
+{
+	if (!core_apply_sparse_checkout ||
+	    istate->sparse_checkout_patterns)
+		return 0;
+
+	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
+
+	if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {
+		FREE_AND_NULL(istate->sparse_checkout_patterns);
+		return -1;
+	}
+
+	return 0;
+}
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate)
+{
+	const char *base;
+	int dtype = DT_REG;
+	init_sparse_checkout_patterns(istate);
+
+	if (!istate->sparse_checkout_patterns)
+		return MATCHED;
+
+	base = strrchr(path, '/');
+	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
+					 &dtype,
+					 istate->sparse_checkout_patterns,
+					 istate) > 0;
+}
+
 static struct path_pattern *last_matching_pattern_from_lists(
 		struct dir_struct *dir, struct index_state *istate,
 		const char *pathname, int pathlen,
diff --git a/dir.h b/dir.h
index b3e1a54a971..b899ee43d81 100644
--- a/dir.h
+++ b/dir.h
@@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,
 				const char *basename, int *dtype,
 				struct pattern_list *pl,
 				struct index_state *istate);
+
+int init_sparse_checkout_patterns(struct index_state *state);
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate);
+
 struct dir_entry *dir_add_ignored(struct dir_struct *dir,
 				  struct index_state *istate,
 				  const char *pathname, int len);
diff --git a/sparse-index.c b/sparse-index.c
index b6e90417556..2efc9fd4910 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,
 	int i, can_convert = 1;
 	int start_converted = num_converted;
 	enum pattern_match_result match;
-	int dtype = DT_UNKNOWN;
 	struct strbuf child_path = STRBUF_INIT;
-	struct pattern_list *pl = istate->sparse_checkout_patterns;
 
 	/*
 	 * Is the current path outside of the sparse cone?
 	 * Then check if the region can be replaced by a sparse
 	 * directory entry (everything is sparse and merged).
 	 */
-	match = path_matches_pattern_list(ct_path, ct_pathlen,
-					  NULL, &dtype, pl, istate);
+	match = path_in_sparse_checkout(ct_path, istate);
 	if (match != NOT_MATCHED)
 		can_convert = 0;
 
@@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo->settings.sparse_index)
 		return 0;
 
-	if (!istate->sparse_checkout_patterns) {
-		istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
-		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
-			return 0;
-	}
+	if (init_sparse_checkout_patterns(istate) < 0)
+		return 0;
 
 	/*
 	 * We need cone-mode patterns to use sparse-index. If a user edits
-- 
gitgitgadget

Re: [PATCH v3 5/8] sparse-checkout: create helper methods

[Johannes Schindelin]

Hi Stolee,

On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> As we integrate the sparse index into more builtins, we occasionally
> need to check the sparse-checkout patterns to see if a path is within
> the sparse-checkout cone. Create some helper methods that help
> initialize the patterns and check for pattern matching to make this
> easier.
>
> The existing callers of commands like get_sparse_checkout_patterns() use
> a custom 'struct pattern_list' that is not necessarily the one in the
> 'struct index_state', so there are not many previous uses that could
> adopt these helpers. There are just two in builtin/add.c and
> sparse-index.c that can use path_in_sparse_checkout().

Very good!

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  builtin/add.c  |  8 ++------
>  dir.c          | 33 +++++++++++++++++++++++++++++++++
>  dir.h          |  6 ++++++
>  sparse-index.c | 12 +++---------
>  4 files changed, 44 insertions(+), 15 deletions(-)
>
> diff --git a/builtin/add.c b/builtin/add.c
> index 17528e8f922..f675bdeae4a 100644
> --- a/builtin/add.c
> +++ b/builtin/add.c
> @@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)
>  	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
>  	int flags = REFRESH_IGNORE_SKIP_WORKTREE |
>  		    (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);
> -	struct pattern_list pl = { 0 };
> -	int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);
>
>  	seen = xcalloc(pathspec->nr, 1);
>  	refresh_index(&the_index, flags, pathspec, seen,
> @@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)
>  	for (i = 0; i < pathspec->nr; i++) {
>  		if (!seen[i]) {
>  			const char *path = pathspec->items[i].original;
> -			int dtype = DT_REG;
>
>  			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
> -			    (sparse_checkout_enabled &&
> -			     !path_matches_pattern_list(path, strlen(path), NULL,
> -							&dtype, &pl, &the_index))) {
> +			    (core_apply_sparse_checkout &&

Do we need to test for `core_apply_sparse_checkout` here? Or does the `if
(!istate->sparse_checkout_patterns) return MATCHED;` early return in
`path_in_sparse_checkout()` suffice to catch this?

The remainder of the patch looks good to me.

Thank you,
Dscho

> +			     path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {
>  				string_list_append(&only_match_skip_worktree,
>  						   pathspec->items[i].original);
>  			} else {
> diff --git a/dir.c b/dir.c
> index 03c4d212672..6fd4f2e0f27 100644
> --- a/dir.c
> +++ b/dir.c
> @@ -1439,6 +1439,39 @@ done:
>  	return result;
>  }
>
> +int init_sparse_checkout_patterns(struct index_state *istate)
> +{
> +	if (!core_apply_sparse_checkout ||
> +	    istate->sparse_checkout_patterns)
> +		return 0;
> +
> +	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
> +
> +	if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {
> +		FREE_AND_NULL(istate->sparse_checkout_patterns);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +int path_in_sparse_checkout(const char *path,
> +			    struct index_state *istate)
> +{
> +	const char *base;
> +	int dtype = DT_REG;
> +	init_sparse_checkout_patterns(istate);
> +
> +	if (!istate->sparse_checkout_patterns)
> +		return MATCHED;
> +
> +	base = strrchr(path, '/');
> +	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
> +					 &dtype,
> +					 istate->sparse_checkout_patterns,
> +					 istate) > 0;
> +}
> +
>  static struct path_pattern *last_matching_pattern_from_lists(
>  		struct dir_struct *dir, struct index_state *istate,
>  		const char *pathname, int pathlen,
> diff --git a/dir.h b/dir.h
> index b3e1a54a971..b899ee43d81 100644
> --- a/dir.h
> +++ b/dir.h
> @@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,
>  				const char *basename, int *dtype,
>  				struct pattern_list *pl,
>  				struct index_state *istate);
> +
> +int init_sparse_checkout_patterns(struct index_state *state);
> +
> +int path_in_sparse_checkout(const char *path,
> +			    struct index_state *istate);
> +
>  struct dir_entry *dir_add_ignored(struct dir_struct *dir,
>  				  struct index_state *istate,
>  				  const char *pathname, int len);
> diff --git a/sparse-index.c b/sparse-index.c
> index b6e90417556..2efc9fd4910 100644
> --- a/sparse-index.c
> +++ b/sparse-index.c
> @@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,
>  	int i, can_convert = 1;
>  	int start_converted = num_converted;
>  	enum pattern_match_result match;
> -	int dtype = DT_UNKNOWN;
>  	struct strbuf child_path = STRBUF_INIT;
> -	struct pattern_list *pl = istate->sparse_checkout_patterns;
>
>  	/*
>  	 * Is the current path outside of the sparse cone?
>  	 * Then check if the region can be replaced by a sparse
>  	 * directory entry (everything is sparse and merged).
>  	 */
> -	match = path_matches_pattern_list(ct_path, ct_pathlen,
> -					  NULL, &dtype, pl, istate);
> +	match = path_in_sparse_checkout(ct_path, istate);
>  	if (match != NOT_MATCHED)
>  		can_convert = 0;
>
> @@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)
>  	if (!istate->repo->settings.sparse_index)
>  		return 0;
>
> -	if (!istate->sparse_checkout_patterns) {
> -		istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
> -		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
> -			return 0;
> -	}
> +	if (init_sparse_checkout_patterns(istate) < 0)
> +		return 0;
>
>  	/*
>  	 * We need cone-mode patterns to use sparse-index. If a user edits
> --
> gitgitgadget
>
>

Re: [PATCH v3 5/8] sparse-checkout: create helper methods

[Derrick Stolee]

On 8/19/2021 4:07 AM, Johannes Schindelin wrote:
...
>>  			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
>> -			    (sparse_checkout_enabled &&
>> -			     !path_matches_pattern_list(path, strlen(path), NULL,
>> -							&dtype, &pl, &the_index))) {
>> +			    (core_apply_sparse_checkout &&
> 
> Do we need to test for `core_apply_sparse_checkout` here? Or does the `if
> (!istate->sparse_checkout_patterns) return MATCHED;` early return in
> `path_in_sparse_checkout()` suffice to catch this?
> 
> The remainder of the patch looks good to me.
> 
> Thank you,
> Dscho
> 
>> +			     path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {

Thank you for pointing out this. This is actually a stale change from an earlier
version where path_in_sparse_checkout returned an 'enum pattern_match_result'
but now casts down to an 'int', meaning '0' is not in the sparse-checkout and '1'
is that it _is_ in the sparse-checkout.

>> +int path_in_sparse_checkout(const char *path,
>> +			    struct index_state *istate)
>> +{
>> +	const char *base;
>> +	int dtype = DT_REG;
>> +	init_sparse_checkout_patterns(istate);
>> +
>> +	if (!istate->sparse_checkout_patterns)
>> +		return MATCHED;

So here, if we do not have sparse-checkout patterns (for example, if
core_apply_sparse_checkout is false), then this returns MATCHED (== 1).

To be extra clear, this should just be 'return 1;'.

>> +	base = strrchr(path, '/');
>> +	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
>> +					 &dtype,
>> +					 istate->sparse_checkout_patterns,
>> +					 istate) > 0;

Here, we are selecting the portion of 'enum pattern_match_result' that
we care about (MATCHED and MATCHED_RECURSIVE). The UNMATCHED (0) and
UNDECIDED (-1) are the other possibilities, but file paths will not
return UNDECIDED, that is instead for directories in non-cone mode
patterns.

Thanks,
-Stolee

[PATCH v3 6/8] attr: be careful about sparse directories

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 attr.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/attr.c b/attr.c
index d029e681f28..a1009f78029 100644
--- a/attr.c
+++ b/attr.c
@@ -14,6 +14,7 @@
 #include "utf8.h"
 #include "quote.h"
 #include "thread-utils.h"
+#include "dir.h"
 
 const char git_attr__true[] = "(builtin)true";
 const char git_attr__false[] = "\0(builtin)false";
@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
 	if (!istate)
 		return NULL;
 
+	/*
+	 * In the case of cone-mode sparse-checkout, getting the
+	 * .gitattributes file from a directory is meaningless: all
+	 * contained paths will be sparse if the .gitattributes is also
+	 * sparse. In the case of a sparse index, it is critical that we
+	 * don't go looking for one as it will expand the index.
+	 */
+	init_sparse_checkout_patterns(istate);
+	if (istate->sparse_checkout_patterns &&
+	    istate->sparse_checkout_patterns->use_cone_patterns &&
+	    path_in_sparse_checkout(path, istate) == NOT_MATCHED)
+		return NULL;
+
 	buf = read_blob_data_from_index(istate, path, NULL);
 	if (!buf)
 		return NULL;
-- 
gitgitgadget

Re: [PATCH v3 6/8] attr: be careful about sparse directories

[Johannes Schindelin]

Hi Stolee,

On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  attr.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/attr.c b/attr.c
> index d029e681f28..a1009f78029 100644
> --- a/attr.c
> +++ b/attr.c
> @@ -14,6 +14,7 @@
>  #include "utf8.h"
>  #include "quote.h"
>  #include "thread-utils.h"
> +#include "dir.h"
>
>  const char git_attr__true[] = "(builtin)true";
>  const char git_attr__false[] = "\0(builtin)false";
> @@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
>  	if (!istate)
>  		return NULL;
>
> +	/*
> +	 * In the case of cone-mode sparse-checkout, getting the
> +	 * .gitattributes file from a directory is meaningless: all
> +	 * contained paths will be sparse if the .gitattributes is also
> +	 * sparse. In the case of a sparse index, it is critical that we
> +	 * don't go looking for one as it will expand the index.
> +	 */
> +	init_sparse_checkout_patterns(istate);
At first I thought that `init_sparse_checkout_patterns()` is called by
`path_in_sparse_checkout()` below, and therefore would not be necessary.

But it is! Without it, we have no way to test whether `use_cone_patterns`
is set, because, well, it gets set by `init_sparse_checkout_patterns()`.

Would it therefore make sense to refactor the code to have a
`path_in_sparse_checkout_cone()` function? Or add a flag
`only_in_cone_mode` as function parameter to `path_in_sparse_checkout()`?

Ciao,
Dscho

> +	if (istate->sparse_checkout_patterns &&
> +	    istate->sparse_checkout_patterns->use_cone_patterns &&
> +	    path_in_sparse_checkout(path, istate) == NOT_MATCHED)

> +		return NULL;
> +
>  	buf = read_blob_data_from_index(istate, path, NULL);
>  	if (!buf)
>  		return NULL;
> --
> gitgitgadget
>
>

Re: [PATCH v3 6/8] attr: be careful about sparse directories

[Derrick Stolee]

On 8/19/2021 4:11 AM, Johannes Schindelin wrote:
> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
>> +	/*
>> +	 * In the case of cone-mode sparse-checkout, getting the
>> +	 * .gitattributes file from a directory is meaningless: all
>> +	 * contained paths will be sparse if the .gitattributes is also
>> +	 * sparse. In the case of a sparse index, it is critical that we
>> +	 * don't go looking for one as it will expand the index.
>> +	 */
>> +	init_sparse_checkout_patterns(istate);
> At first I thought that `init_sparse_checkout_patterns()` is called by
> `path_in_sparse_checkout()` below, and therefore would not be necessary.
> 
> But it is! Without it, we have no way to test whether `use_cone_patterns`
> is set, because, well, it gets set by `init_sparse_checkout_patterns()`.
> 
> Would it therefore make sense to refactor the code to have a
> `path_in_sparse_checkout_cone()` function? Or add a flag
> `only_in_cone_mode` as function parameter to `path_in_sparse_checkout()`?

One way to clean this up as-is would be to replace
 
>> +	if (istate->sparse_checkout_patterns &&
>> +	    istate->sparse_checkout_patterns->use_cone_patterns &&
>> +	    path_in_sparse_checkout(path, istate) == NOT_MATCHED)

with

	if (!path_in_sparse_checkout(path, istate) &&
	    istate->sparse_checkout_patterns->use_cone_patterns)

to get a similar effect. However, it creates wasted pattern-match
checks when not in cone-mode, so you are probably right that a
path_in_cone_mode_sparse_checkout() would be best to short-circuit
in the non-cone-mode case.

This special casing should be rare, so I don't think it is worth
adding a flag to path_in_sparse_checkout() and making those call
sites more complicated.

Thanks,
-Stolee

Re: [PATCH v3 6/8] attr: be careful about sparse directories

[Elijah Newren]

On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  attr.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/attr.c b/attr.c
> index d029e681f28..a1009f78029 100644
> --- a/attr.c
> +++ b/attr.c
> @@ -14,6 +14,7 @@
>  #include "utf8.h"
>  #include "quote.h"
>  #include "thread-utils.h"
> +#include "dir.h"
>
>  const char git_attr__true[] = "(builtin)true";
>  const char git_attr__false[] = "\0(builtin)false";
> @@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
>         if (!istate)
>                 return NULL;
>
> +       /*
> +        * In the case of cone-mode sparse-checkout, getting the
> +        * .gitattributes file from a directory is meaningless: all
> +        * contained paths will be sparse if the .gitattributes is also
> +        * sparse. In the case of a sparse index, it is critical that we
> +        * don't go looking for one as it will expand the index.
> +        */

"all contained paths will be sparse if the .gitattributes is also sparse"?

Do you mean something more like "the .gitattributes only applies for
files under the given directory, and if the directory is sparse, then
neither the .gitattributes file or any other file under that directory
will be present" ?

Also, out of curiosity, I was suggesting in the past we do something
like this for .gitignore files, for the same reason.  Do we have such
logic in place, or is that another area of the code that hasn't been
handled yet?

> +       init_sparse_checkout_patterns(istate);
> +       if (istate->sparse_checkout_patterns &&
> +           istate->sparse_checkout_patterns->use_cone_patterns &&
> +           path_in_sparse_checkout(path, istate) == NOT_MATCHED)
> +               return NULL;
> +
>         buf = read_blob_data_from_index(istate, path, NULL);
>         if (!buf)
>                 return NULL;
> --
> gitgitgadget

Though the code appears correct, I too am curious about the questions
Dscho asked about the code in this patch.

Re: [PATCH v3 6/8] attr: be careful about sparse directories

[Derrick Stolee]

On 8/19/2021 4:53 PM, Elijah Newren wrote:
> On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>> +       /*
>> +        * In the case of cone-mode sparse-checkout, getting the
>> +        * .gitattributes file from a directory is meaningless: all
>> +        * contained paths will be sparse if the .gitattributes is also
>> +        * sparse. In the case of a sparse index, it is critical that we
>> +        * don't go looking for one as it will expand the index.
>> +        */
> 
> "all contained paths will be sparse if the .gitattributes is also sparse"?
> 
> Do you mean something more like "the .gitattributes only applies for
> files under the given directory, and if the directory is sparse, then
> neither the .gitattributes file or any other file under that directory
> will be present" ?

Yes, you understand correctly and explain it better. Thanks.
 
> Also, out of curiosity, I was suggesting in the past we do something
> like this for .gitignore files, for the same reason.  Do we have such
> logic in place, or is that another area of the code that hasn't been
> handled yet?

I don't believe this has been handled. It definitely is less obvious
what to do there, because the point of .gitignore is to skip files that
exist in the working tree even if Git didn't put them there. Meanwhile,
.gitattributes is about how Git writes tracked files, but Git doesn't
write sparse tracked files.

> Though the code appears correct, I too am curious about the questions
> Dscho asked about the code in this patch.
 
Thanks,
-Stolee

Re: [PATCH v3 6/8] attr: be careful about sparse directories

[Elijah Newren]

On Fri, Aug 20, 2021 at 8:39 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/19/2021 4:53 PM, Elijah Newren wrote:
> > On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget
> > <gitgitgadget@gmail.com> wrote:
> >> +       /*
> >> +        * In the case of cone-mode sparse-checkout, getting the
> >> +        * .gitattributes file from a directory is meaningless: all
> >> +        * contained paths will be sparse if the .gitattributes is also
> >> +        * sparse. In the case of a sparse index, it is critical that we
> >> +        * don't go looking for one as it will expand the index.
> >> +        */
> >
> > "all contained paths will be sparse if the .gitattributes is also sparse"?
> >
> > Do you mean something more like "the .gitattributes only applies for
> > files under the given directory, and if the directory is sparse, then
> > neither the .gitattributes file or any other file under that directory
> > will be present" ?
>
> Yes, you understand correctly and explain it better. Thanks.
>
> > Also, out of curiosity, I was suggesting in the past we do something
> > like this for .gitignore files, for the same reason.  Do we have such
> > logic in place, or is that another area of the code that hasn't been
> > handled yet?
>
> I don't believe this has been handled. It definitely is less obvious
> what to do there, because the point of .gitignore is to skip files that
> exist in the working tree even if Git didn't put them there. Meanwhile,
> .gitattributes is about how Git writes tracked files, but Git doesn't
> write sparse tracked files.

Well, one advantage of deleting ignored files in sparse directories
when we sparsify, is that we know if any files are left, they are all
untracked and not ignored.  So we don't need to load the .gitignore
file for those sparse directories.

Sure, there's a small edge case of users adding new untracked files
that would have matched the .gitignore file, but it's also clear that
they removed the .gitignore file when they sparsified, so I don't see
a problem in reporting it as untracked while that directory was
as-sparsified-away-as-possible.

[PATCH v3 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
environment variable or the "index.sparse" config setting before
converting the index to a sparse one. This is for ease of use since all
current consumers are preparing to compress the index before writing it
to disk. If these settings are not enabled, then convert_to_sparse()
silently returns without doing anything.

We will add a consumer in the next change that wants to use the sparse
index as an in-memory data structure, regardless of whether the on-disk
format should be sparse.

To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip
these config checks when enabled. All current consumers are modified to
pass '0' in the new 'flags' parameter.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 read-cache.c   |  4 ++--
 sparse-index.c | 30 ++++++++++++++++--------------
 sparse-index.h |  3 ++-
 3 files changed, 20 insertions(+), 17 deletions(-)

diff --git a/read-cache.c b/read-cache.c
index 9048ef9e905..f5d4385c408 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
 	int ret;
 	int was_full = !istate->sparse_index;
 
-	ret = convert_to_sparse(istate);
+	ret = convert_to_sparse(istate, 0);
 
 	if (ret) {
 		warning(_("failed to convert to a sparse-index"));
@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,
 	int ret, was_full = !istate->sparse_index;
 
 	move_cache_to_base_index(istate);
-	convert_to_sparse(istate);
+	convert_to_sparse(istate, 0);
 
 	trace2_region_enter_printf("index", "shared/do_write_index",
 				   the_repository, "%s", get_tempfile_path(*temp));
diff --git a/sparse-index.c b/sparse-index.c
index 2efc9fd4910..532fd11787e 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -124,7 +124,7 @@ static int index_has_unmerged_entries(struct index_state *istate)
 	return 0;
 }
 
-int convert_to_sparse(struct index_state *istate)
+int convert_to_sparse(struct index_state *istate, int flags)
 {
 	int test_env;
 
@@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo)
 		istate->repo = the_repository;
 
-	/*
-	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
-	 * index.sparse config variable to be on.
-	 */
-	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
-	if (test_env >= 0)
-		set_sparse_index_config(istate->repo, test_env);
+	if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
+		/*
+		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
+		 * index.sparse config variable to be on.
+		 */
+		test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
+		if (test_env >= 0)
+			set_sparse_index_config(istate->repo, test_env);
 
-	/*
-	 * Only convert to sparse if index.sparse is set.
-	 */
-	prepare_repo_settings(istate->repo);
-	if (!istate->repo->settings.sparse_index)
-		return 0;
+		/*
+		 * Only convert to sparse if index.sparse is set.
+		 */
+		prepare_repo_settings(istate->repo);
+		if (!istate->repo->settings.sparse_index)
+			return 0;
+	}
 
 	if (init_sparse_checkout_patterns(istate) < 0)
 		return 0;
diff --git a/sparse-index.h b/sparse-index.h
index 1115a0d7dd9..475f4f0f8da 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -2,7 +2,8 @@
 #define SPARSE_INDEX_H__
 
 struct index_state;
-int convert_to_sparse(struct index_state *istate);
+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)
+int convert_to_sparse(struct index_state *istate, int flags);
 
 /*
  * Some places in the codebase expect to search for a specific path.
-- 
gitgitgadget

Re: [PATCH v3 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag

[Derrick Stolee]

On 8/17/2021 9:23 AM, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
> environment variable or the "index.sparse" config setting before
> converting the index to a sparse one. This is for ease of use since all
> current consumers are preparing to compress the index before writing it
> to disk. If these settings are not enabled, then convert_to_sparse()
> silently returns without doing anything.
> 
> We will add a consumer in the next change that wants to use the sparse
> index as an in-memory data structure, regardless of whether the on-disk
> format should be sparse.
> 
> To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip
> these config checks when enabled. All current consumers are modified to
> pass '0' in the new 'flags' parameter.

...

> -int convert_to_sparse(struct index_state *istate)
> +int convert_to_sparse(struct index_state *istate, int flags)
>  {
>  	int test_env;
>  
> @@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)
>  	if (!istate->repo)
>  		istate->repo = the_repository;
>  

Above this hunk, we fail automatically if the index has a split index.

The purpose of this flag is instead to say "convert to sparse for the
purpose of in-memory computations, not for writing to disk". For such
a case, we could move the split index check to be within the hunk
below. It would be appropriate to rename the flag to something like
SPARSE_INDEX_IN_MEMORY or SPARSE_INDEX_NO_DISK_WRITE to make the
intention more clear.

Thanks to SZEDER for pointing out this failure. I will fix it in the
next version.

> -	/*
> -	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
> -	 * index.sparse config variable to be on.
> -	 */
> -	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
> -	if (test_env >= 0)
> -		set_sparse_index_config(istate->repo, test_env);
> +	if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
> +		/*
> +		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
> +		 * index.sparse config variable to be on.
> +		 */
> +		test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
> +		if (test_env >= 0)
> +			set_sparse_index_config(istate->repo, test_env);
>  
> -	/*
> -	 * Only convert to sparse if index.sparse is set.
> -	 */
> -	prepare_repo_settings(istate->repo);
> -	if (!istate->repo->settings.sparse_index)
> -		return 0;
> +		/*
> +		 * Only convert to sparse if index.sparse is set.
> +		 */
> +		prepare_repo_settings(istate->repo);
> +		if (!istate->repo->settings.sparse_index)
> +			return 0;
> +	}

If you want to try this, here is a diff that can help:

--- >8 ---

diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index b06c8f885ac..c6a512a2107 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -131,7 +131,7 @@ static void clean_tracked_sparse_directories(struct repository *r)
 		 * prevents us from converting to a sparse index, then do
 		 * not try deleting files.
 		 */
-		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
+		if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
 			return;
 		was_full = 1;
 	}
diff --git a/sparse-index.c b/sparse-index.c
index e0a854f9fc3..267503b21fa 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -128,14 +128,20 @@ int convert_to_sparse(struct index_state *istate, int flags)
 {
 	int test_env;
 
-	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
+	if (istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
 	if (!istate->repo)
 		istate->repo = the_repository;
 
-	if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
+	if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {
+		/*
+		 * The sparse index is not integrated with a split index.
+		 */
+		if (istate->sparse_index)
+			return 0;
+
 		/*
 		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
 		 * index.sparse config variable to be on.
diff --git a/sparse-index.h b/sparse-index.h
index 475f4f0f8da..9f3d7bc7faf 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -2,7 +2,7 @@
 #define SPARSE_INDEX_H__
 
 struct index_state;
-#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)
+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)
 int convert_to_sparse(struct index_state *istate, int flags);
 
 /*


--- >8 ---

Thanks,
-Stolee

[PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When changing the scope of a sparse-checkout using cone mode, we might
have some tracked directories go out of scope. The current logic removes
the tracked files from within those directories, but leaves the ignored
files within those directories. This is a bit unexpected to users who
have given input to Git saying they don't need those directories
anymore.

This is something that is new to the cone mode pattern type: the user
has explicitly said "I want these directories and _not_ those
directories." The typical sparse-checkout patterns more generally apply
to "I want files with with these patterns" so it is natural to leave
ignored files as they are. This focus on directories in cone mode
provides us an opportunity to change the behavior.

Leaving these ignored files in the sparse directories makes it
impossible to gain performance benefits in the sparse index. When we
track into these directories, we need to know if the files are ignored
or not, which might depend on the _tracked_ .gitignore file(s) within
the sparse directory. This depends on the indexed version of the file,
so the sparse directory must be expanded.

By deleting the sparse directories when changing scope (or running 'git
sparse-checkout reapply') we regain these performance benefits as if the
repository was in a clean state.

Since these ignored files are frequently build output or helper files
from IDEs, the users should not need the files now that the tracked
files are removed. If the tracked files reappear, then they will have
newer timestamps than the build artifacts, so the artifacts will need to
be regenerated anyway.

Use the sparse-index as a data structure in order to find the sparse
directories that can be safely deleted. Re-expand the index to a full
one if it was full before.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt |  8 +++
 builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 3 files changed, 162 insertions(+)

diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index fdcf43f87cb..f9022b9d555 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the
 'git sparse-checkout set' command to reflect the expected cone in the working
 directory.
 
+The cone mode sparse-checkout patterns will also remove ignored files that
+are not within the sparse-checkout definition. This is important behavior
+to preserve the performance of the sparse index, but also matches that
+cone mode patterns care about directories, not files. If there exist files
+that are untracked and not ignored, then Git will not delete files within
+that directory other than the tracked files that are now out of scope.
+These files should be removed manually to ensure Git can behave optimally.
+
 
 SUBMODULES
 ----------
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 8ba9f13787b..b06c8f885ac 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)
 	return 0;
 }
 
+static void clean_tracked_sparse_directories(struct repository *r)
+{
+	int i, was_full = 0;
+	struct strbuf path = STRBUF_INIT;
+	size_t pathlen;
+	struct string_list_item *item;
+	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
+
+	/*
+	 * If we are not using cone mode patterns, then we cannot
+	 * delete directories outside of the sparse cone.
+	 */
+	if (!r || !r->index || !r->worktree)
+		return;
+	init_sparse_checkout_patterns(r->index);
+	if (!r->index->sparse_checkout_patterns ||
+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
+		return;
+
+	/*
+	 * Use the sparse index as a data structure to assist finding
+	 * directories that are safe to delete. This conversion to a
+	 * sparse index will not delete directories that contain
+	 * conflicted entries or submodules.
+	 */
+	if (!r->index->sparse_index) {
+		/*
+		 * If something, such as a merge conflict or other concern,
+		 * prevents us from converting to a sparse index, then do
+		 * not try deleting files.
+		 */
+		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
+			return;
+		was_full = 1;
+	}
+
+	strbuf_addstr(&path, r->worktree);
+	strbuf_complete(&path, '/');
+	pathlen = path.len;
+
+	/*
+	 * Collect directories that have gone out of scope but also
+	 * exist on disk, so there is some work to be done. We need to
+	 * store the entries in a list before exploring, since that might
+	 * expand the sparse-index again.
+	 */
+	for (i = 0; i < r->index->cache_nr; i++) {
+		struct cache_entry *ce = r->index->cache[i];
+
+		if (S_ISSPARSEDIR(ce->ce_mode) &&
+		    repo_file_exists(r, ce->name))
+			string_list_append(&sparse_dirs, ce->name);
+	}
+
+	for_each_string_list_item(item, &sparse_dirs) {
+		struct dir_struct dir = DIR_INIT;
+		struct pathspec p = { 0 };
+		struct strvec s = STRVEC_INIT;
+
+		strbuf_setlen(&path, pathlen);
+		strbuf_addstr(&path, item->string);
+
+		dir.flags |= DIR_SHOW_IGNORED_TOO;
+
+		setup_standard_excludes(&dir);
+		strvec_push(&s, path.buf);
+
+		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
+		fill_directory(&dir, r->index, &p);
+
+		if (dir.nr) {
+			warning(_("directory '%s' contains untracked files,"
+				  " but is not in the sparse-checkout cone"),
+				item->string);
+		} else if (remove_dir_recursively(&path, 0)) {
+			/*
+			 * Removal is "best effort". If something blocks
+			 * the deletion, then continue with a warning.
+			 */
+			warning(_("failed to remove directory '%s'"),
+				item->string);
+		}
+
+		dir_clear(&dir);
+	}
+
+	string_list_clear(&sparse_dirs, 0);
+	strbuf_release(&path);
+
+	if (was_full)
+		ensure_full_index(r->index);
+}
+
 static int update_working_directory(struct pattern_list *pl)
 {
 	enum update_sparsity_result result;
@@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)
 	else
 		rollback_lock_file(&lock_file);
 
+	clean_tracked_sparse_directories(r);
+
 	r->index->sparse_checkout_patterns = NULL;
 	return result;
 }
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..71236981e64 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
 	check_files repo/deep a deeper1
 '
 
+test_expect_success 'cone mode clears ignored subdirectories' '
+	rm repo/.git/info/sparse-checkout &&
+
+	git -C repo sparse-checkout init --cone &&
+	git -C repo sparse-checkout set deep/deeper1 &&
+
+	cat >repo/.gitignore <<-\EOF &&
+	obj/
+	*.o
+	EOF
+
+	git -C repo add .gitignore &&
+	git -C repo commit -m ".gitignore" &&
+
+	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
+	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
+		    deep/deeper2/obj/a deep/deeper2/file.o file.o
+	do
+		echo ignored >repo/$file || return 1
+	done &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout reapply &&
+	test_path_is_missing repo/folder1 &&
+	test_path_is_missing repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout set deep/deeper2 &&
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	>repo/deep/deeper2/ignored.o &&
+	>repo/deep/deeper2/untracked &&
+
+	# When an untracked file is in the way, all untracked files
+	# (even ignored files) are preserved.
+	git -C repo sparse-checkout set folder1 2>err &&
+	grep "contains untracked files" err &&
+	test_path_is_file repo/deep/deeper2/ignored.o &&
+	test_path_is_file repo/deep/deeper2/untracked &&
+
+	# The rest of the cone matches expectation
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	echo "? deep/deeper2/untracked" >expect &&
+	test_cmp expect out
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Johannes Schindelin]

Hi Stolee,

On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> When changing the scope of a sparse-checkout using cone mode, we might
> have some tracked directories go out of scope. The current logic removes
> the tracked files from within those directories, but leaves the ignored
> files within those directories. This is a bit unexpected to users who
> have given input to Git saying they don't need those directories
> anymore.
>
> This is something that is new to the cone mode pattern type: the user
> has explicitly said "I want these directories and _not_ those
> directories." The typical sparse-checkout patterns more generally apply
> to "I want files with with these patterns" so it is natural to leave
> ignored files as they are. This focus on directories in cone mode
> provides us an opportunity to change the behavior.
>
> Leaving these ignored files in the sparse directories makes it
> impossible to gain performance benefits in the sparse index. When we
> track into these directories, we need to know if the files are ignored
> or not, which might depend on the _tracked_ .gitignore file(s) within
> the sparse directory. This depends on the indexed version of the file,
> so the sparse directory must be expanded.
>
> By deleting the sparse directories when changing scope (or running 'git
> sparse-checkout reapply') we regain these performance benefits as if the
> repository was in a clean state.
>
> Since these ignored files are frequently build output or helper files
> from IDEs, the users should not need the files now that the tracked
> files are removed. If the tracked files reappear, then they will have
> newer timestamps than the build artifacts, so the artifacts will need to
> be regenerated anyway.
>
> Use the sparse-index as a data structure in order to find the sparse
> directories that can be safely deleted. Re-expand the index to a full
> one if it was full before.

This description makes sense, and is easy to explain.

It does not cover the case where untracked files are found and the
directory is _not_ removed as a consequence, though. I would like to ask
to add this to the commit message, because it is kind of important.

The implementation of this behavior looks fine to me.

About this behavior itself: in my experience, the more tricky a feature is
to explain, the more likely the design should have been adjusted in the
first place. And I find myself struggling a little bit to explain what
files `git switch` touches in cone mode in addition to tracked files.

So I wonder whether an easier-to-explain behavior would be the following:
ignored files in directories that fell out of the sparse-checkout cone are
deleted. (Even if there are untracked, unignored files in the same
directory tree.)

This is different than what this patch implements: we would now have to
delete the ignored and out-of-cone files _also_ when there are untracked
files in the same directory, i.e. we could no longer use the sweet
`remove_dir_recursively()` call. Therefore, the implementation of what I
suggested would be much more complicated: you would have to enumerate the
ignored files and remove them individually.

Having said that, even after mulling over this behavior and sleeping over
it, I am unsure what the best way forward would be. Just because it is
easy to explain does not make it right.

It is tricky to decide mostly because "ignored" files are definitely not
always build output. Apart from VIM's temporary files, users like me
frequently write other files and/or directories that we simply do not want
to see tracked in Git. For example, I often test things in an `a1.c` file
that -- for convenience -- lives in the current worktree. Obviously I
don't want Git to track it, but I also don't want it to be deleted, so I
often add corresponding lines to `.git/info/exclude`. Likewise, I
sometimes download additional information related to what I am
implementing, and that also lives in the current worktree (but then, I
usually am too lazy to add an entry to `.git/info/exclude` in those
cases).

Now, I don't want to over-index on my own habits. There are so many users
out there, and most of them have different preferences from mine.

Which leaves me to wonder whether we need at least a flag to turn this
behavior on and off? Something like
`core.ignoredFilesInSparseConesArePrecious = true` (obviously with a
better, shorter name).

Ciao,
Dscho

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/git-sparse-checkout.txt |  8 +++
>  builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
>  t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
>  3 files changed, 162 insertions(+)
>
> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
> index fdcf43f87cb..f9022b9d555 100644
> --- a/Documentation/git-sparse-checkout.txt
> +++ b/Documentation/git-sparse-checkout.txt
> @@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the
>  'git sparse-checkout set' command to reflect the expected cone in the working
>  directory.
>
> +The cone mode sparse-checkout patterns will also remove ignored files that
> +are not within the sparse-checkout definition. This is important behavior
> +to preserve the performance of the sparse index, but also matches that
> +cone mode patterns care about directories, not files. If there exist files
> +that are untracked and not ignored, then Git will not delete files within
> +that directory other than the tracked files that are now out of scope.
> +These files should be removed manually to ensure Git can behave optimally.
> +
>
>  SUBMODULES
>  ----------
> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
> index 8ba9f13787b..b06c8f885ac 100644
> --- a/builtin/sparse-checkout.c
> +++ b/builtin/sparse-checkout.c
> @@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)
>  	return 0;
>  }
>
> +static void clean_tracked_sparse_directories(struct repository *r)
> +{
> +	int i, was_full = 0;
> +	struct strbuf path = STRBUF_INIT;
> +	size_t pathlen;
> +	struct string_list_item *item;
> +	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
> +
> +	/*
> +	 * If we are not using cone mode patterns, then we cannot
> +	 * delete directories outside of the sparse cone.
> +	 */
> +	if (!r || !r->index || !r->worktree)
> +		return;
> +	init_sparse_checkout_patterns(r->index);
> +	if (!r->index->sparse_checkout_patterns ||
> +	    !r->index->sparse_checkout_patterns->use_cone_patterns)
> +		return;
> +
> +	/*
> +	 * Use the sparse index as a data structure to assist finding
> +	 * directories that are safe to delete. This conversion to a
> +	 * sparse index will not delete directories that contain
> +	 * conflicted entries or submodules.
> +	 */
> +	if (!r->index->sparse_index) {
> +		/*
> +		 * If something, such as a merge conflict or other concern,
> +		 * prevents us from converting to a sparse index, then do
> +		 * not try deleting files.
> +		 */
> +		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
> +			return;
> +		was_full = 1;
> +	}
> +
> +	strbuf_addstr(&path, r->worktree);
> +	strbuf_complete(&path, '/');
> +	pathlen = path.len;
> +
> +	/*
> +	 * Collect directories that have gone out of scope but also
> +	 * exist on disk, so there is some work to be done. We need to
> +	 * store the entries in a list before exploring, since that might
> +	 * expand the sparse-index again.
> +	 */
> +	for (i = 0; i < r->index->cache_nr; i++) {
> +		struct cache_entry *ce = r->index->cache[i];
> +
> +		if (S_ISSPARSEDIR(ce->ce_mode) &&
> +		    repo_file_exists(r, ce->name))
> +			string_list_append(&sparse_dirs, ce->name);
> +	}
> +
> +	for_each_string_list_item(item, &sparse_dirs) {
> +		struct dir_struct dir = DIR_INIT;
> +		struct pathspec p = { 0 };
> +		struct strvec s = STRVEC_INIT;
> +
> +		strbuf_setlen(&path, pathlen);
> +		strbuf_addstr(&path, item->string);
> +
> +		dir.flags |= DIR_SHOW_IGNORED_TOO;
> +
> +		setup_standard_excludes(&dir);
> +		strvec_push(&s, path.buf);
> +
> +		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
> +		fill_directory(&dir, r->index, &p);
> +
> +		if (dir.nr) {
> +			warning(_("directory '%s' contains untracked files,"
> +				  " but is not in the sparse-checkout cone"),
> +				item->string);
> +		} else if (remove_dir_recursively(&path, 0)) {
> +			/*
> +			 * Removal is "best effort". If something blocks
> +			 * the deletion, then continue with a warning.
> +			 */
> +			warning(_("failed to remove directory '%s'"),
> +				item->string);
> +		}
> +
> +		dir_clear(&dir);
> +	}
> +
> +	string_list_clear(&sparse_dirs, 0);
> +	strbuf_release(&path);
> +
> +	if (was_full)
> +		ensure_full_index(r->index);
> +}
> +
>  static int update_working_directory(struct pattern_list *pl)
>  {
>  	enum update_sparsity_result result;
> @@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)
>  	else
>  		rollback_lock_file(&lock_file);
>
> +	clean_tracked_sparse_directories(r);
> +
>  	r->index->sparse_checkout_patterns = NULL;
>  	return result;
>  }
> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index 38fc8340f5c..71236981e64 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
>  	check_files repo/deep a deeper1
>  '
>
> +test_expect_success 'cone mode clears ignored subdirectories' '
> +	rm repo/.git/info/sparse-checkout &&
> +
> +	git -C repo sparse-checkout init --cone &&
> +	git -C repo sparse-checkout set deep/deeper1 &&
> +
> +	cat >repo/.gitignore <<-\EOF &&
> +	obj/
> +	*.o
> +	EOF
> +
> +	git -C repo add .gitignore &&
> +	git -C repo commit -m ".gitignore" &&
> +
> +	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
> +	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
> +		    deep/deeper2/obj/a deep/deeper2/file.o file.o
> +	do
> +		echo ignored >repo/$file || return 1
> +	done &&
> +
> +	git -C repo status --porcelain=v2 >out &&
> +	test_must_be_empty out &&
> +
> +	git -C repo sparse-checkout reapply &&
> +	test_path_is_missing repo/folder1 &&
> +	test_path_is_missing repo/deep/deeper2 &&
> +	test_path_is_dir repo/obj &&
> +	test_path_is_file repo/file.o &&
> +
> +	git -C repo status --porcelain=v2 >out &&
> +	test_must_be_empty out &&
> +
> +	git -C repo sparse-checkout set deep/deeper2 &&
> +	test_path_is_missing repo/deep/deeper1 &&
> +	test_path_is_dir repo/deep/deeper2 &&
> +	test_path_is_dir repo/obj &&
> +	test_path_is_file repo/file.o &&
> +
> +	>repo/deep/deeper2/ignored.o &&
> +	>repo/deep/deeper2/untracked &&
> +
> +	# When an untracked file is in the way, all untracked files
> +	# (even ignored files) are preserved.
> +	git -C repo sparse-checkout set folder1 2>err &&
> +	grep "contains untracked files" err &&
> +	test_path_is_file repo/deep/deeper2/ignored.o &&
> +	test_path_is_file repo/deep/deeper2/untracked &&
> +
> +	# The rest of the cone matches expectation
> +	test_path_is_missing repo/deep/deeper1 &&
> +	test_path_is_dir repo/obj &&
> +	test_path_is_file repo/file.o &&
> +
> +	git -C repo status --porcelain=v2 >out &&
> +	echo "? deep/deeper2/untracked" >expect &&
> +	test_cmp expect out
> +'
> +
>  test_done
> --
> gitgitgadget
>

Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee]

On 8/19/2021 4:48 AM, Johannes Schindelin wrote:
> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
> This description makes sense, and is easy to explain.
> 
> It does not cover the case where untracked files are found and the
> directory is _not_ removed as a consequence, though. I would like to ask
> to add this to the commit message, because it is kind of important.

Right. I should have modified the message from my earlier version when
the issues with untracked files came up.
 
> The implementation of this behavior looks fine to me.
> 
> About this behavior itself: in my experience, the more tricky a feature is
> to explain, the more likely the design should have been adjusted in the
> first place. And I find myself struggling a little bit to explain what
> files `git switch` touches in cone mode in addition to tracked files.

Keep in mind that 'git switch' does not change the sparse-checkout cone,
and the activity being described is something that would happen within
'git sparse-checkout set' or 'git sparse-checkout reapply'.

> So I wonder whether an easier-to-explain behavior would be the following:
> ignored files in directories that fell out of the sparse-checkout cone are
> deleted. (Even if there are untracked, unignored files in the same
> directory tree.)
> 
> This is different than what this patch implements: we would now have to
> delete the ignored and out-of-cone files _also_ when there are untracked
> files in the same directory, i.e. we could no longer use the sweet
> `remove_dir_recursively()` call. Therefore, the implementation of what I
> suggested would be much more complicated: you would have to enumerate the
> ignored files and remove them individually.

Outside of "it's harder to write that feature", perhaps I could convince
you that it is better to do nothing in the presence of untracked files.

If a user has an untracked file within a directory that is leaving the
sparse cone, then that means they were doing something in that space and
perhaps has unfinished work. By leaving the files on-disk, they have an
opportunity to revert the change to the sparse-checkout cone and continue
their work interrupted. This includes keeping things like build artifacts
(that are ignored) so they can incrementally build from that position.

The general thought I have here is: having untracked, not-ignored files
in a directory that is leaving the sparse-checkout cone is an unexpected
behavior, so we should do as little as possible in that scenario.

It also makes it more clear to the user what happened: "You had untracked
files here, so we left them alone" is easier to understand than "You had
untracked files here, so we deleted the ones that were ignored and kept
the rest."

> Having said that, even after mulling over this behavior and sleeping over
> it, I am unsure what the best way forward would be. Just because it is
> easy to explain does not make it right.

Of course, you already have a retort to my claim that "simpler is better",
but I'll just focus on the point that "simpler for the user to understand"
is a different point than "simpler to implement".

> Which leaves me to wonder whether we need at least a flag to turn this
> behavior on and off? Something like
> `core.ignoredFilesInSparseConesArePrecious = true` (obviously with a
> better, shorter name).

You are right that there are a lot of users out there, and they have
very different habits. Those habits are shaped by the way Git has worked
for a long time, so adding a way to go back to the previous behavior
would be good to have.

Thanks,
-Stolee

Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Elijah Newren]

On Fri, Aug 20, 2021 at 8:50 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/19/2021 4:48 AM, Johannes Schindelin wrote:
> > On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
> > This description makes sense, and is easy to explain.
> >
> > It does not cover the case where untracked files are found and the
> > directory is _not_ removed as a consequence, though. I would like to ask
> > to add this to the commit message, because it is kind of important.
>
> Right. I should have modified the message from my earlier version when
> the issues with untracked files came up.

:+1:

> > The implementation of this behavior looks fine to me.
> >
> > About this behavior itself: in my experience, the more tricky a feature is
> > to explain, the more likely the design should have been adjusted in the
> > first place. And I find myself struggling a little bit to explain what
> > files `git switch` touches in cone mode in addition to tracked files.
>
> Keep in mind that 'git switch' does not change the sparse-checkout cone,
> and the activity being described is something that would happen within
> 'git sparse-checkout set' or 'git sparse-checkout reapply'.

Good points.

> > So I wonder whether an easier-to-explain behavior would be the following:
> > ignored files in directories that fell out of the sparse-checkout cone are
> > deleted. (Even if there are untracked, unignored files in the same
> > directory tree.)
> >
> > This is different than what this patch implements: we would now have to
> > delete the ignored and out-of-cone files _also_ when there are untracked
> > files in the same directory, i.e. we could no longer use the sweet
> > `remove_dir_recursively()` call. Therefore, the implementation of what I
> > suggested would be much more complicated: you would have to enumerate the
> > ignored files and remove them individually.
>
> Outside of "it's harder to write that feature"

I don't think it is; see my other email.

> perhaps I could convince
> you that it is better to do nothing in the presence of untracked files.
>
> If a user has an untracked file within a directory that is leaving the
> sparse cone, then that means they were doing something in that space and
> perhaps has unfinished work. By leaving the files on-disk, they have an
> opportunity to revert the change to the sparse-checkout cone and continue
> their work interrupted. This includes keeping things like build artifacts
> (that are ignored) so they can incrementally build from that position.
>
> The general thought I have here is: having untracked, not-ignored files
> in a directory that is leaving the sparse-checkout cone is an unexpected
> behavior, so we should do as little as possible in that scenario.
>
> It also makes it more clear to the user what happened: "You had untracked
> files here, so we left them alone" is easier to understand than "You had
> untracked files here, so we deleted the ones that were ignored and kept
> the rest."

This explanation seems reasonable, but it certainly belongs in the
commit message.

However, doesn't it defeat the point of your removing the ignored
files?  Not only do you have directories full of files, but you won't
be able to have sparse directory entries due to the need to have
.gitignore files from underneath them.

Perhaps this is okay because we expect this to be an unusual case.
But, if so, do we want a more verbose warning (not with every
directory that fails to be deleted, but just at the end if there were
any), suggesting to the user that they clean up the relevant
directories and do a sparse-checkout reapply?  And, ultimately, once
the directory is gone, is that good enough to allow us to keep our
indexes sparse and fast, or are we looking at a huge amount of effort
spent on .gitignore files underneath sparse directories?

> > Having said that, even after mulling over this behavior and sleeping over
> > it, I am unsure what the best way forward would be. Just because it is
> > easy to explain does not make it right.
>
> Of course, you already have a retort to my claim that "simpler is better",
> but I'll just focus on the point that "simpler for the user to understand"
> is a different point than "simpler to implement".

I'm confused now.  Which behavior are you arguing is simpler for the
user to understand?

Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Elijah Newren]

On Thu, Aug 19, 2021 at 1:48 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Stolee,
>
> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:
>
> > From: Derrick Stolee <dstolee@microsoft.com>
> >
> > When changing the scope of a sparse-checkout using cone mode, we might
> > have some tracked directories go out of scope. The current logic removes
> > the tracked files from within those directories, but leaves the ignored
> > files within those directories. This is a bit unexpected to users who
> > have given input to Git saying they don't need those directories
> > anymore.
> >
> > This is something that is new to the cone mode pattern type: the user
> > has explicitly said "I want these directories and _not_ those
> > directories." The typical sparse-checkout patterns more generally apply
> > to "I want files with with these patterns" so it is natural to leave
> > ignored files as they are. This focus on directories in cone mode
> > provides us an opportunity to change the behavior.
> >
> > Leaving these ignored files in the sparse directories makes it
> > impossible to gain performance benefits in the sparse index. When we
> > track into these directories, we need to know if the files are ignored
> > or not, which might depend on the _tracked_ .gitignore file(s) within
> > the sparse directory. This depends on the indexed version of the file,
> > so the sparse directory must be expanded.
> >
> > By deleting the sparse directories when changing scope (or running 'git
> > sparse-checkout reapply') we regain these performance benefits as if the
> > repository was in a clean state.
> >
> > Since these ignored files are frequently build output or helper files
> > from IDEs, the users should not need the files now that the tracked
> > files are removed. If the tracked files reappear, then they will have
> > newer timestamps than the build artifacts, so the artifacts will need to
> > be regenerated anyway.
> >
> > Use the sparse-index as a data structure in order to find the sparse
> > directories that can be safely deleted. Re-expand the index to a full
> > one if it was full before.
>
> This description makes sense, and is easy to explain.
>
> It does not cover the case where untracked files are found and the
> directory is _not_ removed as a consequence, though. I would like to ask
> to add this to the commit message, because it is kind of important.
>
> The implementation of this behavior looks fine to me.
>
> About this behavior itself: in my experience, the more tricky a feature is
> to explain, the more likely the design should have been adjusted in the
> first place. And I find myself struggling a little bit to explain what
> files `git switch` touches in cone mode in addition to tracked files.

I share this concern.

> So I wonder whether an easier-to-explain behavior would be the following:
> ignored files in directories that fell out of the sparse-checkout cone are
> deleted. (Even if there are untracked, unignored files in the same
> directory tree.)
>
> This is different than what this patch implements: we would now have to
> delete the ignored and out-of-cone files _also_ when there are untracked
> files in the same directory, i.e. we could no longer use the sweet
> `remove_dir_recursively()` call. Therefore, the implementation of what I
> suggested would be much more complicated: you would have to enumerate the
> ignored files and remove them individually.

The ignored files are already enumerated by the fill_directory call;
you just need to iterate over the dir->ignored_nr entries in
dir->ignored.

> Having said that, even after mulling over this behavior and sleeping over
> it, I am unsure what the best way forward would be. Just because it is
> easy to explain does not make it right.
>
> It is tricky to decide mostly because "ignored" files are definitely not
> always build output. Apart from VIM's temporary files, users like me
> frequently write other files and/or directories that we simply do not want
> to see tracked in Git. For example, I often test things in an `a1.c` file
> that -- for convenience -- lives in the current worktree. Obviously I
> don't want Git to track it, but I also don't want it to be deleted, so I
> often add corresponding lines to `.git/info/exclude`. Likewise, I
> sometimes download additional information related to what I am
> implementing, and that also lives in the current worktree (but then, I
> usually am too lazy to add an entry to `.git/info/exclude` in those
> cases).

I do the same thing, and I know other users that do as well...but I
don't put such files in directories that are irrelevant to me.  I
create cruft files near other files that I'm working on, or in a
special directory of its own, but not under some directory that is
irrelevant to the areas I'm working on.

For reference, we implemented something like this in our `sparsify`
wrapper we have internally, where 'git clean -fdX <all sparse
directories>` is executed whenever folks sparsify.  (We have our own
tool and don't have users use sparse-checkout directly, because our
tool computes dependencies to determine which directories are needed.)
 I was really hesitant to add that cleaning behavior by default, and
just made it an option.  My colleagues tired of all the bug reports
about left-around directories and made it the default, waiting to hear
complaints.  We never got one.  It's been over a year.

> Now, I don't want to over-index on my own habits. There are so many users
> out there, and most of them have different preferences from mine.
>
> Which leaves me to wonder whether we need at least a flag to turn this
> behavior on and off? Something like
> `core.ignoredFilesInSparseConesArePrecious = true` (obviously with a
> better, shorter name).

That seems fine, but I suspect you're projecting the same concerns I
had, and it turns out much less useful than you'd expect (perhaps even
totally useless).

Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs

[Johannes Schindelin]

Hi Elijah,

On Fri, 20 Aug 2021, Elijah Newren wrote:

> On Thu, Aug 19, 2021 at 1:48 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > Having said that, even after mulling over this behavior and sleeping
> > over it, I am unsure what the best way forward would be. Just because
> > it is easy to explain does not make it right.
> >
> > It is tricky to decide mostly because "ignored" files are definitely
> > not always build output. Apart from VIM's temporary files, users like
> > me frequently write other files and/or directories that we simply do
> > not want to see tracked in Git. For example, I often test things in an
> > `a1.c` file that -- for convenience -- lives in the current worktree.
> > Obviously I don't want Git to track it, but I also don't want it to be
> > deleted, so I often add corresponding lines to `.git/info/exclude`.
> > Likewise, I sometimes download additional information related to what
> > I am implementing, and that also lives in the current worktree (but
> > then, I usually am too lazy to add an entry to `.git/info/exclude` in
> > those cases).
>
> I do the same thing, and I know other users that do as well...but I
> don't put such files in directories that are irrelevant to me.  I create
> cruft files near other files that I'm working on, or in a special
> directory of its own, but not under some directory that is irrelevant to
> the areas I'm working on.
>
> For reference, we implemented something like this in our `sparsify`
> wrapper we have internally, where 'git clean -fdX <all sparse
> directories>` is executed whenever folks sparsify.  (We have our own
> tool and don't have users use sparse-checkout directly, because our tool
> computes dependencies to determine which directories are needed.) I was
> really hesitant to add that cleaning behavior by default, and just made
> it an option.  My colleagues tired of all the bug reports about
> left-around directories and made it the default, waiting to hear
> complaints.  We never got one.  It's been over a year.

It is really nice to hear that you have evidence from users using this in
practice. I find that very convincing, and it weighs much more than all of
my (very theoretical) considerations.

Thank you,
Dscho

Re: [PATCH v3 0/8] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
> and v2.33.0-rc1 because of some changes to dir.h that became important!
>
> We launched an experimental release [1] of the sparse-index feature to our
> internal users. We immediately discovered a problem due to the isolated way
> in which we tested the sparse index: we were never building the project and
> changing our sparse-checkout definition.
>
> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp
>
> Users who ran a build in one project and then moved to another still had
> build artifacts in their worktree that lived inside the old directories.
> Since the files are marked by the .gitignore patterns, these files were not
> removed by the 'git sparse-checkout set' command. However, they make the
> sparse-index unusable because every 'git status' command needs to expand the
> sparse-directory entries in order to see if the files are tracked or not.
> This made the first experimental release actually slower for all users
> because of this cost.
>
> The solution we shipped to these customers was to change the way our fork
> handles these ignored files. Specifically, instead of Git completely
> ignoring the files, we changed Git to understand that with cone-mode
> sparse-checkout patterns, the users is asking for entire directories to be
> removed from the worktree. The link [1] included earlier has this change.
>
> I believe that this is a reasonable expectation, though I recognize that it
> might look like breaking the expectations of how .gitignore files work.
>
> Since feedback demonstrated that this is a desired behavior, v2 includes
> this behavior for all "cone mode" repositories.
>
> I'm interested in the community's thoughts about this change, as it seems
> like one that we should make carefully and intentionally.
>
> While the rewrite of the t7519 test seems unrelated, it is required to avoid
> a test failure with this change that deletes files outside of the cone. By
> moving the test into repositories not at $TRASH_DIRECTORY, we gain more
> control over the repository structure.
>
>
> Update in V3
> ============
>
>  * As promised [2], the helper methods are fixed to work with non-cone-mode
>    patterns. A later series will use them to their fullest potential
>    (changing git add, git rm, and git mv when interacting with sparse
>    entries).

Sorry I didn't review V2.  I'll find some time in the next few days to
review V3.

>
> [2]
> https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/
>
>
> Updates in V2
> =============
>
>  * This version correctly leaves untracked files alone. If untracked files
>    are found, then the directory is left as-is, in case those ignored files
>    are important to the user's work resolving those untracked files.
>
>  * This behavior is now enabled by core.sparseCheckoutCone=true.
>
>  * To use a sparse index as an in-memory data structure even when
>    index.sparse is disabled, a new patch is included to modify the prototype
>    of convert_to_sparse() to include a flags parameter.
>
>  * A few cleanup patches that I was collecting based on feedback from the
>    experimental release and intending for my next series were necessary for
>    this implementation.
>
>  * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
>    implementation that used run_subcommand().
>
> Thanks, -Stolee
>
> Derrick Stolee (8):
>   t7519: rewrite sparse index test
>   sparse-index: silently return when not using cone-mode patterns
>   sparse-index: silently return when cache tree fails
>   unpack-trees: fix nested sparse-dir search
>   sparse-checkout: create helper methods
>   attr: be careful about sparse directories
>   sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
>   sparse-checkout: clear tracked sparse dirs
>
>  Documentation/git-sparse-checkout.txt |  8 +++
>  attr.c                                | 14 ++++
>  builtin/add.c                         |  8 +--
>  builtin/sparse-checkout.c             | 95 +++++++++++++++++++++++++++
>  dir.c                                 | 33 ++++++++++
>  dir.h                                 |  6 ++
>  read-cache.c                          |  4 +-
>  sparse-index.c                        | 70 +++++++++++---------
>  sparse-index.h                        |  3 +-
>  t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
>  t/t7519-status-fsmonitor.sh           | 38 ++++++-----
>  unpack-trees.c                        | 18 +++--
>  12 files changed, 294 insertions(+), 62 deletions(-)
>
>
> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v3
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v3
> Pull-Request: https://github.com/gitgitgadget/git/pull/1009
>
> Range-diff vs v2:
>
>  1:  e66106f7a99 = 1:  e66106f7a99 t7519: rewrite sparse index test
>  2:  fb3ff9108bf = 2:  fb3ff9108bf sparse-index: silently return when not using cone-mode patterns
>  3:  37198535268 = 3:  37198535268 sparse-index: silently return when cache tree fails
>  4:  10bcadb284e = 4:  10bcadb284e unpack-trees: fix nested sparse-dir search
>  5:  e9ed5cd2830 ! 5:  5d28570c82a sparse-checkout: create helper methods
>      @@ dir.c: done:
>
>       +int init_sparse_checkout_patterns(struct index_state *istate)
>       +{
>      -+ if (istate->sparse_checkout_patterns)
>      ++ if (!core_apply_sparse_checkout ||
>      ++     istate->sparse_checkout_patterns)
>       +         return 0;
>       +
>       + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
>      @@ dir.c: done:
>       + return 0;
>       +}
>       +
>      -+enum pattern_match_result path_in_sparse_checkout(const char *path,
>      -+                                           struct index_state *istate)
>      ++int path_in_sparse_checkout(const char *path,
>      ++                     struct index_state *istate)
>       +{
>      ++ const char *base;
>       + int dtype = DT_REG;
>       + init_sparse_checkout_patterns(istate);
>       +
>       + if (!istate->sparse_checkout_patterns)
>       +         return MATCHED;
>       +
>      -+ return path_matches_pattern_list(path, strlen(path), NULL, &dtype,
>      ++ base = strrchr(path, '/');
>      ++ return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
>      ++                                  &dtype,
>       +                                  istate->sparse_checkout_patterns,
>      -+                                  istate);
>      ++                                  istate) > 0;
>       +}
>       +
>        static struct path_pattern *last_matching_pattern_from_lists(
>      @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
>       +
>       +int init_sparse_checkout_patterns(struct index_state *state);
>       +
>      -+enum pattern_match_result path_in_sparse_checkout(const char *path,
>      -+                                           struct index_state *istate);
>      ++int path_in_sparse_checkout(const char *path,
>      ++                     struct index_state *istate);
>       +
>        struct dir_entry *dir_add_ignored(struct dir_struct *dir,
>                                   struct index_state *istate,
>  6:  5680df62e1c = 6:  c9e100e68f8 attr: be careful about sparse directories
>  7:  1dd73b36eb4 = 7:  b0ece4b7dcc sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
>  8:  f74015a2be9 = 8:  febef675f05 sparse-checkout: clear tracked sparse dirs
>
> --
> gitgitgadget

[PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee via GitGitGadget]

UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
and v2.33.0-rc1 because of some changes to dir.h that became important!

We launched an experimental release [1] of the sparse-index feature to our
internal users. We immediately discovered a problem due to the isolated way
in which we tested the sparse index: we were never building the project and
changing our sparse-checkout definition.

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Users who ran a build in one project and then moved to another still had
build artifacts in their worktree that lived inside the old directories.
Since the files are marked by the .gitignore patterns, these files were not
removed by the 'git sparse-checkout set' command. However, they make the
sparse-index unusable because every 'git status' command needs to expand the
sparse-directory entries in order to see if the files are tracked or not.
This made the first experimental release actually slower for all users
because of this cost.

The solution we shipped to these customers was to change the way our fork
handles these ignored files. Specifically, instead of Git completely
ignoring the files, we changed Git to understand that with cone-mode
sparse-checkout patterns, the users is asking for entire directories to be
removed from the worktree. The link [1] included earlier has this change.

I believe that this is a reasonable expectation, though I recognize that it
might look like breaking the expectations of how .gitignore files work.

Since feedback demonstrated that this is a desired behavior, v2 includes
this behavior for all "cone mode" repositories.

I'm interested in the community's thoughts about this change, as it seems
like one that we should make carefully and intentionally.

While the rewrite of the t7519 test seems unrelated, it is required to avoid
a test failure with this change that deletes files outside of the cone. By
moving the test into repositories not at $TRASH_DIRECTORY, we gain more
control over the repository structure.


Updates in V4
=============

 * Fixed an issue with the split index.

 * The helper methods are used more consistently.

 * The helper method path_in_cone_mode_sparse_checkout() is introduced.

 * Commit messages are edited for clarity.

 * A new config option is added to disable the behavior being added in this
   series.

 * I split the commit that involves cache_tree_update(). I have not yet
   succeeded in creating tests to demonstrate why this is required outside
   of needing it in the Scalar functional tests, which includes a version of
   partial clone. I will continue to investigate recreating this scenario in
   the Git test suite, but I wanted to send this version out to get feedback
   on the things that have changed.


Update in V3
============

 * As promised [2], the helper methods are fixed to work with non-cone-mode
   patterns. A later series will use them to their fullest potential
   (changing git add, git rm, and git mv when interacting with sparse
   entries).

[2]
https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/


Updates in V2
=============

 * This version correctly leaves untracked files alone. If untracked files
   are found, then the directory is left as-is, in case those ignored files
   are important to the user's work resolving those untracked files.

 * This behavior is now enabled by core.sparseCheckoutCone=true.

 * To use a sparse index as an in-memory data structure even when
   index.sparse is disabled, a new patch is included to modify the prototype
   of convert_to_sparse() to include a flags parameter.

 * A few cleanup patches that I was collecting based on feedback from the
   experimental release and intending for my next series were necessary for
   this implementation.

 * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
   implementation that used run_subcommand().

Thanks, -Stolee

Derrick Stolee (10):
  t7519: rewrite sparse index test
  sparse-index: silently return when not using cone-mode patterns
  sparse-index: silently return when cache tree fails
  sparse-index: use WRITE_TREE_MISSING_OK
  unpack-trees: fix nested sparse-dir search
  sparse-checkout: create helper methods
  attr: be careful about sparse directories
  sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
  sparse-checkout: clear tracked sparse dirs
  sparse-checkout: add config to disable deleting dirs

 Documentation/config/index.txt        |   6 ++
 Documentation/git-sparse-checkout.txt |  10 +++
 attr.c                                |  14 ++++
 builtin/add.c                         |   7 +-
 builtin/sparse-checkout.c             | 101 ++++++++++++++++++++++++++
 dir.c                                 |  52 +++++++++++++
 dir.h                                 |   8 ++
 read-cache.c                          |   4 +-
 sparse-index.c                        |  76 ++++++++++---------
 sparse-index.h                        |   3 +-
 t/t1091-sparse-checkout-builtin.sh    |  63 ++++++++++++++++
 t/t7519-status-fsmonitor.sh           |  38 +++++-----
 unpack-trees.c                        |   7 +-
 13 files changed, 328 insertions(+), 61 deletions(-)


base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1009

Range-diff vs v3:

  1:  e66106f7a99 !  1:  c407b2cb346 t7519: rewrite sparse index test
     @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/
       test_expect_success 'status succeeds with sparse index' '
      -	git reset --hard &&
      +	git clone . full &&
     -+	git clone . sparse &&
     ++	git clone --sparse . sparse &&
      +	git -C sparse sparse-checkout init --cone --sparse-index &&
      +	git -C sparse sparse-checkout set dir1 dir2 &&
       
  2:  fb3ff9108bf !  2:  8660877ba7a sparse-index: silently return when not using cone-mode patterns
     @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate
       {
       	int test_env;
      -	if (istate->split_index || istate->sparse_index ||
     -+
      +	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
       	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
       		return 0;
  3:  37198535268 !  3:  a669740af9a sparse-index: silently return when cache tree fails
     @@ Commit message
          convert_to_sparse(), then we could still recover from this scenario and
          have a sparse index.
      
     -    When constructing the cache-tree extension in convert_to_sparse(), it is
     -    possible that we construct a tree object that is new to the object
     -    database. Without the WRITE_TREE_MISSING_OK flag, this results in an
     -    error that halts our conversion to a sparse index. Add this flag to
     -    remove this limitation.
     -
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
       ## sparse-index.c ##
     @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
      +	/*
      +	 * Silently return if there is a problem with the cache tree update,
      +	 * which might just be due to a conflict state in some entry.
     -+	 *
     -+	 * This might create new tree objects, so be sure to use
     -+	 * WRITE_TREE_MISSING_OK.
      +	 */
     -+	if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
     ++	if (cache_tree_update(istate, 0))
      +		return 0;
       
       	remove_fsmonitor(istate);
  -:  ----------- >  4:  b379b8fc61a sparse-index: use WRITE_TREE_MISSING_OK
  4:  10bcadb284e !  5:  acdded0f762 unpack-trees: fix nested sparse-dir search
     @@ Commit message
          diff that modified files within the sparse directory entry, but we could
          not correctly find the entry.
      
     +    Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
     +    Helped-by: René Scharfe <l.s.r@web.de>
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
       ## unpack-trees.c ##
     @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,
       					    const struct name_entry *p)
       {
      -	struct cache_entry *ce;
     ++	const char *path;
      +	struct cache_entry *ce = NULL;
       	int pos = find_cache_pos(info, p->path, p->pathlen);
       	struct unpack_trees_options *o = info->data;
     -+	struct strbuf full_path = STRBUF_INIT;
       
     - 	if (0 <= pos)
     - 		return o->src_index->cache[pos];
     -@@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
     - 	if (pos < 0 || pos >= o->src_index->cache_nr)
     - 		return NULL;
     - 
     -+	strbuf_addstr(&full_path, info->traverse_path);
     -+	strbuf_add(&full_path, p->path, p->pathlen);
     -+	strbuf_addch(&full_path, '/');
     -+
     - 	/*
     - 	 * Due to lexicographic sorting and sparse directory
     - 	 * entries ending with a trailing slash, our path as a
      @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
       	while (pos >= 0) {
       		ce = o->src_index->cache[pos];
       
      -		if (strncmp(ce->name, p->path, p->pathlen))
     --			return NULL;
     -+		if (strncmp(ce->name, full_path.buf, full_path.len)) {
     -+			ce = NULL;
     -+			break;
     -+		}
     ++		if (!skip_prefix(ce->name, info->traverse_path, &path) ||
     ++		    strncmp(path, p->path, p->pathlen) ||
     ++		    path[p->pathlen] != '/')
     + 			return NULL;
       
       		if (S_ISSPARSEDIR(ce->ce_mode) &&
     - 		    sparse_dir_matches_path(ce, info, p))
     --			return ce;
     -+			break;
     - 
     - 		pos--;
     - 	}
     - 
     --	return NULL;
     -+	strbuf_release(&full_path);
     -+	return ce;
     - }
     - 
     - static void debug_path(struct traverse_info *info)
  5:  5d28570c82a !  6:  1958751aa0e sparse-checkout: create helper methods
     @@ Commit message
          adopt these helpers. There are just two in builtin/add.c and
          sparse-index.c that can use path_in_sparse_checkout().
      
     +    We add a path_in_cone_mode_sparse_checkout() as well that will only
     +    return false if the path is outside of the sparse-checkout definition
     +    _and_ the sparse-checkout patterns are in cone mode.
     +
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
       ## builtin/add.c ##
     @@ builtin/add.c: static int refresh(int verbose, const struct pathspec *pathspec)
      -			    (sparse_checkout_enabled &&
      -			     !path_matches_pattern_list(path, strlen(path), NULL,
      -							&dtype, &pl, &the_index))) {
     -+			    (core_apply_sparse_checkout &&
     -+			     path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {
     ++			    !path_in_sparse_checkout(path, &the_index)) {
       				string_list_append(&only_match_skip_worktree,
       						   pathspec->items[i].original);
       			} else {
     @@ dir.c: done:
       
      +int init_sparse_checkout_patterns(struct index_state *istate)
      +{
     -+	if (!core_apply_sparse_checkout ||
     -+	    istate->sparse_checkout_patterns)
     ++	if (!core_apply_sparse_checkout)
     ++		return 1;
     ++	if (istate->sparse_checkout_patterns)
      +		return 0;
      +
      +	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
     @@ dir.c: done:
      +	return 0;
      +}
      +
     -+int path_in_sparse_checkout(const char *path,
     -+			    struct index_state *istate)
     ++static int path_in_sparse_checkout_1(const char *path,
     ++				     struct index_state *istate,
     ++				     int require_cone_mode)
      +{
      +	const char *base;
      +	int dtype = DT_REG;
     -+	init_sparse_checkout_patterns(istate);
      +
     -+	if (!istate->sparse_checkout_patterns)
     -+		return MATCHED;
     ++	/*
     ++	 * We default to accepting a path if there are no patterns or
     ++	 * they are of the wrong type.
     ++	 */
     ++	if (init_sparse_checkout_patterns(istate) ||
     ++	    (require_cone_mode &&
     ++	     !istate->sparse_checkout_patterns->use_cone_patterns))
     ++		return 1;
      +
      +	base = strrchr(path, '/');
      +	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
     @@ dir.c: done:
      +					 istate->sparse_checkout_patterns,
      +					 istate) > 0;
      +}
     ++
     ++int path_in_sparse_checkout(const char *path,
     ++			    struct index_state *istate)
     ++{
     ++	return path_in_sparse_checkout_1(path, istate, 0);
     ++}
     ++
     ++int path_in_cone_mode_sparse_checkout(const char *path,
     ++				     struct index_state *istate)
     ++{
     ++	return path_in_sparse_checkout_1(path, istate, 1);
     ++}
      +
       static struct path_pattern *last_matching_pattern_from_lists(
       		struct dir_struct *dir, struct index_state *istate,
     @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
      +
      +int path_in_sparse_checkout(const char *path,
      +			    struct index_state *istate);
     ++int path_in_cone_mode_sparse_checkout(const char *path,
     ++				      struct index_state *istate);
      +
       struct dir_entry *dir_add_ignored(struct dir_struct *dir,
       				  struct index_state *istate,
     @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
      
       ## sparse-index.c ##
      @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,
     + {
       	int i, can_convert = 1;
       	int start_converted = num_converted;
     - 	enum pattern_match_result match;
     +-	enum pattern_match_result match;
      -	int dtype = DT_UNKNOWN;
       	struct strbuf child_path = STRBUF_INIT;
      -	struct pattern_list *pl = istate->sparse_checkout_patterns;
     @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,
       	 */
      -	match = path_matches_pattern_list(ct_path, ct_pathlen,
      -					  NULL, &dtype, pl, istate);
     -+	match = path_in_sparse_checkout(ct_path, istate);
     - 	if (match != NOT_MATCHED)
     +-	if (match != NOT_MATCHED)
     ++	if (path_in_sparse_checkout(ct_path, istate))
       		can_convert = 0;
       
     + 	for (i = start; can_convert && i < end; i++) {
      @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
       	if (!istate->repo->settings.sparse_index)
       		return 0;
     @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
      -		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
      -			return 0;
      -	}
     -+	if (init_sparse_checkout_patterns(istate) < 0)
     ++	if (init_sparse_checkout_patterns(istate))
      +		return 0;
       
       	/*
  6:  c9e100e68f8 !  7:  e496f3cee66 attr: be careful about sparse directories
     @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat
       		return NULL;
       
      +	/*
     -+	 * In the case of cone-mode sparse-checkout, getting the
     -+	 * .gitattributes file from a directory is meaningless: all
     -+	 * contained paths will be sparse if the .gitattributes is also
     -+	 * sparse. In the case of a sparse index, it is critical that we
     -+	 * don't go looking for one as it will expand the index.
     ++	 * The .gitattributes file only applies to files within its
     ++	 * parent directory. In the case of cone-mode sparse-checkout,
     ++	 * the .gitattributes file is sparse if and only if all paths
     ++	 * within that directory are also sparse. Thus, don't load the
     ++	 * .gitattributes file since it will not matter.
     ++	 *
     ++	 * In the case of a sparse index, it is critical that we don't go
     ++	 * looking for a .gitattributes file, as the index will expand.
      +	 */
     -+	init_sparse_checkout_patterns(istate);
     -+	if (istate->sparse_checkout_patterns &&
     -+	    istate->sparse_checkout_patterns->use_cone_patterns &&
     -+	    path_in_sparse_checkout(path, istate) == NOT_MATCHED)
     ++	if (!path_in_cone_mode_sparse_checkout(path, istate))
      +		return NULL;
      +
       	buf = read_blob_data_from_index(istate, path, NULL);
  7:  b0ece4b7dcc !  8:  cab9360b1e9 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
     @@ Metadata
      Author: Derrick Stolee <dstolee@microsoft.com>
      
       ## Commit message ##
     -    sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
     +    sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
      
          The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
          environment variable or the "index.sparse" config setting before
     @@ Commit message
          index as an in-memory data structure, regardless of whether the on-disk
          format should be sparse.
      
     -    To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip
     +    To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip
          these config checks when enabled. All current consumers are modified to
          pass '0' in the new 'flags' parameter.
      
     @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate
      +int convert_to_sparse(struct index_state *istate, int flags)
       {
       	int test_env;
     +-	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
     ++	if (istate->sparse_index || !istate->cache_nr ||
     + 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
     + 		return 0;
       
     -@@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
       	if (!istate->repo)
       		istate->repo = the_repository;
       
     @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
      -	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
      -	if (test_env >= 0)
      -		set_sparse_index_config(istate->repo, test_env);
     -+	if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
     ++	if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {
     ++		/*
     ++		 * The sparse index is not (yet) integrated with a split index.
     ++		 */
     ++		if (istate->split_index)
     ++			return 0;
      +		/*
      +		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
      +		 * index.sparse config variable to be on.
     @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
      +			return 0;
      +	}
       
     - 	if (init_sparse_checkout_patterns(istate) < 0)
     + 	if (init_sparse_checkout_patterns(istate))
       		return 0;
      
       ## sparse-index.h ##
     @@ sparse-index.h
       
       struct index_state;
      -int convert_to_sparse(struct index_state *istate);
     -+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)
     ++#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)
      +int convert_to_sparse(struct index_state *istate, int flags);
       
       /*
  8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
     @@ Commit message
          the sparse directory. This depends on the indexed version of the file,
          so the sparse directory must be expanded.
      
     +    We must take special care to look for untracked, non-ignored files in
     +    these directories before deleting them. We do not want to delete any
     +    meaningful work that the users were doing in those directories and
     +    perhaps forgot to add and commit before switching sparse-checkout
     +    definitions. Since those untracked files might be code files that
     +    generated ignored build output, also do not delete any ignored files
     +    from these directories in that case. The users can recover their state
     +    by resetting their sparse-checkout definition to include that directory
     +    and continue. Alternatively, they can see the warning that is presented
     +    and delete the directory themselves to regain the performance they
     +    expect.
     +
          By deleting the sparse directories when changing scope (or running 'git
          sparse-checkout reapply') we regain these performance benefits as if the
          repository was in a clean state.
     @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for
       'git sparse-checkout set' command to reflect the expected cone in the working
       directory.
       
     -+The cone mode sparse-checkout patterns will also remove ignored files that
     -+are not within the sparse-checkout definition. This is important behavior
     -+to preserve the performance of the sparse index, but also matches that
     -+cone mode patterns care about directories, not files. If there exist files
     -+that are untracked and not ignored, then Git will not delete files within
     -+that directory other than the tracked files that are now out of scope.
     -+These files should be removed manually to ensure Git can behave optimally.
     ++When changing the sparse-checkout patterns in cone mode, Git will inspect each
     ++tracked directory that is not within the sparse-checkout cone to see if it
     ++contains any untracked files. If all of those files are ignored due to the
     ++`.gitignore` patterns, then the directory will be deleted. If any of the
     ++untracked files within that directory is not ignored, then no deletions will
     ++occur within that directory and a warning message will appear. If these files
     ++are important, then reset your sparse-checkout definition so they are included,
     ++use `git add` and `git commit` to store them, then remove any remaining files
     ++manually to ensure Git can behave optimally.
      +
       
       SUBMODULES
     @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char
      +	 */
      +	if (!r || !r->index || !r->worktree)
      +		return;
     -+	init_sparse_checkout_patterns(r->index);
     -+	if (!r->index->sparse_checkout_patterns ||
     ++	if (init_sparse_checkout_patterns(r->index) ||
      +	    !r->index->sparse_checkout_patterns->use_cone_patterns)
      +		return;
      +
     @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char
      +		 * prevents us from converting to a sparse index, then do
      +		 * not try deleting files.
      +		 */
     -+		if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
     ++		if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
      +			return;
      +		was_full = 1;
      +	}
  -:  ----------- > 10:  8d55a6ba2fd sparse-checkout: add config to disable deleting dirs

-- 
gitgitgadget

[PATCH v4 01/10] t7519: rewrite sparse index test

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The sparse index is tested with the FS Monitor hook and extension since
f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
was very fragile because it shared an index across sparse and non-sparse
behavior. Since that expansion and contraction could cause the index to
lose its FS Monitor bitmap and token, behavior is fragile to changes in
'git sparse-checkout set'.

Rewrite the test to use two clones of the original repo: full and
sparse. This allows us to also keep the test files (actual, expect,
trace2.txt) out of the repos we are testing with 'git status'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index deea88d4431..f1463197b99 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
 # If "!" is supplied, then we verify that we do not call ensure_full_index
 # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
 check_sparse_index_behavior () {
-	git status --porcelain=v2 >expect &&
-	git sparse-checkout init --cone --sparse-index &&
-	git sparse-checkout set dir1 dir2 &&
+	git -C full status --porcelain=v2 >expect &&
 	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-		git status --porcelain=v2 >actual &&
+		git -C sparse status --porcelain=v2 >actual &&
 	test_region $1 index ensure_full_index trace2.txt &&
 	test_region fsm_hook query trace2.txt &&
 	test_cmp expect actual &&
-	rm trace2.txt &&
-	git sparse-checkout disable
+	rm trace2.txt
 }
 
 test_expect_success 'status succeeds with sparse index' '
-	git reset --hard &&
+	git clone . full &&
+	git clone --sparse . sparse &&
+	git -C sparse sparse-checkout init --cone --sparse-index &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
-	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
-	check_sparse_index_behavior ! &&
-
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 	EOF
-	git config core.fsmonitor .git/hooks/fsmonitor-test &&
+	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
+	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
 	check_sparse_index_behavior ! &&
 
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1/modified\0"
 	EOF
 	check_sparse_index_behavior ! &&
 
-	cp -r dir1 dir1a &&
-	git add dir1a &&
-	git commit -m "add dir1a" &&
+	git -C sparse sparse-checkout add dir1a &&
+
+	for repo in full sparse
+	do
+		cp -r $repo/dir1 $repo/dir1a &&
+		git -C $repo add dir1a &&
+		git -C $repo commit -m "add dir1a" || return 1
+	done &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
 	# This one modifies outside the sparse-checkout definition
 	# and hence we expect to expand the sparse-index.
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1a/modified\0"
 	EOF
-- 
gitgitgadget

[PATCH v4 02/10] sparse-index: silently return when not using cone-mode patterns

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

While the sparse-index is only enabled when core.sparseCheckoutCone is
also enabled, it is possible for the user to modify the sparse-checkout
file manually in a way that does not match cone-mode patterns. In this
case, we should refuse to convert an index into a sparse index, since
the sparse_checkout_patterns will not be initialized with recursive and
parent path hashsets.

Also silently return if there are no cache entries, which is a simple
case: there are no paths to make sparse!

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index c6b4feec413..cd6e0d5f408 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate)
 int convert_to_sparse(struct index_state *istate)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index ||
+	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
@@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate)
 			return 0;
 	}
 
-	if (!istate->sparse_checkout_patterns->use_cone_patterns) {
-		warning(_("attempting to use sparse-index without cone mode"));
-		return -1;
-	}
+	/*
+	 * We need cone-mode patterns to use sparse-index. If a user edits
+	 * their sparse-checkout file manually, then we can detect during
+	 * parsing that they are not actually using cone-mode patterns and
+	 * hence we need to abort this conversion _without error_. Warnings
+	 * already exist in the pattern parsing to inform the user of their
+	 * bad patterns.
+	 */
+	if (!istate->sparse_checkout_patterns->use_cone_patterns)
+		return 0;
 
 	/*
 	 * NEEDSWORK: If we have unmerged entries, then stay full.
-- 
gitgitgadget

[PATCH v4 03/10] sparse-index: silently return when cache tree fails

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

If cache_tree_update() returns a non-zero value, then it could not
create the cache tree. This is likely due to a path having a merge
conflict. Since we are already returning early, let's return silently to
avoid making it seem like we failed to write the index at all.

If we remove our dependence on the cache tree within
convert_to_sparse(), then we could still recover from this scenario and
have a sparse index.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index cd6e0d5f408..d9b07695953 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate)
 
 	/* Clear and recompute the cache-tree */
 	cache_tree_free(&istate->cache_tree);
-	if (cache_tree_update(istate, 0)) {
-		warning(_("unable to update cache-tree, staying full"));
-		return -1;
-	}
+	/*
+	 * Silently return if there is a problem with the cache tree update,
+	 * which might just be due to a conflict state in some entry.
+	 */
+	if (cache_tree_update(istate, 0))
+		return 0;
 
 	remove_fsmonitor(istate);
 
-- 
gitgitgadget

[PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When updating the cache tree in convert_to_sparse(), the
WRITE_TREE_MISSING_OK flag indicates that trees might be computed that
do not already exist within the object database. This happens in cases
such as 'git add' creating new trees that it wants to store in
anticipation of a following 'git commit'. If this flag is not specified,
then it might trigger a promisor fetch or a failure due to the object
not existing locally.

Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these
possible reasons for the cache_tree_update() to fail.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sparse-index.c b/sparse-index.c
index d9b07695953..880c5f72338 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)
 	/*
 	 * Silently return if there is a problem with the cache tree update,
 	 * which might just be due to a conflict state in some entry.
+	 *
+	 * This might create new tree objects, so be sure to use
+	 * WRITE_TREE_MISSING_OK.
 	 */
-	if (cache_tree_update(istate, 0))
+	if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
 		return 0;
 
 	remove_fsmonitor(istate);
-- 
gitgitgadget

Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK

[Elijah Newren]

On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> When updating the cache tree in convert_to_sparse(), the
> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that
> do not already exist within the object database.

Okay.

> This happens in cases
> such as 'git add' creating new trees that it wants to store in
> anticipation of a following 'git commit'.

This doesn't make any sense to me.  Does 'git add' call
convert_to_sparse()?  I don't see why it would; wouldn't the calls to
convert_to_sparse() come via sparse-checkout init/set commands?  If
I'm correct on that, and 'git add' wants to create new trees, then by
the time convert_to_sparse() is called in some subsequent git
operation, then convert_to_sparse would already have the trees it
needs.


I thought the reason you would need this is someone modified and
staged a change to a file underneath a directory that will be
sparsified away; at the time of convert_to_sparse(), a tree object may
not have yet been written for the new tree with the newly modified
file (because those tend to be written at commit time), but you'd need
it at the time you sparsified.

> If this flag is not specified,
> then it might trigger a promisor fetch or a failure due to the object
> not existing locally.

Good point.

> Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these
> possible reasons for the cache_tree_update() to fail.
>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  sparse-index.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/sparse-index.c b/sparse-index.c
> index d9b07695953..880c5f72338 100644
> --- a/sparse-index.c
> +++ b/sparse-index.c
> @@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)
>         /*
>          * Silently return if there is a problem with the cache tree update,
>          * which might just be due to a conflict state in some entry.
> +        *
> +        * This might create new tree objects, so be sure to use
> +        * WRITE_TREE_MISSING_OK.
>          */
> -       if (cache_tree_update(istate, 0))
> +       if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
>                 return 0;
>
>         remove_fsmonitor(istate);
> --
> gitgitgadget

Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK

[Derrick Stolee]

On 8/27/2021 5:33 PM, Elijah Newren wrote:
> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> When updating the cache tree in convert_to_sparse(), the
>> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that
>> do not already exist within the object database.
> 
> Okay.
> 
>> This happens in cases
>> such as 'git add' creating new trees that it wants to store in
>> anticipation of a following 'git commit'.
> 
> This doesn't make any sense to me.  Does 'git add' call
> convert_to_sparse()?  I don't see why it would; wouldn't the calls to
> convert_to_sparse() come via sparse-checkout init/set commands?  If
> I'm correct on that, and 'git add' wants to create new trees, then by
> the time convert_to_sparse() is called in some subsequent git
> operation, then convert_to_sparse would already have the trees it
> needs.

If someone adds a change outside the sparse-checkout cone, then the
index is expanded in-memory, then is converted to sparse when the
index is written again.

> I thought the reason you would need this is someone modified and
> staged a change to a file underneath a directory that will be
> sparsified away; at the time of convert_to_sparse(), a tree object may
> not have yet been written for the new tree with the newly modified
> file (because those tend to be written at commit time), but you'd need
> it at the time you sparsified.

Yes. I think we are trying to say the same thing.

Thanks,
-Stolee

Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK

[Elijah Newren]

On Mon, Aug 30, 2021 at 6:19 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/27/2021 5:33 PM, Elijah Newren wrote:
> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> > <gitgitgadget@gmail.com> wrote:
> >>
> >> From: Derrick Stolee <dstolee@microsoft.com>
> >>
> >> When updating the cache tree in convert_to_sparse(), the
> >> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that
> >> do not already exist within the object database.
> >
> > Okay.
> >
> >> This happens in cases
> >> such as 'git add' creating new trees that it wants to store in
> >> anticipation of a following 'git commit'.
> >
> > This doesn't make any sense to me.  Does 'git add' call
> > convert_to_sparse()?  I don't see why it would; wouldn't the calls to
> > convert_to_sparse() come via sparse-checkout init/set commands?  If
> > I'm correct on that, and 'git add' wants to create new trees, then by
> > the time convert_to_sparse() is called in some subsequent git
> > operation, then convert_to_sparse would already have the trees it
> > needs.
>
> If someone adds a change outside the sparse-checkout cone, then the
> index is expanded in-memory, then is converted to sparse when the
> index is written again.

Ah, I missed that convert_to_sparse() is called from write_index(),
and that we are dealing with a single operation that does a
sparse->full (in memory)->sparse roundtrip.

Thanks for clearing that up.

> > I thought the reason you would need this is someone modified and
> > staged a change to a file underneath a directory that will be
> > sparsified away; at the time of convert_to_sparse(), a tree object may
> > not have yet been written for the new tree with the newly modified
> > file (because those tend to be written at commit time), but you'd need
> > it at the time you sparsified.
>
> Yes. I think we are trying to say the same thing.

While writing this, I was thinking in terms of the `git add` being
done when the index was full (both in memory and on disk), and then a
later `git sparse-checkout ...` command would invoke the
convert_to_sparse().  I didn't fully catch the nuances here.

Thanks for bringing me up to speed.

[PATCH v4 05/10] unpack-trees: fix nested sparse-dir search

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The iterated search in find_cache_entry() was recently modified to
include a loop that searches backwards for a sparse directory entry that
matches the given traverse_info and name_entry. However, the string
comparison failed to actually concatenate those two strings, so this
failed to find a sparse directory when it was not a top-level directory.

This caused some errors in rare cases where a 'git checkout' spanned a
diff that modified files within the sparse directory entry, but we could
not correctly find the entry.

Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Helped-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 unpack-trees.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 5786645f315..47ef0cf4c53 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1255,7 +1255,8 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
 static struct cache_entry *find_cache_entry(struct traverse_info *info,
 					    const struct name_entry *p)
 {
-	struct cache_entry *ce;
+	const char *path;
+	struct cache_entry *ce = NULL;
 	int pos = find_cache_pos(info, p->path, p->pathlen);
 	struct unpack_trees_options *o = info->data;
 
@@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	while (pos >= 0) {
 		ce = o->src_index->cache[pos];
 
-		if (strncmp(ce->name, p->path, p->pathlen))
+		if (!skip_prefix(ce->name, info->traverse_path, &path) ||
+		    strncmp(path, p->path, p->pathlen) ||
+		    path[p->pathlen] != '/')
 			return NULL;
 
 		if (S_ISSPARSEDIR(ce->ce_mode) &&
-- 
gitgitgadget

Re: [PATCH v4 05/10] unpack-trees: fix nested sparse-dir search

[René Scharfe]

Am 24.08.21 um 23:51 schrieb Derrick Stolee via GitGitGadget:
> From: Derrick Stolee <dstolee@microsoft.com>
>
> The iterated search in find_cache_entry() was recently modified to
> include a loop that searches backwards for a sparse directory entry that
> matches the given traverse_info and name_entry. However, the string
> comparison failed to actually concatenate those two strings, so this
> failed to find a sparse directory when it was not a top-level directory.
>
> This caused some errors in rare cases where a 'git checkout' spanned a
> diff that modified files within the sparse directory entry, but we could
> not correctly find the entry.
>
> Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
> Helped-by: René Scharfe <l.s.r@web.de>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  unpack-trees.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 5786645f315..47ef0cf4c53 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -1255,7 +1255,8 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
>  static struct cache_entry *find_cache_entry(struct traverse_info *info,
>  					    const struct name_entry *p)
>  {
> -	struct cache_entry *ce;
> +	const char *path;
> +	struct cache_entry *ce = NULL;
                              ^^^^^^^
This added initialization doesn't seem to be needed...

>  	int pos = find_cache_pos(info, p->path, p->pathlen);
>  	struct unpack_trees_options *o = info->data;
>
> @@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>  	while (pos >= 0) {
>  		ce = o->src_index->cache[pos];

... because ce is set here before it's used.

>
> -		if (strncmp(ce->name, p->path, p->pathlen))
> +		if (!skip_prefix(ce->name, info->traverse_path, &path) ||
> +		    strncmp(path, p->path, p->pathlen) ||
> +		    path[p->pathlen] != '/')
>  			return NULL;
>
>  		if (S_ISSPARSEDIR(ce->ce_mode) &&
>

Re: [PATCH v4 05/10] unpack-trees: fix nested sparse-dir search

[Derrick Stolee]

On 8/24/2021 6:21 PM, René Scharfe wrote:
> Am 24.08.21 um 23:51 schrieb Derrick Stolee via GitGitGadget:
>> -	struct cache_entry *ce;
>> +	const char *path;
>> +	struct cache_entry *ce = NULL;
>                               ^^^^^^^
> This added initialization doesn't seem to be needed...
> 
>>  	int pos = find_cache_pos(info, p->path, p->pathlen);
>>  	struct unpack_trees_options *o = info->data;
>>
>> @@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
>>  	while (pos >= 0) {
>>  		ce = o->src_index->cache[pos];
> 
> ... because ce is set here before it's used.

Better yet, 'ce' is only used within the while loop, so its
declaration can be moved to this line instead.

Thanks,
-Stolee

[PATCH v4 06/10] sparse-checkout: create helper methods

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

As we integrate the sparse index into more builtins, we occasionally
need to check the sparse-checkout patterns to see if a path is within
the sparse-checkout cone. Create some helper methods that help
initialize the patterns and check for pattern matching to make this
easier.

The existing callers of commands like get_sparse_checkout_patterns() use
a custom 'struct pattern_list' that is not necessarily the one in the
'struct index_state', so there are not many previous uses that could
adopt these helpers. There are just two in builtin/add.c and
sparse-index.c that can use path_in_sparse_checkout().

We add a path_in_cone_mode_sparse_checkout() as well that will only
return false if the path is outside of the sparse-checkout definition
_and_ the sparse-checkout patterns are in cone mode.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 builtin/add.c  |  7 +------
 dir.c          | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++
 dir.h          |  8 ++++++++
 sparse-index.c | 14 +++-----------
 4 files changed, 64 insertions(+), 17 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 17528e8f922..88a6c0c69fb 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 	int flags = REFRESH_IGNORE_SKIP_WORKTREE |
 		    (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);
-	struct pattern_list pl = { 0 };
-	int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);
 
 	seen = xcalloc(pathspec->nr, 1);
 	refresh_index(&the_index, flags, pathspec, seen,
@@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	for (i = 0; i < pathspec->nr; i++) {
 		if (!seen[i]) {
 			const char *path = pathspec->items[i].original;
-			int dtype = DT_REG;
 
 			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
-			    (sparse_checkout_enabled &&
-			     !path_matches_pattern_list(path, strlen(path), NULL,
-							&dtype, &pl, &the_index))) {
+			    !path_in_sparse_checkout(path, &the_index)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec->items[i].original);
 			} else {
diff --git a/dir.c b/dir.c
index 03c4d212672..86afa2eae00 100644
--- a/dir.c
+++ b/dir.c
@@ -1439,6 +1439,58 @@ done:
 	return result;
 }
 
+int init_sparse_checkout_patterns(struct index_state *istate)
+{
+	if (!core_apply_sparse_checkout)
+		return 1;
+	if (istate->sparse_checkout_patterns)
+		return 0;
+
+	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
+
+	if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {
+		FREE_AND_NULL(istate->sparse_checkout_patterns);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int path_in_sparse_checkout_1(const char *path,
+				     struct index_state *istate,
+				     int require_cone_mode)
+{
+	const char *base;
+	int dtype = DT_REG;
+
+	/*
+	 * We default to accepting a path if there are no patterns or
+	 * they are of the wrong type.
+	 */
+	if (init_sparse_checkout_patterns(istate) ||
+	    (require_cone_mode &&
+	     !istate->sparse_checkout_patterns->use_cone_patterns))
+		return 1;
+
+	base = strrchr(path, '/');
+	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
+					 &dtype,
+					 istate->sparse_checkout_patterns,
+					 istate) > 0;
+}
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate)
+{
+	return path_in_sparse_checkout_1(path, istate, 0);
+}
+
+int path_in_cone_mode_sparse_checkout(const char *path,
+				     struct index_state *istate)
+{
+	return path_in_sparse_checkout_1(path, istate, 1);
+}
+
 static struct path_pattern *last_matching_pattern_from_lists(
 		struct dir_struct *dir, struct index_state *istate,
 		const char *pathname, int pathlen,
diff --git a/dir.h b/dir.h
index b3e1a54a971..6823312521e 100644
--- a/dir.h
+++ b/dir.h
@@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,
 				const char *basename, int *dtype,
 				struct pattern_list *pl,
 				struct index_state *istate);
+
+int init_sparse_checkout_patterns(struct index_state *state);
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate);
+int path_in_cone_mode_sparse_checkout(const char *path,
+				      struct index_state *istate);
+
 struct dir_entry *dir_add_ignored(struct dir_struct *dir,
 				  struct index_state *istate,
 				  const char *pathname, int len);
diff --git a/sparse-index.c b/sparse-index.c
index 880c5f72338..23f7c3bd361 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate,
 {
 	int i, can_convert = 1;
 	int start_converted = num_converted;
-	enum pattern_match_result match;
-	int dtype = DT_UNKNOWN;
 	struct strbuf child_path = STRBUF_INIT;
-	struct pattern_list *pl = istate->sparse_checkout_patterns;
 
 	/*
 	 * Is the current path outside of the sparse cone?
 	 * Then check if the region can be replaced by a sparse
 	 * directory entry (everything is sparse and merged).
 	 */
-	match = path_matches_pattern_list(ct_path, ct_pathlen,
-					  NULL, &dtype, pl, istate);
-	if (match != NOT_MATCHED)
+	if (path_in_sparse_checkout(ct_path, istate))
 		can_convert = 0;
 
 	for (i = start; can_convert && i < end; i++) {
@@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo->settings.sparse_index)
 		return 0;
 
-	if (!istate->sparse_checkout_patterns) {
-		istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
-		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
-			return 0;
-	}
+	if (init_sparse_checkout_patterns(istate))
+		return 0;
 
 	/*
 	 * We need cone-mode patterns to use sparse-index. If a user edits
-- 
gitgitgadget

[PATCH v4 07/10] attr: be careful about sparse directories

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 attr.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/attr.c b/attr.c
index d029e681f28..62db4aed443 100644
--- a/attr.c
+++ b/attr.c
@@ -14,6 +14,7 @@
 #include "utf8.h"
 #include "quote.h"
 #include "thread-utils.h"
+#include "dir.h"
 
 const char git_attr__true[] = "(builtin)true";
 const char git_attr__false[] = "\0(builtin)false";
@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
 	if (!istate)
 		return NULL;
 
+	/*
+	 * The .gitattributes file only applies to files within its
+	 * parent directory. In the case of cone-mode sparse-checkout,
+	 * the .gitattributes file is sparse if and only if all paths
+	 * within that directory are also sparse. Thus, don't load the
+	 * .gitattributes file since it will not matter.
+	 *
+	 * In the case of a sparse index, it is critical that we don't go
+	 * looking for a .gitattributes file, as the index will expand.
+	 */
+	if (!path_in_cone_mode_sparse_checkout(path, istate))
+		return NULL;
+
 	buf = read_blob_data_from_index(istate, path, NULL);
 	if (!buf)
 		return NULL;
-- 
gitgitgadget

[PATCH v4 08/10] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
environment variable or the "index.sparse" config setting before
converting the index to a sparse one. This is for ease of use since all
current consumers are preparing to compress the index before writing it
to disk. If these settings are not enabled, then convert_to_sparse()
silently returns without doing anything.

We will add a consumer in the next change that wants to use the sparse
index as an in-memory data structure, regardless of whether the on-disk
format should be sparse.

To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip
these config checks when enabled. All current consumers are modified to
pass '0' in the new 'flags' parameter.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 read-cache.c   |  4 ++--
 sparse-index.c | 37 ++++++++++++++++++++++---------------
 sparse-index.h |  3 ++-
 3 files changed, 26 insertions(+), 18 deletions(-)

diff --git a/read-cache.c b/read-cache.c
index 9048ef9e905..f5d4385c408 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
 	int ret;
 	int was_full = !istate->sparse_index;
 
-	ret = convert_to_sparse(istate);
+	ret = convert_to_sparse(istate, 0);
 
 	if (ret) {
 		warning(_("failed to convert to a sparse-index"));
@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,
 	int ret, was_full = !istate->sparse_index;
 
 	move_cache_to_base_index(istate);
-	convert_to_sparse(istate);
+	convert_to_sparse(istate, 0);
 
 	trace2_region_enter_printf("index", "shared/do_write_index",
 				   the_repository, "%s", get_tempfile_path(*temp));
diff --git a/sparse-index.c b/sparse-index.c
index 23f7c3bd361..0bc45f60ac5 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate)
 	return 0;
 }
 
-int convert_to_sparse(struct index_state *istate)
+int convert_to_sparse(struct index_state *istate, int flags)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
+	if (istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
 	if (!istate->repo)
 		istate->repo = the_repository;
 
-	/*
-	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
-	 * index.sparse config variable to be on.
-	 */
-	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
-	if (test_env >= 0)
-		set_sparse_index_config(istate->repo, test_env);
+	if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {
+		/*
+		 * The sparse index is not (yet) integrated with a split index.
+		 */
+		if (istate->split_index)
+			return 0;
+		/*
+		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
+		 * index.sparse config variable to be on.
+		 */
+		test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
+		if (test_env >= 0)
+			set_sparse_index_config(istate->repo, test_env);
 
-	/*
-	 * Only convert to sparse if index.sparse is set.
-	 */
-	prepare_repo_settings(istate->repo);
-	if (!istate->repo->settings.sparse_index)
-		return 0;
+		/*
+		 * Only convert to sparse if index.sparse is set.
+		 */
+		prepare_repo_settings(istate->repo);
+		if (!istate->repo->settings.sparse_index)
+			return 0;
+	}
 
 	if (init_sparse_checkout_patterns(istate))
 		return 0;
diff --git a/sparse-index.h b/sparse-index.h
index 1115a0d7dd9..9f3d7bc7faf 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -2,7 +2,8 @@
 #define SPARSE_INDEX_H__
 
 struct index_state;
-int convert_to_sparse(struct index_state *istate);
+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)
+int convert_to_sparse(struct index_state *istate, int flags);
 
 /*
  * Some places in the codebase expect to search for a specific path.
-- 
gitgitgadget

[PATCH v4 09/10] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When changing the scope of a sparse-checkout using cone mode, we might
have some tracked directories go out of scope. The current logic removes
the tracked files from within those directories, but leaves the ignored
files within those directories. This is a bit unexpected to users who
have given input to Git saying they don't need those directories
anymore.

This is something that is new to the cone mode pattern type: the user
has explicitly said "I want these directories and _not_ those
directories." The typical sparse-checkout patterns more generally apply
to "I want files with with these patterns" so it is natural to leave
ignored files as they are. This focus on directories in cone mode
provides us an opportunity to change the behavior.

Leaving these ignored files in the sparse directories makes it
impossible to gain performance benefits in the sparse index. When we
track into these directories, we need to know if the files are ignored
or not, which might depend on the _tracked_ .gitignore file(s) within
the sparse directory. This depends on the indexed version of the file,
so the sparse directory must be expanded.

We must take special care to look for untracked, non-ignored files in
these directories before deleting them. We do not want to delete any
meaningful work that the users were doing in those directories and
perhaps forgot to add and commit before switching sparse-checkout
definitions. Since those untracked files might be code files that
generated ignored build output, also do not delete any ignored files
from these directories in that case. The users can recover their state
by resetting their sparse-checkout definition to include that directory
and continue. Alternatively, they can see the warning that is presented
and delete the directory themselves to regain the performance they
expect.

By deleting the sparse directories when changing scope (or running 'git
sparse-checkout reapply') we regain these performance benefits as if the
repository was in a clean state.

Since these ignored files are frequently build output or helper files
from IDEs, the users should not need the files now that the tracked
files are removed. If the tracked files reappear, then they will have
newer timestamps than the build artifacts, so the artifacts will need to
be regenerated anyway.

Use the sparse-index as a data structure in order to find the sparse
directories that can be safely deleted. Re-expand the index to a full
one if it was full before.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt | 10 +++
 builtin/sparse-checkout.c             | 94 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 3 files changed, 163 insertions(+)

diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index fdcf43f87cb..42056ee9ff9 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the
 'git sparse-checkout set' command to reflect the expected cone in the working
 directory.
 
+When changing the sparse-checkout patterns in cone mode, Git will inspect each
+tracked directory that is not within the sparse-checkout cone to see if it
+contains any untracked files. If all of those files are ignored due to the
+`.gitignore` patterns, then the directory will be deleted. If any of the
+untracked files within that directory is not ignored, then no deletions will
+occur within that directory and a warning message will appear. If these files
+are important, then reset your sparse-checkout definition so they are included,
+use `git add` and `git commit` to store them, then remove any remaining files
+manually to ensure Git can behave optimally.
+
 
 SUBMODULES
 ----------
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 8ba9f13787b..d0f5c4702be 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv)
 	return 0;
 }
 
+static void clean_tracked_sparse_directories(struct repository *r)
+{
+	int i, was_full = 0;
+	struct strbuf path = STRBUF_INIT;
+	size_t pathlen;
+	struct string_list_item *item;
+	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
+
+	/*
+	 * If we are not using cone mode patterns, then we cannot
+	 * delete directories outside of the sparse cone.
+	 */
+	if (!r || !r->index || !r->worktree)
+		return;
+	if (init_sparse_checkout_patterns(r->index) ||
+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
+		return;
+
+	/*
+	 * Use the sparse index as a data structure to assist finding
+	 * directories that are safe to delete. This conversion to a
+	 * sparse index will not delete directories that contain
+	 * conflicted entries or submodules.
+	 */
+	if (!r->index->sparse_index) {
+		/*
+		 * If something, such as a merge conflict or other concern,
+		 * prevents us from converting to a sparse index, then do
+		 * not try deleting files.
+		 */
+		if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
+			return;
+		was_full = 1;
+	}
+
+	strbuf_addstr(&path, r->worktree);
+	strbuf_complete(&path, '/');
+	pathlen = path.len;
+
+	/*
+	 * Collect directories that have gone out of scope but also
+	 * exist on disk, so there is some work to be done. We need to
+	 * store the entries in a list before exploring, since that might
+	 * expand the sparse-index again.
+	 */
+	for (i = 0; i < r->index->cache_nr; i++) {
+		struct cache_entry *ce = r->index->cache[i];
+
+		if (S_ISSPARSEDIR(ce->ce_mode) &&
+		    repo_file_exists(r, ce->name))
+			string_list_append(&sparse_dirs, ce->name);
+	}
+
+	for_each_string_list_item(item, &sparse_dirs) {
+		struct dir_struct dir = DIR_INIT;
+		struct pathspec p = { 0 };
+		struct strvec s = STRVEC_INIT;
+
+		strbuf_setlen(&path, pathlen);
+		strbuf_addstr(&path, item->string);
+
+		dir.flags |= DIR_SHOW_IGNORED_TOO;
+
+		setup_standard_excludes(&dir);
+		strvec_push(&s, path.buf);
+
+		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
+		fill_directory(&dir, r->index, &p);
+
+		if (dir.nr) {
+			warning(_("directory '%s' contains untracked files,"
+				  " but is not in the sparse-checkout cone"),
+				item->string);
+		} else if (remove_dir_recursively(&path, 0)) {
+			/*
+			 * Removal is "best effort". If something blocks
+			 * the deletion, then continue with a warning.
+			 */
+			warning(_("failed to remove directory '%s'"),
+				item->string);
+		}
+
+		dir_clear(&dir);
+	}
+
+	string_list_clear(&sparse_dirs, 0);
+	strbuf_release(&path);
+
+	if (was_full)
+		ensure_full_index(r->index);
+}
+
 static int update_working_directory(struct pattern_list *pl)
 {
 	enum update_sparsity_result result;
@@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl)
 	else
 		rollback_lock_file(&lock_file);
 
+	clean_tracked_sparse_directories(r);
+
 	r->index->sparse_checkout_patterns = NULL;
 	return result;
 }
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..71236981e64 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
 	check_files repo/deep a deeper1
 '
 
+test_expect_success 'cone mode clears ignored subdirectories' '
+	rm repo/.git/info/sparse-checkout &&
+
+	git -C repo sparse-checkout init --cone &&
+	git -C repo sparse-checkout set deep/deeper1 &&
+
+	cat >repo/.gitignore <<-\EOF &&
+	obj/
+	*.o
+	EOF
+
+	git -C repo add .gitignore &&
+	git -C repo commit -m ".gitignore" &&
+
+	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
+	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
+		    deep/deeper2/obj/a deep/deeper2/file.o file.o
+	do
+		echo ignored >repo/$file || return 1
+	done &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout reapply &&
+	test_path_is_missing repo/folder1 &&
+	test_path_is_missing repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout set deep/deeper2 &&
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	>repo/deep/deeper2/ignored.o &&
+	>repo/deep/deeper2/untracked &&
+
+	# When an untracked file is in the way, all untracked files
+	# (even ignored files) are preserved.
+	git -C repo sparse-checkout set folder1 2>err &&
+	grep "contains untracked files" err &&
+	test_path_is_file repo/deep/deeper2/ignored.o &&
+	test_path_is_file repo/deep/deeper2/untracked &&
+
+	# The rest of the cone matches expectation
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	echo "? deep/deeper2/untracked" >expect &&
+	test_cmp expect out
+'
+
 test_done
-- 
gitgitgadget

[PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The clean_tracked_sparse_directories() method deletes the tracked
directories that go out of scope when the sparse-checkout cone changes,
at least in cone mode. This is new behavior, but is recommended based on
our understanding of how users are interacting with the feature in most
cases.

It is possible that some users will object to the new behavior, so
create a new configuration option 'index.deleteSparseDirectories' that
can be set to 'false' to make clean_tracked_sparse_directories() do
nothing. This will keep all untracked files in the working tree and
cause performance problems with the sparse index, but those trade-offs
are for the user to decide.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/config/index.txt     | 6 ++++++
 builtin/sparse-checkout.c          | 9 ++++++++-
 t/t1091-sparse-checkout-builtin.sh | 4 ++++
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt
index 75f3a2d1054..c65da20a931 100644
--- a/Documentation/config/index.txt
+++ b/Documentation/config/index.txt
@@ -1,3 +1,9 @@
+index.deleteSparseDirectories::
+	When enabled, the cone mode sparse-checkout feature will delete
+	directories that are outside of the sparse-checkout cone, unless
+	such a directory contains an untracked, non-ignored file. Defaults
+	to true.
+
 index.recordEndOfIndexEntries::
 	Specifies whether the index file should include an "End Of Index
 	Entry" section. This reduces index load time on multiprocessor
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index d0f5c4702be..33ec729d9de 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -102,7 +102,7 @@ static int sparse_checkout_list(int argc, const char **argv)
 
 static void clean_tracked_sparse_directories(struct repository *r)
 {
-	int i, was_full = 0;
+	int i, value, was_full = 0;
 	struct strbuf path = STRBUF_INIT;
 	size_t pathlen;
 	struct string_list_item *item;
@@ -118,6 +118,13 @@ static void clean_tracked_sparse_directories(struct repository *r)
 	    !r->index->sparse_checkout_patterns->use_cone_patterns)
 		return;
 
+	/*
+	 * Users can disable this behavior.
+	 */
+	if (!repo_config_get_bool(r, "index.deletesparsedirectories", &value) &&
+	    !value)
+		return;
+
 	/*
 	 * Use the sparse index as a data structure to assist finding
 	 * directories that are safe to delete. This conversion to a
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 71236981e64..e0f31186d89 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -666,6 +666,10 @@ test_expect_success 'cone mode clears ignored subdirectories' '
 	git -C repo status --porcelain=v2 >out &&
 	test_must_be_empty out &&
 
+	git -C repo -c index.deleteSparseDirectories=false sparse-checkout reapply &&
+	test_path_is_dir repo/folder1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+
 	git -C repo sparse-checkout reapply &&
 	test_path_is_missing repo/folder1 &&
 	test_path_is_missing repo/deep/deeper2 &&
-- 
gitgitgadget

Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs

[Elijah Newren]

On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> The clean_tracked_sparse_directories() method deletes the tracked
> directories that go out of scope when the sparse-checkout cone changes,
> at least in cone mode. This is new behavior, but is recommended based on
> our understanding of how users are interacting with the feature in most
> cases.
>
> It is possible that some users will object to the new behavior, so
> create a new configuration option 'index.deleteSparseDirectories' that
> can be set to 'false' to make clean_tracked_sparse_directories() do
> nothing. This will keep all untracked files in the working tree and
> cause performance problems with the sparse index, but those trade-offs
> are for the user to decide.

I'm not sold that we need anything here, and it sounds like this is
all being added based on a theoretical concern.  I might not object
too much normally to trying to address theoretical conerns with new
features, but:

* I'm a little concerned that we're adding a configuration option
(which live forever and are harder to work with
backward-compatibility-wise) rather than a command line flag.

* It's not clear to me that the option name is what we want.  For
example, git checkout has a --[no-]overwrite-ignored for overwriting
ignored files (or not) when switching to a different branch.  git
merge has the same flags (though only the fast-forwarding backend
heeds it currently).  This behavior is quite similar to that flag, and
has the same default of treating ignored files as expendable.  Should
the naming be more similar?

* It's not clear to me that the option has the right level of
implementation granularity.  If someone wants ignored files to be
treated as important, should they need to set a
merge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a
index.deleteSparseDirectories AND other names, or have one high-level
option that sets all of these behaviors?

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/config/index.txt     | 6 ++++++
>  builtin/sparse-checkout.c          | 9 ++++++++-
>  t/t1091-sparse-checkout-builtin.sh | 4 ++++
>  3 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt
> index 75f3a2d1054..c65da20a931 100644
> --- a/Documentation/config/index.txt
> +++ b/Documentation/config/index.txt
> @@ -1,3 +1,9 @@
> +index.deleteSparseDirectories::
> +       When enabled, the cone mode sparse-checkout feature will delete
> +       directories that are outside of the sparse-checkout cone, unless
> +       such a directory contains an untracked, non-ignored file. Defaults
> +       to true.
> +
>  index.recordEndOfIndexEntries::
>         Specifies whether the index file should include an "End Of Index
>         Entry" section. This reduces index load time on multiprocessor
> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
> index d0f5c4702be..33ec729d9de 100644
> --- a/builtin/sparse-checkout.c
> +++ b/builtin/sparse-checkout.c
> @@ -102,7 +102,7 @@ static int sparse_checkout_list(int argc, const char **argv)
>
>  static void clean_tracked_sparse_directories(struct repository *r)
>  {
> -       int i, was_full = 0;
> +       int i, value, was_full = 0;
>         struct strbuf path = STRBUF_INIT;
>         size_t pathlen;
>         struct string_list_item *item;
> @@ -118,6 +118,13 @@ static void clean_tracked_sparse_directories(struct repository *r)
>             !r->index->sparse_checkout_patterns->use_cone_patterns)
>                 return;
>
> +       /*
> +        * Users can disable this behavior.
> +        */
> +       if (!repo_config_get_bool(r, "index.deletesparsedirectories", &value) &&
> +           !value)
> +               return;
> +
>         /*
>          * Use the sparse index as a data structure to assist finding
>          * directories that are safe to delete. This conversion to a
> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index 71236981e64..e0f31186d89 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -666,6 +666,10 @@ test_expect_success 'cone mode clears ignored subdirectories' '
>         git -C repo status --porcelain=v2 >out &&
>         test_must_be_empty out &&
>
> +       git -C repo -c index.deleteSparseDirectories=false sparse-checkout reapply &&
> +       test_path_is_dir repo/folder1 &&
> +       test_path_is_dir repo/deep/deeper2 &&
> +
>         git -C repo sparse-checkout reapply &&
>         test_path_is_missing repo/folder1 &&
>         test_path_is_missing repo/deep/deeper2 &&
> --
> gitgitgadget

Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs

[Derrick Stolee]

On 8/27/2021 4:58 PM, Elijah Newren wrote:
> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> The clean_tracked_sparse_directories() method deletes the tracked
>> directories that go out of scope when the sparse-checkout cone changes,
>> at least in cone mode. This is new behavior, but is recommended based on
>> our understanding of how users are interacting with the feature in most
>> cases.
>>
>> It is possible that some users will object to the new behavior, so
>> create a new configuration option 'index.deleteSparseDirectories' that
>> can be set to 'false' to make clean_tracked_sparse_directories() do
>> nothing. This will keep all untracked files in the working tree and
>> cause performance problems with the sparse index, but those trade-offs
>> are for the user to decide.
> 
> I'm not sold that we need anything here, and it sounds like this is
> all being added based on a theoretical concern.  I might not object
> too much normally to trying to address theoretical conerns with new
> features, but:
> 
> * I'm a little concerned that we're adding a configuration option
> (which live forever and are harder to work with
> backward-compatibility-wise) rather than a command line flag.

The issue with a command-line flag is that it would need to be added
to all the commands that reapply the sparse-checkout definition.
Maybe that's just the 'git sparse-checkout (init|set|add|reapply)'
and 'git read-tree' commands, but are there other places where this
logic might be applied in the future?

Also, as more third-party tools integrate with sparse-checkout, I
don't expect users to be directly involved in changing their cone,
so any use of a command-line flag would need to be integrated into
those tools. A config option applies this logic universally, when
needed.

> * It's not clear to me that the option name is what we want.  For
> example, git checkout has a --[no-]overwrite-ignored for overwriting
> ignored files (or not) when switching to a different branch.  git
> merge has the same flags (though only the fast-forwarding backend
> heeds it currently).  This behavior is quite similar to that flag, and
> has the same default of treating ignored files as expendable.  Should
> the naming be more similar?

I'm open to changing the name to more closely match existing naming
conventions, once we've decided if this should be included at all.

> * It's not clear to me that the option has the right level of
> implementation granularity.  If someone wants ignored files to be
> treated as important, should they need to set a
> merge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a
> index.deleteSparseDirectories AND other names, or have one high-level
> option that sets all of these behaviors?

These cases have different meanings for why an ignored file is
important. For merge and checkout, the argument is saying "don't
overwrite ignored files if they are to be replaced by a tracked
file". For sparse-checkout, the config is saying "don't delete
ignored files that are leaving the sparse-checkout scope". I think
it is meaningful to say that files that match the .gitignore
patterns but are still tracked are deleted, because sparse-checkout
removes all tracked files.

I'm less convinced that there needs to be a meta-setting that covers
all of these cases simultaneously.

---

As for moving forward, I'm fine skipping this patch if there is no
support for it. I just wanted to demonstrate that it could be done.
Perhaps we wait for the theoretical concern to be a real one, as
requested by a user with a legitimate reason to care?

Thanks,
-Stolee

Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs

[Elijah Newren]

On Mon, Aug 30, 2021 at 6:30 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/27/2021 4:58 PM, Elijah Newren wrote:
> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> > <gitgitgadget@gmail.com> wrote:
> >>
> >> From: Derrick Stolee <dstolee@microsoft.com>
> >>
> >> The clean_tracked_sparse_directories() method deletes the tracked
> >> directories that go out of scope when the sparse-checkout cone changes,
> >> at least in cone mode. This is new behavior, but is recommended based on
> >> our understanding of how users are interacting with the feature in most
> >> cases.
> >>
> >> It is possible that some users will object to the new behavior, so
> >> create a new configuration option 'index.deleteSparseDirectories' that
> >> can be set to 'false' to make clean_tracked_sparse_directories() do
> >> nothing. This will keep all untracked files in the working tree and
> >> cause performance problems with the sparse index, but those trade-offs
> >> are for the user to decide.
> >
> > I'm not sold that we need anything here, and it sounds like this is
> > all being added based on a theoretical concern.  I might not object
> > too much normally to trying to address theoretical conerns with new
> > features, but:
> >
> > * I'm a little concerned that we're adding a configuration option
> > (which live forever and are harder to work with
> > backward-compatibility-wise) rather than a command line flag.
>
> The issue with a command-line flag is that it would need to be added
> to all the commands that reapply the sparse-checkout definition.
> Maybe that's just the 'git sparse-checkout (init|set|add|reapply)'
> and 'git read-tree' commands, but are there other places where this
> logic might be applied in the future?
>
> Also, as more third-party tools integrate with sparse-checkout, I
> don't expect users to be directly involved in changing their cone,
> so any use of a command-line flag would need to be integrated into
> those tools. A config option applies this logic universally, when
> needed.
>
> > * It's not clear to me that the option name is what we want.  For
> > example, git checkout has a --[no-]overwrite-ignored for overwriting
> > ignored files (or not) when switching to a different branch.  git
> > merge has the same flags (though only the fast-forwarding backend
> > heeds it currently).  This behavior is quite similar to that flag, and
> > has the same default of treating ignored files as expendable.  Should
> > the naming be more similar?
>
> I'm open to changing the name to more closely match existing naming
> conventions, once we've decided if this should be included at all.
>
> > * It's not clear to me that the option has the right level of
> > implementation granularity.  If someone wants ignored files to be
> > treated as important, should they need to set a
> > merge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a
> > index.deleteSparseDirectories AND other names, or have one high-level
> > option that sets all of these behaviors?
>
> These cases have different meanings for why an ignored file is
> important. For merge and checkout, the argument is saying "don't
> overwrite ignored files if they are to be replaced by a tracked
> file". For sparse-checkout, the config is saying "don't delete
> ignored files that are leaving the sparse-checkout scope". I think
> it is meaningful to say that files that match the .gitignore
> patterns but are still tracked are deleted, because sparse-checkout
> removes all tracked files.
>
> I'm less convinced that there needs to be a meta-setting that covers
> all of these cases simultaneously.
>
> ---
>
> As for moving forward, I'm fine skipping this patch if there is no
> support for it. I just wanted to demonstrate that it could be done.

That's fair, and you certainly achieved that.

> Perhaps we wait for the theoretical concern to be a real one, as
> requested by a user with a legitimate reason to care?

Sounds good to me.  Then we'll have a better frame of reference for
judging how the feature should be tailored as well.

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
> and v2.33.0-rc1 because of some changes to dir.h that became important!

Isn't this stale (i.e. just copied from v2/v3)?

...
> Updates in V4
> =============
>
>  * Fixed an issue with the split index.
>
>  * The helper methods are used more consistently.
>
>  * The helper method path_in_cone_mode_sparse_checkout() is introduced.
>
>  * Commit messages are edited for clarity.
>
>  * A new config option is added to disable the behavior being added in this
>    series.
>
>  * I split the commit that involves cache_tree_update(). I have not yet
>    succeeded in creating tests to demonstrate why this is required outside
>    of needing it in the Scalar functional tests, which includes a version of
>    partial clone. I will continue to investigate recreating this scenario in
>    the Git test suite, but I wanted to send this version out to get feedback
>    on the things that have changed.

Are you talking about patch 3 or patch 4?

For patch 4, as I commented on that patch, my guess would be that you
start with a full tree, pick some directory that you will sparsify
away, and before sparsifying it away, change some file under that
directory and stage the change.  Then when you sparsify the directory
away with cone mode, I think the cache_tree_update() will need to
write the tree.  At least, some quick greps of cache_tree_update
suggest there aren't that many codepaths that attempt to update it, so
I don't think anything else would have written it yet at that point.

For patch 3, since it comes before patch 5, it would be triggerable
with the same case.  Is it possible you originally found both patches
based off similar initial conditions that just resulted in slightly
different behavioral reports, and thus that patch 4 is good enough
without patch 3?  Trying to ignore that possibility and looking at the
cache_tree_update() code, the only other reasons for failure of
cache_tree_update() that I can see are (a) someone having invalid
hashes recorded in the index (e.g. someone using `git update-index
--index-info` with some invalid hashes, or someone manually deleting
some loose blobs from under their .git/ directory), or (b) disk being
full or read-only so that writing new tree objects to the object store
fail.

> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4
> Pull-Request: https://github.com/gitgitgadget/git/pull/1009

A super micro nit:

It appears this is based on your other branches that have since been
incorporated into master.  Not a big deal, but rebasing on master
might make it easier for others to take a look at the series (I was
confused at first why this series depended on other unmerged series,
before realizing those other series had since merged down).  I know we
normally don't like rebasing series, but I suspect it wouldn't change
the range-diff (as I don't think others are touching these areas of
the code), and thus be transparent to Junio.



I commented on some individual patches, but have some small comments
that are easier to post in response to the range-diff:

> Range-diff vs v3:
>
>   1:  e66106f7a99 !  1:  c407b2cb346 t7519: rewrite sparse index test
>      @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/
>        test_expect_success 'status succeeds with sparse index' '
>       - git reset --hard &&
>       + git clone . full &&
>      -+ git clone . sparse &&
>      ++ git clone --sparse . sparse &&
>       + git -C sparse sparse-checkout init --cone --sparse-index &&
>       + git -C sparse sparse-checkout set dir1 dir2 &&
>
>   2:  fb3ff9108bf !  2:  8660877ba7a sparse-index: silently return when not using cone-mode patterns
>      @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate
>        {
>         int test_env;
>       - if (istate->split_index || istate->sparse_index ||
>      -+
>       + if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
>             !core_apply_sparse_checkout || !core_sparse_checkout_cone)
>                 return 0;
>   3:  37198535268 !  3:  a669740af9a sparse-index: silently return when cache tree fails
>      @@ Commit message
>           convert_to_sparse(), then we could still recover from this scenario and
>           have a sparse index.
>
>      -    When constructing the cache-tree extension in convert_to_sparse(), it is
>      -    possible that we construct a tree object that is new to the object
>      -    database. Without the WRITE_TREE_MISSING_OK flag, this results in an
>      -    error that halts our conversion to a sparse index. Add this flag to
>      -    remove this limitation.
>      -
>           Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>
>        ## sparse-index.c ##
>      @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>       + /*
>       +  * Silently return if there is a problem with the cache tree update,
>       +  * which might just be due to a conflict state in some entry.
>      -+  *
>      -+  * This might create new tree objects, so be sure to use
>      -+  * WRITE_TREE_MISSING_OK.
>       +  */
>      -+ if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
>      ++ if (cache_tree_update(istate, 0))
>       +         return 0;
>
>         remove_fsmonitor(istate);
>   -:  ----------- >  4:  b379b8fc61a sparse-index: use WRITE_TREE_MISSING_OK

Thanks for splitting patch 3 & 4.  I left some comments on patch 5.
For patch 4, my comments above about testcases might be of interest.

>   4:  10bcadb284e !  5:  acdded0f762 unpack-trees: fix nested sparse-dir search
>      @@ Commit message
>           diff that modified files within the sparse directory entry, but we could
>           not correctly find the entry.
>
>      +    Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
>      +    Helped-by: René Scharfe <l.s.r@web.de>
>           Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>
>        ## unpack-trees.c ##
>      @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,
>                                             const struct name_entry *p)
>        {
>       - struct cache_entry *ce;
>      ++ const char *path;
>       + struct cache_entry *ce = NULL;
>         int pos = find_cache_pos(info, p->path, p->pathlen);
>         struct unpack_trees_options *o = info->data;
>      -+ struct strbuf full_path = STRBUF_INIT;
>
>      -  if (0 <= pos)
>      -          return o->src_index->cache[pos];
>      -@@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
>      -  if (pos < 0 || pos >= o->src_index->cache_nr)
>      -          return NULL;
>      -
>      -+ strbuf_addstr(&full_path, info->traverse_path);
>      -+ strbuf_add(&full_path, p->path, p->pathlen);
>      -+ strbuf_addch(&full_path, '/');
>      -+
>      -  /*
>      -   * Due to lexicographic sorting and sparse directory
>      -   * entries ending with a trailing slash, our path as a
>       @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
>         while (pos >= 0) {
>                 ce = o->src_index->cache[pos];
>
>       -         if (strncmp(ce->name, p->path, p->pathlen))
>      --                 return NULL;
>      -+         if (strncmp(ce->name, full_path.buf, full_path.len)) {
>      -+                 ce = NULL;
>      -+                 break;
>      -+         }
>      ++         if (!skip_prefix(ce->name, info->traverse_path, &path) ||
>      ++             strncmp(path, p->path, p->pathlen) ||
>      ++             path[p->pathlen] != '/')
>      +                  return NULL;
>
>                 if (S_ISSPARSEDIR(ce->ce_mode) &&
>      -              sparse_dir_matches_path(ce, info, p))
>      --                 return ce;
>      -+                 break;
>      -
>      -          pos--;
>      -  }
>      -
>      -- return NULL;
>      -+ strbuf_release(&full_path);
>      -+ return ce;
>      - }
>      -
>      - static void debug_path(struct traverse_info *info)

René's suggestion definitely made this patch simpler (which sticks out
more when you look at the patch rather than the range-diff).

>   5:  5d28570c82a !  6:  1958751aa0e sparse-checkout: create helper methods
>      @@ Commit message
>           adopt these helpers. There are just two in builtin/add.c and
>           sparse-index.c that can use path_in_sparse_checkout().
>
>      +    We add a path_in_cone_mode_sparse_checkout() as well that will only
>      +    return false if the path is outside of the sparse-checkout definition
>      +    _and_ the sparse-checkout patterns are in cone mode.
>      +
>           Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>
>        ## builtin/add.c ##
>      @@ builtin/add.c: static int refresh(int verbose, const struct pathspec *pathspec)
>       -                     (sparse_checkout_enabled &&
>       -                      !path_matches_pattern_list(path, strlen(path), NULL,
>       -                                                 &dtype, &pl, &the_index))) {
>      -+                     (core_apply_sparse_checkout &&
>      -+                      path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {
>      ++                     !path_in_sparse_checkout(path, &the_index)) {
>                                 string_list_append(&only_match_skip_worktree,
>                                                    pathspec->items[i].original);
>                         } else {
>      @@ dir.c: done:
>
>       +int init_sparse_checkout_patterns(struct index_state *istate)
>       +{
>      -+ if (!core_apply_sparse_checkout ||
>      -+     istate->sparse_checkout_patterns)
>      ++ if (!core_apply_sparse_checkout)
>      ++         return 1;
>      ++ if (istate->sparse_checkout_patterns)
>       +         return 0;
>       +
>       + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
>      @@ dir.c: done:
>       + return 0;
>       +}
>       +
>      -+int path_in_sparse_checkout(const char *path,
>      -+                     struct index_state *istate)
>      ++static int path_in_sparse_checkout_1(const char *path,
>      ++                              struct index_state *istate,
>      ++                              int require_cone_mode)
>       +{
>       + const char *base;
>       + int dtype = DT_REG;
>      -+ init_sparse_checkout_patterns(istate);
>       +
>      -+ if (!istate->sparse_checkout_patterns)
>      -+         return MATCHED;
>      ++ /*
>      ++  * We default to accepting a path if there are no patterns or
>      ++  * they are of the wrong type.
>      ++  */
>      ++ if (init_sparse_checkout_patterns(istate) ||
>      ++     (require_cone_mode &&
>      ++      !istate->sparse_checkout_patterns->use_cone_patterns))
>      ++         return 1;
>       +
>       + base = strrchr(path, '/');
>       + return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
>      @@ dir.c: done:
>       +                                  istate->sparse_checkout_patterns,
>       +                                  istate) > 0;
>       +}
>      ++
>      ++int path_in_sparse_checkout(const char *path,
>      ++                     struct index_state *istate)
>      ++{
>      ++ return path_in_sparse_checkout_1(path, istate, 0);
>      ++}
>      ++
>      ++int path_in_cone_mode_sparse_checkout(const char *path,
>      ++                              struct index_state *istate)
>      ++{
>      ++ return path_in_sparse_checkout_1(path, istate, 1);
>      ++}
>       +
>        static struct path_pattern *last_matching_pattern_from_lists(
>                 struct dir_struct *dir, struct index_state *istate,
>      @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
>       +
>       +int path_in_sparse_checkout(const char *path,
>       +                     struct index_state *istate);
>      ++int path_in_cone_mode_sparse_checkout(const char *path,
>      ++                               struct index_state *istate);
>       +
>        struct dir_entry *dir_add_ignored(struct dir_struct *dir,
>                                   struct index_state *istate,
>      @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,
>
>        ## sparse-index.c ##
>       @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,
>      + {
>         int i, can_convert = 1;
>         int start_converted = num_converted;
>      -  enum pattern_match_result match;
>      +- enum pattern_match_result match;
>       - int dtype = DT_UNKNOWN;
>         struct strbuf child_path = STRBUF_INIT;
>       - struct pattern_list *pl = istate->sparse_checkout_patterns;
>      @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,
>          */
>       - match = path_matches_pattern_list(ct_path, ct_pathlen,
>       -                                   NULL, &dtype, pl, istate);
>      -+ match = path_in_sparse_checkout(ct_path, istate);
>      -  if (match != NOT_MATCHED)
>      +- if (match != NOT_MATCHED)
>      ++ if (path_in_sparse_checkout(ct_path, istate))
>                 can_convert = 0;
>
>      +  for (i = start; can_convert && i < end; i++) {
>       @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>         if (!istate->repo->settings.sparse_index)
>                 return 0;
>      @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>       -         if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
>       -                 return 0;
>       - }
>      -+ if (init_sparse_checkout_patterns(istate) < 0)
>      ++ if (init_sparse_checkout_patterns(istate))
>       +         return 0;
>
>         /*
>   6:  c9e100e68f8 !  7:  e496f3cee66 attr: be careful about sparse directories
>      @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat
>                 return NULL;
>
>       + /*
>      -+  * In the case of cone-mode sparse-checkout, getting the
>      -+  * .gitattributes file from a directory is meaningless: all
>      -+  * contained paths will be sparse if the .gitattributes is also
>      -+  * sparse. In the case of a sparse index, it is critical that we
>      -+  * don't go looking for one as it will expand the index.
>      ++  * The .gitattributes file only applies to files within its
>      ++  * parent directory. In the case of cone-mode sparse-checkout,
>      ++  * the .gitattributes file is sparse if and only if all paths
>      ++  * within that directory are also sparse. Thus, don't load the
>      ++  * .gitattributes file since it will not matter.
>      ++  *
>      ++  * In the case of a sparse index, it is critical that we don't go
>      ++  * looking for a .gitattributes file, as the index will expand.

This comment is better, and I appreciate the thought behind adding the
second paragraph, but the wording seems fuzzy and potentially
confusing to future readers.  Perhaps changing the wording to:

...looking for a .gitattributes file, as doing so would cause the
index to expand.

>       +  */
>      -+ init_sparse_checkout_patterns(istate);
>      -+ if (istate->sparse_checkout_patterns &&
>      -+     istate->sparse_checkout_patterns->use_cone_patterns &&
>      -+     path_in_sparse_checkout(path, istate) == NOT_MATCHED)
>      ++ if (!path_in_cone_mode_sparse_checkout(path, istate))
>       +         return NULL;
>       +
>         buf = read_blob_data_from_index(istate, path, NULL);
>   7:  b0ece4b7dcc !  8:  cab9360b1e9 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
>      @@ Metadata
>       Author: Derrick Stolee <dstolee@microsoft.com>
>
>        ## Commit message ##
>      -    sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag
>      +    sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
>
>           The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
>           environment variable or the "index.sparse" config setting before
>      @@ Commit message
>           index as an in-memory data structure, regardless of whether the on-disk
>           format should be sparse.
>
>      -    To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip
>      +    To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip
>           these config checks when enabled. All current consumers are modified to
>           pass '0' in the new 'flags' parameter.
>
>      @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate
>       +int convert_to_sparse(struct index_state *istate, int flags)
>        {
>         int test_env;
>      +- if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
>      ++ if (istate->sparse_index || !istate->cache_nr ||
>      +      !core_apply_sparse_checkout || !core_sparse_checkout_cone)
>      +          return 0;
>
>      -@@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>         if (!istate->repo)
>                 istate->repo = the_repository;
>
>      @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>       - test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
>       - if (test_env >= 0)
>       -         set_sparse_index_config(istate->repo, test_env);
>      -+ if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {
>      ++ if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {
>      ++         /*
>      ++          * The sparse index is not (yet) integrated with a split index.
>      ++          */
>      ++         if (istate->split_index)
>      ++                 return 0;
>       +         /*
>       +          * The GIT_TEST_SPARSE_INDEX environment variable triggers the
>       +          * index.sparse config variable to be on.
>      @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)
>       +                 return 0;
>       + }
>
>      -  if (init_sparse_checkout_patterns(istate) < 0)
>      +  if (init_sparse_checkout_patterns(istate))
>                 return 0;
>
>        ## sparse-index.h ##
>      @@ sparse-index.h
>
>        struct index_state;
>       -int convert_to_sparse(struct index_state *istate);
>      -+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)
>      ++#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)
>       +int convert_to_sparse(struct index_state *istate, int flags);
>
>        /*
>   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
>      @@ Commit message
>           the sparse directory. This depends on the indexed version of the file,
>           so the sparse directory must be expanded.
>
>      +    We must take special care to look for untracked, non-ignored files in
>      +    these directories before deleting them. We do not want to delete any
>      +    meaningful work that the users were doing in those directories and
>      +    perhaps forgot to add and commit before switching sparse-checkout
>      +    definitions. Since those untracked files might be code files that
>      +    generated ignored build output, also do not delete any ignored files
>      +    from these directories in that case. The users can recover their state
>      +    by resetting their sparse-checkout definition to include that directory
>      +    and continue. Alternatively, they can see the warning that is presented
>      +    and delete the directory themselves to regain the performance they
>      +    expect.
>      +
>           By deleting the sparse directories when changing scope (or running 'git
>           sparse-checkout reapply') we regain these performance benefits as if the
>           repository was in a clean state.
>      @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for
>        'git sparse-checkout set' command to reflect the expected cone in the working
>        directory.
>
>      -+The cone mode sparse-checkout patterns will also remove ignored files that
>      -+are not within the sparse-checkout definition. This is important behavior
>      -+to preserve the performance of the sparse index, but also matches that
>      -+cone mode patterns care about directories, not files. If there exist files
>      -+that are untracked and not ignored, then Git will not delete files within
>      -+that directory other than the tracked files that are now out of scope.
>      -+These files should be removed manually to ensure Git can behave optimally.
>      ++When changing the sparse-checkout patterns in cone mode, Git will inspect each
>      ++tracked directory that is not within the sparse-checkout cone to see if it
>      ++contains any untracked files. If all of those files are ignored due to the
>      ++`.gitignore` patterns, then the directory will be deleted. If any of the
>      ++untracked files within that directory is not ignored, then no deletions will
>      ++occur within that directory and a warning message will appear. If these files
>      ++are important, then reset your sparse-checkout definition so they are included,
>      ++use `git add` and `git commit` to store them, then remove any remaining files
>      ++manually to ensure Git can behave optimally.
>       +
>
>        SUBMODULES
>      @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char
>       +  */
>       + if (!r || !r->index || !r->worktree)
>       +         return;
>      -+ init_sparse_checkout_patterns(r->index);
>      -+ if (!r->index->sparse_checkout_patterns ||
>      ++ if (init_sparse_checkout_patterns(r->index) ||
>       +     !r->index->sparse_checkout_patterns->use_cone_patterns)
>       +         return;
>       +
>      @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char
>       +          * prevents us from converting to a sparse index, then do
>       +          * not try deleting files.
>       +          */
>      -+         if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))
>      ++         if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
>       +                 return;
>       +         was_full = 1;
>       + }

Thanks for the extra explanations; those help.

You haven't yet addressed how this choice will interact with
.gitignore files, though (as I commented on v3).  You alluded in patch
7 to how it was critical to avoid looking for .gitattributes files
because doing so would require the index to expand, possibly
recursively all the way down.  Don't .gitignore files have the same
problem?  Nuking all the ignored files so that you can delete extra
directories and thus not need to walk into them during operations
helps, but does the fact that we sometimes avoid removing directories
mean we have to walk into them anyway?  Do we only need to walk into
such directories when they are (still? again?) present in the working
copy?  Does there need to be some code added somewhere that checks for
the presence of a directory in the working copy and unsparsifies
sparse directory entries in the index when such directories are found?

Also, does non-cone mode sparsity paths have weird bugs around letting
users specify patterns that would happen to also exclude .gitignore
files from the working copy, but keep other sibling files within the
same directory?  In particular, if the .gitignore files aren't
present, does git add -A add a bunch of supposed-to-be-ignored files?
(If so, that's an issue that's probably tangential to this series
since it's not about cone mode or the sparse index, but it's certainly
an interesting thought.)

Maybe the whole .gitignore situation with cone mode has a simple
answer, but since this whole series is about gaining performance by
avoiding recursing into directories corresponding to sparse directory
entries, it feels the answers to how .gitignore will be handled should
at least be addressed.

>   -:  ----------- > 10:  8d55a6ba2fd sparse-checkout: add config to disable deleting dirs

I left some comments.


Thanks, as always, for all your hard work on this!

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

Sorry, one more thing...

On Fri, Aug 27, 2021 at 2:56 PM Elijah Newren <newren@gmail.com> wrote:
>
> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:

> >   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
> >      @@ Commit message
> >           the sparse directory. This depends on the indexed version of the file,
> >           so the sparse directory must be expanded.
> >
> >      +    We must take special care to look for untracked, non-ignored files in
> >      +    these directories before deleting them. We do not want to delete any
> >      +    meaningful work that the users were doing in those directories and
> >      +    perhaps forgot to add and commit before switching sparse-checkout
> >      +    definitions. Since those untracked files might be code files that
> >      +    generated ignored build output, also do not delete any ignored files
> >      +    from these directories in that case. The users can recover their state
> >      +    by resetting their sparse-checkout definition to include that directory
> >      +    and continue. Alternatively, they can see the warning that is presented
> >      +    and delete the directory themselves to regain the performance they
> >      +    expect.

Will deleting the directory regain the performance they expect, or is
another step needed?  In other worse, will the sparsification of the
paths under the no-longer-needed directory into a sparse directory
entry just happen automatically as part of some other command like
commit/add, or do they need to manually run `git sparse-checkout
reapply`?

> >      +
> >           By deleting the sparse directories when changing scope (or running 'git
> >           sparse-checkout reapply') we regain these performance benefits as if the
> >           repository was in a clean state.
> >      @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for
> >        'git sparse-checkout set' command to reflect the expected cone in the working
> >        directory.
> >
> >      -+The cone mode sparse-checkout patterns will also remove ignored files that
> >      -+are not within the sparse-checkout definition. This is important behavior
> >      -+to preserve the performance of the sparse index, but also matches that
> >      -+cone mode patterns care about directories, not files. If there exist files
> >      -+that are untracked and not ignored, then Git will not delete files within
> >      -+that directory other than the tracked files that are now out of scope.
> >      -+These files should be removed manually to ensure Git can behave optimally.
> >      ++When changing the sparse-checkout patterns in cone mode, Git will inspect each
> >      ++tracked directory that is not within the sparse-checkout cone to see if it
> >      ++contains any untracked files. If all of those files are ignored due to the
> >      ++`.gitignore` patterns, then the directory will be deleted. If any of the
> >      ++untracked files within that directory is not ignored, then no deletions will
> >      ++occur within that directory and a warning message will appear. If these files
> >      ++are important, then reset your sparse-checkout definition so they are included,
> >      ++use `git add` and `git commit` to store them, then remove any remaining files
> >      ++manually to ensure Git can behave optimally.

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee]

On 8/27/2021 6:01 PM, Elijah Newren wrote:
> Sorry, one more thing...
> 
> On Fri, Aug 27, 2021 at 2:56 PM Elijah Newren <newren@gmail.com> wrote:
>>
>> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
>> <gitgitgadget@gmail.com> wrote:
> 
>>>   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
>>>      @@ Commit message
>>>           the sparse directory. This depends on the indexed version of the file,
>>>           so the sparse directory must be expanded.
>>>
>>>      +    We must take special care to look for untracked, non-ignored files in
>>>      +    these directories before deleting them. We do not want to delete any
>>>      +    meaningful work that the users were doing in those directories and
>>>      +    perhaps forgot to add and commit before switching sparse-checkout
>>>      +    definitions. Since those untracked files might be code files that
>>>      +    generated ignored build output, also do not delete any ignored files
>>>      +    from these directories in that case. The users can recover their state
>>>      +    by resetting their sparse-checkout definition to include that directory
>>>      +    and continue. Alternatively, they can see the warning that is presented
>>>      +    and delete the directory themselves to regain the performance they
>>>      +    expect.
> 
> Will deleting the directory regain the performance they expect, or is
> another step needed?  In other worse, will the sparsification of the
> paths under the no-longer-needed directory into a sparse directory
> entry just happen automatically as part of some other command like
> commit/add, or do they need to manually run `git sparse-checkout
> reapply`?

If the directory is gone, then their performance returns to as they
expect, since the next Git command will not need to expand the index
to detect the difference between tracked, untracked, and ignored files
within the sparse directory.

(Recall that the index is stored as sparse even if it expands to a
full one in-memory during execution.)

Thanks,
-Stolee

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Mon, Aug 30, 2021 at 6:34 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/27/2021 6:01 PM, Elijah Newren wrote:
> > Sorry, one more thing...
> >
> > On Fri, Aug 27, 2021 at 2:56 PM Elijah Newren <newren@gmail.com> wrote:
> >>
> >> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> >> <gitgitgadget@gmail.com> wrote:
> >
> >>>   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
> >>>      @@ Commit message
> >>>           the sparse directory. This depends on the indexed version of the file,
> >>>           so the sparse directory must be expanded.
> >>>
> >>>      +    We must take special care to look for untracked, non-ignored files in
> >>>      +    these directories before deleting them. We do not want to delete any
> >>>      +    meaningful work that the users were doing in those directories and
> >>>      +    perhaps forgot to add and commit before switching sparse-checkout
> >>>      +    definitions. Since those untracked files might be code files that
> >>>      +    generated ignored build output, also do not delete any ignored files
> >>>      +    from these directories in that case. The users can recover their state
> >>>      +    by resetting their sparse-checkout definition to include that directory
> >>>      +    and continue. Alternatively, they can see the warning that is presented
> >>>      +    and delete the directory themselves to regain the performance they
> >>>      +    expect.
> >
> > Will deleting the directory regain the performance they expect, or is
> > another step needed?  In other worse, will the sparsification of the
> > paths under the no-longer-needed directory into a sparse directory
> > entry just happen automatically as part of some other command like
> > commit/add, or do they need to manually run `git sparse-checkout
> > reapply`?
>
> If the directory is gone, then their performance returns to as they
> expect, since the next Git command will not need to expand the index
> to detect the difference between tracked, untracked, and ignored files
> within the sparse directory.
>
> (Recall that the index is stored as sparse even if it expands to a
> full one in-memory during execution.)

Right, as with the other email, this was the piece that hadn't quite
stuck or registered for me.  Thanks for clarifying.

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee]

On 8/27/2021 5:56 PM, Elijah Newren wrote:
> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index
>> and v2.33.0-rc1 because of some changes to dir.h that became important!
> 
> Isn't this stale (i.e. just copied from v2/v3)?

Sorry, yes. It was an update to the original cover letter, but was not
updated in later versions.

>>  * I split the commit that involves cache_tree_update(). I have not yet
>>    succeeded in creating tests to demonstrate why this is required outside
>>    of needing it in the Scalar functional tests, which includes a version of
>>    partial clone. I will continue to investigate recreating this scenario in
>>    the Git test suite, but I wanted to send this version out to get feedback
>>    on the things that have changed.
> 
> Are you talking about patch 3 or patch 4?

I'm talking about patches 3 and 4, which (at the time that I created
them a few months ago, before later merging them) they each had solved
some problems in the Scalar functional tests. The issue I'm seeing now
is that since so many commands have integrated with the sparse index,
those functional tests are not testing convert_to_sparse() as heavily,
so simply reverting each does not show the problem any more.

> For patch 4, as I commented on that patch, my guess would be that you
> start with a full tree, pick some directory that you will sparsify
> away, and before sparsifying it away, change some file under that
> directory and stage the change.  Then when you sparsify the directory
> away with cone mode, I think the cache_tree_update() will need to
> write the tree.  At least, some quick greps of cache_tree_update
> suggest there aren't that many codepaths that attempt to update it, so
> I don't think anything else would have written it yet at that point.
> 
> For patch 3, since it comes before patch 5, it would be triggerable
> with the same case.  Is it possible you originally found both patches
> based off similar initial conditions that just resulted in slightly
> different behavioral reports, and thus that patch 4 is good enough
> without patch 3?  Trying to ignore that possibility and looking at the
> cache_tree_update() code, the only other reasons for failure of
> cache_tree_update() that I can see are (a) someone having invalid
> hashes recorded in the index (e.g. someone using `git update-index
> --index-info` with some invalid hashes, or someone manually deleting
> some loose blobs from under their .git/ directory), or (b) disk being
> full or read-only so that writing new tree objects to the object store
> fail.

Just for full context:

* Patch 3 removes a warning and "return -1" when cache_tree_update()
  fails. This makes convert_to_sparse() resilient to any failure that
  might happen in cache_tree_update(), whatever it might be.

* Patch 4 adds the WRITE_TREE_MISSING_OK, which removes one possible
  reason that cache_tree_update() could fail (and removes an attempted
  promisor fetch in the case of partial clone).

So I could see that either of the two patches would be sufficient to
make tests pass in all normally-testable scenarios. However, patch 3
is valuable for possibly more complicated failure scenarios such as
corrupted data, while patch 4 is valuable for performance reasons in
a very specific environment.

>> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
>> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4
>> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4
>> Pull-Request: https://github.com/gitgitgadget/git/pull/1009
> 
> A super micro nit:
> 
> It appears this is based on your other branches that have since been
> incorporated into master.  Not a big deal, but rebasing on master
> might make it easier for others to take a look at the series (I was
> confused at first why this series depended on other unmerged series,
> before realizing those other series had since merged down).  I know we
> normally don't like rebasing series, but I suspect it wouldn't change
> the range-diff (as I don't think others are touching these areas of
> the code), and thus be transparent to Junio.

The 'git add' integration only recently hit 'master', so I have not
had a chance to base on a version of that branch with the necessary
dependencies.

>>   6:  c9e100e68f8 !  7:  e496f3cee66 attr: be careful about sparse directories
>>      @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat
>>                 return NULL;
>>
>>       + /*
>>      -+  * In the case of cone-mode sparse-checkout, getting the
>>      -+  * .gitattributes file from a directory is meaningless: all
>>      -+  * contained paths will be sparse if the .gitattributes is also
>>      -+  * sparse. In the case of a sparse index, it is critical that we
>>      -+  * don't go looking for one as it will expand the index.
>>      ++  * The .gitattributes file only applies to files within its
>>      ++  * parent directory. In the case of cone-mode sparse-checkout,
>>      ++  * the .gitattributes file is sparse if and only if all paths
>>      ++  * within that directory are also sparse. Thus, don't load the
>>      ++  * .gitattributes file since it will not matter.
>>      ++  *
>>      ++  * In the case of a sparse index, it is critical that we don't go
>>      ++  * looking for a .gitattributes file, as the index will expand.
> 
> This comment is better, and I appreciate the thought behind adding the
> second paragraph, but the wording seems fuzzy and potentially
> confusing to future readers.  Perhaps changing the wording to:
> 
> ...looking for a .gitattributes file, as doing so would cause the
> index to expand.

Thanks.

>>   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
>>      @@ Commit message
>>           the sparse directory. This depends on the indexed version of the file,
>>           so the sparse directory must be expanded.
>>
>>      +    We must take special care to look for untracked, non-ignored files in
>>      +    these directories before deleting them. We do not want to delete any
>>      +    meaningful work that the users were doing in those directories and
>>      +    perhaps forgot to add and commit before switching sparse-checkout
>>      +    definitions. Since those untracked files might be code files that
>>      +    generated ignored build output, also do not delete any ignored files
>>      +    from these directories in that case. The users can recover their state
>>      +    by resetting their sparse-checkout definition to include that directory
>>      +    and continue. Alternatively, they can see the warning that is presented
>>      +    and delete the directory themselves to regain the performance they
>>      +    expect.
>>      +
...
>>      ++When changing the sparse-checkout patterns in cone mode, Git will inspect each
>>      ++tracked directory that is not within the sparse-checkout cone to see if it
>>      ++contains any untracked files. If all of those files are ignored due to the
>>      ++`.gitignore` patterns, then the directory will be deleted. If any of the
>>      ++untracked files within that directory is not ignored, then no deletions will
>>      ++occur within that directory and a warning message will appear. If these files
>>      ++are important, then reset your sparse-checkout definition so they are included,
>>      ++use `git add` and `git commit` to store them, then remove any remaining files
>>      ++manually to ensure Git can behave optimally.
...
> Thanks for the extra explanations; those help.
> 
> You haven't yet addressed how this choice will interact with
> .gitignore files, though (as I commented on v3).  You alluded in patch
> 7 to how it was critical to avoid looking for .gitattributes files
> because doing so would require the index to expand, possibly
> recursively all the way down.  Don't .gitignore files have the same
> problem?  Nuking all the ignored files so that you can delete extra
> directories and thus not need to walk into them during operations
> helps, but does the fact that we sometimes avoid removing directories
> mean we have to walk into them anyway?  Do we only need to walk into
> such directories when they are (still? again?) present in the working
> copy?  Does there need to be some code added somewhere that checks for
> the presence of a directory in the working copy and unsparsifies
> sparse directory entries in the index when such directories are found?
> 
> Also, does non-cone mode sparsity paths have weird bugs around letting
> users specify patterns that would happen to also exclude .gitignore
> files from the working copy, but keep other sibling files within the
> same directory?  In particular, if the .gitignore files aren't
> present, does git add -A add a bunch of supposed-to-be-ignored files?
> (If so, that's an issue that's probably tangential to this series
> since it's not about cone mode or the sparse index, but it's certainly
> an interesting thought.)
> 
> Maybe the whole .gitignore situation with cone mode has a simple
> answer, but since this whole series is about gaining performance by
> avoiding recursing into directories corresponding to sparse directory
> entries, it feels the answers to how .gitignore will be handled should
> at least be addressed.

The goal of this series is to allow Git to operate as normal (respecting
.gitattributes and .gitignore files as part of that) without needing to
expand a sparse index. The way it achieves this is by deleting the
directories that leave the sparse-checkout cone. If those directories
don't exist, then we don't care about the .gitattributes or .gitignore
files in those locations. When the directory _does_ exist, then we need
to look for the .gitignore file which expands the index to full. This
is done naturally by Git using index_name_pos() for "<dir>/.gitignore"
which finds that "<dir>" (or one of its parent directories) is a
sparse directory, then expands the index and searches for
"<dir>/.gitignore" again, returning the position if it exists in the
fully-expanded index.

The difference between the .gitattributes and .gitignore files is that
we don't need to do anything with the .gitignore files to prevent Git
from accessing them with their contained directories don't exist. The
.gitattributes change in attr.c is reachable from some code path even
when the directory doesn't exist, so it needs special logic to avoid
that expansion.

When we don't succeed in deleting sparse directories, performance
suffers. That's the reason behind having a very-visible warnings to
users that something is blocking the removal. We will be monitoring
performance of users where it is _very clear_ when this happens. From
that experience, we might discover new ways to assist them to get
into a better state (perhaps a warning during "git status" would help?).

Thanks,
-Stolee

Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Mon, Aug 30, 2021 at 6:54 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/27/2021 5:56 PM, Elijah Newren wrote:
> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget
> > <gitgitgadget@gmail.com> wrote:
> >>
...
> >>   8:  febef675f05 !  9:  c19d93ec5d7 sparse-checkout: clear tracked sparse dirs
> >>      @@ Commit message
> >>           the sparse directory. This depends on the indexed version of the file,
> >>           so the sparse directory must be expanded.
> >>
> >>      +    We must take special care to look for untracked, non-ignored files in
> >>      +    these directories before deleting them. We do not want to delete any
> >>      +    meaningful work that the users were doing in those directories and
> >>      +    perhaps forgot to add and commit before switching sparse-checkout
> >>      +    definitions. Since those untracked files might be code files that
> >>      +    generated ignored build output, also do not delete any ignored files
> >>      +    from these directories in that case. The users can recover their state
> >>      +    by resetting their sparse-checkout definition to include that directory
> >>      +    and continue. Alternatively, they can see the warning that is presented
> >>      +    and delete the directory themselves to regain the performance they
> >>      +    expect.
> >>      +
> ...
> >>      ++When changing the sparse-checkout patterns in cone mode, Git will inspect each
> >>      ++tracked directory that is not within the sparse-checkout cone to see if it
> >>      ++contains any untracked files. If all of those files are ignored due to the
> >>      ++`.gitignore` patterns, then the directory will be deleted. If any of the
> >>      ++untracked files within that directory is not ignored, then no deletions will
> >>      ++occur within that directory and a warning message will appear. If these files
> >>      ++are important, then reset your sparse-checkout definition so they are included,
> >>      ++use `git add` and `git commit` to store them, then remove any remaining files
> >>      ++manually to ensure Git can behave optimally.
> ...
> > Thanks for the extra explanations; those help.
> >
> > You haven't yet addressed how this choice will interact with
> > .gitignore files, though (as I commented on v3).
...
> The difference between the .gitattributes and .gitignore files is that
> we don't need to do anything with the .gitignore files to prevent Git
> from accessing them with their contained directories don't exist. The
> .gitattributes change in attr.c is reachable from some code path even
> when the directory doesn't exist, so it needs special logic to avoid
> that expansion.

Ah, this is what I was looking for and couldn't quite work out --
whether .gitignore files might need to be loaded (and whether our
other UI decisions were affecting that).  Sounds like you have it
covered.

Thanks for patiently explaining these various items I was not
understanding or missing.  Sorry I missed a couple key things despite
reading the earlier rounds.  Anyway, looks like we're in pretty good
shape and there's only a few minor things left, mostly stuff you
brought up yourself that you wanted to tweak for v5.

[PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee via GitGitGadget]

We launched an experimental release [1] of the sparse-index feature to our
internal users. We immediately discovered a problem due to the isolated way
in which we tested the sparse index: we were never building the project and
changing our sparse-checkout definition.

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Users who ran a build in one project and then moved to another still had
build artifacts in their worktree that lived inside the old directories.
Since the files are marked by the .gitignore patterns, these files were not
removed by the 'git sparse-checkout set' command. However, they make the
sparse-index unusable because every 'git status' command needs to expand the
sparse-directory entries in order to see if the files are tracked or not.
This made the first experimental release actually slower for all users
because of this cost.

The solution we shipped to these customers was to change the way our fork
handles these ignored files. Specifically, instead of Git completely
ignoring the files, we changed Git to understand that with cone-mode
sparse-checkout patterns, the users is asking for entire directories to be
removed from the worktree. The link [1] included earlier has this change.

I believe that this is a reasonable expectation, though I recognize that it
might look like breaking the expectations of how .gitignore files work.

Since feedback demonstrated that this is a desired behavior, v2 includes
this behavior for all "cone mode" repositories.

I'm interested in the community's thoughts about this change, as it seems
like one that we should make carefully and intentionally.

While the rewrite of the t7519 test seems unrelated, it is required to avoid
a test failure with this change that deletes files outside of the cone. By
moving the test into repositories not at $TRASH_DIRECTORY, we gain more
control over the repository structure.


Updates in V5
=============

 * Updated the locality of a cache_entry pointer.

 * Rephrased a comment.

 * Removed the patch adding a config option.

 * I tried, but failed, to create a scenario where the call to
   cache_tree_update() causes a test to fail. I still think this is valuable
   as defensive programming for the reasons mentioned in the patch, which is
   why I didn't remove them here.


Updates in V4
=============

 * Fixed an issue with the split index.

 * The helper methods are used more consistently.

 * The helper method path_in_cone_mode_sparse_checkout() is introduced.

 * Commit messages are edited for clarity.

 * A new config option is added to disable the behavior being added in this
   series.

 * I split the commit that involves cache_tree_update(). I have not yet
   succeeded in creating tests to demonstrate why this is required outside
   of needing it in the Scalar functional tests, which includes a version of
   partial clone. I will continue to investigate recreating this scenario in
   the Git test suite, but I wanted to send this version out to get feedback
   on the things that have changed.


Update in V3
============

 * As promised [2], the helper methods are fixed to work with non-cone-mode
   patterns. A later series will use them to their fullest potential
   (changing git add, git rm, and git mv when interacting with sparse
   entries).

[2]
https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/


Updates in V2
=============

 * This version correctly leaves untracked files alone. If untracked files
   are found, then the directory is left as-is, in case those ignored files
   are important to the user's work resolving those untracked files.

 * This behavior is now enabled by core.sparseCheckoutCone=true.

 * To use a sparse index as an in-memory data structure even when
   index.sparse is disabled, a new patch is included to modify the prototype
   of convert_to_sparse() to include a flags parameter.

 * A few cleanup patches that I was collecting based on feedback from the
   experimental release and intending for my next series were necessary for
   this implementation.

 * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
   implementation that used run_subcommand().

Thanks, -Stolee

Derrick Stolee (9):
  t7519: rewrite sparse index test
  sparse-index: silently return when not using cone-mode patterns
  unpack-trees: fix nested sparse-dir search
  sparse-index: silently return when cache tree fails
  sparse-index: use WRITE_TREE_MISSING_OK
  sparse-checkout: create helper methods
  attr: be careful about sparse directories
  sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
  sparse-checkout: clear tracked sparse dirs

 Documentation/git-sparse-checkout.txt | 10 +++
 attr.c                                | 15 +++++
 builtin/add.c                         |  7 +-
 builtin/sparse-checkout.c             | 94 +++++++++++++++++++++++++++
 dir.c                                 | 52 +++++++++++++++
 dir.h                                 |  8 +++
 read-cache.c                          |  4 +-
 sparse-index.c                        | 76 ++++++++++++----------
 sparse-index.h                        |  3 +-
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 t/t7519-status-fsmonitor.sh           | 38 ++++++-----
 unpack-trees.c                        |  8 ++-
 12 files changed, 312 insertions(+), 62 deletions(-)


base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1009

Range-diff vs v4:

  1:  c407b2cb346 =  1:  c407b2cb346 t7519: rewrite sparse index test
  2:  8660877ba7a =  2:  8660877ba7a sparse-index: silently return when not using cone-mode patterns
  5:  acdded0f762 !  3:  edb00d3f9aa unpack-trees: fix nested sparse-dir search
     @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,
       {
      -	struct cache_entry *ce;
      +	const char *path;
     -+	struct cache_entry *ce = NULL;
       	int pos = find_cache_pos(info, p->path, p->pathlen);
       	struct unpack_trees_options *o = info->data;
       
      @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
     + 	 * paths (e.g. "subdir-").
     + 	 */
       	while (pos >= 0) {
     - 		ce = o->src_index->cache[pos];
     +-		ce = o->src_index->cache[pos];
     ++		struct cache_entry *ce = o->src_index->cache[pos];
       
      -		if (strncmp(ce->name, p->path, p->pathlen))
      +		if (!skip_prefix(ce->name, info->traverse_path, &path) ||
  3:  a669740af9a =  4:  c8620de61eb sparse-index: silently return when cache tree fails
  4:  b379b8fc61a =  5:  37171612424 sparse-index: use WRITE_TREE_MISSING_OK
  6:  1958751aa0e =  6:  98b4cae2297 sparse-checkout: create helper methods
  7:  e496f3cee66 !  7:  6ec3cb2042e attr: be careful about sparse directories
     @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat
      +	 * .gitattributes file since it will not matter.
      +	 *
      +	 * In the case of a sparse index, it is critical that we don't go
     -+	 * looking for a .gitattributes file, as the index will expand.
     ++	 * looking for a .gitattributes file, as doing so would cause the
     ++	 * index to expand.
      +	 */
      +	if (!path_in_cone_mode_sparse_checkout(path, istate))
      +		return NULL;
  8:  cab9360b1e9 =  8:  d57f48c445c sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
  9:  c19d93ec5d7 =  9:  91b53f20109 sparse-checkout: clear tracked sparse dirs
 10:  8d55a6ba2fd <  -:  ----------- sparse-checkout: add config to disable deleting dirs

-- 
gitgitgadget

[PATCH v5 1/9] t7519: rewrite sparse index test

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The sparse index is tested with the FS Monitor hook and extension since
f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test
was very fragile because it shared an index across sparse and non-sparse
behavior. Since that expansion and contraction could cause the index to
lose its FS Monitor bitmap and token, behavior is fragile to changes in
'git sparse-checkout set'.

Rewrite the test to use two clones of the original repo: full and
sparse. This allows us to also keep the test files (actual, expect,
trace2.txt) out of the repos we are testing with 'git status'.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index deea88d4431..f1463197b99 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '
 # If "!" is supplied, then we verify that we do not call ensure_full_index
 # during a call to 'git status'. Otherwise, we verify that we _do_ call it.
 check_sparse_index_behavior () {
-	git status --porcelain=v2 >expect &&
-	git sparse-checkout init --cone --sparse-index &&
-	git sparse-checkout set dir1 dir2 &&
+	git -C full status --porcelain=v2 >expect &&
 	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-		git status --porcelain=v2 >actual &&
+		git -C sparse status --porcelain=v2 >actual &&
 	test_region $1 index ensure_full_index trace2.txt &&
 	test_region fsm_hook query trace2.txt &&
 	test_cmp expect actual &&
-	rm trace2.txt &&
-	git sparse-checkout disable
+	rm trace2.txt
 }
 
 test_expect_success 'status succeeds with sparse index' '
-	git reset --hard &&
+	git clone . full &&
+	git clone --sparse . sparse &&
+	git -C sparse sparse-checkout init --cone --sparse-index &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
-	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" &&
-	check_sparse_index_behavior ! &&
-
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 	EOF
-	git config core.fsmonitor .git/hooks/fsmonitor-test &&
+	git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&
+	git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&
 	check_sparse_index_behavior ! &&
 
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1/modified\0"
 	EOF
 	check_sparse_index_behavior ! &&
 
-	cp -r dir1 dir1a &&
-	git add dir1a &&
-	git commit -m "add dir1a" &&
+	git -C sparse sparse-checkout add dir1a &&
+
+	for repo in full sparse
+	do
+		cp -r $repo/dir1 $repo/dir1a &&
+		git -C $repo add dir1a &&
+		git -C $repo commit -m "add dir1a" || return 1
+	done &&
+	git -C sparse sparse-checkout set dir1 dir2 &&
 
 	# This one modifies outside the sparse-checkout definition
 	# and hence we expect to expand the sparse-index.
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	write_script .git/hooks/fsmonitor-test <<-\EOF &&
 		printf "last_update_token\0"
 		printf "dir1a/modified\0"
 	EOF
-- 
gitgitgadget

[PATCH v5 2/9] sparse-index: silently return when not using cone-mode patterns

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

While the sparse-index is only enabled when core.sparseCheckoutCone is
also enabled, it is possible for the user to modify the sparse-checkout
file manually in a way that does not match cone-mode patterns. In this
case, we should refuse to convert an index into a sparse index, since
the sparse_checkout_patterns will not be initialized with recursive and
parent path hashsets.

Also silently return if there are no cache entries, which is a simple
case: there are no paths to make sparse!

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index c6b4feec413..cd6e0d5f408 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate)
 int convert_to_sparse(struct index_state *istate)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index ||
+	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
@@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate)
 			return 0;
 	}
 
-	if (!istate->sparse_checkout_patterns->use_cone_patterns) {
-		warning(_("attempting to use sparse-index without cone mode"));
-		return -1;
-	}
+	/*
+	 * We need cone-mode patterns to use sparse-index. If a user edits
+	 * their sparse-checkout file manually, then we can detect during
+	 * parsing that they are not actually using cone-mode patterns and
+	 * hence we need to abort this conversion _without error_. Warnings
+	 * already exist in the pattern parsing to inform the user of their
+	 * bad patterns.
+	 */
+	if (!istate->sparse_checkout_patterns->use_cone_patterns)
+		return 0;
 
 	/*
 	 * NEEDSWORK: If we have unmerged entries, then stay full.
-- 
gitgitgadget

[PATCH v5 3/9] unpack-trees: fix nested sparse-dir search

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The iterated search in find_cache_entry() was recently modified to
include a loop that searches backwards for a sparse directory entry that
matches the given traverse_info and name_entry. However, the string
comparison failed to actually concatenate those two strings, so this
failed to find a sparse directory when it was not a top-level directory.

This caused some errors in rare cases where a 'git checkout' spanned a
diff that modified files within the sparse directory entry, but we could
not correctly find the entry.

Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Helped-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 unpack-trees.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 5786645f315..b78d7ee6c09 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1255,7 +1255,7 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,
 static struct cache_entry *find_cache_entry(struct traverse_info *info,
 					    const struct name_entry *p)
 {
-	struct cache_entry *ce;
+	const char *path;
 	int pos = find_cache_pos(info, p->path, p->pathlen);
 	struct unpack_trees_options *o = info->data;
 
@@ -1281,9 +1281,11 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,
 	 * paths (e.g. "subdir-").
 	 */
 	while (pos >= 0) {
-		ce = o->src_index->cache[pos];
+		struct cache_entry *ce = o->src_index->cache[pos];
 
-		if (strncmp(ce->name, p->path, p->pathlen))
+		if (!skip_prefix(ce->name, info->traverse_path, &path) ||
+		    strncmp(path, p->path, p->pathlen) ||
+		    path[p->pathlen] != '/')
 			return NULL;
 
 		if (S_ISSPARSEDIR(ce->ce_mode) &&
-- 
gitgitgadget

[PATCH v5 4/9] sparse-index: silently return when cache tree fails

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

If cache_tree_update() returns a non-zero value, then it could not
create the cache tree. This is likely due to a path having a merge
conflict. Since we are already returning early, let's return silently to
avoid making it seem like we failed to write the index at all.

If we remove our dependence on the cache tree within
convert_to_sparse(), then we could still recover from this scenario and
have a sparse index.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/sparse-index.c b/sparse-index.c
index cd6e0d5f408..d9b07695953 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate)
 
 	/* Clear and recompute the cache-tree */
 	cache_tree_free(&istate->cache_tree);
-	if (cache_tree_update(istate, 0)) {
-		warning(_("unable to update cache-tree, staying full"));
-		return -1;
-	}
+	/*
+	 * Silently return if there is a problem with the cache tree update,
+	 * which might just be due to a conflict state in some entry.
+	 */
+	if (cache_tree_update(istate, 0))
+		return 0;
 
 	remove_fsmonitor(istate);
 
-- 
gitgitgadget

[PATCH v5 5/9] sparse-index: use WRITE_TREE_MISSING_OK

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When updating the cache tree in convert_to_sparse(), the
WRITE_TREE_MISSING_OK flag indicates that trees might be computed that
do not already exist within the object database. This happens in cases
such as 'git add' creating new trees that it wants to store in
anticipation of a following 'git commit'. If this flag is not specified,
then it might trigger a promisor fetch or a failure due to the object
not existing locally.

Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these
possible reasons for the cache_tree_update() to fail.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 sparse-index.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sparse-index.c b/sparse-index.c
index d9b07695953..880c5f72338 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)
 	/*
 	 * Silently return if there is a problem with the cache tree update,
 	 * which might just be due to a conflict state in some entry.
+	 *
+	 * This might create new tree objects, so be sure to use
+	 * WRITE_TREE_MISSING_OK.
 	 */
-	if (cache_tree_update(istate, 0))
+	if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))
 		return 0;
 
 	remove_fsmonitor(istate);
-- 
gitgitgadget

[PATCH v5 6/9] sparse-checkout: create helper methods

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

As we integrate the sparse index into more builtins, we occasionally
need to check the sparse-checkout patterns to see if a path is within
the sparse-checkout cone. Create some helper methods that help
initialize the patterns and check for pattern matching to make this
easier.

The existing callers of commands like get_sparse_checkout_patterns() use
a custom 'struct pattern_list' that is not necessarily the one in the
'struct index_state', so there are not many previous uses that could
adopt these helpers. There are just two in builtin/add.c and
sparse-index.c that can use path_in_sparse_checkout().

We add a path_in_cone_mode_sparse_checkout() as well that will only
return false if the path is outside of the sparse-checkout definition
_and_ the sparse-checkout patterns are in cone mode.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 builtin/add.c  |  7 +------
 dir.c          | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++
 dir.h          |  8 ++++++++
 sparse-index.c | 14 +++-----------
 4 files changed, 64 insertions(+), 17 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 17528e8f922..88a6c0c69fb 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;
 	int flags = REFRESH_IGNORE_SKIP_WORKTREE |
 		    (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);
-	struct pattern_list pl = { 0 };
-	int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);
 
 	seen = xcalloc(pathspec->nr, 1);
 	refresh_index(&the_index, flags, pathspec, seen,
@@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec)
 	for (i = 0; i < pathspec->nr; i++) {
 		if (!seen[i]) {
 			const char *path = pathspec->items[i].original;
-			int dtype = DT_REG;
 
 			if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||
-			    (sparse_checkout_enabled &&
-			     !path_matches_pattern_list(path, strlen(path), NULL,
-							&dtype, &pl, &the_index))) {
+			    !path_in_sparse_checkout(path, &the_index)) {
 				string_list_append(&only_match_skip_worktree,
 						   pathspec->items[i].original);
 			} else {
diff --git a/dir.c b/dir.c
index 03c4d212672..86afa2eae00 100644
--- a/dir.c
+++ b/dir.c
@@ -1439,6 +1439,58 @@ done:
 	return result;
 }
 
+int init_sparse_checkout_patterns(struct index_state *istate)
+{
+	if (!core_apply_sparse_checkout)
+		return 1;
+	if (istate->sparse_checkout_patterns)
+		return 0;
+
+	CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);
+
+	if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {
+		FREE_AND_NULL(istate->sparse_checkout_patterns);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int path_in_sparse_checkout_1(const char *path,
+				     struct index_state *istate,
+				     int require_cone_mode)
+{
+	const char *base;
+	int dtype = DT_REG;
+
+	/*
+	 * We default to accepting a path if there are no patterns or
+	 * they are of the wrong type.
+	 */
+	if (init_sparse_checkout_patterns(istate) ||
+	    (require_cone_mode &&
+	     !istate->sparse_checkout_patterns->use_cone_patterns))
+		return 1;
+
+	base = strrchr(path, '/');
+	return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,
+					 &dtype,
+					 istate->sparse_checkout_patterns,
+					 istate) > 0;
+}
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate)
+{
+	return path_in_sparse_checkout_1(path, istate, 0);
+}
+
+int path_in_cone_mode_sparse_checkout(const char *path,
+				     struct index_state *istate)
+{
+	return path_in_sparse_checkout_1(path, istate, 1);
+}
+
 static struct path_pattern *last_matching_pattern_from_lists(
 		struct dir_struct *dir, struct index_state *istate,
 		const char *pathname, int pathlen,
diff --git a/dir.h b/dir.h
index b3e1a54a971..6823312521e 100644
--- a/dir.h
+++ b/dir.h
@@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,
 				const char *basename, int *dtype,
 				struct pattern_list *pl,
 				struct index_state *istate);
+
+int init_sparse_checkout_patterns(struct index_state *state);
+
+int path_in_sparse_checkout(const char *path,
+			    struct index_state *istate);
+int path_in_cone_mode_sparse_checkout(const char *path,
+				      struct index_state *istate);
+
 struct dir_entry *dir_add_ignored(struct dir_struct *dir,
 				  struct index_state *istate,
 				  const char *pathname, int len);
diff --git a/sparse-index.c b/sparse-index.c
index 880c5f72338..23f7c3bd361 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate,
 {
 	int i, can_convert = 1;
 	int start_converted = num_converted;
-	enum pattern_match_result match;
-	int dtype = DT_UNKNOWN;
 	struct strbuf child_path = STRBUF_INIT;
-	struct pattern_list *pl = istate->sparse_checkout_patterns;
 
 	/*
 	 * Is the current path outside of the sparse cone?
 	 * Then check if the region can be replaced by a sparse
 	 * directory entry (everything is sparse and merged).
 	 */
-	match = path_matches_pattern_list(ct_path, ct_pathlen,
-					  NULL, &dtype, pl, istate);
-	if (match != NOT_MATCHED)
+	if (path_in_sparse_checkout(ct_path, istate))
 		can_convert = 0;
 
 	for (i = start; can_convert && i < end; i++) {
@@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate)
 	if (!istate->repo->settings.sparse_index)
 		return 0;
 
-	if (!istate->sparse_checkout_patterns) {
-		istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));
-		if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)
-			return 0;
-	}
+	if (init_sparse_checkout_patterns(istate))
+		return 0;
 
 	/*
 	 * We need cone-mode patterns to use sparse-index. If a user edits
-- 
gitgitgadget

[PATCH v5 7/9] attr: be careful about sparse directories

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 attr.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/attr.c b/attr.c
index d029e681f28..79adaa50ea1 100644
--- a/attr.c
+++ b/attr.c
@@ -14,6 +14,7 @@
 #include "utf8.h"
 #include "quote.h"
 #include "thread-utils.h"
+#include "dir.h"
 
 const char git_attr__true[] = "(builtin)true";
 const char git_attr__false[] = "\0(builtin)false";
@@ -744,6 +745,20 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,
 	if (!istate)
 		return NULL;
 
+	/*
+	 * The .gitattributes file only applies to files within its
+	 * parent directory. In the case of cone-mode sparse-checkout,
+	 * the .gitattributes file is sparse if and only if all paths
+	 * within that directory are also sparse. Thus, don't load the
+	 * .gitattributes file since it will not matter.
+	 *
+	 * In the case of a sparse index, it is critical that we don't go
+	 * looking for a .gitattributes file, as doing so would cause the
+	 * index to expand.
+	 */
+	if (!path_in_cone_mode_sparse_checkout(path, istate))
+		return NULL;
+
 	buf = read_blob_data_from_index(istate, path, NULL);
 	if (!buf)
 		return NULL;
-- 
gitgitgadget

[PATCH v5 8/9] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX
environment variable or the "index.sparse" config setting before
converting the index to a sparse one. This is for ease of use since all
current consumers are preparing to compress the index before writing it
to disk. If these settings are not enabled, then convert_to_sparse()
silently returns without doing anything.

We will add a consumer in the next change that wants to use the sparse
index as an in-memory data structure, regardless of whether the on-disk
format should be sparse.

To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip
these config checks when enabled. All current consumers are modified to
pass '0' in the new 'flags' parameter.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 read-cache.c   |  4 ++--
 sparse-index.c | 37 ++++++++++++++++++++++---------------
 sparse-index.h |  3 ++-
 3 files changed, 26 insertions(+), 18 deletions(-)

diff --git a/read-cache.c b/read-cache.c
index 9048ef9e905..f5d4385c408 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
 	int ret;
 	int was_full = !istate->sparse_index;
 
-	ret = convert_to_sparse(istate);
+	ret = convert_to_sparse(istate, 0);
 
 	if (ret) {
 		warning(_("failed to convert to a sparse-index"));
@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,
 	int ret, was_full = !istate->sparse_index;
 
 	move_cache_to_base_index(istate);
-	convert_to_sparse(istate);
+	convert_to_sparse(istate, 0);
 
 	trace2_region_enter_printf("index", "shared/do_write_index",
 				   the_repository, "%s", get_tempfile_path(*temp));
diff --git a/sparse-index.c b/sparse-index.c
index 23f7c3bd361..0bc45f60ac5 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate)
 	return 0;
 }
 
-int convert_to_sparse(struct index_state *istate)
+int convert_to_sparse(struct index_state *istate, int flags)
 {
 	int test_env;
-	if (istate->split_index || istate->sparse_index || !istate->cache_nr ||
+	if (istate->sparse_index || !istate->cache_nr ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
 
 	if (!istate->repo)
 		istate->repo = the_repository;
 
-	/*
-	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
-	 * index.sparse config variable to be on.
-	 */
-	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
-	if (test_env >= 0)
-		set_sparse_index_config(istate->repo, test_env);
+	if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {
+		/*
+		 * The sparse index is not (yet) integrated with a split index.
+		 */
+		if (istate->split_index)
+			return 0;
+		/*
+		 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
+		 * index.sparse config variable to be on.
+		 */
+		test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
+		if (test_env >= 0)
+			set_sparse_index_config(istate->repo, test_env);
 
-	/*
-	 * Only convert to sparse if index.sparse is set.
-	 */
-	prepare_repo_settings(istate->repo);
-	if (!istate->repo->settings.sparse_index)
-		return 0;
+		/*
+		 * Only convert to sparse if index.sparse is set.
+		 */
+		prepare_repo_settings(istate->repo);
+		if (!istate->repo->settings.sparse_index)
+			return 0;
+	}
 
 	if (init_sparse_checkout_patterns(istate))
 		return 0;
diff --git a/sparse-index.h b/sparse-index.h
index 1115a0d7dd9..9f3d7bc7faf 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -2,7 +2,8 @@
 #define SPARSE_INDEX_H__
 
 struct index_state;
-int convert_to_sparse(struct index_state *istate);
+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)
+int convert_to_sparse(struct index_state *istate, int flags);
 
 /*
  * Some places in the codebase expect to search for a specific path.
-- 
gitgitgadget

[PATCH v5 9/9] sparse-checkout: clear tracked sparse dirs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

When changing the scope of a sparse-checkout using cone mode, we might
have some tracked directories go out of scope. The current logic removes
the tracked files from within those directories, but leaves the ignored
files within those directories. This is a bit unexpected to users who
have given input to Git saying they don't need those directories
anymore.

This is something that is new to the cone mode pattern type: the user
has explicitly said "I want these directories and _not_ those
directories." The typical sparse-checkout patterns more generally apply
to "I want files with with these patterns" so it is natural to leave
ignored files as they are. This focus on directories in cone mode
provides us an opportunity to change the behavior.

Leaving these ignored files in the sparse directories makes it
impossible to gain performance benefits in the sparse index. When we
track into these directories, we need to know if the files are ignored
or not, which might depend on the _tracked_ .gitignore file(s) within
the sparse directory. This depends on the indexed version of the file,
so the sparse directory must be expanded.

We must take special care to look for untracked, non-ignored files in
these directories before deleting them. We do not want to delete any
meaningful work that the users were doing in those directories and
perhaps forgot to add and commit before switching sparse-checkout
definitions. Since those untracked files might be code files that
generated ignored build output, also do not delete any ignored files
from these directories in that case. The users can recover their state
by resetting their sparse-checkout definition to include that directory
and continue. Alternatively, they can see the warning that is presented
and delete the directory themselves to regain the performance they
expect.

By deleting the sparse directories when changing scope (or running 'git
sparse-checkout reapply') we regain these performance benefits as if the
repository was in a clean state.

Since these ignored files are frequently build output or helper files
from IDEs, the users should not need the files now that the tracked
files are removed. If the tracked files reappear, then they will have
newer timestamps than the build artifacts, so the artifacts will need to
be regenerated anyway.

Use the sparse-index as a data structure in order to find the sparse
directories that can be safely deleted. Re-expand the index to a full
one if it was full before.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt | 10 +++
 builtin/sparse-checkout.c             | 94 +++++++++++++++++++++++++++
 t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
 3 files changed, 163 insertions(+)

diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index fdcf43f87cb..42056ee9ff9 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the
 'git sparse-checkout set' command to reflect the expected cone in the working
 directory.
 
+When changing the sparse-checkout patterns in cone mode, Git will inspect each
+tracked directory that is not within the sparse-checkout cone to see if it
+contains any untracked files. If all of those files are ignored due to the
+`.gitignore` patterns, then the directory will be deleted. If any of the
+untracked files within that directory is not ignored, then no deletions will
+occur within that directory and a warning message will appear. If these files
+are important, then reset your sparse-checkout definition so they are included,
+use `git add` and `git commit` to store them, then remove any remaining files
+manually to ensure Git can behave optimally.
+
 
 SUBMODULES
 ----------
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 8ba9f13787b..d0f5c4702be 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv)
 	return 0;
 }
 
+static void clean_tracked_sparse_directories(struct repository *r)
+{
+	int i, was_full = 0;
+	struct strbuf path = STRBUF_INIT;
+	size_t pathlen;
+	struct string_list_item *item;
+	struct string_list sparse_dirs = STRING_LIST_INIT_DUP;
+
+	/*
+	 * If we are not using cone mode patterns, then we cannot
+	 * delete directories outside of the sparse cone.
+	 */
+	if (!r || !r->index || !r->worktree)
+		return;
+	if (init_sparse_checkout_patterns(r->index) ||
+	    !r->index->sparse_checkout_patterns->use_cone_patterns)
+		return;
+
+	/*
+	 * Use the sparse index as a data structure to assist finding
+	 * directories that are safe to delete. This conversion to a
+	 * sparse index will not delete directories that contain
+	 * conflicted entries or submodules.
+	 */
+	if (!r->index->sparse_index) {
+		/*
+		 * If something, such as a merge conflict or other concern,
+		 * prevents us from converting to a sparse index, then do
+		 * not try deleting files.
+		 */
+		if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))
+			return;
+		was_full = 1;
+	}
+
+	strbuf_addstr(&path, r->worktree);
+	strbuf_complete(&path, '/');
+	pathlen = path.len;
+
+	/*
+	 * Collect directories that have gone out of scope but also
+	 * exist on disk, so there is some work to be done. We need to
+	 * store the entries in a list before exploring, since that might
+	 * expand the sparse-index again.
+	 */
+	for (i = 0; i < r->index->cache_nr; i++) {
+		struct cache_entry *ce = r->index->cache[i];
+
+		if (S_ISSPARSEDIR(ce->ce_mode) &&
+		    repo_file_exists(r, ce->name))
+			string_list_append(&sparse_dirs, ce->name);
+	}
+
+	for_each_string_list_item(item, &sparse_dirs) {
+		struct dir_struct dir = DIR_INIT;
+		struct pathspec p = { 0 };
+		struct strvec s = STRVEC_INIT;
+
+		strbuf_setlen(&path, pathlen);
+		strbuf_addstr(&path, item->string);
+
+		dir.flags |= DIR_SHOW_IGNORED_TOO;
+
+		setup_standard_excludes(&dir);
+		strvec_push(&s, path.buf);
+
+		parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);
+		fill_directory(&dir, r->index, &p);
+
+		if (dir.nr) {
+			warning(_("directory '%s' contains untracked files,"
+				  " but is not in the sparse-checkout cone"),
+				item->string);
+		} else if (remove_dir_recursively(&path, 0)) {
+			/*
+			 * Removal is "best effort". If something blocks
+			 * the deletion, then continue with a warning.
+			 */
+			warning(_("failed to remove directory '%s'"),
+				item->string);
+		}
+
+		dir_clear(&dir);
+	}
+
+	string_list_clear(&sparse_dirs, 0);
+	strbuf_release(&path);
+
+	if (was_full)
+		ensure_full_index(r->index);
+}
+
 static int update_working_directory(struct pattern_list *pl)
 {
 	enum update_sparsity_result result;
@@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl)
 	else
 		rollback_lock_file(&lock_file);
 
+	clean_tracked_sparse_directories(r);
+
 	r->index->sparse_checkout_patterns = NULL;
 	return result;
 }
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..71236981e64 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '
 	check_files repo/deep a deeper1
 '
 
+test_expect_success 'cone mode clears ignored subdirectories' '
+	rm repo/.git/info/sparse-checkout &&
+
+	git -C repo sparse-checkout init --cone &&
+	git -C repo sparse-checkout set deep/deeper1 &&
+
+	cat >repo/.gitignore <<-\EOF &&
+	obj/
+	*.o
+	EOF
+
+	git -C repo add .gitignore &&
+	git -C repo commit -m ".gitignore" &&
+
+	mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&
+	for file in folder1/obj/a obj/a folder1/file.o folder1.o \
+		    deep/deeper2/obj/a deep/deeper2/file.o file.o
+	do
+		echo ignored >repo/$file || return 1
+	done &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout reapply &&
+	test_path_is_missing repo/folder1 &&
+	test_path_is_missing repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	test_must_be_empty out &&
+
+	git -C repo sparse-checkout set deep/deeper2 &&
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/deep/deeper2 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	>repo/deep/deeper2/ignored.o &&
+	>repo/deep/deeper2/untracked &&
+
+	# When an untracked file is in the way, all untracked files
+	# (even ignored files) are preserved.
+	git -C repo sparse-checkout set folder1 2>err &&
+	grep "contains untracked files" err &&
+	test_path_is_file repo/deep/deeper2/ignored.o &&
+	test_path_is_file repo/deep/deeper2/untracked &&
+
+	# The rest of the cone matches expectation
+	test_path_is_missing repo/deep/deeper1 &&
+	test_path_is_dir repo/obj &&
+	test_path_is_file repo/file.o &&
+
+	git -C repo status --porcelain=v2 >out &&
+	echo "? deep/deeper2/untracked" >expect &&
+	test_cmp expect out
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Updates in V5
> =============
>
>  * Updated the locality of a cache_entry pointer.
>
>  * Rephrased a comment.
>
>  * Removed the patch adding a config option.
>
>  * I tried, but failed, to create a scenario where the call to
>    cache_tree_update() causes a test to fail. I still think this is valuable
>    as defensive programming for the reasons mentioned in the patch, which is
>    why I didn't remove them here.

OK.  

The 'sg/test-split-index-fix' topic that has been queued in 'seen'
textually depended on the earlier iteration of this topic, so I'll
discard it while replacing this topic.

Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> Updates in V5
>> =============
>>
>>  * Updated the locality of a cache_entry pointer.
>>
>>  * Rephrased a comment.
>>
>>  * Removed the patch adding a config option.
>>
>>  * I tried, but failed, to create a scenario where the call to
>>    cache_tree_update() causes a test to fail. I still think this is valuable
>>    as defensive programming for the reasons mentioned in the patch, which is
>>    why I didn't remove them here.
>
> OK.  
>
> The 'sg/test-split-index-fix' topic that has been queued in 'seen'
> textually depended on the earlier iteration of this topic, so I'll
> discard it while replacing this topic.

I rebased the dependent topic myself, but I would not be surprised
if I weren't careful enough and introduced a new bug.  Please double
check when a new 'seen' is pushed out in several hours.

Thanks.

Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Derrick Stolee]

On 9/8/2021 2:56 AM, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>>
>>> Updates in V5
>>> =============
>>>
>>>  * Updated the locality of a cache_entry pointer.
>>>
>>>  * Rephrased a comment.
>>>
>>>  * Removed the patch adding a config option.
>>>
>>>  * I tried, but failed, to create a scenario where the call to
>>>    cache_tree_update() causes a test to fail. I still think this is valuable
>>>    as defensive programming for the reasons mentioned in the patch, which is
>>>    why I didn't remove them here.
>>
>> OK.  
>>
>> The 'sg/test-split-index-fix' topic that has been queued in 'seen'
>> textually depended on the earlier iteration of this topic, so I'll
>> discard it while replacing this topic.
> 
> I rebased the dependent topic myself, but I would not be surprised
> if I weren't careful enough and introduced a new bug.  Please double
> check when a new 'seen' is pushed out in several hours.

Did you have any conflicts? I don't think this version changed any
of the textual dependencies. Anyway, the test suite passes with
GIT_TEST_SPARSE_INDEX=1 on my machine at 'seen' (except for an issue
with t7900-maintenance.sh that is unrelated).

Thanks,
-Stolee

Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Junio C Hamano]

Derrick Stolee <stolee@gmail.com> writes:

>> I rebased the dependent topic myself, but I would not be surprised
>> if I weren't careful enough and introduced a new bug.  Please double
>> check when a new 'seen' is pushed out in several hours.
>
> Did you have any conflicts? I don't think this version changed any
> of the textual dependencies. Anyway, the test suite passes with
> GIT_TEST_SPARSE_INDEX=1 on my machine at 'seen' (except for an issue
> with t7900-maintenance.sh that is unrelated).

There was a bit of test refactoring in the other topic, so I
initially punted but then I found a bit of energy to look at
the conflict and realize it was not all that bad.

Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone

[Elijah Newren]

On Tue, Sep 7, 2021 at 6:42 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> We launched an experimental release [1] of the sparse-index feature to our
> internal users. We immediately discovered a problem due to the isolated way
> in which we tested the sparse index: we were never building the project and
> changing our sparse-checkout definition.
>
> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp
>
> Users who ran a build in one project and then moved to another still had
> build artifacts in their worktree that lived inside the old directories.
> Since the files are marked by the .gitignore patterns, these files were not
> removed by the 'git sparse-checkout set' command. However, they make the
> sparse-index unusable because every 'git status' command needs to expand the
> sparse-directory entries in order to see if the files are tracked or not.
> This made the first experimental release actually slower for all users
> because of this cost.
>
> The solution we shipped to these customers was to change the way our fork
> handles these ignored files. Specifically, instead of Git completely
> ignoring the files, we changed Git to understand that with cone-mode
> sparse-checkout patterns, the users is asking for entire directories to be
> removed from the worktree. The link [1] included earlier has this change.
>
> I believe that this is a reasonable expectation, though I recognize that it
> might look like breaking the expectations of how .gitignore files work.
>
> Since feedback demonstrated that this is a desired behavior, v2 includes
> this behavior for all "cone mode" repositories.
>
> I'm interested in the community's thoughts about this change, as it seems
> like one that we should make carefully and intentionally.
>
> While the rewrite of the t7519 test seems unrelated, it is required to avoid
> a test failure with this change that deletes files outside of the cone. By
> moving the test into repositories not at $TRASH_DIRECTORY, we gain more
> control over the repository structure.
>
>
> Updates in V5
> =============
>
>  * Updated the locality of a cache_entry pointer.
>
>  * Rephrased a comment.
>
>  * Removed the patch adding a config option.
>
>  * I tried, but failed, to create a scenario where the call to
>    cache_tree_update() causes a test to fail. I still think this is valuable
>    as defensive programming for the reasons mentioned in the patch, which is
>    why I didn't remove them here.
>
>
> Updates in V4
> =============
>
>  * Fixed an issue with the split index.
>
>  * The helper methods are used more consistently.
>
>  * The helper method path_in_cone_mode_sparse_checkout() is introduced.
>
>  * Commit messages are edited for clarity.
>
>  * A new config option is added to disable the behavior being added in this
>    series.
>
>  * I split the commit that involves cache_tree_update(). I have not yet
>    succeeded in creating tests to demonstrate why this is required outside
>    of needing it in the Scalar functional tests, which includes a version of
>    partial clone. I will continue to investigate recreating this scenario in
>    the Git test suite, but I wanted to send this version out to get feedback
>    on the things that have changed.
>
>
> Update in V3
> ============
>
>  * As promised [2], the helper methods are fixed to work with non-cone-mode
>    patterns. A later series will use them to their fullest potential
>    (changing git add, git rm, and git mv when interacting with sparse
>    entries).
>
> [2]
> https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/
>
>
> Updates in V2
> =============
>
>  * This version correctly leaves untracked files alone. If untracked files
>    are found, then the directory is left as-is, in case those ignored files
>    are important to the user's work resolving those untracked files.
>
>  * This behavior is now enabled by core.sparseCheckoutCone=true.
>
>  * To use a sparse index as an in-memory data structure even when
>    index.sparse is disabled, a new patch is included to modify the prototype
>    of convert_to_sparse() to include a flags parameter.
>
>  * A few cleanup patches that I was collecting based on feedback from the
>    experimental release and intending for my next series were necessary for
>    this implementation.
>
>  * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous
>    implementation that used run_subcommand().
>
> Thanks, -Stolee
>
> Derrick Stolee (9):
>   t7519: rewrite sparse index test
>   sparse-index: silently return when not using cone-mode patterns
>   unpack-trees: fix nested sparse-dir search
>   sparse-index: silently return when cache tree fails
>   sparse-index: use WRITE_TREE_MISSING_OK
>   sparse-checkout: create helper methods
>   attr: be careful about sparse directories
>   sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
>   sparse-checkout: clear tracked sparse dirs
>
>  Documentation/git-sparse-checkout.txt | 10 +++
>  attr.c                                | 15 +++++
>  builtin/add.c                         |  7 +-
>  builtin/sparse-checkout.c             | 94 +++++++++++++++++++++++++++
>  dir.c                                 | 52 +++++++++++++++
>  dir.h                                 |  8 +++
>  read-cache.c                          |  4 +-
>  sparse-index.c                        | 76 ++++++++++++----------
>  sparse-index.h                        |  3 +-
>  t/t1091-sparse-checkout-builtin.sh    | 59 +++++++++++++++++
>  t/t7519-status-fsmonitor.sh           | 38 ++++++-----
>  unpack-trees.c                        |  8 ++-
>  12 files changed, 312 insertions(+), 62 deletions(-)
>
>
> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v5
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v5
> Pull-Request: https://github.com/gitgitgadget/git/pull/1009
>
> Range-diff vs v4:
>
>   1:  c407b2cb346 =  1:  c407b2cb346 t7519: rewrite sparse index test
>   2:  8660877ba7a =  2:  8660877ba7a sparse-index: silently return when not using cone-mode patterns
>   5:  acdded0f762 !  3:  edb00d3f9aa unpack-trees: fix nested sparse-dir search
>      @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,
>        {
>       - struct cache_entry *ce;
>       + const char *path;
>      -+ struct cache_entry *ce = NULL;
>         int pos = find_cache_pos(info, p->path, p->pathlen);
>         struct unpack_trees_options *o = info->data;
>
>       @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,
>      +   * paths (e.g. "subdir-").
>      +   */
>         while (pos >= 0) {
>      -          ce = o->src_index->cache[pos];
>      +-         ce = o->src_index->cache[pos];
>      ++         struct cache_entry *ce = o->src_index->cache[pos];
>
>       -         if (strncmp(ce->name, p->path, p->pathlen))
>       +         if (!skip_prefix(ce->name, info->traverse_path, &path) ||
>   3:  a669740af9a =  4:  c8620de61eb sparse-index: silently return when cache tree fails
>   4:  b379b8fc61a =  5:  37171612424 sparse-index: use WRITE_TREE_MISSING_OK
>   6:  1958751aa0e =  6:  98b4cae2297 sparse-checkout: create helper methods
>   7:  e496f3cee66 !  7:  6ec3cb2042e attr: be careful about sparse directories
>      @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat
>       +  * .gitattributes file since it will not matter.
>       +  *
>       +  * In the case of a sparse index, it is critical that we don't go
>      -+  * looking for a .gitattributes file, as the index will expand.
>      ++  * looking for a .gitattributes file, as doing so would cause the
>      ++  * index to expand.
>       +  */
>       + if (!path_in_cone_mode_sparse_checkout(path, istate))
>       +         return NULL;
>   8:  cab9360b1e9 =  8:  d57f48c445c sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag
>   9:  c19d93ec5d7 =  9:  91b53f20109 sparse-checkout: clear tracked sparse dirs
>  10:  8d55a6ba2fd <  -:  ----------- sparse-checkout: add config to disable deleting dirs
>

Thanks for all your diligence working on this, and your patience in
explaining all the details to me in the previous rounds.  There wasn't
much left to address after the last round, and this round addresses
the few loose ends.

Reviewed-by: Elijah Newren <newren@gmail.com>