[PATCH] Provide config option to expect files outside sparse patterns

[PATCH] Provide config option to expect files outside sparse patterns

[Elijah Newren via GitGitGadget]

From: Elijah Newren <newren@gmail.com>

Typically with sparse checkouts, we expect files outside the sparsity
patterns to be marked as SKIP_WORKTREE and be missing from the working
tree.  VFS for Git can be used to turn this expectation on its head:
all files are considered present in the working copy, though they are
not vivified until first access access.  With VFS for Git, most of the
files do not match the sparsity patterns at first, and the VFS layer
automatically updates the sparsity patterns to add more files whenever
files are written.

With this background, this special usecase does not play well with the
safety check we added in commit 11d46a399d ("repo_read_index: clear
SKIP_WORKTREE bit from files present in worktree", 2022-01-06).
Checking SKIP_WORKTREE files to see if they are present in the working
tree causes them all to be immediately vivified.  Further, the special
VFS layer, by virtue of automatically updating the sparsity patterns and
catching all accesses, isn't in need of that safety check either.
Provide a configuration option, core.expectFilesOutsideSparsePatterns
so that those with this special usecase can turn off the safety check.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
    Provide config option to expect files outside sparse patterns
    
    Builds on en/present-despite-skipped, and addresses issue reported at
    https://lore.kernel.org/git/YhBCsg2DCEd9FXjE@google.com/

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1153%2Fnewren%2Fpresent-despite-skip-worktree-toggle-for-vfs-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1153/newren/present-despite-skip-worktree-toggle-for-vfs-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1153

 cache.h                          |  1 +
 config.c                         |  5 +++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 5 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/cache.h b/cache.h
index 281f00ab1b1..1f35d8a59ea 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int core_expect_files_outside_sparse_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a01..68e877a1d80 100644
--- a/config.c
+++ b/config.c
@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
+		core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.precomposeunicode")) {
 		precomposed_unicode = git_config_bool(var, value);
 		return 0;
diff --git a/environment.c b/environment.c
index fd0501e77a5..c86ebe9ce74 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int core_expect_files_outside_sparse_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f7..8e2a3973856 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    core_expect_files_outside_sparse_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb4901874..feef883523b 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -84,4 +84,23 @@ test_expect_success 'in partial clone, sparse checkout only fetches needed blobs
 	test_cmp expect actual
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config core.expectFilesOutsideSparsePatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_done

base-commit: d79d29935262f8422a6e620bd33cc2fce28e669d
-- 
gitgitgadget

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Derrick Stolee]

On 2/20/2022 12:05 AM, Elijah Newren via GitGitGadget wrote:
> From: Elijah Newren <newren@gmail.com>
> 
> Typically with sparse checkouts, we expect files outside the sparsity
> patterns to be marked as SKIP_WORKTREE and be missing from the working
> tree.  VFS for Git can be used to turn this expectation on its head:
> all files are considered present in the working copy, though they are
> not vivified until first access access.  With VFS for Git, most of the
> files do not match the sparsity patterns at first, and the VFS layer
> automatically updates the sparsity patterns to add more files whenever
> files are written.
> 
> With this background, this special usecase does not play well with the
> safety check we added in commit 11d46a399d ("repo_read_index: clear
> SKIP_WORKTREE bit from files present in worktree", 2022-01-06).
> Checking SKIP_WORKTREE files to see if they are present in the working
> tree causes them all to be immediately vivified.  Further, the special
> VFS layer, by virtue of automatically updating the sparsity patterns and
> catching all accesses, isn't in need of that safety check either.
> Provide a configuration option, core.expectFilesOutsideSparsePatterns
> so that those with this special usecase can turn off the safety check.

This patch looks like a good solution to the concerns brought up by
Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with
its own custom config to protect things like this. I imagine that we
will start setting your core_expect_files_outside_sparse_patterns
variable when reading the virtual filesystem info. We might even modify
some of our custom checks to use this variable instead. That would make
them appropriate to send upstream.

Should we update Documentation/config/core.txt describing this config
key? Or is this intended to be an internal detail only for something
like vfsd?

The only concern here really is if we want to be picky about the "VFS
for Git" references instead of "vfsd" references in the commit message.

Thanks,
-Stolee

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> This patch looks like a good solution to the concerns brought up by
> Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with
> its own custom config to protect things like this. I imagine that we
> will start setting your core_expect_files_outside_sparse_patterns
> variable when reading the virtual filesystem info. We might even modify
> some of our custom checks to use this variable instead. That would make
> them appropriate to send upstream.
>
> Should we update Documentation/config/core.txt describing this config
> key? Or is this intended to be an internal detail only for something
> like vfsd?
>
> The only concern here really is if we want to be picky about the "VFS
> for Git" references instead of "vfsd" references in the commit message.

Thanks for a quick response to make sure what our left hand is
trying to do will not interfere with what our right hand has already
been doing ;-)

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Johannes Schindelin]

Hi Elijah,

In addition to Stolee's feedback...

On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:

> diff --git a/config.c b/config.c
> index 2bffa8d4a01..68e877a1d80 100644
> --- a/config.c
> +++ b/config.c
> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>  		return 0;
>  	}
>
> +	if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
> +		core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
> +		return 0;
> +	}

The `core` section is already quite crowded (for which I am partially
responsible, of course).

Maybe it would be a good idea to introduce the `sparse` section, using
`sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
false`?

Ciao,
Dscho

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 21 2022, Johannes Schindelin wrote:

> Hi Elijah,
>
> In addition to Stolee's feedback...
>
> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
>
>> diff --git a/config.c b/config.c
>> index 2bffa8d4a01..68e877a1d80 100644
>> --- a/config.c
>> +++ b/config.c
>> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>>  		return 0;
>>  	}
>>
>> +	if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
>> +		core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
>> +		return 0;
>> +	}
>
> The `core` section is already quite crowded (for which I am partially
> responsible, of course).
>
> Maybe it would be a good idea to introduce the `sparse` section, using
> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
> false`?

There's a large list of exceptions to this, but generally we have core.*
for cross-command configuration, and <cmd>.* for command-specific
configuration.

Excetions include http.*, mailmap.*, pack.* (arguably), trace2.*.

In this case though we have 2x /^core\.sparse[A-Z].*/ variables in
git-config(1) already.

So perhaps we could name it like that, then at least you'd find all of
these in the same place when looking through that documentation,
although a name starting with "core.sparse" might not be otherwise
ideal.

There's also no law that says we can't have a three-level variable as
core.sparse.*, perhaps that would be the start of a good trend, and it
would sort adjacent to core.sparse[A-Z].* ...

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Elijah Newren]

On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee
<derrickstolee@github.com> wrote:
>
> On 2/20/2022 12:05 AM, Elijah Newren via GitGitGadget wrote:
> > From: Elijah Newren <newren@gmail.com>
> >
> > Typically with sparse checkouts, we expect files outside the sparsity
> > patterns to be marked as SKIP_WORKTREE and be missing from the working
> > tree.  VFS for Git can be used to turn this expectation on its head:
> > all files are considered present in the working copy, though they are
> > not vivified until first access access.  With VFS for Git, most of the
> > files do not match the sparsity patterns at first, and the VFS layer
> > automatically updates the sparsity patterns to add more files whenever
> > files are written.
> >
> > With this background, this special usecase does not play well with the
> > safety check we added in commit 11d46a399d ("repo_read_index: clear
> > SKIP_WORKTREE bit from files present in worktree", 2022-01-06).
> > Checking SKIP_WORKTREE files to see if they are present in the working
> > tree causes them all to be immediately vivified.  Further, the special
> > VFS layer, by virtue of automatically updating the sparsity patterns and
> > catching all accesses, isn't in need of that safety check either.
> > Provide a configuration option, core.expectFilesOutsideSparsePatterns
> > so that those with this special usecase can turn off the safety check.
>
> This patch looks like a good solution to the concerns brought up by
> Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with
> its own custom config to protect things like this. I imagine that we
> will start setting your core_expect_files_outside_sparse_patterns
> variable when reading the virtual filesystem info. We might even modify
> some of our custom checks to use this variable instead. That would make
> them appropriate to send upstream.
>
> Should we update Documentation/config/core.txt describing this config
> key? Or is this intended to be an internal detail only for something
> like vfsd?

I think it's probably only for vfsd, but vfsd users probably deserve
to be able to look it up in the documentation, so it's probably worth
documenting.   Once we figure out it's name and section, that is (as
per Dscho's email -- comments from you on that would be appreciated).

> The only concern here really is if we want to be picky about the "VFS
> for Git" references instead of "vfsd" references in the commit message.

I'm not sure I'm understanding the distinction.  Was "VFS for Git"
renamed to "vfsd"?  Is "vfsd" Google's modified version?  Something
else?

Also, vfsd doesn't seem to be Google-able whereas "VFS for Git" is,
and since it's not an internal git project, it might be nice to use a
name that users can find.

But anyway, I have no idea what the "correct" text here is (as
probably evidenced by my questions), so if anyone wants to provide
suggestions or corrections to the commit message, I'm happy to take
them.

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Elijah Newren]

On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Elijah,
>
> In addition to Stolee's feedback...
>
> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
>
> > diff --git a/config.c b/config.c
> > index 2bffa8d4a01..68e877a1d80 100644
> > --- a/config.c
> > +++ b/config.c
> > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> >               return 0;
> >       }
> >
> > +     if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
> > +             core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
> > +             return 0;
> > +     }
>
> The `core` section is already quite crowded (for which I am partially
> responsible, of course).
>
> Maybe it would be a good idea to introduce the `sparse` section, using
> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
> false`?

That's a fair point.  At one point Stolee wanted to change from
core.sparse* to sparse.* -- but by that point we already had users and
would have had to deal with a bit of a migration story (and wondering
what to do if people had both old and new config variables set
inconsistently).

I'm not sure if it's optimal to try to keep the sparse settings
together (thus put new ones under core), or try to avoid filling core.
I guess if we moved towards sparse.* now, it might be an easier
migration story if we only have two options to move.  And besides,
we're already split between multiple sections with
extensions.worktreeConfig, core.sparseCheckout{,Cone}, and
index.sparse already...so maybe adding one more section would be par
for the course.  ;-)

So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd
like to hear Stolee's thoughts too.

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Elijah Newren]

On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> On Mon, Feb 21 2022, Johannes Schindelin wrote:
>
> > Hi Elijah,
> >
> > In addition to Stolee's feedback...
> >
> > On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
> >
> >> diff --git a/config.c b/config.c
> >> index 2bffa8d4a01..68e877a1d80 100644
> >> --- a/config.c
> >> +++ b/config.c
> >> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> >>              return 0;
> >>      }
> >>
> >> +    if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
> >> +            core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
> >> +            return 0;
> >> +    }
> >
> > The `core` section is already quite crowded (for which I am partially
> > responsible, of course).
> >
> > Maybe it would be a good idea to introduce the `sparse` section, using
> > `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
> > false`?
>
> There's a large list of exceptions to this, but generally we have core.*
> for cross-command configuration, and <cmd>.* for command-specific
> configuration.
>
> Excetions include http.*, mailmap.*, pack.* (arguably), trace2.*.
>
> In this case though we have 2x /^core\.sparse[A-Z].*/ variables in
> git-config(1) already.

Those wouldn't be an exception as worded.  They are very much
cross-command configuration variables, affecting basically every
command that touches the working tree.

> So perhaps we could name it like that, then at least you'd find all of
> these in the same place when looking through that documentation,
> although a name starting with "core.sparse" might not be otherwise
> ideal.
>
> There's also no law that says we can't have a three-level variable as
> core.sparse.*, perhaps that would be the start of a good trend, and it
> would sort adjacent to core.sparse[A-Z].* ...

Interesting thought.  I'm curious what others think of this.

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 21 2022, Elijah Newren wrote:

> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>>
>> Hi Elijah,
>>
>> In addition to Stolee's feedback...
>>
>> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
>>
>> > diff --git a/config.c b/config.c
>> > index 2bffa8d4a01..68e877a1d80 100644
>> > --- a/config.c
>> > +++ b/config.c
>> > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>> >               return 0;
>> >       }
>> >
>> > +     if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
>> > +             core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
>> > +             return 0;
>> > +     }
>>
>> The `core` section is already quite crowded (for which I am partially
>> responsible, of course).
>>
>> Maybe it would be a good idea to introduce the `sparse` section, using
>> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
>> false`?
>
> That's a fair point.  At one point Stolee wanted to change from
> core.sparse* to sparse.* -- but by that point we already had users and
> would have had to deal with a bit of a migration story (and wondering
> what to do if people had both old and new config variables set
> inconsistently).
>
> I'm not sure if it's optimal to try to keep the sparse settings
> together (thus put new ones under core), or try to avoid filling core.
> I guess if we moved towards sparse.* now, it might be an easier
> migration story if we only have two options to move.  And besides,
> we're already split between multiple sections with
> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and
> index.sparse already...so maybe adding one more section would be par
> for the course.  ;-)
>
> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd
> like to hear Stolee's thoughts too.

Yes, maybe sparse.* etc. is best. I'm fine with whatever you two come up
with.

I just wanted to raise the point that it's less important to consider
names/namespaces for variables one-by-one, and more important that all
the variables in a "family" fit well together, the users can find them
all in one place etc. (ideally, not always possible).

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Johannes Schindelin]

Hi Elijah,

On Mon, 21 Feb 2022, Elijah Newren wrote:

> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > In addition to Stolee's feedback...
> >
> > On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
> >
> > > diff --git a/config.c b/config.c
> > > index 2bffa8d4a01..68e877a1d80 100644
> > > --- a/config.c
> > > +++ b/config.c
> > > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> > >               return 0;
> > >       }
> > >
> > > +     if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
> > > +             core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
> > > +             return 0;
> > > +     }
> >
> > The `core` section is already quite crowded (for which I am partially
> > responsible, of course).
> >
> > Maybe it would be a good idea to introduce the `sparse` section, using
> > `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
> > false`?
>
> That's a fair point.  At one point Stolee wanted to change from
> core.sparse* to sparse.* -- but by that point we already had users and
> would have had to deal with a bit of a migration story (and wondering
> what to do if people had both old and new config variables set
> inconsistently).

Right, migration is always hard.

And it's outside of the scope of this here patch series, of course.

> I'm not sure if it's optimal to try to keep the sparse settings
> together (thus put new ones under core), or try to avoid filling core.
> I guess if we moved towards sparse.* now, it might be an easier
> migration story if we only have two options to move.  And besides,
> we're already split between multiple sections with
> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and
> index.sparse already...so maybe adding one more section would be par
> for the course.  ;-)

FWIW as a potential #leftoverbits, we could migrate those to `sparse.*`
where `sparse.*` would take precendence over `core.sparse*` and the usual
deprecation notice would be shown via the `advice` mechanism.

> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd
> like to hear Stolee's thoughts too.

Indeed, his opinion weighs more than mine on this matter.

Ciao,
Dscho

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

Hi Elijah,

On Mon, 21 Feb 2022, Elijah Newren wrote:

> On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>
> > There's also no law that says we can't have a three-level variable as
> > core.sparse.*, perhaps that would be the start of a good trend, and it
> > would sort adjacent to core.sparse[A-Z].* ...
>
> Interesting thought.  I'm curious what others think of this.

What would make it `core`?

In other words, while it is valuable to consider this option, I don't
think that it solves anything, for the price of additional complexity.

Let's not go there.

Ciao,
Dscho

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Johannes Schindelin]

Hi Elijah,

On Mon, 21 Feb 2022, Elijah Newren wrote:

> On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee
> <derrickstolee@github.com> wrote:
>
> > The only concern here really is if we want to be picky about the "VFS
> > for Git" references instead of "vfsd" references in the commit
> > message.
>
> I'm not sure I'm understanding the distinction.  Was "VFS for Git"
> renamed to "vfsd"?  Is "vfsd" Google's modified version?  Something
> else?

VFS for Git is the existing project at
https://github.com/microsoft/VFSforGit which is pretty much in maintenance
mode.

vfsd was mentioned by Jonathan Tan in
https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/.
From what I gather, it is a completely separate implementation of the same
ideas of VFS for Git, but from what I see it does not share any code with
it (and it is unclear how much vfsd tries/tried to learn from VFS for Git,
it looks like it's being done from scratch but that impression could be
incorrect).

> Also, vfsd doesn't seem to be Google-able whereas "VFS for Git" is, and
> since it's not an internal git project, it might be nice to use a name
> that users can find.
>
> But anyway, I have no idea what the "correct" text here is (as
> probably evidenced by my questions), so if anyone wants to provide
> suggestions or corrections to the commit message, I'm happy to take
> them.

How about this?

	Typically with sparse checkouts, we expect files outside the
	sparsity patterns to be marked as SKIP_WORKTREE and be missing
	from the working tree.

	There is currently an effort code-named `vfsd` (see
	https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/
	for details) to implement a Git-aware virtual file system layer
	(similar in spirit to the VFS for Git project) that can be used to
	turn this expectation on its head: all files are considered
	present in the working copy, though they are not vivified until
	actually accessed.

	With such a virtual file system (VFS) layer, most of the files do
	not match the sparsity patterns at first, and the VFS layer
	automatically updates the sparsity patterns to add more files whenever
	files are written.

	Side note: VFS for Git itself requires the Microsoft fork of Git
	to work, therefore this patch would only be needed in that fork,
	as far as VFS for Git is concerned.

Ciao,
Dscho

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Ævar Arnfjörð Bjarmason]

On Tue, Feb 22 2022, Johannes Schindelin wrote:

> Hi Elijah,
>
> On Mon, 21 Feb 2022, Elijah Newren wrote:
>
>> On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason
>> <avarab@gmail.com> wrote:
>>
>> > There's also no law that says we can't have a three-level variable as
>> > core.sparse.*, perhaps that would be the start of a good trend, and it
>> > would sort adjacent to core.sparse[A-Z].* ...
>>
>> Interesting thought.  I'm curious what others think of this.
>
> What would make it `core`?
>
> In other words, while it is valuable to consider this option, I don't
> think that it solves anything, for the price of additional complexity.

To a first approximation what makes it core.* is that we have a
core.sparseCheckout.

It's hard come up with some definition of config namespacing where it
belongs there, but not a setting that tweaks it further, no?

I.e. we could quibble with the benefit of hindsight whether
core.sshCommand was the best idea at the time, or if it should be
remote.sshCommand or whatever.

But if we introduced a more specific sshCommandSomething having it there
instead of ... well, damn, I forgot that we have ssh.variant :)

Anyway, as noted (and I feel I have to comment as the originator of this
thread) I think the bikeshedding about namespacing is less important
than making sure from a user perspective that these are discoverable.

E.g. core.sshCommand doesn't reference ssh.variant or the other way
around, and in my "man git-config" they're at ~25% and ~90% in to what's
now a *huge* document. You need to read the whole thing or have the
foresight to search through it to discover both.

So for core.sshCommand and ssh.variant, and core.sparse* and sparse.*
cross-linking etc. would go a long way...

Aside: I do have some local patches (waiting on a couple of other things
I have outstanding) to add a "CONFIGURATION" section to all of the
built-in commands with some including/splitting up of
Documentation/config**.txt, so "man git-sparse-checkout" would have a
section listing the subset of "git-config" applicable to the command.

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Derrick Stolee]

On 2/22/2022 7:28 AM, Johannes Schindelin wrote:
> Hi Elijah,
> 
> On Mon, 21 Feb 2022, Elijah Newren wrote:
> 
>> On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee
>> <derrickstolee@github.com> wrote:
>>
>>> The only concern here really is if we want to be picky about the "VFS
>>> for Git" references instead of "vfsd" references in the commit
>>> message.
>>
>> I'm not sure I'm understanding the distinction.  Was "VFS for Git"
>> renamed to "vfsd"?  Is "vfsd" Google's modified version?  Something
>> else?
> 
> VFS for Git is the existing project at
> https://github.com/microsoft/VFSforGit which is pretty much in maintenance
> mode.
> 
> vfsd was mentioned by Jonathan Tan in
> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/.
> From what I gather, it is a completely separate implementation of the same
> ideas of VFS for Git, but from what I see it does not share any code with
> it (and it is unclear how much vfsd tries/tried to learn from VFS for Git,
> it looks like it's being done from scratch but that impression could be
> incorrect).

Elijah is correct that "vfsd" is not publicly visible outside of this
message, so that makes it difficult for us to verify that these patches
being made for it actually work as we intend.

> 	Side note: VFS for Git itself requires the Microsoft fork of Git
> 	to work, therefore this patch would only be needed in that fork,
> 	as far as VFS for Git is concerned.

I would drop this side note, since a big reason for the microsoft/git
fork is so we can create custom patches that handle these issues. We
would probably have created a similar patch during release integration,
but we are grateful to use this patch as a base for making these
integrations simpler in the future.

Thanks,
-Stolee

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Derrick Stolee]

On 2/21/2022 9:23 PM, Elijah Newren wrote:
> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>>
>> Hi Elijah,
>>
>> In addition to Stolee's feedback...
>>
>> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:
>>
>>> diff --git a/config.c b/config.c
>>> index 2bffa8d4a01..68e877a1d80 100644
>>> --- a/config.c
>>> +++ b/config.c
>>> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>>>               return 0;
>>>       }
>>>
>>> +     if (!strcmp(var, "core.expectfilesoutsidesparsepatterns")) {
>>> +             core_expect_files_outside_sparse_patterns = git_config_bool(var, value);
>>> +             return 0;
>>> +     }
>>
>> The `core` section is already quite crowded (for which I am partially
>> responsible, of course).
>>
>> Maybe it would be a good idea to introduce the `sparse` section, using
>> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =
>> false`?
> 
> That's a fair point.  At one point Stolee wanted to change from
> core.sparse* to sparse.* -- but by that point we already had users and
> would have had to deal with a bit of a migration story (and wondering
> what to do if people had both old and new config variables set
> inconsistently).
> 
> I'm not sure if it's optimal to try to keep the sparse settings
> together (thus put new ones under core), or try to avoid filling core.
> I guess if we moved towards sparse.* now, it might be an easier
> migration story if we only have two options to move.  And besides,
> we're already split between multiple sections with
> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and
> index.sparse already...so maybe adding one more section would be par
> for the course.  ;-)
> 
> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd
> like to hear Stolee's thoughts too.

This has been an interesting discussion.

I think that the existence of core.sparseCheckout[Cone] isn't a good
reason to keep adding to the core.* namespace. Creating a sparse.*
namespace is a good idea. I doubt this will be the last time we want
a config option for some behavior custom to sparse-checkouts (or
virtual environments).

I agree that migrating core.sparseCheckout[Cone] to the sparse.*
namespace would need to be done carefully. In particular, we would
probably need to _always_ understand the core.* versions, but start
to prefer the sparse.* versions if there is a conflict. This work
could be delayed indefinitely, or could be offered as a project for
a new contributor (if we experienced contributors could agree on a
clear path to make this transition).

Thanks,
-Stolee

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Jonathan Nieder]

Ævar Arnfjörð Bjarmason wrote:

> To a first approximation what makes it core.* is that we have a
> core.sparseCheckout.
[...]
> Anyway, as noted (and I feel I have to comment as the originator of this
> thread) I think the bikeshedding about namespacing is less important
> than making sure from a user perspective that these are discoverable.

Agreed.  Though there's already also advice.updateSparsePath; even
though I suspect that ctrl+F for "sparse" is what people may already
be using in practice, a pointer from the git-sparse-checkout(1) page
would be welcome.

> E.g. core.sshCommand doesn't reference ssh.variant or the other way
> around, and in my "man git-config" they're at ~25% and ~90% in to what's
> now a *huge* document. You need to read the whole thing or have the
> foresight to search through it to discover both.
>
> So for core.sshCommand and ssh.variant, and core.sparse* and sparse.*
> cross-linking etc. would go a long way...

Agreed as well.

> Aside: I do have some local patches (waiting on a couple of other things
> I have outstanding) to add a "CONFIGURATION" section to all of the
> built-in commands with some including/splitting up of
> Documentation/config**.txt, so "man git-sparse-checkout" would have a
> section listing the subset of "git-config" applicable to the command.

Oh!  I was about to do a quick patch for git-sparse-checkout.txt, but
this is even better, so I think I'll wait for you to send those. :)

Thanks,
Jonathan

[PATCH v2] repo_read_index: add config to expect files outside sparse patterns

[Jonathan Nieder]

From: Elijah Newren <newren@gmail.com>

Typically with sparse checkouts, we expect files outside the sparsity
patterns to be marked as SKIP_WORKTREE and be missing from the working
tree.  In edge cases, this can be violated and cause confusion, so in
a sparse checkout, since 11d46a399d ("repo_read_index: clear
SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git
automatically clears the SKIP_WORKTREE bit at read time for entries
corresponding to files that are present in the working tree.

However, there is a more atypical situation where this situation would
be expected.  A Git-aware virtual file system[1] takes advantage of
its position as a file system driver to expose all files in the
working tree, fetch them on demand using partial clone on access, and
tell Git to pay attention to them on demand by updating the sparse
checkout pattern on writes.  This means that commands like "git
status" only has to examine files that have potentially been modified,
whereas commands like "ls" are able to show the entire codebase
without requiring manual updates to the sparse checkout pattern.

Thus since 11d46a399d, Git with such Git-aware virtual file systems
unsets the SKIP_WORKTREE bit for all files and commands like "git
status" have to fetch and examine them all.

Introduce a configuration setting sparse.expectFilesOutsideOfPatterns
to allow limiting the tracked set of files to a small set once again.
A Git-aware virtual file system or other application that wants to
maintain files outside of the sparse checkout can set this in a
repository to instruct Git not to check for the presence of
SKIP_WORKTREE files.  The setting defaults to false, so most users of
sparse checkout will still get the benefit of an automatically
updating index to recover from interrupted updates that forget to
delete some files or unset SKIP_WORKTREE for them.

[1] such as the vfsd described in
https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

[jn: fleshed out commit message and documentation, added missing
 include to config.txt, moved to a separate config callback]

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Some minor updates, but this is basically the same as the patch you
sent.  Thoughts?

Thanks,
Jonathan

 Documentation/config.txt         |  2 ++
 Documentation/config/sparse.txt  | 24 ++++++++++++++++++++++++
 cache.h                          |  1 +
 config.c                         | 14 ++++++++++++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 7 files changed, 63 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/sparse.txt

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b168f02dc3d..8628ae2634d 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -468,6 +468,8 @@ include::config/sequencer.txt[]
 
 include::config/showbranch.txt[]
 
+include::config/sparse.txt[]
+
 include::config/splitindex.txt[]
 
 include::config/ssh.txt[]
diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
new file mode 100644
index 00000000000..c790c728276
--- /dev/null
+++ b/Documentation/config/sparse.txt
@@ -0,0 +1,24 @@
+sparse.expectFilesOutsideOfPatterns::
+	Typically with sparse checkouts, files not matching any
+	sparsity patterns are marked as such in the index file and
+	missing from the working tree.  Accordingly, Git will
+	ordinarily check whether files that the index indicates are
+	outside of the sparse area are present in the working tree and
+	mark them as present in the index if so.  This option can be
+	used to tell Git that such present-but-unmatching files are
+	expected and to stop checking for them.
++
+The default is `false`.  Leaving this set to `false` is recommended in
+most situations because it allows Git to recover from an interrupted
+operation that updated the working tree without updating the index or
+vice versa.
++
+A Git-based virtual file system (VFS) can turn the usual expectation
+on its head: files are present in the working copy but do not take
+up much disk space because their contents are not downloaded until
+they are accessed.  With such a virtual file system layer, most files
+do not match the sparsity patterns at first, and the VFS layer
+updates the sparsity patterns to add more files whenever files are
+written.  Setting this to `true` supports such a setup where files are
+expected to be present outside the sparse area and a separate, robust
+mechanism is responsible for keeping the sparsity patterns up to date.
diff --git a/cache.h b/cache.h
index 281f00ab1b1..b6b8e83ae35 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int sparse_expect_files_outside_of_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a01..9b9ad1500aa 100644
--- a/config.c
+++ b/config.c
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 	return platform_core_config(var, value, cb);
 }
 
+static int git_default_sparse_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
+		sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
+	/* Add other config variables here and to Documentation/config/sparse.txt. */
+	return 0;
+}
+
 static int git_default_i18n_config(const char *var, const char *value)
 {
 	if (!strcmp(var, "i18n.commitencoding"))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (starts_with(var, "sparse."))
+		return git_default_sparse_config(var, value);
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index fd0501e77a5..fb55bf61290 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int sparse_expect_files_outside_of_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f7..daeb5112a18 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    sparse_expect_files_outside_of_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb4901874..d1833c0f31b 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
 	test "$(cat b)" = "modified"
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config sparse.expectFilesOutsideOfPatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
 	test_create_repo server &&
 	git clone "file://$(pwd)/server" client &&
-- 
2.35.1.574.g5d30c73bfb

Re: [PATCH v2] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

On Tue, Feb 22, 2022 at 6:26 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
>
> From: Elijah Newren <newren@gmail.com>
>
> Typically with sparse checkouts, we expect files outside the sparsity
> patterns to be marked as SKIP_WORKTREE and be missing from the working
> tree.  In edge cases, this can be violated and cause confusion, so in
> a sparse checkout, since 11d46a399d ("repo_read_index: clear
> SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git
> automatically clears the SKIP_WORKTREE bit at read time for entries
> corresponding to files that are present in the working tree.
>
> However, there is a more atypical situation where this situation would
> be expected.  A Git-aware virtual file system[1] takes advantage of
> its position as a file system driver to expose all files in the
> working tree, fetch them on demand using partial clone on access, and
> tell Git to pay attention to them on demand by updating the sparse
> checkout pattern on writes.  This means that commands like "git
> status" only has to examine files that have potentially been modified,
> whereas commands like "ls" are able to show the entire codebase
> without requiring manual updates to the sparse checkout pattern.

Should that be s/commands/a command/ or else s/only has/only have/?

> Thus since 11d46a399d, Git with such Git-aware virtual file systems
> unsets the SKIP_WORKTREE bit for all files and commands like "git
> status" have to fetch and examine them all.
>
> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns
> to allow limiting the tracked set of files to a small set once again.
> A Git-aware virtual file system or other application that wants to
> maintain files outside of the sparse checkout can set this in a
> repository to instruct Git not to check for the presence of
> SKIP_WORKTREE files.  The setting defaults to false, so most users of
> sparse checkout will still get the benefit of an automatically
> updating index to recover from interrupted updates that forget to

Please don't presume that these only come from interrupted updates.
As per the referenced 11d46a399d that started all this:

"""
    There are various ways for users to get files to be present in the
    working copy despite having the SKIP_WORKTREE bit set for that file in
    the index.  This may come from:
      * various git commands not really supporting the SKIP_WORKTREE bit[1,2]
      * users grabbing files from elsewhere and writing them to the worktree
        (perhaps even cached in their editor)
      * users attempting to "abort" a sparse-checkout operation with a
        not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
        working tree is not atomic)[3].
"""

> delete some files or unset SKIP_WORKTREE for them.

The only problem 11d46a399d corrects is having the SKIP_WORKTREE being
*set* despite the file being present.  So the "recover from...updates
that...unset SKIP_WORKTREE" doesn't make any sense.

> [1] such as the vfsd described in
> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/
>
> [jn: fleshed out commit message and documentation, added missing
>  include to config.txt, moved to a separate config callback]
>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
> Some minor updates, but this is basically the same as the patch you
> sent.  Thoughts?

...and it looks even more like the v2 I was about to send, so it seems
like we're pretty much on the same page.  :-)

Overall, it looks good, and your version has some things that are
nicer than mine, but I did have a couple small notes on the commit
message above and a similar one on the config description below.

> Thanks,
> Jonathan
>
>  Documentation/config.txt         |  2 ++
>  Documentation/config/sparse.txt  | 24 ++++++++++++++++++++++++
>  cache.h                          |  1 +
>  config.c                         | 14 ++++++++++++++
>  environment.c                    |  1 +
>  sparse-index.c                   |  3 ++-
>  t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
>  7 files changed, 63 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/config/sparse.txt
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index b168f02dc3d..8628ae2634d 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -468,6 +468,8 @@ include::config/sequencer.txt[]
>
>  include::config/showbranch.txt[]
>
> +include::config/sparse.txt[]
> +
>  include::config/splitindex.txt[]
>
>  include::config/ssh.txt[]
> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
> new file mode 100644
> index 00000000000..c790c728276
> --- /dev/null
> +++ b/Documentation/config/sparse.txt
> @@ -0,0 +1,24 @@
> +sparse.expectFilesOutsideOfPatterns::
> +       Typically with sparse checkouts, files not matching any
> +       sparsity patterns are marked as such in the index file and
> +       missing from the working tree.  Accordingly, Git will
> +       ordinarily check whether files that the index indicates are
> +       outside of the sparse area are present in the working tree and
> +       mark them as present in the index if so.  This option can be
> +       used to tell Git that such present-but-unmatching files are
> +       expected and to stop checking for them.
> ++
> +The default is `false`.  Leaving this set to `false` is recommended in
> +most situations because it allows Git to recover from an interrupted
> +operation that updated the working tree without updating the index or
> +vice versa.

Again, please don't claim this is only for recovering from interrupted
operations; there are other cases -- Git commands that write the
working tree but not the index (checkout-index, git apply), and users
mucking around with files, for example.

> ++
> +A Git-based virtual file system (VFS) can turn the usual expectation
> +on its head: files are present in the working copy but do not take
> +up much disk space because their contents are not downloaded until
> +they are accessed.  With such a virtual file system layer, most files
> +do not match the sparsity patterns at first, and the VFS layer
> +updates the sparsity patterns to add more files whenever files are
> +written.  Setting this to `true` supports such a setup where files are
> +expected to be present outside the sparse area and a separate, robust
> +mechanism is responsible for keeping the sparsity patterns up to date.
> diff --git a/cache.h b/cache.h
> index 281f00ab1b1..b6b8e83ae35 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
>
>  extern int core_apply_sparse_checkout;
>  extern int core_sparse_checkout_cone;
> +extern int sparse_expect_files_outside_of_patterns;
>
>  /*
>   * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
> diff --git a/config.c b/config.c
> index 2bffa8d4a01..9b9ad1500aa 100644
> --- a/config.c
> +++ b/config.c
> @@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>         return platform_core_config(var, value, cb);
>  }
>
> +static int git_default_sparse_config(const char *var, const char *value)
> +{
> +       if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
> +               sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
> +               return 0;
> +       }
> +
> +       /* Add other config variables here and to Documentation/config/sparse.txt. */
> +       return 0;
> +}
> +
>  static int git_default_i18n_config(const char *var, const char *value)
>  {
>         if (!strcmp(var, "i18n.commitencoding"))
> @@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
>                 return 0;
>         }
>
> +       if (starts_with(var, "sparse."))
> +               return git_default_sparse_config(var, value);
> +
>         /* Add other config variables here and to Documentation/config.txt. */
>         return 0;
>  }
> diff --git a/environment.c b/environment.c
> index fd0501e77a5..fb55bf61290 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -70,6 +70,7 @@ char *notes_ref_name;
>  int grafts_replace_parents = 1;
>  int core_apply_sparse_checkout;
>  int core_sparse_checkout_cone;
> +int sparse_expect_files_outside_of_patterns;
>  int merge_log_config = -1;
>  int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
>  unsigned long pack_size_limit_cfg;
> diff --git a/sparse-index.c b/sparse-index.c
> index eed170cd8f7..daeb5112a18 100644
> --- a/sparse-index.c
> +++ b/sparse-index.c
> @@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
>
>         int i;
>
> -       if (!core_apply_sparse_checkout)
> +       if (!core_apply_sparse_checkout ||
> +           sparse_expect_files_outside_of_patterns)
>                 return;
>
>  restart:
> diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
> index 3deb4901874..d1833c0f31b 100755
> --- a/t/t1090-sparse-checkout-scope.sh
> +++ b/t/t1090-sparse-checkout-scope.sh
> @@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
>         test "$(cat b)" = "modified"
>  '
>
> +test_expect_success 'skip-worktree on files outside sparse patterns' '
> +       git sparse-checkout disable &&
> +       git sparse-checkout set --no-cone "a*" &&
> +       git checkout-index --all --ignore-skip-worktree-bits &&
> +
> +       git ls-files -t >output &&
> +       ! grep ^S output >actual &&
> +       test_must_be_empty actual &&
> +
> +       test_config sparse.expectFilesOutsideOfPatterns true &&
> +       cat <<-\EOF >expect &&
> +       S b
> +       S c
> +       EOF
> +       git ls-files -t >output &&
> +       grep ^S output >actual &&
> +       test_cmp expect actual
> +'
> +
>  test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
>         test_create_repo server &&
>         git clone "file://$(pwd)/server" client &&
> --
> 2.35.1.574.g5d30c73bfb

[PATCH v3] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

Typically with sparse checkouts, we expect files outside the sparsity
patterns to be marked as SKIP_WORKTREE and be missing from the working
tree.  In edge cases, this can be violated and cause confusion, so in a
sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE
bit from files present in worktree", 2022-01-06), Git automatically
clears the SKIP_WORKTREE bit at read time for entries corresponding to
files that are present in the working tree.

However, there is a more atypical situation where this situation would
be expected.  A Git-aware virtual file system[1] takes advantage of its
position as a file system driver to expose all files in the working
tree, fetch them on demand using partial clone on access, and tell Git
to pay attention to them on demand by updating the sparse checkout
pattern on writes.  This means that commands like "git status" only have
to examine files that have potentially been modified, whereas commands
like "ls" are able to show the entire codebase without requiring manual
updates to the sparse checkout pattern.

Thus since 11d46a399d, Git with such Git-aware virtual file systems
unsets the SKIP_WORKTREE bit for all files and commands like "git
status" have to fetch and examine them all.

Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
allow limiting the tracked set of files to a small set once again.  A
Git-aware virtual file system or other application that wants to
maintain files outside of the sparse checkout can set this in a
repository to instruct Git not to check for the presence of
SKIP_WORKTREE files.  The setting defaults to false, so most users of
sparse checkout will still get the benefit of an automatically updating
index to recover from the variety of difficult issues detailed in
11d46a399d for paths with SKIP_WORKTREE set despite the path being
present.

[1] such as the vfsd described in
https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Elijah Newren <newren@gmail.com>
---
Changes since v2:
  * Made tweaks to the commit message and the text of the config option as
    highlighted in my response to Jonathan's v2.

I'm guessing that since there are no code (only documentation) changes since
Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs.
I'm also guessing it matches what Stolee and Dscho stated in their comments
on v1.  But it'd be nice to have an ack from each side just to make sure.
    
 Documentation/config.txt         |  2 ++
 Documentation/config/sparse.txt  | 28 ++++++++++++++++++++++++++++
 cache.h                          |  1 +
 config.c                         | 14 ++++++++++++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 7 files changed, 67 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/sparse.txt

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b168f02dc3..8628ae2634 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -468,6 +468,8 @@ include::config/sequencer.txt[]
 
 include::config/showbranch.txt[]
 
+include::config/sparse.txt[]
+
 include::config/splitindex.txt[]
 
 include::config/ssh.txt[]
diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
new file mode 100644
index 0000000000..fba504173c
--- /dev/null
+++ b/Documentation/config/sparse.txt
@@ -0,0 +1,28 @@
+sparse.expectFilesOutsideOfPatterns::
+	Typically with sparse checkouts, files not matching any
+	sparsity patterns are marked as such in the index file and
+	missing from the working tree.  Accordingly, Git will
+	ordinarily check whether files that the index indicates are
+	outside of the sparse area are present in the working tree and
+	mark them as present in the index if so.  This option can be
+	used to tell Git that such present-but-unmatching files are
+	expected and to stop checking for them.
++
+The default is `false`.  Paths which are marked as SKIP_WORKTREE
+despite being present (which can occur for a few different reasons)
+typically present a range of problems which are difficult for users to
+discover and recover from.  The default setting avoids such issues.
++
+A Git-based virtual file system (VFS) can turn the usual expectation
+on its head: files are present in the working copy but do not take
+up much disk space because their contents are not downloaded until
+they are accessed.  With such a virtual file system layer, most files
+do not match the sparsity patterns at first, and the VFS layer
+updates the sparsity patterns to add more files whenever files are
+written.  Setting this to `true` supports such a setup where files are
+expected to be present outside the sparse area and a separate, robust
+mechanism is responsible for keeping the sparsity patterns up to date.
++
+Note that the checking and clearing of the SKIP_WORKTREE bit only
+happens when core.sparseCheckout is true, so this config option has no
+effect unless core.sparseCheckout is true.
diff --git a/cache.h b/cache.h
index 281f00ab1b..b6b8e83ae3 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int sparse_expect_files_outside_of_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a0..9b9ad1500a 100644
--- a/config.c
+++ b/config.c
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 	return platform_core_config(var, value, cb);
 }
 
+static int git_default_sparse_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
+		sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
+	/* Add other config variables here and to Documentation/config/sparse.txt. */
+	return 0;
+}
+
 static int git_default_i18n_config(const char *var, const char *value)
 {
 	if (!strcmp(var, "i18n.commitencoding"))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (starts_with(var, "sparse."))
+		return git_default_sparse_config(var, value);
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index fd0501e77a..fb55bf6129 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int sparse_expect_files_outside_of_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f..daeb5112a1 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    sparse_expect_files_outside_of_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb490187..d1833c0f31 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
 	test "$(cat b)" = "modified"
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config sparse.expectFilesOutsideOfPatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
 	test_create_repo server &&
 	git clone "file://$(pwd)/server" client &&
-- 
2.34.0.22.g587c05af5b.dirty

Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Typically with sparse checkouts, we expect files outside the sparsity
> patterns to be marked as SKIP_WORKTREE and be missing from the working
> tree.  In edge cases, this can be violated and cause confusion, so in a
> sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE

I think this refers to af6a5187 (repo_read_index: clear
SKIP_WORKTREE bit from files present in worktree, 2022-01-14).

> bit from files present in worktree", 2022-01-06), Git automatically
> clears the SKIP_WORKTREE bit at read time for entries corresponding to
> files that are present in the working tree.

So, this is a workflow where the user deliberately "creates" these
files outside the sparsity cone or pattern (by various non-automated
means like editing, copying/renaming, or untarring).  If they did so
on purpose, they may be interested in comparing them with existing
commits, or even including them as a newer version in the next
commit they create.  To help that workflow, clearing the bit makes
sense.

Am I on the right path?  I am wondering if mentioning some of that
would help understanding by the reader when it is contrasted with
the (competing) goal of supporting VFS use case mentioned next.

> However, there is a more atypical situation where this situation would

I wonder if that is "more atypical" (read: makes me wonder if it
depends on who the reader is what is typoical), and more importantly,
if it helps understanding of the reader (read: whether which one is
more common, we'd want to support both camps anyway).  

    There is another workflow, however, that it is expected that
    paths outside the sparsity patterns appear to exist in the
    working tree and that they do not lose the SKIP_WORKTREE bit, at
    least until they get modified.

or something?

> be expected.  A Git-aware virtual file system[1] takes advantage of its
> position as a file system driver to expose all files in the working
> tree, fetch them on demand using partial clone on access, and tell Git
> to pay attention to them on demand by updating the sparse checkout
> pattern on writes.  This means that commands like "git status" only have
> to examine files that have potentially been modified, whereas commands
> like "ls" are able to show the entire codebase without requiring manual
> updates to the sparse checkout pattern.

Well explained.

> Thus since 11d46a399d, Git with such Git-aware virtual file systems

The same stale reference.

> unsets the SKIP_WORKTREE bit for all files and commands like "git
> status" have to fetch and examine them all.
>
> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
> allow limiting the tracked set of files to a small set once again.  A
> Git-aware virtual file system or other application that wants to
> maintain files outside of the sparse checkout can set this in a
> repository to instruct Git not to check for the presence of
> SKIP_WORKTREE files.  The setting defaults to false, so most users of
> sparse checkout will still get the benefit of an automatically updating
> index to recover from the variety of difficult issues detailed in
> 11d46a399d for paths with SKIP_WORKTREE set despite the path being

Ditto.

> I'm guessing that since there are no code (only documentation) changes since
> Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs.
> I'm also guessing it matches what Stolee and Dscho stated in their comments
> on v1.  But it'd be nice to have an ack from each side just to make sure.

True.  Let me queue but leave it just outside 'next' until that
happens.

I think the name of the knob is what Jonathan suggested, so I
presume that their side would be fine with it, but I am curious (I
do not wonder, though) what the plan on the Microsoft's side going
forward.  When they update the version of Git bundled in their vfsd,
would this be reverted and an equivalent they have (and they may
have more such "workaround" in other areas as well?) will be kept,
so whatever we do here will add a minor inconvenience to them but
will not hurt them otherwise?

> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
> new file mode 100644
> index 0000000000..fba504173c
> --- /dev/null
> +++ b/Documentation/config/sparse.txt
> @@ -0,0 +1,28 @@
> +sparse.expectFilesOutsideOfPatterns::
> +	Typically with sparse checkouts, files not matching any
> +	sparsity patterns are marked as such in the index file and

s/index file/index/ perhaps.

> +	missing from the working tree.  Accordingly, Git will
> +	ordinarily check whether files that the index indicates are
> +	outside of the sparse area are present in the working tree and
> +	mark them as present in the index if so.  This option can be

Just an observation.  According to this sentence, "sparse area" is
"paths that ought to be present in the working tree", so paths
"outside of the sparse area" that are present need to be corrected
to be "in" the sparse area by futzing bits.  I always get confused
when I hear "sparse area" if the author meant "paths that ought to
be missing" or "present", but maybe it is just me.

> +	used to tell Git that such present-but-unmatching files are
> +	expected and to stop checking for them.

OK.

> ++
> +The default is `false`.  Paths which are marked as SKIP_WORKTREE
> +despite being present (which can occur for a few different reasons)
> +typically present a range of problems which are difficult for users to

s/typically // perhaps.

> +discover and recover from.  The default setting avoids such issues.
> ++
> +A Git-based virtual file system (VFS) can turn the usual expectation
> +on its head: files are present in the working copy but do not take
> +up much disk space because their contents are not downloaded until
> +they are accessed.  With such a virtual file system layer, most files
> +do not match the sparsity patterns at first, and the VFS layer
> +updates the sparsity patterns to add more files whenever files are
> +written.  Setting this to `true` supports such a setup where files are
> +expected to be present outside the sparse area and a separate, robust
> +mechanism is responsible for keeping the sparsity patterns up to date.

s/separate, robust/separate/ I would think.

We make the outside mechanism that makes these files appear to be
present to also be responsible for maintaining the sparse bit and
patterns.

When the user (or IDE) sets this knob to 'true', do we even have to
expect that files appear to be present?  In the use case we intend
to support with this feature, i.e. some VFS, we might expect all
paths to appear to be present, but if that VFS also allows users to
configure to expose only a subset of paths, not all paths may appear
to be present.  And we are perfectly OK with that, becuase we do not
expect anything about the working tree paths outside the sparsity
pattern.  Am I mistaken?

So, "... supports such a setup where some external system releaves
us of the responsibility of maintaining the consistency between the
presence of working tree files and sparsity patterns, so we stop
expecting whether files are present or missing outside the sparse
area", might be closer to the truth?

> +Note that the checking and clearing of the SKIP_WORKTREE bit only
> +happens when core.sparseCheckout is true, so this config option has no
> +effect unless core.sparseCheckout is true.

Good note to have.  There is no mention of "cone" mode in the entire
description; it is unclear if this only applies to "pattern" mode or
to both "pattern" and "cone" modes, which may want to be clarified.

Thanks.

Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns

[Jonathan Nieder]

Hi,

Elijah Newren wrote:

> Signed-off-by: Elijah Newren <newren@gmail.com>

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks, and sorry for the slow review.  My one remaining area for nits
is the documentation, but that can be improved iteratively via patches
on top.

[...]
> --- /dev/null
> +++ b/Documentation/config/sparse.txt
> @@ -0,0 +1,28 @@
> +sparse.expectFilesOutsideOfPatterns::
> +	Typically with sparse checkouts, files not matching any
> +	sparsity patterns are marked as such in the index file and
> +	missing from the working tree.  Accordingly, Git will
> +	ordinarily check whether files that the index indicates are
> +	outside of the sparse area are present in the working tree and

Junio mentioned the "sparse area" could suggest that the area is
itself sparse and devoid of files, so it might not have been the best
choice of words on my part.  Perhaps "whether files that the index
indicates are not checked out are present in the working tree" would
work here?

> +	mark them as present in the index if so.  This option can be
> +	used to tell Git that such present-but-unmatching files are
> +	expected and to stop checking for them.
> ++
> +The default is `false`.  Paths which are marked as SKIP_WORKTREE
> +despite being present (which can occur for a few different reasons)
> +typically present a range of problems which are difficult for users to
> +discover and recover from.  The default setting avoids such issues.

The git-sparse-checkout(1) page never describes what SKIP_WORKTREE
means, so it might not be obvious to them what this means.  Also, the
"can occur for a few different reasons" may leave the user wondering
whether they are subject to those reasons.  What the reader wants to
know is "I should keep using the default because it makes Git work
better", so how about something like

 The default is `false`, which allows Git to automatically recover
 from the list of files in the index and working tree falling out of
 sync.
 +

?

> ++
> +A Git-based virtual file system (VFS) can turn the usual expectation
> +on its head: files are present in the working copy but do not take
> +up much disk space because their contents are not downloaded until
> +they are accessed.  With such a virtual file system layer, most files
> +do not match the sparsity patterns at first, and the VFS layer
> +updates the sparsity patterns to add more files whenever files are
> +written.  Setting this to `true` supports such a setup where files are
> +expected to be present outside the sparse area and a separate, robust
> +mechanism is responsible for keeping the sparsity patterns up to date.

Here I spent most of the words explaining what a Git-based VFS layer
is, which is also not too relevant to most users (who are just
interested in "is `true` the right value for me?").  How about
reducing it to the following?

 Set this to `true` if you are in a setup where extra files are expected
 to be present and a separate, robust mechanism is responsible for
 keeping the sparsity patterns up to date, such as a Git-aware virtual
 file system.

?

> ++
> +Note that the checking and clearing of the SKIP_WORKTREE bit only
> +happens when core.sparseCheckout is true, so this config option has no
> +effect unless core.sparseCheckout is true.

Good note.  Same nit about the user not necessarily knowing what
SKIP_WORKTREE means applies.  Also, we can remove the extra words
"Note that" since the dutiful reader should be noting everything we
say. :)  I think that would make

 +
 Regardless of this setting, Git does not check for
 present-but-unmatching files unless sparse checkout is enabled, so
 this config option has no effect unless `core.sparseCheckout` is
 `true`.

Thanks,
Jonathan

Re: [PATCH] Provide config option to expect files outside sparse patterns

[Ævar Arnfjörð Bjarmason]

On Tue, Feb 22 2022, Jonathan Nieder wrote:

> Ævar Arnfjörð Bjarmason wrote:
>
>> To a first approximation what makes it core.* is that we have a
>> core.sparseCheckout.
> [...]
>> Anyway, as noted (and I feel I have to comment as the originator of this
>> thread) I think the bikeshedding about namespacing is less important
>> than making sure from a user perspective that these are discoverable.
>
> Agreed.  Though there's already also advice.updateSparsePath; even
> though I suspect that ctrl+F for "sparse" is what people may already
> be using in practice, a pointer from the git-sparse-checkout(1) page
> would be welcome.
>
>> E.g. core.sshCommand doesn't reference ssh.variant or the other way
>> around, and in my "man git-config" they're at ~25% and ~90% in to what's
>> now a *huge* document. You need to read the whole thing or have the
>> foresight to search through it to discover both.
>>
>> So for core.sshCommand and ssh.variant, and core.sparse* and sparse.*
>> cross-linking etc. would go a long way...
>
> Agreed as well.
>
>> Aside: I do have some local patches (waiting on a couple of other things
>> I have outstanding) to add a "CONFIGURATION" section to all of the
>> built-in commands with some including/splitting up of
>> Documentation/config**.txt, so "man git-sparse-checkout" would have a
>> section listing the subset of "git-config" applicable to the command.
>
> Oh!  I was about to do a quick patch for git-sparse-checkout.txt, but
> this is even better, so I think I'll wait for you to send those. :)

I think if you're interested in fixing just that case you shouldn't let
me stop you.

The thing I have is currently dependant on [1], so a review would speed
it up. I.e. needing to create new sub-dirs in Documentation/config that
we lint etc. needs some of those Makefile bits (or rather, is a lot
easier with them).

1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20211224T173558Z-avarab@gmail.com/

The diff for the ad-hoc series I have following that locally follows, to
give you some idea.

I hacked it up before there was git-sparse*, so if I submit it I'd
revisit that, and I think even for the below I stopped at some point and
would need to finish it.

 Documentation/Makefile                             |   2 +
 Documentation/config/color.txt                     | 155 ++-------------------
 Documentation/config/color/blame.txt               |  22 +++
 Documentation/config/color/branch.txt              |  13 ++
 Documentation/config/color/diff.txt                |  27 ++++
 Documentation/config/color/grep.txt                |  70 ++++++++++
 Documentation/config/color/log.txt                 |   9 ++
 Documentation/config/color/push.txt                |   8 ++
 Documentation/config/color/showbranch.txt          |   6 +
 Documentation/config/color/status.txt              |  20 +++
 Documentation/config/core.txt                      |  13 +-
 Documentation/config/core/bigFileThreshold.txt     |  12 ++
 Documentation/config/diff.txt                      |  15 --
 Documentation/config/diff/difftool.txt             |  14 ++
 Documentation/config/difftool.txt                  |   7 +
 Documentation/config/gc.txt                        |  12 +-
 Documentation/config/gc/rerere.txt                 |  11 ++
 Documentation/config/grep.txt                      |   7 +-
 Documentation/config/log.txt                       |  18 ++-
 Documentation/config/notes.txt                     |  43 +++---
 Documentation/config/sendemail.txt                 |  40 +++++-
 Documentation/config/uploadpack.txt                |  26 +---
 Documentation/config/uploadpack/pack-objects.txt   |  25 ++++
 Documentation/git-add.txt                          |   7 +
 Documentation/git-am.txt                           |   7 +
 Documentation/git-apply.txt                        |  10 +-
 Documentation/git-blame.txt                        |  10 ++
 Documentation/git-branch.txt                       |   6 +
 Documentation/git-checkout.txt                     |   7 +
 Documentation/git-clean.txt                        |   7 +
 Documentation/git-clone.txt                        |   9 ++
 Documentation/git-commit-graph.txt                 |   8 ++
 Documentation/git-commit.txt                       |   4 +
 Documentation/git-credential.txt                   |   7 +
 Documentation/git-diff.txt                         |   9 ++
 Documentation/git-difftool.txt                     |  28 +---
 Documentation/git-fast-import.txt                  |   7 +
 Documentation/git-fetch.txt                        |   9 ++
 Documentation/git-gc.txt                           |  14 +-
 Documentation/git-grep.txt                         |  29 +---
 Documentation/git-log.txt                          |  50 ++-----
 Documentation/git-mergetool.txt                    |   3 +
 Documentation/git-notes.txt                        |  54 +------
 Documentation/git-pack-objects.txt                 |   8 +-
 Documentation/git-remote.txt                       |  15 +-
 Documentation/git-repack.txt                       |  19 +--
 Documentation/git-rerere.txt                       |   9 ++
 Documentation/git-reset.txt                        |   7 +
 Documentation/git-send-email.txt                   |  36 +----
 Documentation/git-show-branch.txt                  |   7 +
 Documentation/githooks.txt                         |   4 +
 Documentation/includes/cmd-config-section-all.txt  |   3 +
 Documentation/includes/cmd-config-section-rest.txt |   3 +
 .../templates/configuration-include-pack.txt       |  10 ++
 builtin/help.c                                     |   2 +-
 builtin/log.c                                      |   3 +-
 generate-configlist.sh                             |   2 +-
 t/t4207-log-decoration-colors.sh                   |  12 +-
 58 files changed, 544 insertions(+), 456 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 1eb9192dae8..15b71f24735 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -50,7 +50,9 @@ MAN7_TXT += gitworkflows.txt
 HOWTO_TXT += $(wildcard howto/*.txt)
 
 DOC_DEP_TXT += $(wildcard *.txt)
+DOC_DEP_TXT += $(wildcard includes/*.txt)
 DOC_DEP_TXT += $(wildcard config/*.txt)
+DOC_DEP_TXT += $(wildcard config/*/*.txt)
 
 ifdef MAN_FILTER
 MAN_TXT = $(filter $(MAN_FILTER),$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT))
diff --git a/Documentation/config/color.txt b/Documentation/config/color.txt
index 1795b2d16be..4ca476b99ac 100644
--- a/Documentation/config/color.txt
+++ b/Documentation/config/color.txt
@@ -8,113 +8,15 @@ color.advice::
 color.advice.hint::
 	Use customized color for hints.
 
-color.blame.highlightRecent::
-	Specify the line annotation color for `git blame --color-by-age`
-	depending upon the age of the line.
-+
-This setting should be set to a comma-separated list of color and
-date settings, starting and ending with a color, the dates should be
-set from oldest to newest. The metadata will be colored with the
-specified colors if the line was introduced before the given
-timestamp, overwriting older timestamped colors.
-+
-Instead of an absolute timestamp relative timestamps work as well,
-e.g. `2.weeks.ago` is valid to address anything older than 2 weeks.
-+
-It defaults to `blue,12 month ago,white,1 month ago,red`, which
-colors everything older than one year blue, recent changes between
-one month and one year old are kept white, and lines introduced
-within the last month are colored red.
-
-color.blame.repeatedLines::
-	Use the specified color to colorize line annotations for
-	`git blame --color-lines`, if they come from the same commit as the
-	preceding line. Defaults to cyan.
-
-color.branch::
-	A boolean to enable/disable color in the output of
-	linkgit:git-branch[1]. May be set to `always`,
-	`false` (or `never`) or `auto` (or `true`), in which case colors are used
-	only when the output is to a terminal. If unset, then the
-	value of `color.ui` is used (`auto` by default).
-
-color.branch.<slot>::
-	Use customized color for branch coloration. `<slot>` is one of
-	`current` (the current branch), `local` (a local branch),
-	`remote` (a remote-tracking branch in refs/remotes/),
-	`upstream` (upstream tracking branch), `plain` (other
-	refs).
-
-color.diff::
-	Whether to use ANSI escape sequences to add color to patches.
-	If this is set to `always`, linkgit:git-diff[1],
-	linkgit:git-log[1], and linkgit:git-show[1] will use color
-	for all patches.  If it is set to `true` or `auto`, those
-	commands will only use color when output is to the terminal.
-	If unset, then the value of `color.ui` is used (`auto` by
-	default).
-+
-This does not affect linkgit:git-format-patch[1] or the
-'git-diff-{asterisk}' plumbing commands.  Can be overridden on the
-command line with the `--color[=<when>]` option.
-
-color.diff.<slot>::
-	Use customized color for diff colorization.  `<slot>` specifies
-	which part of the patch to use the specified color, and is one
-	of `context` (context text - `plain` is a historical synonym),
-	`meta` (metainformation), `frag`
-	(hunk header), 'func' (function in hunk header), `old` (removed lines),
-	`new` (added lines), `commit` (commit headers), `whitespace`
-	(highlighting whitespace errors), `oldMoved` (deleted lines),
-	`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,
-	`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`
-	`newMovedAlternativeDimmed` (See the '<mode>'
-	setting of '--color-moved' in linkgit:git-diff[1] for details),
-	`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,
-	`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).
-
-color.decorate.<slot>::
-	Use customized color for 'git log --decorate' output.  `<slot>` is one
-	of `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local
-	branches, remote-tracking branches, tags, stash and HEAD, respectively
-	and `grafted` for grafted commits.
-
-color.grep::
-	When set to `always`, always highlight matches.  When `false` (or
-	`never`), never.  When set to `true` or `auto`, use color only
-	when the output is written to the terminal.  If unset, then the
-	value of `color.ui` is used (`auto` by default).
-
-color.grep.<slot>::
-	Use customized color for grep colorization.  `<slot>` specifies which
-	part of the line to use the specified color, and is one of
-+
---
-`context`;;
-	non-matching text in context lines (when using `-A`, `-B`, or `-C`)
-`filename`;;
-	filename prefix (when not using `-h`)
-`function`;;
-	function name lines (when using `-p`)
-`lineNumber`;;
-	line number prefix (when using `-n`)
-`column`;;
-	column number prefix (when using `--column`)
-`match`;;
-	matching text (same as setting `matchContext` and `matchSelected`)
-`matchContext`;;
-	matching text in context lines
-`matchSelected`;;
-	matching text in selected lines. Also, used to customize the following
-	linkgit:git-log[1] subcommands: `--grep`, `--author` and `--committer`.
-`selected`;;
-	non-matching text in selected lines. Also, used to customize the
-	following linkgit:git-log[1] subcommands: `--grep`, `--author` and
-	`--committer`.
-`separator`;;
-	separators between fields on a line (`:`, `-`, and `=`)
-	and between hunks (`--`)
---
+include::color/blame.txt[]
+
+include::color/branch.txt[]
+
+include::color/diff.txt[]
+
+include::color/log.txt[]
+
+include::color/grep.txt[]
 
 color.interactive::
 	When set to `always`, always use colors for interactive prompts
@@ -135,14 +37,7 @@ color.pager::
 	output going to the pager. Defaults to true; set this to false
 	if your pager does not understand ANSI color codes.
 
-color.push::
-	A boolean to enable/disable color in push errors. May be set to
-	`always`, `false` (or `never`) or `auto` (or `true`), in which
-	case colors are used only when the error output goes to a terminal.
-	If unset, then the value of `color.ui` is used (`auto` by default).
-
-color.push.error::
-	Use customized color for push errors.
+include::color/push.txt[]
 
 color.remote::
 	If set, keywords at the start of the line are highlighted. The
@@ -156,33 +51,9 @@ color.remote.<slot>::
 	`hint`, `warning`, `success` or `error` which match the
 	corresponding keyword.
 
-color.showBranch::
-	A boolean to enable/disable color in the output of
-	linkgit:git-show-branch[1]. May be set to `always`,
-	`false` (or `never`) or `auto` (or `true`), in which case colors are used
-	only when the output is to a terminal. If unset, then the
-	value of `color.ui` is used (`auto` by default).
-
-color.status::
-	A boolean to enable/disable color in the output of
-	linkgit:git-status[1]. May be set to `always`,
-	`false` (or `never`) or `auto` (or `true`), in which case colors are used
-	only when the output is to a terminal. If unset, then the
-	value of `color.ui` is used (`auto` by default).
-
-color.status.<slot>::
-	Use customized color for status colorization. `<slot>` is
-	one of `header` (the header text of the status message),
-	`added` or `updated` (files which are added but not committed),
-	`changed` (files which are changed but not added in the index),
-	`untracked` (files which are not tracked by Git),
-	`branch` (the current branch),
-	`nobranch` (the color the 'no branch' warning is shown in, defaulting
-	to red),
-	`localBranch` or `remoteBranch` (the local and remote branch names,
-	respectively, when branch and tracking information is displayed in the
-	status short-format), or
-	`unmerged` (files which have unmerged changes).
+include::color/showbranch.txt[]
+
+include::color/status.txt[]
 
 color.transport::
 	A boolean to enable/disable color when pushes are rejected. May be
diff --git a/Documentation/config/color/blame.txt b/Documentation/config/color/blame.txt
new file mode 100644
index 00000000000..423290c597b
--- /dev/null
+++ b/Documentation/config/color/blame.txt
@@ -0,0 +1,22 @@
+color.blame.highlightRecent::
+	Specify the line annotation color for `git blame --color-by-age`
+	depending upon the age of the line.
++
+This setting should be set to a comma-separated list of color and
+date settings, starting and ending with a color, the dates should be
+set from oldest to newest. The metadata will be colored with the
+specified colors if the line was introduced before the given
+timestamp, overwriting older timestamped colors.
++
+Instead of an absolute timestamp relative timestamps work as well,
+e.g. `2.weeks.ago` is valid to address anything older than 2 weeks.
++
+It defaults to `blue,12 month ago,white,1 month ago,red`, which
+colors everything older than one year blue, recent changes between
+one month and one year old are kept white, and lines introduced
+within the last month are colored red.
+
+color.blame.repeatedLines::
+	Use the specified color to colorize line annotations for
+	`git blame --color-lines`, if they come from the same commit as the
+	preceding line. Defaults to cyan.
diff --git a/Documentation/config/color/branch.txt b/Documentation/config/color/branch.txt
new file mode 100644
index 00000000000..30bf0c3d108
--- /dev/null
+++ b/Documentation/config/color/branch.txt
@@ -0,0 +1,13 @@
+color.branch::
+	A boolean to enable/disable color in the output of
+	linkgit:git-branch[1]. May be set to `always`,
+	`false` (or `never`) or `auto` (or `true`), in which case colors are used
+	only when the output is to a terminal. If unset, then the
+	value of `color.ui` is used (`auto` by default).
+
+color.branch.<slot>::
+	Use customized color for branch coloration. `<slot>` is one of
+	`current` (the current branch), `local` (a local branch),
+	`remote` (a remote-tracking branch in refs/remotes/),
+	`upstream` (upstream tracking branch), `plain` (other
+	refs).
diff --git a/Documentation/config/color/diff.txt b/Documentation/config/color/diff.txt
new file mode 100644
index 00000000000..609c7a5ca91
--- /dev/null
+++ b/Documentation/config/color/diff.txt
@@ -0,0 +1,27 @@
+color.diff::
+	Whether to use ANSI escape sequences to add color to patches.
+	If this is set to `always`, linkgit:git-diff[1],
+	linkgit:git-log[1], and linkgit:git-show[1] will use color
+	for all patches.  If it is set to `true` or `auto`, those
+	commands will only use color when output is to the terminal.
+	If unset, then the value of `color.ui` is used (`auto` by
+	default).
++
+This does not affect linkgit:git-format-patch[1] or the
+'git-diff-{asterisk}' plumbing commands.  Can be overridden on the
+command line with the `--color[=<when>]` option.
+
+color.diff.<slot>::
+	Use customized color for diff colorization.  `<slot>` specifies
+	which part of the patch to use the specified color, and is one
+	of `context` (context text - `plain` is a historical synonym),
+	`meta` (metainformation), `frag`
+	(hunk header), 'func' (function in hunk header), `old` (removed lines),
+	`new` (added lines), `commit` (commit headers), `whitespace`
+	(highlighting whitespace errors), `oldMoved` (deleted lines),
+	`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,
+	`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`
+	`newMovedAlternativeDimmed` (See the '<mode>'
+	setting of '--color-moved' in linkgit:git-diff[1] for details),
+	`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,
+	`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).
diff --git a/Documentation/config/color/grep.txt b/Documentation/config/color/grep.txt
new file mode 100644
index 00000000000..390838450b3
--- /dev/null
+++ b/Documentation/config/color/grep.txt
@@ -0,0 +1,70 @@
+color.diff::
+	Whether to use ANSI escape sequences to add color to patches.
+	If this is set to `always`, linkgit:git-diff[1],
+	linkgit:git-log[1], and linkgit:git-show[1] will use color
+	for all patches.  If it is set to `true` or `auto`, those
+	commands will only use color when output is to the terminal.
+	If unset, then the value of `color.ui` is used (`auto` by
+	default).
++
+This does not affect linkgit:git-format-patch[1] or the
+'git-diff-{asterisk}' plumbing commands.  Can be overridden on the
+command line with the `--color[=<when>]` option.
+
+color.diff.<slot>::
+	Use customized color for diff colorization.  `<slot>` specifies
+	which part of the patch to use the specified color, and is one
+	of `context` (context text - `plain` is a historical synonym),
+	`meta` (metainformation), `frag`
+	(hunk header), 'func' (function in hunk header), `old` (removed lines),
+	`new` (added lines), `commit` (commit headers), `whitespace`
+	(highlighting whitespace errors), `oldMoved` (deleted lines),
+	`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,
+	`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`
+	`newMovedAlternativeDimmed` (See the '<mode>'
+	setting of '--color-moved' in linkgit:git-diff[1] for details),
+	`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,
+	`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).
+
+color.decorate.<slot>::
+	Use customized color for 'git log --decorate' output.  `<slot>` is one
+	of `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local
+	branches, remote-tracking branches, tags, stash and HEAD, respectively
+	and `grafted` for grafted commits.
+
+color.grep::
+	When set to `always`, always highlight matches.  When `false` (or
+	`never`), never.  When set to `true` or `auto`, use color only
+	when the output is written to the terminal.  If unset, then the
+	value of `color.ui` is used (`auto` by default).
+
+color.grep.<slot>::
+	Use customized color for grep colorization.  `<slot>` specifies which
+	part of the line to use the specified color, and is one of
++
+--
+`context`;;
+	non-matching text in context lines (when using `-A`, `-B`, or `-C`)
+`filename`;;
+	filename prefix (when not using `-h`)
+`function`;;
+	function name lines (when using `-p`)
+`lineNumber`;;
+	line number prefix (when using `-n`)
+`column`;;
+	column number prefix (when using `--column`)
+`match`;;
+	matching text (same as setting `matchContext` and `matchSelected`)
+`matchContext`;;
+	matching text in context lines
+`matchSelected`;;
+	matching text in selected lines. Also, used to customize the following
+	linkgit:git-log[1] subcommands: `--grep`, `--author` and `--committer`.
+`selected`;;
+	non-matching text in selected lines. Also, used to customize the
+	following linkgit:git-log[1] subcommands: `--grep`, `--author` and
+	`--committer`.
+`separator`;;
+	separators between fields on a line (`:`, `-`, and `=`)
+	and between hunks (`--`)
+--
diff --git a/Documentation/config/color/log.txt b/Documentation/config/color/log.txt
new file mode 100644
index 00000000000..056a027804c
--- /dev/null
+++ b/Documentation/config/color/log.txt
@@ -0,0 +1,9 @@
+color.decorate.<slot>::
+color.logDecorate.<slot>::
+	Use customized color for 'git log --decorate' output.  `<slot>` is one
+	of `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local
+	branches, remote-tracking branches, tags, stash and HEAD, respectively
+	and `grafted` for grafted commits.
++
+The `color.logDecorate` name is preferred to the historical
+`color.decorate.
diff --git a/Documentation/config/color/push.txt b/Documentation/config/color/push.txt
new file mode 100644
index 00000000000..8c5f19d12df
--- /dev/null
+++ b/Documentation/config/color/push.txt
@@ -0,0 +1,8 @@
+color.push::
+	A boolean to enable/disable color in push errors. May be set to
+	`always`, `false` (or `never`) or `auto` (or `true`), in which
+	case colors are used only when the error output goes to a terminal.
+	If unset, then the value of `color.ui` is used (`auto` by default).
+
+color.push.error::
+	Use customized color for push errors.
diff --git a/Documentation/config/color/showbranch.txt b/Documentation/config/color/showbranch.txt
new file mode 100644
index 00000000000..af17e085213
--- /dev/null
+++ b/Documentation/config/color/showbranch.txt
@@ -0,0 +1,6 @@
+color.showBranch::
+	A boolean to enable/disable color in the output of
+	linkgit:git-show-branch[1]. May be set to `always`,
+	`false` (or `never`) or `auto` (or `true`), in which case colors are used
+	only when the output is to a terminal. If unset, then the
+	value of `color.ui` is used (`auto` by default).
diff --git a/Documentation/config/color/status.txt b/Documentation/config/color/status.txt
new file mode 100644
index 00000000000..d37bec5f282
--- /dev/null
+++ b/Documentation/config/color/status.txt
@@ -0,0 +1,20 @@
+color.status::
+	A boolean to enable/disable color in the output of
+	linkgit:git-status[1]. May be set to `always`,
+	`false` (or `never`) or `auto` (or `true`), in which case colors are used
+	only when the output is to a terminal. If unset, then the
+	value of `color.ui` is used (`auto` by default).
+
+color.status.<slot>::
+	Use customized color for status colorization. `<slot>` is
+	one of `header` (the header text of the status message),
+	`added` or `updated` (files which are added but not committed),
+	`changed` (files which are changed but not added in the index),
+	`untracked` (files which are not tracked by Git),
+	`branch` (the current branch),
+	`nobranch` (the color the 'no branch' warning is shown in, defaulting
+	to red),
+	`localBranch` or `remoteBranch` (the local and remote branch names,
+	respectively, when branch and tracking information is displayed in the
+	status short-format), or
+	`unmerged` (files which have unmerged changes).
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..6540008f319 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -411,18 +411,7 @@ You probably do not need to adjust this value.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
-core.bigFileThreshold::
-	Files larger than this size are stored deflated, without
-	attempting delta compression.  Storing large files without
-	delta compression avoids excessive memory usage, at the
-	slight expense of increased disk usage. Additionally files
-	larger than this size are always treated as binary.
-+
-Default is 512 MiB on all platforms.  This should be reasonable
-for most projects as source code and other text files can still
-be delta compressed, but larger binary media files won't be.
-+
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+include::core/bigFileThreshold.txt[]
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
diff --git a/Documentation/config/core/bigFileThreshold.txt b/Documentation/config/core/bigFileThreshold.txt
new file mode 100644
index 00000000000..4eafd44013b
--- /dev/null
+++ b/Documentation/config/core/bigFileThreshold.txt
@@ -0,0 +1,12 @@
+core.bigFileThreshold::
+	Files larger than this size are stored deflated, without
+	attempting delta compression.  Storing large files without
+	delta compression avoids excessive memory usage, at the
+	slight expense of increased disk usage. Additionally files
+	larger than this size are always treated as binary.
++
+Default is 512 MiB on all platforms.  This should be reasonable
+for most projects as source code and other text files can still
+be delta compressed, but larger binary media files won't be.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt
index 32f84838ac1..35a7bf86d77 100644
--- a/Documentation/config/diff.txt
+++ b/Documentation/config/diff.txt
@@ -178,21 +178,6 @@ diff.<driver>.cachetextconv::
 	Set this option to true to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
-diff.tool::
-	Controls which diff tool is used by linkgit:git-difftool[1].
-	This variable overrides the value configured in `merge.tool`.
-	The list below shows the valid built-in values.
-	Any other value is treated as a custom diff tool and requires
-	that a corresponding difftool.<tool>.cmd variable is defined.
-
-diff.guitool::
-	Controls which diff tool is used by linkgit:git-difftool[1] when
-	the -g/--gui flag is specified. This variable overrides the value
-	configured in `merge.guitool`. The list below shows the valid
-	built-in values. Any other value is treated as a custom diff tool
-	and requires that a corresponding difftool.<guitool>.cmd variable
-	is defined.
-
 include::../mergetools-diff.txt[]
 
 diff.indentHeuristic::
diff --git a/Documentation/config/diff/difftool.txt b/Documentation/config/diff/difftool.txt
new file mode 100644
index 00000000000..652bc2bd8e1
--- /dev/null
+++ b/Documentation/config/diff/difftool.txt
@@ -0,0 +1,14 @@
+diff.tool::
+	Controls which diff tool is used by linkgit:git-difftool[1].
+	This variable overrides the value configured in `merge.tool`.
+	The list below shows the valid built-in values.
+	Any other value is treated as a custom diff tool and requires
+	that a corresponding difftool.<tool>.cmd variable is defined.
+
+diff.guitool::
+	Controls which diff tool is used by linkgit:git-difftool[1] when
+	the -g/--gui flag is specified. This variable overrides the value
+	configured in `merge.guitool`. The list below shows the valid
+	built-in values. Any other value is treated as a custom diff tool
+	and requires that a corresponding difftool.<guitool>.cmd variable
+	is defined.
diff --git a/Documentation/config/difftool.txt b/Documentation/config/difftool.txt
index 67625944804..dcf6213d31d 100644
--- a/Documentation/config/difftool.txt
+++ b/Documentation/config/difftool.txt
@@ -9,6 +9,13 @@ difftool.<tool>.cmd::
 	file containing the contents of the diff pre-image and 'REMOTE'
 	is set to the name of the temporary file containing the contents
 	of the diff post-image.
++
+See the `--tool=<tool>` option linkgit:git-difftool[1] for more details.
 
 difftool.prompt::
 	Prompt before each invocation of the diff tool.
+
+difftool.trustExitCode::
+	Exit difftool if the invoked diff tool returns a non-zero exit status.
++
+See the `--trust-exit-code` option in linkgit:git-difftool[1] for more details.
diff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt
index c834e07991f..a8873dca580 100644
--- a/Documentation/config/gc.txt
+++ b/Documentation/config/gc.txt
@@ -123,14 +123,4 @@ or rebase occurring.  Since these changes are not part of the current
 project most users will want to expire them sooner, which is why the
 default is more aggressive than `gc.reflogExpire`.
 
-gc.rerereResolved::
-	Records of conflicted merge you resolved earlier are
-	kept for this many days when 'git rerere gc' is run.
-	You can also use more human-readable "1.month.ago", etc.
-	The default is 60 days.  See linkgit:git-rerere[1].
-
-gc.rerereUnresolved::
-	Records of conflicted merge you have not resolved are
-	kept for this many days when 'git rerere gc' is run.
-	You can also use more human-readable "1.month.ago", etc.
-	The default is 15 days.  See linkgit:git-rerere[1].
+include::gc/rerere.txt[]
diff --git a/Documentation/config/gc/rerere.txt b/Documentation/config/gc/rerere.txt
new file mode 100644
index 00000000000..6d0544771d8
--- /dev/null
+++ b/Documentation/config/gc/rerere.txt
@@ -0,0 +1,11 @@
+gc.rerereResolved::
+	Records of conflicted merge you resolved earlier are
+	kept for this many days when 'git rerere gc' is run.
+	You can also use more human-readable "1.month.ago", etc.
+	The default is 60 days.  See linkgit:git-rerere[1].
+
+gc.rerereUnresolved::
+	Records of conflicted merge you have not resolved are
+	kept for this many days when 'git rerere gc' is run.
+	You can also use more human-readable "1.month.ago", etc.
+	The default is 15 days.  See linkgit:git-rerere[1].
diff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt
index 182edd813a5..e521f20390c 100644
--- a/Documentation/config/grep.txt
+++ b/Documentation/config/grep.txt
@@ -17,8 +17,11 @@ grep.extendedRegexp::
 	other than 'default'.
 
 grep.threads::
-	Number of grep worker threads to use.
-	See `grep.threads` in linkgit:git-grep[1] for more information.
+	Number of grep worker threads to use. If unset (or set to 0), Git will
+	use as many threads as the number of logical cores available.
+
+grep.fullName::
+	If set to true, enable `--full-name` option by default.
 
 grep.fallbackToNoIndex::
 	If set to true, fall back to git grep --no-index if git grep
diff --git a/Documentation/config/log.txt b/Documentation/config/log.txt
index 456eb07800c..12ede8e31b8 100644
--- a/Documentation/config/log.txt
+++ b/Documentation/config/log.txt
@@ -1,12 +1,11 @@
-log.abbrevCommit::
-	If true, makes linkgit:git-log[1], linkgit:git-show[1], and
-	linkgit:git-whatchanged[1] assume `--abbrev-commit`. You may
-	override this option with `--no-abbrev-commit`.
-
 log.date::
 	Set the default date-time mode for the 'log' command.
 	Setting a value for log.date is similar to using 'git log''s
 	`--date` option.  See linkgit:git-log[1] for details.
++
+If the format is set to "auto:foo" and the pager is in use, format
+"foo" will be the used for the date format. Otherwise "default" will
+be used.
 
 log.decorate::
 	Print out the ref names of any commits that are shown by the log
@@ -45,6 +44,11 @@ log.showRoot::
 	Tools like linkgit:git-log[1] or linkgit:git-whatchanged[1], which
 	normally hide the root commit will now show it. True by default.
 
+log.abbrevCommit::
+	If true, makes linkgit:git-log[1], linkgit:git-show[1], and
+	linkgit:git-whatchanged[1] assume `--abbrev-commit`. You may
+	override this option with `--no-abbrev-commit`.
+
 log.showSignature::
 	If true, makes linkgit:git-log[1], linkgit:git-show[1], and
 	linkgit:git-whatchanged[1] assume `--show-signature`.
@@ -53,3 +57,7 @@ log.mailmap::
 	If true, makes linkgit:git-log[1], linkgit:git-show[1], and
 	linkgit:git-whatchanged[1] assume `--use-mailmap`, otherwise
 	assume `--no-use-mailmap`. True by default.
++
+The behavior of `log.mailmap` is impacted by the setting of the more
+general 'mailmap.*' configuration variables, i.e. `mailmap.file` and
+`mailmap.blob`.
diff --git a/Documentation/config/notes.txt b/Documentation/config/notes.txt
index aeef56d49ae..c7c4811734b 100644
--- a/Documentation/config/notes.txt
+++ b/Documentation/config/notes.txt
@@ -3,6 +3,9 @@ notes.mergeStrategy::
 	conflicts.  Must be one of `manual`, `ours`, `theirs`, `union`, or
 	`cat_sort_uniq`.  Defaults to `manual`.  See "NOTES MERGE STRATEGIES"
 	section of linkgit:git-notes[1] for more information on each strategy.
++
+This setting can be overridden by passing the `--strategy` option to
+linkgit:git-notes[1].
 
 notes.<name>.mergeStrategy::
 	Which merge strategy to choose when doing a notes merge into
@@ -11,28 +14,35 @@ notes.<name>.mergeStrategy::
 	linkgit:git-notes[1] for more information on the available strategies.
 
 notes.displayRef::
-	The (fully qualified) refname from which to show notes when
-	showing commit messages.  The value of this variable can be set
-	to a glob, in which case notes from all matching refs will be
-	shown.  You may also specify this configuration variable
-	several times.  A warning will be issued for refs that do not
-	exist, but a glob that does not match any refs is silently
-	ignored.
+	Which ref (or refs, if a glob or specified more than once), in
+	addition to the default set by `core.notesRef` or
+	`GIT_NOTES_REF`, to read notes from when showing commit
+	messages with the 'git log' family of commands.
 +
 This setting can be overridden with the `GIT_NOTES_DISPLAY_REF`
 environment variable, which must be a colon separated list of refs or
 globs.
 +
+A warning will be issued for refs that do not exist,
+but a glob that does not match any refs is silently ignored.
++
+This setting can be disabled by the `--no-notes` option to the 'git
+log' family of commands, or by the `--notes=<ref>` option accepted by
+those commands.
++
 The effective value of "core.notesRef" (possibly overridden by
 GIT_NOTES_REF) is also implicitly added to the list of refs to be
 displayed.
 
 notes.rewrite.<command>::
 	When rewriting commits with <command> (currently `amend` or
-	`rebase`) and this variable is set to `true`, Git
-	automatically copies your notes from the original to the
-	rewritten commit.  Defaults to `true`, but see
-	"notes.rewriteRef" below.
+	`rebase`), if this variable is `false`, git will not copy
+	notes from the original to the rewritten commit.  Defaults to
+	`true`.  See also "`notes.rewriteRef`" below.
++
+This setting can be overridden with the `GIT_NOTES_REWRITE_REF`
+environment variable, which must be a colon separated list of refs or
+globs.
 
 notes.rewriteMode::
 	When copying notes during a rewrite (see the
@@ -46,14 +56,13 @@ environment variable.
 
 notes.rewriteRef::
 	When copying notes during a rewrite, specifies the (fully
-	qualified) ref whose notes should be copied.  The ref may be a
-	glob, in which case notes in all matching refs will be copied.
-	You may also specify this configuration several times.
+	qualified) ref whose notes should be copied.  May be a glob,
+	in which case notes in all matching refs will be copied.  You
+	may also specify this configuration several times.
 +
 Does not have a default value; you must configure this variable to
 enable note rewriting.  Set it to `refs/notes/commits` to enable
 rewriting for the default commit notes.
 +
-This setting can be overridden with the `GIT_NOTES_REWRITE_REF`
-environment variable, which must be a colon separated list of refs or
-globs.
+Can be overridden with the `GIT_NOTES_REWRITE_REF` environment variable.
+See `notes.rewrite.<command>` above for a further description of its format.
diff --git a/Documentation/config/sendemail.txt b/Documentation/config/sendemail.txt
index 50baa5d6bfb..51da7088a84 100644
--- a/Documentation/config/sendemail.txt
+++ b/Documentation/config/sendemail.txt
@@ -18,17 +18,49 @@ sendemail.<identity>.*::
 	identity is selected, through either the command-line or
 	`sendemail.identity`.
 
+sendemail.multiEdit::
+	If true (default), a single editor instance will be spawned to edit
+	files you have to edit (patches when `--annotate` is used, and the
+	summary when `--compose` is used). If false, files will be edited one
+	after the other, spawning a new editor each time.
+
+sendemail.confirm::
+	Sets the default for whether to confirm before sending. Must be
+	one of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm`
+	in the linkgit:git-send-email[1] documentation for the meaning of these
+	values.
+
 sendemail.aliasesFile::
+	To avoid typing long email addresses, point this to one or more
+	email aliases files.  You must also supply `sendemail.aliasFileType`.
+
 sendemail.aliasFileType::
+	Format of the file(s) specified in sendemail.aliasesFile. Must be
+	one of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'.
++
+What an alias file in each format looks like can be found in
+the documentation of the email program of the same name. The
+differences and limitations from the standard formats are
+described below:
++
+--
+sendmail;;
+*	Quoted aliases and quoted addresses are not supported: lines that
+	contain a `"` symbol are ignored.
+*	Redirection to a file (`/path/name`) or pipe (`|command`) is not
+	supported.
+*	File inclusion (`:include: /path/name`) is not supported.
+*	Warnings are printed on the standard error output for any
+	explicitly unsupported constructs, and any other lines that are not
+	recognized by the parser.
+--
 sendemail.annotate::
 sendemail.bcc::
 sendemail.cc::
 sendemail.ccCmd::
 sendemail.chainReplyTo::
-sendemail.confirm::
 sendemail.envelopeSender::
 sendemail.from::
-sendemail.multiEdit::
 sendemail.signedoffbycc::
 sendemail.smtpPass::
 sendemail.suppresscc::
@@ -44,7 +76,9 @@ sendemail.thread::
 sendemail.transferEncoding::
 sendemail.validate::
 sendemail.xmailer::
-	See linkgit:git-send-email[1] for description.
+	These configuration variables all provide a default for
+	linkgit:git-send-email[1] command-line options. See its
+	documentation for details.
 
 sendemail.signedoffcc (deprecated)::
 	Deprecated alias for `sendemail.signedoffbycc`.
diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt
index 32fad5bbe81..d3e5ce7236d 100644
--- a/Documentation/config/uploadpack.txt
+++ b/Documentation/config/uploadpack.txt
@@ -27,31 +27,7 @@ uploadpack.allowAnySHA1InWant::
 	object at all.
 	Defaults to `false`.
 
-uploadpack.keepAlive::
-	When `upload-pack` has started `pack-objects`, there may be a
-	quiet period while `pack-objects` prepares the pack. Normally
-	it would output progress information, but if `--quiet` was used
-	for the fetch, `pack-objects` will output nothing at all until
-	the pack data begins. Some clients and networks may consider
-	the server to be hung and give up. Setting this option instructs
-	`upload-pack` to send an empty keepalive packet every
-	`uploadpack.keepAlive` seconds. Setting this option to 0
-	disables keepalive packets entirely. The default is 5 seconds.
-
-uploadpack.packObjectsHook::
-	If this option is set, when `upload-pack` would run
-	`git pack-objects` to create a packfile for a client, it will
-	run this shell command instead.  The `pack-objects` command and
-	arguments it _would_ have run (including the `git pack-objects`
-	at the beginning) are appended to the shell command. The stdin
-	and stdout of the hook are treated as if `pack-objects` itself
-	was run. I.e., `upload-pack` will feed input intended for
-	`pack-objects` to the hook, and expects a completed packfile on
-	stdout.
-+
-Note that this configuration variable is ignored if it is seen in the
-repository-level config (this is a safety measure against fetching from
-untrusted repositories).
+include::uploadpack/pack-objects.txt[]
 
 uploadpack.allowFilter::
 	If this option is set, `upload-pack` will support partial
diff --git a/Documentation/config/uploadpack/pack-objects.txt b/Documentation/config/uploadpack/pack-objects.txt
new file mode 100644
index 00000000000..0447c112811
--- /dev/null
+++ b/Documentation/config/uploadpack/pack-objects.txt
@@ -0,0 +1,25 @@
+uploadpack.keepAlive::
+	When `upload-pack` has started `pack-objects`, there may be a
+	quiet period while `pack-objects` prepares the pack. Normally
+	it would output progress information, but if `--quiet` was used
+	for the fetch, `pack-objects` will output nothing at all until
+	the pack data begins. Some clients and networks may consider
+	the server to be hung and give up. Setting this option instructs
+	`upload-pack` to send an empty keepalive packet every
+	`uploadpack.keepAlive` seconds. Setting this option to 0
+	disables keepalive packets entirely. The default is 5 seconds.
+
+uploadpack.packObjectsHook::
+	If this option is set, when `upload-pack` would run
+	`git pack-objects` to create a packfile for a client, it will
+	run this shell command instead.  The `pack-objects` command and
+	arguments it _would_ have run (including the `git pack-objects`
+	at the beginning) are appended to the shell command. The stdin
+	and stdout of the hook are treated as if `pack-objects` itself
+	was run. I.e., `upload-pack` will feed input intended for
+	`pack-objects` to the hook, and expects a completed packfile on
+	stdout.
++
+Note that this configuration variable is ignored if it is seen in the
+repository-level config (this is a safety measure against fetching from
+untrusted repositories).
diff --git a/Documentation/git-add.txt b/Documentation/git-add.txt
index 11eb70f16c7..36da8bf9b51 100644
--- a/Documentation/git-add.txt
+++ b/Documentation/git-add.txt
@@ -431,6 +431,13 @@ they will make the patch impossible to apply:
 * deleting context or removal lines
 * modifying the contents of context or removal lines
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/add.txt[]
+
 SEE ALSO
 --------
 linkgit:git-status[1]
diff --git a/Documentation/git-am.txt b/Documentation/git-am.txt
index 09107fb1067..7ccb46b660c 100644
--- a/Documentation/git-am.txt
+++ b/Documentation/git-am.txt
@@ -261,6 +261,13 @@ This command can run `applypatch-msg`, `pre-applypatch`,
 and `post-applypatch` hooks.  See linkgit:githooks[5] for more
 information.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/am.txt[]
+
 SEE ALSO
 --------
 linkgit:git-apply[1].
diff --git a/Documentation/git-apply.txt b/Documentation/git-apply.txt
index b6d77f42068..1d478cbe9b5 100644
--- a/Documentation/git-apply.txt
+++ b/Documentation/git-apply.txt
@@ -263,13 +263,9 @@ has no effect when `--index` or `--cached` is in use.
 CONFIGURATION
 -------------
 
-apply.ignoreWhitespace::
-	Set to 'change' if you want changes in whitespace to be ignored by default.
-	Set to one of: no, none, never, false if you want changes in
-	whitespace to be significant.
-apply.whitespace::
-	When no `--whitespace` flag is given from the command
-	line, this configuration item is used as the default.
+include::includes/cmd-config-section-all.txt[]
+
+include::config/apply.txt[]
 
 SUBMODULES
 ----------
diff --git a/Documentation/git-blame.txt b/Documentation/git-blame.txt
index d7a46cc6744..67942b6964d 100644
--- a/Documentation/git-blame.txt
+++ b/Documentation/git-blame.txt
@@ -242,6 +242,16 @@ MAPPING AUTHORS
 See linkgit:gitmailmap[5].
 
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/blame.txt[]
+
+include::config/color/blame.txt[]
+
+
 SEE ALSO
 --------
 linkgit:git-annotate[1]
diff --git a/Documentation/git-branch.txt b/Documentation/git-branch.txt
index c8b4f9ce3c7..601a84451ed 100644
--- a/Documentation/git-branch.txt
+++ b/Documentation/git-branch.txt
@@ -332,6 +332,12 @@ CONFIGURATION
 `--list` is used or implied. The default is to use a pager.
 See linkgit:git-config[1].
 
+include::includes/cmd-config-section-rest.txt[]
+
+include::config/branch.txt[]
+
+include::config/color/branch.txt[]
+
 EXAMPLES
 --------
 
diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 9f37e22e130..4cb9d555b4b 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -600,6 +600,13 @@ $ edit frotz
 $ git add frotz
 ------------
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/checkout.txt[]
+
 SEE ALSO
 --------
 linkgit:git-switch[1],
diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
index a7f309dff5a..91742633fa8 100644
--- a/Documentation/git-clean.txt
+++ b/Documentation/git-clean.txt
@@ -133,6 +133,13 @@ help::
 
   Show brief usage of interactive git-clean.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/clean.txt[]
+
 SEE ALSO
 --------
 linkgit:gitignore[5]
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 984d194934f..7547cbd5266 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -358,6 +358,15 @@ $ cd my-linux
 $ git clone --bare -l /home/proj/.git /pub/scm/proj.git
 ------------
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/init.txt[]
+
+include::config/clone.txt[]
+
 
 GIT
 ---
diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt
index e1f48c95b3c..c8143e9a3ba 100644
--- a/Documentation/git-commit-graph.txt
+++ b/Documentation/git-commit-graph.txt
@@ -143,6 +143,14 @@ $ git rev-parse HEAD | git commit-graph write --stdin-commits --append
 ------------------------------------------------
 
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/commitgraph.txt[]
+
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-commit.txt b/Documentation/git-commit.txt
index 6c60bf98f9f..225c6c9f2e5 100644
--- a/Documentation/git-commit.txt
+++ b/Documentation/git-commit.txt
@@ -557,6 +557,10 @@ The editor used to edit the commit log message will be chosen from the
 `VISUAL` environment variable, or the `EDITOR` environment variable (in that
 order).  See linkgit:git-var[1] for details.
 
+include::includes/cmd-config-section-rest.txt[]
+
+include::config/commit.txt[]
+
 HOOKS
 -----
 This command can run `commit-msg`, `prepare-commit-msg`, `pre-commit`,
diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f18673017f5..d80d9f671f9 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -160,6 +160,13 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/credential.txt[]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-diff.txt b/Documentation/git-diff.txt
index 6236c75c9b8..c9f39225fb1 100644
--- a/Documentation/git-diff.txt
+++ b/Documentation/git-diff.txt
@@ -213,6 +213,15 @@ $ git diff -R                          <2>
     rewrites (very expensive).
 <2> Output diff in reverse.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/diff.txt[]
+
+include::config/color/diff.txt[]
+
 SEE ALSO
 --------
 diff(1),
diff --git a/Documentation/git-difftool.txt b/Documentation/git-difftool.txt
index 143b0c49d73..9f6fae615a4 100644
--- a/Documentation/git-difftool.txt
+++ b/Documentation/git-difftool.txt
@@ -113,33 +113,17 @@ instead.  `--no-symlinks` is the default on Windows.
 
 See linkgit:git-diff[1] for the full list of supported options.
 
-CONFIG VARIABLES
-----------------
+CONFIGURATION
+-------------
+
 'git difftool' falls back to 'git mergetool' config variables when the
 difftool equivalents have not been defined.
 
-diff.tool::
-	The default diff tool to use.
-
-diff.guitool::
-	The default diff tool to use when `--gui` is specified.
+include::includes/cmd-config-section-rest.txt[]
 
-difftool.<tool>.path::
-	Override the path for the given tool.  This is useful in case
-	your tool is not in the PATH.
-
-difftool.<tool>.cmd::
-	Specify the command to invoke the specified diff tool.
-+
-See the `--tool=<tool>` option above for more details.
+include::config/diff/difftool.txt[]
 
-difftool.prompt::
-	Prompt before each invocation of the diff tool.
-
-difftool.trustExitCode::
-	Exit difftool if the invoked diff tool returns a non-zero exit status.
-+
-See the `--trust-exit-code` option above for more details.
+include::config/difftool.txt[]
 
 SEE ALSO
 --------
diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 39cfa05b28b..8b5dd6add00 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -1564,6 +1564,13 @@ operator can use this facility to peek at the objects and refs from an
 import in progress, at the cost of some added running time and worse
 compression.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/fastimport.txt[]
+
 SEE ALSO
 --------
 linkgit:git-fast-export[1]
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index 550c16ca616..3c66dd584ff 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -285,6 +285,15 @@ linkgit:git-gc[1]).
 
 include::transfer-data-leaks.txt[]
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-rest.txt[]
+
+include::config/fetch.txt[]
+
+include::config/remote.txt[]
+
 BUGS
 ----
 Using --recurse-submodules can only fetch new commits in already checked
diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 853967dea01..2d2b142058e 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -103,14 +103,6 @@ more time, and the resulting space/delta optimization may or may not
 be worth it. Not using this at all is the right trade-off for most
 users and their repositories.
 
-CONFIGURATION
--------------
-
-The below documentation is the same as what's found in
-linkgit:git-config[1]:
-
-include::config/gc.txt[]
-
 NOTES
 -----
 
@@ -150,6 +142,12 @@ HOOKS
 The 'git gc --auto' command will run the 'pre-auto-gc' hook.  See
 linkgit:githooks[5] for more information.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/gc.txt[]
 
 SEE ALSO
 --------
diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt
index 3d393fbac1b..70f3f7993e7 100644
--- a/Documentation/git-grep.txt
+++ b/Documentation/git-grep.txt
@@ -334,34 +334,9 @@ performance in this case, it might be desirable to use `--threads=1`.
 CONFIGURATION
 -------------
 
-grep.lineNumber::
-	If set to true, enable `-n` option by default.
-
-grep.column::
-	If set to true, enable the `--column` option by default.
-
-grep.patternType::
-	Set the default matching behavior. Using a value of 'basic', 'extended',
-	'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,
-	`--fixed-strings`, or `--perl-regexp` option accordingly, while the
-	value 'default' will return to the default matching behavior.
-
-grep.extendedRegexp::
-	If set to true, enable `--extended-regexp` option by default. This
-	option is ignored when the `grep.patternType` option is set to a value
-	other than 'default'.
-
-grep.threads::
-	Number of grep worker threads to use. If unset (or set to 0), Git will
-	use as many threads as the number of logical cores available.
-
-grep.fullName::
-	If set to true, enable `--full-name` option by default.
-
-grep.fallbackToNoIndex::
-	If set to true, fall back to git grep --no-index if git grep
-	is executed outside of a git repository.  Defaults to false.
+include::includes/cmd-config-section-all.txt[]
 
+include::config/grep.txt[]
 
 GIT
 ---
diff --git a/Documentation/git-log.txt b/Documentation/git-log.txt
index 20e87cecf49..cb66f0a9cac 100644
--- a/Documentation/git-log.txt
+++ b/Documentation/git-log.txt
@@ -42,6 +42,10 @@ OPTIONS
 	shown. The option `--decorate` is short-hand for `--decorate=short`.
 	Default to configuration value of `log.decorate` if configured,
 	otherwise, `auto`.
++
+This output may be colored. See `color.ui` in linkgit:git-config[1]
+and `color.logDecorate` (also known as `color.decorate`) there and in
+the "CONFIGURATION" section below.
 
 --decorate-refs=<pattern>::
 --decorate-refs-exclude=<pattern>::
@@ -199,47 +203,11 @@ i18n.logOutputEncoding::
 	Defaults to the value of `i18n.commitEncoding` if set, and UTF-8
 	otherwise.
 
-log.date::
-	Default format for human-readable dates.  (Compare the
-	`--date` option.)  Defaults to "default", which means to write
-	dates like `Sat May 8 19:35:34 2010 -0500`.
-+
-If the format is set to "auto:foo" and the pager is in use, format
-"foo" will be the used for the date format. Otherwise "default" will
-be used.
-
-log.follow::
-	If `true`, `git log` will act as if the `--follow` option was used when
-	a single <path> is given.  This has the same limitations as `--follow`,
-	i.e. it cannot be used to follow multiple files and does not work well
-	on non-linear history.
-
-log.showRoot::
-	If `false`, `git log` and related commands will not treat the
-	initial commit as a big creation event.  Any root commits in
-	`git log -p` output would be shown without a diff attached.
-	The default is `true`.
-
-log.showSignature::
-	If `true`, `git log` and related commands will act as if the
-	`--show-signature` option was passed to them.
-
-mailmap.*::
-	See linkgit:git-shortlog[1].
-
-notes.displayRef::
-	Which refs, in addition to the default set by `core.notesRef`
-	or `GIT_NOTES_REF`, to read notes from when showing commit
-	messages with the `log` family of commands.  See
-	linkgit:git-notes[1].
-+
-May be an unabbreviated ref name or a glob and may be specified
-multiple times.  A warning will be issued for refs that do not exist,
-but a glob that does not match any refs is silently ignored.
-+
-This setting can be disabled by the `--no-notes` option,
-overridden by the `GIT_NOTES_DISPLAY_REF` environment variable,
-and overridden by the `--notes=<ref>` option.
+include::includes/cmd-config-section-rest.txt[]
+
+include::config/log.txt[]
+
+include::config/notes.txt[]
 
 GIT
 ---
diff --git a/Documentation/git-mergetool.txt b/Documentation/git-mergetool.txt
index e587c7763a7..5a9d94d3d17 100644
--- a/Documentation/git-mergetool.txt
+++ b/Documentation/git-mergetool.txt
@@ -101,6 +101,9 @@ success of the resolution after the custom tool has exited.
 
 CONFIGURATION
 -------------
+
+include::includes/cmd-config-section-all.txt[]
+
 include::config/mergetool.txt[]
 
 TEMPORARY FILES
diff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt
index 0a4200674c4..68d547e2474 100644
--- a/Documentation/git-notes.txt
+++ b/Documentation/git-notes.txt
@@ -44,7 +44,7 @@ using the `--notes` option. Such notes are added as a patch commentary
 after a three dash separator line.
 
 To change which notes are shown by 'git log', see the
-"notes.displayRef" configuration in linkgit:git-log[1].
+"notes.displayRef" configuration in linkgit:git-config[1].
 
 See the "notes.rewrite.<command>" configuration for a way to carry
 notes across commands that rewrite commits.
@@ -316,57 +316,9 @@ core.notesRef::
 	This setting can be overridden through the environment and
 	command line.
 
-notes.mergeStrategy::
-	Which merge strategy to choose by default when resolving notes
-	conflicts.  Must be one of `manual`, `ours`, `theirs`, `union`, or
-	`cat_sort_uniq`.  Defaults to `manual`.  See "NOTES MERGE STRATEGIES"
-	section above for more information on each strategy.
-+
-This setting can be overridden by passing the `--strategy` option.
-
-notes.<name>.mergeStrategy::
-	Which merge strategy to choose when doing a notes merge into
-	refs/notes/<name>.  This overrides the more general
-	"notes.mergeStrategy".  See the "NOTES MERGE STRATEGIES" section above
-	for more information on each available strategy.
-
-notes.displayRef::
-	Which ref (or refs, if a glob or specified more than once), in
-	addition to the default set by `core.notesRef` or
-	`GIT_NOTES_REF`, to read notes from when showing commit
-	messages with the 'git log' family of commands.
-	This setting can be overridden on the command line or by the
-	`GIT_NOTES_DISPLAY_REF` environment variable.
-	See linkgit:git-log[1].
-
-notes.rewrite.<command>::
-	When rewriting commits with <command> (currently `amend` or
-	`rebase`), if this variable is `false`, git will not copy
-	notes from the original to the rewritten commit.  Defaults to
-	`true`.  See also "`notes.rewriteRef`" below.
-+
-This setting can be overridden by the `GIT_NOTES_REWRITE_REF`
-environment variable.
+include::includes/cmd-config-section-rest.txt[]
 
-notes.rewriteMode::
-	When copying notes during a rewrite, what to do if the target
-	commit already has a note.  Must be one of `overwrite`,
-	`concatenate`, `cat_sort_uniq`, or `ignore`.  Defaults to
-	`concatenate`.
-+
-This setting can be overridden with the `GIT_NOTES_REWRITE_MODE`
-environment variable.
-
-notes.rewriteRef::
-	When copying notes during a rewrite, specifies the (fully
-	qualified) ref whose notes should be copied.  May be a glob,
-	in which case notes in all matching refs will be copied.  You
-	may also specify this configuration several times.
-+
-Does not have a default value; you must configure this variable to
-enable note rewriting.
-+
-Can be overridden with the `GIT_NOTES_REWRITE_REF` environment variable.
+include::config/notes.txt[]
 
 
 ENVIRONMENT
diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
index f8344e1e5ba..41042ebcc7d 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -414,12 +414,10 @@ over user-wide config, and so forth).
 CONFIGURATION
 -------------
 
-Various configuration variables affect packing, see
-linkgit:git-config[1] (search for "pack" and "delta").
+include::templates/configuration-include-pack.txt[]
+
+include::config/uploadpack/pack-objects.txt[]
 
-Notably, delta compression is not used on objects larger than the
-`core.bigFileThreshold` configuration variable and on files with the
-attribute `delta` set to false.
 
 SEE ALSO
 --------
diff --git a/Documentation/git-remote.txt b/Documentation/git-remote.txt
index 2bebc32566b..03ae8399bae 100644
--- a/Documentation/git-remote.txt
+++ b/Documentation/git-remote.txt
@@ -196,13 +196,6 @@ be updated.  (See linkgit:git-config[1]).
 With `--prune` option, run pruning against all the remotes that are updated.
 
 
-DISCUSSION
-----------
-
-The remote configuration is achieved using the `remote.origin.url` and
-`remote.origin.fetch` configuration variables.  (See
-linkgit:git-config[1]).
-
 EXIT STATUS
 -----------
 
@@ -255,6 +248,14 @@ $ git remote add -f -t master -m master origin git://example.com/git.git/
 $ git merge origin
 ------------
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/remote.txt[]
+
+include::config/remotes.txt[]
 
 SEE ALSO
 --------
diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
index ee30edc178a..893a2a7a281 100644
--- a/Documentation/git-repack.txt
+++ b/Documentation/git-repack.txt
@@ -204,22 +204,9 @@ linkgit:git-multi-pack-index[1]).
 CONFIGURATION
 -------------
 
-Various configuration variables affect packing, see
-linkgit:git-config[1] (search for "pack" and "delta").
-
-By default, the command passes `--delta-base-offset` option to
-'git pack-objects'; this typically results in slightly smaller packs,
-but the generated packs are incompatible with versions of Git older than
-version 1.4.4. If you need to share your repository with such ancient Git
-versions, either directly or via the dumb http protocol, then you
-need to set the configuration variable `repack.UseDeltaBaseOffset` to
-"false" and repack. Access from old Git versions over the native protocol
-is unaffected by this option as the conversion is performed on the fly
-as needed in that case.
-
-Delta compression is not used on objects larger than the
-`core.bigFileThreshold` configuration variable and on files with the
-attribute `delta` set to false.
+include::templates/configuration-include-pack.txt[]
+
+include::config/repack.txt[]
 
 SEE ALSO
 --------
diff --git a/Documentation/git-rerere.txt b/Documentation/git-rerere.txt
index 4cfc8833780..1a32ca368c1 100644
--- a/Documentation/git-rerere.txt
+++ b/Documentation/git-rerere.txt
@@ -217,6 +217,15 @@ same as lines with conflict markers, 'git rerere' may fail to record a
 conflict resolution.  To work around this, the `conflict-marker-size`
 setting in linkgit:gitattributes[5] can be used.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/rerere.txt[]
+
+include::config/gc.txt[]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt
index 6f7685f53d5..a193024d39f 100644
--- a/Documentation/git-reset.txt
+++ b/Documentation/git-reset.txt
@@ -499,6 +499,13 @@ working index HEAD target         working index HEAD
 
 `X` means any state and `U` means an unmerged index.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/reset.txt[]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-send-email.txt b/Documentation/git-send-email.txt
index 41cd8cb4247..3290043053a 100644
--- a/Documentation/git-send-email.txt
+++ b/Documentation/git-send-email.txt
@@ -456,41 +456,9 @@ Information
 CONFIGURATION
 -------------
 
-sendemail.aliasesFile::
-	To avoid typing long email addresses, point this to one or more
-	email aliases files.  You must also supply `sendemail.aliasFileType`.
+include::includes/cmd-config-section-all.txt[]
 
-sendemail.aliasFileType::
-	Format of the file(s) specified in sendemail.aliasesFile. Must be
-	one of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'.
-+
-What an alias file in each format looks like can be found in
-the documentation of the email program of the same name. The
-differences and limitations from the standard formats are
-described below:
-+
---
-sendmail;;
-*	Quoted aliases and quoted addresses are not supported: lines that
-	contain a `"` symbol are ignored.
-*	Redirection to a file (`/path/name`) or pipe (`|command`) is not
-	supported.
-*	File inclusion (`:include: /path/name`) is not supported.
-*	Warnings are printed on the standard error output for any
-	explicitly unsupported constructs, and any other lines that are not
-	recognized by the parser.
---
-
-sendemail.multiEdit::
-	If true (default), a single editor instance will be spawned to edit
-	files you have to edit (patches when `--annotate` is used, and the
-	summary when `--compose` is used). If false, files will be edited one
-	after the other, spawning a new editor each time.
-
-sendemail.confirm::
-	Sets the default for whether to confirm before sending. Must be
-	one of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm`
-	in the previous section for the meaning of these values.
+include::config/sendemail.txt[]
 
 EXAMPLES
 --------
diff --git a/Documentation/git-show-branch.txt b/Documentation/git-show-branch.txt
index 5cc2fcefbab..e5ec6b467f9 100644
--- a/Documentation/git-show-branch.txt
+++ b/Documentation/git-show-branch.txt
@@ -199,6 +199,13 @@ shows 10 reflog entries going back from the tip as of 1 hour ago.
 Without `--list`, the output also shows how these tips are
 topologically related with each other.
 
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/showbranch.txt[]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt
index a16e62bc8c8..fa1ebb13dbe 100644
--- a/Documentation/githooks.txt
+++ b/Documentation/githooks.txt
@@ -702,6 +702,10 @@ SEE ALSO
 --------
 linkgit:git-hook[1]
 
+The `uploadpack.packObjectsHook` configuration variable can configure
+a hook for linkgit:git-pack-objects[1] to use, but not in a way that
+uses the hook facility discussed above.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/includes/cmd-config-section-all.txt b/Documentation/includes/cmd-config-section-all.txt
new file mode 100644
index 00000000000..296a239f2ab
--- /dev/null
+++ b/Documentation/includes/cmd-config-section-all.txt
@@ -0,0 +1,3 @@
+Everything below this line in this section is selectively included
+from the linkgit:git-config[1] documentation. The content is the same
+as what's found there:
diff --git a/Documentation/includes/cmd-config-section-rest.txt b/Documentation/includes/cmd-config-section-rest.txt
new file mode 100644
index 00000000000..fe4721dac76
--- /dev/null
+++ b/Documentation/includes/cmd-config-section-rest.txt
@@ -0,0 +1,3 @@
+The discussion above this line is unique to this documentation, but
+everything below is included from section(s) in linkgit:git-config[1]
+aplicable to this command:
diff --git a/Documentation/templates/configuration-include-pack.txt b/Documentation/templates/configuration-include-pack.txt
new file mode 100644
index 00000000000..fa80f68b658
--- /dev/null
+++ b/Documentation/templates/configuration-include-pack.txt
@@ -0,0 +1,10 @@
+Various configuration variables affect packing. Those variables and
+more are discussed in linkgit:git-config[1].
+
+The below documentation is selectively included from
+linkgit:git-config[1]. It should be all the configuration variables
+relevant to this command, but some may have been missed. When in doubt
+doubt consult linkgit:git-config[1] for the full listing:
+
+include::../config/core/bigFileThreshold.txt[]
+include::../config/pack.txt[]
diff --git a/builtin/help.c b/builtin/help.c
index d387131dd83..d03fc7aaac4 100644
--- a/builtin/help.c
+++ b/builtin/help.c
@@ -94,7 +94,7 @@ static void list_config_help(enum show_config_type type)
 	struct slot_expansion slot_expansions[] = {
 		{ "advice", "*", list_config_advices },
 		{ "color.branch", "<slot>", list_config_color_branch_slots },
-		{ "color.decorate", "<slot>", list_config_color_decorate_slots },
+		{ "color.logDecorate", "<slot>", list_config_color_decorate_slots },
 		{ "color.diff", "<slot>", list_config_color_diff_slots },
 		{ "color.grep", "<slot>", list_config_color_grep_slots },
 		{ "color.interactive", "<slot>", list_config_color_interactive_slots },
diff --git a/builtin/log.c b/builtin/log.c
index 093d0d26553..3240785a7ee 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -522,7 +522,8 @@ static int git_log_config(const char *var, const char *value, void *cb)
 		default_follow = git_config_bool(var, value);
 		return 0;
 	}
-	if (skip_prefix(var, "color.decorate.", &slot_name))
+	if (skip_prefix(var, "color.logDecorate.", &slot_name) ||
+	    skip_prefix(var, "color.decorate.", &slot_name))
 		return parse_decorate_color_config(var, slot_name, value);
 	if (!strcmp(var, "log.mailmap")) {
 		use_mailmap_config = git_config_bool(var, value);
diff --git a/generate-configlist.sh b/generate-configlist.sh
index 8692fe5cf4d..de418d2b8bd 100755
--- a/generate-configlist.sh
+++ b/generate-configlist.sh
@@ -7,7 +7,7 @@ print_config_list () {
 	cat <<EOF
 static const char *config_name_list[] = {
 EOF
-	grep -h '^[a-zA-Z].*\..*::$' Documentation/*config.txt Documentation/config/*.txt |
+	grep -h '^[a-zA-Z].*\..*::$' Documentation/*config.txt Documentation/config/*.txt Documentation/config/*/*.txt|
 	sed '/deprecated/d; s/::$//; s/,  */\n/g' |
 	sort |
 	sed 's/^.*$/	"&",/'
diff --git a/t/t4207-log-decoration-colors.sh b/t/t4207-log-decoration-colors.sh
index b8709424981..f125ce4fa7a 100755
--- a/t/t4207-log-decoration-colors.sh
+++ b/t/t4207-log-decoration-colors.sh
@@ -56,10 +56,16 @@ EOF
 # We want log to show all, but the second parent to refs/stash is irrelevant
 # to this test since it does not contain any decoration, hence --first-parent
 test_expect_success 'Commit Decorations Colored Correctly' '
-	git log --first-parent --abbrev=10 --all --decorate --oneline --color=always |
-	sed "s/[0-9a-f]\{10,10\}/COMMIT_ID/" |
-	test_decode_color >out &&
+	git log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw &&
+	sed "s/[0-9a-f]\{10,10\}/COMMIT_ID/" <out.raw >out.color &&
+	test_decode_color <out.color >out &&
 	test_cmp expected out
 '
 
+test_expect_success 'color.logDecorate.* is preferred to color.decorate.*' '
+	git config --rename-section color.decorate color.logDecorate &&
+	git log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw2 &&
+	test_cmp out.raw out.raw2
+'
+
 test_done

Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

On Thu, Feb 24, 2022 at 10:24 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> > Typically with sparse checkouts, we expect files outside the sparsity
> > patterns to be marked as SKIP_WORKTREE and be missing from the working
> > tree.  In edge cases, this can be violated and cause confusion, so in a
> > sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE
>
> I think this refers to af6a5187 (repo_read_index: clear
> SKIP_WORKTREE bit from files present in worktree, 2022-01-14).

Yes, I'm usually pretty good about grabbing the commits you created
and merged rather than the local copies I submitted, but I messed it
up here.  Thanks for catching.

>
> > bit from files present in worktree", 2022-01-06), Git automatically
> > clears the SKIP_WORKTREE bit at read time for entries corresponding to
> > files that are present in the working tree.
>
> So, this is a workflow where the user deliberately "creates" these
> files outside the sparsity cone or pattern (by various non-automated
> means like editing, copying/renaming, or untarring).  If they did so
> on purpose, they may be interested in comparing them with existing
> commits, or even including them as a newer version in the next
> commit they create.  To help that workflow, clearing the bit makes
> sense.
>
> Am I on the right path?  I am wondering if mentioning some of that
> would help understanding by the reader when it is contrasted with
> the (competing) goal of supporting VFS use case mentioned next.

Yes, this is one of three ways that things can get out of sync.  Since
this commit was being added to en/present-despite-skipped which
spelled this out in detail and thus would appear just a few commits
before, I thought it wasn't worth repeating these details, but I
guessed wrong.  I'll include them here again.

> > However, there is a more atypical situation where this situation would
>
> I wonder if that is "more atypical" (read: makes me wonder if it
> depends on who the reader is what is typoical), and more importantly,
> if it helps understanding of the reader (read: whether which one is
> more common, we'd want to support both camps anyway).
>
>     There is another workflow, however, that it is expected that
>     paths outside the sparsity patterns appear to exist in the
>     working tree and that they do not lose the SKIP_WORKTREE bit, at
>     least until they get modified.
>
> or something?

I like it.

> > be expected.  A Git-aware virtual file system[1] takes advantage of its
> > position as a file system driver to expose all files in the working
> > tree, fetch them on demand using partial clone on access, and tell Git
> > to pay attention to them on demand by updating the sparse checkout
> > pattern on writes.  This means that commands like "git status" only have
> > to examine files that have potentially been modified, whereas commands
> > like "ls" are able to show the entire codebase without requiring manual
> > updates to the sparse checkout pattern.
>
> Well explained.
>
> > Thus since 11d46a399d, Git with such Git-aware virtual file systems
>
> The same stale reference.
>
> > unsets the SKIP_WORKTREE bit for all files and commands like "git
> > status" have to fetch and examine them all.
> >
> > Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
> > allow limiting the tracked set of files to a small set once again.  A
> > Git-aware virtual file system or other application that wants to
> > maintain files outside of the sparse checkout can set this in a
> > repository to instruct Git not to check for the presence of
> > SKIP_WORKTREE files.  The setting defaults to false, so most users of
> > sparse checkout will still get the benefit of an automatically updating
> > index to recover from the variety of difficult issues detailed in
> > 11d46a399d for paths with SKIP_WORKTREE set despite the path being
>
> Ditto.

Will fix all three.

> > I'm guessing that since there are no code (only documentation) changes since
> > Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs.
> > I'm also guessing it matches what Stolee and Dscho stated in their comments
> > on v1.  But it'd be nice to have an ack from each side just to make sure.
>
> True.  Let me queue but leave it just outside 'next' until that
> happens.
>
> I think the name of the knob is what Jonathan suggested, so I
> presume that their side would be fine with it, but I am curious (I
> do not wonder, though) what the plan on the Microsoft's side going
> forward.  When they update the version of Git bundled in their vfsd,
> would this be reverted and an equivalent they have (and they may
> have more such "workaround" in other areas as well?) will be kept,
> so whatever we do here will add a minor inconvenience to them but
> will not hurt them otherwise?
>
> > diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
> > new file mode 100644
> > index 0000000000..fba504173c
> > --- /dev/null
> > +++ b/Documentation/config/sparse.txt
> > @@ -0,0 +1,28 @@
> > +sparse.expectFilesOutsideOfPatterns::
> > +     Typically with sparse checkouts, files not matching any
> > +     sparsity patterns are marked as such in the index file and
>
> s/index file/index/ perhaps.

Will fix.

> > +     missing from the working tree.  Accordingly, Git will
> > +     ordinarily check whether files that the index indicates are
> > +     outside of the sparse area are present in the working tree and
> > +     mark them as present in the index if so.  This option can be
>
> Just an observation.  According to this sentence, "sparse area" is
> "paths that ought to be present in the working tree", so paths
> "outside of the sparse area" that are present need to be corrected
> to be "in" the sparse area by futzing bits.  I always get confused
> when I hear "sparse area" if the author meant "paths that ought to
> be missing" or "present", but maybe it is just me.

I reworded this based on a combination of the feedback from you and
Jonathan.  I think it's clearer now; I'll resubmit soon.

>
> > +     used to tell Git that such present-but-unmatching files are
> > +     expected and to stop checking for them.
>
> OK.
>
> > ++
> > +The default is `false`.  Paths which are marked as SKIP_WORKTREE
> > +despite being present (which can occur for a few different reasons)
> > +typically present a range of problems which are difficult for users to
>
> s/typically // perhaps.

Sure.

> > +discover and recover from.  The default setting avoids such issues.
> > ++
> > +A Git-based virtual file system (VFS) can turn the usual expectation
> > +on its head: files are present in the working copy but do not take
> > +up much disk space because their contents are not downloaded until
> > +they are accessed.  With such a virtual file system layer, most files
> > +do not match the sparsity patterns at first, and the VFS layer
> > +updates the sparsity patterns to add more files whenever files are
> > +written.  Setting this to `true` supports such a setup where files are
> > +expected to be present outside the sparse area and a separate, robust
> > +mechanism is responsible for keeping the sparsity patterns up to date.
>
> s/separate, robust/separate/ I would think.
>
> We make the outside mechanism that makes these files appear to be
> present to also be responsible for maintaining the sparse bit and
> patterns.
>
> When the user (or IDE) sets this knob to 'true', do we even have to
> expect that files appear to be present?  In the use case we intend
> to support with this feature, i.e. some VFS, we might expect all
> paths to appear to be present, but if that VFS also allows users to
> configure to expose only a subset of paths, not all paths may appear
> to be present.  And we are perfectly OK with that, becuase we do not
> expect anything about the working tree paths outside the sparsity
> pattern.  Am I mistaken?
>
> So, "... supports such a setup where some external system releaves
> us of the responsibility of maintaining the consistency between the
> presence of working tree files and sparsity patterns, so we stop
> expecting whether files are present or missing outside the sparse
> area", might be closer to the truth?

Good point, and thanks for the suggested wording.

> > +Note that the checking and clearing of the SKIP_WORKTREE bit only
> > +happens when core.sparseCheckout is true, so this config option has no
> > +effect unless core.sparseCheckout is true.
>
> Good note to have.  There is no mention of "cone" mode in the entire
> description; it is unclear if this only applies to "pattern" mode or
> to both "pattern" and "cone" modes, which may want to be clarified.

Yeah, it applies to both pattern and cone modes.  I went with
Jonathan's wording, which I think sounded more precise and suggested
that only core.sparseCheckout=true matters:

"""
Regardless of this setting, Git does not check for
 present-but-unmatching files unless sparse checkout is enabled, so
 this config option has no effect unless `core.sparseCheckout` is
 `true`.
"""

Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

On Fri, Feb 25, 2022 at 8:33 AM Jonathan Nieder <jrnieder@gmail.com> wrote:
>
> Hi,
>
> Elijah Newren wrote:
>
> > Signed-off-by: Elijah Newren <newren@gmail.com>
>
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
> Thanks, and sorry for the slow review.  My one remaining area for nits
> is the documentation, but that can be improved iteratively via patches
> on top.
>
> [...]
> > --- /dev/null
> > +++ b/Documentation/config/sparse.txt
> > @@ -0,0 +1,28 @@
> > +sparse.expectFilesOutsideOfPatterns::
> > +     Typically with sparse checkouts, files not matching any
> > +     sparsity patterns are marked as such in the index file and
> > +     missing from the working tree.  Accordingly, Git will
> > +     ordinarily check whether files that the index indicates are
> > +     outside of the sparse area are present in the working tree and
>
> Junio mentioned the "sparse area" could suggest that the area is
> itself sparse and devoid of files, so it might not have been the best
> choice of words on my part.  Perhaps "whether files that the index
> indicates are not checked out are present in the working tree" would
> work here?

I rewrote the paragraph.  I think it's more clear now; I'll resubmit
it here soon.

> > +     mark them as present in the index if so.  This option can be
> > +     used to tell Git that such present-but-unmatching files are
> > +     expected and to stop checking for them.
> > ++
> > +The default is `false`.  Paths which are marked as SKIP_WORKTREE
> > +despite being present (which can occur for a few different reasons)
> > +typically present a range of problems which are difficult for users to
> > +discover and recover from.  The default setting avoids such issues.
>
> The git-sparse-checkout(1) page never describes what SKIP_WORKTREE
> means, so it might not be obvious to them what this means.  Also, the
> "can occur for a few different reasons" may leave the user wondering
> whether they are subject to those reasons.  What the reader wants to
> know is "I should keep using the default because it makes Git work
> better", so how about something like
>
>  The default is `false`, which allows Git to automatically recover
>  from the list of files in the index and working tree falling out of
>  sync.
>  +
>
> ?

I like this.

> > ++
> > +A Git-based virtual file system (VFS) can turn the usual expectation
> > +on its head: files are present in the working copy but do not take
> > +up much disk space because their contents are not downloaded until
> > +they are accessed.  With such a virtual file system layer, most files
> > +do not match the sparsity patterns at first, and the VFS layer
> > +updates the sparsity patterns to add more files whenever files are
> > +written.  Setting this to `true` supports such a setup where files are
> > +expected to be present outside the sparse area and a separate, robust
> > +mechanism is responsible for keeping the sparsity patterns up to date.
>
> Here I spent most of the words explaining what a Git-based VFS layer
> is, which is also not too relevant to most users (who are just
> interested in "is `true` the right value for me?").  How about
> reducing it to the following?
>
>  Set this to `true` if you are in a setup where extra files are expected
>  to be present and a separate, robust mechanism is responsible for
>  keeping the sparsity patterns up to date, such as a Git-aware virtual
>  file system.
>
> ?

I like this, but I also added in some of the wording suggestions from
Junio here, so it's
a bit longer but has both some of his suggested wording and yours for
slightly different aspects that I think works well together.

>
> > ++
> > +Note that the checking and clearing of the SKIP_WORKTREE bit only
> > +happens when core.sparseCheckout is true, so this config option has no
> > +effect unless core.sparseCheckout is true.
>
> Good note.  Same nit about the user not necessarily knowing what
> SKIP_WORKTREE means applies.  Also, we can remove the extra words
> "Note that" since the dutiful reader should be noting everything we
> say. :)  I think that would make
>
>  +
>  Regardless of this setting, Git does not check for
>  present-but-unmatching files unless sparse checkout is enabled, so
>  this config option has no effect unless `core.sparseCheckout` is
>  `true`.

I like this too.  Thanks for the suggestions, the proposed changes,
and the review.

[PATCH v4] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

Typically with sparse checkouts, we expect files outside the sparsity
patterns to be marked as SKIP_WORKTREE and be missing from the working
tree.  Sometimes this expectation would be violated however; including
in cases such as:
  * users grabbing files from elsewhere and writing them to the worktree
    (perhaps by editing a cached copy in an editor, copying/renaming, or
     even untarring)
  * various git commands having incomplete or no support for the
    SKIP_WORKTREE bit[1,2]
  * users attempting to "abort" a sparse-checkout operation with a
    not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
    working tree is not atomic)[3].
When the SKIP_WORKTREE bit in the index did not reflect the presence of
the file in the working tree, it traditionally caused confusion and was
difficult to detect and recover from.  So, in a sparse checkout, since
af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE
bit at index read time for entries corresponding to files that are
present in the working tree.

There is another workflow, however, where it is expected that paths
outside the sparsity patterns appear to exist in the working tree and
that they do not lose the SKIP_WORKTREE bit, at least until they get
modified.  A Git-aware virtual file system[4] takes advantage of its
position as a file system driver to expose all files in the working
tree, fetch them on demand using partial clone on access, and tell Git
to pay attention to them on demand by updating the sparse checkout
pattern on writes.  This means that commands like "git status" only have
to examine files that have potentially been modified, whereas commands
like "ls" are able to show the entire codebase without requiring manual
updates to the sparse checkout pattern.

Thus since af6a51875a, Git with such Git-aware virtual file systems
unsets the SKIP_WORKTREE bit for all files and commands like "git
status" have to fetch and examine them all.

Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
allow limiting the tracked set of files to a small set once again.  A
Git-aware virtual file system or other application that wants to
maintain files outside of the sparse checkout can set this in a
repository to instruct Git not to check for the presence of
SKIP_WORKTREE files.  The setting defaults to false, so most users of
sparse checkout will still get the benefit of an automatically updating
index to recover from the variety of difficult issues detailed in
af6a51875a for paths with SKIP_WORKTREE set despite the path being
present.

[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/
[2] The three long paragraphs in the middle of
    https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[1] such as the vfsd described in
https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Elijah Newren <newren@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
---
Changes since v3:
  * Many wording updates based on feedback and suggestions from both
    Jonathan and Junio.  Thanks for providing concrete suggestions!

Range-diff:
1:  92a13a8f08 ! 1:  5ded1532cd repo_read_index: add config to expect files outside sparse patterns
    @@ Commit message
     
         Typically with sparse checkouts, we expect files outside the sparsity
         patterns to be marked as SKIP_WORKTREE and be missing from the working
    -    tree.  In edge cases, this can be violated and cause confusion, so in a
    -    sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE
    -    bit from files present in worktree", 2022-01-06), Git automatically
    -    clears the SKIP_WORKTREE bit at read time for entries corresponding to
    -    files that are present in the working tree.
    +    tree.  Sometimes this expectation would be violated however; including
    +    in cases such as:
    +      * users grabbing files from elsewhere and writing them to the worktree
    +        (perhaps by editing a cached copy in an editor, copying/renaming, or
    +         even untarring)
    +      * various git commands having incomplete or no support for the
    +        SKIP_WORKTREE bit[1,2]
    +      * users attempting to "abort" a sparse-checkout operation with a
    +        not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
    +        working tree is not atomic)[3].
    +    When the SKIP_WORKTREE bit in the index did not reflect the presence of
    +    the file in the working tree, it traditionally caused confusion and was
    +    difficult to detect and recover from.  So, in a sparse checkout, since
    +    af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
    +    in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE
    +    bit at index read time for entries corresponding to files that are
    +    present in the working tree.
     
    -    However, there is a more atypical situation where this situation would
    -    be expected.  A Git-aware virtual file system[1] takes advantage of its
    +    There is another workflow, however, where it is expected that paths
    +    outside the sparsity patterns appear to exist in the working tree and
    +    that they do not lose the SKIP_WORKTREE bit, at least until they get
    +    modified.  A Git-aware virtual file system[4] takes advantage of its
         position as a file system driver to expose all files in the working
         tree, fetch them on demand using partial clone on access, and tell Git
         to pay attention to them on demand by updating the sparse checkout
    @@ Commit message
         like "ls" are able to show the entire codebase without requiring manual
         updates to the sparse checkout pattern.
     
    -    Thus since 11d46a399d, Git with such Git-aware virtual file systems
    +    Thus since af6a51875a, Git with such Git-aware virtual file systems
         unsets the SKIP_WORKTREE bit for all files and commands like "git
         status" have to fetch and examine them all.
     
    @@ Commit message
         SKIP_WORKTREE files.  The setting defaults to false, so most users of
         sparse checkout will still get the benefit of an automatically updating
         index to recover from the variety of difficult issues detailed in
    -    11d46a399d for paths with SKIP_WORKTREE set despite the path being
    +    af6a51875a for paths with SKIP_WORKTREE set despite the path being
         present.
     
    +    [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/
    +    [2] The three long paragraphs in the middle of
    +        https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
    +    [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
         [1] such as the vfsd described in
         https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/
     
    -    Helped-by: Jonathan Nieder <jrnieder@gmail.com>
         Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
         Signed-off-by: Elijah Newren <newren@gmail.com>
    +    Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
     
      ## Documentation/config.txt ##
     @@ Documentation/config.txt: include::config/sequencer.txt[]
    @@ Documentation/config/sparse.txt (new)
     @@
     +sparse.expectFilesOutsideOfPatterns::
     +	Typically with sparse checkouts, files not matching any
    -+	sparsity patterns are marked as such in the index file and
    -+	missing from the working tree.  Accordingly, Git will
    -+	ordinarily check whether files that the index indicates are
    -+	outside of the sparse area are present in the working tree and
    -+	mark them as present in the index if so.  This option can be
    -+	used to tell Git that such present-but-unmatching files are
    -+	expected and to stop checking for them.
    ++	sparsity patterns are marked with a SKIP_WORKTREE bit in the
    ++	index and are missing from the working tree.  Accordingly, Git
    ++	will ordinarily check whether files with the SKIP_WORKTREE bit
    ++	are in fact present in the working tree contrary to
    ++	expectations.  If Git finds any, it marks those paths as
    ++	present by clearing the relevant SKIP_WORKTREE bits.  This
    ++	option can be used to tell Git that such
    ++	present-despite-skipped files are expected and to stop
    ++	checking for them.
     ++
    -+The default is `false`.  Paths which are marked as SKIP_WORKTREE
    -+despite being present (which can occur for a few different reasons)
    -+typically present a range of problems which are difficult for users to
    -+discover and recover from.  The default setting avoids such issues.
    ++The default is `false`, which allows Git to automatically recover
    ++from the list of files in the index and working tree falling out of
    ++sync.
     ++
    -+A Git-based virtual file system (VFS) can turn the usual expectation
    -+on its head: files are present in the working copy but do not take
    -+up much disk space because their contents are not downloaded until
    -+they are accessed.  With such a virtual file system layer, most files
    -+do not match the sparsity patterns at first, and the VFS layer
    -+updates the sparsity patterns to add more files whenever files are
    -+written.  Setting this to `true` supports such a setup where files are
    -+expected to be present outside the sparse area and a separate, robust
    -+mechanism is responsible for keeping the sparsity patterns up to date.
    ++Set this to `true` if you are in a setup where some external factor
    ++relieves Git of the responsibility for maintaining the consistency
    ++between the presence of working tree files and sparsity patterns.  For
    ++example, if you have a Git-aware virtual file system that has a robust
    ++mechanism for keeping the working tree and the sparsity patterns up to
    ++date based on access patterns.
     ++
    -+Note that the checking and clearing of the SKIP_WORKTREE bit only
    -+happens when core.sparseCheckout is true, so this config option has no
    -+effect unless core.sparseCheckout is true.
    ++Regardless of this setting, Git does not check for
    ++present-despite-skipped files unless sparse checkout is enabled, so
    ++this config option has no effect unless `core.sparseCheckout` is
    ++`true`.
     
      ## cache.h ##
     @@ cache.h: extern const char *core_fsmonitor;

 Documentation/config.txt         |  2 ++
 Documentation/config/sparse.txt  | 27 +++++++++++++++++++++++++++
 cache.h                          |  1 +
 config.c                         | 14 ++++++++++++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/sparse.txt

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b168f02dc3..8628ae2634 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -468,6 +468,8 @@ include::config/sequencer.txt[]
 
 include::config/showbranch.txt[]
 
+include::config/sparse.txt[]
+
 include::config/splitindex.txt[]
 
 include::config/ssh.txt[]
diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
new file mode 100644
index 0000000000..aff49a8d3a
--- /dev/null
+++ b/Documentation/config/sparse.txt
@@ -0,0 +1,27 @@
+sparse.expectFilesOutsideOfPatterns::
+	Typically with sparse checkouts, files not matching any
+	sparsity patterns are marked with a SKIP_WORKTREE bit in the
+	index and are missing from the working tree.  Accordingly, Git
+	will ordinarily check whether files with the SKIP_WORKTREE bit
+	are in fact present in the working tree contrary to
+	expectations.  If Git finds any, it marks those paths as
+	present by clearing the relevant SKIP_WORKTREE bits.  This
+	option can be used to tell Git that such
+	present-despite-skipped files are expected and to stop
+	checking for them.
++
+The default is `false`, which allows Git to automatically recover
+from the list of files in the index and working tree falling out of
+sync.
++
+Set this to `true` if you are in a setup where some external factor
+relieves Git of the responsibility for maintaining the consistency
+between the presence of working tree files and sparsity patterns.  For
+example, if you have a Git-aware virtual file system that has a robust
+mechanism for keeping the working tree and the sparsity patterns up to
+date based on access patterns.
++
+Regardless of this setting, Git does not check for
+present-despite-skipped files unless sparse checkout is enabled, so
+this config option has no effect unless `core.sparseCheckout` is
+`true`.
diff --git a/cache.h b/cache.h
index 281f00ab1b..b6b8e83ae3 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int sparse_expect_files_outside_of_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a0..9b9ad1500a 100644
--- a/config.c
+++ b/config.c
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 	return platform_core_config(var, value, cb);
 }
 
+static int git_default_sparse_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
+		sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
+	/* Add other config variables here and to Documentation/config/sparse.txt. */
+	return 0;
+}
+
 static int git_default_i18n_config(const char *var, const char *value)
 {
 	if (!strcmp(var, "i18n.commitencoding"))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (starts_with(var, "sparse."))
+		return git_default_sparse_config(var, value);
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index fd0501e77a..fb55bf6129 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int sparse_expect_files_outside_of_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f..daeb5112a1 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    sparse_expect_files_outside_of_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb490187..d1833c0f31 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
 	test "$(cat b)" = "modified"
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config sparse.expectFilesOutsideOfPatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
 	test_create_repo server &&
 	git clone "file://$(pwd)/server" client &&
-- 
2.35.1.278.g4f05fa7020.dirty

[PATCH v5] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

Typically with sparse checkouts, we expect files outside the sparsity
patterns to be marked as SKIP_WORKTREE and be missing from the working
tree.  Sometimes this expectation would be violated however; including
in cases such as:
  * users grabbing files from elsewhere and writing them to the worktree
    (perhaps by editing a cached copy in an editor, copying/renaming, or
     even untarring)
  * various git commands having incomplete or no support for the
    SKIP_WORKTREE bit[1,2]
  * users attempting to "abort" a sparse-checkout operation with a
    not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
    working tree is not atomic)[3].
When the SKIP_WORKTREE bit in the index did not reflect the presence of
the file in the working tree, it traditionally caused confusion and was
difficult to detect and recover from.  So, in a sparse checkout, since
af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE
bit at index read time for entries corresponding to files that are
present in the working tree.

There is another workflow, however, where it is expected that paths
outside the sparsity patterns appear to exist in the working tree and
that they do not lose the SKIP_WORKTREE bit, at least until they get
modified.  A Git-aware virtual file system[4] takes advantage of its
position as a file system driver to expose all files in the working
tree, fetch them on demand using partial clone on access, and tell Git
to pay attention to them on demand by updating the sparse checkout
pattern on writes.  This means that commands like "git status" only have
to examine files that have potentially been modified, whereas commands
like "ls" are able to show the entire codebase without requiring manual
updates to the sparse checkout pattern.

Thus since af6a51875a, Git with such Git-aware virtual file systems
unsets the SKIP_WORKTREE bit for all files and commands like "git
status" have to fetch and examine them all.

Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
allow limiting the tracked set of files to a small set once again.  A
Git-aware virtual file system or other application that wants to
maintain files outside of the sparse checkout can set this in a
repository to instruct Git not to check for the presence of
SKIP_WORKTREE files.  The setting defaults to false, so most users of
sparse checkout will still get the benefit of an automatically updating
index to recover from the variety of difficult issues detailed in
af6a51875a for paths with SKIP_WORKTREE set despite the path being
present.

[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/
[2] The three long paragraphs in the middle of
    https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[4] such as the vfsd described in
https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Elijah Newren <newren@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
---
Changes since v4:
  - Corrected an incorrect footnote

Range-diff:
1:  5ded1532cd ! 1:  d348cc766f repo_read_index: add config to expect files outside sparse patterns
    @@ Commit message
         [2] The three long paragraphs in the middle of
             https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
         [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
    -    [1] such as the vfsd described in
    +    [4] such as the vfsd described in
         https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/
     
         Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

 Documentation/config.txt         |  2 ++
 Documentation/config/sparse.txt  | 27 +++++++++++++++++++++++++++
 cache.h                          |  1 +
 config.c                         | 14 ++++++++++++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 7 files changed, 66 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/sparse.txt

diff --git a/Documentation/config.txt b/Documentation/config.txt
index b168f02dc3..8628ae2634 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -468,6 +468,8 @@ include::config/sequencer.txt[]
 
 include::config/showbranch.txt[]
 
+include::config/sparse.txt[]
+
 include::config/splitindex.txt[]
 
 include::config/ssh.txt[]
diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
new file mode 100644
index 0000000000..aff49a8d3a
--- /dev/null
+++ b/Documentation/config/sparse.txt
@@ -0,0 +1,27 @@
+sparse.expectFilesOutsideOfPatterns::
+	Typically with sparse checkouts, files not matching any
+	sparsity patterns are marked with a SKIP_WORKTREE bit in the
+	index and are missing from the working tree.  Accordingly, Git
+	will ordinarily check whether files with the SKIP_WORKTREE bit
+	are in fact present in the working tree contrary to
+	expectations.  If Git finds any, it marks those paths as
+	present by clearing the relevant SKIP_WORKTREE bits.  This
+	option can be used to tell Git that such
+	present-despite-skipped files are expected and to stop
+	checking for them.
++
+The default is `false`, which allows Git to automatically recover
+from the list of files in the index and working tree falling out of
+sync.
++
+Set this to `true` if you are in a setup where some external factor
+relieves Git of the responsibility for maintaining the consistency
+between the presence of working tree files and sparsity patterns.  For
+example, if you have a Git-aware virtual file system that has a robust
+mechanism for keeping the working tree and the sparsity patterns up to
+date based on access patterns.
++
+Regardless of this setting, Git does not check for
+present-despite-skipped files unless sparse checkout is enabled, so
+this config option has no effect unless `core.sparseCheckout` is
+`true`.
diff --git a/cache.h b/cache.h
index 281f00ab1b..b6b8e83ae3 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int sparse_expect_files_outside_of_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a0..9b9ad1500a 100644
--- a/config.c
+++ b/config.c
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 	return platform_core_config(var, value, cb);
 }
 
+static int git_default_sparse_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
+		sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
+	/* Add other config variables here and to Documentation/config/sparse.txt. */
+	return 0;
+}
+
 static int git_default_i18n_config(const char *var, const char *value)
 {
 	if (!strcmp(var, "i18n.commitencoding"))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (starts_with(var, "sparse."))
+		return git_default_sparse_config(var, value);
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index fd0501e77a..fb55bf6129 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int sparse_expect_files_outside_of_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f..daeb5112a1 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    sparse_expect_files_outside_of_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb490187..d1833c0f31 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
 	test "$(cat b)" = "modified"
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config sparse.expectFilesOutsideOfPatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
 	test_create_repo server &&
 	git clone "file://$(pwd)/server" client &&
-- 
2.35.1.355.g49d31b1b5a.dirty

Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Typically with sparse checkouts, we expect files outside the sparsity
> patterns to be marked as SKIP_WORKTREE and be missing from the working
> tree.  Sometimes this expectation would be violated however; including
> in cases such as:
>   * users grabbing files from elsewhere and writing them to the worktree
>     (perhaps by editing a cached copy in an editor, copying/renaming, or
>      even untarring)
>   * various git commands having incomplete or no support for the
>     SKIP_WORKTREE bit[1,2]
>   * users attempting to "abort" a sparse-checkout operation with a
>     not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
>     working tree is not atomic)[3].
> When the SKIP_WORKTREE bit in the index did not reflect the presence of
> the file in the working tree, it traditionally caused confusion and was
> difficult to detect and recover from.  So, in a sparse checkout, since
> af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
> in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE

The reference is a bit off here.  Here is what I get locally:

    af6a51875a (repo_read_index: clear SKIP_WORKTREE bit from files
    present in worktree, 2022-01-14)

and that is in the version I have locally in 'next'.

> [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/
> [2] The three long paragraphs in the middle of
>     https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
> [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
> [4] such as the vfsd described in

Here is another difference from the version I have locally in
'next', which I didn't notice that this [4] was misspelt as [1]
before applying.

Everything else seems the same, so let's not bother reverting the
old one out of 'next' and merging this version after fixing this
version up.  What we have is good enough modulo [4] vs [1].

Thanks.

Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns

[Elijah Newren]

On Tue, Mar 1, 2022 at 11:37 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> > Typically with sparse checkouts, we expect files outside the sparsity
> > patterns to be marked as SKIP_WORKTREE and be missing from the working
> > tree.  Sometimes this expectation would be violated however; including
> > in cases such as:
> >   * users grabbing files from elsewhere and writing them to the worktree
> >     (perhaps by editing a cached copy in an editor, copying/renaming, or
> >      even untarring)
> >   * various git commands having incomplete or no support for the
> >     SKIP_WORKTREE bit[1,2]
> >   * users attempting to "abort" a sparse-checkout operation with a
> >     not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
> >     working tree is not atomic)[3].
> > When the SKIP_WORKTREE bit in the index did not reflect the presence of
> > the file in the working tree, it traditionally caused confusion and was
> > difficult to detect and recover from.  So, in a sparse checkout, since
> > af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
> > in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE
>
> The reference is a bit off here.  Here is what I get locally:
>
>     af6a51875a (repo_read_index: clear SKIP_WORKTREE bit from files
>     present in worktree, 2022-01-14)
>
> and that is in the version I have locally in 'next'.

Ugh, forgot to update the date when I updated the reference when you
pointed that out.

> > [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/
> > [2] The three long paragraphs in the middle of
> >     https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
> > [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
> > [4] such as the vfsd described in
>
> Here is another difference from the version I have locally in
> 'next', which I didn't notice that this [4] was misspelt as [1]
> before applying.

Sorry, I hadn't noticed you merging to next, and I saw in the irc logs
the discussion about this 1 vs. 4 between you and jrnieder so I
thought I'd fix it.

> Everything else seems the same, so let's not bother reverting the
> old one out of 'next' and merging this version after fixing this
> version up.  What we have is good enough modulo [4] vs [1].

Sounds good.

Re: [PATCH v4] repo_read_index: add config to expect files outside sparse patterns

[Derrick Stolee]

On 2/26/2022 1:12 AM, Elijah Newren wrote:
> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to
> allow limiting the tracked set of files to a small set once again.  A
> Git-aware virtual file system or other application that wants to
> maintain files outside of the sparse checkout can set this in a
> repository to instruct Git not to check for the presence of
> SKIP_WORKTREE files.  The setting defaults to false, so most users of
> sparse checkout will still get the benefit of an automatically updating
> index to recover from the variety of difficult issues detailed in
> af6a51875a for paths with SKIP_WORKTREE set despite the path being
> present.

Sorry that I lost track of this thread recently. I re-read this
version that made it to 'next' already and have no issues.

This approach will work well for our needs in the microsoft/git
fork. Likely, we'll enable the sparse_expect_files_outside_of_patterns
global when we initialize our core_virtualfilesystem global.
Then, we'll probably replace some of the uses of core_virtualfilesytem
with this new global in some of the conditions that we inserted to
work with previous restrictions in this area. Then, those changes will
be fit for submitting here. If anyone needs those changes sooner, then
do not hesitate to make similar changes. We will adapt in our fork to
whatever the Git community thinks is best.

Thanks,
-Stolee