[PATCH] experimental: default to fetch.writeCommitGraph=false

[PATCH] experimental: default to fetch.writeCommitGraph=false

[Jonathan Nieder]

The fetch.writeCommitGraph feature makes fetches write out a commit
graph file for the newly downloaded pack on fetch.  This improves the
performance of various commands that would perform a revision walk and
eventually ought to be the default for everyone.  To prepare for that
future, it's enabled by default for users that set
feature.experimental=true to experience such future defaults.

Alas, for --unshallow fetches from a shallow clone it runs into a
snag: by the time Git has fetched the new objects and is writing a
commit graph, it has performed a revision walk and r->parsed_objects
contains information about the shallow boundary from *before* the
fetch.  The commit graph writing code is careful to avoid writing a
commit graph file in shallow repositories, but the new state is not
shallow, and the result is that from that point on, commands like "git
log" make use of a newly written commit graph file representing a
fictional history with the old shallow boundary.

We could fix this by making the commit graph writing code more careful
to avoid writing a commit graph that could have used any grafts or
shallow state, but it is possible that there are other pieces of
mutated state that fetch's commit graph writing code may be relying
on.  So disable it in the feature.experimental configuration.

Google developers have been running in this configuration (by setting
fetch.writeCommitGraph=false in the system config) to work around this
bug since it was discovered in April.  Once the fix lands, we'll
enable fetch.writeCommitGraph=true again to give it some early testing
before rolling out to a wider audience.

In other words:

- this patch only affects behavior with feature.experimental=true

- it makes feature.experimental match the configuration Google has
  been using for the last few months, meaning it would leave users in
  a better tested state than without it

- this should improve testing for other features guarded by
  feature.experimental, by making feature.experimental safer to use

Reported-by: Jay Conrod <jayconrod@google.com>
Helped-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
I realize this is late to send.  That said, as described above, I
think it's a good way to buy time by minimizing user exposure to
fetch.writeCommitGraph=true until a fix for it is well cooked.

In other words, I'd like to see this patch in Git 2.28-rc0.

Thanks of all kinds welcome, as always.  Previous discussion:
https://lore.kernel.org/git/20200603034213.GB253041@google.com/

 Documentation/config/feature.txt | 8 --------
 Documentation/config/fetch.txt   | 3 +--
 repo-settings.c                  | 8 ++++----
 3 files changed, 5 insertions(+), 14 deletions(-)

diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt
index 28c33602d52..c0cbf2bb1cd 100644
--- a/Documentation/config/feature.txt
+++ b/Documentation/config/feature.txt
@@ -15,14 +15,6 @@ feature.experimental::
 * `fetch.negotiationAlgorithm=skipping` may improve fetch negotiation times by
 skipping more commits at a time, reducing the number of round trips.
 +
-* `fetch.writeCommitGraph=true` writes a commit-graph after every `git fetch`
-command that downloads a pack-file from a remote. Using the `--split` option,
-most executions will create a very small commit-graph file on top of the
-existing commit-graph file(s). Occasionally, these files will merge and the
-write may take longer. Having an updated commit-graph file helps performance
-of many Git commands, including `git merge-base`, `git push -f`, and
-`git log --graph`.
-+
 * `protocol.version=2` speeds up fetches from repositories with many refs by
 allowing the client to specify which refs to list before the server lists
 them.
diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt
index b1a9b1461d3..b20394038d1 100644
--- a/Documentation/config/fetch.txt
+++ b/Documentation/config/fetch.txt
@@ -90,5 +90,4 @@ fetch.writeCommitGraph::
 	the existing commit-graph file(s). Occasionally, these files will
 	merge and the write may take longer. Having an updated commit-graph
 	file helps performance of many Git commands, including `git merge-base`,
-	`git push -f`, and `git log --graph`. Defaults to false, unless
-	`feature.experimental` is true.
+	`git push -f`, and `git log --graph`. Defaults to false.
diff --git a/repo-settings.c b/repo-settings.c
index dc6817daa95..0918408b344 100644
--- a/repo-settings.c
+++ b/repo-settings.c
@@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
 		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
 		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
 	}
+
 	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
 		r->settings.fetch_write_commit_graph = value;
-	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
-	}
 	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
 
+	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
+		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
+
 	/* Hack for test programs like test-dump-untracked-cache */
 	if (ignore_untracked_cache_config)
 		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;
-- 
2.27.0.383.g050319c2ae

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Derrick Stolee]

On 7/7/2020 2:20 AM, Jonathan Nieder wrote:
> The fetch.writeCommitGraph feature makes fetches write out a commit
> graph file for the newly downloaded pack on fetch.  This improves the
> performance of various commands that would perform a revision walk and
> eventually ought to be the default for everyone.  To prepare for that
> future, it's enabled by default for users that set
> feature.experimental=true to experience such future defaults.
> 
> Alas, for --unshallow fetches from a shallow clone it runs into a
> snag: by the time Git has fetched the new objects and is writing a
> commit graph, it has performed a revision walk and r->parsed_objects
> contains information about the shallow boundary from *before* the
> fetch.  The commit graph writing code is careful to avoid writing a
> commit graph file in shallow repositories, but the new state is not
> shallow, and the result is that from that point on, commands like "git
> log" make use of a newly written commit graph file representing a
> fictional history with the old shallow boundary.
> 
> We could fix this by making the commit graph writing code more careful
> to avoid writing a commit graph that could have used any grafts or
> shallow state, but it is possible that there are other pieces of
> mutated state that fetch's commit graph writing code may be relying
> on.  So disable it in the feature.experimental configuration.
> 
> Google developers have been running in this configuration (by setting
> fetch.writeCommitGraph=false in the system config) to work around this
> bug since it was discovered in April.  Once the fix lands, we'll
> enable fetch.writeCommitGraph=true again to give it some early testing
> before rolling out to a wider audience.
> 
> In other words:
> 
> - this patch only affects behavior with feature.experimental=true
> 
> - it makes feature.experimental match the configuration Google has
>   been using for the last few months, meaning it would leave users in
>   a better tested state than without it
> 
> - this should improve testing for other features guarded by
>   feature.experimental, by making feature.experimental safer to use
> 
> Reported-by: Jay Conrod <jayconrod@google.com>
> Helped-by: Taylor Blau <me@ttaylorr.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
> I realize this is late to send.  That said, as described above, I
> think it's a good way to buy time by minimizing user exposure to
> fetch.writeCommitGraph=true until a fix for it is well cooked.

While it would certainly be better to fix the commit_graph_compatible()
method, I understand that that is a more complicated problem.

> In other words, I'd like to see this patch in Git 2.28-rc0.

This patch is 100% correct.

Normally, I would say "this is experimental, and a user can always
disable fetch.writeCommitGraph manually if they are running into this."
This is especially true because unshallowing a repo is (probably)
a rare operation. At least, I expect that very few users actually do
it, and those who do are expert users.

But, it is best to reduce user pain, even in rare cases like this.

Further, I hope to submit new maintenance tasks soon which can
replace fetch.writeCommitGraph.

Thanks,
-Stolee

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> The fetch.writeCommitGraph feature makes fetches write out a commit

> In other words:
>
> - this patch only affects behavior with feature.experimental=true
>
> - it makes feature.experimental match the configuration Google has
>   been using for the last few months, meaning it would leave users in
>   a better tested state than without it
>
> - this should improve testing for other features guarded by
>   feature.experimental, by making feature.experimental safer to use


In other words, fetch.writeCommitGraph in its current form is too
broken to be recommended even for brave souls with "experimental"
bit on.

I wonder if we perhaps wnat to add to the documentation for
writeCommitGraph configuration that its use is currently not
recommended in a shallow clone or something (I know it is not
a problem just to use it with shallow but the breakage needs
to involve unshallowing, but by definition those who do not
use shallow would not hit the unshallowing bug, so...).

> Reported-by: Jay Conrod <jayconrod@google.com>
> Helped-by: Taylor Blau <me@ttaylorr.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
> I realize this is late to send.  That said, as described above, I
> think it's a good way to buy time by minimizing user exposure to
> fetch.writeCommitGraph=true until a fix for it is well cooked.
>
> In other words, I'd like to see this patch in Git 2.28-rc0.

Yes, I do, too.

Thanks.

> diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt
> index b1a9b1461d3..b20394038d1 100644
> --- a/Documentation/config/fetch.txt
> +++ b/Documentation/config/fetch.txt
> @@ -90,5 +90,4 @@ fetch.writeCommitGraph::
>  	the existing commit-graph file(s). Occasionally, these files will
>  	merge and the write may take longer. Having an updated commit-graph
>  	file helps performance of many Git commands, including `git merge-base`,
> -	`git push -f`, and `git log --graph`. Defaults to false, unless
> -	`feature.experimental` is true.
> +	`git push -f`, and `git log --graph`. Defaults to false.
> diff --git a/repo-settings.c b/repo-settings.c
> index dc6817daa95..0918408b344 100644
> --- a/repo-settings.c
> +++ b/repo-settings.c
> @@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
>  		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
>  		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
>  	}
> +
>  	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
>  		r->settings.fetch_write_commit_graph = value;
> -	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
> -		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
> -		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
> -	}
>  	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
>  
> +	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
> +		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
> +
>  	/* Hack for test programs like test-dump-untracked-cache */
>  	if (ignore_untracked_cache_config)
>  		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Taylor Blau]

Hi Junio,

On Tue, Jul 07, 2020 at 08:09:36AM -0700, Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
> > The fetch.writeCommitGraph feature makes fetches write out a commit
>
> > In other words:
> >
> > - this patch only affects behavior with feature.experimental=true
> >
> > - it makes feature.experimental match the configuration Google has
> >   been using for the last few months, meaning it would leave users in
> >   a better tested state than without it
> >
> > - this should improve testing for other features guarded by
> >   feature.experimental, by making feature.experimental safer to use
>
>
> In other words, fetch.writeCommitGraph in its current form is too
> broken to be recommended even for brave souls with "experimental"
> bit on.
>
> I wonder if we perhaps wnat to add to the documentation for
> writeCommitGraph configuration that its use is currently not
> recommended in a shallow clone or something (I know it is not
> a problem just to use it with shallow but the breakage needs
> to involve unshallowing, but by definition those who do not
> use shallow would not hit the unshallowing bug, so...).

I think this is a good direction if you don't want to take the patch I
sent in [1] for v2.28.0. If you do, though, I don't think that this
would be necessary.

> > Reported-by: Jay Conrod <jayconrod@google.com>
> > Helped-by: Taylor Blau <me@ttaylorr.com>
> > Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> > ---
> > I realize this is late to send.  That said, as described above, I
> > think it's a good way to buy time by minimizing user exposure to
> > fetch.writeCommitGraph=true until a fix for it is well cooked.
> >
> > In other words, I'd like to see this patch in Git 2.28-rc0.
>
> Yes, I do, too.
>
> Thanks.
>
> > diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt
> > index b1a9b1461d3..b20394038d1 100644
> > --- a/Documentation/config/fetch.txt
> > +++ b/Documentation/config/fetch.txt
> > @@ -90,5 +90,4 @@ fetch.writeCommitGraph::
> >  	the existing commit-graph file(s). Occasionally, these files will
> >  	merge and the write may take longer. Having an updated commit-graph
> >  	file helps performance of many Git commands, including `git merge-base`,
> > -	`git push -f`, and `git log --graph`. Defaults to false, unless
> > -	`feature.experimental` is true.
> > +	`git push -f`, and `git log --graph`. Defaults to false.
> > diff --git a/repo-settings.c b/repo-settings.c
> > index dc6817daa95..0918408b344 100644
> > --- a/repo-settings.c
> > +++ b/repo-settings.c
> > @@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
> >  		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
> >  		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
> >  	}
> > +
> >  	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
> >  		r->settings.fetch_write_commit_graph = value;
> > -	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
> > -		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
> > -		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
> > -	}
> >  	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
> >
> > +	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
> > +		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
> > +
> >  	/* Hack for test programs like test-dump-untracked-cache */
> >  	if (ignore_untracked_cache_config)
> >  		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;

Thanks,
Taylor

[1]: https://lore.kernel.org/git/20200707144338.GA26342@syl.lan/T/#t

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

>> I wonder if we perhaps wnat to add to the documentation for
>> writeCommitGraph configuration that its use is currently not
>> recommended in a shallow clone or something (I know it is not
>> a problem just to use it with shallow but the breakage needs
>> to involve unshallowing, but by definition those who do not
>> use shallow would not hit the unshallowing bug, so...).
>
> I think this is a good direction if you don't want to take the patch I
> sent in [1] for v2.28.0. If you do, though, I don't think that this
> would be necessary.

Good timing.  I didn't know a "fix" was already being worked on ([1]
is the patch from this morning, right?  I haven't seen it except for
its subject).

We could obviously do both excluding it from the usual experimental
set and applying your fix, so that those who are really curious can
help us make sure your fix would be all that is needed.  Let's see
what Jonathan says...

Thanks.

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Taylor Blau]

On Tue, Jul 07, 2020 at 09:50:00AM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> >> I wonder if we perhaps wnat to add to the documentation for
> >> writeCommitGraph configuration that its use is currently not
> >> recommended in a shallow clone or something (I know it is not
> >> a problem just to use it with shallow but the breakage needs
> >> to involve unshallowing, but by definition those who do not
> >> use shallow would not hit the unshallowing bug, so...).
> >
> > I think this is a good direction if you don't want to take the patch I
> > sent in [1] for v2.28.0. If you do, though, I don't think that this
> > would be necessary.
>
> Good timing.  I didn't know a "fix" was already being worked on ([1]
> is the patch from this morning, right?  I haven't seen it except for
> its subject).

[1] is the fix. Jonathan wrote it a month or so ago, I just added a test
on top. (Independently, I tested it with the reproduction in the
original bug report, and it worked properly).

> We could obviously do both excluding it from the usual experimental
> set and applying your fix, so that those who are really curious can
> help us make sure your fix would be all that is needed.  Let's see
> what Jonathan says...

Either of those sound good to me.

> Thanks.

Thanks,
Taylor

Re: [PATCH] experimental: default to fetch.writeCommitGraph=false

[Jonathan Nieder]

Junio C Hamano wrote:

> We could obviously do both excluding it from the usual experimental
> set and applying your fix, so that those who are really curious can
> help us make sure your fix would be all that is needed.  Let's see
> what Jonathan says...

Yes, that would be my preference.

That is, both:

* applying the fix to provide a good experience to users interested in
  fetch.writeCommitGraph

* disabling fetch.writeCommitGraph in the experimental set, since it
  has not had much production exposure yet.  The experimental set is
  relatively young, so I want to ensure people's initial experiences
  with it are positive so that they stick with it if they're
  interested in experimental features (or in other words, I think
  there's still a place for features that are not yet proven enough
  to go in the experimental set).

Regardless of what is put in the experimental set, at $DAYJOB we will
run with the fix applied and with fetch.writeCommitGraph=true starting
next week.  I'd encourage anyone else with a similarly controlled
setup to try the same.

Thanks,
Jonathan