[PATCH] Documentation: mark `--object-format=sha256` as experimental

[PATCH] Documentation: mark `--object-format=sha256` as experimental

[Martin Ågren]

After eff45daab8 ("repository: enable SHA-256 support by default",
2020-07-29), vanilla builds of Git enable the user to run, e.g.,

  git init --object-format=sha256

and hack away. This can be a good way to gain experience with the
SHA-256 world, e.g., to find bugs that

  GIT_TEST_DEFAULT_HASH=sha256 make test

doesn't spot.

But it really is a separate world: Such SHA-256 repos will live entirely
separate from the (by now fairly large) set of SHA-1 repos. Interacting
across the border is possible in principle, e.g., through "diff + apply"
(or "format-patch + am"), but even that has its limitations: Applying a
SHA-256 diff in a SHA-1 repo works in the simple case, but if you need
to resort to `-3`, you're out of luck.

Similarly, "push + pull" should work, but you really will be operating
mostly offset from the rest of the world. That might be ok by the time
you initialize your repository, and it might be ok for several months
after that, but there might come a day when you're starting to regret
your use of `git init --object-format=sha256` and have dug yourself into
a fairly deep hole.

Workflows aside, let's consider a more technical aspect. Pack index
files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition
document foresees a v3, which we do not yet support (and for all we
know, the final v3 might end up different from the one sketched in the
hash transition document).

When the test suite is run with SHA-1 as the default hash algo, it
creates and consumes v2 pack files. But with SHA-256, we use an
undocumented, hybrid format where the header looks like v2, but where
the payload is not only "not correct SHA1", but where even the data
sizes are different. The trailing checksum is different, meaning no-one
(except us!) should/would try to interpret this file as a proper v2 pack
index.

We could certainly (re)define v2 to match our SHA-256 behavior, but we
do foresee v3 for a reason. And that would still just fix this specific
issue. And even when everything around SHA-256 is well-defined and we
have SHA-1--SHA-256 interoperability, there's a risk, at least
initially, that somewhere we'd be permeating buggy data that we'd then
feel responsible for and need to be able to handle for a long time to
come.

In short: we need some time and leeway.

Wherever `--object-format` is mentioned in our documentation, let's make
it clear that using it with "sha256" is experimental. If we later need
to explain why we can't handle data we generated back in 2020, we can
always point to this paragraph we're adding here.

By "include::"-ing a small blurb, we should be able to be consistent
throughout the documentation and can eventually gradually tone down the
severity of this text. One day, we might even use it to start phasing
out `--object-format=sha1`, but let's not get ahead of ourselves...

There's also `extensions.objectFormat`, but it's only mentioned three
times. Twice where we're adding this new disclaimer and in the third
spot we already have a "do not edit" warning. From there, interested
readers should eventually find this new one that we're adding here.

Because `GIT_DEFAULT_HASH` provides another entry point to this
functionality, document the experimental nature of it too.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/git-index-pack.txt           | 2 ++
 Documentation/git-init.txt                 | 2 ++
 Documentation/git-show-index.txt           | 2 ++
 Documentation/git.txt                      | 3 ++-
 Documentation/object-format-disclaimer.txt | 6 ++++++
 5 files changed, 14 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/object-format-disclaimer.txt

diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt
index 9316d9a80b..139285dd0f 100644
--- a/Documentation/git-index-pack.txt
+++ b/Documentation/git-index-pack.txt
@@ -100,6 +100,8 @@ OPTIONS
 	value is set or outside a repository.
 +
 This option cannot be used with --stdin.
++
+include::object-format-disclaimer.txt[]
 
 NOTES
 -----
diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt
index ddfe265da5..f35f70f13d 100644
--- a/Documentation/git-init.txt
+++ b/Documentation/git-init.txt
@@ -53,6 +53,8 @@ current working directory.
 
 Specify the given object format (hash algorithm) for the repository.  The valid
 values are 'sha1' and (if enabled) 'sha256'.  'sha1' is the default.
++
+include::object-format-disclaimer.txt[]
 
 --template=<template_directory>::
 
diff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt
index 39b1d8eaa1..e49318a5a0 100644
--- a/Documentation/git-show-index.txt
+++ b/Documentation/git-show-index.txt
@@ -44,6 +44,8 @@ OPTIONS
 	valid values are 'sha1' and (if enabled) 'sha256'.  The default is the
 	algorithm for the current repository (set by `extensions.objectFormat`), or
 	'sha1' if no value is set or outside a repository..
++
+include::object-format-disclaimer.txt[]
 
 GIT
 ---
diff --git a/Documentation/git.txt b/Documentation/git.txt
index 3e50065198..e0a2690ba2 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -497,7 +497,8 @@ double-quotes and respecting backslash escapes. E.g., the value
 	If this variable is set, the default hash algorithm for new
 	repositories will be set to this value. This value is currently
 	ignored when cloning; the setting of the remote repository
-	is used instead. The default is "sha1".
+	is used instead. The default is "sha1". THIS VARIABLE IS
+	EXPERIMENTAL! See `--object-format` in linkgit:git-init[1].
 
 Git Commits
 ~~~~~~~~~~~
diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt
new file mode 100644
index 0000000000..4cb106f0d1
--- /dev/null
+++ b/Documentation/object-format-disclaimer.txt
@@ -0,0 +1,6 @@
+THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still
+in an early stage.  A SHA-256 repository will in general not be able to
+share work with "regular" SHA-1 repositories.  It should be assumed
+that, e.g., Git internal file formats in relation to SHA-256
+repositories may change in backwards-incompatible ways.  Only use
+`--object-format=sha256` for testing purposes.
-- 
2.28.0.199.g4234a9100e

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5077 bytes --]

On 2020-08-06 at 20:23:58, Martin Ågren wrote:
> After eff45daab8 ("repository: enable SHA-256 support by default",
> 2020-07-29), vanilla builds of Git enable the user to run, e.g.,
> 
>   git init --object-format=sha256
> 
> and hack away. This can be a good way to gain experience with the
> SHA-256 world, e.g., to find bugs that
> 
>   GIT_TEST_DEFAULT_HASH=sha256 make test
> 
> doesn't spot.
> 
> But it really is a separate world: Such SHA-256 repos will live entirely
> separate from the (by now fairly large) set of SHA-1 repos. Interacting
> across the border is possible in principle, e.g., through "diff + apply"
> (or "format-patch + am"), but even that has its limitations: Applying a
> SHA-256 diff in a SHA-1 repo works in the simple case, but if you need
> to resort to `-3`, you're out of luck.
> 
> Similarly, "push + pull" should work, but you really will be operating
> mostly offset from the rest of the world. That might be ok by the time
> you initialize your repository, and it might be ok for several months
> after that, but there might come a day when you're starting to regret
> your use of `git init --object-format=sha256` and have dug yourself into
> a fairly deep hole.

I do agree that they don't interoperate right now, and that we'd like it
to in the future.  But there are definitely people who can use SHA-256
support for new projects without problems.  I'm aware of certain
government agencies who very much do not want to use SHA-1 at all (and
at some point will be legally prohibited from doing so), and they will
be completely fine with the status quo.  Some of those same
organizations are unhappy about prohibited algorithms even being linked
into the binaries they use.  These folks can use a suitably new version
of Git everywhere and not care about the lack of backwards
compatibility.

I am, of course, in favor of abandoning SHA-1 as fast as practically
possible, but I understand that backwards compatibility is obviously a
concern.

> Workflows aside, let's consider a more technical aspect. Pack index
> files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition
> document foresees a v3, which we do not yet support (and for all we
> know, the final v3 might end up different from the one sketched in the
> hash transition document).
> 
> When the test suite is run with SHA-1 as the default hash algo, it
> creates and consumes v2 pack files. But with SHA-256, we use an
> undocumented, hybrid format where the header looks like v2, but where
> the payload is not only "not correct SHA1", but where even the data
> sizes are different. The trailing checksum is different, meaning no-one
> (except us!) should/would try to interpret this file as a proper v2 pack
> index.
>
> We could certainly (re)define v2 to match our SHA-256 behavior, but we
> do foresee v3 for a reason. And that would still just fix this specific
> issue. And even when everything around SHA-256 is well-defined and we
> have SHA-1--SHA-256 interoperability, there's a risk, at least
> initially, that somewhere we'd be permeating buggy data that we'd then
> feel responsible for and need to be able to handle for a long time to
> come.

These are valid index v1 and v2 files, just with a different hash
algorithm.  v3 is there for the point where we do interoperate and need
to store hash values of multiple algorithms at once.  There's little to
no benefit to v3 if you don't need multiple algorithm support, other
than the fact that they declare the algorithms in them.

This is no different than saying that our commit or tree objects are in
a different form; they are syntactically identical, just with a
different hash algorithm.  That's how everything is in the .git
directory.

> diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt
> new file mode 100644
> index 0000000000..4cb106f0d1
> --- /dev/null
> +++ b/Documentation/object-format-disclaimer.txt
> @@ -0,0 +1,6 @@
> +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still
> +in an early stage.  A SHA-256 repository will in general not be able to
> +share work with "regular" SHA-1 repositories.  It should be assumed
> +that, e.g., Git internal file formats in relation to SHA-256
> +repositories may change in backwards-incompatible ways.  Only use
> +`--object-format=sha256` for testing purposes.

I'm fine with marking the functionality experimental for a few releases,
since it is possible we have bugs people haven't found, and adding a
note about interoperability after that point, since I think that's a
fair and valuable issue.  I think if we go a few releases without any
major issues, we can change this to the following:

  Note that a SHA-256 repository cannot yet share work with "regular"
  SHA-1 repositories.  Many tools do not yet understand SHA-256
  repositories, so users may wish to take this into account when
  creating new repositories.
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[Martin Ågren]

On Fri, 7 Aug 2020 at 01:08, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2020-08-06 at 20:23:58, Martin Ågren wrote:
> > After eff45daab8 ("repository: enable SHA-256 support by default",
> > 2020-07-29), vanilla builds of Git enable the user to run, e.g.,
> >
> >   git init --object-format=sha256
> >
> > and hack away. [...]
[...]
> > Similarly, "push + pull" should work, but you really will be operating
> > mostly offset from the rest of the world. That might be ok by the time
> > you initialize your repository, and it might be ok for several months
> > after that, but there might come a day when you're starting to regret
> > your use of `git init --object-format=sha256` and have dug yourself into
> > a fairly deep hole.
>
> I do agree that they don't interoperate right now, and that we'd like it
> to in the future.  But there are definitely people who can use SHA-256
> support for new projects without problems.  I'm aware of certain
> government agencies who very much do not want to use SHA-1 at all (and
> at some point will be legally prohibited from doing so), and they will
> be completely fine with the status quo.  Some of those same
> organizations are unhappy about prohibited algorithms even being linked
> into the binaries they use.  These folks can use a suitably new version
> of Git everywhere and not care about the lack of backwards
> compatibility.
>
> I am, of course, in favor of abandoning SHA-1 as fast as practically
> possible, but I understand that backwards compatibility is obviously a
> concern.

Yeah, I'd prefer them to know that they are early adopters and that they
should be prepared for a situation where there's some incompatibility
across versions. I don't just mean "I can't read my old SHA-1 data any
more", I mean "I used Git v2.29.0 to create a SHA-256 repo and now Git
v3.15.0 won't read it". (Or v2.31.0?)

I've followed the work on the commit graph functionality and file format
mostly from the sidelines. It's been lots of good work with lots of good
outcome, but there also seems to have been (of course) a few
incompatibilities, bugs and "argh, if only we'd have done it like this
from the beginning". I'd assume the effort -- and potential for bugs and
"ooh, we should have done it that way" -- for SHA-256--SHA-1
interoperability to be larger than what's been put into the commit graph
so far.

> > Workflows aside, let's consider a more technical aspect. Pack index
> > files (pack-*.idx) exist in two flavours: v1 and v2. [...]
[...]
> > We could certainly (re)define v2 to match our SHA-256 behavior, but we
> > do foresee v3 for a reason. And that would still just fix this specific
> > issue. And even when everything around SHA-256 is well-defined and we
> > have SHA-1--SHA-256 interoperability, there's a risk, at least
> > initially, that somewhere we'd be permeating buggy data that we'd then
> > feel responsible for and need to be able to handle for a long time to
> > come.
>
> These are valid index v1 and v2 files, just with a different hash
> algorithm.

I claim that they are not valid, precisely because they use a different
hash algorithm.

> v3 is there for the point where we do interoperate and need
> to store hash values of multiple algorithms at once.  There's little to
> no benefit to v3 if you don't need multiple algorithm support, other
> than the fact that they declare the algorithms in them.

One additional benefit: they'd correspond to a specification. :-)

> This is no different than saying that our commit or tree objects are in
> a different form; they are syntactically identical, just with a
> different hash algorithm.  That's how everything is in the .git
> directory.

For objects, I could perhaps accept that the format outlined in the hash
transition document is the specification.

That document says that pack indices "use a new v3 format that supports
multiple hash functions." It then goes on to draft such a format. (Maybe
it's a specification, but until there exists at least one
implementation, I'd rather see it as a draft.) No mention of v2 pack
indices with SHA-256 data, neither in that document, nor anywhere else
in our documentation that I could find.

The "v2 but with SHA-256" packfile index format we're producing contains
lots of 32 B SHA-256s instead of 20 B SHA-1s, ok, that much can be
guessed in one try. The index file ends with a 32 B SHA-256, after
referencing the 32 B packfile SHA-256. Ok, maybe that could also be
guessed. If we're committed to maintaining that format, we should put it
down in writing. And if we're not committed to it, we should make that
clear.

The hash transition document foresees a packfile index format v3.
Notably, it uses a _20_ B SHA-256 checksum and references a _20_ B
SHA-256 packfile SHA-256. In light of that, are we certain that the "v2
with SHA-256" format outlined above is not a maintenance burden? Or that
if there is any kind of cost, that it's worth it? Or, for that matter,
that guessing the details of "v2 but with SHA-256" is trivial?

I fully respect the effort that has gone into making the test suite
run with 32 B SHA-256 instead of 20 B SHA-1. But do we really intend to
support for many years to come the new file formats that such a test run
produces and consumes?

Bundles v3, yeah I guess so. Thanks for making that move!

Pack index "v2 but with SHA-256", maybe. At the very least, we should
set down our feet consciously.

> > +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still
> > +in an early stage.  A SHA-256 repository will in general not be able to
> > +share work with "regular" SHA-1 repositories.  It should be assumed
> > +that, e.g., Git internal file formats in relation to SHA-256
> > +repositories may change in backwards-incompatible ways.  Only use
> > +`--object-format=sha256` for testing purposes.
>
> I'm fine with marking the functionality experimental for a few releases,
> since it is possible we have bugs people haven't found, and adding a
> note about interoperability after that point, since I think that's a
> fair and valuable issue.  I think if we go a few releases without any
> major issues, we can change this to the following:
>
>   Note that a SHA-256 repository cannot yet share work with "regular"
>   SHA-1 repositories.  Many tools do not yet understand SHA-256
>   repositories, so users may wish to take this into account when
>   creating new repositories.

With respect, I think that's too aggressive. By that time, we may
conclude that, e.g., the "v2 pack indices with SHA-256" file handling is
robust. But I'd be surprised if using `git init --object-format=sha256`
in June 2021 won't cause *some* extra work for users or ourselves
further down the line compared to using a regular SHA-1 `git init`.
Pushing to a SHA-1 hosting service will become *possible* at some point,
but maybe it won't be *efficient enough to be practical in the real
world* until some time after that.

All those other, *new* file formats outlined in the hash
transition document won't exist at that time (at least not in master).

Now would probably be a good time to update the hash transition
documents, first of all to tick off what we've already done, and second,
to reassess the rest. Quoting:

  The first user-visible change is the introduction of the objectFormat
  extension (without compatObjectFormat). This requires:
  - implementing the loose-object-idx
  - teaching fsck about this mode of operation
  - using the hash function API (vtable) when computing object names
  - signing objects and verifying signatures
  - rejecting attempts to fetch from or push to an incompatible
    repository

I don't think we're there yet. Maybe, e.g., the new loose-object-idx
isn't strictly needed, in which case this part of the plan could be
updated. (Or maybe who wrote the above thought there'd be some value in
knowing that *all* SHA-256 repos *always* have loose-object-idx tables
to save us from some file-discovery dancing?) We do say elsewhere in the
document that

  Alongside the packfile, a SHA-256 repository stores a bidirectional
  mapping between SHA-256 and SHA-1 object names.

So at the time we do not seem to be producing correct, proper,
as-specified (or at least as-drafted) SHA-256 repositories. Or maybe in
2030, we can stop insisting on such a mapping, because everyone uses
SHA-256 anyway, so then maybe it shouldn't be mandatory now, either.

"Signing objects" is a bit vague, but under "Signed Commits", I see:

  [...]

  This means commits can be signed
  1. using SHA-1 only, as in existing signed commit objects
  2. using both SHA-1 and SHA-256, by using both gpgsig-sha256 and gpgsig
     fields.
  3. using only SHA-256, by only using the gpgsig-sha256 field.

Right now, we can do either 1 or 3. Maybe that's enough. I do think
there's a bug in git-replace where we'll only remove the last signature,
but as we'll currently only create one signature, that's perhaps "ok".
I still believe we should think hard before saying (even if we only say
so by omission) that, e.g., file structures are known-good and will be
supported for a long time to come.

Martin

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[Junio C Hamano]

Martin Ågren <martin.agren@gmail.com> writes:

>> I'm fine with marking the functionality experimental for a few releases,
>> since it is possible we have bugs people haven't found, and adding a
>> note about interoperability after that point, since I think that's a
>> fair and valuable issue.  I think if we go a few releases without any
>> major issues, we can change this to the following:
>>
>>   Note that a SHA-256 repository cannot yet share work with "regular"
>>   SHA-1 repositories.  Many tools do not yet understand SHA-256
>>   repositories, so users may wish to take this into account when
>>   creating new repositories.
>
> With respect, I think that's too aggressive. By that time, we may
> conclude that, e.g., the "v2 pack indices with SHA-256" file handling is
> robust. But I'd be surprised if using `git init --object-format=sha256`
> in June 2021 won't cause *some* extra work for users or ourselves
> further down the line compared to using a regular SHA-1 `git init`.
> Pushing to a SHA-1 hosting service will become *possible* at some point,
> but maybe it won't be *efficient enough to be practical in the real
> world* until some time after that.

IOW, you question "if we go a few releases without any major issues"
part?  I tend to agree that for a large change like this, a few
releases may not be sufficiently long time for a feature that is
marked as experimental in big flashing red letters to get exercised
enough to get major issues noticed.

> All those other, *new* file formats outlined in the hash
> transition document won't exist at that time (at least not in master).
>
> Now would probably be a good time to update the hash transition
> documents, first of all to tick off what we've already done, and second,
> to reassess the rest.

Yes, it is a good idea to stop and see where in the overall large
picture we currently are.

Thanks.

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[Martin Ågren]

On Fri, 7 Aug 2020 at 22:50, Junio C Hamano <gitster@pobox.com> wrote:
>
> Martin Ågren <martin.agren@gmail.com> writes:
>
> >> I'm fine with marking the functionality experimental for a few releases,
> >> since it is possible we have bugs people haven't found, and adding a
> >> note about interoperability after that point, since I think that's a
> >> fair and valuable issue.  I think if we go a few releases without any
> >> major issues, we can change this to the following:
> >>
> >>   Note that a SHA-256 repository cannot yet share work with "regular"
> >>   SHA-1 repositories.  Many tools do not yet understand SHA-256
> >>   repositories, so users may wish to take this into account when
> >>   creating new repositories.
> >
> > With respect, I think that's too aggressive. By that time, we may
> > conclude that, e.g., the "v2 pack indices with SHA-256" file handling is
> > robust. But I'd be surprised if using `git init --object-format=sha256`
> > in June 2021 won't cause *some* extra work for users or ourselves
> > further down the line compared to using a regular SHA-1 `git init`.
> > Pushing to a SHA-1 hosting service will become *possible* at some point,
> > but maybe it won't be *efficient enough to be practical in the real
> > world* until some time after that.
>
> IOW, you question "if we go a few releases without any major issues"
> part?  I tend to agree that for a large change like this, a few
> releases may not be sufficiently long time for a feature that is
> marked as experimental in big flashing red letters to get exercised
> enough to get major issues noticed.

Yeah, thanks for summarizing what I failed to express using so many
words.

I'm fully open to the idea that some people want to leave SHA-1 behind
and that they can do it today, in some "local" sense. If those people
are fully aware that they are guinea pigs, it might actually be ok for
us to subject them to a few rounds of "oops, Git v2.32.0 produces data
that v2.34.0 and later will barf on". Or at least it would be on our
table whether we wanted to be that cavalier.

Once SHA-256 repos as such are no longer experimental, I fear that we
can only buy ourselves that leeway by introducing fiftyeleven different
config flags for "please produce auxiliary files X even if you don't
actually use them", "please do use X, and I'm fully expecting to trip on
them if you decide to tweak them in backwards-incompatible ways", and so
on. The alternative to buying such leeway might be to establish, pretty
early on, a respectable set of things we support "for compatibility
reasons".

> > Now would probably be a good time to update the hash transition
> > documents, first of all to tick off what we've already done, and second,
> > to reassess the rest.
>
> Yes, it is a good idea to stop and see where in the overall large
> picture we currently are.

Martin

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[Junio C Hamano]

Martin Ågren <martin.agren@gmail.com> writes:

>> IOW, you question "if we go a few releases without any major issues"
>> part?  I tend to agree that for a large change like this, a few
>> releases may not be sufficiently long time for a feature that is
>> marked as experimental in big flashing red letters to get exercised
>> enough to get major issues noticed.
>
> Yeah, thanks for summarizing what I failed to express using so many
> words.
>
> I'm fully open to the idea that some people want to leave SHA-1 behind
> and that they can do it today, in some "local" sense. If those people
> are fully aware that they are guinea pigs, it might actually be ok for
> us to subject them to a few rounds of "oops, Git v2.32.0 produces data
> that v2.34.0 and later will barf on". Or at least it would be on our
> table whether we wanted to be that cavalier.
>
> Once SHA-256 repos as such are no longer experimental, I fear that we
> can only buy ourselves that leeway by introducing fiftyeleven different
> config flags for "please produce auxiliary files X even if you don't
> actually use them", "please do use X, and I'm fully expecting to trip on
> them if you decide to tweak them in backwards-incompatible ways", and so
> on. The alternative to buying such leeway might be to establish, pretty
> early on, a respectable set of things we support "for compatibility
> reasons".

OK, so can we resolve this one way or the other and move on?

For now, I'd vote for applying this warning patch, but with or
without such warning, it is more important to iron out those details
we fear might have to change.

Thanks.

Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2977 bytes --]

On 2020-08-13 at 20:34:10, Junio C Hamano wrote:
> Martin Ågren <martin.agren@gmail.com> writes:
> 
> >> IOW, you question "if we go a few releases without any major issues"
> >> part?  I tend to agree that for a large change like this, a few
> >> releases may not be sufficiently long time for a feature that is
> >> marked as experimental in big flashing red letters to get exercised
> >> enough to get major issues noticed.
> >
> > Yeah, thanks for summarizing what I failed to express using so many
> > words.
> >
> > I'm fully open to the idea that some people want to leave SHA-1 behind
> > and that they can do it today, in some "local" sense. If those people
> > are fully aware that they are guinea pigs, it might actually be ok for
> > us to subject them to a few rounds of "oops, Git v2.32.0 produces data
> > that v2.34.0 and later will barf on". Or at least it would be on our
> > table whether we wanted to be that cavalier.
> >
> > Once SHA-256 repos as such are no longer experimental, I fear that we
> > can only buy ourselves that leeway by introducing fiftyeleven different
> > config flags for "please produce auxiliary files X even if you don't
> > actually use them", "please do use X, and I'm fully expecting to trip on
> > them if you decide to tweak them in backwards-incompatible ways", and so
> > on. The alternative to buying such leeway might be to establish, pretty
> > early on, a respectable set of things we support "for compatibility
> > reasons".
> 
> OK, so can we resolve this one way or the other and move on?
> 
> For now, I'd vote for applying this warning patch, but with or
> without such warning, it is more important to iron out those details
> we fear might have to change.

I'm fine with applying this patch.

As for changes, I don't think there's any changes we need to make for a
stage 4 implementation.  It works and it passes the testsuite.  I've
verified the block and gcrypt SHA-256 implementations produce identical
results, which was my major worry.  Other than the philosophical
disagreement over whether index v1 and v2 should support SHA-256, I
don't think there's any points of contention.

When we add support for SHA-1 interoperability, then we'll need pack
index v3, proper multi-pack index support, and the loose object index.
Those are being written at the moment, and I think it's fine to add a
tool to generate the necessary files once that code is in place (which,
it's looking like, can just be a shell script that pipes every loose
object to git hash-object and rebuilds the pack indexes).  Support for
interoperability is an additional extension, so we don't have any
compatibility concerns and we can generate all of those files based on
that option.

I don't plan to enable support for extensions.compatObjectFormat without
all of the required pieces in place.  There won't be any incremental
steps there.
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

[PATCH v2] Documentation: mark `--object-format=sha256` as experimental

[Martin Ågren]

After eff45daab8 ("repository: enable SHA-256 support by default",
2020-07-29), vanilla builds of Git enable the user to run, e.g.,

  git init --object-format=sha256

and hack away. This can be a good way to gain experience with the
SHA-256 world, e.g., to find bugs that

  GIT_TEST_DEFAULT_HASH=sha256 make test

doesn't spot.

But it really is a separate world: Such SHA-256 repos will live entirely
separate from the (by now fairly large) set of SHA-1 repos. Interacting
across the border is possible in principle, e.g., through "diff + apply"
(or "format-patch + am"), but even that has its limitations: Applying a
SHA-256 diff in a SHA-1 repo works in the simple case, but if you need
to resort to `-3`, you're out of luck.

Similarly, "push + pull" should work, but you really will be operating
mostly offset from the rest of the world. That might be ok by the time
you initialize your repository, and it might be ok for several months
after that, but there might come a day when you're starting to regret
your use of `git init --object-format=sha256` and have dug yourself into
a fairly deep hole.

There are currently topics in flight to document our data formats and
protocols regarding SHA-256 and in some cases (midx and commit-graph),
we're considering adjusting how the file formats indicate which object
format to use.

Wherever `--object-format` is mentioned in our documentation, let's make
it clear that using it with "sha256" is experimental. If we later need
to explain why we can't handle data we generated back in 2020, we can
always point to this paragraph we're adding here.

By "include::"-ing a small blurb, we should be able to be consistent
throughout the documentation and can eventually gradually tone down the
severity of this text. One day, we might even use it to start phasing
out `--object-format=sha1`, but let's not get ahead of ourselves...

There's also `extensions.objectFormat`, but it's only mentioned three
times. Twice where we're adding this new disclaimer and in the third
spot we already have a "do not edit" warning. From there, interested
readers should eventually find this new one that we're adding here.

Because `GIT_DEFAULT_HASH` provides another entry point to this
functionality, document the experimental nature of it too.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Given the in-flight topics to document various SHA-256 details, I think
 it makes sense to simplify the commit message a bit. The diff is the
 same as in the original submission of this patch [1].

 [1] https://lore.kernel.org/git/20200806202358.2265705-1-martin.agren@gmail.com/

Range-diff against v1:
1:  19d5bb5028 ! 1:  0c2b124cfd Documentation: mark `--object-format=sha256` as experimental
    @@ Commit message
         your use of `git init --object-format=sha256` and have dug yourself into
         a fairly deep hole.
     
    -    Workflows aside, let's consider a more technical aspect. Pack index
    -    files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition
    -    document foresees a v3, which we do not yet support (and for all we
    -    know, the final v3 might end up different from the one sketched in the
    -    hash transition document).
    -
    -    When the test suite is run with SHA-1 as the default hash algo, it
    -    creates and consumes v2 pack files. But with SHA-256, we use an
    -    undocumented, hybrid format where the header looks like v2, but where
    -    the payload is not only "not correct SHA1", but where even the data
    -    sizes are different. The trailing checksum is different, meaning no-one
    -    (except us!) should/would try to interpret this file as a proper v2 pack
    -    index.
    -
    -    We could certainly (re)define v2 to match our SHA-256 behavior, but we
    -    do foresee v3 for a reason. And that would still just fix this specific
    -    issue. And even when everything around SHA-256 is well-defined and we
    -    have SHA-1--SHA-256 interoperability, there's a risk, at least
    -    initially, that somewhere we'd be permeating buggy data that we'd then
    -    feel responsible for and need to be able to handle for a long time to
    -    come.
    -
    -    In short: we need some time and leeway.
    +    There are currently topics in flight to document our data formats and
    +    protocols regarding SHA-256 and in some cases (midx and commit-graph),
    +    we're considering adjusting how the file formats indicate which object
    +    format to use.
     
         Wherever `--object-format` is mentioned in our documentation, let's make
         it clear that using it with "sha256" is experimental. If we later need
    @@ Commit message
         functionality, document the experimental nature of it too.
     
         Signed-off-by: Martin Ågren <martin.agren@gmail.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
      ## Documentation/git-index-pack.txt ##
     @@ Documentation/git-index-pack.txt: OPTIONS

 Documentation/git-index-pack.txt           | 2 ++
 Documentation/git-init.txt                 | 2 ++
 Documentation/git-show-index.txt           | 2 ++
 Documentation/git.txt                      | 3 ++-
 Documentation/object-format-disclaimer.txt | 6 ++++++
 5 files changed, 14 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/object-format-disclaimer.txt

diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt
index ac74d058e0..af0c26232c 100644
--- a/Documentation/git-index-pack.txt
+++ b/Documentation/git-index-pack.txt
@@ -100,6 +100,8 @@ OPTIONS
 	value is set or outside a repository.
 +
 This option cannot be used with --stdin.
++
+include::object-format-disclaimer.txt[]
 
 NOTES
 -----
diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt
index ddfe265da5..f35f70f13d 100644
--- a/Documentation/git-init.txt
+++ b/Documentation/git-init.txt
@@ -53,6 +53,8 @@ current working directory.
 
 Specify the given object format (hash algorithm) for the repository.  The valid
 values are 'sha1' and (if enabled) 'sha256'.  'sha1' is the default.
++
+include::object-format-disclaimer.txt[]
 
 --template=<template_directory>::
 
diff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt
index 39b1d8eaa1..e49318a5a0 100644
--- a/Documentation/git-show-index.txt
+++ b/Documentation/git-show-index.txt
@@ -44,6 +44,8 @@ OPTIONS
 	valid values are 'sha1' and (if enabled) 'sha256'.  The default is the
 	algorithm for the current repository (set by `extensions.objectFormat`), or
 	'sha1' if no value is set or outside a repository..
++
+include::object-format-disclaimer.txt[]
 
 GIT
 ---
diff --git a/Documentation/git.txt b/Documentation/git.txt
index 3e50065198..e0a2690ba2 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -497,7 +497,8 @@ double-quotes and respecting backslash escapes. E.g., the value
 	If this variable is set, the default hash algorithm for new
 	repositories will be set to this value. This value is currently
 	ignored when cloning; the setting of the remote repository
-	is used instead. The default is "sha1".
+	is used instead. The default is "sha1". THIS VARIABLE IS
+	EXPERIMENTAL! See `--object-format` in linkgit:git-init[1].
 
 Git Commits
 ~~~~~~~~~~~
diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt
new file mode 100644
index 0000000000..4cb106f0d1
--- /dev/null
+++ b/Documentation/object-format-disclaimer.txt
@@ -0,0 +1,6 @@
+THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still
+in an early stage.  A SHA-256 repository will in general not be able to
+share work with "regular" SHA-1 repositories.  It should be assumed
+that, e.g., Git internal file formats in relation to SHA-256
+repositories may change in backwards-incompatible ways.  Only use
+`--object-format=sha256` for testing purposes.
-- 
2.28.0.297.g1956fa8f8d