Question About Git V2 Protocol & SHA256

Question About Git V2 Protocol & SHA256

[Wu, Zhichen]

Hi Git Community,
 
Hope this email finds you well.
 
I’m Zhichen from AWS CodeCommit. I’m reaching out regarding 2 git features that we are planning for CodeCommit and your input would be critical for our next step.

1. Since v2 protocol has been launched for 2 years, is there any plan in the near future about v1 deprecation?
2. I see v2 has a capability called “object-format” that provides SHA1 option. I’m wondering if that capability will be the only way for client and server to start using SHA256? Or put it as another word, will v2 protocol be the prerequisite of SHA256?
 
Looking forward to hearing back from you.
 
Thanks in advance!
Zhichen Wu

Re: Question About Git V2 Protocol & SHA256

[Jeff King]

On Tue, Sep 29, 2020 at 01:17:59AM +0000, Wu, Zhichen wrote:

> 1. Since v2 protocol has been launched for 2 years, is there any plan
>    in the near future about v1 deprecation?

I suppose it depends what you mean by deprecation.

v2 is not even the default yet (but hopefully will be soon, perhaps in
the next release).

Once that happens, I doubt that we will add new capabilities or features
to v1 in general. Old versions of Git would not be able to take
advantage of them anyway, and new ones would be using v2.

But I don't think v1 will go away for a long time. There will still be
v1 clients and v1 servers hanging around for many years, and modern
versions of Git will need to speak both.

> 2. I see v2 has a capability called “object-format” that provides SHA1
>    option. I’m wondering if that capability will be the only way for
>    client and server to start using SHA256? Or put it as another word,
>    will v2 protocol be the prerequisite of SHA256?

I think it would be impossible to handle object-format via v1, because
the v1 protocol writes the ref advertisement before any capabilities are
negotiated. So I think v1 must implicitly remain sha1-only (and a sha256
repository on the server side would need to either reject a v1 client,
or back-translate as it would for a v2 client which asks for sha1).

-Peff

Re: Question About Git V2 Protocol & SHA256

[Wu, Zhichen]

Hi Peff,

Thank you for the quick response! That is very helpful and I think I have all the information I need.

Appreciate it a lot!
Zhichen

On 9/29/20, 3:13 PM, "Jeff King" <peff@peff.net> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Tue, Sep 29, 2020 at 01:17:59AM +0000, Wu, Zhichen wrote:

    > 1. Since v2 protocol has been launched for 2 years, is there any plan
    >    in the near future about v1 deprecation?

    I suppose it depends what you mean by deprecation.

    v2 is not even the default yet (but hopefully will be soon, perhaps in
    the next release).

    Once that happens, I doubt that we will add new capabilities or features
    to v1 in general. Old versions of Git would not be able to take
    advantage of them anyway, and new ones would be using v2.

    But I don't think v1 will go away for a long time. There will still be
    v1 clients and v1 servers hanging around for many years, and modern
    versions of Git will need to speak both.

    > 2. I see v2 has a capability called “object-format” that provides SHA1
    >    option. I’m wondering if that capability will be the only way for
    >    client and server to start using SHA256? Or put it as another word,
    >    will v2 protocol be the prerequisite of SHA256?

    I think it would be impossible to handle object-format via v1, because
    the v1 protocol writes the ref advertisement before any capabilities are
    negotiated. So I think v1 must implicitly remain sha1-only (and a sha256
    repository on the server side would need to either reject a v1 client,
    or back-translate as it would for a v2 client which asks for sha1).

    -Peff

Re: Question About Git V2 Protocol & SHA256

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1598 bytes --]

On 2020-09-29 at 22:13:11, Jeff King wrote:
> On Tue, Sep 29, 2020 at 01:17:59AM +0000, Wu, Zhichen wrote:
> > 2. I see v2 has a capability called “object-format” that provides SHA1
> >    option. I’m wondering if that capability will be the only way for
> >    client and server to start using SHA256? Or put it as another word,
> >    will v2 protocol be the prerequisite of SHA256?
> 
> I think it would be impossible to handle object-format via v1, because
> the v1 protocol writes the ref advertisement before any capabilities are
> negotiated. So I think v1 must implicitly remain sha1-only (and a sha256
> repository on the server side would need to either reject a v1 client,
> or back-translate as it would for a v2 client which asks for sha1).

I don't think that's the case.  You can indeed use v1 with SHA-256, but
if you have a SHA-1-only Git, it will choke because the object ID is
longer than it expects.  If you want to negotiate the algorithm when we
support both and the client can't deal with translating the initial ref
advertisement, then yes, you'll need v2.

We even support SHA-256 via bundles and the DAV-based HTTP protocol, but
the latter will never support negotiation of hash algorithms because
it's based on static files.

It is required that you understand the object-format capability on the
client side to support SHA-256, since if you fail to announce it, the
default is SHA-1, and right now, the server side will produce an error
if the client doesn't announce it (or sends SHA-1 data).
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Question About Git V2 Protocol & SHA256

[Wu, Zhichen]

Hi Brian,

Thanks for the details!

Now it sounds like although SHA1/SHA256 and v1/v2 are separate features, v2 capability is the only way for the client and server to negotiate the object format so that they won't send out something that is not understandable by the other party.

In that case, I think it's still valid that v2 needs to be supported first so that SHA256 can later be supported with enough flexibility.

Bests,
Zhichen


On 9/29/20, 3:45 PM, "brian m. carlson" <sandals@crustytoothpaste.net> wrote:

    On 2020-09-29 at 22:13:11, Jeff King wrote:
    > On Tue, Sep 29, 2020 at 01:17:59AM +0000, Wu, Zhichen wrote:
    > > 2. I see v2 has a capability called “object-format” that provides SHA1
    > >    option. I’m wondering if that capability will be the only way for
    > >    client and server to start using SHA256? Or put it as another word,
    > >    will v2 protocol be the prerequisite of SHA256?
    > 
    > I think it would be impossible to handle object-format via v1, because
    > the v1 protocol writes the ref advertisement before any capabilities are
    > negotiated. So I think v1 must implicitly remain sha1-only (and a sha256
    > repository on the server side would need to either reject a v1 client,
    > or back-translate as it would for a v2 client which asks for sha1).

    I don't think that's the case.  You can indeed use v1 with SHA-256, but
    if you have a SHA-1-only Git, it will choke because the object ID is
    longer than it expects.  If you want to negotiate the algorithm when we
    support both and the client can't deal with translating the initial ref
    advertisement, then yes, you'll need v2.

    We even support SHA-256 via bundles and the DAV-based HTTP protocol, but
    the latter will never support negotiation of hash algorithms because
    it's based on static files.

    It is required that you understand the object-format capability on the
    client side to support SHA-256, since if you fail to announce it, the
    default is SHA-1, and right now, the server side will produce an error
    if the client doesn't announce it (or sends SHA-1 data).
    -- 
    brian m. carlson: Houston, Texas, US

Re: Question About Git V2 Protocol & SHA256

[Jeff King]

On Tue, Sep 29, 2020 at 10:43:56PM +0000, brian m. carlson wrote:

> On 2020-09-29 at 22:13:11, Jeff King wrote:
> > On Tue, Sep 29, 2020 at 01:17:59AM +0000, Wu, Zhichen wrote:
> > > 2. I see v2 has a capability called “object-format” that provides SHA1
> > >    option. I’m wondering if that capability will be the only way for
> > >    client and server to start using SHA256? Or put it as another word,
> > >    will v2 protocol be the prerequisite of SHA256?
> > 
> > I think it would be impossible to handle object-format via v1, because
> > the v1 protocol writes the ref advertisement before any capabilities are
> > negotiated. So I think v1 must implicitly remain sha1-only (and a sha256
> > repository on the server side would need to either reject a v1 client,
> > or back-translate as it would for a v2 client which asks for sha1).
> 
> I don't think that's the case.  You can indeed use v1 with SHA-256, but
> if you have a SHA-1-only Git, it will choke because the object ID is
> longer than it expects.  If you want to negotiate the algorithm when we
> support both and the client can't deal with translating the initial ref
> advertisement, then yes, you'll need v2.

I agree that we _could_ just dump sha256 within a v1 protocol and wait
for the client to choke. But that seems like an awfully lousy user
experience. By contrast, assuming that the client wants sha1 means that
either:

  - we can reject it with a sensible ERR message that tells the user
    what is going on

  - we can serve them by talking in terms of sha1 if we're willing to
    do the extra conversion work server-side (and/or have a cache of
    sha1-format objects)

The only thing we lose is that a recent client who understands sha256
wouldn't be able to contact us and do a sha256-over-v1 transaction. But
why would they want to do so?

This is all hypothetical at this point, though, right? I tried finding
details in the hash transition document, but protocol changes are listed
as out of scope there. It does say that sha256 servers may just reject
sha1 clients; but even so I'd prefer if we could do it with a nice
message (i.e., my bullet one above).

My suggestion does also require that we have a v2 receive-pack protocol,
which does not yet exist (but following the blueprint for fetch, I don't
expect it to be a big deal).

-Peff

Re: Question About Git V2 Protocol & SHA256

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5532 bytes --]

On 2020-09-30 at 00:46:30, Jeff King wrote:
> I agree that we _could_ just dump sha256 within a v1 protocol and wait
> for the client to choke. But that seems like an awfully lousy user
> experience. By contrast, assuming that the client wants sha1 means that
> either:
> 
>   - we can reject it with a sensible ERR message that tells the user
>     what is going on
> 
>   - we can serve them by talking in terms of sha1 if we're willing to
>     do the extra conversion work server-side (and/or have a cache of
>     sha1-format objects)

If you have a suitably new Git, it will fail with an appropriate
message on both sides.  The client will complain that you are trying to
use a remote that uses the wrong hash algorithm and the server will
complain that you have failed to pass a suitable object-format
extension.  That will look like this:

  fatal: mismatched object format: server sha256; client sha1
  fatal: expected flush after ref listing

That's true if you're using protocol v2 at all, or a suitably new Git
with protocol v1, although the messages may differ slightly.  However,
if you're using an older Git with v1, then you'll get this:

  fatal: protocol error: unexpected '304c98b4860fa40b3e04f3e2e24db1a13ab959922a63209685813908c4fabc83 HEAD'
  fatal: the remote end hung up unexpectedly

That's because, as you pointed out, we don't get to the point where the
client fails to send the value.

> The only thing we lose is that a recent client who understands sha256
> wouldn't be able to contact us and do a sha256-over-v1 transaction. But
> why would they want to do so?

Because v2 isn't implemented for pushes yet.  The testsuite does not
even come close to passing unless you have a fully functional remote
system.  That's why I implemented protocol support, even though it
wasn't originally planned.

That was mentioned in the cover letter of the series that introduced
protocol support.

I should also point out that v2 SSH support doesn't work at all with
OpenSSH unless the server is specifically configured to allow the
GIT_PROTOCOL environment variable.  Since there are many use cases for
people to push Git data over SSH to servers they do not control, there
is a valid and viable use case for v1 support.  I personally know people
with shell accounts who could not use v2 over SSH but could viably use
v1.  Heck, that's even true for our shell host at work.

Absent a significant outreach campaign to all major operating system
vendors to get this environment variable added to all supported versions
of their OpenSSH packages, I don't see how restricting SHA-256 to
protocol v2 is acceptable.  Even then, many systems would still fail to
work correctly due to local config file modifications.

In retrospect, we probably should have sent "GIT_PROTOCOL=version=2" as
part of the shell command and let implementers deal with the fact that
their SSH servers would need to adequately parse and interpret shell
commands to function properly.  But it's probably too late to do that
now without bumping the protocol to v3.

> This is all hypothetical at this point, though, right? I tried finding
> details in the hash transition document, but protocol changes are listed
> as out of scope there. It does say that sha256 servers may just reject
> sha1 clients; but even so I'd prefer if we could do it with a nice
> message (i.e., my bullet one above).

It is not hypothetical.  Git in master has a fully functional SHA-256
implementation that fully interoperates with other SHA-256 repositories.
As I mentioned above and elsewhere, that is a prerequisite for a fully
functional testsuite.  I would not have shipped user-visible SHA-256
support with a testsuite as broken as it would be without protocol
support.

> My suggestion does also require that we have a v2 receive-pack protocol,
> which does not yet exist (but following the blueprint for fetch, I don't
> expect it to be a big deal).

If someone would like to implement protocol v2 for push in time for
2.29, then that could be a viable approach if we can address the serious
problem of the SSH situation in that time frame as well.

I don't plan to implement v2 push support because that is a substantial
amount of work and I have even less time to work on Git than normal due
to taking classes multiple nights a week.  As I know you are aware,
working on Git is not my day job.

I will admit to being a bit annoyed that we're discussing changes like
this at this point.  I've tried to be open about the design and
implementation, answering questions and providing complete branches and
reasonably thorough cover letters that people can peruse to know what's
going on even if they aren't interested in the full series themselves.
LWN has even summarized some of the later work.  It's fine if folks
don't want to participate in that process, but it's hard to incorporate
feedback that comes in well after the fact.  I certainly hope the nature
and content of my SHA-256 work isn't a surprise to regulars on the list,
because I've clearly failed to communicate effectively if that's the
case.

I am at this point not planning any other major changes to SHA-256
support before the interop work starts to land.  I am of course happy to
address any bug reports that come up, and if others would like to send
in patches related to SHA-256, I will review them and provide feedback
as my time allows.
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Question About Git V2 Protocol & SHA256

[Jeff King]

On Wed, Sep 30, 2020 at 02:19:25AM +0000, brian m. carlson wrote:

> If you have a suitably new Git, it will fail with an appropriate
> message on both sides.  The client will complain that you are trying to
> use a remote that uses the wrong hash algorithm and the server will
> complain that you have failed to pass a suitable object-format
> extension.  That will look like this:
> 
>   fatal: mismatched object format: server sha256; client sha1
>   fatal: expected flush after ref listing

That makes sense, and the message isn't too bad.

> That's true if you're using protocol v2 at all, or a suitably new Git
> with protocol v1, although the messages may differ slightly.  However,
> if you're using an older Git with v1, then you'll get this:
> 
>   fatal: protocol error: unexpected '304c98b4860fa40b3e04f3e2e24db1a13ab959922a63209685813908c4fabc83 HEAD'
>   fatal: the remote end hung up unexpectedly
> 
> That's because, as you pointed out, we don't get to the point where the
> client fails to send the value.

That makes switching to sha256 on the server a bit of a flag day, then,
doesn't it? Everybody with older Git will not be able to clone such a
repo. That may be OK for some cases, but I suspect it is going to cause
a delay in servers moving to sha256 themselves. Even if a project is
ready to migrate, it may be hesitant to do so because of third parties
outside of its control (whether external tools like CI, or just wanting
to be open to cloning by random users).

On the other hand, any kind of hash negotiation would require on-the-fly
conversion for that case (i.e., a sha256 server serving an older sha1
client needs to produce the alternate-universe sha1 objects). That's
expensive either in CPU or in disk space, and it may well be that
third-party hosting sites would refuse to enable such an option anyway.

> > The only thing we lose is that a recent client who understands sha256
> > wouldn't be able to contact us and do a sha256-over-v1 transaction. But
> > why would they want to do so?
> 
> Because v2 isn't implemented for pushes yet.  The testsuite does not
> even come close to passing unless you have a fully functional remote
> system.  That's why I implemented protocol support, even though it
> wasn't originally planned.
> 
> That was mentioned in the cover letter of the series that introduced
> protocol support.

Yeah, I think I misunderstood the direction there. I took the
object-format capability to mean that the client and server would
negotiate. But it really is the server saying "I'm going to speak this;
you'd better understand it and be able to translate objects on your
side".

I did realize that getting the test suite to pass meant that we'd be
passing sha256 hashes over the wire. But I had assumed that was part of
the strategy to make everything work in a sha256 world, and that more
interoperability stuff was coming on top. I.e., I thought our current
state was less mature than you ended up implementing.

> I should also point out that v2 SSH support doesn't work at all with
> OpenSSH unless the server is specifically configured to allow the
> GIT_PROTOCOL environment variable.  Since there are many use cases for
> people to push Git data over SSH to servers they do not control, there
> is a valid and viable use case for v1 support.  I personally know people
> with shell accounts who could not use v2 over SSH but could viably use
> v1.  Heck, that's even true for our shell host at work.

Thanks, that's an interesting counter-example that I didn't know about.

> In retrospect, we probably should have sent "GIT_PROTOCOL=version=2" as
> part of the shell command and let implementers deal with the fact that
> their SSH servers would need to adequately parse and interpret shell
> commands to function properly.  But it's probably too late to do that
> now without bumping the protocol to v3.

That was definitely discussed, but rejected because of the
interoperability problems. Not just for servers who actually want v2,
but it would mean that the v2 probe would cause existing v1 servers to
stop working. I.e., it would stop being safe to send v2 probes, which
was the whole point of the scheme.

> > My suggestion does also require that we have a v2 receive-pack protocol,
> > which does not yet exist (but following the blueprint for fetch, I don't
> > expect it to be a big deal).
> 
> If someone would like to implement protocol v2 for push in time for
> 2.29, then that could be a viable approach if we can address the serious
> problem of the SSH situation in that time frame as well.

Hmm. I hadn't really thought of 2.29 as the time-frame. In my mind,
sha256 support is still considered experimental, and we are free to
adjust things as necessary. But maybe that's naive. Anything in the
hands of users may be a source of annoying cross-version problems.

> I will admit to being a bit annoyed that we're discussing changes like
> this at this point.  I've tried to be open about the design and
> implementation, answering questions and providing complete branches and
> reasonably thorough cover letters that people can peruse to know what's
> going on even if they aren't interested in the full series themselves.
> LWN has even summarized some of the later work.  It's fine if folks
> don't want to participate in that process, but it's hard to incorporate
> feedback that comes in well after the fact.  I certainly hope the nature
> and content of my SHA-256 work isn't a surprise to regulars on the list,
> because I've clearly failed to communicate effectively if that's the
> case.

That's a fair criticism. I definitely have not followed your work as
closely as I could have (or perhaps should have). I'm glad we're having
this conversation now at least, though. I had some misunderstandings
about how we planned to proceed. And now I have a much better sense of
where you're at with it (and hopefully Zhichen sees that half of what I
wrote in my original reply is BS).

I can live with the scheme you described. The "flag day" thing is really
my biggest reservation, but it may just be something we have to accept.

-Peff

Re: Question About Git V2 Protocol & SHA256

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5320 bytes --]

On 2020-09-30 at 12:20:06, Jeff King wrote:
> On Wed, Sep 30, 2020 at 02:19:25AM +0000, brian m. carlson wrote:
> 
> > If you have a suitably new Git, it will fail with an appropriate
> > message on both sides.  The client will complain that you are trying to
> > use a remote that uses the wrong hash algorithm and the server will
> > complain that you have failed to pass a suitable object-format
> > extension.  That will look like this:
> > 
> >   fatal: mismatched object format: server sha256; client sha1
> >   fatal: expected flush after ref listing
> 
> That makes sense, and the message isn't too bad.

Yeah, it seems reasonable.  If people like something else better, it can
be adjusted.

> > That's true if you're using protocol v2 at all, or a suitably new Git
> > with protocol v1, although the messages may differ slightly.  However,
> > if you're using an older Git with v1, then you'll get this:
> > 
> >   fatal: protocol error: unexpected '304c98b4860fa40b3e04f3e2e24db1a13ab959922a63209685813908c4fabc83 HEAD'
> >   fatal: the remote end hung up unexpectedly
> > 
> > That's because, as you pointed out, we don't get to the point where the
> > client fails to send the value.
> 
> That makes switching to sha256 on the server a bit of a flag day, then,
> doesn't it? Everybody with older Git will not be able to clone such a
> repo. That may be OK for some cases, but I suspect it is going to cause
> a delay in servers moving to sha256 themselves. Even if a project is
> ready to migrate, it may be hesitant to do so because of third parties
> outside of its control (whether external tools like CI, or just wanting
> to be open to cloning by random users).

Right, but as the code stands now, those people couldn't make use of a
SHA-256 repository anyway because there's no interoperability.

> On the other hand, any kind of hash negotiation would require on-the-fly
> conversion for that case (i.e., a sha256 server serving an older sha1
> client needs to produce the alternate-universe sha1 objects). That's
> expensive either in CPU or in disk space, and it may well be that
> third-party hosting sites would refuse to enable such an option anyway.

We will eventually have some sort of compatibility, and it's
theoretically possible that hosting could offer SHA-1 compatibility for
those repositories, even if that's only for protocol v1.  So the
repository could use SHA-1 for v1 and negotiate for v2.

The desired behavior for a large hoster like GitHub or GitLab may be to
force v2 only for SHA-256; that's a justifiable decision there, because
we know all clients will support it and we don't have to deal with
remote systems that are poorly configured.  So I wouldn't be opposed to
seeing an option that did that.  As mentioned elsewhere, though, it
isn't a 100% viable option everywhere.

> > Because v2 isn't implemented for pushes yet.  The testsuite does not
> > even come close to passing unless you have a fully functional remote
> > system.  That's why I implemented protocol support, even though it
> > wasn't originally planned.
> > 
> > That was mentioned in the cover letter of the series that introduced
> > protocol support.
> 
> Yeah, I think I misunderstood the direction there. I took the
> object-format capability to mean that the client and server would
> negotiate. But it really is the server saying "I'm going to speak this;
> you'd better understand it and be able to translate objects on your
> side".

Yes, at this point it is, and it will remain that way for v1.

> I did realize that getting the test suite to pass meant that we'd be
> passing sha256 hashes over the wire. But I had assumed that was part of
> the strategy to make everything work in a sha256 world, and that more
> interoperability stuff was coming on top. I.e., I thought our current
> state was less mature than you ended up implementing.

More interoperability is coming, but the stage 4 (SHA-256 only)
implementation was easier to implement, so we got that first.

The good news is that the compat code is less involved than I expected
because we providently made the decision to write every new loose object
in the repository using a single function.  The biggest problem is now
the testsuite because we try to stuff all manner of invalid and broken
object IDs in places and that doesn't work anymore because they can't be
mapped.  I'm chipping away at this problem while watching movies on the
weekend, so it's coming.

> > In retrospect, we probably should have sent "GIT_PROTOCOL=version=2" as
> > part of the shell command and let implementers deal with the fact that
> > their SSH servers would need to adequately parse and interpret shell
> > commands to function properly.  But it's probably too late to do that
> > now without bumping the protocol to v3.
> 
> That was definitely discussed, but rejected because of the
> interoperability problems. Not just for servers who actually want v2,
> but it would mean that the v2 probe would cause existing v1 servers to
> stop working. I.e., it would stop being safe to send v2 probes, which
> was the whole point of the scheme.

Yeah.  I guess we can't have everything we want in this case.
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]