[PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[lilinchao]

From: lilinchao <lilinchao@oschina.cn>

When clone with http protocol version 1, the server side
just tells client that "invalid server response, got version 1",
this is not clear enough, because version 0 is ok, and
version 2 is ok, then version 1 should be ok too intuitively,
but the other side just treat it as "invalid response", this
can't explain why is not ok.

>From receive-pack/upload-pack, there is a comment which I think,
can explain it:
"v1 is just the original protocol with a version string".
So I made this patch to try to fix it.

Signed-off-by: lilinchao <lilinchao@oschina.cn>
---
 remote-curl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/remote-curl.c b/remote-curl.c
index 0290b04891..1fe1f3c475 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -434,7 +434,8 @@ static void check_smart_http(struct discovery *d, const char *service,
 		 * be handled elsewhere.
 		 */
 		d->proto_git = 1;
-
+	} else if (!strcmp(reader.line, "version 1")) {
+		die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));
 	} else {
 		die(_("invalid server response; got '%s'"), reader.line);
 	}
-- 
2.30.0.1006.g4a81e96670

Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[Junio C Hamano]

lilinchao@oschina.cn writes:

>  		 * be handled elsewhere.
>  		 */
>  		d->proto_git = 1;
> -

Unrelated removal.

> +	} else if (!strcmp(reader.line, "version 1")) {
> +		die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));

The user may no longer get "invalid response; got 'version 1'", but
the above does not still explain why v1 is bad and v0 or v2 is
welcome, either.  IOW, I do not think the patch improves the message
to achieve what it attempted to do, i.e.

    ... but the other side just treat it as "invalid response", this
    can't explain why is not ok.

I wonder if it is a sensible and better alternative to treat v1
response as if we got v0 (if v1 is truly the same as v0 except for
the initial version advertisement).

Input from those who are familiar with the protocol versions is very
much appreciated.

Thanks.

>  	} else {
>  		die(_("invalid server response; got '%s'"), reader.line);
>  	}

Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>> +	} else if (!strcmp(reader.line, "version 1")) {
>> +		die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));
>
> The user may no longer get "invalid response; got 'version 1'", but
> the above does not still explain why v1 is bad and v0 or v2 is
> welcome, either.  IOW, I do not think the patch improves the message
> to achieve what it attempted to do, i.e.
>
>     ... but the other side just treat it as "invalid response", this
>     can't explain why is not ok.

Alternatively

	v1 is not supported; use v0 or v2

would explain why the connection is refused.  It explains why it is
not ok much clearly than "just the original with a version string".

> I wonder if it is a sensible and better alternative to treat v1
> response as if we got v0 (if v1 is truly the same as v0 except for
> the initial version advertisement).
>
> Input from those who are familiar with the protocol versions is very
> much appreciated.

This still stands; we reject because we don't support, but is it
easy to support it instead, if there is no difference?

Re: Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[lilinchao]

--------------
lilinchao@oschina.cn
>Junio C Hamano <gitster@pobox.com> writes:
>
>>> +	} else if (!strcmp(reader.line, "version 1")) {
>>> +	die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));
>>
>> The user may no longer get "invalid response; got 'version 1'", but
>> the above does not still explain why v1 is bad and v0 or v2 is
>> welcome, either.  IOW, I do not think the patch improves the message
>> to achieve what it attempted to do, i.e.
>>
>>     ... but the other side just treat it as "invalid response", this
>>     can't explain why is not ok.
>
>Alternatively
>
>	v1 is not supported; use v0 or v2
>
>would explain why the connection is refused.  It explains why it is
>not ok much clearly than "just the original with a version string".
>
>> I wonder if it is a sensible and better alternative to treat v1
>> response as if we got v0 (if v1 is truly the same as v0 except for
>> the initial version advertisement).
>>
>> Input from those who are familiar with the protocol versions is very
>> much appreciated.
>
>This still stands; we reject because we don't support, but is it
>easy to support it instead, if there is no difference? 

Yes, if there is no difference, just to support it too. So I don't know
why it is not support until now.

Thanks.

Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[Jeff King]

On Wed, Mar 24, 2021 at 01:28:32PM -0700, Junio C Hamano wrote:

> > +	} else if (!strcmp(reader.line, "version 1")) {
> > +		die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));
> 
> The user may no longer get "invalid response; got 'version 1'", but
> the above does not still explain why v1 is bad and v0 or v2 is
> welcome, either.  IOW, I do not think the patch improves the message
> to achieve what it attempted to do, i.e.
> 
>     ... but the other side just treat it as "invalid response", this
>     can't explain why is not ok.
> 
> I wonder if it is a sensible and better alternative to treat v1
> response as if we got v0 (if v1 is truly the same as v0 except for
> the initial version advertisement).
> 
> Input from those who are familiar with the protocol versions is very
> much appreciated.

Yes, "v1" is supposed to behave just like v0, except with the version
advertisement (it is true that there is no point in normal people using
it, but the purpose was to make sure the version advertisement worked).

I am not sure who is rejecting it, though. Our test suite passes with
GIT_TEST_PROTOCOL_VERSION=1. Running something like:

  $ GIT_TRACE_PACKET=1 git -c protocol.version=1 ls-remote https://github.com/git/git

yields a conversation like (cut down for clarity):

  git< # service=git-upload-pack
  git< 0000
  git< version 1
  git< 1234abcd[...etc, this is a normal v0/v1 advertisement]

So the version string is there, but it does not trigger the problem
described by this patch. That's because check_smart_http(), after seeing
the "# service" line and the flush, takes all the rest of the packetized
data and gives it to parse_git_refs(), which handles the version field
line via discover_version().

  Aside: on gitlab.com, the v1 response looks like a v0 response, with
  no extra header. I guess they did not bother to implement v1, which is
  OK, since it was not useful after the initial experiment.

So everything seems to be working as intended. Is there some particular
server that returns "version 1" in the wrong way, triggering the die()?

One curiosity is that for v2, the response from github.com does include
the "service" line. So it follows the same path as v1, and never hits
the "version 2" line check here. But http-backend omits the "service"
line, due to 237ffedd46 (http: eliminate "# service" line when using
protocol v2, 2018-03-15).

So it's interesting that GitHub behaves differently than http-backend
here. It's not surprising, since the HTTP framing is all done by a
custom server there, which implemented off the spec.  What _is_
surprising is that the client seems perfectly happy to see either form,
and nobody has noticed the difference until just now.

IMHO the spec is very unclear here; it says "client makes a smart
info/refs request as described in http-protocol.txt", but doesn't call
out the difference in the response. It's only implied by the example:

  A v2 server would reply:

     S: 200 OK
     S: <Some headers>
     S: ...
     S:
     S: 000eversion 2\n
     S: <capability-advertisement>

where it is unclear whether the blank line is separating HTTP headers
from the body (and thus "..." is some headers), or if it is separating
the "# service" line and matching flush from the rest of the response
body.

I note that gitlab.com also returns the "service" line for v2 (I don't
know anything about their implementation, but I would not be at all
surprised if they also use a custom HTTP endpoint; apache+http-backend
is not very flexible or scalable).

Anyway, that's all just an interesting side note. The client is happy
with either form (though it might be nice if we had tests for the "#
service" form; I suspect our tests don't cover that because they are all
using http-backend).

Getting back to the patch at hand, if there is a server saying "version
1" without a "service" line, then I think that is a bug in that server.

-Peff

Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[Jeff King]

On Fri, Mar 26, 2021 at 02:24:03AM -0400, Jeff King wrote:

> One curiosity is that for v2, the response from github.com does include
> the "service" line. So it follows the same path as v1, and never hits
> the "version 2" line check here. But http-backend omits the "service"
> line, due to 237ffedd46 (http: eliminate "# service" line when using
> protocol v2, 2018-03-15).
> 
> So it's interesting that GitHub behaves differently than http-backend
> here. It's not surprising, since the HTTP framing is all done by a
> custom server there, which implemented off the spec.  What _is_
> surprising is that the client seems perfectly happy to see either form,
> and nobody has noticed the difference until just now.
> 
> IMHO the spec is very unclear here; it says "client makes a smart
> info/refs request as described in http-protocol.txt", but doesn't call
> out the difference in the response. It's only implied by the example:
> 
>   A v2 server would reply:
> 
>      S: 200 OK
>      S: <Some headers>
>      S: ...
>      S:
>      S: 000eversion 2\n
>      S: <capability-advertisement>
> 
> where it is unclear whether the blank line is separating HTTP headers
> from the body (and thus "..." is some headers), or if it is separating
> the "# service" line and matching flush from the rest of the response
> body.
> 
> I note that gitlab.com also returns the "service" line for v2 (I don't
> know anything about their implementation, but I would not be at all
> surprised if they also use a custom HTTP endpoint; apache+http-backend
> is not very flexible or scalable).

I wondered two things:

  - how other servers behave; jgit is the obvious other one to check. It
    seems to match http-backend in omitting the "service" line. I also
    checked its v1 behavior. It seems to ignore it totally and behave
    like v0 (which again, is OK, since it's not useful). This was based
    on testing against https://android.googlesource.com. In v0, it
    claims agent=JGit/4-google, though curiously in v2 it does not
    advertise an agent at all. :)

  - whether other v2 clients are equally forgiving of either format.
    Again, jgit is probably the most interesting here (libgit2 does not
    speak v2 at all yet). And indeed, it seems to be happy with either
    format (which is not surprising, given how common both types of
    server are).

-Peff

Re: Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[lilinchao]

--------------
lilinchao@oschina.cn
>On Wed, Mar 24, 2021 at 01:28:32PM -0700, Junio C Hamano wrote:
>
>> > +	} else if (!strcmp(reader.line, "version 1")) {
>> > +	die(_("v1 is just the original protocol with a version string, use v0 or v2 instead."));
>>
>> The user may no longer get "invalid response; got 'version 1'", but
>> the above does not still explain why v1 is bad and v0 or v2 is
>> welcome, either.  IOW, I do not think the patch improves the message
>> to achieve what it attempted to do, i.e.
>>
>>     ... but the other side just treat it as "invalid response", this
>>     can't explain why is not ok.
>>
>> I wonder if it is a sensible and better alternative to treat v1
>> response as if we got v0 (if v1 is truly the same as v0 except for
>> the initial version advertisement).
>>
>> Input from those who are familiar with the protocol versions is very
>> much appreciated.
>
>Yes, "v1" is supposed to behave just like v0, except with the version
>advertisement (it is true that there is no point in normal people using
>it, but the purpose was to make sure the version advertisement worked).
>
>I am not sure who is rejecting it, though. Our test suite passes with
>GIT_TEST_PROTOCOL_VERSION=1. Running something like:
>
> $ GIT_TRACE_PACKET=1 git -c protocol.version=1 ls-remote https://github.com/git/git
>
>yields a conversation like (cut down for clarity):
>
>  git< # service=git-upload-pack
>  git< 0000
>  git< version 1
>  git< 1234abcd[...etc, this is a normal v0/v1 advertisement]
>
>So the version string is there, but it does not trigger the problem
>described by this patch. That's because check_smart_http(), after seeing
>the "# service" line and the flush, takes all the rest of the packetized
>data and gives it to parse_git_refs(), which handles the version field
>line via discover_version().
>
>  Aside: on gitlab.com, the v1 response looks like a v0 response, with
>  no extra header. I guess they did not bother to implement v1, which is
>  OK, since it was not useful after the initial experiment.
>
>So everything seems to be working as intended. Is there some particular
>server that returns "version 1" in the wrong way, triggering the die()?
> 
On gitee.com, I got "version 1", and the process died here.

>One curiosity is that for v2, the response from github.com does include
>the "service" line. So it follows the same path as v1, and never hits
>the "version 2" line check here. But http-backend omits the "service"
>line, due to 237ffedd46 (http: eliminate "# service" line when using
>protocol v2, 2018-03-15).
> 
Keen observation :)

>So it's interesting that GitHub behaves differently than http-backend
>here. It's not surprising, since the HTTP framing is all done by a
>custom server there, which implemented off the spec.  What _is_
>surprising is that the client seems perfectly happy to see either form,
>and nobody has noticed the difference until just now.
>
>IMHO the spec is very unclear here; it says "client makes a smart
>info/refs request as described in http-protocol.txt", but doesn't call
>out the difference in the response. It's only implied by the example:
>
>  A v2 server would reply:
>
>     S: 200 OK
>     S: <Some headers>
>     S: ...
>     S:
>     S: 000eversion 2\n
>     S: <capability-advertisement>
>
>where it is unclear whether the blank line is separating HTTP headers
>from the body (and thus "..." is some headers), or if it is separating
>the "# service" line and matching flush from the rest of the response
>body.
>
>I note that gitlab.com also returns the "service" line for v2 (I don't
>know anything about their implementation, but I would not be at all
>surprised if they also use a custom HTTP endpoint; apache+http-backend
>is not very flexible or scalable).
> 
gitee.com returns the "version 1" line for v1, so died for invalid server response
and it returns the "version 2" line for v2, which is expected.

>Anyway, that's all just an interesting side note. The client is happy
>with either form (though it might be nice if we had tests for the "#
>service" form; I suspect our tests don't cover that because they are all
>using http-backend).
>
>Getting back to the patch at hand, if there is a server saying "version
>1" without a "service" line, then I think that is a bug in that server.
> 
If the problem is on the server side, then, is this patch worth continuing?

Thanks!

>-Peff

Re: Re: [PATCH 2/2] remote-curl.c: handle v1 when check_smart_http

[Jeff King]

On Mon, Mar 29, 2021 at 05:30:29PM +0800, lilinchao@oschina.cn wrote:

> >Anyway, that's all just an interesting side note. The client is happy
> >with either form (though it might be nice if we had tests for the "#
> >service" form; I suspect our tests don't cover that because they are all
> >using http-backend).
> >
> >Getting back to the patch at hand, if there is a server saying "version
> >1" without a "service" line, then I think that is a bug in that server.
> > 
> If the problem is on the server side, then, is this patch worth continuing?

IMHO, no. I think the response from gitee.com is violating the protocol
spec. It would be nice to fix, but in practice it isn't all that
important because somebody would have to manually set protocol.version=1
to see the problem.

-Peff