[PATCH] fetch-pack: be more precise in parsing v2 response

[PATCH] fetch-pack: be more precise in parsing v2 response

[Jonathan Tan]

Each section in a protocol v2 response is followed by either a DELIM
packet (indicating more sections to follow) or a FLUSH packet
(indicating none to follow). But when parsing the "acknowledgments"
section, do_fetch_pack_v2() is liberal in accepting both, but determines
whether to continue reading or not based solely on the contents of the
"acknowledgments" section, not on whether DELIM or FLUSH was read.

There is no issue with a protocol-compliant server, but can result in
confusing error messages when communicating with a server that
serves unexpected additional sections. Consider a server that sends
"new-section" after "acknowledgments":

 - client writes request
 - client reads the "acknowledgments" section which contains no "ready",
   then DELIM
 - since there was no "ready", client needs to continue negotiation, and
   writes request
 - client reads "new-section", and reports to the end user "expected
   'acknowledgments', received 'new-section'"

For the person debugging the involved Git implementation(s), the error
message is confusing in that "new-section" was not received in response
to the latest request, but to the first one.

One solution is to always continue reading after DELIM, but in this
case, we can do better. We know from the protocol that "ready" means at
least the packfile section is coming (hence, DELIM) and that no "ready"
means that no sections are to follow (hence, FLUSH). So teach
process_acks() to enforce this.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 fetch-pack.c           | 12 ++++++++++
 t/t5702-protocol-v2.sh | 50 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/fetch-pack.c b/fetch-pack.c
index b3ed7121bc..9691046e64 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1248,6 +1248,18 @@ static int process_acks(struct fetch_negotiator *negotiator,
 	    reader->status != PACKET_READ_DELIM)
 		die(_("error processing acks: %d"), reader->status);
 
+	/*
+	 * If an "acknowledgments" section is sent, a packfile is sent if and
+	 * only if "ready" was sent in this section. The other sections
+	 * ("shallow-info" and "wanted-refs") are sent only if a packfile is
+	 * sent. Therefore, a DELIM is expected if "ready" is sent, and a FLUSH
+	 * otherwise.
+	 */
+	if (received_ready && reader->status != PACKET_READ_DELIM)
+		die(_("expected packfile to be sent after 'ready'"));
+	if (!received_ready && reader->status != PACKET_READ_FLUSH)
+		die(_("expected no other sections to be sent after no 'ready'"));
+
 	/* return 0 if no common, 1 if there are common, or 2 if ready */
 	return received_ready ? 2 : (received_ack ? 1 : 0);
 }
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 8360188c01..51009ca391 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -512,6 +512,56 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '
 	! grep "git< version 2" log
 '
 
+test_expect_success 'when server sends "ready", expect DELIM' '
+	rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child &&
+
+	git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one &&
+
+	git clone "$HTTPD_URL/smart/http_parent" http_child &&
+
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two &&
+
+	# After "ready" in the acknowledgments section, pretend that a FLUSH
+	# (0000) was sent instead of a DELIM (0001).
+	printf "/ready/,$ s/0001/0000/" \
+		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+
+	test_must_fail git -C http_child -c protocol.version=2 \
+		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+	test_i18ngrep "expected packfile to be sent after .ready." err
+'
+
+test_expect_success 'when server does not send "ready", expect FLUSH' '
+	rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child log &&
+
+	git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one &&
+
+	git clone "$HTTPD_URL/smart/http_parent" http_child &&
+
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two &&
+
+	# Create many commits to extend the negotiation phase across multiple
+	# requests, so that the server does not send "ready" in the first
+	# request.
+	for i in $(test_seq 1 32)
+	do
+		test_commit -C http_child c$i
+	done &&
+
+	# After the acknowledgments section, pretend that a DELIM
+	# (0001) was sent instead of a FLUSH (0000).
+	printf "/acknowledgments/,$ s/0000/0001/" \
+		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+
+	GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
+		-c protocol.version=2 \
+		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
+	grep "fetch< acknowledgments" log &&
+	! grep "fetch< ready" log &&
+	test_i18ngrep "expected no other sections to be sent after no .ready." err
+'
 
 stop_httpd
 
-- 
2.19.0.271.gfe8321ec05.dirty

Re: [PATCH] fetch-pack: be more precise in parsing v2 response

[Junio C Hamano]

Jonathan Tan <jonathantanmy@google.com> writes:

> +	GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
> +		-c protocol.version=2 \
> +		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&

Because test_must_fail is a shell function, the above is not a
correct way to say "I want GIT_TRACE_PACKET exported only while this
thing runs".

I'll squash the following in.

 t/t5702-protocol-v2.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 51009ca391..d58fbfa9e5 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -555,7 +555,7 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
 	printf "/acknowledgments/,$ s/0000/0001/" \
 		>"$HTTPD_ROOT_PATH/one-time-sed" &&
 
-	GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
+	test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \
 		-c protocol.version=2 \
 		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
 	grep "fetch< acknowledgments" log &&

Re: [PATCH] fetch-pack: be more precise in parsing v2 response

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Jonathan Tan <jonathantanmy@google.com> writes:
>
>> +	GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
>> +		-c protocol.version=2 \
>> +		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
>
> Because test_must_fail is a shell function, the above is not a
> correct way to say "I want GIT_TRACE_PACKET exported only while this
> thing runs".
>
> I'll squash the following in.
>
>  t/t5702-protocol-v2.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
> index 51009ca391..d58fbfa9e5 100755
> --- a/t/t5702-protocol-v2.sh
> +++ b/t/t5702-protocol-v2.sh
> @@ -555,7 +555,7 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
>  	printf "/acknowledgments/,$ s/0000/0001/" \
>  		>"$HTTPD_ROOT_PATH/one-time-sed" &&
>  
> -	GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
> +	test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \
>  		-c protocol.version=2 \
>  		fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
>  	grep "fetch< acknowledgments" log &&

I know it only has been a few days, but is there any other issue
in the patch, anybody?

Otherwise, I am wondering if we can move this forwared after
squashing the above fix in.

Thanks.

Re: [PATCH] fetch-pack: be more precise in parsing v2 response

[Stefan Beller]

On Thu, Oct 25, 2018 at 2:04 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Jonathan Tan <jonathantanmy@google.com> writes:
> >
> >> +    GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
> >> +            -c protocol.version=2 \
> >> +            fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
> >
> > Because test_must_fail is a shell function, the above is not a
> > correct way to say "I want GIT_TRACE_PACKET exported only while this
> > thing runs".
> >
> > I'll squash the following in.
> >
> >  t/t5702-protocol-v2.sh | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
> > index 51009ca391..d58fbfa9e5 100755
> > --- a/t/t5702-protocol-v2.sh
> > +++ b/t/t5702-protocol-v2.sh
> > @@ -555,7 +555,7 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
> >       printf "/acknowledgments/,$ s/0000/0001/" \
> >               >"$HTTPD_ROOT_PATH/one-time-sed" &&
> >
> > -     GIT_TRACE_PACKET="$(pwd)/log" test_must_fail git -C http_child \
> > +     test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \
> >               -c protocol.version=2 \
> >               fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err &&
> >       grep "fetch< acknowledgments" log &&
>
> I know it only has been a few days, but is there any other issue
> in the patch, anybody?

I have reviewed the patch and I think it is good with the squashed change above.

Thanks,
Stefan