From: Jonathan Nieder <jrnieder@gmail.com>
To: Brandon Williams <bmwill@google.com>
Cc: git@vger.kernel.org, Anton Golubev <anton.golubev@gmail.com>
Subject: Re: [PATCH 1/2] remote-curl: accept all encoding supported by curl
Date: Mon, 21 May 2018 17:01:29 -0700 [thread overview]
Message-ID: <20180522000129.GG10623@aiede.svl.corp.google.com> (raw)
In-Reply-To: <20180521234004.142548-1-bmwill@google.com>
Hi,
Brandon Williams wrote:
> Subject: remote-curl: accept all encoding supported by curl
nit: s/encoding/encodings
> Configure curl to accept all encoding which curl supports instead of
> only accepting gzip responses.
Likewise.
> This is necessary to fix a bug when using an installation of curl which
> doesn't support gzip. Since curl doesn't do any checking to verify that
> it supports the encoding set when calling 'curl_easy_setopt()', curl can
> end up sending an "Accept-Encoding" header indicating that it supports
> a particular encoding when in fact it doesn't. Instead when the empty
> string "" is used when setting `CURLOPT_ENCODING`, curl will send an
> "Accept-Encoding" header containing only the encoding methods curl
> supports.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
Thanks for the analysis and fix.
Reported-by: Anton Golubev <anton.golubev@gmail.com>
Also ccing the reporter so we can hopefully get a tested-by. Anton,
can you test this patch and let us know how it goes? You can apply it
as follows:
curl \
https://public-inbox.org/git/20180521234004.142548-1-bmwill@google.com/raw \
>patch.txt
git am -3 patch.txt
Brandon, can the commit message also say a little more about the
motivating context and symptoms?
$ curl --version
curl 7.52.1 (arm-openwrt-linux-gnu) libcurl/7.52.1 mbedTLS/2.6.0
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL
The issue is that when curl is built without the "zlib" feature, since
v1.8.0-rc0~14^2 (Enable info/refs gzip decompression in HTTP client,
2012-09-19) we end up requesting "gzip" encoding anyway despite
libcurl not being able to decode it. Worse, instead of getting a
clear error message indicating so, we end up falling back to "dumb"
http, producing a confusing and difficult to debug result.
> ---
> http.c | 2 +-
> remote-curl.c | 2 +-
> t/t5551-http-fetch-smart.sh | 4 ++--
> 3 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/http.c b/http.c
> index fed13b216..709150fc7 100644
> --- a/http.c
> +++ b/http.c
> @@ -1788,7 +1788,7 @@ static int http_request(const char *url,
>
> curl_easy_setopt(slot->curl, CURLOPT_URL, url);
> curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
> - curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
> + curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
>
> ret = run_one_slot(slot, &results);
>
> diff --git a/remote-curl.c b/remote-curl.c
> index ceb05347b..565bba104 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -684,7 +684,7 @@ static int post_rpc(struct rpc_state *rpc)
> curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
> curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
> curl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);
> - curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
> + curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
>
> if (large_request) {
> /* The request body is large and the size cannot be predicted.
Makes sense.
> diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
> index f5721b4a5..39c65482c 100755
> --- a/t/t5551-http-fetch-smart.sh
> +++ b/t/t5551-http-fetch-smart.sh
> @@ -26,14 +26,14 @@ setup_askpass_helper
> cat >exp <<EOF
> > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
> > Accept: */*
> -> Accept-Encoding: gzip
> +> Accept-Encoding: deflate, gzip
> > Pragma: no-cache
> < HTTP/1.1 200 OK
> < Pragma: no-cache
> < Cache-Control: no-cache, max-age=0, must-revalidate
> < Content-Type: application/x-git-upload-pack-advertisement
> > POST /smart/repo.git/git-upload-pack HTTP/1.1
> -> Accept-Encoding: gzip
> +> Accept-Encoding: deflate, gzip
> > Content-Type: application/x-git-upload-pack-request
> > Accept: application/x-git-upload-pack-result
> > Content-Length: xxx
If libcurl gains support for another encoding in the future, this test
would start failing. Can we make the matching less strict? For
example, how about something like the following for squashing in?
Thanks,
Jonathan
diff --git i/t/t5551-http-fetch-smart.sh w/t/t5551-http-fetch-smart.sh
index 39c65482ce..913089b144 100755
--- i/t/t5551-http-fetch-smart.sh
+++ w/t/t5551-http-fetch-smart.sh
@@ -26,14 +26,14 @@ setup_askpass_helper
cat >exp <<EOF
> GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
> Accept: */*
-> Accept-Encoding: deflate, gzip
+> Accept-Encoding: ENCODINGS
> Pragma: no-cache
< HTTP/1.1 200 OK
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-advertisement
> POST /smart/repo.git/git-upload-pack HTTP/1.1
-> Accept-Encoding: deflate, gzip
+> Accept-Encoding: ENCODINGS
> Content-Type: application/x-git-upload-pack-request
> Accept: application/x-git-upload-pack-result
> Content-Length: xxx
@@ -79,8 +79,13 @@ test_expect_success 'clone http repository' '
/^< Date: /d
/^< Content-Length: /d
/^< Transfer-Encoding: /d
- " >act &&
- test_cmp exp act
+ " >actual &&
+ sed -e "s/^> Accept-Encoding: .*/> Accept-Encoding: ENCODINGS/" \
+ actual >actual.smudged &&
+ test_cmp exp actual.smudged &&
+
+ grep "Accept-Encoding:.*gzip" actual >actual.gzip &&
+ test_line_count = 2 actual.gzip
'
test_expect_success 'fetch changes via http' '
next prev parent reply other threads:[~2018-05-22 0:01 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-05-21 23:40 [PATCH 1/2] remote-curl: accept all encoding supported by curl Brandon Williams
2018-05-21 23:40 ` [PATCH 2/2] remote-curl: accept compressed responses with protocol v2 Brandon Williams
2018-05-22 0:01 ` Jonathan Nieder [this message]
2018-05-26 11:08 ` [PATCH 1/2] remote-curl: accept all encoding supported by curl anton.golubev
2018-05-22 0:02 ` Stefan Beller
2018-05-22 1:00 ` Jonathan Nieder
2018-05-22 6:32 ` Daniel Stenberg
2018-05-22 18:40 ` Brandon Williams
2018-05-22 18:42 ` [PATCH v2 1/2] remote-curl: accept all encodings " Brandon Williams
2018-05-22 18:42 ` [PATCH v2 2/2] remote-curl: accept compressed responses with protocol v2 Brandon Williams
2018-05-22 18:55 ` Jonathan Nieder
2018-05-22 18:48 ` [PATCH v2 1/2] remote-curl: accept all encodings supported by curl Jonathan Nieder
2018-05-23 1:23 ` Junio C Hamano
2018-05-23 2:17 ` brian m. carlson
2018-05-23 5:55 ` Daniel Stenberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180522000129.GG10623@aiede.svl.corp.google.com \
--to=jrnieder@gmail.com \
--cc=anton.golubev@gmail.com \
--cc=bmwill@google.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).