git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
From: Todd Zullinger <tmz@pobox.com>
To: Jeremy Linton <lintonrjeremy@gmail.com>
Cc: git@vger.kernel.org, jonathantanmy@google.com, gitster@pobox.com,
	Eric Sunshine <sunshine@sunshineco.com>,
	Pavel Cahyna <pcahyna@redhat.com>
Subject: Re: [PATCH] packfile: Correct zlib buffer handling
Date: Fri, 25 May 2018 21:06:11 -0400
Message-ID: <20180526010611.GI26695@zaya.teonanacatl.net> (raw)
In-Reply-To: <20180525231713.23047-1-lintonrjeremy@gmail.com>

Jeremy Linton wrote:
> The buffer being passed to zlib includes a null terminator that
> git needs to keep in place. unpack_compressed_entry() attempts to
> detect the case that the source buffer hasn't been fully consumed
> by checking to see if the destination buffer has been over consumed.
> 
> This yields two problems, first a single byte overrun won't be detected
> properly because the Z_STREAM_END will then be set, but the null
> terminator will have been overwritten. The other problem is that
> more recent zlib patches have been poisoning the unconsumed portions
> of the buffers which also overwrites the null, while correctly
> returning length and status.
> 
> Lets rely on the fact that the source buffer will only be fully
> consumed when the when the destination buffer is inflated to the
> correct size. We can do this by passing zlib the correct buffer size
> and properly checking the return status. The latter check actually
> already exists if the buffer size is correct.
> 
> Signed-off-by: Jeremy Linton <lintonrjeremy@gmail.com>
> ---

As a little background, earlier today Pavel Cahyna filed a
ticket about a regression in a recent zlib update on aarch64
in Fedora[1].

While he was doing that I was just beginning to look at why
the git test suite failed fairly badly a build last
night[2].  The aarch64 build on Fedora 28 failed, while it
succeeded on all other architectures (armv7hl, i686, ppc64,
ppc64le, s390x, x86_64).  It also passed on newer and older
Fedora releases.

The difference was that the Fedora 28 zlib build has some
aarch64 optimization patches applied[3].  (Those patches are
in rawhide/f29 as well, but due to an unrelated issue have
not yet made it into the buildroot used for the git build.)

I'm woefully unqualified to comment on the patch, but if
there are any questions about how this was found, I hope the
above background will be helpful.

A big thanks to Jeremy for dropping whatever he had planned
to do today and dig into this issue.  I can only hope it was
either more fun or less work than what he hoped to do with
his Friday. :)

Thanks also to Pavel for finding the source of the failures
and filing a detailed bug report to get things moving.

[1] https://bugzilla.redhat.com/1582555
[2] https://fedorapeople.org/~tmz/git-aarch64-make-test
[3] https://src.fedoraproject.org/rpms/zlib/c/25e9802

>  packfile.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/packfile.c b/packfile.c
> index 7c1a2519f..245eb3204 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -1416,7 +1416,7 @@ static void *unpack_compressed_entry(struct packed_git *p,
>  		return NULL;
>  	memset(&stream, 0, sizeof(stream));
>  	stream.next_out = buffer;
> -	stream.avail_out = size + 1;
> +	stream.avail_out = size;
>  
>  	git_inflate_init(&stream);
>  	do {
> @@ -1424,7 +1424,7 @@ static void *unpack_compressed_entry(struct packed_git *p,
>  		stream.next_in = in;
>  		st = git_inflate(&stream, Z_FINISH);
>  		if (!stream.avail_out)
> -			break; /* the payload is larger than it should be */
> +			break; /* done, st indicates if source fully consumed */
>  		curpos += stream.next_in - in;
>  	} while (st == Z_OK || st == Z_BUF_ERROR);
>  	git_inflate_end(&stream);

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An election is coming. Universal peace is declared and the foxes have
a sincere interest in prolonging the lives of the poultry.
    -- T.S. Eliot


  parent reply index

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-05-25 23:17 Jeremy Linton
2018-05-25 23:36 ` Eric Sunshine
2018-05-26  1:06 ` Todd Zullinger [this message]
  -- strict thread matches above, loose matches on Subject: below --
2018-05-25 22:56 Jeremy Linton
2018-05-26  5:51 ` Duy Nguyen
2018-05-26 23:57   ` Junio C Hamano
2018-05-27  5:02     ` Duy Nguyen
2018-05-27 11:53       ` Junio C Hamano
2018-05-28  2:41         ` Junio C Hamano
2018-06-13  1:04           ` Jeremy Linton

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180526010611.GI26695@zaya.teonanacatl.net \
    --to=tmz@pobox.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jonathantanmy@google.com \
    --cc=lintonrjeremy@gmail.com \
    --cc=pcahyna@redhat.com \
    --cc=sunshine@sunshineco.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox