git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Han-Wen Nienhuys via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org,
	"Carlo Marcelo Arenas Belón" <carenas@gmail.com>,
	"Han-Wen Nienhuys" <hanwenn@gmail.com>,
	"Han-Wen Nienhuys" <hanwen@google.com>
Subject: Re: [PATCH v2 08/19] reftable: reading/writing blocks
Date: Fri, 24 Sep 2021 13:52:28 +0200	[thread overview]
Message-ID: <87wnn62nhp.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <97f7ee04886d573f6b0f2e0d54a853cf016e7d74.1631213265.git.gitgitgadget@gmail.com>


On Thu, Sep 09 2021, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> The reftable format is structured as a sequence of block. Within a block,
> records are prefix compressed, with an index of offsets for fully expand keys to
> enable binary search within blocks.
> [...]
> +	if (typ == BLOCK_TYPE_LOG) {
> +		int block_header_skip = 4 + header_off;
> +		uLongf dst_len = sz - block_header_skip; /* total size of dest
> +							    buffer. */
> +		uLongf src_len = block->len - block_header_skip;
> +		/* Log blocks specify the *uncompressed* size in their header.
> +		 */
> +		uint8_t *uncompressed = reftable_malloc(sz);
> +
> +		/* Copy over the block header verbatim. It's not compressed. */
> +		memcpy(uncompressed, block->data, block_header_skip);
> +
> +		/* Uncompress */
> +		if (Z_OK !=
> +		    uncompress2(uncompressed + block_header_skip, &dst_len,
> +				block->data + block_header_skip, &src_len)) {
> +			reftable_free(uncompressed);
> +			return REFTABLE_ZLIB_ERROR;
> +		}
> +
> +		if (dst_len + block_header_skip != sz)
> +			return REFTABLE_FORMAT_ERROR;
> +
> +		/* We're done with the input data. */
> +		reftable_block_done(block);
> +		block->data = uncompressed;
> +		block->len = sz;
> +		block->source = malloc_block_source();
> +		full_block_size = src_len + block_header_skip;

I haven't tried to re-implement this, but in 07/19 we're adding an
uncompress2() fallaback function, and here's the only place where it'll
get used.

Looking at the uncompress2() implementation isn't this a rather trivial
wrapper for something we'd get from git.git's zlib.c, i.e. you seem to
just need the "avail_in" here, which we're checking in various other
parts of the codebase.

I suspect that the reason for this is that this code is trying to keep
at arms length from the rest of git.git, but if so some ifdef here if
it's compiling in git.git seems better than having a compat interface in
git.git just for doing something its zlib.c API provides.

Anyway, re Junio's ask if this is ready for "next" in the latest What's
Cooking I don't think this should be a blocker, just wondering if
anyone's interested in some post-cleanup at some point, and whether I'm
missing something about uncompress2() v.s. zlib.c.

  reply	other threads:[~2021-09-24 11:59 UTC|newest]

Thread overview: 111+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-30 14:57 [PATCH 00/19] Adds reftable library code from https://github.com/hanwen/reftable Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 01/19] hash.h: provide constants for the hash IDs Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 02/19] reftable: RFC: add LICENSE Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 03/19] reftable: add error related functionality Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 04/19] reftable: utility functions Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 05/19] reftable: add blocksource, an abstraction for random access reads Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 06/19] reftable: (de)serialization for the polymorphic record type Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 07/19] Provide zlib's uncompress2 from compat/zlib-compat.c Han-Wen Nienhuys via GitGitGadget
2021-09-02  6:12   ` [PATCH] fixup! " Carlo Marcelo Arenas Belón
2021-08-30 14:57 ` [PATCH 08/19] reftable: reading/writing blocks Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 09/19] reftable: a generic binary tree implementation Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 10/19] reftable: write reftable files Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 11/19] reftable: generic interface to tables Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 12/19] reftable: read reftable files Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 13/19] reftable: reftable file level tests Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 14/19] reftable: add a heap-based priority queue for reftable records Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 15/19] reftable: add merged table view Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 16/19] reftable: implement refname validation Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 17/19] reftable: implement stack, a mutable database of reftable files Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 18/19] reftable: add dump utility Han-Wen Nienhuys via GitGitGadget
2021-08-30 14:57 ` [PATCH 19/19] Add "test-tool dump-reftable" command Han-Wen Nienhuys via GitGitGadget
2021-08-30 15:22 ` [PATCH 00/19] Adds reftable library code from https://github.com/hanwen/reftable Han-Wen Nienhuys
2021-09-08  7:45 ` [PATCH 0/4] fixup for hn/reftable Carlo Marcelo Arenas Belón
2021-09-08  7:45   ` [PATCH 1/4] fixup! reftable: reading/writing blocks Carlo Marcelo Arenas Belón
2021-09-08  7:45   ` [PATCH 2/4] fixup! reftable: utility functions Carlo Marcelo Arenas Belón
2021-09-08  7:45   ` [PATCH 3/4] fixup! Provide zlib's uncompress2 from compat/zlib-compat.c Carlo Marcelo Arenas Belón
2021-09-08  7:45   ` [PATCH 4/4] fixup! reftable: utility functions Carlo Marcelo Arenas Belón
2021-09-08 18:50   ` [PATCH 0/4] fixup for hn/reftable Junio C Hamano
2021-09-09 18:47 ` [PATCH v2 00/19] Adds reftable library code from https://github.com/hanwen/reftable Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 01/19] hash.h: provide constants for the hash IDs Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 02/19] reftable: RFC: add LICENSE Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 03/19] reftable: add error related functionality Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 04/19] reftable: utility functions Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 05/19] reftable: add blocksource, an abstraction for random access reads Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 06/19] reftable: (de)serialization for the polymorphic record type Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 07/19] Provide zlib's uncompress2 from compat/zlib-compat.c Han-Wen Nienhuys via GitGitGadget
2021-09-15  7:34     ` Carlo Arenas
2021-09-09 18:47   ` [PATCH v2 08/19] reftable: reading/writing blocks Han-Wen Nienhuys via GitGitGadget
2021-09-24 11:52     ` Ævar Arnfjörð Bjarmason [this message]
2021-09-09 18:47   ` [PATCH v2 09/19] reftable: a generic binary tree implementation Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 10/19] reftable: write reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 11/19] reftable: generic interface to tables Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 12/19] reftable: read reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 13/19] reftable: reftable file level tests Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 14/19] reftable: add a heap-based priority queue for reftable records Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 15/19] reftable: add merged table view Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 16/19] reftable: implement refname validation Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 17/19] reftable: implement stack, a mutable database of reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 18/19] reftable: add dump utility Han-Wen Nienhuys via GitGitGadget
2021-09-09 18:47   ` [PATCH v2 19/19] Add "test-tool dump-reftable" command Han-Wen Nienhuys via GitGitGadget
2021-09-09 20:02   ` [PATCH v2 00/19] Adds reftable library code from https://github.com/hanwen/reftable Junio C Hamano
2021-09-09 20:32   ` Junio C Hamano
2021-09-13 10:14     ` Han-Wen Nienhuys
2021-09-13 18:30       ` Junio C Hamano
2021-09-13 19:29         ` Carlo Arenas
2021-09-13 20:34           ` Junio C Hamano
2021-09-28 15:09   ` [PATCH v3 " Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:09     ` [PATCH v3 01/19] hash.h: provide constants for the hash IDs Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:09     ` [PATCH v3 02/19] reftable: RFC: add LICENSE Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 03/19] reftable: add error related functionality Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 04/19] reftable: utility functions Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 05/19] reftable: add blocksource, an abstraction for random access reads Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 06/19] reftable: (de)serialization for the polymorphic record type Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 07/19] Provide zlib's uncompress2 from compat/zlib-compat.c Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 08/19] reftable: reading/writing blocks Han-Wen Nienhuys via GitGitGadget
2021-09-30 12:23       ` [PATCH] squash! " Carlo Marcelo Arenas Belón
2021-10-07 16:34         ` Han-Wen Nienhuys
2021-09-28 15:10     ` [PATCH v3 09/19] reftable: a generic binary tree implementation Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 10/19] reftable: write reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 11/19] reftable: generic interface to tables Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 12/19] reftable: read reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 13/19] reftable: reftable file level tests Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 14/19] reftable: add a heap-based priority queue for reftable records Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 15/19] reftable: add merged table view Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 16/19] reftable: implement refname validation Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 17/19] reftable: implement stack, a mutable database of reftable files Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 18/19] reftable: add dump utility Han-Wen Nienhuys via GitGitGadget
2021-09-28 15:10     ` [PATCH v3 19/19] Add "test-tool dump-reftable" command Han-Wen Nienhuys via GitGitGadget
2021-09-28 18:17     ` [PATCH v3 00/19] Adds reftable library code from https://github.com/hanwen/reftable Junio C Hamano
2021-10-02  9:20       ` Ævar Arnfjörð Bjarmason
2021-09-30  5:40     ` hn/reftable "fixes" Carlo Marcelo Arenas Belón
2021-09-30  5:40       ` [PATCH 1/4] fixup! reftable: add a heap-based priority queue for reftable records Carlo Marcelo Arenas Belón
2021-09-30  5:40       ` [PATCH 2/4] fixup! reftable: implement stack, a mutable database of reftable files Carlo Marcelo Arenas Belón
2021-10-01 15:37         ` C++(C99)-style comments in git.git Ævar Arnfjörð Bjarmason
2021-09-30  5:40       ` [PATCH 3/4] config.mak.uname: last release and snapshots of Minix still use zlib 1.2.3 Carlo Marcelo Arenas Belón
2021-09-30  5:40       ` [PATCH 4/4] reftable: avoid non portable compile time pointer to function Carlo Marcelo Arenas Belón
2021-09-30 20:35       ` hn/reftable "fixes" Junio C Hamano
2021-10-07 20:24     ` [PATCH v4 00/19] Adds reftable library code from https://github.com/hanwen/reftable Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:24       ` [PATCH v4 01/19] hash.h: provide constants for the hash IDs Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:24       ` [PATCH v4 02/19] reftable: add LICENSE Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:24       ` [PATCH v4 03/19] reftable: add error related functionality Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 04/19] reftable: utility functions Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 05/19] reftable: add blocksource, an abstraction for random access reads Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 06/19] reftable: (de)serialization for the polymorphic record type Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 07/19] Provide zlib's uncompress2 from compat/zlib-compat.c Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 08/19] reftable: reading/writing blocks Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 09/19] reftable: a generic binary tree implementation Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 10/19] reftable: write reftable files Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 11/19] reftable: generic interface to tables Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 12/19] reftable: read reftable files Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 13/19] reftable: reftable file level tests Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 14/19] reftable: add a heap-based priority queue for reftable records Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 15/19] reftable: add merged table view Han-Wen Nienhuys via GitGitGadget
2022-01-13 11:38         ` [PATCH] reftable tests: use C syntax compatible with old xlc Ævar Arnfjörð Bjarmason
2022-01-13 14:23           ` Han-Wen Nienhuys
2022-01-13 16:22             ` Ævar Arnfjörð Bjarmason
2022-01-13 19:09             ` Junio C Hamano
2021-10-07 20:25       ` [PATCH v4 16/19] reftable: implement refname validation Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 17/19] reftable: implement stack, a mutable database of reftable files Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 18/19] reftable: add dump utility Han-Wen Nienhuys via GitGitGadget
2021-10-07 20:25       ` [PATCH v4 19/19] Add "test-tool dump-reftable" command Han-Wen Nienhuys via GitGitGadget

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87wnn62nhp.fsf@evledraar.gmail.com \
    --to=avarab@gmail.com \
    --cc=carenas@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=hanwen@google.com \
    --cc=hanwenn@gmail.com \
    --subject='Re: [PATCH v2 08/19] reftable: reading/writing blocks' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).