Re: [PATCH v2 09/10] reftable/block: reuse `zstream` state on inflation

[Karthik Nayak <karthik.188@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> When calling `inflateInit()` and `inflate()`, the zlib library will
> allocate several data structures for the underlying `zstream` to keep
> track of various information. Thus, when inflating repeatedly, it is
> possible to optimize memory allocation patterns by reusing the `zstream`
> and then calling `inflateReset()` on it to prepare it for the next chunk
> of data to inflate.
>
> This is exactly what the reftable code is doing: when iterating through
> reflogs we need to potentially inflate many log blocks, but we discard
> the `zstream` every single time. Instead, as we reuse the `block_reader`
> for each of the blocks anyway, we can initialize the `zstream` once and
> then reuse it for subsequent inflations.
>
> Refactor the code to do so, which leads to a significant reduction in
> the number of allocations. The following measurements were done when
> iterating through 1 million reflog entries. Before:
>
>   HEAP SUMMARY:
>       in use at exit: 13,473 bytes in 122 blocks
>     total heap usage: 23,028 allocs, 22,906 frees, 162,813,552 bytes allocated
>
> After:
>
>   HEAP SUMMARY:
>       in use at exit: 13,473 bytes in 122 blocks
>     total heap usage: 302 allocs, 180 frees, 88,352 bytes allocated
>

Really nice how just reusing the data structure has such a significant
impact.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]