Re: revisiting zstd timings

[Junio C Hamano <gitster@pobox.com>]

Jeff King <peff@peff.net> writes:

> If we were designing git today, it seems like a no-brainer to use zstd
> over zlib. But given backwards-compatibility issues, I'm not sure.
> 10-20% speedup on reading is awfully nice, but I don't think there's a
> good way to gracefully transition, because zlib is part of the
> on-the-wire format for serving objects. We could re-compress on the fly,
> but that gets expensive (in existing cases, we can quite often serve the
> zlib content straight from disk, but this would require an extra
> inflate/deflate. At least we wouldn't have to reconstitute objects from
> deltas, though).
>
> A transition would probably look something like:
>
>   0. The patch below, or something like it, to teach git to read both
>      zlib and zstd, and optionally write zstd. We'd probably want to
>      make this an unconditional requirement like zlib, because the point
>      is for it to be available everywhere (I assume the zstd code is
>      pretty portable, but I haven't put it to the test).
>
>   1. Another patch to add a "zstd" capability to the protocol. This
>      would require teaching pack-objects an option to convert zstd back
>      to zlib on the fly.
>
>      Servers which handle a limited number of updated clients can switch
>      to zstd immediately to get the benefit, and their clients can
>      handle it directly. Likewise, clients of older servers may wish to
>      repack locally using zstd to get the benefit. They'll have to
>      recompress on the fly during push, but pushes are rare than other
>      operations (and often limited by bandwidth anyway).
>
>   2. After a while, eventually flip the default to zstd=5.
>
>   3. If "a while" is long enough, perhaps add a patch to let servers
>      tell clients "go upgrade" rather than recompressing on the fly.
>
> I don't have immediate plans for any of that, but maybe something to
> think about.

Thanks for a write-up.  This is quite interesting.

Thanks to d98b46f8d, this does not have to impact the object naming
;-)