Re: Can I use CRoaring library in Git?

[Taylor Blau <me@ttaylorr.com>]

On Mon, Jul 18, 2022 at 02:27:59PM +0530, Abhradeep Chakraborty wrote:
> > I assume that Abhradeep will want to pursue some format redesign as part
> > of the transition, though, at least to see if changing the format beyond
> > a version bump and new compression scheme is worthwhile.
>
> I haven't thought much about it until now. As far as I think we don't
> need Xor Flag anymore.

I think that would be an interesting experiment to run. I suspect that
XOR-compression is helping us quite a lot with on-disk file size with
EWAH bitmaps, but that may or may not be true with Roaring.

If Roaring can compress the same selection of bitmaps to a comparable
size without the additional layer of XOR-offsets, then I think they are
additional complexity that can be eschewed for now.

(Keep in mind, we can always revisit that decision if we decide that we
want to add XOR compression back in through another version bump. But it
would be good to make backwards-incompatible changes as infrequently as
possible. So this is a good opportunity for us to be as thorough in our
experimentation as possible).

Thanks,
Taylor