Re: reftable [v3]: new ref storage format

[Stefan Beller <sbeller@google.com>]

On Mon, Jul 24, 2017 at 4:00 PM, Shawn Pearce <spearce@spearce.org> wrote:
> On Mon, Jul 24, 2017 at 3:22 PM, Stefan Beller <sbeller@google.com> wrote:

>>> #### index record
>>>
>>> An index record describes the last entry in another block.
>>> Index records are written as:
>>>
>>>     varint( prefix_length )
>>>     varint( (suffix_length << 3) | 0 )
>>>     suffix
>>>     varint( block_offset )
>>>
>>> Index records use prefix compression exactly like `ref_record`.
>>>
>>> Index records store `block_offset` after the suffix, specifying the
>>> offset in bytes (from the start of the file) of the block that ends
>>> with this reference.
>>
>> Instead of hardcoding the "0" in the last 3 bits, maybe pick one
>> of the reserved bit patterns to be there? I would imagine this
>> makes debugging easier:
>>
>>     0x5? Hah that must be an index block I have been
>>     looking at the wrong block!
>
> This is an excellent suggestion. I'll include it in the next iteration.

I was thinking about this a bit more and wondering if this would
allow mixing all sorts of entries in the same block. Think about one
of the most common cases client side:

    git commit -m "just a lonely ref update, no fancy stuff"

For that we need (a) a single ref update and (b) now that the
format evolved a single reflog entry, maybe (c) an obj entry, too.

I just realize that in this case we'd crank down the block size
such that there is very little padding required, but I still wonder
if it would be possible to omit the second (and third) block,
but rather go for a "m"ixed block containing just these 3 entries.

Additionally by setting the block_size to a special value "0",
we could indicate the omission of a footer, such that a single
ref update reftable is super small.