Re: [PATCH 09/19] documentation: add documentation for the bitmap format

[Shawn Pearce <spearce@spearce.org>]

On Fri, Oct 25, 2013 at 3:21 AM, Jeff King <peff@peff.net> wrote:
> [+cc spearce; sorry, I really should have cc'd the whole series to you
>      in the first place]
>
> On Fri, Oct 25, 2013 at 08:16:18AM +0700, Nguyen Thai Ngoc Duy wrote:
>
>> > +       - A header appears at the beginning:
>> > +
>> > +               4-byte signature: {'B', 'I', 'T', 'M'}
>> > +
>> > +               2-byte version number (network byte order)
>> > +                       The current implementation only supports version 1
>> > +                       of the bitmap index (the same one as JGit).
>>
>> I suppose this means if I want to extend pack bitmaps to be used on
>> shallow clones, I need to step version to 2 before adding the shallow
>> points in .bitmap file as there's no chance of modifying v1 anymore,
>> correct?
>
> It depends on how you want to change it. And what Shawn says. :)

Correct, I think we need to bump to version 2 for any changes.

> This is an attempt to document the JGit format. My feeling is that there
> should be a version bump if an existing implementation would barf on it,
> and I assume Shawn would agree.  But if you want to simply add extra
> data that would be ignored by an existing implementation, it would be OK
> to add the data and mark it with a flag:
>
>> > +               2-byte flags (network byte order)
>> > +
>> > +                       The following flags are supported:
>> [...]
>
> That's how we added the name-hash cache in the final patch.
>
> That being said, JGit is _not_ happy with that, and complains when any
> flags besides OPT_FULL are used.  Whether that is because I am
> misinterpreting the intent of the flags field, or because JGit is being
> overly strict is up for debate.

Flags was created by Colby, I don't think he intended to use it for
determining if an optional section exists in the file or not. There is
insufficient information using only 1 bit to skip over an optional
section. The dircache file uses a length field with each extension
name to allow a reader to skip over sections it does not recognize
(and whose name is lowercase).

I think Colby and I talked about having additional optional sections
in this file, but Colby didn't want to overcomplicate the format early
on. So v1 is probably not very extensible and we may have to go to v2
to safely create an extension with the name hash cache used in this
series.

Given that the JGit v1 bitmap format has been shipping since JGit 3.0
and in Gerrit Code Review 2.6, its in use in the wild. So we aren't
going to go back and redefine v1.