Re: [PATCH v4 01/13] commit-graph: add format document

[Derrick Stolee <stolee@gmail.com>]

On 2/21/2018 2:23 PM, Stefan Beller wrote:
> On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee <stolee@gmail.com> wrote:
>> +In order to allow extensions that add extra data to the graph, we organize
>> +the body into "chunks" and provide a binary lookup table at the beginning
>> +of the body. The header includes certain values, such as number of chunks,
>> +hash lengths and types.
>> +
>> +All 4-byte numbers are in network order.
>> +
>> +HEADER:
>> +
>> +  4-byte signature:
>> +      The signature is: {'C', 'G', 'P', 'H'}
>> +
>> +  1-byte version number:
>> +      Currently, the only valid version is 1.
>> +
>> +  1-byte Object Id Version (1 = SHA-1)
>> +
>> +  1-byte number (C) of "chunks"
>> +
>> +  1-byte (reserved for later use)
> What should clients of today do with it?
> * ignore it completely [as they have no idea what it is] or
> * throw hands up in the air if it is anything other than 0 ?
>    [because clearly we will increment the version
>     or have new information in a new chunk instead of just sneaking
>     in information here?]

They should ignore it completely, which will allow using the value for 
something meaningful later without causing a version change (which we DO 
die() for). A user could downgrade from a version that uses this byte 
for something meaningful and not require a new commit-graph file.

The "commit-graph read" subcommand does output this byte, so we can 
verify that the "write" subcommand places a 0 in this position.

>
>> +CHUNK LOOKUP:
>> +
>> +  (C + 1) * 12 bytes listing the table of contents for the chunks:
>> +      First 4 bytes describe chunk id. Value 0 is a terminating label.
>> +      Other 8 bytes provide offset in current file for chunk to start.
> offset [in bytes? I could imagine having a larger granularity here,
> because chunks don't sound small.]

It is good to specify "offset in bytes".

>
>> +      (Chunks are ordered contiguously in the file, so you can infer
>> +      the length using the next chunk position if necessary.)
>> +
>> +  The remaining data in the body is described one chunk at a time, and
>> +  these chunks may be given in any order. Chunks are required unless
>> +  otherwise specified.
>> +
>> +CHUNK DATA:
>> +
>> +  OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)
>> +      The ith entry, F[i], stores the number of OIDs with first
>> +      byte at most i. Thus F[255] stores the total
>> +      number of commits (N).
> [ so in small repos, where there are fewer than 256 objects,
> F[i] == F[i+1], for all i'th where there is no object starting with i byte]

Correct. I'm not sure this additional information is valuable for the 
document, though.

>
>> +  OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)
>> +      The OIDs for all commits in the graph, sorted in ascending order.
>> +
>> +  Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)
>> +    * The first H bytes are for the OID of the root tree.
>> +    * The next 8 bytes are for the int-ids of the first two parents
>> +      of the ith commit. Stores value 0xffffffff if no parent in that
>> +      position. If there are more than two parents, the second value
>> +      has its most-significant bit on and the other bits store an array
>> +      position into the Large Edge List chunk.
>> +    * The next 8 bytes store the generation number of the commit and
>> +      the commit time in seconds since EPOCH. The generation number
>> +      uses the higher 30 bits of the first 4 bytes, while the commit
>> +      time uses the 32 bits of the second 4 bytes, along with the lowest
>> +      2 bits of the lowest byte, storing the 33rd and 34th bit of the
>> +      commit time.
>> +
>> +  Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]
>> +      This list of 4-byte values store the second through nth parents for
>> +      all octopus merges. The second parent value in the commit data stores
>> +      an array position within this list along with the most-significant bit
>> +      on. Starting at that array position, iterate through this list of int-ids
>> +      for the parents until reaching a value with the most-significant bit on.
>> +      The other bits correspond to the int-id of the last parent.
>> +
>> +TRAILER:
>> +
>> +       H-byte HASH-checksum of all of the above.
>> +
>> --
>> 2.7.4
> Makes sense so far, I'll read on.
> I agree with Junio, that I could read this documentation without
> the urge to point out nits. :)
>
> Thanks,
> Stefan