Re: Using bitmaps to accelerate fetch and clone

[Nguyen Thai Ngoc Duy <pclouds@gmail.com>]

On Thu, Sep 27, 2012 at 9:33 PM, Shawn Pearce <spearce@spearce.org> wrote:
>> I'd like to see some sort of extension mechanism like in
>> $GIT_DIR/index, so that we don't have to increase pack index version
>> often.
>
> This might be worthwhile. I dislike the way $GIT_DIR/index encodes
> extensions. Forcing an extension to fully materialize itself to
> determine its length so the length can be placed before the data is
> painful to work with when writing the file out to disk. I would prefer
> writing an index catalog at the trailer of the file. We already
> require random access to the index file, so its possible for a reader
> to read a fixed size trailer record that has the 2 SHA-1s we normally
> end an index with, and an extension catalog footer that has a length
> and CRC-32 of the catalog. The catalog would immediately appear before
> the footer, so a reader can find the start of the extension catalog by
> subtracting from the end of the file the catalog length and the file
> footer and catalog footer lengths. The catalog can then supply a
> starting offset for each extension section, and writers don't need to
> predict in advance how much data they need to store. Readers trying to
> use extensions aren't really hurt, Git already randomly seeks to read
> the tail of an index file to compare the pack SHA-1 before assuming
> the index is valid.

Yeah, that's exactly what I had in mind. But perhaps a separate file
(or files?) may be better. On that point, should all extensions be in
one new extra file, or one extension per file? I prefer all extensions
in one file, so we only need a single additional stat() for extension
check instead of probing for the entire pack-XXX.* range. In that
case, the catalog trailer idea still applies.
-- 
Duy