Re: [PATCH] pack-format: correct multi-pack-index description

[Derrick Stolee <stolee@gmail.com>]

On 2/10/2020 9:22 AM, Johannes Berg wrote:
> On Mon, 2020-02-10 at 09:18 -0500, Derrick Stolee wrote:
>>
>> Thank you for finding this doc bug. This is a very subtle point,
>> and you have described it very clearly.
> 
> I was going back and forth on the wording a bit, glad I found something
> that you think is a good description :)
> 
> Are you familiar with the multi-pack-index and how it's used, by any
> chance?

Yes. I wrote the first version, and we use it a lot in VFS for Git.

> I came here from bup (https://github.com/bup/bup/) and needed a way to
> store the offset to find objects in "pure bup", today it only stores
> object *presence* and *pack* in its multi-index, but not the offset.
> 
> However, it seems to do a bit better in terms of not requiring a single
> multi-index, but instead storing it in midx-*.midx files and multiple
> can describe the repository state. Why wasn't something like that done
> for git as well? It's a bit annoying to have to recreate the full midx
> every time a pack file is added, and searching in two or three midx
> files wouldn't really be a big deal?

Part of my initial plan was to have this incremental file format.
The commit-graph uses a very similar mechanism. The difference may
be that you likely allow multiple .midx files found by scanning the
pack directory, but I would expect something like the
"commit-graph-chain" file that provides an ordered list of the
incremental files. This can be important for deciding when to merge
layers or delete old files, and would be critical to the possibility
of converting reachability bitmaps to rely on a stable object order
stored in the multi-pack-index instead of pack-order.

The reason the multi-pack-index has not become incremental is that
VFS for Git no longer needs to write it very often. We write the
entire multi-pack-index during a background job that triggers once
per day. If we needed to write it more frequently, then the incremental
format would be more important to us.

That said: if someone wanted to contribute an incremental format,
then I would be happy to review it!

> Anyway, that's just an aside, but during all this investigation I
> stumbled across this small inconsistency - I'm glad the docs exist at
> all! :-)

I'm glad they helped.

Thanks,
-Stolee