Re: Git performance results on a large repository

[David Barr <davidbarr@google.com>]

On Sun, Feb 5, 2012 at 3:30 PM,  <david@lang.hm> wrote:
> On Fri, 3 Feb 2012, Joshua Redstone wrote:
>
>> The test repo has 4 million commits, linear history and about 1.3 million
>> files.  The size of the .git directory is about 15GB, and has been
>> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
>> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
>> lots of ram and flash).  The size of the index file is 191 MB.
>
>
> This may be a silly thought, but what if instead of one pack file of your
> entire history (4 million commits) you create multiple packs (say every half
> million commits) and mark all but the most recent pack as .keep (so that
> they won't be modified by a repack)
>
> that way things that only need to worry about recent history (blame, etc)
> will probably never have to go past the most recent pack file or two
>
> I may be wrong, but I think that when git is looking for 'similar files' for
> delta compression, it limits it's search to the current pack, so this will
> also keep you from searching the entire project history.

I don't know if there is an easy way to determine with the with the
current tools
in git but one useful statistic for tuning packing performance is the
size of the
largest component in the delta-chain graph. The significance of this number is
that the product of window-size and maximum depth need not be larger than it.
I've found that with some older repositories I could have a depth as low as 3
and still get good performance from a moderate window size.

--
David Barr