Re: Figured out how to get Mozilla into git

[Nicolas Pitre <nico@cam.org>]

On Fri, 9 Jun 2006, Linus Torvalds wrote:

> 
> 
> On Fri, 9 Jun 2006, Jakub Narebski wrote:
> > Jon Smirl wrote:
> > 
> > >> git-repack -a -d but it OOMs on my 2GB+2GBswap machine :(
> > > 
> > > We are all having problems getting this to run on 32 bit machines with
> > > the 3-4GB process size limitations.
> > 
> > Is that expected (for 10GB repository if I remember correctly), or is there
> > some way to avoid this OOM?

What was that 10GB related to, exactly?  The original CVS repo, or the 
unpacked GIT repo?

> So a single 2GB pack is already very much pushing it. It's really really 
> hard to map in a 2GB file on a 32-bit platform: your VM is usually 
> fragmented enough that it simply isn't practical. In fact, I think the 
> limit for _practical_ usage of single packs is probably somewhere in the 
> half-gig region, unless you just have 64-bit machines.

Sure, but have we already reached that size?

The historic Linux repo currently repacks itself into a ~175MB pack for 
63428 commits.

The current Linux repo is ~103MB with a much shorter history (27153 
commits).

Given the above we can estimate the size of the kernel repository after 
x commits as follows:

	slope = (175 - 103) / (63428 - 27153) = approx 2KB per commit

	initial size = 175 - .001985 * 63428 = 49MB

So the initial kernel commit is about 49MB in size which is coherent 
with the corresponding compressed tarball.  Subsequent commits are 2KB 
in size on average.  Given that it will take about 233250 commits before 
the kernel reaches the half gigabyte pack file, and given the current 
commit rate (approx 23700 commits per year), that means we still have 
nearly 9 years to go.  And at that point 64-bit machines are likely to 
be the norm.

So given those numbers I don't think this is really an issue.  The Linux 
kernel is a rather huge and pretty active project to base comparisons 
against.  The Mozilla repository might be difficult to import and 
repack, but once repacked it should still be pretty usable now even on a 
32-bit machine even with a single pack.

Otherwise that should be quite easy to add a batch size argument to 
git-repack so git-rev-list and git-pack-objects are called multiple 
times with sequential commit 
ranges to create a repo with multiple packs.


Nicolas