Re: [PATCH] RFC: git lazy clone proof-of-concept

[Nicolas Pitre <nico@cam.org>]

On Sun, 10 Feb 2008, Johannes Schindelin wrote:

> Hi,
> 
> On Sun, 10 Feb 2008, Johannes Schindelin wrote:
> 
> > On Sat, 9 Feb 2008, Nicolas Pitre wrote:
> > 
> > > On Sat, 9 Feb 2008, Jan Holesovsky wrote:
> > > 
> > > > On Friday 08 February 2008 20:00, Jakub Narebski wrote:
> > > > 
> > > > > Both Mozilla import, and GCC import were packed below 0.5 GB. 
> > > > > Warning: you would need machine with large amount of memory to 
> > > > > repack it tightly in sensible time!
> > > > 
> > > > As I answered elsewhere, unfortunately it goes out of memory even on 
> > > > 8G machine (x86-64), so...  But still trying.
> > > 
> > > Try setting the following config variables as follows:
> > > 
> > > 	git config pack.deltaCacheLimit 1
> > > 	git config pack.deltaCacheSize 1
> > > 	git config pack.windowMemory 1g
> > > 
> > > That should help keeping memory usage somewhat bounded.
> > 
> > I tried that:
> > 
> > $ git config pack.deltaCacheLimit 1
> > $ git config pack.deltaCacheSize 1
> > $ git config pack.windowMemory 2g
> > $ #/usr/bin/time git repack -a -d -f --window=250 --depth=250
> > $ du -s objects/
> > 2548137 objects/
> > $ /usr/bin/time git repack -a -d -f --window=250 --depth=250
> > Counting objects: 2477715, done.
> > fatal: Out of memory, malloc failed411764)
> > Command exited with non-zero status 1
> > 9356.95user 53.33system 2:38:58elapsed 98%CPU (0avgtext+0avgdata 
> > 0maxresident)k
> > 0inputs+0outputs (31929major+18088744minor)pagefaults 0swaps
> > 
> > Note that this is on a 2.4GHz Quadcode CPU with 3.5GB RAM.
> > 
> > I'm retrying with smaller values, but at over 2.5 hours per try, this is 
> > getting tedious.
> 
> Now, _that_ is strange.  Using 150 instead of 250 brings it down even 
> quicker!
> 
> $ /usr/bin/time git repack -a -d -f --window=150 --depth=150
> Counting objects: 2477715, done.
> Compressing objects:  19% (481551/2411764)
> Compressing objects:  19% (482333/2411764)
> fatal: Out of memory, malloc failed411764)
> Command exited with non-zero status 1
> 7118.37user 54.15system 2:01:44elapsed 98%CPU (0avgtext+0avgdata 
> 0maxresident)k
> 0inputs+0outputs (29834major+17122977minor)pagefaults 0swaps
> 
> (I hit the Return key twice during the time I suspected it would go out of 
> memory, so it might have been really at 20%.)
> 
> Ideas?

You're probably hitting the same memory allocator fragmentation issue I 
had with the gcc repo.  On my machine with 1GB of ram, I was able to 
repack the 1.5GB source pack just fine, but repacking the 300MB source 
pack was impossible due to memory exhaustion.

My theory is that the smaller pack has many more deltas with deeper 
delta chains, and this is stumping much harder on the memory allocator 
which fails to prevent fragmentation at some point.  When Jon Smirl 
tested Git using the Google memory allocator there was around 1GB less 
allocated, which might indicate that the glibc allocator has issues with 
some of Git's workloads.


Nicolas