Re: Something is broken in repack

["Jon Smirl" <jonsmirl@gmail.com>]

On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Fri, 7 Dec 2007, Jon Smirl wrote:
>
> > Using this config:
> > [pack]
> >         threads = 4
> >         deltacachesize = 256M
> >         deltacachelimit = 0
> >
> > And the 330MB gcc pack for input
> >  git repack -a -d -f  --depth=250 --window=250
> >
> > complete seconds RAM
> > 10%  47 1GB
> > 20%  29 1Gb
> > 30%  24 1Gb
> > 40%  18 1GB
> > 50%  110 1.2GB
> > 60%  85 1.4GB
> > 70%  195 1.5GB
> > 80%  186 2.5GB
> > 90%  489 3.8GB
> > 95%  800 4.8GB
> > I killed it because it started swapping
> >
> > The mmaps are only about 400MB in this case.
> > At the end the git process had 4.4GB of physical RAM allocated.
> >
> > Starting from a highly compressed pack greatly aggravates the problem.
> > Starting with a 2GB pack of the same data my process size only grew to
> > 3GB with 2GB of mmaps.
>
> You said having reproduced the issue, albeit not as severe, with the
> Linux kernel repo.  I did just that:
>
> # to get the default pack:
> $ git repack -a -f -d
>
> # first measurement with a repack from a default pack
> $ /usr/bin/time git repack -a -f --window=256 --depth=256
> 2572.17user 5.87system 22:46.80elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
> 15720inputs+356640outputs (71major+264376minor)pagefaults 0swaps
>
> # do it again to start from a highly packed pack
> $ /usr/bin/time git repack -a -f --window=256 --depth=256
> 2573.53user 5.62system 22:45.60elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
> 29176inputs+356664outputs (210major+274887minor)pagefaults 0swaps
>
> This is with pack.threads=2 on a P4 with HT, and I'm using the machine
> for other tasks as well, but all measured time is sensibly the same for
> both cases.  Virtual memory allocation never reached 700MB in both cases
> either.
>

This is the mail about the kernel pack, the one you quoted is a gcc run.

The kernel repo has the same problem but not nearly as bad.

Starting from a default pack
 git repack -a -d -f  --depth=1000 --window=1000
Uses 1GB of physical memory

Now do the command again.
 git repack -a -d -f  --depth=1000 --window=1000
Uses 1.3GB of physical memory

I suspect the gcc repo has much longer revision chains than the kernel
one since the kernel repo is only a few years old. The Mozilla repo
contained revision chains with over 2,000 revisions. Longer revision
chains result in longer delta chains.

So what is allocating the extra memory? Either a function of the
number of entries in the chain, or related to accessing the chain
since a chain with more entries will need to be accessed more times.

I have a 168MB kernel pack now after 15 minutes of four cores at 100%.

Here's another observation, the gcc objects are larger. Kernel has
650K objects in 190MB, gcc has 870K objects in 330MB. Average gcc
object is 30% larger. How should the average kernel developer
interpret this?



>
> Nicolas
>


-- 
Jon Smirl
jonsmirl@gmail.com