Re: Something is broken in repack

[Nicolas Pitre <nico@cam.org>]

On Mon, 10 Dec 2007, Jon Smirl wrote:

> New run using same configuration. With the addition of the more
> efficient load balancing patches and delta cache accounting.
> 
> Seconds are wall clock time. They are lower since the patch made
> threading better at using all four cores. I am stuck at 380-390% CPU
> utilization for the git process.
> 
> complete seconds RAM
> 10%   60    900M (includes counting)
> 20%   15    900M
> 30%   15    900M
> 40%   50    1.2G
> 50%   80    1.3G
> 60%   70    1.7G
> 70%   140  1.8G
> 80%   180  2.0G
> 90%   280  2.2G
> 95%   530  2.8G - 1,420 total to here, previous was 1,983
> 100% 1390 2.85G
> During the writing phase RAM fell to 1.6G
> What is being freed in the writing phase??

The cached delta results, but you put a cap of 256MB for them.

Could you try again with that cache disabled entirely, with 
pack.deltacachesize = 1 (don't use 0 as that means unbounded).

And then, while still keeping the delta cache disabled, could you try 
with pack.threads = 2, and pack.threads = 1 ?

I'm sorry to ask you to do this but I don't have enough ram to even 
complete a repack with threads=2 so I'm reattempting single threaded at 
the moment.  But I really wonder if the threading has such an effect on 
memory usage.



> 
> I have no explanation for the change in RAM usage. Two guesses come to
> mind. Memory fragmentation. Or the change in the way the work was
> split up altered RAM usage.
> 
> Total CPU time was 195 minutes in 70 minutes clock time. About 70%
> efficient. During the compress phase all four cores were active until
> the last 90 seconds. Writing the objects took over 23 minutes CPU
> bound on one core.
> 
> New pack file is: 270,594,853
> Old one was: 344,543,752
> It still has 828,660 objects

You mean the pack for the gcc repo is now less than 300MB?  Wow.


Nicolas