Re: Something is broken in repack

["Jon Smirl" <jonsmirl@gmail.com>]

On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > New run using same configuration. With the addition of the more
> > efficient load balancing patches and delta cache accounting.
> >
> > Seconds are wall clock time. They are lower since the patch made
> > threading better at using all four cores. I am stuck at 380-390% CPU
> > utilization for the git process.
> >
> > complete seconds RAM
> > 10%   60    900M (includes counting)
> > 20%   15    900M
> > 30%   15    900M
> > 40%   50    1.2G
> > 50%   80    1.3G
> > 60%   70    1.7G
> > 70%   140  1.8G
> > 80%   180  2.0G
> > 90%   280  2.2G
> > 95%   530  2.8G - 1,420 total to here, previous was 1,983
> > 100% 1390 2.85G
> > During the writing phase RAM fell to 1.6G
> > What is being freed in the writing phase??
>
> The cached delta results, but you put a cap of 256MB for them.
>
> Could you try again with that cache disabled entirely, with
> pack.deltacachesize = 1 (don't use 0 as that means unbounded).
>
> And then, while still keeping the delta cache disabled, could you try
> with pack.threads = 2, and pack.threads = 1 ?
>
> I'm sorry to ask you to do this but I don't have enough ram to even
> complete a repack with threads=2 so I'm reattempting single threaded at
> the moment.  But I really wonder if the threading has such an effect on
> memory usage.

I already have a threads = 1 running with this config. Binary and
config were same from threads=4 run.

10% 28min 950M
40% 135min 950M
50% 157min 900M
60% 160min 830M
100% 170min 830M

Something is hurting bad with threads. 170 CPU minutes with one
thread, versus 195 CPU minutes with four threads.

Is there a different memory allocator that can be used when
multithreaded on gcc? This whole problem may be coming from the memory
allocation function. git is hardly interacting at all on the thread
level so it's likely a problem in the C run-time.

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[pack]
        threads = 1
        deltacachesize = 256M
        windowmemory = 256M
        deltacachelimit = 0
[remote "origin"]
        url = git://git.infradead.org/gcc.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "trunk"]
        remote = origin
        merge = refs/heads/trunk




>
>
>
> >
> > I have no explanation for the change in RAM usage. Two guesses come to
> > mind. Memory fragmentation. Or the change in the way the work was
> > split up altered RAM usage.
> >
> > Total CPU time was 195 minutes in 70 minutes clock time. About 70%
> > efficient. During the compress phase all four cores were active until
> > the last 90 seconds. Writing the objects took over 23 minutes CPU
> > bound on one core.
> >
> > New pack file is: 270,594,853
> > Old one was: 344,543,752
> > It still has 828,660 objects
>
> You mean the pack for the gcc repo is now less than 300MB?  Wow.
>
>
> Nicolas
>


-- 
Jon Smirl
jonsmirl@gmail.com