Re: performance on repack

["Jon Smirl" <jonsmirl@gmail.com>]

On 8/13/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> > On 32b there's windowing code for accessing the packfile since we can
> > run out of address space, does this code get turned off for 64b?
>
> The windowing code you are talking about defaults as follows:
>
>   Parameter                  32b      64b
>   -----------------------------------------
>   core.packedGitWindowSize    32M     1G
>   core.packedGitLimit        256M     8G
>
> So I doubt you are having issues with the windowing code on a 64b
> system, unless your repository is just *huge*.  I did not think that
> anyone had a Git repository that exceeded 8G, though the window
> size of 1G might be a tad too small if there are many packfiles
> and they are each larger than 1G.

Why use windows on 64b? Default core.packedGitWindowSize equal to
core.packedGitLimit

I haven't measured it but I suspect the OS calls for moving the
windows are are quite slow on a relative basis since they have to
rewrite a bunch of page tables. Why is the window so small on 32b? I
thought we were up to about a 1GB packfile before running out of
address space with Mozilla. Shouldn't the window simply be set as
large as possible on 32b, this size being a function of the available
address space, not the amount of physical memory?


> > > * On the other hand, we could run all try_delta operations for one object
> > >   parallel. This way, we would need not very much more memory, but
> > >   require more synchronization (and more complex code).
> >
> > This solution was my first thought too. Use the main thread to get
> > everything needed for the object into RAM, then multi-thread the
> > compute bound, in-memory delta search operation. Shared CPU caches
> > might make this very fast.
>
> I have been thinking about doing this, especially now that the
> default window size is much larger.  I think the default is up as
> high as 50, which means we'd keep that shiny new UltraSPARC T2 busy.
> Not that I have one...  so anyone from Sun is welcome to send me
> one if they want.  ;-)
>
> I'm not sure its that complex to run all try_delta calls of the
> current window in parallel.  Might be a simple enough change that
> its actually worth the extra complexity, especially with these
> multi-core systems being so readily available.  Repacking is the
> most CPU intensive operation Git performs, and the one that is also
> the easiest to make parallel.
>
> Maybe someone else will beat me to it, but if not I might give such
> a patch a shot in a few weeks.

I'll test it. But my assignment this week is to figure out how to
program the BestComm DMA engine in a PowerPC chip. It's oh so much fun
trying to figure out how to program these "engines" that ASIC
designers love to build and never fully document.

>
> --
> Shawn.
>


-- 
Jon Smirl
jonsmirl@gmail.com