Re: performance on repack

[Nicolas Pitre <nico@cam.org>]

On Thu, 16 Aug 2007, David Kastrup wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > On Wed, 15 Aug 2007, Jon Smirl wrote:
> >
> >> On 8/15/07, Martin Koegler <mkoegler@auto.tuwien.ac.at> wrote:
> >> > git-pack-objects knows the order, in which it will use the objects.  A
> >> > seperate thread could pre-read the next object and wait until the main
> >> > thread starts processing it. After the read is complete, another
> >> > thread could start computing the delta index.
> >> 
> >> The hope is that the new adaptive read ahead code in the kernel will
> >> get this right and you won't need the second thread. Letting the
> >> kernel handle the read ahead will dynamically scale as other demands
> >> are made on the host. There's effectively only one read ahead cache in
> >> the system, only the kernel really knows how to divide it up between
> >> competing apps.
> >
> > No read ahead will ever help the delta search phase.
> 
> Well, the delta search phase consists of computing a delta index and
> then matching against it.

No, what I mean is what happens at a higher level where one object is 
deltified against several base object candidates to find the best match.  
Those several objects are presorted according to a combination of 
heuristics that makes their actual access completely random, hence no 
kernel read ahead might help here.

> If I understand correctly, delta indices
> for the search window are kept, and the current file is compared
> against them.  Locality might be better if just one delta index gets
> calculated and then compared with all _upcoming_ delta candidates in
> one go.

This appears so obvious that I attempted that a while ago already.

The idea turned up to be so complex to implement correctly and produced 
suboptimal results in practice that I abandoned it.

See http://marc.info/?l=git&m=114610715706599&w=2 for the details if 
you're interested.

PS: please at least CC me when replying to my mails


Nicolas