Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

[Ben Peart <peartben@gmail.com>]

On 11/20/2017 6:51 PM, Ramsay Jones wrote:
> 
> 
> On 20/11/17 14:01, Ben Peart wrote:
>> Further testing has revealed that switching from the regular heap to a refactored version of the mem_pool in fast-import.c produces similar gains as parallelizing do_index_load().  This appears to be a much simpler patch for similar gains so we will be pursuing that path.
>>
>> Combining the two patches resulted in a further 25% reduction of do_index_load() but with index load times getting smaller, that only resulted in a 3% improvement in total git command time.  Given the greater complexity of this patch, we've decided to put it on hold for now.  We'll keep it in our back pocket in case we need it in the future.
> 
> Since you are withdrawing the 'bp/fastindex' branch, I won't send the
> patch to fix up the sparse warnings (unless you would like to see it
> before you put it in your back pocket). ;-)
> 
> ATB,
> Ramsay Jones
> 
> 

I'm happy to take your patches as I am keeping a copy just in case we 
need to revisit it.

I'm actually interested to see how Peff's suggestion to use tcmalloc 
helps because nearly 90% of the remaining execution time was spent 
waiting for a lock in malloc due to all contention from the multi-threading.