Re: slow object packing during push

[David Turner <David.Turner@twosigma.com>]

On 3/9/21 9:14 PM, Ævar Arnfjörð Bjarmason wrote:
> On Tue, Mar 09 2021, David Turner wrote:
>
>> I have a large, funny repository that has very slow pushes unless I
>> set pack.usebitmaps=false to false.
> Good to see you on-list again! :)

Thanks!

>> First, a description of the repo: it's about 175GB, and was created by combining about 40,000 smaller repositories.  Historically, these repos were submodules of one meta repository[2].  I have stitched together the submodules, and this is the repository in which the stitching was done - that is, it contains all of the objects from the smaller repos, plus all of the objects from the meta repository, plus the newly-created trees & commits for the stitched repositories.  As new commits come into the meta repository (which have gitlinks to new submodule commits), we fetch from the meta repository (8s - it would be 2s if we were fetching into a normal clone without all of the other stuff), and the submodules (up to 10s per and embarrassingly parallel). Then we stitch (~0s), and push to the stitched "unity" repository (~2 minutes!!!).  The entire repo fits in RAM (yes, all 175G) and is in fact in the disk cache (I prewarmed the cache before testing anything).  
>>
>> The vast majority of the time appears to be spent in git pack-objects, and in particular in the stack trace in [1].  If I set pack.usebitmaps=false, the push only takes 10s.   This seems like pack bitmaps are a severe pessimization for my purposes.  This is true even immediately after a repack (that is, almost all of the objects are in one giant pack, except the newly-fetched ones).  I also tried setting up pack islands - one for each smaller repo, one for the stitched commits, and one for commits from the meta repo.  I'm not sure if this is necessary, but it's definitely not sufficient (my current config has it turned on, because I didn't feel like repacking again after testing it, and I tested it before testing pack.usebimaps). 
>> [snip]
> Without having carefully re-read it, I believe this issue is the same as
> what I reported here in 2019, and I think you'll find the resulting
> discussion intresting:
> https://lore.kernel.org/git/87zhoz8b9o.fsf@evledraar.gmail.com/
>
> Having skimmed it, I think you're probably omitting that this is a bare
> repo you're pushing from, and thus you're running into the combination
> of repack.writeBitmaps being true by default on bare repos, and
> pack.useBitmaps being true everywhere (but having no effect by default
> unless you have a bare repo, unless you manually make bitmaps blah
> blah).
>
> One of the semi-conclusions from the above thread was that we mostly
> turned this on thinking that bare ~= server that accepts pushes, and
> bitmaps are known to be worse (but not always!) for when you're pushing
> *from* your repo.

Thanks, that does explain it.  We should probably consider a default
that bitmaps aren't used on pushes.