Re: Performance of "git gc..." is extremely bad in some cases

["brian m. carlson" <sandals@crustytoothpaste.net>]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

On 2021-03-08 at 22:29:16, Bryan Turner wrote:
> On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller <anthony@monospace.sh> wrote:
> >
> > What did you do before the bug happened? (Steps to reproduce your issue)
> >
> > git clone https://github.com/notracking/hosts-blocklists
> > cd hosts-blocklists
> > git reflog expire --all --expire=now && git gc --prune=now --aggressive
> 
> --aggressive tells git gc to discard all of its existing delta chains
> and go find new ones, and to be fairly aggressive in how it looks for
> candidates. This is going to be the primary source of the resource
> usage you see, as well as the time.
> 
> Aggressive GCs are something you do once in a (very great) while. If
> you try this without the --aggressive, how does it look?

I should point out that this repository is also rather pathologically
structured.  Almost every commit is an automatic commit updating the
same five files which are text files ranging from 5 MB to 11 MB.

When you use --aggressive, as Bryan pointed out, you're asking to throw
away all the deltas and try really hard to compute all of them fresh.
That's going to use a lot of memory because you're loading many large
text files into memory.  It's also going to use a lot of CPU because
these files do indeed delta extremely well, and since computing deltas
on larger files is more expensive, especially when there are many of
them.

And that's just the blobs.  The trees and commits are also going to be
nearly identically structured and will also delta well with virtually
every other similar object of their type.  Normally Git sorts by size
which helps pick better candidates, but since these are all going to be
identically sized, the performance is going to suffer.

Now, I have the advantage in this case of being a person who's sometimes
on call for the maintenance of Git repositories and in that capacity,
that this is pathologically structured is obvious to me.  But, yeah, I
would definitely not run --aggressive on this repo unless I needed to
and I would not expect it to perform well.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]