Re: Performance of "git gc..." is extremely bad in some cases

[Anthony Muller <anthony@monospace.sh>]

Thank you Brian and Bryan. You both clarified what was happening and now I know what to look for.

I can use a shallow clone for most repos, but there are some I want to keep history for. I don't need a full copy of this repo, but it was a good repo to show the issue I was facing.

Thanks again!


 ---- On Mon, 08 Mar 2021 23:56:53 +0000 brian m. carlson <sandals@crustytoothpaste.net> wrote ----
 > On 2021-03-08 at 22:29:16, Bryan Turner wrote:
 > > On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller <anthony@monospace.sh> wrote:
 > > >
 > > > What did you do before the bug happened? (Steps to reproduce your issue)
 > > >
 > > > git clone https://github.com/notracking/hosts-blocklists
 > > > cd hosts-blocklists
 > > > git reflog expire --all --expire=now && git gc --prune=now --aggressive
 > > 
 > > --aggressive tells git gc to discard all of its existing delta chains
 > > and go find new ones, and to be fairly aggressive in how it looks for
 > > candidates. This is going to be the primary source of the resource
 > > usage you see, as well as the time.
 > > 
 > > Aggressive GCs are something you do once in a (very great) while. If
 > > you try this without the --aggressive, how does it look?
 > 
 > I should point out that this repository is also rather pathologically
 > structured.  Almost every commit is an automatic commit updating the
 > same five files which are text files ranging from 5 MB to 11 MB.
 > 
 > When you use --aggressive, as Bryan pointed out, you're asking to throw
 > away all the deltas and try really hard to compute all of them fresh.
 > That's going to use a lot of memory because you're loading many large
 > text files into memory.  It's also going to use a lot of CPU because
 > these files do indeed delta extremely well, and since computing deltas
 > on larger files is more expensive, especially when there are many of
 > them.
 > 
 > And that's just the blobs.  The trees and commits are also going to be
 > nearly identically structured and will also delta well with virtually
 > every other similar object of their type.  Normally Git sorts by size
 > which helps pick better candidates, but since these are all going to be
 > identically sized, the performance is going to suffer.
 > 
 > Now, I have the advantage in this case of being a person who's sometimes
 > on call for the maintenance of Git repositories and in that capacity,
 > that this is pathologically structured is obvious to me.  But, yeah, I
 > would definitely not run --aggressive on this repo unless I needed to
 > and I would not expect it to perform well.
 > -- 
 > brian m. carlson (he/him or they/them)
 > Houston, Texas, US
 >