git gc ineffective

git gc ineffective

[Michal Suchánek]

Hello,

I am running git 2.29.1

I noticed I am running out of disk space, and one repository taking up
about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
and git would report it does gc after every commit. Did gc without
--aggressive, and the used space almost *doubled* to 75G (about 1G is
checkout and some utracked files).

The expected size is about 20G. Is there some way to debug gc to figure
out what went wrong?

Thanks

Michal

Re: git gc ineffective

[Andreas Schwab]

On Nov 09 2020, Michal Suchánek wrote:

> I noticed I am running out of disk space, and one repository taking up
> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> and git would report it does gc after every commit.

Do you have a lot of loose objects?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: git gc ineffective

[Michal Suchánek]

On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > I noticed I am running out of disk space, and one repository taking up
> > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > and git would report it does gc after every commit.
> 
> Do you have a lot of loose objects?
{ for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
53392

> 
> Andreas.
> 
> -- 
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."

Re: git gc ineffective

[Michal Suchánek]

On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > On Nov 09 2020, Michal Suchánek wrote:
> > 
> > > I noticed I am running out of disk space, and one repository taking up
> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > and git would report it does gc after every commit.
> > 
> > Do you have a lot of loose objects?
> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> 53392
And in the double-size repository it's doubled, too:
 { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
 101167

Re: git gc ineffective

[Michal Suchánek]

On Mon, Nov 09, 2020 at 11:43:27AM +0100, Michal Suchánek wrote:
> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > > On Nov 09 2020, Michal Suchánek wrote:
> > > 
> > > > I noticed I am running out of disk space, and one repository taking up
> > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > > and git would report it does gc after every commit.
> > > 
> > > Do you have a lot of loose objects?
> > { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> > 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167
And server-side bare repository has 16G with 30836 loose objects and git
2.26.2.

Thanks

Michal

Re: git gc ineffective

[Andreas Schwab]

On Nov 09 2020, Michal Suchánek wrote:

> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>> > On Nov 09 2020, Michal Suchánek wrote:
>> > 
>> > > I noticed I am running out of disk space, and one repository taking up
>> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>> > > and git would report it does gc after every commit.
>> > 
>> > Do you have a lot of loose objects?
>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>> 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167

git count-objects also shows the size.

Does it help to prune them --exprire now?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: git gc ineffective

[Michal Suchánek]

Hello,

On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> >> > On Nov 09 2020, Michal Suchánek wrote:
> >> > 
> >> > > I noticed I am running out of disk space, and one repository taking up
> >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> >> > > and git would report it does gc after every commit.
> >> > 
> >> > Do you have a lot of loose objects?
> >> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >> 53392
> > And in the double-size repository it's doubled, too:
> >  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >  101167
> 
> git count-objects also shows the size.
$ git count-objects
59853 objects, 43249880 kilobytes
$ du -hs .git
48G     .git
$ git gc --aggressive
Enumerating objects: 1825080, done.
Counting objects: 100% (1825080/1825080), done.
Delta compression using up to 4 threads
Compressing objects: 100% (1803925/1803925), done.
Writing objects: 100% (1825080/1825080), done.
Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0
Removing duplicate objects: 100% (256/256), done.
Checking connectivity: 2003814, done.
Expanding reachable commits in commit graph: 337512, done.
$ du -hs .git
172G    .git
$ git count-objects
178734 objects, 175309572 kilobytes

> Does it help to prune them --exprire now?

$ git prune
Checking connectivity: 1825478, done.
$ du -hs .git
3.9G    .git
$ git --version
git version 2.26.2

So it is my wrong expectation that 'gc' comand removes garbage. It
creates it en masse.

It just makes is in a way that the 'prune' command that really reoves
garbage can now remove it.

Thanks

Michal

Re: git gc ineffective

[Johannes Sixt]

Am 07.01.21 um 19:35 schrieb Michal Suchánek:
> Hello,
> 
> On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
>> On Nov 09 2020, Michal Suchánek wrote:
>>
>>> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>>>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>>>>> On Nov 09 2020, Michal Suchánek wrote:
>>>>>
>>>>>> I noticed I am running out of disk space, and one repository taking up
>>>>>> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>>>>>> and git would report it does gc after every commit.
>>>>>
>>>>> Do you have a lot of loose objects?
>>>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>> 53392
>>> And in the double-size repository it's doubled, too:
>>>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>  101167
>>
>> git count-objects also shows the size.
> $ git count-objects
> 59853 objects, 43249880 kilobytes
> $ du -hs .git
> 48G     .git
> $ git gc --aggressive
> Enumerating objects: 1825080, done.
> Counting objects: 100% (1825080/1825080), done.
> Delta compression using up to 4 threads
> Compressing objects: 100% (1803925/1803925), done.
> Writing objects: 100% (1825080/1825080), done.
> Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0
> Removing duplicate objects: 100% (256/256), done.
> Checking connectivity: 2003814, done.
> Expanding reachable commits in commit graph: 337512, done.
> $ du -hs .git
> 172G    .git
> $ git count-objects
> 178734 objects, 175309572 kilobytes
> 
>> Does it help to prune them --exprire now?
> 
> $ git prune
> Checking connectivity: 1825478, done.
> $ du -hs .git
> 3.9G    .git
> $ git --version
> git version 2.26.2
> 
> So it is my wrong expectation that 'gc' comand removes garbage. It
> creates it en masse.
> 
> It just makes is in a way that the 'prune' command that really reoves
> garbage can now remove it.

It's an unfortunate default behavior of `git gc`. Set gc.pruneExpire to
'now' to countermand it, but watch out for the caveats.

See
https://stackoverflow.com/questions/55414916/how-to-avoid-that-git-gc-generates-garbage-loose-objects
for more details.

-- Hannes