git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* git gc ineffective
@ 2020-11-09  9:20 Michal Suchánek
  2020-11-09  9:49 ` Andreas Schwab
  0 siblings, 1 reply; 8+ messages in thread
From: Michal Suchánek @ 2020-11-09  9:20 UTC (permalink / raw)
  To: Git List


Hello,

I am running git 2.29.1

I noticed I am running out of disk space, and one repository taking up
about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
and git would report it does gc after every commit. Did gc without
--aggressive, and the used space almost *doubled* to 75G (about 1G is
checkout and some utracked files).

The expected size is about 20G. Is there some way to debug gc to figure
out what went wrong?

Thanks

Michal

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09  9:20 git gc ineffective Michal Suchánek
@ 2020-11-09  9:49 ` Andreas Schwab
  2020-11-09 10:17   ` Michal Suchánek
  0 siblings, 1 reply; 8+ messages in thread
From: Andreas Schwab @ 2020-11-09  9:49 UTC (permalink / raw)
  To: Michal Suchánek; +Cc: Git List

On Nov 09 2020, Michal Suchánek wrote:

> I noticed I am running out of disk space, and one repository taking up
> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> and git would report it does gc after every commit.

Do you have a lot of loose objects?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09  9:49 ` Andreas Schwab
@ 2020-11-09 10:17   ` Michal Suchánek
  2020-11-09 10:43     ` Michal Suchánek
  0 siblings, 1 reply; 8+ messages in thread
From: Michal Suchánek @ 2020-11-09 10:17 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Git List

On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > I noticed I am running out of disk space, and one repository taking up
> > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > and git would report it does gc after every commit.
> 
> Do you have a lot of loose objects?
{ for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
53392

> 
> Andreas.
> 
> -- 
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09 10:17   ` Michal Suchánek
@ 2020-11-09 10:43     ` Michal Suchánek
  2020-11-09 10:48       ` Michal Suchánek
  2020-11-09 11:17       ` Andreas Schwab
  0 siblings, 2 replies; 8+ messages in thread
From: Michal Suchánek @ 2020-11-09 10:43 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Git List

On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > On Nov 09 2020, Michal Suchánek wrote:
> > 
> > > I noticed I am running out of disk space, and one repository taking up
> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > and git would report it does gc after every commit.
> > 
> > Do you have a lot of loose objects?
> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> 53392
And in the double-size repository it's doubled, too:
 { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
 101167

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09 10:43     ` Michal Suchánek
@ 2020-11-09 10:48       ` Michal Suchánek
  2020-11-09 11:17       ` Andreas Schwab
  1 sibling, 0 replies; 8+ messages in thread
From: Michal Suchánek @ 2020-11-09 10:48 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Git List

On Mon, Nov 09, 2020 at 11:43:27AM +0100, Michal Suchánek wrote:
> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> > On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> > > On Nov 09 2020, Michal Suchánek wrote:
> > > 
> > > > I noticed I am running out of disk space, and one repository taking up
> > > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> > > > and git would report it does gc after every commit.
> > > 
> > > Do you have a lot of loose objects?
> > { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> > 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167
And server-side bare repository has 16G with 30836 loose objects and git
2.26.2.

Thanks

Michal

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09 10:43     ` Michal Suchánek
  2020-11-09 10:48       ` Michal Suchánek
@ 2020-11-09 11:17       ` Andreas Schwab
  2021-01-07 18:35         ` Michal Suchánek
  1 sibling, 1 reply; 8+ messages in thread
From: Andreas Schwab @ 2020-11-09 11:17 UTC (permalink / raw)
  To: Michal Suchánek; +Cc: Git List

On Nov 09 2020, Michal Suchánek wrote:

> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>> > On Nov 09 2020, Michal Suchánek wrote:
>> > 
>> > > I noticed I am running out of disk space, and one repository taking up
>> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>> > > and git would report it does gc after every commit.
>> > 
>> > Do you have a lot of loose objects?
>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>> 53392
> And in the double-size repository it's doubled, too:
>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>  101167

git count-objects also shows the size.

Does it help to prune them --exprire now?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2020-11-09 11:17       ` Andreas Schwab
@ 2021-01-07 18:35         ` Michal Suchánek
  2021-01-07 21:48           ` Johannes Sixt
  0 siblings, 1 reply; 8+ messages in thread
From: Michal Suchánek @ 2021-01-07 18:35 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Git List

Hello,

On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
> On Nov 09 2020, Michal Suchánek wrote:
> 
> > On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
> >> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
> >> > On Nov 09 2020, Michal Suchánek wrote:
> >> > 
> >> > > I noticed I am running out of disk space, and one repository taking up
> >> > > about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
> >> > > and git would report it does gc after every commit.
> >> > 
> >> > Do you have a lot of loose objects?
> >> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >> 53392
> > And in the double-size repository it's doubled, too:
> >  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
> >  101167
> 
> git count-objects also shows the size.
$ git count-objects
59853 objects, 43249880 kilobytes
$ du -hs .git
48G     .git
$ git gc --aggressive
Enumerating objects: 1825080, done.
Counting objects: 100% (1825080/1825080), done.
Delta compression using up to 4 threads
Compressing objects: 100% (1803925/1803925), done.
Writing objects: 100% (1825080/1825080), done.
Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0
Removing duplicate objects: 100% (256/256), done.
Checking connectivity: 2003814, done.
Expanding reachable commits in commit graph: 337512, done.
$ du -hs .git
172G    .git
$ git count-objects
178734 objects, 175309572 kilobytes

> Does it help to prune them --exprire now?

$ git prune
Checking connectivity: 1825478, done.
$ du -hs .git
3.9G    .git
$ git --version
git version 2.26.2

So it is my wrong expectation that 'gc' comand removes garbage. It
creates it en masse.

It just makes is in a way that the 'prune' command that really reoves
garbage can now remove it.

Thanks

Michal

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: git gc ineffective
  2021-01-07 18:35         ` Michal Suchánek
@ 2021-01-07 21:48           ` Johannes Sixt
  0 siblings, 0 replies; 8+ messages in thread
From: Johannes Sixt @ 2021-01-07 21:48 UTC (permalink / raw)
  To: Michal Suchánek; +Cc: Git List, Andreas Schwab

Am 07.01.21 um 19:35 schrieb Michal Suchánek:
> Hello,
> 
> On Mon, Nov 09, 2020 at 12:17:57PM +0100, Andreas Schwab wrote:
>> On Nov 09 2020, Michal Suchánek wrote:
>>
>>> On Mon, Nov 09, 2020 at 11:17:38AM +0100, Michal Suchánek wrote:
>>>> On Mon, Nov 09, 2020 at 10:49:21AM +0100, Andreas Schwab wrote:
>>>>> On Nov 09 2020, Michal Suchánek wrote:
>>>>>
>>>>>> I noticed I am running out of disk space, and one repository taking up
>>>>>> about 38G. Did git gc --aggressive, and the used space *raised* to 42G,
>>>>>> and git would report it does gc after every commit.
>>>>>
>>>>> Do you have a lot of loose objects?
>>>> { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>> 53392
>>> And in the double-size repository it's doubled, too:
>>>  { for i in  .git/objects/?? ; do ls $i ; done ; } | wc -l
>>>  101167
>>
>> git count-objects also shows the size.
> $ git count-objects
> 59853 objects, 43249880 kilobytes
> $ du -hs .git
> 48G     .git
> $ git gc --aggressive
> Enumerating objects: 1825080, done.
> Counting objects: 100% (1825080/1825080), done.
> Delta compression using up to 4 threads
> Compressing objects: 100% (1803925/1803925), done.
> Writing objects: 100% (1825080/1825080), done.
> Total 1825080 (delta 1234005), reused 587969 (delta 0), pack-reused 0
> Removing duplicate objects: 100% (256/256), done.
> Checking connectivity: 2003814, done.
> Expanding reachable commits in commit graph: 337512, done.
> $ du -hs .git
> 172G    .git
> $ git count-objects
> 178734 objects, 175309572 kilobytes
> 
>> Does it help to prune them --exprire now?
> 
> $ git prune
> Checking connectivity: 1825478, done.
> $ du -hs .git
> 3.9G    .git
> $ git --version
> git version 2.26.2
> 
> So it is my wrong expectation that 'gc' comand removes garbage. It
> creates it en masse.
> 
> It just makes is in a way that the 'prune' command that really reoves
> garbage can now remove it.

It's an unfortunate default behavior of `git gc`. Set gc.pruneExpire to
'now' to countermand it, but watch out for the caveats.

See
https://stackoverflow.com/questions/55414916/how-to-avoid-that-git-gc-generates-garbage-loose-objects
for more details.

-- Hannes

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-01-07 21:52 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-09  9:20 git gc ineffective Michal Suchánek
2020-11-09  9:49 ` Andreas Schwab
2020-11-09 10:17   ` Michal Suchánek
2020-11-09 10:43     ` Michal Suchánek
2020-11-09 10:48       ` Michal Suchánek
2020-11-09 11:17       ` Andreas Schwab
2021-01-07 18:35         ` Michal Suchánek
2021-01-07 21:48           ` Johannes Sixt

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).