Git very slow ?

Git very slow ?

[Ken Moffat]

Hi, please CC me if that is not your usual fashion, because I am not
subscribed.

I use git for my build scripts - those are accessed over nfs.  Since
I started using 2.1 and later (I don't think I used 2.0) commands
such as 'commit' take a long time before anything happens.  I
assumed that the newer version meant this would take longer.

But today^Wyesterday I was bisecting the kernel on a local
filesystem - even when the number of revisions left to test was in
the single digits, git bisect took a long time to decide which
revision should be the next one to test.  The filesystems are ext4.
Is this sort of delay normal now?

What really prompted me to ask is that I ran git blame on a script,
to see when I made a particular change so that I could add that
information to a ticket, and almost gave up waiting because it felt
as if it was taking for ever.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: Git very slow ?

[Kevin D]

On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
> Hi, please CC me if that is not your usual fashion, because I am not
> subscribed.
> 
> I use git for my build scripts - those are accessed over nfs.  Since
> I started using 2.1 and later (I don't think I used 2.0) commands
> such as 'commit' take a long time before anything happens.  I
> assumed that the newer version meant this would take longer.
> 
> But today^Wyesterday I was bisecting the kernel on a local
> filesystem - even when the number of revisions left to test was in
> the single digits, git bisect took a long time to decide which
> revision should be the next one to test.  The filesystems are ext4.
> Is this sort of delay normal now?
> 
> What really prompted me to ask is that I ran git blame on a script,
> to see when I made a particular change so that I could add that
> information to a ticket, and almost gave up waiting because it felt
> as if it was taking for ever.
> 

What kind of repository are we talking about? How many files, how big?
Git should not have become significantly slower recently.

Also, might there be anti-virus software that slows down file access?

Re: Git very slow ?

[David Kastrup]

Kevin D <me@ikke.info> writes:

> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
>> Hi, please CC me if that is not your usual fashion, because I am not
>> subscribed.
>> 
>> I use git for my build scripts - those are accessed over nfs.  Since
>> I started using 2.1 and later (I don't think I used 2.0) commands
>> such as 'commit' take a long time before anything happens.  I
>> assumed that the newer version meant this would take longer.
>> 
>> But today^Wyesterday I was bisecting the kernel on a local
>> filesystem - even when the number of revisions left to test was in
>> the single digits, git bisect took a long time to decide which
>> revision should be the next one to test.  The filesystems are ext4.
>> Is this sort of delay normal now?
>> 
>> What really prompted me to ask is that I ran git blame on a script,
>> to see when I made a particular change so that I could add that
>> information to a ticket, and almost gave up waiting because it felt
>> as if it was taking for ever.
>> 
>
> What kind of repository are we talking about? How many files, how big?
> Git should not have become significantly slower recently.

Particularly not git-blame in 2.1.  I should be quite surprised to see
any git-blame call running noticeably slower in 2.1 than in any
preceding version.

What may have happened is that the repository recently got repacked
aggressively and thus any access to older revisions got slower.
However, that change would be mostly tied to the repository rather than
the version of Git you access it with.

-- 
David Kastrup

Re: Git very slow ?

[Ken Moffat]

On Sun, Mar 08, 2015 at 04:51:36PM +0100, Kevin D wrote:
> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
> > Hi, please CC me if that is not your usual fashion, because I am not
> > subscribed.
> > 
> > I use git for my build scripts - those are accessed over nfs.  Since
> > I started using 2.1 and later (I don't think I used 2.0) commands
> > such as 'commit' take a long time before anything happens.  I
> > assumed that the newer version meant this would take longer.
> > 
> > But today^Wyesterday I was bisecting the kernel on a local
> > filesystem - even when the number of revisions left to test was in
> > the single digits, git bisect took a long time to decide which
> > revision should be the next one to test.  The filesystems are ext4.
> > Is this sort of delay normal now?
> > 
> > What really prompted me to ask is that I ran git blame on a script,
> > to see when I made a particular change so that I could add that
> > information to a ticket, and almost gave up waiting because it felt
> > as if it was taking for ever.
> > 
> 
> What kind of repository are we talking about? How many files, how big?
> Git should not have become significantly slower recently.
> 

The comments on git bisect were for linus'skernel tree, on a local
disk.  2.3GB of repo, just under 57000 files.

My own repo of build scripts, where I have noticed the delay before
git commit lets me type in the message, is an nfs v3 mount from
another of my machines in the same room - ping between them gives
times of 0.25 to 0.3 seconds and I think the nfs part is irrelevant.
Here, the size is 70MB and 12133 files [ about 1500 scripts total,
so the rest is from the commits ].

Some of this might be the drives - on the desktop with linus's tree
the machine only supports SATA2 (3GB/S), but the machine serving my
scripts goes back further and probably only supports SATA1 (1.5GB/S)

> Also, might there be anti-virus software that slows down file access?

No, this is all local access on linux machines.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: Git very slow ?

[Ken Moffat]

On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:
> Kevin D <me@ikke.info> writes:
> 
> > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
> >> Hi, please CC me if that is not your usual fashion, because I am not
> >> subscribed.
> >> 
> >> I use git for my build scripts - those are accessed over nfs.  Since
> >> I started using 2.1 and later (I don't think I used 2.0) commands
> >> such as 'commit' take a long time before anything happens.  I
> >> assumed that the newer version meant this would take longer.
> >> 
> >> But today^Wyesterday I was bisecting the kernel on a local
> >> filesystem - even when the number of revisions left to test was in
> >> the single digits, git bisect took a long time to decide which
> >> revision should be the next one to test.  The filesystems are ext4.
> >> Is this sort of delay normal now?
> >> 
> >> What really prompted me to ask is that I ran git blame on a script,
> >> to see when I made a particular change so that I could add that
> >> information to a ticket, and almost gave up waiting because it felt
> >> as if it was taking for ever.
> >> 
> >
> > What kind of repository are we talking about? How many files, how big?
> > Git should not have become significantly slower recently.
> 
> Particularly not git-blame in 2.1.  I should be quite surprised to see
> any git-blame call running noticeably slower in 2.1 than in any
> preceding version.
> 
> What may have happened is that the repository recently got repacked
> aggressively and thus any access to older revisions got slower.
> However, that change would be mostly tied to the repository rather than
> the version of Git you access it with.
> 
That is possible - well, not recently-recently, but I might have
repacked my repo of buildscripts some time last year.  Running
 ls -al .git
in that repository gives me:
drwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .
drwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..
-rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION
drwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches
-rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG
-rw-r--r--   1 ken 100    566 May 17  2014 config
-rw-r--r--   1 ken 100     73 May  1  2010 description
-rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache
-rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD
drwxr-xr-x   2 ken 100   4096 May  1  2010 hooks
-rw-r--r--   1 ken 100 218255 Mar  8 16:07 index
drwxr-xr-x   2 ken 100   4096 Sep 16  2013 info
drwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs
drwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects
-rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD
-rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs
drwxr-xr-x   5 ken 100   4096 May 20  2014 refs
-rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF

Running git blame on a script which dates back to when the repo was
created takes between 5 and 6 seconds to show the first screen,
running it on a file first created last August took about 3 seconds.
I have another small repo also on nfs with only a few files, started
last year, and there git blame takes about a second.

In my clone of linus' tree, running git blame scripts/headers.sh (a
random example,I do not normally run git blame there) took 10
seconds to return the first screen.

All of those timings are from preparing the comand, watching my
desktop clock until it changes the second, and hitting enter - so,
only accurate to approx one second.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: Git very slow ?

[David Kastrup]

Ken Moffat <zarniwhoop@ntlworld.com> writes:

> On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:
>
>> Particularly not git-blame in 2.1.  I should be quite surprised to see
>> any git-blame call running noticeably slower in 2.1 than in any
>> preceding version.
>> 
>> What may have happened is that the repository recently got repacked
>> aggressively and thus any access to older revisions got slower.
>> However, that change would be mostly tied to the repository rather than
>> the version of Git you access it with.
>> 
> That is possible - well, not recently-recently, but I might have
> repacked my repo of buildscripts some time last year.  Running
>  ls -al .git
> in that repository gives me:
> drwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .
> drwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..
> -rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION
> drwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches
> -rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG
> -rw-r--r--   1 ken 100    566 May 17  2014 config
> -rw-r--r--   1 ken 100     73 May  1  2010 description
> -rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache
> -rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD
> drwxr-xr-x   2 ken 100   4096 May  1  2010 hooks
> -rw-r--r--   1 ken 100 218255 Mar  8 16:07 index
> drwxr-xr-x   2 ken 100   4096 Sep 16  2013 info
> drwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs
> drwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects
> -rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD
> -rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs
> drwxr-xr-x   5 ken 100   4096 May 20  2014 refs
> -rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF
>
> Running git blame on a script which dates back to when the repo was
> created takes between 5 and 6 seconds to show the first screen,

Since git blame outputs everything once it is finished ("the first
screen" is purely the pager's business), it needs to unpack the entire
history of the file (unless no blameable lines remain at all) and look
at it.  6 seconds tends not to be all that excessive for extracting more
than 5 years of a file's history.

-- 
David Kastrup

Re: Git very slow ?

[Linus Torvalds]

On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
>
> The comments on git bisect were for linus'skernel tree, on a local
> disk.  2.3GB of repo, just under 57000 files.

Ugh. I hope you are talking about checked-out size.

    [torvalds@i7 linux]$ du -sh .git
   850M .git

because otherwise it sounds like that repo hasn't been repacked in forever.

To really pack things (which can slow things down for old history as
people said, but on the whole it tends to be a big win due to less
IO), do

   git repack -adf --window=200 --depth=200

and go away for a while. Oh, and make sure your machine has enough
memory and CPU to make that "for a while" not be *too* long.

You should have a few hundred files (just a few tens of files directly
after the repack) and that roughly 850MB of space for the repository
information itself.

But yeah, fully checked out and built with all the modules etc, you
would have much more.

That said, if you have something fairly that is consistently really
slow (like the "git commit" you mentioned), *before* doing the repack,
do

   strace -o ../trace-file -Ttt git commit

and we can get a much better guess about why it's so slow. Send it to
me in private email if you don't want to make it public, and I can
take a look.

> ping between them gives times of 0.25 to 0.3 seconds

.. and I *really* hope that was not seconds, but ms. Otherwise your
nfsv3 setup is going to be really really painful.

                          Linus

Re: Git very slow ?

[Linus Torvalds]

On Sun, Mar 8, 2015 at 12:37 PM, David Kastrup <dak@gnu.org> wrote:
>
> Since git blame outputs everything once it is finished ("the first
> screen" is purely the pager's business), it needs to unpack the entire
> history of the file (unless no blameable lines remain at all) and look
> at it.  6 seconds tends not to be all that excessive for extracting more
> than 5 years of a file's history.

Yeah, "git blame" can easily be several seconds without anything being wrong.

But "git commit" should be fairly instantaneous. Even over NFS.

That said, on NFS in particular, make sure you don't have

    [core]
        PreloadIndex = false

in your .gitconfig to disable the threaded index preloading.

But "core.preloadindex" _should_ be enabled by default in anything but
the most ancient git versions, and it can make a huge difference on
NFS because it allows the 'lstat()' calls to check that the index is
up-to-date to be done in parallel. Without that, git on NFS can be a
bit sluggish.

On local filesystems it normally doesn't make as much of a difference,
since things tend to be either cached or seek-limited.

                             Linus

Re: Git very slow ?

[Ken Moffat]

On Sun, Mar 08, 2015 at 12:39:10PM -0700, Linus Torvalds wrote:
> On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> >
> > The comments on git bisect were for linus'skernel tree, on a local
> > disk.  2.3GB of repo, just under 57000 files.
> 
> Ugh. I hope you are talking about checked-out size.
> 
>     [torvalds@i7 linux]$ du -sh .git
>    850M .git
> 
> because otherwise it sounds like that repo hasn't been repacked in forever.
> 

 Yes - I had finished bisecting, with the build still present.

> To really pack things (which can slow things down for old history as
> people said, but on the whole it tends to be a big win due to less
> IO), do
> 
>    git repack -adf --window=200 --depth=200
> 
> and go away for a while. Oh, and make sure your machine has enough
> memory and CPU to make that "for a while" not be *too* long.
> 

 For that, many thanks - this desktop has about 7GB (integrated
graphics steal a bit), current AMD desktop, and the repack of my
scripts repo took about 56 seconds.  I'll do that on my copy of
the kernel tomorrow (it's on another machine).

> You should have a few hundred files (just a few tens of files directly
> after the repack) and that roughly 850MB of space for the repository
> information itself.
> 
> But yeah, fully checked out and built with all the modules etc, you
> would have much more.
> 
> That said, if you have something fairly that is consistently really
> slow (like the "git commit" you mentioned), *before* doing the repack,
> do
> 
>    strace -o ../trace-file -Ttt git commit
> 
> and we can get a much better guess about why it's so slow. Send it to
> me in private email if you don't want to make it public, and I can
> take a look.

 I don't think you need to look - it was taking most of the time
(about 8 seconds) looping through the many files below .git/objects.
The trace was just over 9000 lines, repeating after the repack was
less than 1300 lines.  It's available (97K after using xz) if you
think it would be useful, but I think you have already diagnosed the
problem and solution.
> 
> > ping between them gives times of 0.25 to 0.3 seconds
> 
> .. and I *really* hope that was not seconds, but ms. Otherwise your
> nfsv3 setup is going to be really really painful.
> 
>                           Linus

 Yes, I'm not always good at knowing the right units.  Thanks for the
help.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.