git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Linus Torvalds <torvalds@linux-foundation.org>
To: Brandon Casey <casey@nrlssc.navy.mil>
Cc: Alex Riesen <raa.lkml@gmail.com>,
	Bevan Watkiss <bevan.watkiss@cloakware.com>,
	Git Mailing List <git@vger.kernel.org>
Subject: Re:
Date: Fri, 8 May 2009 09:15:50 -0700 (PDT)	[thread overview]
Message-ID: <alpine.LFD.2.01.0905080857130.4983@localhost.localdomain> (raw)
In-Reply-To: <eFUCK0_CEtLa6Qvg6X1SqHmCgRnY3_3dy3OCJK26lGP-_kDRyWtlRA@cipher.nrlssc.navy.mil>



On Fri, 8 May 2009, Brandon Casey wrote:
> 
> plain 'git checkout' on linux kernel over NFS.

Thanks.

> Best time without patch: 1.20 seconds
> 
>   0.45user 0.71system 0:01.20elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (0major+15467minor)pagefaults 0swaps
> 
> Best time with patch (core.preloadindex = true): 1.10 seconds
> 
>   0.43user 4.00system 0:01.10elapsed 402%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (0major+13999minor)pagefaults 0swaps
> 
> Best time with patch (core.preloadindex = false): 0.84 seconds
> 
>   0.42user 0.39system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (0major+13965minor)pagefaults 0swaps

Ok, that is _disgusting_. The parallelism clearly works (402%CPU), but the 
system time overhead is horrible. Going from 0.39s system time to 4s of 
system time is really quite nasty.

Is there any possibility you could oprofile this (run it in a loop to get 
better profiles)? It very much sounds like some serious lock contention, 
and I'd love to hear more about exactly which lock it's hitting.

Also, you're already almost totally CPU-bound, with 96% CPU for the 
single-threaded csase. So you may be running over NFS, but your NFS server 
is likely pretty good and/or the client just captures everything in the 
caches anyway.

I don't recall what the Linux NFS stat cache timeout is, but it's less 
than a minute. I suspect that you ran things in a tight loop, which is why 
you then got effectively the local caching behavior for the best times. 

Can you do a "best time" check but with a 60-second pause between runs 
(and before), to see what happens when the client doesn't do caching?

> Best time with read_cache_preload patch only: 1.38 seconds
> 
>   0.45user 4.42system 0:01.38elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (0major+13990minor)pagefaults 0swaps

Yeah, here you're not getting any advantage of fewer lstats, and you 
show the same "almost entirely CPU-bound on four cores" behavior, and the 
same (probable) lock contention that has pushed the system time way up.

> The read_cache_preload() changes actually slow things down for me for this
> case.
> 
> Reduction in lstat's gives a nice 30% improvement.

Yes, I think the one-liner lstat avoidance is a real fix regardless. And 
the preloading sounds like it hits serialization overhead in the kernel, 
which I'm not at all surprised at, but not being surprised doesn't mean 
that I'm not interested to hear where it is.

The Linux VFS dcache itself should scale better than that (but who knows - 
cacheline ping-pong due to lock contention can easily cause a 10x slowdown 
even without being _totally_ contended all the time). So I would _suspect_ 
that it's some NFS lock that you're seeing, but I'd love to know more.

Btw, those system times are pretty high to begin with, so I'd love to know 
kernel version and see a profile even without the parallel case and 
presumably lock contention. Because while I probably have a faster 
machine anyway, what I see iis:

	[torvalds@nehalem linux]$ /usr/bin/time git checkout
	0.13user 0.05system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+13334minor)pagefaults 0swaps

ie my "system" time is _much_ lower than yours (and lower than your system 
time). This is the 'without patch' time, btw, so this has extra lstat's. 
And my system time is still lower than my user time, so I wonder where all 
_your_ system time comes from. Your system time is much more comparable to 
user time even in the good case, and I wonder why?

Could be just that kernel code tends to have more cache misses, and my 8MB 
cache captures it all, and yours doesn't. Regardless, a profile would be 
very interesting.

			Linus

  reply	other threads:[~2009-05-08 16:18 UTC|newest]

Thread overview: 84+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2009-05-07 17:01 (unknown), Bevan Watkiss
2009-05-07 17:13 ` Alex Riesen
2009-05-07 17:26   ` Bevan Watkiss
2009-05-07 18:18     ` Alex Riesen
2009-05-07 18:48       ` Bevan Watkiss
2009-05-07 19:56         ` Björn Steinbrink
2009-05-07 18:56     ` Linus Torvalds
2009-05-07 19:37       ` RE: Bevan Watkiss
2009-05-07 20:07         ` RE: Linus Torvalds
2009-05-07 20:20           ` RE: Linus Torvalds
2009-05-07 20:43             ` Junio C Hamano
2009-05-07 21:33               ` Re: Linus Torvalds
2009-05-07 21:55             ` Linus Torvalds
2009-05-07 22:27               ` RE: david
2009-05-07 22:36                 ` RE: Linus Torvalds
2009-05-07 22:43                   ` RE: david
2009-05-07 23:00                     ` RE: Linus Torvalds
2009-05-07 23:07                       ` RE: david
2009-05-07 23:18                         ` RE: Linus Torvalds
2009-05-07 23:31                           ` RE: david
2009-05-07 23:57                             ` Johan Herland
2009-05-08 16:14                             ` Bevan Watkiss
2009-05-08  8:17               ` Alex Riesen
2009-05-08 14:39                 ` Re: Linus Torvalds
2009-05-08 15:51                   ` Re: Brandon Casey
2009-05-08 16:15                     ` Linus Torvalds [this message]
2009-05-08 17:27                       ` Re: Brandon Casey
2009-05-08 17:43                         ` Re: Brandon Casey
2009-05-08 21:49                           ` Re: Linus Torvalds
2009-05-08 23:04                             ` Re: Brandon Casey
2009-05-09 16:44                               ` Re: Linus Torvalds
2009-05-08 17:44                         ` Re: Linus Torvalds
2009-05-08 16:47               ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik
2009-05-08 17:57                 ` Linus Torvalds
     [not found] <TXJgqLzlM6oCfTXKSqrSBk@txt.att.net>
2023-08-09  5:12 ` Luna Jernberg
     [not found] <20220301070226.2477769-1-jaydeepjd.8914>
2022-03-06 11:10 ` Jaydeep P Das
2022-03-06 11:22   ` Jaydeep Das
  -- strict thread matches above, loose matches on Subject: below --
2021-08-21 14:40 TECOB270_Ganesh Pawar
2021-08-21 23:52 ` Jeff King
2019-11-15 16:03 Martin Nicolay
2019-11-15 16:29 ` Martin Ågren
2019-11-15 16:37   ` Re: Martin Ågren
2019-08-20 17:23 William Baker
2019-08-20 17:27 ` Yagnatinsky, Mark
2019-03-05 14:57 [GSoC][PATCH v2 3/3] t3600: use helpers to replace test -d/f/e/s <path> Eric Sunshine
2019-03-05 23:38 ` Rohit Ashiwal
2019-01-23 10:50 Christopher Hagler
2019-01-23 14:16 ` Cody Kratzer
2019-01-23 14:25   ` Re: Thomas Braun
2019-01-23 16:00   ` Re: Christopher Hagler
2019-01-23 16:35     ` Randall S. Becker
2019-01-24 17:11   ` Johannes Schindelin
2018-10-08 13:33 Netravnen
2018-10-08 13:34 ` Inderpreet Saini
2018-04-27  0:54 [PATCH v3 2/3] merge: Add merge.renames config setting Ben Peart
2018-04-27 18:19 ` Elijah Newren
2018-04-30 13:11   ` Ben Peart
2018-04-30 16:12     ` Re: Elijah Newren
2018-05-02 14:33       ` Re: Ben Peart
2018-02-27  1:18 Alan Gage
2018-02-27 10:26 ` René Scharfe
2017-11-20 15:10 Viet Nguyen
2017-11-20 20:07 ` Stefan Beller
2017-11-12  2:21 hsed
2017-11-13 18:56 ` Stefan Beller
2017-01-25  0:11 [PATCH 7/7] completion: recognize more long-options Cornelius Weig
2017-01-25  0:21 ` Stefan Beller
2017-01-25  0:43   ` Cornelius Weig
2017-01-25  0:52     ` Re: Stefan Beller
2017-01-25  0:54   ` Re: Linus Torvalds
2017-01-25  1:32     ` Re: Eric Wong
2016-04-11 19:04 (unknown), miwilliams
2016-04-12  4:33 ` Stefan Beller
2015-08-19 19:41 Re: christain147
2015-08-19 11:09 Re: christain147
2015-08-05 12:47 (unknown) Ivan Chernyavsky
2015-08-15  9:19 ` Duy Nguyen
2015-08-17 17:49   ` Re: Junio C Hamano
2015-04-08 20:44 (unknown), Mamta Upadhyay
2015-04-08 21:58 ` Thomas Braun
2015-04-09 11:27   ` Re: Konstantin Khomoutov
     [not found] <CANSxx61FaNp5SBXJ8Y+pWn0eDcunmibKR5g8rttnWGdGwEMHCA@mail.gmail.com>
2015-03-18 20:45 ` Re: Junio C Hamano
2015-03-18 21:06   ` Re: Stefan Beller
2015-03-18 21:17     ` Re: Jeff King
2015-03-18 21:28       ` Re: Jeff King
2015-03-18 21:33         ` Re: Junio C Hamano
2015-03-18 21:45           ` Re: Stefan Beller
2015-03-13  1:34 (unknown) cody.taylor
2015-03-13  2:00 ` Duy Nguyen
2014-09-08 11:36 (unknown), R. Klomp
     [not found] ` <CAOqJoqGSRUw_UT4LhqpYX-WX6AEd2ReAWjgNS76Cra-SMKw3NQ@mail.gmail.com>
2014-09-08 14:36   ` R. Klomp
2014-09-10  0:00     ` Re: David Aguilar
2014-09-15 15:10       ` Re: R. Klomp
2014-02-06 11:54 "Sparse checkout leaves no entry on working directory" all the time on Windows 7 on Git 1.8.5.2.msysgit.0 konstunn
2014-02-06 13:20 ` Johannes Sixt
2014-02-06 19:56   ` Constantine Gorbunov
2012-06-12 21:12 (unknown), rohit sood
2012-06-12 23:51 ` Erik Faye-Lund
2009-11-18  5:03 Re: Anna
2009-05-11 18:57 (unknown) Don Slutz
2009-05-11 20:48 ` Johannes Schindelin
2009-05-12 12:45   ` Re: Don Slutz
2009-03-30  5:03 (unknown), David Aguilar
2009-03-30  7:02 ` Markus Heidelberg
2009-03-30  8:46   ` Re: Junio C Hamano
2007-11-01 20:44 (unknown), Francesco Pretto
2007-11-01 20:48 ` Francesco Pretto
2006-02-02  0:39 [RFC & PATCH] Solaris 8: ENOSYS when mkdir applied to automount., Jason Riedy
2006-02-02  4:18 ` H. Peter Anvin
2005-04-22 22:19 (unknown), atani
2005-04-22 23:16 ` Martin Schlemmer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.LFD.2.01.0905080857130.4983@localhost.localdomain \
    --to=torvalds@linux-foundation.org \
    --cc=bevan.watkiss@cloakware.com \
    --cc=casey@nrlssc.navy.mil \
    --cc=git@vger.kernel.org \
    --cc=raa.lkml@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).