From: Linus Torvalds <torvalds@linux-foundation.org>
To: Brandon Casey <casey@nrlssc.navy.mil>
Cc: Alex Riesen <raa.lkml@gmail.com>,
Bevan Watkiss <bevan.watkiss@cloakware.com>,
Git Mailing List <git@vger.kernel.org>
Subject: Re:
Date: Fri, 8 May 2009 09:15:50 -0700 (PDT) [thread overview]
Message-ID: <alpine.LFD.2.01.0905080857130.4983@localhost.localdomain> (raw)
In-Reply-To: <eFUCK0_CEtLa6Qvg6X1SqHmCgRnY3_3dy3OCJK26lGP-_kDRyWtlRA@cipher.nrlssc.navy.mil>
On Fri, 8 May 2009, Brandon Casey wrote:
>
> plain 'git checkout' on linux kernel over NFS.
Thanks.
> Best time without patch: 1.20 seconds
>
> 0.45user 0.71system 0:01.20elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (0major+15467minor)pagefaults 0swaps
>
> Best time with patch (core.preloadindex = true): 1.10 seconds
>
> 0.43user 4.00system 0:01.10elapsed 402%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (0major+13999minor)pagefaults 0swaps
>
> Best time with patch (core.preloadindex = false): 0.84 seconds
>
> 0.42user 0.39system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (0major+13965minor)pagefaults 0swaps
Ok, that is _disgusting_. The parallelism clearly works (402%CPU), but the
system time overhead is horrible. Going from 0.39s system time to 4s of
system time is really quite nasty.
Is there any possibility you could oprofile this (run it in a loop to get
better profiles)? It very much sounds like some serious lock contention,
and I'd love to hear more about exactly which lock it's hitting.
Also, you're already almost totally CPU-bound, with 96% CPU for the
single-threaded csase. So you may be running over NFS, but your NFS server
is likely pretty good and/or the client just captures everything in the
caches anyway.
I don't recall what the Linux NFS stat cache timeout is, but it's less
than a minute. I suspect that you ran things in a tight loop, which is why
you then got effectively the local caching behavior for the best times.
Can you do a "best time" check but with a 60-second pause between runs
(and before), to see what happens when the client doesn't do caching?
> Best time with read_cache_preload patch only: 1.38 seconds
>
> 0.45user 4.42system 0:01.38elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (0major+13990minor)pagefaults 0swaps
Yeah, here you're not getting any advantage of fewer lstats, and you
show the same "almost entirely CPU-bound on four cores" behavior, and the
same (probable) lock contention that has pushed the system time way up.
> The read_cache_preload() changes actually slow things down for me for this
> case.
>
> Reduction in lstat's gives a nice 30% improvement.
Yes, I think the one-liner lstat avoidance is a real fix regardless. And
the preloading sounds like it hits serialization overhead in the kernel,
which I'm not at all surprised at, but not being surprised doesn't mean
that I'm not interested to hear where it is.
The Linux VFS dcache itself should scale better than that (but who knows -
cacheline ping-pong due to lock contention can easily cause a 10x slowdown
even without being _totally_ contended all the time). So I would _suspect_
that it's some NFS lock that you're seeing, but I'd love to know more.
Btw, those system times are pretty high to begin with, so I'd love to know
kernel version and see a profile even without the parallel case and
presumably lock contention. Because while I probably have a faster
machine anyway, what I see iis:
[torvalds@nehalem linux]$ /usr/bin/time git checkout
0.13user 0.05system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+13334minor)pagefaults 0swaps
ie my "system" time is _much_ lower than yours (and lower than your system
time). This is the 'without patch' time, btw, so this has extra lstat's.
And my system time is still lower than my user time, so I wonder where all
_your_ system time comes from. Your system time is much more comparable to
user time even in the good case, and I wonder why?
Could be just that kernel code tends to have more cache misses, and my 8MB
cache captures it all, and yours doesn't. Regardless, a profile would be
very interesting.
Linus
next prev parent reply other threads:[~2009-05-08 16:18 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-05-07 17:01 (unknown), Bevan Watkiss
2009-05-07 17:13 ` Alex Riesen
2009-05-07 17:26 ` Bevan Watkiss
2009-05-07 18:18 ` Alex Riesen
2009-05-07 18:48 ` Bevan Watkiss
2009-05-07 19:56 ` Björn Steinbrink
2009-05-07 18:56 ` Linus Torvalds
2009-05-07 19:37 ` RE: Bevan Watkiss
2009-05-07 20:07 ` RE: Linus Torvalds
2009-05-07 20:20 ` RE: Linus Torvalds
2009-05-07 20:43 ` Junio C Hamano
2009-05-07 21:33 ` Re: Linus Torvalds
2009-05-07 21:55 ` Linus Torvalds
2009-05-07 22:27 ` RE: david
2009-05-07 22:36 ` RE: Linus Torvalds
2009-05-07 22:43 ` RE: david
2009-05-07 23:00 ` RE: Linus Torvalds
2009-05-07 23:07 ` RE: david
2009-05-07 23:18 ` RE: Linus Torvalds
2009-05-07 23:31 ` RE: david
2009-05-07 23:57 ` Johan Herland
2009-05-08 16:14 ` Bevan Watkiss
2009-05-08 8:17 ` Alex Riesen
2009-05-08 14:39 ` Re: Linus Torvalds
2009-05-08 15:51 ` Re: Brandon Casey
2009-05-08 16:15 ` Linus Torvalds [this message]
2009-05-08 17:27 ` Re: Brandon Casey
2009-05-08 17:43 ` Re: Brandon Casey
2009-05-08 21:49 ` Re: Linus Torvalds
2009-05-08 23:04 ` Re: Brandon Casey
2009-05-09 16:44 ` Re: Linus Torvalds
2009-05-08 17:44 ` Re: Linus Torvalds
2009-05-08 16:47 ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik
2009-05-08 17:57 ` Linus Torvalds
[not found] <TXJgqLzlM6oCfTXKSqrSBk@txt.att.net>
2023-08-09 5:12 ` Luna Jernberg
[not found] <20220301070226.2477769-1-jaydeepjd.8914>
2022-03-06 11:10 ` Jaydeep P Das
2022-03-06 11:22 ` Jaydeep Das
-- strict thread matches above, loose matches on Subject: below --
2021-08-21 14:40 TECOB270_Ganesh Pawar
2021-08-21 23:52 ` Jeff King
2019-11-15 16:03 Martin Nicolay
2019-11-15 16:29 ` Martin Ågren
2019-11-15 16:37 ` Re: Martin Ågren
2019-08-20 17:23 William Baker
2019-08-20 17:27 ` Yagnatinsky, Mark
2019-03-05 14:57 [GSoC][PATCH v2 3/3] t3600: use helpers to replace test -d/f/e/s <path> Eric Sunshine
2019-03-05 23:38 ` Rohit Ashiwal
2019-01-23 10:50 Christopher Hagler
2019-01-23 14:16 ` Cody Kratzer
2019-01-23 14:25 ` Re: Thomas Braun
2019-01-23 16:00 ` Re: Christopher Hagler
2019-01-23 16:35 ` Randall S. Becker
2019-01-24 17:11 ` Johannes Schindelin
2018-10-08 13:33 Netravnen
2018-10-08 13:34 ` Inderpreet Saini
2018-04-27 0:54 [PATCH v3 2/3] merge: Add merge.renames config setting Ben Peart
2018-04-27 18:19 ` Elijah Newren
2018-04-30 13:11 ` Ben Peart
2018-04-30 16:12 ` Re: Elijah Newren
2018-05-02 14:33 ` Re: Ben Peart
2018-02-27 1:18 Alan Gage
2018-02-27 10:26 ` René Scharfe
2017-11-20 15:10 Viet Nguyen
2017-11-20 20:07 ` Stefan Beller
2017-11-12 2:21 hsed
2017-11-13 18:56 ` Stefan Beller
2017-01-25 0:11 [PATCH 7/7] completion: recognize more long-options Cornelius Weig
2017-01-25 0:21 ` Stefan Beller
2017-01-25 0:43 ` Cornelius Weig
2017-01-25 0:52 ` Re: Stefan Beller
2017-01-25 0:54 ` Re: Linus Torvalds
2017-01-25 1:32 ` Re: Eric Wong
2016-04-11 19:04 (unknown), miwilliams
2016-04-12 4:33 ` Stefan Beller
2015-08-19 19:41 Re: christain147
2015-08-19 11:09 Re: christain147
2015-08-05 12:47 (unknown) Ivan Chernyavsky
2015-08-15 9:19 ` Duy Nguyen
2015-08-17 17:49 ` Re: Junio C Hamano
2015-04-08 20:44 (unknown), Mamta Upadhyay
2015-04-08 21:58 ` Thomas Braun
2015-04-09 11:27 ` Re: Konstantin Khomoutov
[not found] <CANSxx61FaNp5SBXJ8Y+pWn0eDcunmibKR5g8rttnWGdGwEMHCA@mail.gmail.com>
2015-03-18 20:45 ` Re: Junio C Hamano
2015-03-18 21:06 ` Re: Stefan Beller
2015-03-18 21:17 ` Re: Jeff King
2015-03-18 21:28 ` Re: Jeff King
2015-03-18 21:33 ` Re: Junio C Hamano
2015-03-18 21:45 ` Re: Stefan Beller
2015-03-13 1:34 (unknown) cody.taylor
2015-03-13 2:00 ` Duy Nguyen
2014-09-08 11:36 (unknown), R. Klomp
[not found] ` <CAOqJoqGSRUw_UT4LhqpYX-WX6AEd2ReAWjgNS76Cra-SMKw3NQ@mail.gmail.com>
2014-09-08 14:36 ` R. Klomp
2014-09-10 0:00 ` Re: David Aguilar
2014-09-15 15:10 ` Re: R. Klomp
2014-02-06 11:54 "Sparse checkout leaves no entry on working directory" all the time on Windows 7 on Git 1.8.5.2.msysgit.0 konstunn
2014-02-06 13:20 ` Johannes Sixt
2014-02-06 19:56 ` Constantine Gorbunov
2012-06-12 21:12 (unknown), rohit sood
2012-06-12 23:51 ` Erik Faye-Lund
2009-11-18 5:03 Re: Anna
2009-05-11 18:57 (unknown) Don Slutz
2009-05-11 20:48 ` Johannes Schindelin
2009-05-12 12:45 ` Re: Don Slutz
2009-03-30 5:03 (unknown), David Aguilar
2009-03-30 7:02 ` Markus Heidelberg
2009-03-30 8:46 ` Re: Junio C Hamano
2007-11-01 20:44 (unknown), Francesco Pretto
2007-11-01 20:48 ` Francesco Pretto
2006-02-02 0:39 [RFC & PATCH] Solaris 8: ENOSYS when mkdir applied to automount., Jason Riedy
2006-02-02 4:18 ` H. Peter Anvin
2005-04-22 22:19 (unknown), atani
2005-04-22 23:16 ` Martin Schlemmer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.LFD.2.01.0905080857130.4983@localhost.localdomain \
--to=torvalds@linux-foundation.org \
--cc=bevan.watkiss@cloakware.com \
--cc=casey@nrlssc.navy.mil \
--cc=git@vger.kernel.org \
--cc=raa.lkml@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).