From: Jeff King <peff@peff.net>
To: Vegard Nossum <vegard.nossum@oracle.com>
Cc: git@vger.kernel.org,
"Quentin Casasnovas" <quentin.casasnovas@oracle.com>,
"Shawn Pearce" <spearce@spearce.org>,
"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>
Subject: Re: Huge performance bottleneck reading packs
Date: Wed, 12 Oct 2016 19:18:07 -0400 [thread overview]
Message-ID: <20161012231807.syockv2emrsjf55r@sigill.intra.peff.net> (raw)
In-Reply-To: <20161012230143.5kxcmtityaasra5j@sigill.intra.peff.net>
On Wed, Oct 12, 2016 at 07:01:43PM -0400, Jeff King wrote:
> On Thu, Oct 13, 2016 at 12:30:52AM +0200, Vegard Nossum wrote:
>
> > However, the commit found by 'git blame' above appears just fine to me,
> > I haven't been able to spot a bug in it.
> >
> > A closer inspection reveals the problem to really be that this is an
> > extremely hot path with more than -- holy cow -- 4,106,756,451
> > iterations on the 'packed_git' list for a single 'git fetch' on my
> > repository. I'm guessing the patch above just made the inner loop
> > ever so slightly slower.
> >
> > My .git/objects/pack/ has ~2088 files (1042 idx files, 1042 pack files,
> > and 4 tmp_pack_* files).
>
> Yeah. I agree that the commit you found makes the check a little more
> expensive, but I think the root of the problem is calling
> prepare_packed_git_one many times. This _should_ happen once for each
> pack at program startup, and possibly again if we need to re-scan the
> pack directory to account for racing with a simultaneous repack.
>
> The latter is generally triggered when we fail to look up an object we
> expect to exist. So I'd suspect 45e8a74 (has_sha1_file: re-check pack
> directory before giving up, 2013-08-30) is playing a part. We dealt with
> that to some degree in 0eeb077 (index-pack: avoid excessive re-reading
> of pack directory, 2015-06-09), but it would not surprise me if there is
> another spot that needs similar treatment.
>
> Does the patch below help?
Also, is it possible to make the repository in question available? I
might be able to reproduce based on your description, but it would save
time if I could directly run gdb on your example.
Specifically, I'm interested in the call graph that leads to calling
reprepare_packed_git().
-Peff
next prev parent reply other threads:[~2016-10-12 23:24 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-12 22:30 Huge performance bottleneck reading packs Vegard Nossum
2016-10-12 22:45 ` Junio C Hamano
2016-10-13 7:17 ` Vegard Nossum
2016-10-13 14:50 ` Jeff King
2016-10-12 23:01 ` Jeff King
2016-10-12 23:18 ` Jeff King [this message]
2016-10-12 23:47 ` Jeff King
2016-10-13 9:04 ` Vegard Nossum
2016-10-14 9:35 ` Jakub Narębski
2016-10-13 7:20 ` Vegard Nossum
2016-10-13 15:26 ` Jeff King
2016-10-13 16:53 ` [PATCH] fetch: use "quick" has_sha1_file for tag following Jeff King
2016-10-13 17:04 ` Jeff King
2016-10-13 20:06 ` Jeff King
2016-10-14 17:39 ` Junio C Hamano
2016-10-14 18:59 ` Jeff King
2016-10-17 17:30 ` Junio C Hamano
2016-10-18 10:28 ` Jeff King
2016-10-13 18:18 ` Huge performance bottleneck reading packs Vegard Nossum
2016-10-13 20:43 ` Jeff King
2016-10-14 6:55 ` Vegard Nossum
2016-10-14 19:00 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161012231807.syockv2emrsjf55r@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=pclouds@gmail.com \
--cc=quentin.casasnovas@oracle.com \
--cc=spearce@spearce.org \
--cc=vegard.nossum@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).