From: Jeff King <peff@peff.net>
To: s@kazlauskas.me
Cc: Michael Haggerty <mhagger@alum.mit.edu>, git@vger.kernel.org
Subject: Re: Fetching new refs gets progressively slower
Date: Sun, 9 Jul 2017 07:29:32 -0400 [thread overview]
Message-ID: <20170709112932.njac5m6jmgmjywoz@sigill.intra.peff.net> (raw)
In-Reply-To: <20170709102506.GA32425@kumabox>
On Sun, Jul 09, 2017 at 01:25:06PM +0300, s@kazlauskas.me wrote:
> I have a weird issue where fetching a large number of refs will start off
> with lines like these:
>
> * [new ref] refs/pull/10000/head -> origin/pr/10000
> * [new ref] refs/pull/10001/head -> origin/pr/10001
>
> going fairly fast, and then progressively getting slower and slower. By the
> time git is working on 40 thousandth such ref, it seems like it is only
> handling about 3-5 such “new ref”s.
>
> These are the steps I used to reproduce:
>
> $ git clone git@github.com:rust-lang/rust
> $ # edit .git/config to add $ # `fetch =
> +refs/pull/*/head:refs/remotes/origin/pr/*` under origin remote
> $ git fetch
Interesting. The CPU is definitely going to user-space, and it does look
like a quadratic case, where we end up re-reading the loose ref
directory for each new ref we write.
The backtrace looks something like:
#1 0x0000555d86bcaf62 in files_read_raw_ref (ref_store=0x555d88d63a60,
refname=0x555d8a4791b0 "refs/remotes/origin/pr/10936",
sha1=0x7ffd29cbe1a0 "\003ŕX[\212*\037\064xE\fU\362!z44!<]U", referent=0x555d86efc630 <sb_refname>,
type=0x7ffd29cbe0c4) at refs/files-backend.c:686
#2 0x0000555d86bc845f in refs_read_raw_ref (ref_store=0x555d88d63a60,
refname=0x555d8a4791b0 "refs/remotes/origin/pr/10936",
sha1=0x7ffd29cbe1a0 "\003ŕX[\212*\037\064xE\fU\362!z44!<]U", referent=0x555d86efc630 <sb_refname>,
type=0x7ffd29cbe0c4) at refs.c:1391
#3 0x0000555d86bc851f in refs_resolve_ref_unsafe (refs=0x555d88d63a60,
refname=0x555d8a4791b0 "refs/remotes/origin/pr/10936", resolve_flags=1,
sha1=0x7ffd29cbe1a0 "\003ŕX[\212*\037\064xE\fU\362!z44!<]U", flags=0x7ffd29cbe19c) at refs.c:1430
#4 0x0000555d86bca9de in loose_fill_ref_dir (ref_store=0x555d88d63a60, dir=0x555d8a478cc8,
dirname=0x555d8a478cf0 "refs/remotes/origin/pr/") at refs/files-backend.c:485
#5 0x0000555d86bd1444 in get_ref_dir (entry=0x555d8a478cc0) at refs/ref-cache.c:28
#6 0x0000555d86bd18d4 in search_for_subdir (dir=0x555d8a478c78,
subdirname=0x555d8a414330 "refs/remotes/origin/pr/13832/", len=23, mkdir=0) at refs/ref-cache.c:172
#7 0x0000555d86bd192d in find_containing_dir (dir=0x555d8a478c78,
refname=0x555d8a414330 "refs/remotes/origin/pr/13832/", mkdir=0) at refs/ref-cache.c:191
#8 0x0000555d86bd23a4 in cache_ref_iterator_begin (cache=0x555d8a418380,
prefix=0x555d8a414330 "refs/remotes/origin/pr/13832/", prime_dir=1) at refs/ref-cache.c:567
#9 0x0000555d86bcb8eb in files_ref_iterator_begin (ref_store=0x555d88d63a60,
prefix=0x555d8a414330 "refs/remotes/origin/pr/13832/", flags=1) at refs/files-backend.c:1120
#10 0x0000555d86bc801a in refs_ref_iterator_begin (refs=0x555d88d63a60,
prefix=0x555d8a414330 "refs/remotes/origin/pr/13832/", trim=0, flags=1) at refs.c:1268
#11 0x0000555d86bc9289 in refs_verify_refname_available (refs=0x555d88d63a60,
refname=0x555d88d64b50 "refs/remotes/origin/pr/13832", extras=0x7ffd29cbe620, skip=0x0, err=0x7ffd29cbe710)
at refs.c:1887
#12 0x0000555d86bcb553 in lock_raw_ref (refs=0x555d88d63a60, refname=0x555d88d64b50 "refs/remotes/origin/pr/13832",
mustexist=0, extras=0x7ffd29cbe620, skip=0x0, lock_p=0x7ffd29cbe588, referent=0x7ffd29cbe590,
type=0x555d88d64b38, err=0x7ffd29cbe710) at refs/files-backend.c:956
#13 0x0000555d86bcf1bc in lock_ref_for_update (refs=0x555d88d63a60, update=0x555d88d64b00,
transaction=0x555d88d64d20, head_ref=0x555d8a435630 "refs/heads/master", affected_refnames=0x7ffd29cbe620,
err=0x7ffd29cbe710) at refs/files-backend.c:2764
The issue seems recent. Bisecting leads to Michael's 524a9fdb5
(refs_verify_refname_available(): use function in more places,
2017-04-16), which makes sense.
-Peff
next prev parent reply other threads:[~2017-07-09 11:29 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-09 10:25 Fetching new refs gets progressively slower s
2017-07-09 11:29 ` Jeff King [this message]
2017-08-17 15:12 ` [PATCH] files-backend: cheapen refname_available check when locking refs Michael Haggerty
2017-08-17 15:22 ` Jeff King
2017-08-17 17:56 ` Brandon Williams
2017-08-17 21:37 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170709112932.njac5m6jmgmjywoz@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=mhagger@alum.mit.edu \
--cc=s@kazlauskas.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).