From: Jeff King <peff@peff.net>
To: Michael Haggerty <mhagger@alum.mit.edu>
Cc: "Torsten Bögershausen" <tboegi@web.de>,
"Junio C Hamano" <gitster@pobox.com>,
"David Turner" <dturner@twopensource.com>,
"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
"Ramsay Jones" <ramsay@ramsayjones.plus.com>,
git@vger.kernel.org
Subject: Re: [PATCH v2 29/33] refs: resolve symbolic refs first
Date: Thu, 12 May 2016 03:45:28 -0400 [thread overview]
Message-ID: <20160512074528.GB10922@sigill.intra.peff.net> (raw)
In-Reply-To: <66d33af21bd1e398973414435af43d06f2e2099c.1462550456.git.mhagger@alum.mit.edu>
On Fri, May 06, 2016 at 06:14:10PM +0200, Michael Haggerty wrote:
> This makes use of a new function, lock_ref_raw(), which is analogous to
> read_ref_raw(), but acquires a lock on the reference before reading it.
Minor nit: the new function is actually called lock_raw_ref(). I don't
care which is used, just an inconsistency.
But my much bigger (non-)nit is that this seems to make large ref
updates much slower. You can see this by running t5551 with "--long".
In t5551.26, we fetch 48000 new tags into a repository that already has
2000 tags. Before this patch, it takes about 2 seconds. After, it chews
CPU for several minutes (I never actually let it finish).
The perf output isn't all that instructive. We seem to spend a lot of
time reading directory entries. Attaching with gdb shows:
#0 0x00007f35e00c2670 in __open_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000000000533982 in read_raw_ref (
refname=0x4e899f0 "refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-11520", sha1=0x7fff7c5aff30 "\002\357\373\070\332{\341\005\366츖\265G\276\332\f\025\271\276\377\177",
referent=0x836300 <sb_refname>, type=0x7fff7c5afe34) at refs/files-backend.c:1468
#2 0x0000000000530bf3 in resolve_ref_unsafe (
refname=0x4e899f0 "refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-11520", resolve_flags=1,
sha1=0x7fff7c5aff30 "\002\357\373\070\332{\341\005\366츖\265G\276\332\f\025\271\276\377\177",
flags=0x7fff7c5aff2c) at refs.c:1209
#3 0x000000000052e56f in read_ref_full (
refname=0x4e899f0 "refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-11520", resolve_flags=1,
sha1=0x7fff7c5aff30 "\002\357\373\070\332{\341\005\366츖\265G\276\332\f\025\271\276\377\177",
flags=0x7fff7c5aff2c) at refs.c:169
#4 0x000000000053316e in read_loose_refs (dirname=0x4e30f80 "refs/tags/", dir=0x4e30f58) at refs/files-backend.c:1216
#5 0x0000000000531435 in get_ref_dir (entry=0x4e30f50) at refs/files-backend.c:174
#6 0x000000000053265c in verify_refname_available_dir (
refname=0x1cd9438 "refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-12016", extras=0x7fff7c5b01d0, skip=0x0, dir=0x4dd6e98, err=0x7fff7c5b02a0) at refs/files-backend.c:789
#7 0x0000000000533e44 in lock_raw_ref (
refname=0x1cd9438 "refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-12016", mustexist=0, extras=0x7fff7c5b01d0, skip=0x0, lock_p=0x1cd9420, referent=0x7fff7c5b0140, type=0x1cd9428,
err=0x7fff7c5b02a0) at refs/files-backend.c:1663
#8 0x00000000005379d7 in lock_ref_for_update (update=0x1cd93f0, transaction=0x4db0150,
head_ref=0x4db0000 "refs/heads/master", affected_refnames=0x7fff7c5b01d0, err=0x7fff7c5b02a0)
at refs/files-backend.c:3416
[...]
So I'd expect us to hit that lock_ref_for_update() for each of the new
refs. But then we end up in verify_refname_available_dir(), which wants
to read all of the loose refs again. So we end up with a quadratic
number of calls to read_ref_full().
I haven't found the actual bug yet. It may be something as simple as not
clearing REF_INCOMPLETE from the loose-ref cache when we ought to. But
that's a wild (optimistic) guess.
-Peff
next prev parent reply other threads:[~2016-05-12 7:45 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-05-06 16:13 [PATCH v2 00/33] Yet more preparation for reference backends Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 01/33] t1404: demonstrate a bug resolving references Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 02/33] commit_ref(): if there is an empty dir in the way, delete it Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 03/33] read_raw_ref(): don't get confused by an empty directory Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 04/33] safe_create_leading_directories(): improve docstring Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 05/33] remove_dir_recursively(): add docstring Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 06/33] refname_is_safe(): use skip_prefix() Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 07/33] refname_is_safe(): don't allow the empty string Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 08/33] refname_is_safe(): insist that the refname already be normalized Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 09/33] commit_ref_update(): write error message to *err, not stderr Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 10/33] rename_ref(): remove unneeded local variable Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 11/33] ref_transaction_commit(): remove local variable n Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 12/33] read_raw_ref(): rename flags argument to type Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 13/33] read_raw_ref(): clear *type at start of function Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 14/33] read_raw_ref(): rename symref argument to referent Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 15/33] read_raw_ref(): improve docstring Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 16/33] read_raw_ref(): move docstring to header file Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 17/33] lock_ref_sha1_basic(): remove unneeded local variable Michael Haggerty
2016-05-06 16:13 ` [PATCH v2 18/33] refs: make error messages more consistent Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 19/33] ref_transaction_create(): disallow recursive pruning Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 20/33] ref_transaction_commit(): correctly report close_ref() failure Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 21/33] delete_branches(): use resolve_refdup() Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 22/33] refs: allow log-only updates Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 23/33] refs: don't dereference on rename Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 24/33] verify_refname_available(): adjust constness in declaration Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 25/33] add_update(): initialize the whole ref_update Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 26/33] lock_ref_for_update(): new function Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 27/33] unlock_ref(): move definition higher in the file Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 28/33] ref_transaction_update(): check refname_is_safe() at a minimum Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 29/33] refs: resolve symbolic refs first Michael Haggerty
2016-05-12 7:45 ` Jeff King [this message]
2016-05-12 8:25 ` Jeff King
2016-05-13 12:33 ` Michael Haggerty
2016-05-13 12:35 ` [PATCH v3 " Michael Haggerty
2016-05-13 12:58 ` Jeff King
2016-05-13 12:51 ` [PATCH v2 " Jeff King
2016-05-14 9:02 ` Torsten Bögershausen
2016-05-06 16:14 ` [PATCH v2 30/33] lock_ref_for_update(): don't re-read non-symbolic references Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 31/33] lock_ref_for_update(): don't resolve symrefs Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 32/33] commit_ref_update(): remove the flags parameter Michael Haggerty
2016-05-06 16:14 ` [PATCH v2 33/33] lock_ref_sha1_basic(): only handle REF_NODEREF mode Michael Haggerty
2016-05-09 20:12 ` [PATCH v2 00/33] Yet more preparation for reference backends David Turner
2016-05-09 21:05 ` Junio C Hamano
2016-05-09 21:50 ` Michael Haggerty
2016-05-09 22:04 ` Junio C Hamano
2016-05-12 7:55 ` Jeff King
2016-05-12 16:10 ` Junio C Hamano
2016-05-10 21:32 ` Junio C Hamano
2016-06-10 12:50 ` Michael Haggerty
2016-06-10 15:43 ` Junio C Hamano
2016-06-13 9:55 ` [ADDENDUM v4] " Michael Haggerty
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160512074528.GB10922@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=dturner@twopensource.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=mhagger@alum.mit.edu \
--cc=pclouds@gmail.com \
--cc=ramsay@ramsayjones.plus.com \
--cc=tboegi@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).