From: "Iucha, Florin" <Florin.Iucha@amd.com>
To: "git@vger.kernel.org" <git@vger.kernel.org>
Subject: High locking contention during repack?
Date: Wed, 12 Dec 2018 03:01:47 +0000 [thread overview]
Message-ID: <SN1PR12MB23840AFE62E41D908A40D1B095A70@SN1PR12MB2384.namprd12.prod.outlook.com> (raw)
Greetings,
I am running “git-repack -A -d -f -F --window=250 --depth=250” on a Git repository converted using git-svn.
The repository contains more than 10 years of development, and a mixture of source code and media assets. The size of the objects directory is around 50GB, and there are around 700k objects. There are several svn branches.
I have ran the repack with Git 2.17.1 (original package from Ubuntu 18.04), 2.19.1 (package from Ubuntu Git PPA) and with git next (bc1bbc6f855c3b5ef7fcbd0f688f647c4e5b208b).
The system is a 16 core / 32 thread Threadripper with 128GB of RAM and NVMe storage. The repack starts strong, with 32 threads but it fairly quickly gets to 99% done and the number of threads drops to 4 then 3 then 2. However, running “dstat 5” I see lots of “sys” time without any IO time (the network traffic you see is caused by SSH).
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read writ| recv send| in out | int csw
24 24 48 4 0| 0 0 | 362B 8392B| 0 0 | 64k 5042
27 25 44 4 0| 0 0 | 289B 4830B| 0 0 | 36k 5017
24 24 48 4 0| 0 0 | 362B 8392B| 0 0 | 64k 5042
27 25 44 4 0| 0 0 | 289B 4830B| 0 0 | 36k 5017
20 22 54 4 0| 0 0 | 330B 4826B| 0 0 | 31k 4657
22 23 51 3 0| 0 0 | 286B 4834B| 0 0 | 27k 4536
17 23 56 4 0| 0 0 | 299B 4820B| 0 0 | 29k 4099
24 23 49 4 0| 0 0 | 347B 4830B| 0 0 | 32k 4436
27 24 45 4 0| 0 0 | 292B 4820B| 0 0 | 22k 4513
26 27 44 3 0| 0 0 | 292B 4820B| 0 0 | 24k 4878
25 28 44 4 0| 0 0 | 329B 4837B| 0 0 | 45k 5023
23 30 44 3 0| 0 9830B| 370B 4836B| 0 0 | 40k 5064
23 26 47 4 0| 0 0 | 373B 4848B| 0 0 | 52k 4746
25 26 45 4 0| 0 0 | 323B 4833B| 0 0 | 31k 4874
24 28 45 3 0| 0 0 | 269B 4842B| 0 0 | 30k 4820
25 28 43 4 0| 0 0 | 280B 4826B| 0 0 | 45k 5237
25 29 43 3 0| 0 0 | 263B 4826B| 0 0 | 26k 5037
22 30 45 4 0| 0 0 | 287B 4823B| 0 0 | 45k 4154
0 53 42 5 0| 0 0 | 275B 1210B| 0 0 |5017 1331
0 53 42 5 0| 0 0 | 97B 0 | 0 0 |5006 1272
0 53 42 5 0| 0 0 | 70B 0 | 0 0 |5008 1272
0 53 42 5 0| 0 0 | 24B 0 | 0 0 |5007 1283
11 43 43 4 0| 0 0 | 644B 19k| 0 0 | 20k 2995
25 28 43 4 0| 0 123k| 304B 4802B| 0 0 | 29k 5118
25 27 44 4 0| 0 0 | 297B 4800B| 0 0 | 41k 5121
26 27 43 4 0| 0 0 | 320B 4786B| 0 0 | 29k 4973
22 30 44 3 0| 0 0 | 332B 4786B| 0 0 | 32k 4610
24 27 46 3 0| 0 0 | 296B 4786B| 0 0 | 56k 4632
35 17 44 3 0| 0 0 | 292B 4776B| 0 0 | 100k 4889
34 20 43 4 0| 0 9830B| 610B 4820B| 0 0 | 59k 4811
This when it gets to two threads phase:
1 40 59 0 0| 0 38k| 373B 2458B| 0 0 | 50k 2874
0 40 59 0 0| 0 0 | 268B 2892B| 0 0 | 10k 2858
0 40 59 0 0| 0 0 | 299B 3102B| 0 0 |8392 2919
1 40 59 0 0| 0 0 | 297B 3176B| 0 0 | 21k 2831
0 40 59 0 0| 0 0 | 281B 3414B| 0 0 |9192 2856
1 40 59 0 0| 0 0 | 264B 3493B| 0 0 | 17k 2905
1 40 59 0 0| 0 35k| 340B 3571B| 0 0 | 13k 2885
The repack still progresses, as I see the count of repacked objects being incremented every several minutes. But I suspect there is some terrible lock contention, because otherwise I can’t explain the high system usage with no IO.
Running a strace on the running git-repack process shows only these:
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
Any idea on how to debug this? I have ran git-repack under gdb, but it seems to spin on builtin/repack.c line 409.
Thank you for Git,
florin
next reply other threads:[~2018-12-12 3:02 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-12 3:01 Iucha, Florin [this message]
2018-12-12 11:24 ` High locking contention during repack? Jeff King
2018-12-12 14:04 ` Ævar Arnfjörð Bjarmason
2018-12-12 16:49 ` Iucha, Florin
2018-12-12 16:54 ` Ævar Arnfjörð Bjarmason
2018-12-12 18:08 ` Iucha, Florin
2018-12-12 18:30 ` Iucha, Florin
2018-12-12 19:05 ` Iucha, Florin
2018-12-12 21:50 ` Iucha, Florin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=SN1PR12MB23840AFE62E41D908A40D1B095A70@SN1PR12MB2384.namprd12.prod.outlook.com \
--to=florin.iucha@amd.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).