From: Jeff King <peff@peff.net>
To: Vegard Nossum <vegard.nossum@oracle.com>
Cc: "Junio C Hamano" <gitster@pobox.com>,
git@vger.kernel.org,
"Quentin Casasnovas" <quentin.casasnovas@oracle.com>,
"Shawn Pearce" <spearce@spearce.org>,
"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: Re: Huge performance bottleneck reading packs
Date: Thu, 13 Oct 2016 10:50:06 -0400 [thread overview]
Message-ID: <20161013145006.mouzqmz3fssrjad4@sigill.intra.peff.net> (raw)
In-Reply-To: <1d5dd36b-7a9e-ac00-352a-d71e0e277002@oracle.com>
On Thu, Oct 13, 2016 at 09:17:34AM +0200, Vegard Nossum wrote:
> Oops. I disabled gc a while ago; one reason I did that is that it takes
> a long time to run and it has a tendency to kick in at the worst time. I
> guess I should really put it in cron then.
>
> I'm not sure if this is related, but I also had a problem with GitPython
> and large pack files in the past (" ValueError: Couldn't obtain fanout
> table or warning: packfile ./objects/pack/....pack cannot be accessed")
Sounds like they didn't correctly implement the extra index fanout that
happens for pack above 2G. The old Grit library had a similar bug.
> and I have pack.packSizeLimit set to 512m to fix that.
> Although the whole repo is 17G so I guess it shouldn't be necessary to
> have that many pack files.
Using packSizeLimit does "solve" that problem, but it comes with its own
set of issues. There is a very good chance that your repository would be
much smaller than 17G as a single packfile, because Git does not allow
deltas across packs, and it does not optimize the placement of objects
to keep delta-related objects in a single pack. So you'll quite often be
storing full copies of objects that could otherwise be stored as a tiny
delta.
You might want to compare the resulting size for a full repack with and
without pack.packSizeLimit.
But I agree that is not the cause of your thousand packs. They are more
likely the accumulated cruft of a thousand fetches.
-Peff
next prev parent reply other threads:[~2016-10-13 14:50 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-12 22:30 Huge performance bottleneck reading packs Vegard Nossum
2016-10-12 22:45 ` Junio C Hamano
2016-10-13 7:17 ` Vegard Nossum
2016-10-13 14:50 ` Jeff King [this message]
2016-10-12 23:01 ` Jeff King
2016-10-12 23:18 ` Jeff King
2016-10-12 23:47 ` Jeff King
2016-10-13 9:04 ` Vegard Nossum
2016-10-14 9:35 ` Jakub Narębski
2016-10-13 7:20 ` Vegard Nossum
2016-10-13 15:26 ` Jeff King
2016-10-13 16:53 ` [PATCH] fetch: use "quick" has_sha1_file for tag following Jeff King
2016-10-13 17:04 ` Jeff King
2016-10-13 20:06 ` Jeff King
2016-10-14 17:39 ` Junio C Hamano
2016-10-14 18:59 ` Jeff King
2016-10-17 17:30 ` Junio C Hamano
2016-10-18 10:28 ` Jeff King
2016-10-13 18:18 ` Huge performance bottleneck reading packs Vegard Nossum
2016-10-13 20:43 ` Jeff King
2016-10-14 6:55 ` Vegard Nossum
2016-10-14 19:00 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161013145006.mouzqmz3fssrjad4@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=pclouds@gmail.com \
--cc=quentin.casasnovas@oracle.com \
--cc=spearce@spearce.org \
--cc=vegard.nossum@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).