git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Jeff King <peff@peff.net>
To: Junio C Hamano <gitster@pobox.com>
Cc: Stefan Beller <sbeller@google.com>,
	David Turner <dturner@twosigma.com>,
	"git@vger.kernel.org" <git@vger.kernel.org>
Subject: Re: [PATCH] docs: update 64-bit core.packedGitLimit default
Date: Wed, 21 Jun 2017 14:53:07 -0400	[thread overview]
Message-ID: <20170621185307.xu6rcnj2y3jvdati@sigill.intra.peff.net> (raw)
In-Reply-To: <xmqqvanpp4n5.fsf@gitster.mtv.corp.google.com>

On Wed, Jun 21, 2017 at 11:38:54AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > So the other direction, instead of avoiding the memory limit in (4), is
> > to stop closing "small" packs in (2). But I don't think that's a good
> > idea. Even with the code after David's patch, you can still trigger the
> > problem by running out of file descriptors. And if we stop closing
> > small packs, that makes it even more likely for that to happen.
> 
> I recall that when we notice that we cannot access a loose one that
> we earlier thought existed we fall back to rescan the packs?  Would
> an approach similar to that can work to deal with the "closed small
> pack goes away" scenario?

Not very well. See the first paragraph of my explanation. Basically,
pack-objects is special because it makes decisions based on (and records
pointers to) the particular packed representation. If that goes away, it
just bails.

Which isn't to say that falling back is impossible. I think in the worst
case that it could say "oops, I can't access the pack that has object X
anymore", fall back to finding _any_ copy of it and including it as a
pure base object (it's too late at that point to make a delta, and
trying to be clever about reusing on-disk deltas is likely just going to
end up with a broken corner case).

So then you have a sub-optimal pack, but at least it didn't die(). If
that happens for one object, I don't think it's that big a deal. But the
resulting pack could end up pretty sub-optimal if you lose access to a
whole pack. And remember, "small" here is just smaller than the window
size, which is a gigabyte on 64 bit systems. So imagine that you lose
access to a 500 MB pack, but we recover by sending base objects. Then
everything that was in that pack gets converted to its full non-delta
representation, which could mean it expands to several gigabytes. The
current behavior to die() and retry the fetch is not that bad an
alternative.

Of course, the best alternative is retaining access to the packs, which
is what typically happens now on 64-bit systems (it's just that the
packedGitLimit was set pointlessly low). I'm not sure if you're asking
in general, or as a last-ditch effort for 32-bit systems.

-Peff

  reply	other threads:[~2017-06-21 18:53 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-20 20:41 [PATCH] Increase core.packedGitLimit David Turner
2017-04-20 21:02 ` Jeff King
2017-04-20 21:58   ` Johannes Schindelin
2017-04-20 22:04     ` David Turner
2017-04-21  9:34       ` Johannes Schindelin
2017-06-21 13:51   ` [PATCH] docs: update 64-bit core.packedGitLimit default Jeff King
2017-06-21 16:25     ` Stefan Beller
2017-06-21 18:06       ` Jeff King
2017-06-21 18:38         ` Junio C Hamano
2017-06-21 18:53           ` Jeff King [this message]
2017-06-21 19:22             ` Junio C Hamano
2017-06-21 19:31               ` Jeff King
2017-06-21 13:44 ` [PATCH] Increase core.packedGitLimit Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170621185307.xu6rcnj2y3jvdati@sigill.intra.peff.net \
    --to=peff@peff.net \
    --cc=dturner@twosigma.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=sbeller@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).