git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / Atom feed
From: Duy Nguyen <pclouds@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Cc: Git Mailing List <git@vger.kernel.org>, Jeff King <peff@peff.net>,
	Eric Wong <e@80x24.org>
Subject: Re: [PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config
Date: Tue, 6 Mar 2018 07:10:07 +0700
Message-ID: <CACsJy8BySv+EWSY8-Uwzf3o14dP8NdpcPTa7Bh2c2wtJ3jny1w@mail.gmail.com> (raw)

On Mon, Mar 5, 2018 at 9:00 PM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> On Thu, Mar 01 2018, Nguyễn Thái Ngọc Duy jotted:
>
>> pack-objects could be a big memory hog especially on large repos,
>> everybody knows that. The suggestion to stick a .keep file on the
>> largest pack to avoid this problem is also known for a long time.
>>
>> Let's do the suggestion automatically instead of waiting for people to
>> come to Git mailing list and get the advice. When a certain condition
>> is met, gc --auto create a .keep file temporary before repack is run,
>> then remove it afterward.
>>
>> gc --auto does this based on an estimation of pack-objects memory
>> usage and whether that fits in one third of system memory (the
>> assumption here is for desktop environment where there are many other
>> applications running).
>>
>> Since the estimation may be inaccurate and that 1/3 threshold is
>> arbitrary, give the user a finer control over this mechanism as well:
>> if the largest pack is larger than gc.bigPackThreshold, it's kept.
>
> This is very promising. Saves lots of memory on my ad-hoc testing of
> adding a *.keep file on an in-house repo.

The good news for you is when we run external rev-list on top of this,
memory consumption seems even better (and I think even peak memory
should be a bit lower too, but I'll need to verify that).

>> +     if (big_pack_threshold)
>> +             return pack->pack_size >= big_pack_threshold;
>> +
>> +     /* First we have to scan through at least one pack */
>> +     mem_want = pack->pack_size + pack->index_size;
>> +     /* then pack-objects needs lots more for book keeping */
>> +     mem_want += sizeof(struct object_entry) * nr_objects;
>> +     /*
>> +      * internal rev-list --all --objects takes up some memory too,
>> +      * let's say half of it is for blobs
>> +      */
>> +     mem_want += sizeof(struct blob) * nr_objects / 2;
>> +     /*
>> +      * and the other half is for trees (commits and tags are
>> +      * usually insignificant)
>> +      */
>> +     mem_want += sizeof(struct tree) * nr_objects / 2;
>> +     /* and then obj_hash[], underestimated in fact */
>> +     mem_want += sizeof(struct object *) * nr_objects;
>> +     /*
>> +      * read_sha1_file() (either at delta calculation phase, or
>> +      * writing phase) also fills up the delta base cache
>> +      */
>> +     mem_want += delta_base_cache_limit;
>> +     /* and of course pack-objects has its own delta cache */
>> +     mem_want += max_delta_cache_size;
>
> I'm not familiar enough with this part to say, but isn't this assuming a
> lot about the distribution of objects in a way that will cause is not to
> repack in some pathological cases?

Yeah this assumes a "normal" case. When the estimation is really off,
either we exclude the base pack or repack everything unnecessarily,
but we always repack. A wrong decision here can only affect
performance, not correctness.

There's one case I probably should address though. This "exclude the
base pack" will create two packs in the end, one big and one small.
But if the second pack is getting as big as the first one, it's time
we merge both into one.

> Probably worth documenting...
>
>> +     /* Only allow 1/3 of memory for pack-objects */
>> +     mem_have = total_ram() / 3;
>
> Would be great to have this be a configurable variable, so you could set
> it to e.g. 33% (like here), 50% etc.

Hmm.. isn't gc.bigPackThreshold enough? I mean in a controlled
environment, you probably already know how much ram is available, and
much of this estimation is based on pack size (well the number of
objects in the pack) anyway, you could avoid all this heuristics by
saying "when the base pack is larger than 1GB, always exclude it in
repack". This estimation should only needed when people do not
configure anything (and still expect reasonable defaults). Or when you
plan multiple 'gc' runs on the same machine?
-- 
Duy

             reply	other threads:[~2018-03-06  0:10 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-06  0:10 Duy Nguyen [this message]
  -- strict thread matches above, loose matches on Subject: below --
2018-02-28  9:27 Reduce pack-objects memory footprint? Duy Nguyen
2018-03-01  9:20 ` [PATCH/RFC 0/1] Avoid expensive 'repack -ad' in gc --auto Nguyễn Thái Ngọc Duy
2018-03-01  9:20   ` [PATCH/RFC 1/1] gc --auto: exclude the largest giant pack in low-memory config Nguyễn Thái Ngọc Duy
2018-03-01 18:14     ` Junio C Hamano
2018-03-02  0:00       ` Duy Nguyen
2018-03-05 14:00     ` Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CACsJy8BySv+EWSY8-Uwzf3o14dP8NdpcPTa7Bh2c2wtJ3jny1w@mail.gmail.com \
    --to=pclouds@gmail.com \
    --cc=avarab@gmail.com \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git