git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org, Johannes.Schindelin@gmx.de,
	sandals@crustytoothpaste.net, steadmon@google.com,
	jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com,
	phillip.wood123@gmail.com, emilyshaffer@google.com,
	sluongng@gmail.com, jonathantanmy@google.com,
	Derrick Stolee <derrickstolee@github.com>,
	Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch
Date: Thu, 23 Jul 2020 15:15:18 -0700	[thread overview]
Message-ID: <xmqqa6zpq3nt.fsf@gitster.c.googlers.com> (raw)
In-Reply-To: <478c7f1d0b858755c2c4b98605405214910b6f4c.1595527000.git.gitgitgadget@gmail.com> (Derrick Stolee via GitGitGadget's message of "Thu, 23 Jul 2020 17:56:33 +0000")

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> When repacking during the 'incremental-repack' task, we use the
> --batch-size option in 'git multi-pack-index repack'. The initial setting
> used --batch-size=0 to repack everything into a single pack-file. This is
> not sustaintable for a large repository. The amount of work required is

sustainable.

> also likely to use too many system resources for a background job.
>
> Update the 'incremental-repack' task by dynamically computing a
> --batch-size option based on the current pack-file structure.

OK.

> The dynamic default size is computed with this idea in mind for a client
> repository that was cloned from a very large remote: there is likely one
> "big" pack-file that was created at clone time. Thus, do not try
> repacking it as it is likely packed efficiently by the server.
>
> Instead, we select the second-largest pack-file, and create a batch size
> that is one larger than that pack-file. If there are three or more
> pack-files, then this guarantees that at least two will be combined into
> a new pack-file.
>
> Of course, this means that the second-largest pack-file size is likely
> to grow over time and may eventually surpass the initially-cloned
> pack-file. Recall that the pack-file batch is selected in a greedy
> manner: the packs are considered from oldest to newest and are selected
> if they have size smaller than the batch size until the total selected
> size is larger than the batch size. Thus, that oldest "clone" pack will
> be first to repack after the new data creates a pack larger than that.
>
> We also want to place some limits on how large these pack-files become,
> in order to bound the amount of time spent repacking. A maximum
> batch-size of two gigabytes means that large repositories will never be
> packed into a single pack-file using this job, but also that repack is
> rather expensive. This is a trade-off that is valuable to have if the
> maintenance is being run automatically or in the background. Users who
> truly want to optimize for space and performance (and are willing to pay
> the upfront cost of a full repack) can use the 'gc' task to do so.

It might be too late to ask this now, but how does the quality of
the resulting combined pack ensured, wrt locality and deltification?

> +#define TWO_GIGABYTES (2147483647)
> +#define UNSET_BATCH_SIZE ((unsigned long)-1)
> +
> +static off_t get_auto_pack_size(void)
> +{
> +	/*
> +	 * The "auto" value is special: we optimize for
> +	 * one large pack-file (i.e. from a clone) and
> +	 * expect the rest to be small and they can be
> +	 * repacked quickly.
> +	 *
> +	 * The strategy we select here is to select a
> +	 * size that is one more than the second largest
> +	 * pack-file. This ensures that we will repack
> +	 * at least two packs if there are three or more
> +	 * packs.
> +	 */
> +	off_t max_size = 0;
> +	off_t second_largest_size = 0;
> +	off_t result_size;
> +	struct packed_git *p;
> +	struct repository *r = the_repository;
> +
> +	reprepare_packed_git(r);
> +	for (p = get_all_packs(r); p; p = p->next) {
> +		if (p->pack_size > max_size) {
> +			second_largest_size = max_size;
> +			max_size = p->pack_size;
> +		} else if (p->pack_size > second_largest_size)
> +			second_largest_size = p->pack_size;
> +	}
> +
> +	result_size = second_largest_size + 1;

We won't worry about this addition wrapping around; I guess we
cannot do anything intelligent when it happens.

> +	/* But limit ourselves to a batch size of 2g */
> +	if (result_size > TWO_GIGABYTES)
> +		result_size = TWO_GIGABYTES;

Well, when it happens, we'd cap to 2G, which must be a reasonable
fallback value, so it would be OK.

> +	return result_size;
> +}
> +
>  static int multi_pack_index_repack(void)
>  {
>  	int result;
>  	struct argv_array cmd = ARGV_ARRAY_INIT;
> +	struct strbuf batch_arg = STRBUF_INIT;
> +
>  	argv_array_pushl(&cmd, "multi-pack-index", "repack", NULL);
>  
>  	if (opts.quiet)
>  		argv_array_push(&cmd, "--no-progress");
>  
> -	argv_array_push(&cmd, "--batch-size=0");
> +	strbuf_addf(&batch_arg, "--batch-size=%"PRIuMAX,
> +		    (uintmax_t)get_auto_pack_size());
> +	argv_array_push(&cmd, batch_arg.buf);
>  
>  	close_object_store(the_repository->objects);
>  	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
> +	strbuf_release(&batch_arg);

I think I saw a suggestion to use xstrfmt() with free()  instead of
the sequence of strbuf_init(), strbuf_addf(), and strbuf_release()
in a similar but different context.  Perhaps we should follow suit
here, too?


Thanks.  That's it for today from me.


  reply	other threads:[~2020-07-23 22:15 UTC|newest]

Thread overview: 164+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-07 14:21 [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 01/21] gc: use the_repository less often Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 02/21] gc: use repository in too_many_loose_objects() Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 03/21] gc: use repo config Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 04/21] gc: drop the_repository in log location Derrick Stolee via GitGitGadget
2020-07-09  2:22   ` Jonathan Tan
2020-07-09 11:13     ` Derrick Stolee
2020-07-07 14:21 ` [PATCH 05/21] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 06/21] maintenance: add --quiet option Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 07/21] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 08/21] maintenance: initialize task array and hashmap Derrick Stolee via GitGitGadget
2020-07-09  2:25   ` Jonathan Tan
2020-07-09 13:15     ` Derrick Stolee
2020-07-09 13:51       ` Junio C Hamano
2020-07-07 14:21 ` [PATCH 09/21] maintenance: add commit-graph task Derrick Stolee via GitGitGadget
2020-07-09  2:29   ` Jonathan Tan
2020-07-09 11:14     ` Derrick Stolee
2020-07-09 22:52       ` Jeff King
2020-07-09 23:41         ` Derrick Stolee
2020-07-07 14:21 ` [PATCH 10/21] maintenance: add --task option Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 11/21] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 12/21] maintenance: add fetch task Derrick Stolee via GitGitGadget
2020-07-09  2:35   ` Jonathan Tan
2020-07-07 14:21 ` [PATCH 13/21] maintenance: add loose-objects task Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 14/21] maintenance: add pack-files task Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 15/21] maintenance: auto-size pack-files batch Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 16/21] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 17/21] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 18/21] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 19/21] maintenance: create auto condition for loose-objects Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 20/21] maintenance: add pack-files auto condition Derrick Stolee via GitGitGadget
2020-07-07 14:21 ` [PATCH 21/21] midx: use start_delayed_progress() Derrick Stolee via GitGitGadget
2020-07-08 23:57 ` [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization Emily Shaffer
2020-07-09 11:21   ` Derrick Stolee
2020-07-09 12:43     ` Derrick Stolee
2020-07-09 23:16       ` Jeff King
2020-07-09 23:45         ` Derrick Stolee
2020-07-10 18:46           ` Emily Shaffer
2020-07-10 19:30             ` Son Luong Ngoc
2020-07-09 14:05     ` Junio C Hamano
2020-07-09 15:54       ` Derrick Stolee
2020-07-09 16:26         ` Junio C Hamano
2020-07-09 16:56           ` Derrick Stolee
2020-07-23 17:56 ` [PATCH v2 00/18] " Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 01/18] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget
2020-07-25  1:26     ` Taylor Blau
2020-07-25  1:47     ` Đoàn Trần Công Danh
2020-07-29 22:19     ` Jonathan Nieder
2020-07-30 13:12       ` Derrick Stolee
2020-07-31  0:30         ` Jonathan Nieder
2020-08-03 17:37           ` Derrick Stolee
2020-08-03 17:46             ` Jonathan Nieder
2020-08-03 22:46               ` Taylor Blau
2020-08-03 23:01                 ` Jonathan Nieder
2020-08-03 23:08                   ` Taylor Blau
2020-08-03 23:17                     ` Jonathan Nieder
2020-08-04  0:07                       ` Junio C Hamano
2020-08-04 13:32                       ` Derrick Stolee
2020-08-04 14:42                         ` Jonathan Nieder
2020-08-04 16:32                           ` Derrick Stolee
2020-08-04 17:02                             ` Jonathan Nieder
2020-08-04 17:51                               ` Derrick Stolee
2020-08-05 15:02               ` Derrick Stolee
2020-07-31 16:40         ` Jonathan Nieder
2020-08-03 23:52         ` Jonathan Nieder
2020-07-23 17:56   ` [PATCH v2 02/18] maintenance: add --quiet option Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 03/18] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget
2020-07-23 20:21     ` Junio C Hamano
2020-07-25  1:33       ` Taylor Blau
2020-07-30 13:29       ` Derrick Stolee
2020-07-30 13:31         ` Derrick Stolee
2020-07-30 19:00           ` Eric Sunshine
2020-07-30 20:21             ` Derrick Stolee
2020-07-23 17:56   ` [PATCH v2 04/18] maintenance: initialize task array Derrick Stolee via GitGitGadget
2020-07-23 19:57     ` Junio C Hamano
2020-07-24 12:23       ` Derrick Stolee
2020-07-24 12:51         ` Derrick Stolee
2020-07-24 19:39           ` Junio C Hamano
2020-07-25  1:46           ` Taylor Blau
2020-07-29 22:19     ` Emily Shaffer
2020-07-23 17:56   ` [PATCH v2 05/18] maintenance: add commit-graph task Derrick Stolee via GitGitGadget
2020-07-23 20:22     ` Junio C Hamano
2020-07-24 13:09       ` Derrick Stolee
2020-07-24 19:47         ` Junio C Hamano
2020-07-25  1:52           ` Taylor Blau
2020-07-30 13:59             ` Derrick Stolee
2020-07-29  0:22     ` Jeff King
2020-07-23 17:56   ` [PATCH v2 06/18] maintenance: add --task option Derrick Stolee via GitGitGadget
2020-07-23 20:21     ` Junio C Hamano
2020-07-23 22:18       ` Junio C Hamano
2020-07-24 13:36       ` Derrick Stolee
2020-07-24 19:50         ` Junio C Hamano
2020-07-23 17:56   ` [PATCH v2 07/18] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 08/18] maintenance: add prefetch task Derrick Stolee via GitGitGadget
2020-07-23 20:53     ` Junio C Hamano
2020-07-24 14:25       ` Derrick Stolee
2020-07-24 20:47         ` Junio C Hamano
2020-07-25  1:37     ` Đoàn Trần Công Danh
2020-07-25  1:48       ` Junio C Hamano
2020-07-27 14:07         ` Derrick Stolee
2020-07-27 16:13           ` Junio C Hamano
2020-07-27 18:27             ` Derrick Stolee
2020-07-28 16:37               ` [PATCH v2] fetch: optionally allow disabling FETCH_HEAD update Junio C Hamano
2020-07-29  9:12                 ` Phillip Wood
2020-07-29  9:17                   ` Phillip Wood
2020-07-30 15:17                 ` Derrick Stolee
2020-07-23 17:56   ` [PATCH v2 09/18] maintenance: add loose-objects task Derrick Stolee via GitGitGadget
2020-07-23 20:59     ` Junio C Hamano
2020-07-24 14:50       ` Derrick Stolee
2020-07-24 19:57         ` Junio C Hamano
2020-07-29 22:21     ` Emily Shaffer
2020-07-30 15:38       ` Derrick Stolee
2020-07-23 17:56   ` [PATCH v2 10/18] maintenance: add incremental-repack task Derrick Stolee via GitGitGadget
2020-07-23 22:00     ` Junio C Hamano
2020-07-24 15:03       ` Derrick Stolee
2020-07-29 22:22     ` Emily Shaffer
2020-07-23 17:56   ` [PATCH v2 11/18] maintenance: auto-size incremental-repack batch Derrick Stolee via GitGitGadget
2020-07-23 22:15     ` Junio C Hamano [this message]
2020-07-23 23:09       ` Eric Sunshine
2020-07-23 23:24         ` Junio C Hamano
2020-07-24 16:09           ` Derrick Stolee
2020-07-24 19:51       ` Derrick Stolee
2020-07-24 20:17         ` Junio C Hamano
2020-07-29 22:23     ` Emily Shaffer
2020-07-30 16:57       ` Derrick Stolee
2020-07-30 19:02         ` Derrick Stolee
2020-07-30 19:24           ` Chris Torek
2020-08-05 12:37           ` Đoàn Trần Công Danh
2020-08-06 13:54             ` Derrick Stolee
2020-07-23 17:56   ` [PATCH v2 12/18] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 13/18] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 14/18] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 15/18] maintenance: create auto condition for loose-objects Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 16/18] maintenance: add incremental-repack auto condition Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 17/18] midx: use start_delayed_progress() Derrick Stolee via GitGitGadget
2020-07-23 17:56   ` [PATCH v2 18/18] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget
2020-07-29 22:03   ` [PATCH v2 00/18] Maintenance builtin, allowing 'gc --auto' customization Emily Shaffer
2020-07-30 22:24   ` [PATCH v3 00/20] " Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 01/20] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 02/20] maintenance: add --quiet option Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 03/20] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 04/20] maintenance: initialize task array Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 05/20] maintenance: add commit-graph task Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 06/20] maintenance: add --task option Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 07/20] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 08/20] fetch: optionally allow disabling FETCH_HEAD update Junio C Hamano via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 09/20] maintenance: add prefetch task Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 10/20] maintenance: add loose-objects task Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 11/20] midx: enable core.multiPackIndex by default Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 12/20] maintenance: add incremental-repack task Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 13/20] maintenance: auto-size incremental-repack batch Derrick Stolee via GitGitGadget
2020-07-30 23:36       ` Chris Torek
2020-08-03 17:43         ` Derrick Stolee
2020-07-30 22:24     ` [PATCH v3 14/20] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 15/20] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 16/20] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 17/20] maintenance: create auto condition for loose-objects Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 18/20] maintenance: add incremental-repack auto condition Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 19/20] midx: use start_delayed_progress() Derrick Stolee via GitGitGadget
2020-07-30 22:24     ` [PATCH v3 20/20] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget
2020-07-30 23:06     ` [PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization Junio C Hamano
2020-07-30 23:31     ` Junio C Hamano
2020-07-31  2:58       ` Junio C Hamano
2020-08-06 17:58     ` Derrick Stolee

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqqa6zpq3nt.fsf@gitster.c.googlers.com \
    --to=gitster@pobox.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=congdanhqx@gmail.com \
    --cc=derrickstolee@github.com \
    --cc=dstolee@microsoft.com \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=jonathantanmy@google.com \
    --cc=jrnieder@gmail.com \
    --cc=peff@peff.net \
    --cc=phillip.wood123@gmail.com \
    --cc=sandals@crustytoothpaste.net \
    --cc=sluongng@gmail.com \
    --cc=steadmon@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).