From: Eric Sunshine <ericsunshine@gmail.com>
To: Stefan Beller <sbeller@google.com>
Cc: Git List <git@vger.kernel.org>,
Jacob Keller <jacob.keller@gmail.com>, Jeff King <peff@peff.net>,
Junio C Hamano <gitster@pobox.com>,
Jonathan Nieder <jrnieder@gmail.com>,
Johannes Schindelin <johannes.schindelin@gmail.com>,
Jens Lehmann <Jens.Lehmann@web.de>
Subject: Re: [PATCHv2 5/8] fetching submodules: Respect `submodule.jobs` config option
Date: Thu, 29 Oct 2015 22:17:12 -0400 [thread overview]
Message-ID: <CAPig+cS5yCzcz6xNyaMLTBFUzPqmbbE8x2_toFxAvXELcc786A@mail.gmail.com> (raw)
In-Reply-To: <1446074504-6014-6-git-send-email-sbeller@google.com>
On Wed, Oct 28, 2015 at 7:21 PM, Stefan Beller <sbeller@google.com> wrote:
> This allows to configure fetching and updating in parallel
> without having the command line option.
>
> This moved the responsibility to determine how many parallel processes
> to start from builtin/fetch to submodule.c as we need a way to communicate
> "The user did not specify the number of parallel processes in the command
> line options" in the builtin fetch. The submodule code takes care of
> the precedence (CLI > config > default)
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 391a0c3..785721a 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -2643,6 +2643,13 @@ submodule.<name>.ignore::
> "--ignore-submodules" option. The 'git submodule' commands are not
> affected by this setting.
>
> +submodule.jobs::
> + This is used to determine how many submodules can be operated on in
> + parallel. Specifying a positive integer allows up to that number
> + of submodules being fetched in parallel. This is used in fetch
> + and clone operations only. A value of 0 will give some reasonable
> + default. The defaults may change with different versions of Git.
I'm not sure that "default" is the correct word here. When you talk
about a "default", you're normally explaining what happens when the
configuration is not provided. (In fact, the default number of jobs is
1, which you may want to document here).
> tag.sort::
> This variable controls the sort ordering of tags when displayed by
> linkgit:git-tag[1]. Without the "--sort=<value>" option provided, the
> diff --git a/submodule-config.c b/submodule-config.c
> index 1cea404..07bdcdf 100644
> --- a/submodule-config.c
> +++ b/submodule-config.c
> @@ -32,6 +32,7 @@ enum lookup_type {
>
> static struct submodule_cache cache;
> static int is_cache_init;
> +static int parallel_jobs = -1;
>
> static int config_path_cmp(const struct submodule_entry *a,
> const struct submodule_entry *b,
> @@ -235,6 +236,9 @@ static int parse_generic_submodule_config(const char *var,
> const char *key,
> const char *value)
> {
> + if (!strcmp(key, "jobs")) {
> + parallel_jobs = strtol(value, NULL, 10);
> + }
Style: unnecessary braces
Why does this allow a negative value? The documentation doesn't
mention anything about it.
> return 0;
> }
>
> diff --git a/submodule.c b/submodule.c
> index 0257ea3..188ba02 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -752,6 +752,11 @@ int fetch_populated_submodules(const struct argv_array *options,
> argv_array_push(&spf.args, "--recurse-submodules-default");
> /* default value, "--submodule-prefix" and its value are added later */
>
> + if (max_parallel_jobs < 0)
> + max_parallel_jobs = config_parallel_submodules();
> + if (max_parallel_jobs < 0)
> + max_parallel_jobs = 1;
run_process_parallel() itself specially handles max_parallel_jobs==0,
so you don't need to consider it here. Okay.
> +
> calculate_changed_submodule_paths();
> run_processes_parallel(max_parallel_jobs,
> get_next_submodule,
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index 1b4ce69..5c3579c 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -470,4 +470,18 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea
> test_i18ncmp expect.err actual.err
> '
>
> +test_expect_success 'fetching submodules respects parallel settings' '
> + git config fetch.recurseSubmodules true &&
> + (
> + cd downstream &&
> + GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&
> + grep "7 children" trace.out &&
> + git config submodule.jobs 8 &&
> + GIT_TRACE=$(pwd)/trace.out git fetch &&
> + grep "8 children" trace.out &&
> + GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
> + grep "9 children" trace.out
> + )
> +'
Not specifically related to this test, but maybe add tests to check
cases when --jobs is not specified, and --jobs=1?
> +
> test_done
> --
> 2.5.0.281.g4ed9cdb
>
next prev parent reply other threads:[~2015-10-30 2:17 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-27 18:15 [PATCH 0/9] Expose the submodule parallelism to the user Stefan Beller
2015-10-27 18:15 ` [PATCH 1/9] submodule-config: "goto" removal in parse_config() Stefan Beller
2015-10-27 21:26 ` Jonathan Nieder
2015-10-27 21:39 ` Junio C Hamano
2015-10-27 18:15 ` [PATCH 2/9] submodule config: keep update strategy around Stefan Beller
2015-10-27 18:15 ` [PATCH 3/9] run_processes_parallel: Add output to tracing messages Stefan Beller
2015-10-27 18:15 ` [PATCH 4/9] git submodule update: have a dedicated helper for cloning Stefan Beller
2015-10-27 18:15 ` [PATCH 5/9] submodule update: expose parallelism to the user Stefan Beller
2015-10-27 20:59 ` Junio C Hamano
2015-10-28 21:40 ` Stefan Beller
2015-10-28 22:20 ` Junio C Hamano
2015-10-27 18:15 ` [PATCH 6/9] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2015-10-27 20:57 ` Junio C Hamano
2015-10-28 20:50 ` Stefan Beller
2015-10-27 18:15 ` [PATCH 7/9] submodule config: remove name_and_item_from_var Stefan Beller
2015-10-27 18:15 ` [PATCH 8/9] submodule-config: parse_config Stefan Beller
2015-10-27 18:15 ` [PATCH 9/9] fetching submodules: Respect `submodule.jobs` config option Stefan Beller
2015-10-27 21:00 ` Junio C Hamano
2015-10-27 19:12 ` [PATCH 0/9] Expose the submodule parallelism to the user Junio C Hamano
2015-10-28 23:21 ` [PATCHv2 0/8] " Stefan Beller
2015-10-28 23:21 ` [PATCHv2 1/8] run_processes_parallel: Add output to tracing messages Stefan Beller
2015-10-30 1:10 ` Eric Sunshine
2015-10-30 17:32 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 2/8] submodule config: keep update strategy around Stefan Beller
2015-10-30 1:14 ` Eric Sunshine
2015-10-30 17:38 ` Stefan Beller
2015-10-30 18:16 ` Eric Sunshine
2015-10-30 18:25 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 3/8] submodule config: remove name_and_item_from_var Stefan Beller
2015-10-30 1:23 ` Eric Sunshine
2015-10-30 18:37 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 4/8] submodule-config: parse_config Stefan Beller
2015-10-30 1:53 ` Eric Sunshine
2015-10-30 19:29 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 5/8] fetching submodules: Respect `submodule.jobs` config option Stefan Beller
2015-10-30 2:17 ` Eric Sunshine [this message]
2015-10-28 23:21 ` [PATCHv2 6/8] git submodule update: have a dedicated helper for cloning Stefan Beller
2015-10-29 22:34 ` Junio C Hamano
2015-10-28 23:21 ` [PATCHv2 7/8] submodule update: expose parallelism to the user Stefan Beller
2015-10-28 23:21 ` [PATCHv2 8/8] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2015-11-01 8:58 ` Eric Sunshine
2015-10-29 13:19 ` [PATCHv2 0/8] Expose the submodule parallelism to the user Ramsay Jones
2015-10-29 15:51 ` Stefan Beller
2015-10-29 17:23 ` Junio C Hamano
2015-10-29 17:30 ` Stefan Beller
2015-10-29 23:50 ` Ramsay Jones
2015-11-03 19:41 ` Stefan Beller
2015-10-29 20:12 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAPig+cS5yCzcz6xNyaMLTBFUzPqmbbE8x2_toFxAvXELcc786A@mail.gmail.com \
--to=ericsunshine@gmail.com \
--cc=Jens.Lehmann@web.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jacob.keller@gmail.com \
--cc=johannes.schindelin@gmail.com \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=sbeller@google.com \
--cc=sunshine@sunshineco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).