From: Ramsay Jones <ramsay@ramsayjones.plus.com>
To: Stefan Beller <sbeller@google.com>, git@vger.kernel.org
Cc: jacob.keller@gmail.com, peff@peff.net, gitster@pobox.com,
jrnieder@gmail.com, johannes.schindelin@gmail.com,
Jens.Lehmann@web.de, ericsunshine@gmail.com
Subject: Re: [PATCHv2 0/8] Expose the submodule parallelism to the user
Date: Thu, 29 Oct 2015 13:19:48 +0000 [thread overview]
Message-ID: <56321CF4.60807@ramsayjones.plus.com> (raw)
In-Reply-To: <1446074504-6014-1-git-send-email-sbeller@google.com>
On 28/10/15 23:21, Stefan Beller wrote:
> This replaces origin/sb/submodule-parallel-update
> (anchoring at 74367d8938, Merge branch 'sb/submodule-parallel-fetch'
> into sb/submodule-parallel-update)
>
> What does it do?
> ---
> This series should finish the on going efforts of parallelizing
> submodule network traffic. The patches contain tests for clone,
> fetch and submodule update to use the actual parallelism both via
> command line as well as a configured option. I decided to go with
> "submodule.jobs" for all three for now.
>
> What is new in v2?
> ---
> * The patches got reordered slightly
> * Documentation was adapted
>
> Interdiff below
>
> Stefan Beller (8):
> run_processes_parallel: Add output to tracing messages
> submodule config: keep update strategy around
> submodule config: remove name_and_item_from_var
> submodule-config: parse_config
> fetching submodules: Respect `submodule.jobs` config option
> git submodule update: have a dedicated helper for cloning
> submodule update: expose parallelism to the user
> clone: allow an explicit argument for parallel submodule clones
>
> Documentation/config.txt | 7 ++
> Documentation/git-clone.txt | 6 +-
> Documentation/git-submodule.txt | 7 +-
> builtin/clone.c | 23 +++-
> builtin/fetch.c | 2 +-
> builtin/submodule--helper.c | 244 ++++++++++++++++++++++++++++++++++++++++
> git-submodule.sh | 54 ++++-----
> run-command.c | 4 +
> submodule-config.c | 98 ++++++++++------
> submodule-config.h | 3 +
> submodule.c | 5 +
> t/t5526-fetch-submodules.sh | 14 +++
> t/t7400-submodule-basic.sh | 4 +-
> t/t7406-submodule-update.sh | 27 +++++
> 14 files changed, 418 insertions(+), 80 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 0de0138..785721a 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -2643,12 +2643,12 @@ submodule.<name>.ignore::
> "--ignore-submodules" option. The 'git submodule' commands are not
> affected by this setting.
>
> -submodule::jobs
> +submodule.jobs::
> This is used to determine how many submodules can be operated on in
> parallel. Specifying a positive integer allows up to that number
> - of submodules being fetched in parallel. Specifying 0 the number
> - of cpus will be taken as the maximum number. Currently this is
> - used in fetch and clone operations only.
> + of submodules being fetched in parallel. This is used in fetch
> + and clone operations only. A value of 0 will give some reasonable
> + default. The defaults may change with different versions of Git.
>
> tag.sort::
> This variable controls the sort ordering of tags when displayed by
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index affa52e..01bd6b7 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -216,9 +216,10 @@ objects from the source repository into a pack in the cloned repository.
> The result is Git repository can be separated from working
> tree.
>
> --j::
> ---jobs::
> +-j <n>::
> +--jobs <n>::
> The number of submodules fetched at the same time.
> + Defaults to the `submodule.jobs` option.
Hmm, is there a way to _not_ fetch in parallel (override the
config) from the command line for a given command?
ATB,
Ramsay Jones
>
> <repository>::
> The (possibly remote) repository to clone from. See the
> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
> index f5429fa..c70fafd 100644
> --- a/Documentation/git-submodule.txt
> +++ b/Documentation/git-submodule.txt
> @@ -374,10 +374,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.
> clone with a history truncated to the specified number of revisions.
> See linkgit:git-clone[1]
>
> --j::
> ---jobs::
> +-j <n>::
> +--jobs <n>::
> This option is only valid for the update command.
> Clone new submodules in parallel with as many jobs.
> + Defaults to the `submodule.jobs` option.
>
> <path>...::
> Paths to submodule(s). When specified this will restrict the command
> diff --git a/builtin/clone.c b/builtin/clone.c
> index 5ac2d89..22b9924 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -727,10 +727,7 @@ static int checkout(void)
> struct argv_array args = ARGV_ARRAY_INIT;
> argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>
> - if (max_jobs == -1)
> - if (git_config_get_int("submodule.jobs", &max_jobs))
> - max_jobs = 1;
> - if (max_jobs != 1) {
> + if (max_jobs != -1) {
> struct strbuf sb = STRBUF_INIT;
> strbuf_addf(&sb, "--jobs=%d", max_jobs);
> argv_array_push(&args, sb.buf);
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index c3d438a..67dba1c 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -476,9 +476,10 @@ static int update_clone(int argc, const char **argv, const char *prefix)
> /* Overlay the parsed .gitmodules file with .git/config */
> git_config(git_submodule_config, NULL);
>
> - if (max_jobs == -1)
> - if (git_config_get_int("submodule.jobs", &max_jobs))
> - max_jobs = 1;
> + if (max_jobs < 0)
> + max_jobs = config_parallel_submodules();
> + if (max_jobs < 0)
> + max_jobs = 1;
>
> run_processes_parallel(max_jobs,
> update_clone_get_next_task,
>
next prev parent reply other threads:[~2015-10-29 13:20 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-27 18:15 [PATCH 0/9] Expose the submodule parallelism to the user Stefan Beller
2015-10-27 18:15 ` [PATCH 1/9] submodule-config: "goto" removal in parse_config() Stefan Beller
2015-10-27 21:26 ` Jonathan Nieder
2015-10-27 21:39 ` Junio C Hamano
2015-10-27 18:15 ` [PATCH 2/9] submodule config: keep update strategy around Stefan Beller
2015-10-27 18:15 ` [PATCH 3/9] run_processes_parallel: Add output to tracing messages Stefan Beller
2015-10-27 18:15 ` [PATCH 4/9] git submodule update: have a dedicated helper for cloning Stefan Beller
2015-10-27 18:15 ` [PATCH 5/9] submodule update: expose parallelism to the user Stefan Beller
2015-10-27 20:59 ` Junio C Hamano
2015-10-28 21:40 ` Stefan Beller
2015-10-28 22:20 ` Junio C Hamano
2015-10-27 18:15 ` [PATCH 6/9] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2015-10-27 20:57 ` Junio C Hamano
2015-10-28 20:50 ` Stefan Beller
2015-10-27 18:15 ` [PATCH 7/9] submodule config: remove name_and_item_from_var Stefan Beller
2015-10-27 18:15 ` [PATCH 8/9] submodule-config: parse_config Stefan Beller
2015-10-27 18:15 ` [PATCH 9/9] fetching submodules: Respect `submodule.jobs` config option Stefan Beller
2015-10-27 21:00 ` Junio C Hamano
2015-10-27 19:12 ` [PATCH 0/9] Expose the submodule parallelism to the user Junio C Hamano
2015-10-28 23:21 ` [PATCHv2 0/8] " Stefan Beller
2015-10-28 23:21 ` [PATCHv2 1/8] run_processes_parallel: Add output to tracing messages Stefan Beller
2015-10-30 1:10 ` Eric Sunshine
2015-10-30 17:32 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 2/8] submodule config: keep update strategy around Stefan Beller
2015-10-30 1:14 ` Eric Sunshine
2015-10-30 17:38 ` Stefan Beller
2015-10-30 18:16 ` Eric Sunshine
2015-10-30 18:25 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 3/8] submodule config: remove name_and_item_from_var Stefan Beller
2015-10-30 1:23 ` Eric Sunshine
2015-10-30 18:37 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 4/8] submodule-config: parse_config Stefan Beller
2015-10-30 1:53 ` Eric Sunshine
2015-10-30 19:29 ` Stefan Beller
2015-10-28 23:21 ` [PATCHv2 5/8] fetching submodules: Respect `submodule.jobs` config option Stefan Beller
2015-10-30 2:17 ` Eric Sunshine
2015-10-28 23:21 ` [PATCHv2 6/8] git submodule update: have a dedicated helper for cloning Stefan Beller
2015-10-29 22:34 ` Junio C Hamano
2015-10-28 23:21 ` [PATCHv2 7/8] submodule update: expose parallelism to the user Stefan Beller
2015-10-28 23:21 ` [PATCHv2 8/8] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2015-11-01 8:58 ` Eric Sunshine
2015-10-29 13:19 ` Ramsay Jones [this message]
2015-10-29 15:51 ` [PATCHv2 0/8] Expose the submodule parallelism to the user Stefan Beller
2015-10-29 17:23 ` Junio C Hamano
2015-10-29 17:30 ` Stefan Beller
2015-10-29 23:50 ` Ramsay Jones
2015-11-03 19:41 ` Stefan Beller
2015-10-29 20:12 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56321CF4.60807@ramsayjones.plus.com \
--to=ramsay@ramsayjones.plus.com \
--cc=Jens.Lehmann@web.de \
--cc=ericsunshine@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jacob.keller@gmail.com \
--cc=johannes.schindelin@gmail.com \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=sbeller@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).