From: Stefan Beller <sbeller@google.com>
To: sbeller@google.com, git@vger.kernel.org, Jens.Lehmann@web.de,
gitster@pobox.com
Cc: peff@peff.net, sunshine@sunshineco.com, jrnieder@gmail.com
Subject: [PATCHv18 10/11] submodule update: expose parallelism to the user
Date: Thu, 25 Feb 2016 15:08:27 -0800 [thread overview]
Message-ID: <1456441708-13512-11-git-send-email-sbeller@google.com> (raw)
In-Reply-To: <1456441708-13512-1-git-send-email-sbeller@google.com>
Expose possible parallelism either via the "--jobs" CLI parameter or
the "submodule.fetchJobs" setting.
By having the variable initialized to -1, we make sure 0 can be passed
into the parallel processing machine, which will then pick as many parallel
workers as there are CPUs.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
Documentation/git-submodule.txt | 7 ++++++-
builtin/submodule--helper.c | 8 +++++++-
git-submodule.sh | 9 +++++++++
t/t7406-submodule-update.sh | 12 ++++++++++++
4 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 1572f05..13adebf 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -16,7 +16,7 @@ SYNOPSIS
'git submodule' [--quiet] deinit [-f|--force] [--] <path>...
'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]
[-f|--force] [--rebase|--merge] [--reference <repository>]
- [--depth <depth>] [--recursive] [--] [<path>...]
+ [--depth <depth>] [--recursive] [--jobs <n>] [--] [<path>...]
'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
[commit] [--] [<path>...]
'git submodule' [--quiet] foreach [--recursive] <command>
@@ -377,6 +377,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.
clone with a history truncated to the specified number of revisions.
See linkgit:git-clone[1]
+-j <n>::
+--jobs <n>::
+ This option is only valid for the update command.
+ Clone new submodules in parallel with as many jobs.
+ Defaults to the `submodule.fetchJobs` option.
<path>...::
Paths to submodule(s). When specified this will restrict the command
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index c852554..882aeca 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -433,6 +433,7 @@ static int update_clone_task_finished(int result,
static int update_clone(int argc, const char **argv, const char *prefix)
{
const char *update = NULL;
+ int max_jobs = -1;
struct string_list_item *item;
struct pathspec pathspec;
struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
@@ -453,6 +454,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)
OPT_STRING(0, "depth", &suc.depth, "<depth>",
N_("Create a shallow clone truncated to the "
"specified number of revisions")),
+ OPT_INTEGER('j', "jobs", &max_jobs,
+ N_("parallel jobs")),
OPT__QUIET(&suc.quiet, N_("don't print cloning progress")),
OPT_END()
};
@@ -479,7 +482,10 @@ static int update_clone(int argc, const char **argv, const char *prefix)
gitmodules_config();
git_config(submodule_config, NULL);
- run_processes_parallel(1,
+ if (max_jobs < 0)
+ max_jobs = parallel_submodules();
+
+ run_processes_parallel(max_jobs,
update_clone_get_next_task,
update_clone_start_failure,
update_clone_task_finished,
diff --git a/git-submodule.sh b/git-submodule.sh
index a6a82d2..86018ee 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -645,6 +645,14 @@ cmd_update()
--depth=*)
depth=$1
;;
+ -j|--jobs)
+ case "$2" in '') usage ;; esac
+ jobs="--jobs=$2"
+ shift
+ ;;
+ --jobs=*)
+ jobs=$1
+ ;;
--)
shift
break
@@ -671,6 +679,7 @@ cmd_update()
${update:+--update "$update"} \
${reference:+--reference "$reference"} \
${depth:+--depth "$depth"} \
+ ${jobs:+$jobs} \
"$@" || echo "#unmatched"
} | {
err=
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index dda3929..7fd5142 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -774,4 +774,16 @@ test_expect_success 'submodule update --recursive drops module name before recur
test_i18ngrep "Submodule path .deeper/submodule/subsubmodule.: checked out" actual
)
'
+
+test_expect_success 'submodule update can be run in parallel' '
+ (cd super2 &&
+ GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 7 &&
+ grep "7 tasks" trace.out &&
+ git config submodule.fetchJobs 8 &&
+ GIT_TRACE=$(pwd)/trace.out git submodule update &&
+ grep "8 tasks" trace.out &&
+ GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 9 &&
+ grep "9 tasks" trace.out
+ )
+'
test_done
--
2.7.0.rc0.36.g75877e4.dirty
next prev parent reply other threads:[~2016-02-25 23:09 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-25 3:06 [PATCHv17 00/11] Expose submodule parallelism to the user Stefan Beller
2016-02-25 3:06 ` [PATCHv17 01/11] submodule-config: keep update strategy around Stefan Beller
2016-02-25 18:06 ` Junio C Hamano
2016-02-25 18:21 ` Stefan Beller
2016-02-25 3:06 ` [PATCHv17 02/11] submodule-config: drop check against NULL Stefan Beller
2016-02-25 3:06 ` [PATCHv17 03/11] fetching submodules: respect `submodule.fetchJobs` config option Stefan Beller
2016-02-25 3:06 ` [PATCHv17 04/11] submodule update: direct error message to stderr Stefan Beller
2016-02-25 3:06 ` [PATCHv17 05/11] run_processes_parallel: treat output of children as byte array Stefan Beller
2016-02-25 18:16 ` Junio C Hamano
2016-02-25 20:35 ` Stefan Beller
2016-02-25 3:06 ` [PATCHv17 06/11] run-command: expose default_{start_failure, task_finished} Stefan Beller
2016-02-25 3:06 ` [PATCHv17 07/11] run_processes_parallel: rename parameters for the callbacks Stefan Beller
2016-02-25 3:06 ` [PATCHv17 08/11] run_processes_parallel: correctly terminate callbacks with an LF Stefan Beller
2016-02-25 3:06 ` [PATCHv17 09/11] git submodule update: have a dedicated helper for cloning Stefan Beller
2016-02-25 3:06 ` [PATCHv17 10/11] submodule update: expose parallelism to the user Stefan Beller
2016-02-25 3:06 ` [PATCHv17 11/11] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2016-02-25 22:26 ` [PATCHv17 00/11] Expose submodule parallelism to the user Junio C Hamano
2016-02-25 23:08 ` [PATCHv18 00/11] Expose Stefan Beller
2016-02-25 23:08 ` [PATCHv18 01/11] submodule-config: keep update strategy around Stefan Beller
2016-02-25 23:08 ` [PATCHv18 02/11] submodule-config: drop check against NULL Stefan Beller
2016-02-25 23:08 ` [PATCHv18 03/11] fetching submodules: respect `submodule.fetchJobs` config option Stefan Beller
2016-02-25 23:08 ` [PATCHv18 04/11] submodule update: direct error message to stderr Stefan Beller
2016-02-25 23:08 ` [PATCHv18 05/11] run_processes_parallel: treat output of children as byte array Stefan Beller
2016-02-25 23:08 ` [PATCHv18 06/11] run-command: expose default_{start_failure, task_finished} Stefan Beller
2016-02-25 23:08 ` [PATCHv18 07/11] run_processes_parallel: rename parameters for the callbacks Stefan Beller
2016-02-25 23:08 ` [PATCHv18 08/11] run_processes_parallel: correctly terminate callbacks with an LF Stefan Beller
2016-02-25 23:08 ` [PATCHv18 09/11] git submodule update: have a dedicated helper for cloning Stefan Beller
2016-02-25 23:08 ` Stefan Beller [this message]
2016-02-25 23:08 ` [PATCHv18 11/11] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2016-02-25 23:11 ` [PATCHv18 00/11] Expose Stefan Beller
2016-02-25 23:19 ` Jonathan Nieder
2016-02-25 23:25 ` Stefan Beller
2016-02-25 23:35 ` Jonathan Nieder
2016-02-25 23:39 ` Junio C Hamano
2016-02-25 23:48 ` [PATCHv19 00/11] Expose submodule parallelism to the user Stefan Beller
2016-02-25 23:48 ` [PATCHv19 01/11] submodule-config: keep update strategy around Stefan Beller
2016-02-25 23:48 ` [PATCHv19 02/11] submodule-config: drop check against NULL Stefan Beller
2016-02-25 23:48 ` [PATCHv19 03/11] fetching submodules: respect `submodule.fetchJobs` config option Stefan Beller
2016-02-25 23:48 ` [PATCHv19 04/11] submodule update: direct error message to stderr Stefan Beller
2016-02-25 23:48 ` [PATCHv19 05/11] run_processes_parallel: treat output of children as byte array Stefan Beller
2016-02-25 23:48 ` [PATCHv19 06/11] run-command: expose default_{start_failure, task_finished} Stefan Beller
2016-02-25 23:48 ` [PATCHv19 07/11] run_processes_parallel: rename parameters for the callbacks Stefan Beller
2016-02-25 23:48 ` [PATCHv19 08/11] run_processes_parallel: correctly terminate callbacks with an LF Stefan Beller
2016-02-25 23:48 ` [PATCHv19 09/11] git submodule update: have a dedicated helper for cloning Stefan Beller
2016-02-27 8:40 ` Duy Nguyen
2016-02-29 19:03 ` Stefan Beller
2016-02-25 23:48 ` [PATCHv19 10/11] submodule update: expose parallelism to the user Stefan Beller
2016-02-25 23:48 ` [PATCHv19 11/11] clone: allow an explicit argument for parallel submodule clones Stefan Beller
2016-02-25 23:50 ` [PATCHv19 00/11] Expose submodule parallelism to the user Jonathan Nieder
2016-02-29 20:48 ` Johannes Sixt
2016-02-29 20:59 ` Stefan Beller
2016-02-29 21:01 ` Junio C Hamano
2016-02-29 21:06 ` Stefan Beller
2016-02-29 21:19 ` Junio C Hamano
2016-02-29 21:22 ` Stefan Beller
2016-02-29 21:28 ` Johannes Sixt
2016-02-29 21:51 ` Junio C Hamano
2016-02-29 21:55 ` Stefan Beller
2016-02-25 23:25 ` [PATCHv18 00/11] Expose Jonathan Nieder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1456441708-13512-11-git-send-email-sbeller@google.com \
--to=sbeller@google.com \
--cc=Jens.Lehmann@web.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=sunshine@sunshineco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).