From: Stefan Beller <sbeller@google.com>
To: gitster@pobox.com
Cc: git@vger.kernel.org, hvoigt@hvoigt.net, jens.lehmann@web.de,
Stefan Beller <sbeller@google.com>
Subject: [PATCH] Add fetch.recurseSubmoduleParallelism config option
Date: Mon, 12 Oct 2015 15:52:30 -0700 [thread overview]
Message-ID: <1444690350-6486-1-git-send-email-sbeller@google.com> (raw)
This allows to configure fetching in parallel without having the annoying
command line option.
This moved the responsibility to determine how many parallel processes
to start from builtin/fetch to submodule.c as we need a way to communicate
"The user did not specify the number of parallel processes in the command
line options" in the builtin fetch. The submodule code takes care of
the precedence (CLI > config > default)
Signed-off-by: Stefan Beller <sbeller@google.com>
---
Documentation/config.txt | 6 ++++++
builtin/fetch.c | 2 +-
submodule.c | 14 ++++++++++++++
3 files changed, 21 insertions(+), 1 deletion(-)
I just monkey tested the code and it worked once! The problem with testing
this parallelizing option is that the expected behavior doesn't change
except for being faster. And I don't want to add timing tests to the test
suite because they are unreliable.
Any idea how to test this properly?
This applies on top of sb/submodule-parallel-fetch
Thanks,
Stefan
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 315f271..1172db0 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1140,6 +1140,12 @@ fetch.recurseSubmodules::
when its superproject retrieves a commit that updates the submodule's
reference.
+fetch.recurseSubmoduleParallelism
+ This is used to determine how many submodules can be fetched in
+ parallel. Specifying a positive integer allows up to that number
+ of submodules being fetched in parallel. Specifying 0 the number
+ of cpus will be taken as the maximum number.
+
fetch.fsckObjects::
If it is set to true, git-fetch-pack will check all fetched
objects. It will abort in the case of a malformed object or a
diff --git a/builtin/fetch.c b/builtin/fetch.c
index f28eac6..b1399dc 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -37,7 +37,7 @@ static int prune = -1; /* unspecified */
static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;
static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
static int tags = TAGS_DEFAULT, unshallow, update_shallow;
-static int max_children = 1;
+static int max_children = -1;
static const char *depth;
static const char *upload_pack;
static struct strbuf default_rla = STRBUF_INIT;
diff --git a/submodule.c b/submodule.c
index c21b265..c85d3ef 100644
--- a/submodule.c
+++ b/submodule.c
@@ -15,6 +15,7 @@
#include "thread-utils.h"
static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;
+static int config_fetch_parallel_submodules = -1;
static struct string_list changed_submodule_paths;
static int initialized_fetch_ref_tips;
static struct sha1_array ref_tips_before_fetch;
@@ -179,6 +180,14 @@ int submodule_config(const char *var, const char *value, void *cb)
else if (!strcmp(var, "fetch.recursesubmodules")) {
config_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);
return 0;
+ } else if (!strcmp(var, "fetch.recursesubmoduleparallelism")) {
+ char *end;
+ int ret;
+ config_fetch_parallel_submodules = strtol(value, &end, 0);
+ ret = (*end == '\0');
+ if (!ret)
+ warning("value for fetch.recurseSubmoduleParallelism not recognized");
+ return ret;
}
return 0;
}
@@ -759,6 +768,11 @@ int fetch_populated_submodules(const struct argv_array *options,
argv_array_push(&spf.args, "--recurse-submodules-default");
/* default value, "--submodule-prefix" and its value are added later */
+ if (max_parallel_jobs < 0)
+ max_parallel_jobs = config_fetch_parallel_submodules;
+ if (max_parallel_jobs < 0)
+ max_parallel_jobs = 1;
+
calculate_changed_submodule_paths();
run_processes_parallel(max_parallel_jobs,
get_next_submodule,
--
2.5.0.267.g8d6e698.dirty
next reply other threads:[~2015-10-12 22:53 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-12 22:52 Stefan Beller [this message]
2015-10-12 23:14 ` [PATCH] Add fetch.recurseSubmoduleParallelism config option Junio C Hamano
2015-10-12 23:31 ` Stefan Beller
2015-10-12 23:50 ` Junio C Hamano
2015-10-16 17:04 ` Stefan Beller
2015-10-16 17:26 ` Junio C Hamano
2015-10-13 7:32 ` Junio C Hamano
2015-10-13 16:03 ` Stefan Beller
2015-10-13 21:07 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1444690350-6486-1-git-send-email-sbeller@google.com \
--to=sbeller@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=hvoigt@hvoigt.net \
--cc=jens.lehmann@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).