[PATCH 00/15] run-command API: pass functions & opts via struct

[PATCH 00/15] run-command API: pass functions & opts via struct

[Ævar Arnfjörð Bjarmason]

This series changes the run-command API so that we pass options via a
struct instead of via the argument list, the end result is that an API
user looks like e.g.:
	
	+       const struct run_process_parallel_opts opts = {
	+               .tr2_category = "submodule",
	+               .tr2_label = "parallel/update",
	+
	+               .jobs = update_data->max_jobs,
	+
	+               .get_next_task = update_clone_get_next_task,
	+               .start_failure = update_clone_start_failure,
	+               .task_finished = update_clone_task_finished,
	+               .data = &suc,
	+       };
	[...]
	-       run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
	-                                  update_clone_start_failure,
	-                                  update_clone_task_finished, &suc, "submodule",
	-                                  "parallel/update");
	+       run_processes_parallel(&opts);


These patches are derived from earlier patches I sent for passing the
the "ungroup" parameter to this API[1], that's currently done with a
global variable, because we needed a minimal change for a regression
fix.

I'm submitting this now in the rc phase because there's another
concurrent series that could make use of this[2]. The alternative
would be for it to add an extra parameter to the two functions (one
after this series).

The upcoming migration to the new hook API and config-based hooks[3]
will also benefit significantly from this. I have a version of that
topic rebased on top of this, having this first gets rid of a lot of
churn, adding an optional callback just requires adding things to the
struct introduced here, not changing every single caller.

(Passing) CI at:
https://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing

1. https://lore.kernel.org/git/cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com/
2. https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/
3. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/

Brief commentary on individual patches below:

Ævar Arnfjörð Bjarmason (15):
  hook tests: fix redirection logic error in 96e7225b310
  submodule tests: reset "trace.out" between "grep" invocations

These bugfixes could be split out I suppose, but since we're changing
this area 1/15 seemed prudent, 2/15 is required for a later test
addition

  run-command tests: test stdout of run_command_parallel()

Tighten up tests, was ejected from the "ungroup" topic.

  run-command test helper: use "else if" pattern
  run-command tests: use "return", not "exit"

Just setup for later changes.

  run-command API: have "run_processes_parallel{,_tr2}()" return void

Turns out we've never used the boilerplate return value for anything
useful ever.

  run-command API: make "jobs" parameter an "unsigned int"
  run-command API: don't fall back on online_cpus()

This allows us to make the "opts" struct "const", which helps a lot in
passing it around later on.

  run-command.c: add an initializer for "struct parallel_processes"
  run-command API: add nascent "struct run_process_parallel_opts"
  run-command API: make run_process_parallel{,_tr2}() thin wrappers
  run-command API: have run_process_parallel() take an "opts" struct
  run-command API: move *_tr2() users to "run_processes_parallel()"

This is arguably one logical change (and at some point I had it as
such), but as the diff would be really large I've tried to split this
into easily digestable steps.

  run-command.c: don't copy *_fn to "struct parallel_processes"
  run-command.c: don't copy "ungroup" to "struct parallel_processes"

The only reason for copying various parameters into "struct
parallel_processes" was because we passed them as positionals, now
that we have an "opts" struct we can just pass that along instead.

This leaves the "struct parallel_processes" in run-command.c purely
for managing our own internal state.

By avoiding this copying we also cut down a lot on the boilerplate
needed to add a new parameter.

 builtin/fetch.c             |  25 +++---
 builtin/submodule--helper.c |  16 +++-
 hook.c                      |  23 +++---
 run-command.c               | 152 ++++++++++++++----------------------
 run-command.h               |  71 ++++++++++++-----
 submodule-config.c          |   2 +
 submodule.c                 |  18 +++--
 t/helper/test-run-command.c |  77 +++++++++++-------
 t/t0061-run-command.sh      |  25 +++---
 t/t1800-hook.sh             |   2 +-
 t/t5526-fetch-submodules.sh |  16 +++-
 11 files changed, 247 insertions(+), 180 deletions(-)

-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 01/15] hook tests: fix redirection logic error in 96e7225b310

[Ævar Arnfjörð Bjarmason]

The tests added in 96e7225b310 (hook: add 'run' subcommand,
2021-12-22) were redirecting to "actual" both in the body of the hook
itself and in the testing code below.

The net result was that the "2>>actual" redirection later in the test
wasn't doing anything. Let's have those redirection do what it looks
like they're doing.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t1800-hook.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh
index 43fcb7c0bfc..2ef3579fa7c 100755
--- a/t/t1800-hook.sh
+++ b/t/t1800-hook.sh
@@ -95,7 +95,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '
 test_expect_success 'git -c core.hooksPath=<PATH> hook run' '
 	mkdir my-hooks &&
 	write_script my-hooks/test-hook <<-\EOF &&
-	echo Hook ran $1 >>actual
+	echo Hook ran $1
 	EOF
 
 	cat >expect <<-\EOF &&
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 02/15] submodule tests: reset "trace.out" between "grep" invocations

[Ævar Arnfjörð Bjarmason]

Fix test patterns added in 62104ba14af (submodules: allow parallel
fetching, add tests and documentation, 2015-12-15) and
a028a1930c6 (fetching submodules: respect `submodule.fetchJobs` config
option, 2016-02-29).

In the former case we were leaving a trace.out file at the top-level
for any subsequent tests (there are none, currently). Let's clean the
file up instead.

In the latter case we were testing that a given configuration would
result in "N tasks" in the log, but we were grepping through the log
for all previous such tests, when we really meant to clear the logs
between the "grep" invocations.

In practice this resulted in no logic error, as e.g. "--fetch 7" would
not print out a "9 tasks" line, but let's be paranoid and stop
implicitly assuming that that's the case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t5526-fetch-submodules.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index a301b56db89..e36f9fdf242 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -177,6 +177,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" '
 '
 
 test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
+	test_when_finished "rm -f trace.out" &&
 	add_submodule_commits &&
 	(
 		cd downstream &&
@@ -704,17 +705,25 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git
 
 test_expect_success 'fetching submodules respects parallel settings' '
 	git config fetch.recurseSubmodules true &&
+	test_when_finished "rm -f downstream/trace.out" &&
 	(
 		cd downstream &&
 		GIT_TRACE=$(pwd)/trace.out git fetch &&
 		grep "1 tasks" trace.out &&
+		>trace.out &&
+
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&
 		grep "7 tasks" trace.out &&
+		>trace.out &&
+
 		git config submodule.fetchJobs 8 &&
 		GIT_TRACE=$(pwd)/trace.out git fetch &&
 		grep "8 tasks" trace.out &&
+		>trace.out &&
+
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
-		grep "9 tasks" trace.out
+		grep "9 tasks" trace.out &&
+		>trace.out &&
 	)
 '
 
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 03/15] run-command tests: test stdout of run_command_parallel()

[Ævar Arnfjörð Bjarmason]

Extend the tests added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) to test stdout in
addition to stderr.

When the "ungroup" feature was added in fd3aaf53f71 (run-command: add
an "ungroup" option to run_process_parallel(), 2022-06-07) its tests
were made to test both the stdout and stderr, but these existing tests
were left alone. Let's also exhaustively test our expected output
here.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t0061-run-command.sh | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 7b5423eebda..19af082750a 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -130,8 +130,9 @@ World
 EOF
 
 test_expect_success 'run_command runs in parallel with more jobs available than tasks' '
-	test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '
@@ -141,8 +142,9 @@ test_expect_success 'run_command runs ungrouped in parallel with more jobs avail
 '
 
 test_expect_success 'run_command runs in parallel with as many jobs as tasks' '
-	test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '
@@ -152,8 +154,9 @@ test_expect_success 'run_command runs ungrouped in parallel with as many jobs as
 '
 
 test_expect_success 'run_command runs in parallel with more tasks than jobs available' '
-	test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '
@@ -172,8 +175,9 @@ asking for a quick stop
 EOF
 
 test_expect_success 'run_command is asked to abort gracefully' '
-	test-tool run-command run-command-abort 3 false 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-abort 3 false >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command is asked to abort gracefully (ungroup)' '
@@ -187,8 +191,9 @@ no further jobs available
 EOF
 
 test_expect_success 'run_command outputs ' '
-	test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command outputs (ungroup) ' '
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 04/15] run-command test helper: use "else if" pattern

[Ævar Arnfjörð Bjarmason]

Adjust the cmd__run_command() to use an "if/else if" chain rather than
mutually exclusive "if" statements. This non-functional change makes a
subsequent commit smaller.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index c9283b47afa..390fa4fb724 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_clear(&proc.args);
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
-	if (!strcmp(argv[1], "run-command-parallel"))
+	if (!strcmp(argv[1], "run-command-parallel")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, NULL, &proc));
-
-	if (!strcmp(argv[1], "run-command-abort"))
+	} else if (!strcmp(argv[1], "run-command-abort")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, task_finished, &proc));
-
-	if (!strcmp(argv[1], "run-command-no-jobs"))
+	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		exit(run_processes_parallel(jobs, no_job,
 					    NULL, task_finished, &proc));
-
-	fprintf(stderr, "check usage\n");
-	return 1;
+	} else {
+		fprintf(stderr, "check usage\n");
+		return 1;
+	}
 }
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 05/15] run-command tests: use "return", not "exit"

[Ævar Arnfjörð Bjarmason]

Change the "run-command" test helper to "return" instead of calling
"exit", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)

Because we'd previously gotten past the SANITIZE=leak check by using
exit() here we need to move to "goto cleanup" pattern. See
fdc8f79f1f1 (leak tests: run various "test-tool" tests in t00*.sh
SANITIZE=leak, 2021-10-12) for prior art. for when this code was opted
into the "linux-leaks" job.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 44 +++++++++++++++++++++++--------------
 1 file changed, 28 insertions(+), 16 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 390fa4fb724..ebda2203408 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)
 {
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
+	int ret;
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
-		exit(testsuite(argc - 1, argv + 1));
+		return testsuite(argc - 1, argv + 1);
 	if (!strcmp(argv[1], "inherited-handle"))
-		exit(inherit_handle(argv[0]));
+		return inherit_handle(argv[0]);
 	if (!strcmp(argv[1], "inherited-handle-child"))
-		exit(inherit_handle_child());
+		return inherit_handle_child();
 
 	if (argc >= 2 && !strcmp(argv[1], "quote-stress-test"))
 		return !!quote_stress_test(argc - 1, argv + 1);
@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)
 		argv += 2;
 		argc -= 2;
 	}
-	if (argc < 3)
-		return 1;
+	if (argc < 3) {
+		ret = 1;
+		goto cleanup;
+	}
 	strvec_pushv(&proc.args, (const char **)argv + 2);
 
 	if (!strcmp(argv[1], "start-command-ENOENT")) {
-		if (start_command(&proc) < 0 && errno == ENOENT)
-			return 0;
+		if (start_command(&proc) < 0 && errno == ENOENT) {
+			ret = 0;
+			goto cleanup;
+		}
 		fprintf(stderr, "FAIL %s\n", argv[1]);
 		return 1;
 	}
-	if (!strcmp(argv[1], "run-command"))
-		exit(run_command(&proc));
+	if (!strcmp(argv[1], "run-command")) {
+		ret = run_command(&proc);
+		goto cleanup;
+	}
 
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
@@ -428,16 +435,21 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, NULL, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL, NULL,
+				       &proc);
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL,
+				       task_finished, &proc);
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		exit(run_processes_parallel(jobs, no_job,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, no_job, NULL, task_finished,
+				       &proc);
 	} else {
+		ret = 1;
 		fprintf(stderr, "check usage\n");
-		return 1;
+		goto cleanup;
 	}
+	ret = 0;
+cleanup:
+	child_process_clear(&proc);
+	return ret;
 }
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 05/15] run-command tests: use "return", not "exit"

[Phillip Wood]

Hi Ævar

On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
> Change the "run-command" test helper to "return" instead of calling
> "exit", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)
> 
> Because we'd previously gotten past the SANITIZE=leak check by using
> exit() here we need to move to "goto cleanup" pattern. See
> fdc8f79f1f1 (leak tests: run various "test-tool" tests in t00*.sh
> SANITIZE=leak, 2021-10-12) for prior art. for when this code was opted
> into the "linux-leaks" job.

That commit just adds some TEST_PASSES_SANITIZE_LEAK=true lines, it's 
not nothing to do with "goto cleanup", I don't think we need to 
reference any prior art, just explain why we need to add the cleanup 
which you already do.

>   	if (!strcmp(argv[1], "run-command-parallel")) {
> -		exit(run_processes_parallel(jobs, parallel_next,
> -					    NULL, NULL, &proc));
> +		run_processes_parallel(jobs, parallel_next, NULL, NULL,
> +				       &proc);

There is no explanation of why it is safe to discard the return value 
here. The answer is in the next commit message, but needs to be 
mentioned here as well.

Best Wishes

Phillip

[PATCH 06/15] run-command API: have "run_processes_parallel{,_tr2}()" return void

[Ævar Arnfjörð Bjarmason]

Change the "run_processes_parallel{,_tr2}()" functions to return void,
instead of int. Ever since c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) they have
unconditionally returned 0.

To get a "real" return value out of this function the caller needs to
get it via the "task_finished_fn" callback, see the example in hook.c
added in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).

So the "result = " and "if (!result)" code added to "builtin/fetch.c"
d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
2019-10-05) has always been redundant, we always took that "if"
path. Likewise the "ret =" in "t/helper/test-run-command.c" added in
be5d88e1128 (test-tool run-command: learn to run (parts of) the
testsuite, 2019-10-04) wasn't used, instead we got the return value
from the "if (suite.failed.nr > 0)" block seen in the context.

Subsequent commits will alter this API interface, getting rid of this
always-zero return value makes it easier to understand those changes.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 17 ++++++++---------
 run-command.c               | 27 +++++++++++----------------
 run-command.h               | 16 ++++++++--------
 t/helper/test-run-command.c |  4 ++--
 4 files changed, 29 insertions(+), 35 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index a0fca93bb6a..78043fb67ef 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
 
 		strvec_push(&argv, "--end-of-options");
-		result = run_processes_parallel_tr2(max_children,
-						    &fetch_next_remote,
-						    &fetch_failed_to_start,
-						    &fetch_finished,
-						    &state,
-						    "fetch", "parallel/fetch");
-
-		if (!result)
-			result = state.result;
+		run_processes_parallel_tr2(max_children,
+					   &fetch_next_remote,
+					   &fetch_failed_to_start,
+					   &fetch_finished,
+					   &state,
+					   "fetch", "parallel/fetch");
+
+		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
 			const char *name = list->items[i].string;
diff --git a/run-command.c b/run-command.c
index 5ec3a46dccf..642e6b6e057 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-int run_processes_parallel(int n,
-			   get_next_task_fn get_next_task,
-			   start_failure_fn start_failure,
-			   task_finished_fn task_finished,
-			   void *pp_cb)
+void run_processes_parallel(int n,
+			    get_next_task_fn get_next_task,
+			    start_failure_fn start_failure,
+			    task_finished_fn task_finished,
+			    void *pp_cb)
 {
 	int i, code;
 	int output_timeout = 100;
@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,
 	}
 
 	pp_cleanup(&pp);
-	return 0;
 }
 
-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
-			       start_failure_fn start_failure,
-			       task_finished_fn task_finished, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label)
+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+				start_failure_fn start_failure,
+				task_finished_fn task_finished, void *pp_cb,
+				const char *tr2_category, const char *tr2_label)
 {
-	int result;
-
 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
 				   ((n < 1) ? online_cpus() : n));
 
-	result = run_processes_parallel(n, get_next_task, start_failure,
-					task_finished, pp_cb);
+	run_processes_parallel(n, get_next_task, start_failure,
+			       task_finished, pp_cb);
 
 	trace2_region_leave(tr2_category, tr2_label, NULL);
-
-	return result;
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 0e85e5846a5..e76a1b6b5b3 100644
--- a/run-command.h
+++ b/run-command.h
@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-int run_processes_parallel(int n,
-			   get_next_task_fn,
-			   start_failure_fn,
-			   task_finished_fn,
-			   void *pp_cb);
-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
-			       task_finished_fn, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label);
+void run_processes_parallel(int n,
+			    get_next_task_fn,
+			    start_failure_fn,
+			    task_finished_fn,
+			    void *pp_cb);
+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+				task_finished_fn, void *pp_cb,
+				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index ebda2203408..c431e3094df 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	ret = run_processes_parallel(max_jobs, next_test, test_failed,
-				     test_finished, &suite);
+	run_processes_parallel(max_jobs, next_test, test_failed,
+			       test_finished, &suite);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 06/15] run-command API: have "run_processes_parallel{,_tr2}()" return void

[Phillip Wood]

On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
> Change the "run_processes_parallel{,_tr2}()" functions to return void,
> instead of int. Ever since c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) they have
> unconditionally returned 0.
> 
> To get a "real" return value out of this function the caller needs to
> get it via the "task_finished_fn" callback, see the example in hook.c
> added in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).
> 
> So the "result = " and "if (!result)" code added to "builtin/fetch.c"
> d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
> 2019-10-05) has always been redundant, we always took that "if"
> path. Likewise the "ret =" in "t/helper/test-run-command.c" added in
> be5d88e1128 (test-tool run-command: learn to run (parts of) the
> testsuite, 2019-10-04) wasn't used, instead we got the return value
> from the "if (suite.failed.nr > 0)" block seen in the context.
> 
> Subsequent commits will alter this API interface, getting rid of this
> always-zero return value makes it easier to understand those changes.

This looks good, having a fixed return value is confusing.

Phillip

> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   builtin/fetch.c             | 17 ++++++++---------
>   run-command.c               | 27 +++++++++++----------------
>   run-command.h               | 16 ++++++++--------
>   t/helper/test-run-command.c |  4 ++--
>   4 files changed, 29 insertions(+), 35 deletions(-)
> 
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index a0fca93bb6a..78043fb67ef 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)
>   		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
>   
>   		strvec_push(&argv, "--end-of-options");
> -		result = run_processes_parallel_tr2(max_children,
> -						    &fetch_next_remote,
> -						    &fetch_failed_to_start,
> -						    &fetch_finished,
> -						    &state,
> -						    "fetch", "parallel/fetch");
> -
> -		if (!result)
> -			result = state.result;
> +		run_processes_parallel_tr2(max_children,
> +					   &fetch_next_remote,
> +					   &fetch_failed_to_start,
> +					   &fetch_finished,
> +					   &state,
> +					   "fetch", "parallel/fetch");
> +
> +		result = state.result;
>   	} else
>   		for (i = 0; i < list->nr; i++) {
>   			const char *name = list->items[i].string;
> diff --git a/run-command.c b/run-command.c
> index 5ec3a46dccf..642e6b6e057 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)
>   	return result;
>   }
>   
> -int run_processes_parallel(int n,
> -			   get_next_task_fn get_next_task,
> -			   start_failure_fn start_failure,
> -			   task_finished_fn task_finished,
> -			   void *pp_cb)
> +void run_processes_parallel(int n,
> +			    get_next_task_fn get_next_task,
> +			    start_failure_fn start_failure,
> +			    task_finished_fn task_finished,
> +			    void *pp_cb)
>   {
>   	int i, code;
>   	int output_timeout = 100;
> @@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,
>   	}
>   
>   	pp_cleanup(&pp);
> -	return 0;
>   }
>   
> -int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
> -			       start_failure_fn start_failure,
> -			       task_finished_fn task_finished, void *pp_cb,
> -			       const char *tr2_category, const char *tr2_label)
> +void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
> +				start_failure_fn start_failure,
> +				task_finished_fn task_finished, void *pp_cb,
> +				const char *tr2_category, const char *tr2_label)
>   {
> -	int result;
> -
>   	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
>   				   ((n < 1) ? online_cpus() : n));
>   
> -	result = run_processes_parallel(n, get_next_task, start_failure,
> -					task_finished, pp_cb);
> +	run_processes_parallel(n, get_next_task, start_failure,
> +			       task_finished, pp_cb);
>   
>   	trace2_region_leave(tr2_category, tr2_label, NULL);
> -
> -	return result;
>   }
>   
>   int run_auto_maintenance(int quiet)
> diff --git a/run-command.h b/run-command.h
> index 0e85e5846a5..e76a1b6b5b3 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,
>    * API reads that setting.
>    */
>   extern int run_processes_parallel_ungroup;
> -int run_processes_parallel(int n,
> -			   get_next_task_fn,
> -			   start_failure_fn,
> -			   task_finished_fn,
> -			   void *pp_cb);
> -int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
> -			       task_finished_fn, void *pp_cb,
> -			       const char *tr2_category, const char *tr2_label);
> +void run_processes_parallel(int n,
> +			    get_next_task_fn,
> +			    start_failure_fn,
> +			    task_finished_fn,
> +			    void *pp_cb);
> +void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
> +				task_finished_fn, void *pp_cb,
> +				const char *tr2_category, const char *tr2_label);
>   
>   /**
>    * Convenience function which prepares env for a command to be run in a
> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
> index ebda2203408..c431e3094df 100644
> --- a/t/helper/test-run-command.c
> +++ b/t/helper/test-run-command.c
> @@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)
>   	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
>   		(uintmax_t)suite.tests.nr, max_jobs);
>   
> -	ret = run_processes_parallel(max_jobs, next_test, test_failed,
> -				     test_finished, &suite);
> +	run_processes_parallel(max_jobs, next_test, test_failed,
> +			       test_finished, &suite);
>   
>   	if (suite.failed.nr > 0) {
>   		ret = 1;

[PATCH 07/15] run-command API: make "jobs" parameter an "unsigned int"

[Ævar Arnfjörð Bjarmason]

The rename the "n" variable added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) to "jobs", and
change the type to an "unsigned int". As we'll see in a subsequent
commit we do pass "0" here, but never "jobs < 0".

The only users of the "jobs" parameter are:

 * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
   config. As seen in the code that parses the config added in
   d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
   2019-10-05) will die if the git_config_int() return value is < 0.

   It will however pass us jobs = 0, as we'll see in a subsequent
   commit.

 * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
   config. Read via code originally added in a028a1930c6 (fetching
   submodules: respect `submodule.fetchJobs` config option, 2016-02-29).

   It now piggy-backs on the the submodule.fetchJobs code and
   validation added in f20e7c1ea24 (submodule: remove
   submodule.fetchjobs from submodule-config parsing, 2017-08-02).

   Like builtin/fetch.c it will die if the git_config_int() return
   value is < 0, but like builtin/fetch.c it will pass us jobs = 0.

 * builtin/submodule--helper.c: defaults to 1. Read via code
   originally added in 2335b870fa7 (submodule update: expose parallelism
   to the user, 2016-02-29).

   Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
   submodule-config parsing, 2017-08-02) it shares a config parser and
   semantics with the submodule.c caller.

 * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
   subcommand, 2021-12-22).

 * t/helper/test-run-command.c: can be -1 after parsing the arguments,
   but will then be overridden to online_cpus() before passing it to
   this API. See be5d88e1128 (test-tool run-command: learn to run (parts
   of) the testsuite, 2019-10-04).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 33 +++++++++++++++++----------------
 run-command.h |  6 +++---
 2 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/run-command.c b/run-command.c
index 642e6b6e057..80d282dbdb6 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *data;
 
-	int max_processes;
-	int nr_processes;
+	unsigned int max_processes;
+	unsigned int nr_processes;
 
 	get_next_task_fn get_next_task;
 	start_failure_fn start_failure;
@@ -1560,20 +1560,21 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    int n,
+		    unsigned int jobs,
 		    get_next_task_fn get_next_task,
 		    start_failure_fn start_failure,
 		    task_finished_fn task_finished,
 		    void *data, int ungroup)
 {
-	int i;
+	unsigned int i;
 
-	if (n < 1)
-		n = online_cpus();
+	if (jobs < 1)
+		jobs = online_cpus();
 
-	pp->max_processes = n;
+	pp->max_processes = jobs;
 
-	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
+	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
+		     jobs);
 
 	pp->data = data;
 	if (!get_next_task)
@@ -1587,14 +1588,14 @@ static void pp_init(struct parallel_processes *pp,
 	pp->output_owner = 0;
 	pp->shutdown = 0;
 	pp->ungroup = ungroup;
-	CALLOC_ARRAY(pp->children, n);
+	CALLOC_ARRAY(pp->children, jobs);
 	if (pp->ungroup)
 		pp->pfd = NULL;
 	else
-		CALLOC_ARRAY(pp->pfd, n);
+		CALLOC_ARRAY(pp->pfd, jobs);
 	strbuf_init(&pp->buffered_output, 0);
 
-	for (i = 0; i < n; i++) {
+	for (i = 0; i < jobs; i++) {
 		strbuf_init(&pp->children[i].err, 0);
 		child_process_init(&pp->children[i].process);
 		if (pp->pfd) {
@@ -1783,7 +1784,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(int n,
+void run_processes_parallel(unsigned int jobs,
 			    get_next_task_fn get_next_task,
 			    start_failure_fn start_failure,
 			    task_finished_fn task_finished,
@@ -1798,7 +1799,7 @@ void run_processes_parallel(int n,
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
+	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
 		ungroup);
 	while (1) {
 		for (i = 0;
@@ -1836,15 +1837,15 @@ void run_processes_parallel(int n,
 	pp_cleanup(&pp);
 }
 
-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
-				   ((n < 1) ? online_cpus() : n));
+				   ((jobs < 1) ? online_cpus() : jobs));
 
-	run_processes_parallel(n, get_next_task, start_failure,
+	run_processes_parallel(jobs, get_next_task, start_failure,
 			       task_finished, pp_cb);
 
 	trace2_region_leave(tr2_category, tr2_label, NULL);
diff --git a/run-command.h b/run-command.h
index e76a1b6b5b3..4502bdc64dc 100644
--- a/run-command.h
+++ b/run-command.h
@@ -459,7 +459,7 @@ typedef int (*task_finished_fn)(int result,
 				void *pp_task_cb);
 
 /**
- * Runs up to n processes at the same time. Whenever a process can be
+ * Runs up to 'jobs' processes at the same time. Whenever a process can be
  * started, the callback get_next_task_fn is called to obtain the data
  * required to start another child process.
  *
@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-void run_processes_parallel(int n,
+void run_processes_parallel(unsigned int jobs,
 			    get_next_task_fn,
 			    start_failure_fn,
 			    task_finished_fn,
 			    void *pp_cb);
-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
 				task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
 
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 07/15] run-command API: make "jobs" parameter an "unsigned int"

[Calvin Wan]

> The rename the "n" variable added in c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) to "jobs", and
> change the type to an "unsigned int". As we'll see in a subsequent
> commit we do pass "0" here, but never "jobs < 0".

> -       pp->max_processes = n;
> +       pp->max_processes = jobs;
>
> -       trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
> +       trace_printf("run_processes_parallel: preparing to run up to %d tasks",
> +                    jobs);

Should we normalize what we call processes/jobs/tasks? They all
seem to mean the same thing. I'm leaning towards processes
since the function name itself is run_processes_parallel.

> - * Runs up to n processes at the same time. Whenever a process can be
> + * Runs up to 'jobs' processes at the same time. Whenever a process can be

Also should "jobs" be changed to "max_jobs" or whatever normalized name
we call it? The comment here can then be changed to:
"Runs up to 'max_processes' at the same time."

Re: [PATCH 07/15] run-command API: make "jobs" parameter an "unsigned int"

[Phillip Wood]

On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
> The rename the "n" variable added in c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) to "jobs", and
> change the type to an "unsigned int". As we'll see in a subsequent
> commit we do pass "0" here, but never "jobs < 0".
> 
> The only users of the "jobs" parameter are:
> 
>   * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
>     config. As seen in the code that parses the config added in
>     d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
>     2019-10-05) will die if the git_config_int() return value is < 0.
> 
>     It will however pass us jobs = 0, as we'll see in a subsequent
>     commit.
> 
>   * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
>     config. Read via code originally added in a028a1930c6 (fetching
>     submodules: respect `submodule.fetchJobs` config option, 2016-02-29).
> 
>     It now piggy-backs on the the submodule.fetchJobs code and
>     validation added in f20e7c1ea24 (submodule: remove
>     submodule.fetchjobs from submodule-config parsing, 2017-08-02).
> 
>     Like builtin/fetch.c it will die if the git_config_int() return
>     value is < 0, but like builtin/fetch.c it will pass us jobs = 0.
> 
>   * builtin/submodule--helper.c: defaults to 1. Read via code
>     originally added in 2335b870fa7 (submodule update: expose parallelism
>     to the user, 2016-02-29).
> 
>     Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
>     submodule-config parsing, 2017-08-02) it shares a config parser and
>     semantics with the submodule.c caller.
> 
>   * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
>     subcommand, 2021-12-22).
> 
>   * t/helper/test-run-command.c: can be -1 after parsing the arguments,
>     but will then be overridden to online_cpus() before passing it to
>     this API. See be5d88e1128 (test-tool run-command: learn to run (parts
>     of) the testsuite, 2019-10-04).

I'm afraid I think this commit is a complete waste of reviewers time. As 
far as I con see in a later commit all the changes here are rewritten. 
Why do we need to go from n -> jobs -> opts->jobs rather than just n -> 
opts->jobs later.

Best Wishes

Phillip

> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 33 +++++++++++++++++----------------
>   run-command.h |  6 +++---
>   2 files changed, 20 insertions(+), 19 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index 642e6b6e057..80d282dbdb6 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;
>   struct parallel_processes {
>   	void *data;
>   
> -	int max_processes;
> -	int nr_processes;
> +	unsigned int max_processes;
> +	unsigned int nr_processes;
>   
>   	get_next_task_fn get_next_task;
>   	start_failure_fn start_failure;
> @@ -1560,20 +1560,21 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    int n,
> +		    unsigned int jobs,
>   		    get_next_task_fn get_next_task,
>   		    start_failure_fn start_failure,
>   		    task_finished_fn task_finished,
>   		    void *data, int ungroup)
>   {
> -	int i;
> +	unsigned int i;
>   
> -	if (n < 1)
> -		n = online_cpus();
> +	if (jobs < 1)
> +		jobs = online_cpus();
>   
> -	pp->max_processes = n;
> +	pp->max_processes = jobs;
>   
> -	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
> +	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
> +		     jobs);
>   
>   	pp->data = data;
>   	if (!get_next_task)
> @@ -1587,14 +1588,14 @@ static void pp_init(struct parallel_processes *pp,
>   	pp->output_owner = 0;
>   	pp->shutdown = 0;
>   	pp->ungroup = ungroup;
> -	CALLOC_ARRAY(pp->children, n);
> +	CALLOC_ARRAY(pp->children, jobs);
>   	if (pp->ungroup)
>   		pp->pfd = NULL;
>   	else
> -		CALLOC_ARRAY(pp->pfd, n);
> +		CALLOC_ARRAY(pp->pfd, jobs);
>   	strbuf_init(&pp->buffered_output, 0);
>   
> -	for (i = 0; i < n; i++) {
> +	for (i = 0; i < jobs; i++) {
>   		strbuf_init(&pp->children[i].err, 0);
>   		child_process_init(&pp->children[i].process);
>   		if (pp->pfd) {
> @@ -1783,7 +1784,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
>   	return result;
>   }
>   
> -void run_processes_parallel(int n,
> +void run_processes_parallel(unsigned int jobs,
>   			    get_next_task_fn get_next_task,
>   			    start_failure_fn start_failure,
>   			    task_finished_fn task_finished,
> @@ -1798,7 +1799,7 @@ void run_processes_parallel(int n,
>   	/* unset for the next API user */
>   	run_processes_parallel_ungroup = 0;
>   
> -	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
> +	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
>   		ungroup);
>   	while (1) {
>   		for (i = 0;
> @@ -1836,15 +1837,15 @@ void run_processes_parallel(int n,
>   	pp_cleanup(&pp);
>   }
>   
> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
> +void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
>   				start_failure_fn start_failure,
>   				task_finished_fn task_finished, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label)
>   {
>   	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
> -				   ((n < 1) ? online_cpus() : n));
> +				   ((jobs < 1) ? online_cpus() : jobs));
>   
> -	run_processes_parallel(n, get_next_task, start_failure,
> +	run_processes_parallel(jobs, get_next_task, start_failure,
>   			       task_finished, pp_cb);
>   
>   	trace2_region_leave(tr2_category, tr2_label, NULL);
> diff --git a/run-command.h b/run-command.h
> index e76a1b6b5b3..4502bdc64dc 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -459,7 +459,7 @@ typedef int (*task_finished_fn)(int result,
>   				void *pp_task_cb);
>   
>   /**
> - * Runs up to n processes at the same time. Whenever a process can be
> + * Runs up to 'jobs' processes at the same time. Whenever a process can be
>    * started, the callback get_next_task_fn is called to obtain the data
>    * required to start another child process.
>    *
> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
>    * API reads that setting.
>    */
>   extern int run_processes_parallel_ungroup;
> -void run_processes_parallel(int n,
> +void run_processes_parallel(unsigned int jobs,
>   			    get_next_task_fn,
>   			    start_failure_fn,
>   			    task_finished_fn,
>   			    void *pp_cb);
> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
> +void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
>   				task_finished_fn, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label);
>   

[PATCH 08/15] run-command API: don't fall back on online_cpus()

[Ævar Arnfjörð Bjarmason]

When a "jobs = 0" is passed let's BUG() out rather than fall back on
online_cpus(). The default behavior was added when this API was
implemented in c553c72eed6 (run-command: add an asynchronous parallel
child processor, 2015-12-15).

Most of our code in-tree that scales up to "online_cpus()" by default
calls that function by itself. By having these callers of the
"run_processes_parallel()" API do the same we can in subsequent
commits pass all arguments down as a "const struct".

The preceding commit has an overview of the API callers that passed
"jobs = 0". There were only two of them (actually three, but they
resolved to these two config parsing codepaths).

The "fetch.parallel" caller already had a test for the
"fetch.parallel=0" case added in 0353c688189 (fetch: do not run a
redundant fetch from submodule, 2022-05-16), but there was no such
test for "submodule.fetchJobs". Let's add one here.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 2 ++
 run-command.c               | 6 +++---
 submodule-config.c          | 2 ++
 t/t5526-fetch-submodules.sh | 5 +++++
 4 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 78043fb67ef..82f1da14ec1 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
 		fetch_parallel_config = git_config_int(k, v);
 		if (fetch_parallel_config < 0)
 			die(_("fetch.parallel cannot be negative"));
+		if (!fetch_parallel_config)
+			fetch_parallel_config = online_cpus();
 		return 0;
 	}
 
diff --git a/run-command.c b/run-command.c
index 80d282dbdb6..1a604af14fb 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1568,8 +1568,8 @@ static void pp_init(struct parallel_processes *pp,
 {
 	unsigned int i;
 
-	if (jobs < 1)
-		jobs = online_cpus();
+	if (!jobs)
+		BUG("you must provide a non-zero number of jobs!");
 
 	pp->max_processes = jobs;
 
@@ -1843,7 +1843,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
 				const char *tr2_category, const char *tr2_label)
 {
 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
-				   ((jobs < 1) ? online_cpus() : jobs));
+				   jobs);
 
 	run_processes_parallel(jobs, get_next_task, start_failure,
 			       task_finished, pp_cb);
diff --git a/submodule-config.c b/submodule-config.c
index cd7ee236a12..4dc61b3a78a 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)
 	int fetchjobs = git_config_int(var, value);
 	if (fetchjobs < 0)
 		die(_("negative values not allowed for submodule.fetchJobs"));
+	if (!fetchjobs)
+		fetchjobs = online_cpus();
 	return fetchjobs;
 }
 
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index e36f9fdf242..98a287ffb90 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
 		grep "9 tasks" trace.out &&
 		>trace.out &&
+
+		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
+		grep "preparing to run up to [0-9]* tasks" trace.out &&
+		! grep "up to 0 tasks" trace.out &&
+		>trace.out
 	)
 '
 
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 08/15] run-command API: don't fall back on online_cpus()

[Phillip Wood]

On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
> When a "jobs = 0" is passed let's BUG() out rather than fall back on
> online_cpus().

Why? It seems like a useful default behavior.

> The default behavior was added when this API was
> implemented in c553c72eed6 (run-command: add an asynchronous parallel
> child processor, 2015-12-15).
> 
> Most of our code in-tree that scales up to "online_cpus()" by default
> calls that function by itself. By having these callers of the
> "run_processes_parallel()" API do the same we can in subsequent
> commits pass all arguments down as a "const struct".

This is not convincing, you can have a function with a const struct 
interface that sets the number of jobs by doing

f(const struct *pp) {
	struct pp mutable;

	if (!pp->n) {
		mutable = *pp;
		mutable.n = online_cpus();
		pp = &mutable;
	}
	...
}

I'm not convinced by the reasoning given for removing the default behavior

Best Wishes

Phillip

> The preceding commit has an overview of the API callers that passed
> "jobs = 0". There were only two of them (actually three, but they
> resolved to these two config parsing codepaths).
> 
> The "fetch.parallel" caller already had a test for the
> "fetch.parallel=0" case added in 0353c688189 (fetch: do not run a
> redundant fetch from submodule, 2022-05-16), but there was no such
> test for "submodule.fetchJobs". Let's add one here.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   builtin/fetch.c             | 2 ++
>   run-command.c               | 6 +++---
>   submodule-config.c          | 2 ++
>   t/t5526-fetch-submodules.sh | 5 +++++
>   4 files changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 78043fb67ef..82f1da14ec1 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
>   		fetch_parallel_config = git_config_int(k, v);
>   		if (fetch_parallel_config < 0)
>   			die(_("fetch.parallel cannot be negative"));
> +		if (!fetch_parallel_config)
> +			fetch_parallel_config = online_cpus();
>   		return 0;
>   	}
>   
> diff --git a/run-command.c b/run-command.c
> index 80d282dbdb6..1a604af14fb 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1568,8 +1568,8 @@ static void pp_init(struct parallel_processes *pp,
>   {
>   	unsigned int i;
>   
> -	if (jobs < 1)
> -		jobs = online_cpus();
> +	if (!jobs)
> +		BUG("you must provide a non-zero number of jobs!");
>   
>   	pp->max_processes = jobs;
>   
> @@ -1843,7 +1843,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
>   				const char *tr2_category, const char *tr2_label)
>   {
>   	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
> -				   ((jobs < 1) ? online_cpus() : jobs));
> +				   jobs);
>   
>   	run_processes_parallel(jobs, get_next_task, start_failure,
>   			       task_finished, pp_cb);
> diff --git a/submodule-config.c b/submodule-config.c
> index cd7ee236a12..4dc61b3a78a 100644
> --- a/submodule-config.c
> +++ b/submodule-config.c
> @@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)
>   	int fetchjobs = git_config_int(var, value);
>   	if (fetchjobs < 0)
>   		die(_("negative values not allowed for submodule.fetchJobs"));
> +	if (!fetchjobs)
> +		fetchjobs = online_cpus();
>   	return fetchjobs;
>   }
>   
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index e36f9fdf242..98a287ffb90 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '
>   		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
>   		grep "9 tasks" trace.out &&
>   		>trace.out &&
> +
> +		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
> +		grep "preparing to run up to [0-9]* tasks" trace.out &&
> +		! grep "up to 0 tasks" trace.out &&
> +		>trace.out
>   	)
>   '
>   

[PATCH 09/15] run-command.c: add an initializer for "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

Add a PARALLEL_PROCESSES_INIT macro for the "struct
parallel_processes" used in run-command.c.

This allows us to do away with a call to strbuf_init(), and to rely on
other fields being NULL'd.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/run-command.c b/run-command.c
index 1a604af14fb..31a856f8b9a 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1525,6 +1525,9 @@ struct parallel_processes {
 	int output_owner;
 	struct strbuf buffered_output; /* of finished children */
 };
+#define PARALLEL_PROCESSES_INIT { \
+	.buffered_output = STRBUF_INIT, \
+}
 
 static int default_start_failure(struct strbuf *out,
 				 void *pp_cb,
@@ -1589,11 +1592,8 @@ static void pp_init(struct parallel_processes *pp,
 	pp->shutdown = 0;
 	pp->ungroup = ungroup;
 	CALLOC_ARRAY(pp->children, jobs);
-	if (pp->ungroup)
-		pp->pfd = NULL;
-	else
+	if (!pp->ungroup)
 		CALLOC_ARRAY(pp->pfd, jobs);
-	strbuf_init(&pp->buffered_output, 0);
 
 	for (i = 0; i < jobs; i++) {
 		strbuf_init(&pp->children[i].err, 0);
@@ -1794,7 +1794,7 @@ void run_processes_parallel(unsigned int jobs,
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	int ungroup = run_processes_parallel_ungroup;
-	struct parallel_processes pp;
+	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 10/15] run-command API: add nascent "struct run_process_parallel_opts"

[Ævar Arnfjörð Bjarmason]

As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
run_process_parallel(), 2022-06-07) which added the "ungroup" passing
it to "run_process_parallel()" via the global
"run_processes_parallel_ungroup" variable was a compromise to get the
smallest possible regression fix for "maint" at the time.

This follow-up to that is a start at passing that parameter and others
via a new "struct run_process_parallel_opts", as the earlier
version[1] of what became fd3aaf53f71 did.

For now we're only changing how data is passed internally to
"run-command.c", i.e. from "run_process_parallel()" to
pp_init(). Subsequent commits will change "run_processes_parallel()"
itself, as well as the "run_processes_parallel_tr2()" wrapper
function.

1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 38 ++++++++++++++++++++++----------------
 run-command.h | 44 +++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 63 insertions(+), 19 deletions(-)

diff --git a/run-command.c b/run-command.c
index 31a856f8b9a..f82fc7f1515 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1563,21 +1563,21 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    unsigned int jobs,
-		    get_next_task_fn get_next_task,
-		    start_failure_fn start_failure,
-		    task_finished_fn task_finished,
-		    void *data, int ungroup)
+		    const struct run_process_parallel_opts *opts)
 {
 	unsigned int i;
+	void *data = opts->data;
+	get_next_task_fn get_next_task = opts->get_next_task;
+	start_failure_fn start_failure = opts->start_failure;
+	task_finished_fn task_finished = opts->task_finished;
 
-	if (!jobs)
+	if (!opts->jobs)
 		BUG("you must provide a non-zero number of jobs!");
 
-	pp->max_processes = jobs;
+	pp->max_processes = opts->jobs;
 
 	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
-		     jobs);
+		     opts->jobs);
 
 	pp->data = data;
 	if (!get_next_task)
@@ -1590,12 +1590,12 @@ static void pp_init(struct parallel_processes *pp,
 	pp->nr_processes = 0;
 	pp->output_owner = 0;
 	pp->shutdown = 0;
-	pp->ungroup = ungroup;
-	CALLOC_ARRAY(pp->children, jobs);
+	pp->ungroup = opts->ungroup;
+	CALLOC_ARRAY(pp->children, opts->jobs);
 	if (!pp->ungroup)
-		CALLOC_ARRAY(pp->pfd, jobs);
+		CALLOC_ARRAY(pp->pfd, opts->jobs);
 
-	for (i = 0; i < jobs; i++) {
+	for (i = 0; i < opts->jobs; i++) {
 		strbuf_init(&pp->children[i].err, 0);
 		child_process_init(&pp->children[i].process);
 		if (pp->pfd) {
@@ -1793,14 +1793,20 @@ void run_processes_parallel(unsigned int jobs,
 	int i, code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
-	int ungroup = run_processes_parallel_ungroup;
 	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
+	const struct run_process_parallel_opts opts = {
+		.jobs = jobs,
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+		.ungroup = run_processes_parallel_ungroup,
+		.data = pp_cb,
+	};
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
-		ungroup);
+	pp_init(&pp, &opts);
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1817,7 +1823,7 @@ void run_processes_parallel(unsigned int jobs,
 		}
 		if (!pp.nr_processes)
 			break;
-		if (ungroup) {
+		if (opts.ungroup) {
 			int i;
 
 			for (i = 0; i < pp.max_processes; i++)
diff --git a/run-command.h b/run-command.h
index 4502bdc64dc..210fb9e8bc4 100644
--- a/run-command.h
+++ b/run-command.h
@@ -458,6 +458,47 @@ typedef int (*task_finished_fn)(int result,
 				void *pp_cb,
 				void *pp_task_cb);
 
+/**
+ * Option used by run_processes_parallel(), { 0 }-initialized means no
+ * options.
+ */
+struct run_process_parallel_opts
+{
+	/**
+	 * jobs: see 'jobs' in run_processes_parallel() below.
+	 */
+	int jobs;
+
+	/**
+	 * ungroup: see 'ungroup' in run_processes_parallel() below.
+	 */
+	unsigned int ungroup:1;
+
+	/**
+	 * get_next_task: See get_next_task_fn() above. This must be
+	 * specified.
+	 */
+	get_next_task_fn get_next_task;
+
+	/**
+	 * start_failure: See start_failure_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	start_failure_fn start_failure;
+
+	/**
+	 * task_finished: See task_finished_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	task_finished_fn task_finished;
+
+	/**
+	 * data: user data, will be passed as "pp_cb" to the callback
+	 * parameters.
+	 */
+	void *data;
+};
+
 /**
  * Runs up to 'jobs' processes at the same time. Whenever a process can be
  * started, the callback get_next_task_fn is called to obtain the data
@@ -467,9 +508,6 @@ typedef int (*task_finished_fn)(int result,
  * (both stdout and stderr) is routed to stderr in a manner that output
  * from different tasks does not interleave (but see "ungroup" below).
  *
- * start_failure_fn and task_finished_fn can be NULL to omit any
- * special handling.
- *
  * If the "ungroup" option isn't specified, the API will set the
  * "stdout_to_stderr" parameter in "struct child_process" and provide
  * the callbacks with a "struct strbuf *out" parameter to write output
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 10/15] run-command API: add nascent "struct run_process_parallel_opts"

[Phillip Wood]

On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
> As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
> run_process_parallel(), 2022-06-07) which added the "ungroup" passing
> it to "run_process_parallel()" via the global
> "run_processes_parallel_ungroup" variable was a compromise to get the
> smallest possible regression fix for "maint" at the time.
> 
> This follow-up to that is a start at passing that parameter and others
> via a new "struct run_process_parallel_opts", as the earlier
> version[1] of what became fd3aaf53f71 did.
> 
> For now we're only changing how data is passed internally to
> "run-command.c", i.e. from "run_process_parallel()" to
> pp_init(). Subsequent commits will change "run_processes_parallel()"
> itself, as well as the "run_processes_parallel_tr2()" wrapper
> function.

This is a very welcome change, but I'd have found it easier to follow if 
patches 10-12 were squashed together. There is a lot of code that gets 
added only to be deleted or moved in a following commit. A single patch 
would be bigger but they'd be less changes to review overall. As far as 
I can tell the overall change is good.

Best Wishes

Phillip

> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 38 ++++++++++++++++++++++----------------
>   run-command.h | 44 +++++++++++++++++++++++++++++++++++++++++---
>   2 files changed, 63 insertions(+), 19 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index 31a856f8b9a..f82fc7f1515 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1563,21 +1563,21 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    unsigned int jobs,
> -		    get_next_task_fn get_next_task,
> -		    start_failure_fn start_failure,
> -		    task_finished_fn task_finished,
> -		    void *data, int ungroup)
> +		    const struct run_process_parallel_opts *opts)
>   {
>   	unsigned int i;
> +	void *data = opts->data;
> +	get_next_task_fn get_next_task = opts->get_next_task;
> +	start_failure_fn start_failure = opts->start_failure;
> +	task_finished_fn task_finished = opts->task_finished;
>   
> -	if (!jobs)
> +	if (!opts->jobs)
>   		BUG("you must provide a non-zero number of jobs!");
>   
> -	pp->max_processes = jobs;
> +	pp->max_processes = opts->jobs;
>   
>   	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
> -		     jobs);
> +		     opts->jobs);
>   
>   	pp->data = data;
>   	if (!get_next_task)
> @@ -1590,12 +1590,12 @@ static void pp_init(struct parallel_processes *pp,
>   	pp->nr_processes = 0;
>   	pp->output_owner = 0;
>   	pp->shutdown = 0;
> -	pp->ungroup = ungroup;
> -	CALLOC_ARRAY(pp->children, jobs);
> +	pp->ungroup = opts->ungroup;
> +	CALLOC_ARRAY(pp->children, opts->jobs);
>   	if (!pp->ungroup)
> -		CALLOC_ARRAY(pp->pfd, jobs);
> +		CALLOC_ARRAY(pp->pfd, opts->jobs);
>   
> -	for (i = 0; i < jobs; i++) {
> +	for (i = 0; i < opts->jobs; i++) {
>   		strbuf_init(&pp->children[i].err, 0);
>   		child_process_init(&pp->children[i].process);
>   		if (pp->pfd) {
> @@ -1793,14 +1793,20 @@ void run_processes_parallel(unsigned int jobs,
>   	int i, code;
>   	int output_timeout = 100;
>   	int spawn_cap = 4;
> -	int ungroup = run_processes_parallel_ungroup;
>   	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
> +	const struct run_process_parallel_opts opts = {
> +		.jobs = jobs,
> +		.get_next_task = get_next_task,
> +		.start_failure = start_failure,
> +		.task_finished = task_finished,
> +		.ungroup = run_processes_parallel_ungroup,
> +		.data = pp_cb,
> +	};
>   
>   	/* unset for the next API user */
>   	run_processes_parallel_ungroup = 0;
>   
> -	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
> -		ungroup);
> +	pp_init(&pp, &opts);
>   	while (1) {
>   		for (i = 0;
>   		    i < spawn_cap && !pp.shutdown &&
> @@ -1817,7 +1823,7 @@ void run_processes_parallel(unsigned int jobs,
>   		}
>   		if (!pp.nr_processes)
>   			break;
> -		if (ungroup) {
> +		if (opts.ungroup) {
>   			int i;
>   
>   			for (i = 0; i < pp.max_processes; i++)
> diff --git a/run-command.h b/run-command.h
> index 4502bdc64dc..210fb9e8bc4 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -458,6 +458,47 @@ typedef int (*task_finished_fn)(int result,
>   				void *pp_cb,
>   				void *pp_task_cb);
>   
> +/**
> + * Option used by run_processes_parallel(), { 0 }-initialized means no
> + * options.
> + */
> +struct run_process_parallel_opts
> +{
> +	/**
> +	 * jobs: see 'jobs' in run_processes_parallel() below.
> +	 */
> +	int jobs;
> +
> +	/**
> +	 * ungroup: see 'ungroup' in run_processes_parallel() below.
> +	 */
> +	unsigned int ungroup:1;
> +
> +	/**
> +	 * get_next_task: See get_next_task_fn() above. This must be
> +	 * specified.
> +	 */
> +	get_next_task_fn get_next_task;
> +
> +	/**
> +	 * start_failure: See start_failure_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	start_failure_fn start_failure;
> +
> +	/**
> +	 * task_finished: See task_finished_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	task_finished_fn task_finished;
> +
> +	/**
> +	 * data: user data, will be passed as "pp_cb" to the callback
> +	 * parameters.
> +	 */
> +	void *data;
> +};
> +
>   /**
>    * Runs up to 'jobs' processes at the same time. Whenever a process can be
>    * started, the callback get_next_task_fn is called to obtain the data
> @@ -467,9 +508,6 @@ typedef int (*task_finished_fn)(int result,
>    * (both stdout and stderr) is routed to stderr in a manner that output
>    * from different tasks does not interleave (but see "ungroup" below).
>    *
> - * start_failure_fn and task_finished_fn can be NULL to omit any
> - * special handling.
> - *
>    * If the "ungroup" option isn't specified, the API will set the
>    * "stdout_to_stderr" parameter in "struct child_process" and provide
>    * the callbacks with a "struct strbuf *out" parameter to write output

[PATCH 11/15] run-command API: make run_process_parallel{,_tr2}() thin wrappers

[Ævar Arnfjörð Bjarmason]

Make the "run_process_parallel()" and "run_process_parallel_tr2()"
functions thin wrappers that construct a "struct
run_process_parallel_opts" struct, this is in preparation for changing
the API users to use a "struct run_process_parallel_opts" directly.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 67 ++++++++++++++++++++++++++++++++++++---------------
 run-command.h |  7 ++++++
 2 files changed, 54 insertions(+), 20 deletions(-)

diff --git a/run-command.c b/run-command.c
index f82fc7f1515..7a138234b40 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1784,29 +1784,25 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(unsigned int jobs,
-			    get_next_task_fn get_next_task,
-			    start_failure_fn start_failure,
-			    task_finished_fn task_finished,
-			    void *pp_cb)
+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
-	const struct run_process_parallel_opts opts = {
-		.jobs = jobs,
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-		.ungroup = run_processes_parallel_ungroup,
-		.data = pp_cb,
-	};
+	/* options */
+	const char *tr2_category = opts->tr2_category;
+	const char *tr2_label = opts->tr2_label;
+	const int do_trace2 = tr2_category && tr2_label;
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, &opts);
+	if (do_trace2)
+		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
+					   "max:%d", opts->jobs);
+
+	pp_init(&pp, opts);
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1823,7 +1819,7 @@ void run_processes_parallel(unsigned int jobs,
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts.ungroup) {
+		if (opts->ungroup) {
 			int i;
 
 			for (i = 0; i < pp.max_processes; i++)
@@ -1841,6 +1837,29 @@ void run_processes_parallel(unsigned int jobs,
 	}
 
 	pp_cleanup(&pp);
+
+	if (do_trace2)
+		trace2_region_leave(tr2_category, tr2_label, NULL);
+}
+
+void run_processes_parallel(unsigned int jobs,
+			    get_next_task_fn get_next_task,
+			    start_failure_fn start_failure,
+			    task_finished_fn task_finished,
+			    void *pp_cb)
+{
+	const struct run_process_parallel_opts opts = {
+		.jobs = jobs,
+		.ungroup = run_processes_parallel_ungroup,
+
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+
+		.data = pp_cb,
+	};
+
+	run_processes_parallel_1(&opts);
 }
 
 void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
@@ -1848,13 +1867,21 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
-				   jobs);
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = tr2_category,
+		.tr2_label = tr2_label,
+
+		.jobs = jobs,
+		.ungroup = run_processes_parallel_ungroup,
 
-	run_processes_parallel(jobs, get_next_task, start_failure,
-			       task_finished, pp_cb);
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+
+		.data = pp_cb,
+	};
 
-	trace2_region_leave(tr2_category, tr2_label, NULL);
+	run_processes_parallel_1(&opts);
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 210fb9e8bc4..7151312592e 100644
--- a/run-command.h
+++ b/run-command.h
@@ -464,6 +464,13 @@ typedef int (*task_finished_fn)(int result,
  */
 struct run_process_parallel_opts
 {
+	/**
+	 * tr2_category & tr2_label: sets the trace2 category and label for
+	 * logging. These must either be unset, or both of them must be set.
+	 */
+	const char *tr2_category;
+	const char *tr2_label;
+
 	/**
 	 * jobs: see 'jobs' in run_processes_parallel() below.
 	 */
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 12/15] run-command API: have run_process_parallel() take an "opts" struct

[Ævar Arnfjörð Bjarmason]

Have the "run_process_parallel()" function take an "opts" struct,
which allows us to eliminate it as a wrapper for
"run_processes_parallel_1()".

Since the new "run_processes_parallel()" function is able to take an
optional "tr2_category" and "tr2_label" via the struct we can at this
point migrate all of the users of "run_processes_parallel_tr2()" over
to it.

But let's not migrate all the API users, only the two users that
passed the "ungroup" parameter via the
"run_processes_parallel_ungroup" global, allowing us to delete that
global in favor of passing "ungroup" via the "opts" struct. As noted
in fd3aaf53f71 (run-command: add an "ungroup" option to
run_process_parallel(), 2022-06-07) which added
"run_processes_parallel_ungroup" passing this as a global was a hack
to produce a small regression fix for "maint".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c                      | 23 ++++++++++++++---------
 run-command.c               | 29 ++---------------------------
 run-command.h               | 15 ++++-----------
 t/helper/test-run-command.c | 32 +++++++++++++++++++++-----------
 4 files changed, 41 insertions(+), 58 deletions(-)

diff --git a/hook.c b/hook.c
index a493939a4fc..ce91a5d1f80 100644
--- a/hook.c
+++ b/hook.c
@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		.options = options,
 	};
 	const char *const hook_path = find_hook(hook_name);
-	int jobs = 1;
 	int ret = 0;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "hook",
+		.tr2_label = hook_name,
+
+		.jobs = 1,
+		.ungroup = 1,
+
+		.get_next_task = pick_next_hook,
+		.start_failure = notify_start_failure,
+		.task_finished = notify_hook_finished,
+
+		.data = &cb_data,
+	};
 
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		cb_data.hook_path = abs_path.buf;
 	}
 
-	run_processes_parallel_ungroup = 1;
-	run_processes_parallel_tr2(jobs,
-				   pick_next_hook,
-				   notify_start_failure,
-				   notify_hook_finished,
-				   &cb_data,
-				   "hook",
-				   hook_name);
+	run_processes_parallel(&opts);
 	ret = cb_data.rc;
 cleanup:
 	strbuf_release(&abs_path);
diff --git a/run-command.c b/run-command.c
index 7a138234b40..cdbba15a0ba 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1496,7 +1496,6 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
-int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *data;
 
@@ -1784,7 +1783,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)
+void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
 	int output_timeout = 100;
@@ -1795,9 +1794,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
 	const char *tr2_label = opts->tr2_label;
 	const int do_trace2 = tr2_category && tr2_label;
 
-	/* unset for the next API user */
-	run_processes_parallel_ungroup = 0;
-
 	if (do_trace2)
 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
 					   "max:%d", opts->jobs);
@@ -1842,26 +1838,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel(unsigned int jobs,
-			    get_next_task_fn get_next_task,
-			    start_failure_fn start_failure,
-			    task_finished_fn task_finished,
-			    void *pp_cb)
-{
-	const struct run_process_parallel_opts opts = {
-		.jobs = jobs,
-		.ungroup = run_processes_parallel_ungroup,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-
-		.data = pp_cb,
-	};
-
-	run_processes_parallel_1(&opts);
-}
-
 void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
@@ -1872,7 +1848,6 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
 		.tr2_label = tr2_label,
 
 		.jobs = jobs,
-		.ungroup = run_processes_parallel_ungroup,
 
 		.get_next_task = get_next_task,
 		.start_failure = start_failure,
@@ -1881,7 +1856,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
 		.data = pp_cb,
 	};
 
-	run_processes_parallel_1(&opts);
+	run_processes_parallel(&opts);
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 7151312592e..0f1d4081dcd 100644
--- a/run-command.h
+++ b/run-command.h
@@ -507,8 +507,10 @@ struct run_process_parallel_opts
 };
 
 /**
+ * Options are passed via the "struct run_process_parallel_opts" above.
+ *
  * Runs up to 'jobs' processes at the same time. Whenever a process can be
- * started, the callback get_next_task_fn is called to obtain the data
+ * started, the callback opts.get_next_task is called to obtain the data
  * required to start another child process.
  *
  * The children started via this function run in parallel. Their output
@@ -524,17 +526,8 @@ struct run_process_parallel_opts
  * NULL "struct strbuf *out" parameter, and are responsible for
  * emitting their own output, including dealing with any race
  * conditions due to writing in parallel to stdout and stderr.
- * The "ungroup" option can be enabled by setting the global
- * "run_processes_parallel_ungroup" to "1" before invoking
- * run_processes_parallel(), it will be set back to "0" as soon as the
- * API reads that setting.
  */
-extern int run_processes_parallel_ungroup;
-void run_processes_parallel(unsigned int jobs,
-			    get_next_task_fn,
-			    start_failure_fn,
-			    task_finished_fn,
-			    void *pp_cb);
+void run_processes_parallel(const struct run_process_parallel_opts *opts);
 void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
 				task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index c431e3094df..46bac2bb700 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {
 static int testsuite(int argc, const char **argv)
 {
 	struct testsuite suite = TESTSUITE_INIT;
-	int max_jobs = 1, i, ret;
+	int max_jobs = 1, i, ret = 0;
 	DIR *dir;
 	struct dirent *d;
 	struct option options[] = {
@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)
 			 "write JUnit-style XML files"),
 		OPT_END()
 	};
+	struct run_process_parallel_opts opts = {
+		.get_next_task = next_test,
+		.start_failure = test_failed,
+		.task_finished = test_finished,
+		.data = &suite,
+	};
 
 	argc = parse_options(argc, argv, NULL, options,
 			testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	run_processes_parallel(max_jobs, next_test, test_failed,
-			       test_finished, &suite);
+	opts.jobs = max_jobs;
+	run_processes_parallel(&opts);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)
 	string_list_clear(&suite.tests, 0);
 	string_list_clear(&suite.failed, 0);
 
-	return !!ret;
+	return ret;
 }
 
 static uint64_t my_random_next = 1234;
@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
 	int ret;
+	struct run_process_parallel_opts opts = {
+		.data = &proc,
+	};
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
 		return testsuite(argc - 1, argv + 1);
@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
 		argc -= 1;
-		run_processes_parallel_ungroup = 1;
+		opts.ungroup = 1;
 	}
 
 	jobs = atoi(argv[2]);
@@ -435,19 +444,20 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		run_processes_parallel(jobs, parallel_next, NULL, NULL,
-				       &proc);
+		opts.get_next_task = parallel_next;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		run_processes_parallel(jobs, parallel_next, NULL,
-				       task_finished, &proc);
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		run_processes_parallel(jobs, no_job, NULL, task_finished,
-				       &proc);
+		opts.get_next_task = no_job;
+		opts.task_finished = task_finished;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
 		goto cleanup;
 	}
+	opts.jobs = jobs;
+	run_processes_parallel(&opts);
 	ret = 0;
 cleanup:
 	child_process_clear(&proc);
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 13/15] run-command API: move *_tr2() users to "run_processes_parallel()"

[Ævar Arnfjörð Bjarmason]

Have the users of the "run_processes_parallel_tr2()" function use
"run_processes_parallel()" instead. In preceding commits the latter
was refactored to take a "struct run_process_parallel_opts" argument,
since the only reason for "run_processes_parallel_tr2()" to exist was
to take arguments that are now a part of that struct we can do away
with it.

See ee4512ed481 (trace2: create new combined trace facility,
2019-02-22) for the addition of the "*_tr2()" variant of the function,
it was used by every caller except "t/helper/test-run-command.c"..

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 18 ++++++++++++------
 builtin/submodule--helper.c | 16 ++++++++++++----
 run-command.c               | 21 ---------------------
 run-command.h               |  3 ---
 submodule.c                 | 18 ++++++++++++------
 5 files changed, 36 insertions(+), 40 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 82f1da14ec1..a4eea27cdae 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	if (max_children != 1 && list->nr != 1) {
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		const struct run_process_parallel_opts opts = {
+			.tr2_category = "fetch",
+			.tr2_label = "parallel/fetch",
+
+			.jobs = max_children,
+
+			.get_next_task = &fetch_next_remote,
+			.start_failure = &fetch_failed_to_start,
+			.task_finished = &fetch_finished,
+			.data = &state,
+		};
 
 		strvec_push(&argv, "--end-of-options");
-		run_processes_parallel_tr2(max_children,
-					   &fetch_next_remote,
-					   &fetch_failed_to_start,
-					   &fetch_finished,
-					   &state,
-					   "fetch", "parallel/fetch");
 
+		run_processes_parallel(&opts);
 		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 0b4acb442b2..cd34d021b70 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)
 {
 	int i, ret = 0;
 	struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/update",
+
+		.jobs = update_data->max_jobs,
+
+		.get_next_task = update_clone_get_next_task,
+		.start_failure = update_clone_start_failure,
+		.task_finished = update_clone_task_finished,
+		.data = &suc,
+	};
 
 	suc.update_data = update_data;
-	run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
-				   update_clone_start_failure,
-				   update_clone_task_finished, &suc, "submodule",
-				   "parallel/update");
+	run_processes_parallel(&opts);
 
 	/*
 	 * We saved the output and put it out all at once now.
diff --git a/run-command.c b/run-command.c
index cdbba15a0ba..1d5a500ecce 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1838,27 +1838,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
-				start_failure_fn start_failure,
-				task_finished_fn task_finished, void *pp_cb,
-				const char *tr2_category, const char *tr2_label)
-{
-	const struct run_process_parallel_opts opts = {
-		.tr2_category = tr2_category,
-		.tr2_label = tr2_label,
-
-		.jobs = jobs,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-
-		.data = pp_cb,
-	};
-
-	run_processes_parallel(&opts);
-}
-
 int run_auto_maintenance(int quiet)
 {
 	int enabled;
diff --git a/run-command.h b/run-command.h
index 0f1d4081dcd..075bd9b9de4 100644
--- a/run-command.h
+++ b/run-command.h
@@ -528,9 +528,6 @@ struct run_process_parallel_opts
  * conditions due to writing in parallel to stdout and stderr.
  */
 void run_processes_parallel(const struct run_process_parallel_opts *opts);
-void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
-				task_finished_fn, void *pp_cb,
-				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/submodule.c b/submodule.c
index bf7a2c79183..72b295b87b7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,
 {
 	int i;
 	struct submodule_parallel_fetch spf = SPF_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/fetch",
+
+		.jobs = max_parallel_jobs,
+
+		.get_next_task = get_next_submodule,
+		.start_failure = fetch_start_failure,
+		.task_finished = fetch_finish,
+		.data = &spf,
+	};
 
 	spf.r = r;
 	spf.command_line_option = command_line_option;
@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,
 
 	calculate_changed_submodule_paths(r, &spf.changed_submodule_names);
 	string_list_sort(&spf.changed_submodule_names);
-	run_processes_parallel_tr2(max_parallel_jobs,
-				   get_next_submodule,
-				   fetch_start_failure,
-				   fetch_finish,
-				   &spf,
-				   "submodule", "parallel/fetch");
+	run_processes_parallel(&opts);
 
 	if (spf.submodules_with_errors.len > 0)
 		fprintf(stderr, _("Errors during submodule fetch:\n%s"),
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 14/15] run-command.c: don't copy *_fn to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

The only remaining reason for copying the callbacks in the "struct
run_process_parallel_opts" over to the "struct parallel_processes" was
to avoid two if/else statements in case the "start_failure" and
"task_finished" callbacks were NULL.

Let's handle those cases in pp_start_one() and pp_collect_finished()
instead, and avoid the default_* stub functions, and the need to copy
this data around.

Organizing the code like this made more sense before the "struct
run_parallel_parallel_opts" existed, as we'd have needed to pass each
of these as a separate parameter.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 67 +++++++++++++++++++--------------------------------
 1 file changed, 25 insertions(+), 42 deletions(-)

diff --git a/run-command.c b/run-command.c
index 1d5a500ecce..b25e74fcc06 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1502,10 +1502,6 @@ struct parallel_processes {
 	unsigned int max_processes;
 	unsigned int nr_processes;
 
-	get_next_task_fn get_next_task;
-	start_failure_fn start_failure;
-	task_finished_fn task_finished;
-
 	struct {
 		enum child_state state;
 		struct child_process process;
@@ -1528,21 +1524,6 @@ struct parallel_processes {
 	.buffered_output = STRBUF_INIT, \
 }
 
-static int default_start_failure(struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
-static int default_task_finished(int result,
-				 struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
 static void kill_children(struct parallel_processes *pp, int signo)
 {
 	int i, n = pp->max_processes;
@@ -1566,9 +1547,6 @@ static void pp_init(struct parallel_processes *pp,
 {
 	unsigned int i;
 	void *data = opts->data;
-	get_next_task_fn get_next_task = opts->get_next_task;
-	start_failure_fn start_failure = opts->start_failure;
-	task_finished_fn task_finished = opts->task_finished;
 
 	if (!opts->jobs)
 		BUG("you must provide a non-zero number of jobs!");
@@ -1579,12 +1557,8 @@ static void pp_init(struct parallel_processes *pp,
 		     opts->jobs);
 
 	pp->data = data;
-	if (!get_next_task)
+	if (!opts->get_next_task)
 		BUG("you need to specify a get_next_task function");
-	pp->get_next_task = get_next_task;
-
-	pp->start_failure = start_failure ? start_failure : default_start_failure;
-	pp->task_finished = task_finished ? task_finished : default_task_finished;
 
 	pp->nr_processes = 0;
 	pp->output_owner = 0;
@@ -1637,7 +1611,8 @@ static void pp_cleanup(struct parallel_processes *pp)
  * <0 no new job was started, user wishes to shutdown early. Use negative code
  *    to signal the children.
  */
-static int pp_start_one(struct parallel_processes *pp)
+static int pp_start_one(struct parallel_processes *pp,
+			const struct run_process_parallel_opts *opts)
 {
 	int i, code;
 
@@ -1647,10 +1622,10 @@ static int pp_start_one(struct parallel_processes *pp)
 	if (i == pp->max_processes)
 		BUG("bookkeeping is hard");
 
-	code = pp->get_next_task(&pp->children[i].process,
-				 pp->ungroup ? NULL : &pp->children[i].err,
-				 pp->data,
-				 &pp->children[i].data);
+	code = opts->get_next_task(&pp->children[i].process,
+				   pp->ungroup ? NULL : &pp->children[i].err,
+				   pp->data,
+				   &pp->children[i].data);
 	if (!code) {
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
@@ -1665,10 +1640,14 @@ static int pp_start_one(struct parallel_processes *pp)
 	pp->children[i].process.no_stdin = 1;
 
 	if (start_command(&pp->children[i].process)) {
-		code = pp->start_failure(pp->ungroup ? NULL :
-					 &pp->children[i].err,
-					 pp->data,
-					 pp->children[i].data);
+		if (opts->start_failure)
+			code = opts->start_failure(pp->ungroup ? NULL :
+						   &pp->children[i].err,
+						   pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
+
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
@@ -1723,7 +1702,8 @@ static void pp_output(struct parallel_processes *pp)
 	}
 }
 
-static int pp_collect_finished(struct parallel_processes *pp)
+static int pp_collect_finished(struct parallel_processes *pp,
+			       const struct run_process_parallel_opts *opts)
 {
 	int i, code;
 	int n = pp->max_processes;
@@ -1738,9 +1718,12 @@ static int pp_collect_finished(struct parallel_processes *pp)
 
 		code = finish_command(&pp->children[i].process);
 
-		code = pp->task_finished(code, pp->ungroup ? NULL :
-					 &pp->children[i].err, pp->data,
-					 pp->children[i].data);
+		if (opts->task_finished)
+			code = opts->task_finished(code, pp->ungroup ? NULL :
+						   &pp->children[i].err, pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
 
 		if (code)
 			result = code;
@@ -1804,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		    i < spawn_cap && !pp.shutdown &&
 		    pp.nr_processes < pp.max_processes;
 		    i++) {
-			code = pp_start_one(&pp);
+			code = pp_start_one(&pp, opts);
 			if (!code)
 				continue;
 			if (code < 0) {
@@ -1824,7 +1807,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 			pp_buffer_stderr(&pp, output_timeout);
 			pp_output(&pp);
 		}
-		code = pp_collect_finished(&pp);
+		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
 			if (code < 0)
-- 
2.38.0.rc2.935.g6b421ae1592

[PATCH 15/15] run-command.c: don't copy "ungroup" to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in the preceding commit, let's not
copy the "ungroup" member of the "struct run_process_parallel_opts"
over to the "struct parallel_processes". Now that we're passing the
"opts" down there's no reason to do so.

This makes the code easier to follow, as we have a "const" attribute
on the "struct run_process_parallel_opts", but not "struct
parallel_processes". We do not alter the "ungroup" argument, so
storing it in the non-const structure would make this control flow
less obvious.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index b25e74fcc06..da026319335 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1515,7 +1515,6 @@ struct parallel_processes {
 	struct pollfd *pfd;
 
 	unsigned shutdown : 1;
-	unsigned ungroup : 1;
 
 	int output_owner;
 	struct strbuf buffered_output; /* of finished children */
@@ -1563,9 +1562,8 @@ static void pp_init(struct parallel_processes *pp,
 	pp->nr_processes = 0;
 	pp->output_owner = 0;
 	pp->shutdown = 0;
-	pp->ungroup = opts->ungroup;
 	CALLOC_ARRAY(pp->children, opts->jobs);
-	if (!pp->ungroup)
+	if (!opts->ungroup)
 		CALLOC_ARRAY(pp->pfd, opts->jobs);
 
 	for (i = 0; i < opts->jobs; i++) {
@@ -1623,17 +1621,17 @@ static int pp_start_one(struct parallel_processes *pp,
 		BUG("bookkeeping is hard");
 
 	code = opts->get_next_task(&pp->children[i].process,
-				   pp->ungroup ? NULL : &pp->children[i].err,
+				   opts->ungroup ? NULL : &pp->children[i].err,
 				   pp->data,
 				   &pp->children[i].data);
 	if (!code) {
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
 		return 1;
 	}
-	if (!pp->ungroup) {
+	if (!opts->ungroup) {
 		pp->children[i].process.err = -1;
 		pp->children[i].process.stdout_to_stderr = 1;
 	}
@@ -1641,14 +1639,14 @@ static int pp_start_one(struct parallel_processes *pp,
 
 	if (start_command(&pp->children[i].process)) {
 		if (opts->start_failure)
-			code = opts->start_failure(pp->ungroup ? NULL :
+			code = opts->start_failure(opts->ungroup ? NULL :
 						   &pp->children[i].err,
 						   pp->data,
 						   pp->children[i].data);
 		else
 			code = 0;
 
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
@@ -1719,7 +1717,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		code = finish_command(&pp->children[i].process);
 
 		if (opts->task_finished)
-			code = opts->task_finished(code, pp->ungroup ? NULL :
+			code = opts->task_finished(code, opts->ungroup ? NULL :
 						   &pp->children[i].err, pp->data,
 						   pp->children[i].data);
 		else
@@ -1736,7 +1734,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			pp->pfd[i].fd = -1;
 		child_process_init(&pp->children[i].process);
 
-		if (pp->ungroup) {
+		if (opts->ungroup) {
 			; /* no strbuf_*() work to do here */
 		} else if (i != pp->output_owner) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
-- 
2.38.0.rc2.935.g6b421ae1592

Re: [PATCH 00/15] run-command API: pass functions & opts via struct

[Calvin Wan]

Thanks for sending this patch! I looked it over and all of the
refactoring changes you made look good to me. I'll go ahead
and rebase my patch off this.

Re: [PATCH 00/15] run-command API: pass functions & opts via struct

[Phillip Wood]

On 30/09/2022 12:27, Ævar Arnfjörð Bjarmason wrote:
> This series changes the run-command API so that we pass options via a
> struct instead of via the argument list, the end result is that an API
> user looks like e.g.:
> 	
> 	+       const struct run_process_parallel_opts opts = {
> 	+               .tr2_category = "submodule",
> 	+               .tr2_label = "parallel/update",
> 	+
> 	+               .jobs = update_data->max_jobs,
> 	+
> 	+               .get_next_task = update_clone_get_next_task,
> 	+               .start_failure = update_clone_start_failure,
> 	+               .task_finished = update_clone_task_finished,
> 	+               .data = &suc,
> 	+       };
> 	[...]
> 	-       run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
> 	-                                  update_clone_start_failure,
> 	-                                  update_clone_task_finished, &suc, "submodule",
> 	-                                  "parallel/update");
> 	+       run_processes_parallel(&opts);
> 

Overall this looks good, I've left a few comments as I feel some of the 
patches are either redundant or could be helpfully squashed into an 
adjacent patch to reduce the amount of churn within this series.

Best Wishes

Phillip

> These patches are derived from earlier patches I sent for passing the
> the "ungroup" parameter to this API[1], that's currently done with a
> global variable, because we needed a minimal change for a regression
> fix.
> 
> I'm submitting this now in the rc phase because there's another
> concurrent series that could make use of this[2]. The alternative
> would be for it to add an extra parameter to the two functions (one
> after this series).
> 
> The upcoming migration to the new hook API and config-based hooks[3]
> will also benefit significantly from this. I have a version of that
> topic rebased on top of this, having this first gets rid of a lot of
> churn, adding an optional callback just requires adding things to the
> struct introduced here, not changing every single caller.
> 
> (Passing) CI at:
> https://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing
> 
> 1. https://lore.kernel.org/git/cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com/
> 2. https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/
> 3. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/
> 
> Brief commentary on individual patches below:
> 
> Ævar Arnfjörð Bjarmason (15):
>    hook tests: fix redirection logic error in 96e7225b310
>    submodule tests: reset "trace.out" between "grep" invocations
> 
> These bugfixes could be split out I suppose, but since we're changing
> this area 1/15 seemed prudent, 2/15 is required for a later test
> addition
> 
>    run-command tests: test stdout of run_command_parallel()
> 
> Tighten up tests, was ejected from the "ungroup" topic.
> 
>    run-command test helper: use "else if" pattern
>    run-command tests: use "return", not "exit"
> 
> Just setup for later changes.
> 
>    run-command API: have "run_processes_parallel{,_tr2}()" return void
> 
> Turns out we've never used the boilerplate return value for anything
> useful ever.
> 
>    run-command API: make "jobs" parameter an "unsigned int"
>    run-command API: don't fall back on online_cpus()
> 
> This allows us to make the "opts" struct "const", which helps a lot in
> passing it around later on.
> 
>    run-command.c: add an initializer for "struct parallel_processes"
>    run-command API: add nascent "struct run_process_parallel_opts"
>    run-command API: make run_process_parallel{,_tr2}() thin wrappers
>    run-command API: have run_process_parallel() take an "opts" struct
>    run-command API: move *_tr2() users to "run_processes_parallel()"
> 
> This is arguably one logical change (and at some point I had it as
> such), but as the diff would be really large I've tried to split this
> into easily digestable steps.
> 
>    run-command.c: don't copy *_fn to "struct parallel_processes"
>    run-command.c: don't copy "ungroup" to "struct parallel_processes"
> 
> The only reason for copying various parameters into "struct
> parallel_processes" was because we passed them as positionals, now
> that we have an "opts" struct we can just pass that along instead.
> 
> This leaves the "struct parallel_processes" in run-command.c purely
> for managing our own internal state.
> 
> By avoiding this copying we also cut down a lot on the boilerplate
> needed to add a new parameter.
> 
>   builtin/fetch.c             |  25 +++---
>   builtin/submodule--helper.c |  16 +++-
>   hook.c                      |  23 +++---
>   run-command.c               | 152 ++++++++++++++----------------------
>   run-command.h               |  71 ++++++++++++-----
>   submodule-config.c          |   2 +
>   submodule.c                 |  18 +++--
>   t/helper/test-run-command.c |  77 +++++++++++-------
>   t/t0061-run-command.sh      |  25 +++---
>   t/t1800-hook.sh             |   2 +-
>   t/t5526-fetch-submodules.sh |  16 +++-
>   11 files changed, 247 insertions(+), 180 deletions(-)
> 

Re: [PATCH 00/15] run-command API: pass functions & opts via struct

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> Overall this looks good, I've left a few comments as I feel some of
> the patches are either redundant or could be helpfully squashed into
> an adjacent patch to reduce the amount of churn within this series.
>
> Best Wishes
>
> Phillip

Thanks for a review.

[PATCH v2 00/22] run-command API: pass functions & opts via struct

[Ævar Arnfjörð Bjarmason]

This series changes the run-command API so that we pass options via a
struct instead of via the argument list, the end result is that an API
user looks like e.g.:

     +       const struct run_process_parallel_opts opts = {
     +               .tr2_category = "submodule",
     +               .tr2_label = "parallel/update",
     +
     +               .processes = update_data->max_jobs,
     +
     +               .get_next_task = update_clone_get_next_task,
     +               .start_failure = update_clone_start_failure,
     +               .task_finished = update_clone_task_finished,
     +               .data = &suc,
     +       };
     [...]
     -       run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
     -                                  update_clone_start_failure,
     -                                  update_clone_task_finished, &suc, "submodule",
     -                                  "parallel/update");
     +       run_processes_parallel(&opts);

For v1 see
https://lore.kernel.org/git/cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com/;
Changes since v1 (inline comments on listed patches) 

For (passing CI) & a branch for this:
https://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing-2

I've locally rebased Calvin's
https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/
on this, which is based on v1 of this series. There are conflicts, but
they're trivial/easy to solve.

Ævar Arnfjörð Bjarmason (22):
  hook tests: fix redirection logic error in 96e7225b310
  submodule tests: reset "trace.out" between "grep" invocations
  run-command tests: test stdout of run_command_parallel()
  run-command test helper: use "else if" pattern

These are the same.

  run-command API: have "run_processes_parallel{,_tr2}()" return void

Early test change needed to exit(), before we adjust exit() behavior.

  run-command tests: use "return", not "exit"

Ditto.

  run-command.c: remove dead assignment in while-loop
  run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful

These two are new, the C99 for-loop syntax isn't just being used
willy-nilly, but as we were changing a previous "int i" to a "int" and
a "size_t" in some cases this really made things simpler, the
alternative being to split "i" into "i", "j" etc.

  run-command API: make "n" parameter a "size_t"

That "n" to "size_t" change, which the preceding change was prepping
for.

  run-command API: don't fall back on online_cpus()

Phillip Wood wondered why we couldn't keep this, hopefully the later
parts of the series now make the case for it, i.e. we can now nicely
pass all parameters "const" all the way down (not at this point, but
soon...).

  run-command.c: use designated init for pp_init(), add "const"

Starting some of that "const"-ing.

  run-command API: add nascent "struct run_process_parallel_opts"

Renamed the .jobs member to .processes, per feedback.

  run-command API: make run_process_parallel{,_tr2}() thin wrappers
  run-command API: have run_process_parallel() take an "opts" struct
  run-command API: move *_tr2() users to "run_processes_parallel()"
  run-command.c: make "struct parallel_processes" const if possible

Mostly rebasing changes on the above.

  run-command.c: don't copy *_fn to "struct parallel_processes"
  run-command.c: don't copy "ungroup" to "struct parallel_processes"
  run-command.c: don't copy "data" to "struct parallel_processes"
  run-command.c: use "opts->processes", not "pp->max_processes"
  run-command.c: pass "opts" further down, and use "opts->processes"

Only the "ungroup" was here in v1, these all make the API much easier
to extend, as we don't need to copy various data as boilerplate, and
make the ownership clearer.

  run-command.c: remove "pp->max_processes", add "const" to signal()
    handler

In particular here we narrow down a special-case where the signal
handler needs different const-ing.

 builtin/fetch.c             |  25 ++--
 builtin/submodule--helper.c |  16 ++-
 hook.c                      |  23 ++--
 run-command.c               | 249 ++++++++++++++++--------------------
 run-command.h               |  71 +++++++---
 submodule-config.c          |   2 +
 submodule.c                 |  18 ++-
 t/helper/test-run-command.c |  77 +++++++----
 t/t0061-run-command.sh      |  25 ++--
 t/t1800-hook.sh             |   2 +-
 t/t5526-fetch-submodules.sh |  16 ++-
 11 files changed, 296 insertions(+), 228 deletions(-)

Range-diff against v1:
 1:  2e213e70038 =  1:  bc51dfcb1be hook tests: fix redirection logic error in 96e7225b310
 2:  605b15fceab =  2:  3027f5587a7 submodule tests: reset "trace.out" between "grep" invocations
 3:  d8042722e49 =  3:  c4923358bbd run-command tests: test stdout of run_command_parallel()
 4:  6d15fbe6c26 =  4:  26e28086252 run-command test helper: use "else if" pattern
 6:  c86dc59d07c !  5:  5e09dc68fd9 run-command API: have "run_processes_parallel{,_tr2}()" return void
    @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
      
      	if (suite.failed.nr > 0) {
      		ret = 1;
    +@@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
    + 	strvec_pushv(&proc.args, (const char **)argv + 3);
    + 
    + 	if (!strcmp(argv[1], "run-command-parallel")) {
    +-		exit(run_processes_parallel(jobs, parallel_next,
    +-					    NULL, NULL, &proc));
    ++		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
    + 	} else if (!strcmp(argv[1], "run-command-abort")) {
    +-		exit(run_processes_parallel(jobs, parallel_next,
    +-					    NULL, task_finished, &proc));
    ++		run_processes_parallel(jobs, parallel_next, NULL,
    ++				       task_finished, &proc);
    + 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
    +-		exit(run_processes_parallel(jobs, no_job,
    +-					    NULL, task_finished, &proc));
    ++		run_processes_parallel(jobs, no_job, NULL, task_finished,
    ++				       &proc);
    + 	} else {
    + 		fprintf(stderr, "check usage\n");
    + 		return 1;
    + 	}
    ++	exit(0);
    + }
 5:  4ebbf6207fe !  6:  e4e91dbbf9e run-command tests: use "return", not "exit"
    @@ Commit message
         run-command tests: use "return", not "exit"
     
         Change the "run-command" test helper to "return" instead of calling
    -    "exit", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)
    +    "exit", see 338abb0f045 (builtins + test helpers: use return instead
    +    of exit() in cmd_*, 2021-06-08)
     
         Because we'd previously gotten past the SANITIZE=leak check by using
    -    exit() here we need to move to "goto cleanup" pattern. See
    -    fdc8f79f1f1 (leak tests: run various "test-tool" tests in t00*.sh
    -    SANITIZE=leak, 2021-10-12) for prior art. for when this code was opted
    -    into the "linux-leaks" job.
    +    exit() here we need to move to "goto cleanup" pattern.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
      	if (!strcmp(argv[1], "--ungroup")) {
      		argv += 1;
     @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
    - 	strvec_pushv(&proc.args, (const char **)argv + 3);
    - 
    - 	if (!strcmp(argv[1], "run-command-parallel")) {
    --		exit(run_processes_parallel(jobs, parallel_next,
    --					    NULL, NULL, &proc));
    -+		run_processes_parallel(jobs, parallel_next, NULL, NULL,
    -+				       &proc);
    - 	} else if (!strcmp(argv[1], "run-command-abort")) {
    --		exit(run_processes_parallel(jobs, parallel_next,
    --					    NULL, task_finished, &proc));
    -+		run_processes_parallel(jobs, parallel_next, NULL,
    -+				       task_finished, &proc);
    - 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
    --		exit(run_processes_parallel(jobs, no_job,
    --					    NULL, task_finished, &proc));
    -+		run_processes_parallel(jobs, no_job, NULL, task_finished,
    -+				       &proc);
    + 		run_processes_parallel(jobs, no_job, NULL, task_finished,
    + 				       &proc);
      	} else {
     +		ret = 1;
      		fprintf(stderr, "check usage\n");
     -		return 1;
     +		goto cleanup;
      	}
    +-	exit(0);
     +	ret = 0;
     +cleanup:
     +	child_process_clear(&proc);
 -:  ----------- >  7:  b90961ae76d run-command.c: remove dead assignment in while-loop
 -:  ----------- >  8:  279b0430c5d run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful
 7:  a9810aaa852 !  9:  a900711270c run-command API: make "jobs" parameter an "unsigned int"
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    run-command API: make "jobs" parameter an "unsigned int"
    +    run-command API: make "n" parameter a "size_t"
     
    -    The rename the "n" variable added in c553c72eed6 (run-command: add an
    -    asynchronous parallel child processor, 2015-12-15) to "jobs", and
    -    change the type to an "unsigned int". As we'll see in a subsequent
    -    commit we do pass "0" here, but never "jobs < 0".
    +    Make the "n" variable added in c553c72eed6 (run-command: add an
    +    asynchronous parallel child processor, 2015-12-15) a "size_t". As
    +    we'll see in a subsequent commit we do pass "0" here, but never "jobs
    +    < 0".
     
    -    The only users of the "jobs" parameter are:
    +    We could have made it an "unsigned int", but as we're having to change
    +    this let's not leave another case in the codebase where a size_t and
    +    "unsigned int" size differ on some platforms. In this case it's likely
    +    to never matter, but it's easier to not need to worry about it.
    +
    +    After this and preceding changes:
    +
    +            make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
    +
    +    Only has one (and new) -Wsigned-compare warning, about a comparison of
    +    "i" to online_cpus(), a subsequent commit will adjust & deal with
    +    online_cpus() and that warning.
    +
    +    The only users of the "n" parameter are:
     
          * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
            config. As seen in the code that parses the config added in
            d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
            2019-10-05) will die if the git_config_int() return value is < 0.
     
    -       It will however pass us jobs = 0, as we'll see in a subsequent
    -       commit.
    +       It will however pass us n = 0, as we'll see in a subsequent commit.
     
          * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
            config. Read via code originally added in a028a1930c6 (fetching
    @@ Commit message
            submodule.fetchjobs from submodule-config parsing, 2017-08-02).
     
            Like builtin/fetch.c it will die if the git_config_int() return
    -       value is < 0, but like builtin/fetch.c it will pass us jobs = 0.
    +       value is < 0, but like builtin/fetch.c it will pass us n = 0.
     
          * builtin/submodule--helper.c: defaults to 1. Read via code
            originally added in 2335b870fa7 (submodule update: expose parallelism
    @@ run-command.c: int run_processes_parallel_ungroup;
      
     -	int max_processes;
     -	int nr_processes;
    -+	unsigned int max_processes;
    -+	unsigned int nr_processes;
    ++	size_t max_processes;
    ++	size_t nr_processes;
      
      	get_next_task_fn get_next_task;
      	start_failure_fn start_failure;
    +@@ run-command.c: struct parallel_processes {
    + 	unsigned shutdown : 1;
    + 	unsigned ungroup : 1;
    + 
    +-	int output_owner;
    ++	size_t output_owner;
    + 	struct strbuf buffered_output; /* of finished children */
    + };
    + 
    +@@ run-command.c: static int default_task_finished(int result,
    + 
    + static void kill_children(struct parallel_processes *pp, int signo)
    + {
    +-	int i, n = pp->max_processes;
    +-
    +-	for (i = 0; i < n; i++)
    ++	for (size_t i = 0; i < pp->max_processes; i++)
    + 		if (pp->children[i].state == GIT_CP_WORKING)
    + 			kill(pp->children[i].process.pid, signo);
    + }
     @@ run-command.c: static void handle_children_on_signal(int signo)
      }
      
      static void pp_init(struct parallel_processes *pp,
     -		    int n,
    -+		    unsigned int jobs,
    ++		    size_t n,
      		    get_next_task_fn get_next_task,
      		    start_failure_fn start_failure,
      		    task_finished_fn task_finished,
      		    void *data, int ungroup)
      {
     -	int i;
    -+	unsigned int i;
    - 
    --	if (n < 1)
    --		n = online_cpus();
    -+	if (jobs < 1)
    -+		jobs = online_cpus();
    +-
    + 	if (n < 1)
    + 		n = online_cpus();
      
    --	pp->max_processes = n;
    -+	pp->max_processes = jobs;
    + 	pp->max_processes = n;
      
     -	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
    -+	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
    -+		     jobs);
    ++	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
    ++		     (uintmax_t)n);
      
      	pp->data = data;
      	if (!get_next_task)
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    - 	pp->output_owner = 0;
    - 	pp->shutdown = 0;
    - 	pp->ungroup = ungroup;
    --	CALLOC_ARRAY(pp->children, n);
    -+	CALLOC_ARRAY(pp->children, jobs);
    - 	if (pp->ungroup)
    - 		pp->pfd = NULL;
    - 	else
    --		CALLOC_ARRAY(pp->pfd, n);
    -+		CALLOC_ARRAY(pp->pfd, jobs);
    + 		CALLOC_ARRAY(pp->pfd, n);
      	strbuf_init(&pp->buffered_output, 0);
      
     -	for (i = 0; i < n; i++) {
    -+	for (i = 0; i < jobs; i++) {
    ++	for (size_t i = 0; i < n; i++) {
      		strbuf_init(&pp->children[i].err, 0);
      		child_process_init(&pp->children[i].process);
      		if (pp->pfd) {
    +@@ run-command.c: static void pp_init(struct parallel_processes *pp,
    + 
    + static void pp_cleanup(struct parallel_processes *pp)
    + {
    +-	int i;
    +-
    + 	trace_printf("run_processes_parallel: done");
    +-	for (i = 0; i < pp->max_processes; i++) {
    ++	for (size_t i = 0; i < pp->max_processes; i++) {
    + 		strbuf_release(&pp->children[i].err);
    + 		child_process_clear(&pp->children[i].process);
    + 	}
    +@@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
    +  */
    + static int pp_start_one(struct parallel_processes *pp)
    + {
    +-	int i, code;
    ++	size_t i;
    ++	int code;
    + 
    + 	for (i = 0; i < pp->max_processes; i++)
    + 		if (pp->children[i].state == GIT_CP_FREE)
    +@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
    + 
    + static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
    + {
    +-	int i;
    +-
    + 	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
    + 		if (errno == EINTR)
    + 			continue;
    +@@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
    + 	}
    + 
    + 	/* Buffer output from all pipes. */
    +-	for (i = 0; i < pp->max_processes; i++) {
    ++	for (size_t i = 0; i < pp->max_processes; i++) {
    + 		if (pp->children[i].state == GIT_CP_WORKING &&
    + 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
    + 			int n = strbuf_read_once(&pp->children[i].err,
    +@@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
    + 
    + static void pp_output(struct parallel_processes *pp)
    + {
    +-	int i = pp->output_owner;
    ++	size_t i = pp->output_owner;
    + 
    + 	if (pp->children[i].state == GIT_CP_WORKING &&
    + 	    pp->children[i].err.len) {
    +@@ run-command.c: static void pp_output(struct parallel_processes *pp)
    + 
    + static int pp_collect_finished(struct parallel_processes *pp)
    + {
    +-	int i, code;
    +-	int n = pp->max_processes;
    ++	int code;
    ++	size_t i, n = pp->max_processes;
    + 	int result = 0;
    + 
    + 	while (pp->nr_processes > 0) {
     @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
      	return result;
      }
      
     -void run_processes_parallel(int n,
    -+void run_processes_parallel(unsigned int jobs,
    ++void run_processes_parallel(size_t n,
      			    get_next_task_fn get_next_task,
      			    start_failure_fn start_failure,
      			    task_finished_fn task_finished,
     @@ run-command.c: void run_processes_parallel(int n,
    - 	/* unset for the next API user */
    - 	run_processes_parallel_ungroup = 0;
    - 
    --	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
    -+	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
    - 		ungroup);
    - 	while (1) {
    - 		for (i = 0;
    + 		if (!pp.nr_processes)
    + 			break;
    + 		if (ungroup) {
    +-			int i;
    +-
    +-			for (i = 0; i < pp.max_processes; i++)
    ++			for (size_t i = 0; i < pp.max_processes; i++)
    + 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
    + 		} else {
    + 			pp_buffer_stderr(&pp, output_timeout);
     @@ run-command.c: void run_processes_parallel(int n,
      	pp_cleanup(&pp);
      }
      
     -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
    -+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
    ++void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
      				start_failure_fn start_failure,
      				task_finished_fn task_finished, void *pp_cb,
      				const char *tr2_category, const char *tr2_label)
    - {
    - 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
    --				   ((n < 1) ? online_cpus() : n));
    -+				   ((jobs < 1) ? online_cpus() : jobs));
    - 
    --	run_processes_parallel(n, get_next_task, start_failure,
    -+	run_processes_parallel(jobs, get_next_task, start_failure,
    - 			       task_finished, pp_cb);
    - 
    - 	trace2_region_leave(tr2_category, tr2_label, NULL);
     
      ## run-command.h ##
    -@@ run-command.h: typedef int (*task_finished_fn)(int result,
    - 				void *pp_task_cb);
    - 
    - /**
    -- * Runs up to n processes at the same time. Whenever a process can be
    -+ * Runs up to 'jobs' processes at the same time. Whenever a process can be
    -  * started, the callback get_next_task_fn is called to obtain the data
    -  * required to start another child process.
    -  *
     @@ run-command.h: typedef int (*task_finished_fn)(int result,
       * API reads that setting.
       */
      extern int run_processes_parallel_ungroup;
     -void run_processes_parallel(int n,
    -+void run_processes_parallel(unsigned int jobs,
    ++void run_processes_parallel(size_t n,
      			    get_next_task_fn,
      			    start_failure_fn,
      			    task_finished_fn,
      			    void *pp_cb);
     -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
    -+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
    ++void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
      				task_finished_fn, void *pp_cb,
      				const char *tr2_category, const char *tr2_label);
      
 8:  4223980f296 ! 10:  eb9d672b0d8 run-command API: don't fall back on online_cpus()
    @@ builtin/fetch.c: static int git_fetch_config(const char *k, const char *v, void
     
      ## run-command.c ##
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    + 		    task_finished_fn task_finished,
    + 		    void *data, int ungroup)
      {
    - 	unsigned int i;
    +-	if (n < 1)
    +-		n = online_cpus();
    ++	if (!n)
    ++		BUG("you must provide a non-zero number of processes!");
      
    --	if (jobs < 1)
    --		jobs = online_cpus();
    -+	if (!jobs)
    -+		BUG("you must provide a non-zero number of jobs!");
    + 	pp->max_processes = n;
      
    - 	pp->max_processes = jobs;
    - 
    -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
    +@@ run-command.c: void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
    + 				task_finished_fn task_finished, void *pp_cb,
      				const char *tr2_category, const char *tr2_label)
      {
    - 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
    --				   ((jobs < 1) ? online_cpus() : jobs));
    -+				   jobs);
    +-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
    +-				   ((n < 1) ? online_cpus() : n));
    ++	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
      
    - 	run_processes_parallel(jobs, get_next_task, start_failure,
    + 	run_processes_parallel(n, get_next_task, start_failure,
      			       task_finished, pp_cb);
     
      ## submodule-config.c ##
 9:  015edcc42f9 ! 11:  aedda10d8e1 run-command.c: add an initializer for "struct parallel_processes"
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    run-command.c: add an initializer for "struct parallel_processes"
    +    run-command.c: use designated init for pp_init(), add "const"
     
    -    Add a PARALLEL_PROCESSES_INIT macro for the "struct
    -    parallel_processes" used in run-command.c.
    +    Use a designated initializer to initialize those parts of pp_init()
    +    that don't need any conditionals for their initialization, this sets
    +    us on a path to pp_init() itself into mostly a validation and
    +    allocation function.
     
    -    This allows us to do away with a call to strbuf_init(), and to rely on
    -    other fields being NULL'd.
    +    Since we're doing that we can add "const" to some of the members of
    +    the "struct parallel_processes", which helps to clarify and
    +    self-document this code. E.g. we never alter the "data" pointer we
    +    pass t user callbacks, nor (after the preceding change to stop
    +    invoking online_cpus()) do we change "max_processes", the same goes
    +    for the "ungroup" option.
    +
    +    We can also do away with a call to strbuf_init() in favor of macro
    +    initialization, and to rely on other fields being NULL'd or zero'd.
    +
    +    Making members of a struct "const" rather that the pointer to the
    +    struct itself is usually painful, as e.g. it precludes us from
    +    incrementally setting up the structure. In this case we only set it up
    +    with the assignment in run_process_parallel() and pp_init(), and don't
    +    pass the struct pointer around as "const", so making individual
    +    members "const" is worth the potential hassle for extra safety.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## run-command.c ##
    +@@ run-command.c: enum child_state {
    + 
    + int run_processes_parallel_ungroup;
    + struct parallel_processes {
    +-	void *data;
    ++	void *const data;
    + 
    +-	size_t max_processes;
    ++	const size_t max_processes;
    + 	size_t nr_processes;
    + 
    + 	get_next_task_fn get_next_task;
     @@ run-command.c: struct parallel_processes {
    - 	int output_owner;
    + 	struct pollfd *pfd;
    + 
    + 	unsigned shutdown : 1;
    +-	unsigned ungroup : 1;
    ++	const unsigned ungroup : 1;
    + 
    + 	size_t output_owner;
      	struct strbuf buffered_output; /* of finished children */
    - };
    -+#define PARALLEL_PROCESSES_INIT { \
    -+	.buffered_output = STRBUF_INIT, \
    -+}
    +@@ run-command.c: static void handle_children_on_signal(int signo)
    + }
    + 
    + static void pp_init(struct parallel_processes *pp,
    +-		    size_t n,
    + 		    get_next_task_fn get_next_task,
    + 		    start_failure_fn start_failure,
    +-		    task_finished_fn task_finished,
    +-		    void *data, int ungroup)
    ++		    task_finished_fn task_finished)
    + {
    ++	const size_t n = pp->max_processes;
    ++
    + 	if (!n)
    + 		BUG("you must provide a non-zero number of processes!");
    + 
    +-	pp->max_processes = n;
    +-
    + 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
    + 		     (uintmax_t)n);
      
    - static int default_start_failure(struct strbuf *out,
    - 				 void *pp_cb,
    +-	pp->data = data;
    + 	if (!get_next_task)
    + 		BUG("you need to specify a get_next_task function");
    + 	pp->get_next_task = get_next_task;
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    - 	pp->shutdown = 0;
    - 	pp->ungroup = ungroup;
    - 	CALLOC_ARRAY(pp->children, jobs);
    + 	pp->start_failure = start_failure ? start_failure : default_start_failure;
    + 	pp->task_finished = task_finished ? task_finished : default_task_finished;
    + 
    +-	pp->nr_processes = 0;
    +-	pp->output_owner = 0;
    +-	pp->shutdown = 0;
    +-	pp->ungroup = ungroup;
    + 	CALLOC_ARRAY(pp->children, n);
     -	if (pp->ungroup)
     -		pp->pfd = NULL;
     -	else
     +	if (!pp->ungroup)
    - 		CALLOC_ARRAY(pp->pfd, jobs);
    + 		CALLOC_ARRAY(pp->pfd, n);
     -	strbuf_init(&pp->buffered_output, 0);
      
    - 	for (i = 0; i < jobs; i++) {
    + 	for (size_t i = 0; i < n; i++) {
      		strbuf_init(&pp->children[i].err, 0);
    -@@ run-command.c: void run_processes_parallel(unsigned int jobs,
    +@@ run-command.c: void run_processes_parallel(size_t n,
      	int output_timeout = 100;
      	int spawn_cap = 4;
      	int ungroup = run_processes_parallel_ungroup;
     -	struct parallel_processes pp;
    -+	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
    ++	struct parallel_processes pp = {
    ++		.max_processes = n,
    ++		.data = pp_cb,
    ++		.buffered_output = STRBUF_INIT,
    ++		.ungroup = ungroup,
    ++	};
      
      	/* unset for the next API user */
      	run_processes_parallel_ungroup = 0;
    + 
    +-	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
    +-		ungroup);
    ++	pp_init(&pp, get_next_task, start_failure, task_finished);
    + 	while (1) {
    + 		for (int i = 0;
    + 		    i < spawn_cap && !pp.shutdown &&
10:  613ccb85fa2 ! 12:  fde2af11579 run-command API: add nascent "struct run_process_parallel_opts"
    @@ Commit message
         itself, as well as the "run_processes_parallel_tr2()" wrapper
         function.
     
    +    Since we need to change all of the occurrences of "n" to
    +    "opt->SOMETHING" let's take the opportunity and rename the terse "n"
    +    to "processes". We could also have picked "max_processes", "jobs",
    +    "threads" etc., but as the API is named "run_processes_parallel()"
    +    let's go with "processes".
    +
         1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
    @@ run-command.c: static void handle_children_on_signal(int signo)
      }
      
      static void pp_init(struct parallel_processes *pp,
    --		    unsigned int jobs,
     -		    get_next_task_fn get_next_task,
     -		    start_failure_fn start_failure,
    --		    task_finished_fn task_finished,
    --		    void *data, int ungroup)
    +-		    task_finished_fn task_finished)
     +		    const struct run_process_parallel_opts *opts)
      {
    - 	unsigned int i;
    -+	void *data = opts->data;
    +-	const size_t n = pp->max_processes;
    ++	const size_t n = opts->processes;
     +	get_next_task_fn get_next_task = opts->get_next_task;
     +	start_failure_fn start_failure = opts->start_failure;
     +	task_finished_fn task_finished = opts->task_finished;
      
    --	if (!jobs)
    -+	if (!opts->jobs)
    - 		BUG("you must provide a non-zero number of jobs!");
    - 
    --	pp->max_processes = jobs;
    -+	pp->max_processes = opts->jobs;
    - 
    - 	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
    --		     jobs);
    -+		     opts->jobs);
    - 
    - 	pp->data = data;
    - 	if (!get_next_task)
    -@@ run-command.c: static void pp_init(struct parallel_processes *pp,
    - 	pp->nr_processes = 0;
    - 	pp->output_owner = 0;
    - 	pp->shutdown = 0;
    --	pp->ungroup = ungroup;
    --	CALLOC_ARRAY(pp->children, jobs);
    -+	pp->ungroup = opts->ungroup;
    -+	CALLOC_ARRAY(pp->children, opts->jobs);
    - 	if (!pp->ungroup)
    --		CALLOC_ARRAY(pp->pfd, jobs);
    -+		CALLOC_ARRAY(pp->pfd, opts->jobs);
    + 	if (!n)
    + 		BUG("you must provide a non-zero number of processes!");
    +@@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
    + 	return result;
    + }
      
    --	for (i = 0; i < jobs; i++) {
    -+	for (i = 0; i < opts->jobs; i++) {
    - 		strbuf_init(&pp->children[i].err, 0);
    - 		child_process_init(&pp->children[i].process);
    - 		if (pp->pfd) {
    -@@ run-command.c: void run_processes_parallel(unsigned int jobs,
    - 	int i, code;
    +-void run_processes_parallel(size_t n,
    ++void run_processes_parallel(size_t processes,
    + 			    get_next_task_fn get_next_task,
    + 			    start_failure_fn start_failure,
    + 			    task_finished_fn task_finished,
    +@@ run-command.c: void run_processes_parallel(size_t n,
    + 	int code;
      	int output_timeout = 100;
      	int spawn_cap = 4;
     -	int ungroup = run_processes_parallel_ungroup;
    - 	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
    + 	struct parallel_processes pp = {
    +-		.max_processes = n,
    ++		.max_processes = processes,
    + 		.data = pp_cb,
    + 		.buffered_output = STRBUF_INIT,
    +-		.ungroup = ungroup,
    ++		.ungroup = run_processes_parallel_ungroup,
    ++	};
     +	const struct run_process_parallel_opts opts = {
    -+		.jobs = jobs,
    ++		.processes = processes,
    ++
     +		.get_next_task = get_next_task,
     +		.start_failure = start_failure,
     +		.task_finished = task_finished,
    ++
     +		.ungroup = run_processes_parallel_ungroup,
    -+		.data = pp_cb,
    -+	};
    + 	};
      
      	/* unset for the next API user */
      	run_processes_parallel_ungroup = 0;
      
    --	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
    --		ungroup);
    +-	pp_init(&pp, get_next_task, start_failure, task_finished);
     +	pp_init(&pp, &opts);
    ++
      	while (1) {
    - 		for (i = 0;
    + 		for (int i = 0;
      		    i < spawn_cap && !pp.shutdown &&
    -@@ run-command.c: void run_processes_parallel(unsigned int jobs,
    +@@ run-command.c: void run_processes_parallel(size_t n,
      		}
      		if (!pp.nr_processes)
      			break;
     -		if (ungroup) {
     +		if (opts.ungroup) {
    - 			int i;
    - 
    - 			for (i = 0; i < pp.max_processes; i++)
    + 			for (size_t i = 0; i < pp.max_processes; i++)
    + 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
    + 		} else {
     
      ## run-command.h ##
     @@ run-command.h: typedef int (*task_finished_fn)(int result,
    - 				void *pp_cb,
      				void *pp_task_cb);
      
    -+/**
    + /**
    +- * Runs up to n processes at the same time. Whenever a process can be
     + * Option used by run_processes_parallel(), { 0 }-initialized means no
     + * options.
     + */
     +struct run_process_parallel_opts
     +{
     +	/**
    -+	 * jobs: see 'jobs' in run_processes_parallel() below.
    ++	 * processes: see 'processes' in run_processes_parallel() below.
     +	 */
    -+	int jobs;
    ++	size_t processes;
     +
     +	/**
     +	 * ungroup: see 'ungroup' in run_processes_parallel() below.
    @@ run-command.h: typedef int (*task_finished_fn)(int result,
     +	 * NULL to omit any special handling.
     +	 */
     +	task_finished_fn task_finished;
    -+
    -+	/**
    -+	 * data: user data, will be passed as "pp_cb" to the callback
    -+	 * parameters.
    -+	 */
    -+	void *data;
     +};
     +
    - /**
    -  * Runs up to 'jobs' processes at the same time. Whenever a process can be
    ++/**
    ++ * Runs N 'processes' at the same time. Whenever a process can be
       * started, the callback get_next_task_fn is called to obtain the data
    +  * required to start another child process.
    +  *
     @@ run-command.h: typedef int (*task_finished_fn)(int result,
       * (both stdout and stderr) is routed to stderr in a manner that output
       * from different tasks does not interleave (but see "ungroup" below).
    @@ run-command.h: typedef int (*task_finished_fn)(int result,
       * If the "ungroup" option isn't specified, the API will set the
       * "stdout_to_stderr" parameter in "struct child_process" and provide
       * the callbacks with a "struct strbuf *out" parameter to write output
    +@@ run-command.h: typedef int (*task_finished_fn)(int result,
    +  * API reads that setting.
    +  */
    + extern int run_processes_parallel_ungroup;
    +-void run_processes_parallel(size_t n,
    ++void run_processes_parallel(size_t processes,
    + 			    get_next_task_fn,
    + 			    start_failure_fn,
    + 			    task_finished_fn,
    + 			    void *pp_cb);
    +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
    +-				task_finished_fn, void *pp_cb,
    ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
    ++				start_failure_fn, task_finished_fn, void *pp_cb,
    + 				const char *tr2_category, const char *tr2_label);
    + 
    + /**
11:  bad36eabfe9 ! 13:  01e894bed90 run-command API: make run_process_parallel{,_tr2}() thin wrappers
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
      	return result;
      }
      
    --void run_processes_parallel(unsigned int jobs,
    +-void run_processes_parallel(size_t processes,
     -			    get_next_task_fn get_next_task,
     -			    start_failure_fn start_failure,
     -			    task_finished_fn task_finished,
     -			    void *pp_cb)
    -+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)
    ++static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
      {
    - 	int i, code;
    + 	int code;
      	int output_timeout = 100;
      	int spawn_cap = 4;
    - 	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
    + 	struct parallel_processes pp = {
    +-		.max_processes = processes,
    ++		.max_processes = opts->processes,
    + 		.data = pp_cb,
    + 		.buffered_output = STRBUF_INIT,
    + 		.ungroup = run_processes_parallel_ungroup,
    + 	};
     -	const struct run_process_parallel_opts opts = {
    --		.jobs = jobs,
    +-		.processes = processes,
    +-
     -		.get_next_task = get_next_task,
     -		.start_failure = start_failure,
     -		.task_finished = task_finished,
    +-
     -		.ungroup = run_processes_parallel_ungroup,
    --		.data = pp_cb,
     -	};
     +	/* options */
     +	const char *tr2_category = opts->tr2_category;
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
     -	pp_init(&pp, &opts);
     +	if (do_trace2)
     +		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
    -+					   "max:%d", opts->jobs);
    -+
    ++					   "max:%d", opts->processes);
    + 
     +	pp_init(&pp, opts);
      	while (1) {
    - 		for (i = 0;
    + 		for (int i = 0;
      		    i < spawn_cap && !pp.shutdown &&
    -@@ run-command.c: void run_processes_parallel(unsigned int jobs,
    +@@ run-command.c: void run_processes_parallel(size_t processes,
      		}
      		if (!pp.nr_processes)
      			break;
     -		if (opts.ungroup) {
     +		if (opts->ungroup) {
    - 			int i;
    - 
    - 			for (i = 0; i < pp.max_processes; i++)
    -@@ run-command.c: void run_processes_parallel(unsigned int jobs,
    + 			for (size_t i = 0; i < pp.max_processes; i++)
    + 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
    + 		} else {
    +@@ run-command.c: void run_processes_parallel(size_t processes,
      	}
      
      	pp_cleanup(&pp);
    @@ run-command.c: void run_processes_parallel(unsigned int jobs,
     +		trace2_region_leave(tr2_category, tr2_label, NULL);
     +}
     +
    -+void run_processes_parallel(unsigned int jobs,
    ++void run_processes_parallel(size_t processes,
     +			    get_next_task_fn get_next_task,
     +			    start_failure_fn start_failure,
     +			    task_finished_fn task_finished,
     +			    void *pp_cb)
     +{
     +	const struct run_process_parallel_opts opts = {
    -+		.jobs = jobs,
    ++		.processes = processes,
     +		.ungroup = run_processes_parallel_ungroup,
     +
     +		.get_next_task = get_next_task,
     +		.start_failure = start_failure,
     +		.task_finished = task_finished,
    -+
    -+		.data = pp_cb,
     +	};
     +
    -+	run_processes_parallel_1(&opts);
    ++	run_processes_parallel_1(&opts, pp_cb);
      }
      
    - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
    -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
    +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
    ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
    + 				start_failure_fn start_failure,
      				task_finished_fn task_finished, void *pp_cb,
      				const char *tr2_category, const char *tr2_label)
      {
    --	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
    --				   jobs);
    +-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
     +	const struct run_process_parallel_opts opts = {
     +		.tr2_category = tr2_category,
     +		.tr2_label = tr2_label,
     +
    -+		.jobs = jobs,
    ++		.processes = processes,
     +		.ungroup = run_processes_parallel_ungroup,
      
    --	run_processes_parallel(jobs, get_next_task, start_failure,
    +-	run_processes_parallel(n, get_next_task, start_failure,
     -			       task_finished, pp_cb);
     +		.get_next_task = get_next_task,
     +		.start_failure = start_failure,
     +		.task_finished = task_finished,
    -+
    -+		.data = pp_cb,
     +	};
      
     -	trace2_region_leave(tr2_category, tr2_label, NULL);
    -+	run_processes_parallel_1(&opts);
    ++	run_processes_parallel_1(&opts, pp_cb);
      }
      
      int run_auto_maintenance(int quiet)
    @@ run-command.h: typedef int (*task_finished_fn)(int result,
     +	const char *tr2_label;
     +
      	/**
    - 	 * jobs: see 'jobs' in run_processes_parallel() below.
    + 	 * processes: see 'processes' in run_processes_parallel() below.
      	 */
12:  eeaac80fbac ! 14:  41c2886b44b run-command API: have run_process_parallel() take an "opts" struct
    @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
     +		.tr2_category = "hook",
     +		.tr2_label = hook_name,
     +
    -+		.jobs = 1,
    ++		.processes = 1,
     +		.ungroup = 1,
     +
     +		.get_next_task = pick_next_hook,
    @@ run-command.c: enum child_state {
      
     -int run_processes_parallel_ungroup;
      struct parallel_processes {
    - 	void *data;
    + 	void *const data;
      
     @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
      	return result;
      }
      
    --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)
    +-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
     +void run_processes_parallel(const struct run_process_parallel_opts *opts)
      {
    - 	int i, code;
    + 	int code;
      	int output_timeout = 100;
    -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
    + 	int spawn_cap = 4;
    + 	struct parallel_processes pp = {
    + 		.max_processes = opts->processes,
    +-		.data = pp_cb,
    ++		.data = opts->data,
    + 		.buffered_output = STRBUF_INIT,
    +-		.ungroup = run_processes_parallel_ungroup,
    ++		.ungroup = opts->ungroup,
    + 	};
    + 	/* options */
    + 	const char *tr2_category = opts->tr2_category;
      	const char *tr2_label = opts->tr2_label;
      	const int do_trace2 = tr2_category && tr2_label;
      
    @@ run-command.c: static void run_processes_parallel_1(const struct run_process_par
     -
      	if (do_trace2)
      		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
    - 					   "max:%d", opts->jobs);
    + 					   "max:%d", opts->processes);
     @@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
      		trace2_region_leave(tr2_category, tr2_label, NULL);
      }
      
    --void run_processes_parallel(unsigned int jobs,
    +-void run_processes_parallel(size_t processes,
     -			    get_next_task_fn get_next_task,
     -			    start_failure_fn start_failure,
     -			    task_finished_fn task_finished,
     -			    void *pp_cb)
     -{
     -	const struct run_process_parallel_opts opts = {
    --		.jobs = jobs,
    +-		.processes = processes,
     -		.ungroup = run_processes_parallel_ungroup,
     -
     -		.get_next_task = get_next_task,
     -		.start_failure = start_failure,
     -		.task_finished = task_finished,
    --
    --		.data = pp_cb,
     -	};
     -
    --	run_processes_parallel_1(&opts);
    +-	run_processes_parallel_1(&opts, pp_cb);
     -}
     -
    - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
    + void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
      				start_failure_fn start_failure,
      				task_finished_fn task_finished, void *pp_cb,
    -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
    +@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task
    + 	const struct run_process_parallel_opts opts = {
    + 		.tr2_category = tr2_category,
      		.tr2_label = tr2_label,
    - 
    - 		.jobs = jobs,
    +-
    + 		.processes = processes,
     -		.ungroup = run_processes_parallel_ungroup,
      
      		.get_next_task = get_next_task,
      		.start_failure = start_failure,
    -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas
    - 		.data = pp_cb,
    + 		.task_finished = task_finished,
      	};
      
    --	run_processes_parallel_1(&opts);
    +-	run_processes_parallel_1(&opts, pp_cb);
     +	run_processes_parallel(&opts);
      }
      
    @@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_
     
      ## run-command.h ##
     @@ run-command.h: struct run_process_parallel_opts
    + 	 * NULL to omit any special handling.
    + 	 */
    + 	task_finished_fn task_finished;
    ++
    ++	/**
    ++	 * data: user data, will be passed as "pp_cb" to the callback
    ++	 * parameters.
    ++	 */
    ++	void *data;
      };
      
      /**
     + * Options are passed via the "struct run_process_parallel_opts" above.
     + *
    -  * Runs up to 'jobs' processes at the same time. Whenever a process can be
    +  * Runs N 'processes' at the same time. Whenever a process can be
     - * started, the callback get_next_task_fn is called to obtain the data
     + * started, the callback opts.get_next_task is called to obtain the data
       * required to start another child process.
    @@ run-command.h: struct run_process_parallel_opts
     - * API reads that setting.
       */
     -extern int run_processes_parallel_ungroup;
    --void run_processes_parallel(unsigned int jobs,
    +-void run_processes_parallel(size_t processes,
     -			    get_next_task_fn,
     -			    start_failure_fn,
     -			    task_finished_fn,
     -			    void *pp_cb);
     +void run_processes_parallel(const struct run_process_parallel_opts *opts);
    - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
    - 				task_finished_fn, void *pp_cb,
    + void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
    + 				start_failure_fn, task_finished_fn, void *pp_cb,
      				const char *tr2_category, const char *tr2_label);
     
      ## t/helper/test-run-command.c ##
    @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
      
     -	run_processes_parallel(max_jobs, next_test, test_failed,
     -			       test_finished, &suite);
    -+	opts.jobs = max_jobs;
    ++	opts.processes = max_jobs;
     +	run_processes_parallel(&opts);
      
      	if (suite.failed.nr > 0) {
    @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
      	strvec_pushv(&proc.args, (const char **)argv + 3);
      
      	if (!strcmp(argv[1], "run-command-parallel")) {
    --		run_processes_parallel(jobs, parallel_next, NULL, NULL,
    --				       &proc);
    +-		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
     +		opts.get_next_task = parallel_next;
      	} else if (!strcmp(argv[1], "run-command-abort")) {
     -		run_processes_parallel(jobs, parallel_next, NULL,
    @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)
      		fprintf(stderr, "check usage\n");
      		goto cleanup;
      	}
    -+	opts.jobs = jobs;
    ++	opts.processes = jobs;
     +	run_processes_parallel(&opts);
      	ret = 0;
      cleanup:
13:  37d194a338f ! 15:  391d1d99d91 run-command API: move *_tr2() users to "run_processes_parallel()"
    @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi
     +			.tr2_category = "fetch",
     +			.tr2_label = "parallel/fetch",
     +
    -+			.jobs = max_children,
    ++			.processes = max_children,
     +
     +			.get_next_task = &fetch_next_remote,
     +			.start_failure = &fetch_failed_to_start,
    @@ builtin/submodule--helper.c: static int update_submodules(struct update_data *up
     +		.tr2_category = "submodule",
     +		.tr2_label = "parallel/update",
     +
    -+		.jobs = update_data->max_jobs,
    ++		.processes = update_data->max_jobs,
     +
     +		.get_next_task = update_clone_get_next_task,
     +		.start_failure = update_clone_start_failure,
    @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
      		trace2_region_leave(tr2_category, tr2_label, NULL);
      }
      
    --void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,
    +-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
     -				start_failure_fn start_failure,
     -				task_finished_fn task_finished, void *pp_cb,
     -				const char *tr2_category, const char *tr2_label)
    @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
     -	const struct run_process_parallel_opts opts = {
     -		.tr2_category = tr2_category,
     -		.tr2_label = tr2_label,
    --
    --		.jobs = jobs,
    +-		.processes = processes,
     -
     -		.get_next_task = get_next_task,
     -		.start_failure = start_failure,
     -		.task_finished = task_finished,
    --
    --		.data = pp_cb,
     -	};
     -
     -	run_processes_parallel(&opts);
    @@ run-command.h: struct run_process_parallel_opts
       * conditions due to writing in parallel to stdout and stderr.
       */
      void run_processes_parallel(const struct run_process_parallel_opts *opts);
    --void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,
    --				task_finished_fn, void *pp_cb,
    +-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
    +-				start_failure_fn, task_finished_fn, void *pp_cb,
     -				const char *tr2_category, const char *tr2_label);
      
      /**
    @@ submodule.c: int fetch_submodules(struct repository *r,
     +		.tr2_category = "submodule",
     +		.tr2_label = "parallel/fetch",
     +
    -+		.jobs = max_parallel_jobs,
    ++		.processes = max_parallel_jobs,
     +
     +		.get_next_task = get_next_submodule,
     +		.start_failure = fetch_start_failure,
 -:  ----------- > 16:  acac50cc1a5 run-command.c: make "struct parallel_processes" const if possible
14:  68605f4034b ! 17:  fdd64236985 run-command.c: don't copy *_fn to "struct parallel_processes"
    @@ Commit message
     
      ## run-command.c ##
     @@ run-command.c: struct parallel_processes {
    - 	unsigned int max_processes;
    - 	unsigned int nr_processes;
    + 	const size_t max_processes;
    + 	size_t nr_processes;
      
     -	get_next_task_fn get_next_task;
     -	start_failure_fn start_failure;
    @@ run-command.c: struct parallel_processes {
      		enum child_state state;
      		struct child_process process;
     @@ run-command.c: struct parallel_processes {
    - 	.buffered_output = STRBUF_INIT, \
    - }
    + 	struct strbuf buffered_output; /* of finished children */
    + };
      
     -static int default_start_failure(struct strbuf *out,
     -				 void *pp_cb,
    @@ run-command.c: struct parallel_processes {
     -	return 0;
     -}
     -
    - static void kill_children(struct parallel_processes *pp, int signo)
    + static void kill_children(const struct parallel_processes *pp, int signo)
      {
    - 	int i, n = pp->max_processes;
    + 	for (size_t i = 0; i < pp->max_processes; i++)
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    + 		    const struct run_process_parallel_opts *opts)
      {
    - 	unsigned int i;
    - 	void *data = opts->data;
    + 	const size_t n = opts->processes;
     -	get_next_task_fn get_next_task = opts->get_next_task;
     -	start_failure_fn start_failure = opts->start_failure;
     -	task_finished_fn task_finished = opts->task_finished;
      
    - 	if (!opts->jobs)
    - 		BUG("you must provide a non-zero number of jobs!");
    + 	if (!n)
    + 		BUG("you must provide a non-zero number of processes!");
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    - 		     opts->jobs);
    + 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
    + 		     (uintmax_t)n);
      
    - 	pp->data = data;
     -	if (!get_next_task)
     +	if (!opts->get_next_task)
      		BUG("you need to specify a get_next_task function");
    @@ run-command.c: static void pp_init(struct parallel_processes *pp,
     -	pp->start_failure = start_failure ? start_failure : default_start_failure;
     -	pp->task_finished = task_finished ? task_finished : default_task_finished;
      
    - 	pp->nr_processes = 0;
    - 	pp->output_owner = 0;
    + 	CALLOC_ARRAY(pp->children, n);
    + 	if (!pp->ungroup)
     @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
       * <0 no new job was started, user wishes to shutdown early. Use negative code
       *    to signal the children.
    @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
     +static int pp_start_one(struct parallel_processes *pp,
     +			const struct run_process_parallel_opts *opts)
      {
    - 	int i, code;
    - 
    + 	size_t i;
    + 	int code;
     @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
      	if (i == pp->max_processes)
      		BUG("bookkeeping is hard");
    @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
      		if (!pp->ungroup) {
      			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
      			strbuf_reset(&pp->children[i].err);
    -@@ run-command.c: static void pp_output(struct parallel_processes *pp)
    +@@ run-command.c: static void pp_output(const struct parallel_processes *pp)
      	}
      }
      
    @@ run-command.c: static void pp_output(struct parallel_processes *pp)
     +static int pp_collect_finished(struct parallel_processes *pp,
     +			       const struct run_process_parallel_opts *opts)
      {
    - 	int i, code;
    - 	int n = pp->max_processes;
    + 	int code;
    + 	size_t i, n = pp->max_processes;
     @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
      
      		code = finish_command(&pp->children[i].process);
15:  60d66fbd90a ! 18:  17f34d81ecd run-command.c: don't copy "ungroup" to "struct parallel_processes"
    @@ run-command.c: struct parallel_processes {
      	struct pollfd *pfd;
      
      	unsigned shutdown : 1;
    --	unsigned ungroup : 1;
    +-	const unsigned ungroup : 1;
      
    - 	int output_owner;
    + 	size_t output_owner;
      	struct strbuf buffered_output; /* of finished children */
     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
    - 	pp->nr_processes = 0;
    - 	pp->output_owner = 0;
    - 	pp->shutdown = 0;
    --	pp->ungroup = opts->ungroup;
    - 	CALLOC_ARRAY(pp->children, opts->jobs);
    + 		BUG("you need to specify a get_next_task function");
    + 
    + 	CALLOC_ARRAY(pp->children, n);
     -	if (!pp->ungroup)
     +	if (!opts->ungroup)
    - 		CALLOC_ARRAY(pp->pfd, opts->jobs);
    + 		CALLOC_ARRAY(pp->pfd, n);
      
    - 	for (i = 0; i < opts->jobs; i++) {
    + 	for (size_t i = 0; i < n; i++) {
     @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
      		BUG("bookkeeping is hard");
      
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
      			; /* no strbuf_*() work to do here */
      		} else if (i != pp->output_owner) {
      			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
    +@@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
    + 		.max_processes = opts->processes,
    + 		.data = opts->data,
    + 		.buffered_output = STRBUF_INIT,
    +-		.ungroup = opts->ungroup,
    + 	};
    + 	/* options */
    + 	const char *tr2_category = opts->tr2_category;
 -:  ----------- > 19:  9cbee2dfe76 run-command.c: don't copy "data" to "struct parallel_processes"
 -:  ----------- > 20:  2dabed9e155 run-command.c: use "opts->processes", not "pp->max_processes"
 -:  ----------- > 21:  c1a286a8ebb run-command.c: pass "opts" further down, and use "opts->processes"
 -:  ----------- > 22:  541f41566e7 run-command.c: remove "pp->max_processes", add "const" to signal() handler
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 01/22] hook tests: fix redirection logic error in 96e7225b310

[Ævar Arnfjörð Bjarmason]

The tests added in 96e7225b310 (hook: add 'run' subcommand,
2021-12-22) were redirecting to "actual" both in the body of the hook
itself and in the testing code below.

The net result was that the "2>>actual" redirection later in the test
wasn't doing anything. Let's have those redirection do what it looks
like they're doing.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t1800-hook.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh
index 43fcb7c0bfc..2ef3579fa7c 100755
--- a/t/t1800-hook.sh
+++ b/t/t1800-hook.sh
@@ -95,7 +95,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '
 test_expect_success 'git -c core.hooksPath=<PATH> hook run' '
 	mkdir my-hooks &&
 	write_script my-hooks/test-hook <<-\EOF &&
-	echo Hook ran $1 >>actual
+	echo Hook ran $1
 	EOF
 
 	cat >expect <<-\EOF &&
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 02/22] submodule tests: reset "trace.out" between "grep" invocations

[Ævar Arnfjörð Bjarmason]

Fix test patterns added in 62104ba14af (submodules: allow parallel
fetching, add tests and documentation, 2015-12-15) and
a028a1930c6 (fetching submodules: respect `submodule.fetchJobs` config
option, 2016-02-29).

In the former case we were leaving a trace.out file at the top-level
for any subsequent tests (there are none, currently). Let's clean the
file up instead.

In the latter case we were testing that a given configuration would
result in "N tasks" in the log, but we were grepping through the log
for all previous such tests, when we really meant to clear the logs
between the "grep" invocations.

In practice this resulted in no logic error, as e.g. "--fetch 7" would
not print out a "9 tasks" line, but let's be paranoid and stop
implicitly assuming that that's the case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t5526-fetch-submodules.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index a301b56db89..e36f9fdf242 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -177,6 +177,7 @@ test_expect_success "submodule.recurse option triggers recursive fetch" '
 '
 
 test_expect_success "fetch --recurse-submodules -j2 has the same output behaviour" '
+	test_when_finished "rm -f trace.out" &&
 	add_submodule_commits &&
 	(
 		cd downstream &&
@@ -704,17 +705,25 @@ test_expect_success "'fetch.recurseSubmodules=on-demand' works also without .git
 
 test_expect_success 'fetching submodules respects parallel settings' '
 	git config fetch.recurseSubmodules true &&
+	test_when_finished "rm -f downstream/trace.out" &&
 	(
 		cd downstream &&
 		GIT_TRACE=$(pwd)/trace.out git fetch &&
 		grep "1 tasks" trace.out &&
+		>trace.out &&
+
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&
 		grep "7 tasks" trace.out &&
+		>trace.out &&
+
 		git config submodule.fetchJobs 8 &&
 		GIT_TRACE=$(pwd)/trace.out git fetch &&
 		grep "8 tasks" trace.out &&
+		>trace.out &&
+
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
-		grep "9 tasks" trace.out
+		grep "9 tasks" trace.out &&
+		>trace.out &&
 	)
 '
 
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 03/22] run-command tests: test stdout of run_command_parallel()

[Ævar Arnfjörð Bjarmason]

Extend the tests added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) to test stdout in
addition to stderr.

When the "ungroup" feature was added in fd3aaf53f71 (run-command: add
an "ungroup" option to run_process_parallel(), 2022-06-07) its tests
were made to test both the stdout and stderr, but these existing tests
were left alone. Let's also exhaustively test our expected output
here.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t0061-run-command.sh | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh
index 7b5423eebda..19af082750a 100755
--- a/t/t0061-run-command.sh
+++ b/t/t0061-run-command.sh
@@ -130,8 +130,9 @@ World
 EOF
 
 test_expect_success 'run_command runs in parallel with more jobs available than tasks' '
-	test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '
@@ -141,8 +142,9 @@ test_expect_success 'run_command runs ungrouped in parallel with more jobs avail
 '
 
 test_expect_success 'run_command runs in parallel with as many jobs as tasks' '
-	test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '
@@ -152,8 +154,9 @@ test_expect_success 'run_command runs ungrouped in parallel with as many jobs as
 '
 
 test_expect_success 'run_command runs in parallel with more tasks than jobs available' '
-	test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '
@@ -172,8 +175,9 @@ asking for a quick stop
 EOF
 
 test_expect_success 'run_command is asked to abort gracefully' '
-	test-tool run-command run-command-abort 3 false 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-abort 3 false >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command is asked to abort gracefully (ungroup)' '
@@ -187,8 +191,9 @@ no further jobs available
 EOF
 
 test_expect_success 'run_command outputs ' '
-	test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" 2>actual &&
-	test_cmp expect actual
+	test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err &&
+	test_must_be_empty out &&
+	test_cmp expect err
 '
 
 test_expect_success 'run_command outputs (ungroup) ' '
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 04/22] run-command test helper: use "else if" pattern

[Ævar Arnfjörð Bjarmason]

Adjust the cmd__run_command() to use an "if/else if" chain rather than
mutually exclusive "if" statements. This non-functional change makes a
subsequent commit smaller.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index c9283b47afa..390fa4fb724 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_clear(&proc.args);
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
-	if (!strcmp(argv[1], "run-command-parallel"))
+	if (!strcmp(argv[1], "run-command-parallel")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, NULL, &proc));
-
-	if (!strcmp(argv[1], "run-command-abort"))
+	} else if (!strcmp(argv[1], "run-command-abort")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, task_finished, &proc));
-
-	if (!strcmp(argv[1], "run-command-no-jobs"))
+	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		exit(run_processes_parallel(jobs, no_job,
 					    NULL, task_finished, &proc));
-
-	fprintf(stderr, "check usage\n");
-	return 1;
+	} else {
+		fprintf(stderr, "check usage\n");
+		return 1;
+	}
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 05/22] run-command API: have "run_processes_parallel{,_tr2}()" return void

[Ævar Arnfjörð Bjarmason]

Change the "run_processes_parallel{,_tr2}()" functions to return void,
instead of int. Ever since c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) they have
unconditionally returned 0.

To get a "real" return value out of this function the caller needs to
get it via the "task_finished_fn" callback, see the example in hook.c
added in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).

So the "result = " and "if (!result)" code added to "builtin/fetch.c"
d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
2019-10-05) has always been redundant, we always took that "if"
path. Likewise the "ret =" in "t/helper/test-run-command.c" added in
be5d88e1128 (test-tool run-command: learn to run (parts of) the
testsuite, 2019-10-04) wasn't used, instead we got the return value
from the "if (suite.failed.nr > 0)" block seen in the context.

Subsequent commits will alter this API interface, getting rid of this
always-zero return value makes it easier to understand those changes.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 17 ++++++++---------
 run-command.c               | 27 +++++++++++----------------
 run-command.h               | 16 ++++++++--------
 t/helper/test-run-command.c | 16 ++++++++--------
 4 files changed, 35 insertions(+), 41 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index a0fca93bb6a..78043fb67ef 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
 
 		strvec_push(&argv, "--end-of-options");
-		result = run_processes_parallel_tr2(max_children,
-						    &fetch_next_remote,
-						    &fetch_failed_to_start,
-						    &fetch_finished,
-						    &state,
-						    "fetch", "parallel/fetch");
-
-		if (!result)
-			result = state.result;
+		run_processes_parallel_tr2(max_children,
+					   &fetch_next_remote,
+					   &fetch_failed_to_start,
+					   &fetch_finished,
+					   &state,
+					   "fetch", "parallel/fetch");
+
+		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
 			const char *name = list->items[i].string;
diff --git a/run-command.c b/run-command.c
index 5ec3a46dccf..642e6b6e057 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-int run_processes_parallel(int n,
-			   get_next_task_fn get_next_task,
-			   start_failure_fn start_failure,
-			   task_finished_fn task_finished,
-			   void *pp_cb)
+void run_processes_parallel(int n,
+			    get_next_task_fn get_next_task,
+			    start_failure_fn start_failure,
+			    task_finished_fn task_finished,
+			    void *pp_cb)
 {
 	int i, code;
 	int output_timeout = 100;
@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,
 	}
 
 	pp_cleanup(&pp);
-	return 0;
 }
 
-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
-			       start_failure_fn start_failure,
-			       task_finished_fn task_finished, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label)
+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+				start_failure_fn start_failure,
+				task_finished_fn task_finished, void *pp_cb,
+				const char *tr2_category, const char *tr2_label)
 {
-	int result;
-
 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
 				   ((n < 1) ? online_cpus() : n));
 
-	result = run_processes_parallel(n, get_next_task, start_failure,
-					task_finished, pp_cb);
+	run_processes_parallel(n, get_next_task, start_failure,
+			       task_finished, pp_cb);
 
 	trace2_region_leave(tr2_category, tr2_label, NULL);
-
-	return result;
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 0e85e5846a5..e76a1b6b5b3 100644
--- a/run-command.h
+++ b/run-command.h
@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-int run_processes_parallel(int n,
-			   get_next_task_fn,
-			   start_failure_fn,
-			   task_finished_fn,
-			   void *pp_cb);
-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
-			       task_finished_fn, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label);
+void run_processes_parallel(int n,
+			    get_next_task_fn,
+			    start_failure_fn,
+			    task_finished_fn,
+			    void *pp_cb);
+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+				task_finished_fn, void *pp_cb,
+				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 390fa4fb724..30c474f3243 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	ret = run_processes_parallel(max_jobs, next_test, test_failed,
-				     test_finished, &suite);
+	run_processes_parallel(max_jobs, next_test, test_failed,
+			       test_finished, &suite);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
@@ -428,16 +428,16 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, NULL, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL,
+				       task_finished, &proc);
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		exit(run_processes_parallel(jobs, no_job,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, no_job, NULL, task_finished,
+				       &proc);
 	} else {
 		fprintf(stderr, "check usage\n");
 		return 1;
 	}
+	exit(0);
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 06/22] run-command tests: use "return", not "exit"

[Ævar Arnfjörð Bjarmason]

Change the "run-command" test helper to "return" instead of calling
"exit", see 338abb0f045 (builtins + test helpers: use return instead
of exit() in cmd_*, 2021-06-08)

Because we'd previously gotten past the SANITIZE=leak check by using
exit() here we need to move to "goto cleanup" pattern.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 30c474f3243..ee509aefa2f 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)
 {
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
+	int ret;
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
-		exit(testsuite(argc - 1, argv + 1));
+		return testsuite(argc - 1, argv + 1);
 	if (!strcmp(argv[1], "inherited-handle"))
-		exit(inherit_handle(argv[0]));
+		return inherit_handle(argv[0]);
 	if (!strcmp(argv[1], "inherited-handle-child"))
-		exit(inherit_handle_child());
+		return inherit_handle_child();
 
 	if (argc >= 2 && !strcmp(argv[1], "quote-stress-test"))
 		return !!quote_stress_test(argc - 1, argv + 1);
@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)
 		argv += 2;
 		argc -= 2;
 	}
-	if (argc < 3)
-		return 1;
+	if (argc < 3) {
+		ret = 1;
+		goto cleanup;
+	}
 	strvec_pushv(&proc.args, (const char **)argv + 2);
 
 	if (!strcmp(argv[1], "start-command-ENOENT")) {
-		if (start_command(&proc) < 0 && errno == ENOENT)
-			return 0;
+		if (start_command(&proc) < 0 && errno == ENOENT) {
+			ret = 0;
+			goto cleanup;
+		}
 		fprintf(stderr, "FAIL %s\n", argv[1]);
 		return 1;
 	}
-	if (!strcmp(argv[1], "run-command"))
-		exit(run_command(&proc));
+	if (!strcmp(argv[1], "run-command")) {
+		ret = run_command(&proc);
+		goto cleanup;
+	}
 
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
@@ -436,8 +443,12 @@ int cmd__run_command(int argc, const char **argv)
 		run_processes_parallel(jobs, no_job, NULL, task_finished,
 				       &proc);
 	} else {
+		ret = 1;
 		fprintf(stderr, "check usage\n");
-		return 1;
+		goto cleanup;
 	}
-	exit(0);
+	ret = 0;
+cleanup:
+	child_process_clear(&proc);
+	return ret;
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 07/22] run-command.c: remove dead assignment in while-loop

[Ævar Arnfjörð Bjarmason]

Remove code that's been unused since it was added in
c553c72eed6 (run-command: add an asynchronous parallel child
processor, 2015-12-15), the next use of "i" in this function is:

	for (i = 0; ...

So we'll always clobber the "i" that's set here. Presumably the "i"
assignment is an artifact of WIP code that made it into our tree.

A subsequent commit will need to adjust the type of the "i" variable
in the otherwise unrelated for-loop, which is why this is being
removed now.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/run-command.c b/run-command.c
index 642e6b6e057..bd45828fe2c 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1689,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 {
 	int i;
 
-	while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
+	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
 		if (errno == EINTR)
 			continue;
 		pp_cleanup(pp);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 08/22] run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful

[Ævar Arnfjörð Bjarmason]

Refactor code in run-command.c where the "i" iteration variable is
being compared to an unsigned type, or where it's being shadowed.

In a subsequent commit the type of the "n" variable will be changed,
this also helps to avoid some of the warnings we've had under
"DEVOPTS=extra-all" since 8d133a4653a (strvec: use size_t to store nr
and alloc, 2021-09-11).

Per the thread ending at [1] we already have this C99 syntax in the
v2.38.0 release, per 6563706568b (CodingGuidelines: give deadline for
"for (int i = 0; ...", 2022-03-30) we should re-visit the wider use of
this syntax for November 2022, meaning within the window of v2.39.0.

As of writing it's earlier than that deadline and per [1] we want to
"avoid open[ing] the floodgate and deliberately add more [this C99
syntax]". But in this case the use of the syntax solves a real problem
of conflating types, which we'd otherwise need to avoid by using
differently named iteration variables (and the associated larger
refactoring).

1. https://lore.kernel.org/git/xmqqy1wgwkbj.fsf@gitster.g/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/run-command.c b/run-command.c
index bd45828fe2c..7b27e4de78d 100644
--- a/run-command.c
+++ b/run-command.c
@@ -443,7 +443,6 @@ static char **prep_childenv(const char *const *deltaenv)
 	struct string_list env = STRING_LIST_INIT_DUP;
 	struct strbuf key = STRBUF_INIT;
 	const char *const *p;
-	int i;
 
 	/* Construct a sorted string list consisting of the current environ */
 	for (p = (const char *const *) environ; p && *p; p++) {
@@ -476,7 +475,7 @@ static char **prep_childenv(const char *const *deltaenv)
 
 	/* Create an array of 'char *' to be used as the childenv */
 	ALLOC_ARRAY(childenv, env.nr + 1);
-	for (i = 0; i < env.nr; i++)
+	for (size_t i = 0; i < env.nr; i++)
 		childenv[i] = env.items[i].util;
 	childenv[env.nr] = NULL;
 
@@ -581,7 +580,6 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
 {
 	struct string_list envs = STRING_LIST_INIT_DUP;
 	const char *const *e;
-	int i;
 	int printed_unset = 0;
 
 	/* Last one wins, see run-command.c:prep_childenv() for context */
@@ -599,7 +597,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
 	}
 
 	/* "unset X Y...;" */
-	for (i = 0; i < envs.nr; i++) {
+	for (size_t i = 0; i < envs.nr; i++) {
 		const char *var = envs.items[i].string;
 		const char *val = envs.items[i].util;
 
@@ -616,7 +614,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
 		strbuf_addch(dst, ';');
 
 	/* ... followed by "A=B C=D ..." */
-	for (i = 0; i < envs.nr; i++) {
+	for (size_t i = 0; i < envs.nr; i++) {
 		const char *var = envs.items[i].string;
 		const char *val = envs.items[i].util;
 		const char *oldval;
@@ -1789,7 +1787,7 @@ void run_processes_parallel(int n,
 			    task_finished_fn task_finished,
 			    void *pp_cb)
 {
-	int i, code;
+	int code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	int ungroup = run_processes_parallel_ungroup;
@@ -1801,7 +1799,7 @@ void run_processes_parallel(int n,
 	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
 		ungroup);
 	while (1) {
-		for (i = 0;
+		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
 		    pp.nr_processes < pp.max_processes;
 		    i++) {
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v2 08/22] run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful

[Phillip Wood]

Hi Ævar

On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:
> Refactor code in run-command.c where the "i" iteration variable is
> being compared to an unsigned type, or where it's being shadowed.
> 
> In a subsequent commit the type of the "n" variable will be changed,
> this also helps to avoid some of the warnings we've had under
> "DEVOPTS=extra-all" since 8d133a4653a (strvec: use size_t to store nr
> and alloc, 2021-09-11).
> 
> Per the thread ending at [1] we already have this C99 syntax in the
> v2.38.0 release, per 6563706568b (CodingGuidelines: give deadline for
> "for (int i = 0; ...", 2022-03-30) we should re-visit the wider use of
> this syntax for November 2022, meaning within the window of v2.39.0.
> 
> As of writing it's earlier than that deadline and per [1] we want to
> "avoid open[ing] the floodgate and deliberately add more [this C99
> syntax]". But in this case the use of the syntax solves a real problem
> of conflating types, which we'd otherwise need to avoid by using
> differently named iteration variables (and the associated larger
> refactoring).
> 
> 1. https://lore.kernel.org/git/xmqqy1wgwkbj.fsf@gitster.g/
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 12 +++++-------
>   1 file changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index bd45828fe2c..7b27e4de78d 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -443,7 +443,6 @@ static char **prep_childenv(const char *const *deltaenv)
>   	struct string_list env = STRING_LIST_INIT_DUP;
>   	struct strbuf key = STRBUF_INIT;
>   	const char *const *p;
> -	int i;
>   
>   	/* Construct a sorted string list consisting of the current environ */
>   	for (p = (const char *const *) environ; p && *p; p++) {
> @@ -476,7 +475,7 @@ static char **prep_childenv(const char *const *deltaenv)
>   
>   	/* Create an array of 'char *' to be used as the childenv */
>   	ALLOC_ARRAY(childenv, env.nr + 1);
> -	for (i = 0; i < env.nr; i++)
> +	for (size_t i = 0; i < env.nr; i++)

In this case we're changing the type to avoid a signed/unsigned 
comparison. We could achieve this by just changing the type of i above.

>   		childenv[i] = env.items[i].util;
>   	childenv[env.nr] = NULL;
>   
> @@ -581,7 +580,6 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
>   {
>   	struct string_list envs = STRING_LIST_INIT_DUP;
>   	const char *const *e;
> -	int i;
>   	int printed_unset = 0;
>   
>   	/* Last one wins, see run-command.c:prep_childenv() for context */
> @@ -599,7 +597,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
>   	}
>   
>   	/* "unset X Y...;" */
> -	for (i = 0; i < envs.nr; i++) {
> +	for (size_t i = 0; i < envs.nr; i++) {

Ditto

>   		const char *var = envs.items[i].string;
>   		const char *val = envs.items[i].util;
>   
> @@ -616,7 +614,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)
>   		strbuf_addch(dst, ';');
>   
>   	/* ... followed by "A=B C=D ..." */
> -	for (i = 0; i < envs.nr; i++) {
> +	for (size_t i = 0; i < envs.nr; i++) {

Ditto

>   		const char *var = envs.items[i].string;
>   		const char *val = envs.items[i].util;
>   		const char *oldval;
> @@ -1789,7 +1787,7 @@ void run_processes_parallel(int n,
>   			    task_finished_fn task_finished,
>   			    void *pp_cb)
>   {
> -	int i, code;
> +	int code;
>   	int output_timeout = 100;
>   	int spawn_cap = 4;
>   	int ungroup = run_processes_parallel_ungroup;
> @@ -1801,7 +1799,7 @@ void run_processes_parallel(int n,
>   	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
>   		ungroup);
>   	while (1) {
> -		for (i = 0;
> +		for (int i = 0;

Here we are moving the declaration to the loop.

Am I missing something, I don't understand how these changes match this 
description in the commit message

 > But in this case the use of the syntax solves a real problem
 > of conflating types, which we'd otherwise need to avoid by using
 > differently named iteration variables (and the associated larger
 > refactoring).

Where are the cases where we'd need differently named variables if we 
did not use this syntax. In each case we delete the old declaration.

While I can see the point in avoiding the signed/unsigned comparisons I 
don't think it is strictly related to the topic of this series.

Best Wishes

Phillip

>   		    i < spawn_cap && !pp.shutdown &&
>   		    pp.nr_processes < pp.max_processes;
>   		    i++) {

Re: [PATCH v2 08/22] run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

>>   	while (1) {
>> -		for (i = 0;
>> +		for (int i = 0;
>
> Here we are moving the declaration to the loop.
>
> Am I missing something, I don't understand how these changes match
> this description in the commit message

It matches the commit title, and makes it clear that this does not
have to be part of this series.

Ævar, try to do a focused series that achieves the objective of the
topic well, without succumbing to the temptation of creature feep.
It is especially important when you are taking other peoples' series
hostage by requesting them to rebase on you.

Thanks.

[PATCH v2 09/22] run-command API: make "n" parameter a "size_t"

[Ævar Arnfjörð Bjarmason]

Make the "n" variable added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) a "size_t". As
we'll see in a subsequent commit we do pass "0" here, but never "jobs
< 0".

We could have made it an "unsigned int", but as we're having to change
this let's not leave another case in the codebase where a size_t and
"unsigned int" size differ on some platforms. In this case it's likely
to never matter, but it's easier to not need to worry about it.

After this and preceding changes:

	make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter

Only has one (and new) -Wsigned-compare warning, about a comparison of
"i" to online_cpus(), a subsequent commit will adjust & deal with
online_cpus() and that warning.

The only users of the "n" parameter are:

 * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
   config. As seen in the code that parses the config added in
   d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
   2019-10-05) will die if the git_config_int() return value is < 0.

   It will however pass us n = 0, as we'll see in a subsequent commit.

 * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
   config. Read via code originally added in a028a1930c6 (fetching
   submodules: respect `submodule.fetchJobs` config option, 2016-02-29).

   It now piggy-backs on the the submodule.fetchJobs code and
   validation added in f20e7c1ea24 (submodule: remove
   submodule.fetchjobs from submodule-config parsing, 2017-08-02).

   Like builtin/fetch.c it will die if the git_config_int() return
   value is < 0, but like builtin/fetch.c it will pass us n = 0.

 * builtin/submodule--helper.c: defaults to 1. Read via code
   originally added in 2335b870fa7 (submodule update: expose parallelism
   to the user, 2016-02-29).

   Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
   submodule-config parsing, 2017-08-02) it shares a config parser and
   semantics with the submodule.c caller.

 * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
   subcommand, 2021-12-22).

 * t/helper/test-run-command.c: can be -1 after parsing the arguments,
   but will then be overridden to online_cpus() before passing it to
   this API. See be5d88e1128 (test-tool run-command: learn to run (parts
   of) the testsuite, 2019-10-04).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 44 ++++++++++++++++++--------------------------
 run-command.h |  4 ++--
 2 files changed, 20 insertions(+), 28 deletions(-)

diff --git a/run-command.c b/run-command.c
index 7b27e4de78d..5a63008b6e1 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1498,8 +1498,8 @@ int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *data;
 
-	int max_processes;
-	int nr_processes;
+	size_t max_processes;
+	size_t nr_processes;
 
 	get_next_task_fn get_next_task;
 	start_failure_fn start_failure;
@@ -1520,7 +1520,7 @@ struct parallel_processes {
 	unsigned shutdown : 1;
 	unsigned ungroup : 1;
 
-	int output_owner;
+	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
 };
 
@@ -1541,9 +1541,7 @@ static int default_task_finished(int result,
 
 static void kill_children(struct parallel_processes *pp, int signo)
 {
-	int i, n = pp->max_processes;
-
-	for (i = 0; i < n; i++)
+	for (size_t i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
 			kill(pp->children[i].process.pid, signo);
 }
@@ -1558,20 +1556,19 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    int n,
+		    size_t n,
 		    get_next_task_fn get_next_task,
 		    start_failure_fn start_failure,
 		    task_finished_fn task_finished,
 		    void *data, int ungroup)
 {
-	int i;
-
 	if (n < 1)
 		n = online_cpus();
 
 	pp->max_processes = n;
 
-	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
+	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
+		     (uintmax_t)n);
 
 	pp->data = data;
 	if (!get_next_task)
@@ -1592,7 +1589,7 @@ static void pp_init(struct parallel_processes *pp,
 		CALLOC_ARRAY(pp->pfd, n);
 	strbuf_init(&pp->buffered_output, 0);
 
-	for (i = 0; i < n; i++) {
+	for (size_t i = 0; i < n; i++) {
 		strbuf_init(&pp->children[i].err, 0);
 		child_process_init(&pp->children[i].process);
 		if (pp->pfd) {
@@ -1607,10 +1604,8 @@ static void pp_init(struct parallel_processes *pp,
 
 static void pp_cleanup(struct parallel_processes *pp)
 {
-	int i;
-
 	trace_printf("run_processes_parallel: done");
-	for (i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < pp->max_processes; i++) {
 		strbuf_release(&pp->children[i].err);
 		child_process_clear(&pp->children[i].process);
 	}
@@ -1637,7 +1632,8 @@ static void pp_cleanup(struct parallel_processes *pp)
  */
 static int pp_start_one(struct parallel_processes *pp)
 {
-	int i, code;
+	size_t i;
+	int code;
 
 	for (i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_FREE)
@@ -1685,8 +1681,6 @@ static int pp_start_one(struct parallel_processes *pp)
 
 static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 {
-	int i;
-
 	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
 		if (errno == EINTR)
 			continue;
@@ -1695,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 	}
 
 	/* Buffer output from all pipes. */
-	for (i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < pp->max_processes; i++) {
 		if (pp->children[i].state == GIT_CP_WORKING &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
@@ -1712,7 +1706,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 
 static void pp_output(struct parallel_processes *pp)
 {
-	int i = pp->output_owner;
+	size_t i = pp->output_owner;
 
 	if (pp->children[i].state == GIT_CP_WORKING &&
 	    pp->children[i].err.len) {
@@ -1723,8 +1717,8 @@ static void pp_output(struct parallel_processes *pp)
 
 static int pp_collect_finished(struct parallel_processes *pp)
 {
-	int i, code;
-	int n = pp->max_processes;
+	int code;
+	size_t i, n = pp->max_processes;
 	int result = 0;
 
 	while (pp->nr_processes > 0) {
@@ -1781,7 +1775,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(int n,
+void run_processes_parallel(size_t n,
 			    get_next_task_fn get_next_task,
 			    start_failure_fn start_failure,
 			    task_finished_fn task_finished,
@@ -1815,9 +1809,7 @@ void run_processes_parallel(int n,
 		if (!pp.nr_processes)
 			break;
 		if (ungroup) {
-			int i;
-
-			for (i = 0; i < pp.max_processes; i++)
+			for (size_t i = 0; i < pp.max_processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
 			pp_buffer_stderr(&pp, output_timeout);
@@ -1834,7 +1826,7 @@ void run_processes_parallel(int n,
 	pp_cleanup(&pp);
 }
 
-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
diff --git a/run-command.h b/run-command.h
index e76a1b6b5b3..6f7604e1146 100644
--- a/run-command.h
+++ b/run-command.h
@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-void run_processes_parallel(int n,
+void run_processes_parallel(size_t n,
 			    get_next_task_fn,
 			    start_failure_fn,
 			    task_finished_fn,
 			    void *pp_cb);
-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
 				task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
 
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v2 09/22] run-command API: make "n" parameter a "size_t"

[Phillip Wood]

Hi Ævar

On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:
> Make the "n" variable added in c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) a "size_t". As
> we'll see in a subsequent commit we do pass "0" here, but never "jobs
> < 0".

This feels like a pretty arbitrary change that is not related to 
converting run_processes_parallel() to take a struct of options. There 
must be lots of places in the code base where we use int to pass a 
non-negative number around and int is more than wide enough to hold the 
number of jobs.

Best Wishes

Phillip

> We could have made it an "unsigned int", but as we're having to change
> this let's not leave another case in the codebase where a size_t and
> "unsigned int" size differ on some platforms. In this case it's likely
> to never matter, but it's easier to not need to worry about it.
> 
> After this and preceding changes:
> 
> 	make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
> 
> Only has one (and new) -Wsigned-compare warning, about a comparison of
> "i" to online_cpus(), a subsequent commit will adjust & deal with
> online_cpus() and that warning.
> 
> The only users of the "n" parameter are:
> 
>   * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
>     config. As seen in the code that parses the config added in
>     d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
>     2019-10-05) will die if the git_config_int() return value is < 0.
> 
>     It will however pass us n = 0, as we'll see in a subsequent commit.
> 
>   * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
>     config. Read via code originally added in a028a1930c6 (fetching
>     submodules: respect `submodule.fetchJobs` config option, 2016-02-29).
> 
>     It now piggy-backs on the the submodule.fetchJobs code and
>     validation added in f20e7c1ea24 (submodule: remove
>     submodule.fetchjobs from submodule-config parsing, 2017-08-02).
> 
>     Like builtin/fetch.c it will die if the git_config_int() return
>     value is < 0, but like builtin/fetch.c it will pass us n = 0.
> 
>   * builtin/submodule--helper.c: defaults to 1. Read via code
>     originally added in 2335b870fa7 (submodule update: expose parallelism
>     to the user, 2016-02-29).
> 
>     Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
>     submodule-config parsing, 2017-08-02) it shares a config parser and
>     semantics with the submodule.c caller.
> 
>   * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
>     subcommand, 2021-12-22).
> 
>   * t/helper/test-run-command.c: can be -1 after parsing the arguments,
>     but will then be overridden to online_cpus() before passing it to
>     this API. See be5d88e1128 (test-tool run-command: learn to run (parts
>     of) the testsuite, 2019-10-04).
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 44 ++++++++++++++++++--------------------------
>   run-command.h |  4 ++--
>   2 files changed, 20 insertions(+), 28 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index 7b27e4de78d..5a63008b6e1 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1498,8 +1498,8 @@ int run_processes_parallel_ungroup;
>   struct parallel_processes {
>   	void *data;
>   
> -	int max_processes;
> -	int nr_processes;
> +	size_t max_processes;
> +	size_t nr_processes;
>   
>   	get_next_task_fn get_next_task;
>   	start_failure_fn start_failure;
> @@ -1520,7 +1520,7 @@ struct parallel_processes {
>   	unsigned shutdown : 1;
>   	unsigned ungroup : 1;
>   
> -	int output_owner;
> +	size_t output_owner;
>   	struct strbuf buffered_output; /* of finished children */
>   };
>   
> @@ -1541,9 +1541,7 @@ static int default_task_finished(int result,
>   
>   static void kill_children(struct parallel_processes *pp, int signo)
>   {
> -	int i, n = pp->max_processes;
> -
> -	for (i = 0; i < n; i++)
> +	for (size_t i = 0; i < pp->max_processes; i++)
>   		if (pp->children[i].state == GIT_CP_WORKING)
>   			kill(pp->children[i].process.pid, signo);
>   }
> @@ -1558,20 +1556,19 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    int n,
> +		    size_t n,
>   		    get_next_task_fn get_next_task,
>   		    start_failure_fn start_failure,
>   		    task_finished_fn task_finished,
>   		    void *data, int ungroup)
>   {
> -	int i;
> -
>   	if (n < 1)
>   		n = online_cpus();
>   
>   	pp->max_processes = n;
>   
> -	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
> +	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
> +		     (uintmax_t)n);
>   
>   	pp->data = data;
>   	if (!get_next_task)
> @@ -1592,7 +1589,7 @@ static void pp_init(struct parallel_processes *pp,
>   		CALLOC_ARRAY(pp->pfd, n);
>   	strbuf_init(&pp->buffered_output, 0);
>   
> -	for (i = 0; i < n; i++) {
> +	for (size_t i = 0; i < n; i++) {
>   		strbuf_init(&pp->children[i].err, 0);
>   		child_process_init(&pp->children[i].process);
>   		if (pp->pfd) {
> @@ -1607,10 +1604,8 @@ static void pp_init(struct parallel_processes *pp,
>   
>   static void pp_cleanup(struct parallel_processes *pp)
>   {
> -	int i;
> -
>   	trace_printf("run_processes_parallel: done");
> -	for (i = 0; i < pp->max_processes; i++) {
> +	for (size_t i = 0; i < pp->max_processes; i++) {
>   		strbuf_release(&pp->children[i].err);
>   		child_process_clear(&pp->children[i].process);
>   	}
> @@ -1637,7 +1632,8 @@ static void pp_cleanup(struct parallel_processes *pp)
>    */
>   static int pp_start_one(struct parallel_processes *pp)
>   {
> -	int i, code;
> +	size_t i;
> +	int code;
>   
>   	for (i = 0; i < pp->max_processes; i++)
>   		if (pp->children[i].state == GIT_CP_FREE)
> @@ -1685,8 +1681,6 @@ static int pp_start_one(struct parallel_processes *pp)
>   
>   static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>   {
> -	int i;
> -
>   	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
>   		if (errno == EINTR)
>   			continue;
> @@ -1695,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>   	}
>   
>   	/* Buffer output from all pipes. */
> -	for (i = 0; i < pp->max_processes; i++) {
> +	for (size_t i = 0; i < pp->max_processes; i++) {
>   		if (pp->children[i].state == GIT_CP_WORKING &&
>   		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
>   			int n = strbuf_read_once(&pp->children[i].err,
> @@ -1712,7 +1706,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>   
>   static void pp_output(struct parallel_processes *pp)
>   {
> -	int i = pp->output_owner;
> +	size_t i = pp->output_owner;
>   
>   	if (pp->children[i].state == GIT_CP_WORKING &&
>   	    pp->children[i].err.len) {
> @@ -1723,8 +1717,8 @@ static void pp_output(struct parallel_processes *pp)
>   
>   static int pp_collect_finished(struct parallel_processes *pp)
>   {
> -	int i, code;
> -	int n = pp->max_processes;
> +	int code;
> +	size_t i, n = pp->max_processes;
>   	int result = 0;
>   
>   	while (pp->nr_processes > 0) {
> @@ -1781,7 +1775,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
>   	return result;
>   }
>   
> -void run_processes_parallel(int n,
> +void run_processes_parallel(size_t n,
>   			    get_next_task_fn get_next_task,
>   			    start_failure_fn start_failure,
>   			    task_finished_fn task_finished,
> @@ -1815,9 +1809,7 @@ void run_processes_parallel(int n,
>   		if (!pp.nr_processes)
>   			break;
>   		if (ungroup) {
> -			int i;
> -
> -			for (i = 0; i < pp.max_processes; i++)
> +			for (size_t i = 0; i < pp.max_processes; i++)
>   				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
>   		} else {
>   			pp_buffer_stderr(&pp, output_timeout);
> @@ -1834,7 +1826,7 @@ void run_processes_parallel(int n,
>   	pp_cleanup(&pp);
>   }
>   
> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
> +void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
>   				start_failure_fn start_failure,
>   				task_finished_fn task_finished, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label)
> diff --git a/run-command.h b/run-command.h
> index e76a1b6b5b3..6f7604e1146 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
>    * API reads that setting.
>    */
>   extern int run_processes_parallel_ungroup;
> -void run_processes_parallel(int n,
> +void run_processes_parallel(size_t n,
>   			    get_next_task_fn,
>   			    start_failure_fn,
>   			    task_finished_fn,
>   			    void *pp_cb);
> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
> +void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
>   				task_finished_fn, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label);
>   

[PATCH v2 10/22] run-command API: don't fall back on online_cpus()

[Ævar Arnfjörð Bjarmason]

When a "jobs = 0" is passed let's BUG() out rather than fall back on
online_cpus(). The default behavior was added when this API was
implemented in c553c72eed6 (run-command: add an asynchronous parallel
child processor, 2015-12-15).

Most of our code in-tree that scales up to "online_cpus()" by default
calls that function by itself. By having these callers of the
"run_processes_parallel()" API do the same we can in subsequent
commits pass all arguments down as a "const struct".

The preceding commit has an overview of the API callers that passed
"jobs = 0". There were only two of them (actually three, but they
resolved to these two config parsing codepaths).

The "fetch.parallel" caller already had a test for the
"fetch.parallel=0" case added in 0353c688189 (fetch: do not run a
redundant fetch from submodule, 2022-05-16), but there was no such
test for "submodule.fetchJobs". Let's add one here.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 2 ++
 run-command.c               | 7 +++----
 submodule-config.c          | 2 ++
 t/t5526-fetch-submodules.sh | 5 +++++
 4 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 78043fb67ef..82f1da14ec1 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
 		fetch_parallel_config = git_config_int(k, v);
 		if (fetch_parallel_config < 0)
 			die(_("fetch.parallel cannot be negative"));
+		if (!fetch_parallel_config)
+			fetch_parallel_config = online_cpus();
 		return 0;
 	}
 
diff --git a/run-command.c b/run-command.c
index 5a63008b6e1..21c00a48823 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1562,8 +1562,8 @@ static void pp_init(struct parallel_processes *pp,
 		    task_finished_fn task_finished,
 		    void *data, int ungroup)
 {
-	if (n < 1)
-		n = online_cpus();
+	if (!n)
+		BUG("you must provide a non-zero number of processes!");
 
 	pp->max_processes = n;
 
@@ -1831,8 +1831,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
-				   ((n < 1) ? online_cpus() : n));
+	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
 
 	run_processes_parallel(n, get_next_task, start_failure,
 			       task_finished, pp_cb);
diff --git a/submodule-config.c b/submodule-config.c
index cd7ee236a12..4dc61b3a78a 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)
 	int fetchjobs = git_config_int(var, value);
 	if (fetchjobs < 0)
 		die(_("negative values not allowed for submodule.fetchJobs"));
+	if (!fetchjobs)
+		fetchjobs = online_cpus();
 	return fetchjobs;
 }
 
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index e36f9fdf242..98a287ffb90 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
 		grep "9 tasks" trace.out &&
 		>trace.out &&
+
+		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
+		grep "preparing to run up to [0-9]* tasks" trace.out &&
+		! grep "up to 0 tasks" trace.out &&
+		>trace.out
 	)
 '
 
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v2 10/22] run-command API: don't fall back on online_cpus()

[Phillip Wood]

Hi Ævar

On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:
> When a "jobs = 0" is passed let's BUG() out rather than fall back on
> online_cpus(). The default behavior was added when this API was
> implemented in c553c72eed6 (run-command: add an asynchronous parallel
> child processor, 2015-12-15).

This commit message still does not explain why this change is necessary. 
The cover letter says you hope it is obvious but it isn't. As I pointed 
out in my previous review[1] I think it is possible for 
run_processes_parallel() to take a const struct without this change.

Best Wishes

Phillip

[1] 
https://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com

> Most of our code in-tree that scales up to "online_cpus()" by default
> calls that function by itself. By having these callers of the
> "run_processes_parallel()" API do the same we can in subsequent
> commits pass all arguments down as a "const struct".
> 
> The preceding commit has an overview of the API callers that passed
> "jobs = 0". There were only two of them (actually three, but they
> resolved to these two config parsing codepaths).
> 
> The "fetch.parallel" caller already had a test for the
> "fetch.parallel=0" case added in 0353c688189 (fetch: do not run a
> redundant fetch from submodule, 2022-05-16), but there was no such
> test for "submodule.fetchJobs". Let's add one here.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   builtin/fetch.c             | 2 ++
>   run-command.c               | 7 +++----
>   submodule-config.c          | 2 ++
>   t/t5526-fetch-submodules.sh | 5 +++++
>   4 files changed, 12 insertions(+), 4 deletions(-)
> 
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 78043fb67ef..82f1da14ec1 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
>   		fetch_parallel_config = git_config_int(k, v);
>   		if (fetch_parallel_config < 0)
>   			die(_("fetch.parallel cannot be negative"));
> +		if (!fetch_parallel_config)
> +			fetch_parallel_config = online_cpus();
>   		return 0;
>   	}
>   
> diff --git a/run-command.c b/run-command.c
> index 5a63008b6e1..21c00a48823 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1562,8 +1562,8 @@ static void pp_init(struct parallel_processes *pp,
>   		    task_finished_fn task_finished,
>   		    void *data, int ungroup)
>   {
> -	if (n < 1)
> -		n = online_cpus();
> +	if (!n)
> +		BUG("you must provide a non-zero number of processes!");
>   
>   	pp->max_processes = n;
>   
> @@ -1831,8 +1831,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
>   				task_finished_fn task_finished, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label)
>   {
> -	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
> -				   ((n < 1) ? online_cpus() : n));
> +	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
>   
>   	run_processes_parallel(n, get_next_task, start_failure,
>   			       task_finished, pp_cb);
> diff --git a/submodule-config.c b/submodule-config.c
> index cd7ee236a12..4dc61b3a78a 100644
> --- a/submodule-config.c
> +++ b/submodule-config.c
> @@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)
>   	int fetchjobs = git_config_int(var, value);
>   	if (fetchjobs < 0)
>   		die(_("negative values not allowed for submodule.fetchJobs"));
> +	if (!fetchjobs)
> +		fetchjobs = online_cpus();
>   	return fetchjobs;
>   }
>   
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index e36f9fdf242..98a287ffb90 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '
>   		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
>   		grep "9 tasks" trace.out &&
>   		>trace.out &&
> +
> +		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
> +		grep "preparing to run up to [0-9]* tasks" trace.out &&
> +		! grep "up to 0 tasks" trace.out &&
> +		>trace.out
>   	)
>   '
>   

[PATCH v2 11/22] run-command.c: use designated init for pp_init(), add "const"

[Ævar Arnfjörð Bjarmason]

Use a designated initializer to initialize those parts of pp_init()
that don't need any conditionals for their initialization, this sets
us on a path to pp_init() itself into mostly a validation and
allocation function.

Since we're doing that we can add "const" to some of the members of
the "struct parallel_processes", which helps to clarify and
self-document this code. E.g. we never alter the "data" pointer we
pass t user callbacks, nor (after the preceding change to stop
invoking online_cpus()) do we change "max_processes", the same goes
for the "ungroup" option.

We can also do away with a call to strbuf_init() in favor of macro
initialization, and to rely on other fields being NULL'd or zero'd.

Making members of a struct "const" rather that the pointer to the
struct itself is usually painful, as e.g. it precludes us from
incrementally setting up the structure. In this case we only set it up
with the assignment in run_process_parallel() and pp_init(), and don't
pass the struct pointer around as "const", so making individual
members "const" is worth the potential hassle for extra safety.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 34 ++++++++++++++--------------------
 1 file changed, 14 insertions(+), 20 deletions(-)

diff --git a/run-command.c b/run-command.c
index 21c00a48823..6acfd29d8c1 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1496,9 +1496,9 @@ enum child_state {
 
 int run_processes_parallel_ungroup;
 struct parallel_processes {
-	void *data;
+	void *const data;
 
-	size_t max_processes;
+	const size_t max_processes;
 	size_t nr_processes;
 
 	get_next_task_fn get_next_task;
@@ -1518,7 +1518,7 @@ struct parallel_processes {
 	struct pollfd *pfd;
 
 	unsigned shutdown : 1;
-	unsigned ungroup : 1;
+	const unsigned ungroup : 1;
 
 	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
@@ -1556,21 +1556,18 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    size_t n,
 		    get_next_task_fn get_next_task,
 		    start_failure_fn start_failure,
-		    task_finished_fn task_finished,
-		    void *data, int ungroup)
+		    task_finished_fn task_finished)
 {
+	const size_t n = pp->max_processes;
+
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
 
-	pp->max_processes = n;
-
 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
 		     (uintmax_t)n);
 
-	pp->data = data;
 	if (!get_next_task)
 		BUG("you need to specify a get_next_task function");
 	pp->get_next_task = get_next_task;
@@ -1578,16 +1575,9 @@ static void pp_init(struct parallel_processes *pp,
 	pp->start_failure = start_failure ? start_failure : default_start_failure;
 	pp->task_finished = task_finished ? task_finished : default_task_finished;
 
-	pp->nr_processes = 0;
-	pp->output_owner = 0;
-	pp->shutdown = 0;
-	pp->ungroup = ungroup;
 	CALLOC_ARRAY(pp->children, n);
-	if (pp->ungroup)
-		pp->pfd = NULL;
-	else
+	if (!pp->ungroup)
 		CALLOC_ARRAY(pp->pfd, n);
-	strbuf_init(&pp->buffered_output, 0);
 
 	for (size_t i = 0; i < n; i++) {
 		strbuf_init(&pp->children[i].err, 0);
@@ -1785,13 +1775,17 @@ void run_processes_parallel(size_t n,
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	int ungroup = run_processes_parallel_ungroup;
-	struct parallel_processes pp;
+	struct parallel_processes pp = {
+		.max_processes = n,
+		.data = pp_cb,
+		.buffered_output = STRBUF_INIT,
+		.ungroup = ungroup,
+	};
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
-		ungroup);
+	pp_init(&pp, get_next_task, start_failure, task_finished);
 	while (1) {
 		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 12/22] run-command API: add nascent "struct run_process_parallel_opts"

[Ævar Arnfjörð Bjarmason]

As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
run_process_parallel(), 2022-06-07) which added the "ungroup" passing
it to "run_process_parallel()" via the global
"run_processes_parallel_ungroup" variable was a compromise to get the
smallest possible regression fix for "maint" at the time.

This follow-up to that is a start at passing that parameter and others
via a new "struct run_process_parallel_opts", as the earlier
version[1] of what became fd3aaf53f71 did.

For now we're only changing how data is passed internally to
"run-command.c", i.e. from "run_process_parallel()" to
pp_init(). Subsequent commits will change "run_processes_parallel()"
itself, as well as the "run_processes_parallel_tr2()" wrapper
function.

Since we need to change all of the occurrences of "n" to
"opt->SOMETHING" let's take the opportunity and rename the terse "n"
to "processes". We could also have picked "max_processes", "jobs",
"threads" etc., but as the API is named "run_processes_parallel()"
let's go with "processes".

1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 30 ++++++++++++++++++++----------
 run-command.h | 46 +++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 59 insertions(+), 17 deletions(-)

diff --git a/run-command.c b/run-command.c
index 6acfd29d8c1..3042cb26172 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1556,11 +1556,12 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    get_next_task_fn get_next_task,
-		    start_failure_fn start_failure,
-		    task_finished_fn task_finished)
+		    const struct run_process_parallel_opts *opts)
 {
-	const size_t n = pp->max_processes;
+	const size_t n = opts->processes;
+	get_next_task_fn get_next_task = opts->get_next_task;
+	start_failure_fn start_failure = opts->start_failure;
+	task_finished_fn task_finished = opts->task_finished;
 
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
@@ -1765,7 +1766,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(size_t n,
+void run_processes_parallel(size_t processes,
 			    get_next_task_fn get_next_task,
 			    start_failure_fn start_failure,
 			    task_finished_fn task_finished,
@@ -1774,18 +1775,27 @@ void run_processes_parallel(size_t n,
 	int code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
-	int ungroup = run_processes_parallel_ungroup;
 	struct parallel_processes pp = {
-		.max_processes = n,
+		.max_processes = processes,
 		.data = pp_cb,
 		.buffered_output = STRBUF_INIT,
-		.ungroup = ungroup,
+		.ungroup = run_processes_parallel_ungroup,
+	};
+	const struct run_process_parallel_opts opts = {
+		.processes = processes,
+
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+
+		.ungroup = run_processes_parallel_ungroup,
 	};
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, get_next_task, start_failure, task_finished);
+	pp_init(&pp, &opts);
+
 	while (1) {
 		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1802,7 +1812,7 @@ void run_processes_parallel(size_t n,
 		}
 		if (!pp.nr_processes)
 			break;
-		if (ungroup) {
+		if (opts.ungroup) {
 			for (size_t i = 0; i < pp.max_processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
diff --git a/run-command.h b/run-command.h
index 6f7604e1146..03be48e7ce3 100644
--- a/run-command.h
+++ b/run-command.h
@@ -459,7 +459,42 @@ typedef int (*task_finished_fn)(int result,
 				void *pp_task_cb);
 
 /**
- * Runs up to n processes at the same time. Whenever a process can be
+ * Option used by run_processes_parallel(), { 0 }-initialized means no
+ * options.
+ */
+struct run_process_parallel_opts
+{
+	/**
+	 * processes: see 'processes' in run_processes_parallel() below.
+	 */
+	size_t processes;
+
+	/**
+	 * ungroup: see 'ungroup' in run_processes_parallel() below.
+	 */
+	unsigned int ungroup:1;
+
+	/**
+	 * get_next_task: See get_next_task_fn() above. This must be
+	 * specified.
+	 */
+	get_next_task_fn get_next_task;
+
+	/**
+	 * start_failure: See start_failure_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	start_failure_fn start_failure;
+
+	/**
+	 * task_finished: See task_finished_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	task_finished_fn task_finished;
+};
+
+/**
+ * Runs N 'processes' at the same time. Whenever a process can be
  * started, the callback get_next_task_fn is called to obtain the data
  * required to start another child process.
  *
@@ -467,9 +502,6 @@ typedef int (*task_finished_fn)(int result,
  * (both stdout and stderr) is routed to stderr in a manner that output
  * from different tasks does not interleave (but see "ungroup" below).
  *
- * start_failure_fn and task_finished_fn can be NULL to omit any
- * special handling.
- *
  * If the "ungroup" option isn't specified, the API will set the
  * "stdout_to_stderr" parameter in "struct child_process" and provide
  * the callbacks with a "struct strbuf *out" parameter to write output
@@ -485,13 +517,13 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-void run_processes_parallel(size_t n,
+void run_processes_parallel(size_t processes,
 			    get_next_task_fn,
 			    start_failure_fn,
 			    task_finished_fn,
 			    void *pp_cb);
-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
-				task_finished_fn, void *pp_cb,
+void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
+				start_failure_fn, task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
 
 /**
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 13/22] run-command API: make run_process_parallel{,_tr2}() thin wrappers

[Ævar Arnfjörð Bjarmason]

Make the "run_process_parallel()" and "run_process_parallel_tr2()"
functions thin wrappers that construct a "struct
run_process_parallel_opts" struct, this is in preparation for changing
the API users to use a "struct run_process_parallel_opts" directly.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 66 ++++++++++++++++++++++++++++++++++-----------------
 run-command.h |  7 ++++++
 2 files changed, 51 insertions(+), 22 deletions(-)

diff --git a/run-command.c b/run-command.c
index 3042cb26172..3cdf85876c1 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1766,36 +1766,30 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(size_t processes,
-			    get_next_task_fn get_next_task,
-			    start_failure_fn start_failure,
-			    task_finished_fn task_finished,
-			    void *pp_cb)
+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
 {
 	int code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	struct parallel_processes pp = {
-		.max_processes = processes,
+		.max_processes = opts->processes,
 		.data = pp_cb,
 		.buffered_output = STRBUF_INIT,
 		.ungroup = run_processes_parallel_ungroup,
 	};
-	const struct run_process_parallel_opts opts = {
-		.processes = processes,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-
-		.ungroup = run_processes_parallel_ungroup,
-	};
+	/* options */
+	const char *tr2_category = opts->tr2_category;
+	const char *tr2_label = opts->tr2_label;
+	const int do_trace2 = tr2_category && tr2_label;
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, &opts);
+	if (do_trace2)
+		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
+					   "max:%d", opts->processes);
 
+	pp_init(&pp, opts);
 	while (1) {
 		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1812,7 +1806,7 @@ void run_processes_parallel(size_t processes,
 		}
 		if (!pp.nr_processes)
 			break;
-		if (opts.ungroup) {
+		if (opts->ungroup) {
 			for (size_t i = 0; i < pp.max_processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
@@ -1828,19 +1822,47 @@ void run_processes_parallel(size_t processes,
 	}
 
 	pp_cleanup(&pp);
+
+	if (do_trace2)
+		trace2_region_leave(tr2_category, tr2_label, NULL);
+}
+
+void run_processes_parallel(size_t processes,
+			    get_next_task_fn get_next_task,
+			    start_failure_fn start_failure,
+			    task_finished_fn task_finished,
+			    void *pp_cb)
+{
+	const struct run_process_parallel_opts opts = {
+		.processes = processes,
+		.ungroup = run_processes_parallel_ungroup,
+
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+	};
+
+	run_processes_parallel_1(&opts, pp_cb);
 }
 
-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
+void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = tr2_category,
+		.tr2_label = tr2_label,
+
+		.processes = processes,
+		.ungroup = run_processes_parallel_ungroup,
 
-	run_processes_parallel(n, get_next_task, start_failure,
-			       task_finished, pp_cb);
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+	};
 
-	trace2_region_leave(tr2_category, tr2_label, NULL);
+	run_processes_parallel_1(&opts, pp_cb);
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 03be48e7ce3..bd0b7b70b27 100644
--- a/run-command.h
+++ b/run-command.h
@@ -464,6 +464,13 @@ typedef int (*task_finished_fn)(int result,
  */
 struct run_process_parallel_opts
 {
+	/**
+	 * tr2_category & tr2_label: sets the trace2 category and label for
+	 * logging. These must either be unset, or both of them must be set.
+	 */
+	const char *tr2_category;
+	const char *tr2_label;
+
 	/**
 	 * processes: see 'processes' in run_processes_parallel() below.
 	 */
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v2 13/22] run-command API: make run_process_parallel{,_tr2}() thin wrappers

[Phillip Wood]

On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:
> -	const struct run_process_parallel_opts opts = {
> -		.processes = processes,
> -
> -		.get_next_task = get_next_task,
> -		.start_failure = start_failure,
> -		.task_finished = task_finished,
> -
> -		.ungroup = run_processes_parallel_ungroup,
> -	};
[...]
> +	const struct run_process_parallel_opts opts = {
> +		.processes = processes,
> +		.ungroup = run_processes_parallel_ungroup,
> +
> +		.get_next_task = get_next_task,
> +		.start_failure = start_failure,
> +		.task_finished = task_finished,
> +	};
> +
> +	run_processes_parallel_1(&opts, pp_cb);
>   }

I complained before that this is moving things around that you added in 
the last commit, but not only are you moving the designated initializer 
added in the last commit you are rearranging it.

Best Wishes

Phillip

[PATCH v2 14/22] run-command API: have run_process_parallel() take an "opts" struct

[Ævar Arnfjörð Bjarmason]

Have the "run_process_parallel()" function take an "opts" struct,
which allows us to eliminate it as a wrapper for
"run_processes_parallel_1()".

Since the new "run_processes_parallel()" function is able to take an
optional "tr2_category" and "tr2_label" via the struct we can at this
point migrate all of the users of "run_processes_parallel_tr2()" over
to it.

But let's not migrate all the API users, only the two users that
passed the "ungroup" parameter via the
"run_processes_parallel_ungroup" global, allowing us to delete that
global in favor of passing "ungroup" via the "opts" struct. As noted
in fd3aaf53f71 (run-command: add an "ungroup" option to
run_process_parallel(), 2022-06-07) which added
"run_processes_parallel_ungroup" passing this as a global was a hack
to produce a small regression fix for "maint".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c                      | 23 ++++++++++++++---------
 run-command.c               | 32 ++++----------------------------
 run-command.h               | 21 ++++++++++-----------
 t/helper/test-run-command.c | 31 +++++++++++++++++++++----------
 4 files changed, 49 insertions(+), 58 deletions(-)

diff --git a/hook.c b/hook.c
index a493939a4fc..a4fa1031f28 100644
--- a/hook.c
+++ b/hook.c
@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		.options = options,
 	};
 	const char *const hook_path = find_hook(hook_name);
-	int jobs = 1;
 	int ret = 0;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "hook",
+		.tr2_label = hook_name,
+
+		.processes = 1,
+		.ungroup = 1,
+
+		.get_next_task = pick_next_hook,
+		.start_failure = notify_start_failure,
+		.task_finished = notify_hook_finished,
+
+		.data = &cb_data,
+	};
 
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		cb_data.hook_path = abs_path.buf;
 	}
 
-	run_processes_parallel_ungroup = 1;
-	run_processes_parallel_tr2(jobs,
-				   pick_next_hook,
-				   notify_start_failure,
-				   notify_hook_finished,
-				   &cb_data,
-				   "hook",
-				   hook_name);
+	run_processes_parallel(&opts);
 	ret = cb_data.rc;
 cleanup:
 	strbuf_release(&abs_path);
diff --git a/run-command.c b/run-command.c
index 3cdf85876c1..0956cc572b7 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1494,7 +1494,6 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
-int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *const data;
 
@@ -1766,25 +1765,22 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
+void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	struct parallel_processes pp = {
 		.max_processes = opts->processes,
-		.data = pp_cb,
+		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
-		.ungroup = run_processes_parallel_ungroup,
+		.ungroup = opts->ungroup,
 	};
 	/* options */
 	const char *tr2_category = opts->tr2_category;
 	const char *tr2_label = opts->tr2_label;
 	const int do_trace2 = tr2_category && tr2_label;
 
-	/* unset for the next API user */
-	run_processes_parallel_ungroup = 0;
-
 	if (do_trace2)
 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
 					   "max:%d", opts->processes);
@@ -1827,24 +1823,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel(size_t processes,
-			    get_next_task_fn get_next_task,
-			    start_failure_fn start_failure,
-			    task_finished_fn task_finished,
-			    void *pp_cb)
-{
-	const struct run_process_parallel_opts opts = {
-		.processes = processes,
-		.ungroup = run_processes_parallel_ungroup,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-	};
-
-	run_processes_parallel_1(&opts, pp_cb);
-}
-
 void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
@@ -1853,16 +1831,14 @@ void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task
 	const struct run_process_parallel_opts opts = {
 		.tr2_category = tr2_category,
 		.tr2_label = tr2_label,
-
 		.processes = processes,
-		.ungroup = run_processes_parallel_ungroup,
 
 		.get_next_task = get_next_task,
 		.start_failure = start_failure,
 		.task_finished = task_finished,
 	};
 
-	run_processes_parallel_1(&opts, pp_cb);
+	run_processes_parallel(&opts);
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index bd0b7b70b27..aabdaf684db 100644
--- a/run-command.h
+++ b/run-command.h
@@ -498,11 +498,19 @@ struct run_process_parallel_opts
 	 * NULL to omit any special handling.
 	 */
 	task_finished_fn task_finished;
+
+	/**
+	 * data: user data, will be passed as "pp_cb" to the callback
+	 * parameters.
+	 */
+	void *data;
 };
 
 /**
+ * Options are passed via the "struct run_process_parallel_opts" above.
+ *
  * Runs N 'processes' at the same time. Whenever a process can be
- * started, the callback get_next_task_fn is called to obtain the data
+ * started, the callback opts.get_next_task is called to obtain the data
  * required to start another child process.
  *
  * The children started via this function run in parallel. Their output
@@ -518,17 +526,8 @@ struct run_process_parallel_opts
  * NULL "struct strbuf *out" parameter, and are responsible for
  * emitting their own output, including dealing with any race
  * conditions due to writing in parallel to stdout and stderr.
- * The "ungroup" option can be enabled by setting the global
- * "run_processes_parallel_ungroup" to "1" before invoking
- * run_processes_parallel(), it will be set back to "0" as soon as the
- * API reads that setting.
  */
-extern int run_processes_parallel_ungroup;
-void run_processes_parallel(size_t processes,
-			    get_next_task_fn,
-			    start_failure_fn,
-			    task_finished_fn,
-			    void *pp_cb);
+void run_processes_parallel(const struct run_process_parallel_opts *opts);
 void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
 				start_failure_fn, task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index ee509aefa2f..3ecb830f4a8 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {
 static int testsuite(int argc, const char **argv)
 {
 	struct testsuite suite = TESTSUITE_INIT;
-	int max_jobs = 1, i, ret;
+	int max_jobs = 1, i, ret = 0;
 	DIR *dir;
 	struct dirent *d;
 	struct option options[] = {
@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)
 			 "write JUnit-style XML files"),
 		OPT_END()
 	};
+	struct run_process_parallel_opts opts = {
+		.get_next_task = next_test,
+		.start_failure = test_failed,
+		.task_finished = test_finished,
+		.data = &suite,
+	};
 
 	argc = parse_options(argc, argv, NULL, options,
 			testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	run_processes_parallel(max_jobs, next_test, test_failed,
-			       test_finished, &suite);
+	opts.processes = max_jobs;
+	run_processes_parallel(&opts);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)
 	string_list_clear(&suite.tests, 0);
 	string_list_clear(&suite.failed, 0);
 
-	return !!ret;
+	return ret;
 }
 
 static uint64_t my_random_next = 1234;
@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
 	int ret;
+	struct run_process_parallel_opts opts = {
+		.data = &proc,
+	};
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
 		return testsuite(argc - 1, argv + 1);
@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
 		argc -= 1;
-		run_processes_parallel_ungroup = 1;
+		opts.ungroup = 1;
 	}
 
 	jobs = atoi(argv[2]);
@@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
+		opts.get_next_task = parallel_next;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		run_processes_parallel(jobs, parallel_next, NULL,
-				       task_finished, &proc);
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		run_processes_parallel(jobs, no_job, NULL, task_finished,
-				       &proc);
+		opts.get_next_task = no_job;
+		opts.task_finished = task_finished;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
 		goto cleanup;
 	}
+	opts.processes = jobs;
+	run_processes_parallel(&opts);
 	ret = 0;
 cleanup:
 	child_process_clear(&proc);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 15/22] run-command API: move *_tr2() users to "run_processes_parallel()"

[Ævar Arnfjörð Bjarmason]

Have the users of the "run_processes_parallel_tr2()" function use
"run_processes_parallel()" instead. In preceding commits the latter
was refactored to take a "struct run_process_parallel_opts" argument,
since the only reason for "run_processes_parallel_tr2()" to exist was
to take arguments that are now a part of that struct we can do away
with it.

See ee4512ed481 (trace2: create new combined trace facility,
2019-02-22) for the addition of the "*_tr2()" variant of the function,
it was used by every caller except "t/helper/test-run-command.c"..

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 18 ++++++++++++------
 builtin/submodule--helper.c | 16 ++++++++++++----
 run-command.c               | 18 ------------------
 run-command.h               |  3 ---
 submodule.c                 | 18 ++++++++++++------
 5 files changed, 36 insertions(+), 37 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 82f1da14ec1..b06e454cbdd 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	if (max_children != 1 && list->nr != 1) {
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		const struct run_process_parallel_opts opts = {
+			.tr2_category = "fetch",
+			.tr2_label = "parallel/fetch",
+
+			.processes = max_children,
+
+			.get_next_task = &fetch_next_remote,
+			.start_failure = &fetch_failed_to_start,
+			.task_finished = &fetch_finished,
+			.data = &state,
+		};
 
 		strvec_push(&argv, "--end-of-options");
-		run_processes_parallel_tr2(max_children,
-					   &fetch_next_remote,
-					   &fetch_failed_to_start,
-					   &fetch_finished,
-					   &state,
-					   "fetch", "parallel/fetch");
 
+		run_processes_parallel(&opts);
 		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 0b4acb442b2..df526525c1e 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)
 {
 	int i, ret = 0;
 	struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/update",
+
+		.processes = update_data->max_jobs,
+
+		.get_next_task = update_clone_get_next_task,
+		.start_failure = update_clone_start_failure,
+		.task_finished = update_clone_task_finished,
+		.data = &suc,
+	};
 
 	suc.update_data = update_data;
-	run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
-				   update_clone_start_failure,
-				   update_clone_task_finished, &suc, "submodule",
-				   "parallel/update");
+	run_processes_parallel(&opts);
 
 	/*
 	 * We saved the output and put it out all at once now.
diff --git a/run-command.c b/run-command.c
index 0956cc572b7..64fa0299175 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1823,24 +1823,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
-				start_failure_fn start_failure,
-				task_finished_fn task_finished, void *pp_cb,
-				const char *tr2_category, const char *tr2_label)
-{
-	const struct run_process_parallel_opts opts = {
-		.tr2_category = tr2_category,
-		.tr2_label = tr2_label,
-		.processes = processes,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-	};
-
-	run_processes_parallel(&opts);
-}
-
 int run_auto_maintenance(int quiet)
 {
 	int enabled;
diff --git a/run-command.h b/run-command.h
index aabdaf684db..e3e1ea01ad9 100644
--- a/run-command.h
+++ b/run-command.h
@@ -528,9 +528,6 @@ struct run_process_parallel_opts
  * conditions due to writing in parallel to stdout and stderr.
  */
 void run_processes_parallel(const struct run_process_parallel_opts *opts);
-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
-				start_failure_fn, task_finished_fn, void *pp_cb,
-				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/submodule.c b/submodule.c
index bf7a2c79183..f7c71f1f4b1 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,
 {
 	int i;
 	struct submodule_parallel_fetch spf = SPF_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/fetch",
+
+		.processes = max_parallel_jobs,
+
+		.get_next_task = get_next_submodule,
+		.start_failure = fetch_start_failure,
+		.task_finished = fetch_finish,
+		.data = &spf,
+	};
 
 	spf.r = r;
 	spf.command_line_option = command_line_option;
@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,
 
 	calculate_changed_submodule_paths(r, &spf.changed_submodule_names);
 	string_list_sort(&spf.changed_submodule_names);
-	run_processes_parallel_tr2(max_parallel_jobs,
-				   get_next_submodule,
-				   fetch_start_failure,
-				   fetch_finish,
-				   &spf,
-				   "submodule", "parallel/fetch");
+	run_processes_parallel(&opts);
 
 	if (spf.submodules_with_errors.len > 0)
 		fprintf(stderr, _("Errors during submodule fetch:\n%s"),
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 16/22] run-command.c: make "struct parallel_processes" const if possible

[Ævar Arnfjörð Bjarmason]

Add a "const" to two "struct parallel_processes" parameters where
we're not modifying anything in "pp". For kill_children() we'll call
it from both the signal handler, and from run_processes_parallel()
itself. Adding a "const" there makes it clear that we don't need to
modify any state when killing our children.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/run-command.c b/run-command.c
index 64fa0299175..6c343b9b77b 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1538,7 +1538,7 @@ static int default_task_finished(int result,
 	return 0;
 }
 
-static void kill_children(struct parallel_processes *pp, int signo)
+static void kill_children(const struct parallel_processes *pp, int signo)
 {
 	for (size_t i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
@@ -1694,7 +1694,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 	}
 }
 
-static void pp_output(struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp)
 {
 	size_t i = pp->output_owner;
 
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 17/22] run-command.c: don't copy *_fn to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

The only remaining reason for copying the callbacks in the "struct
run_process_parallel_opts" over to the "struct parallel_processes" was
to avoid two if/else statements in case the "start_failure" and
"task_finished" callbacks were NULL.

Let's handle those cases in pp_start_one() and pp_collect_finished()
instead, and avoid the default_* stub functions, and the need to copy
this data around.

Organizing the code like this made more sense before the "struct
run_parallel_parallel_opts" existed, as we'd have needed to pass each
of these as a separate parameter.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 67 +++++++++++++++++++--------------------------------
 1 file changed, 25 insertions(+), 42 deletions(-)

diff --git a/run-command.c b/run-command.c
index 6c343b9b77b..4fce1193b66 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1500,10 +1500,6 @@ struct parallel_processes {
 	const size_t max_processes;
 	size_t nr_processes;
 
-	get_next_task_fn get_next_task;
-	start_failure_fn start_failure;
-	task_finished_fn task_finished;
-
 	struct {
 		enum child_state state;
 		struct child_process process;
@@ -1523,21 +1519,6 @@ struct parallel_processes {
 	struct strbuf buffered_output; /* of finished children */
 };
 
-static int default_start_failure(struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
-static int default_task_finished(int result,
-				 struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
 static void kill_children(const struct parallel_processes *pp, int signo)
 {
 	for (size_t i = 0; i < pp->max_processes; i++)
@@ -1558,9 +1539,6 @@ static void pp_init(struct parallel_processes *pp,
 		    const struct run_process_parallel_opts *opts)
 {
 	const size_t n = opts->processes;
-	get_next_task_fn get_next_task = opts->get_next_task;
-	start_failure_fn start_failure = opts->start_failure;
-	task_finished_fn task_finished = opts->task_finished;
 
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
@@ -1568,12 +1546,8 @@ static void pp_init(struct parallel_processes *pp,
 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
 		     (uintmax_t)n);
 
-	if (!get_next_task)
+	if (!opts->get_next_task)
 		BUG("you need to specify a get_next_task function");
-	pp->get_next_task = get_next_task;
-
-	pp->start_failure = start_failure ? start_failure : default_start_failure;
-	pp->task_finished = task_finished ? task_finished : default_task_finished;
 
 	CALLOC_ARRAY(pp->children, n);
 	if (!pp->ungroup)
@@ -1620,7 +1594,8 @@ static void pp_cleanup(struct parallel_processes *pp)
  * <0 no new job was started, user wishes to shutdown early. Use negative code
  *    to signal the children.
  */
-static int pp_start_one(struct parallel_processes *pp)
+static int pp_start_one(struct parallel_processes *pp,
+			const struct run_process_parallel_opts *opts)
 {
 	size_t i;
 	int code;
@@ -1631,10 +1606,10 @@ static int pp_start_one(struct parallel_processes *pp)
 	if (i == pp->max_processes)
 		BUG("bookkeeping is hard");
 
-	code = pp->get_next_task(&pp->children[i].process,
-				 pp->ungroup ? NULL : &pp->children[i].err,
-				 pp->data,
-				 &pp->children[i].data);
+	code = opts->get_next_task(&pp->children[i].process,
+				   pp->ungroup ? NULL : &pp->children[i].err,
+				   pp->data,
+				   &pp->children[i].data);
 	if (!code) {
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
@@ -1649,10 +1624,14 @@ static int pp_start_one(struct parallel_processes *pp)
 	pp->children[i].process.no_stdin = 1;
 
 	if (start_command(&pp->children[i].process)) {
-		code = pp->start_failure(pp->ungroup ? NULL :
-					 &pp->children[i].err,
-					 pp->data,
-					 pp->children[i].data);
+		if (opts->start_failure)
+			code = opts->start_failure(pp->ungroup ? NULL :
+						   &pp->children[i].err,
+						   pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
+
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
@@ -1705,7 +1684,8 @@ static void pp_output(const struct parallel_processes *pp)
 	}
 }
 
-static int pp_collect_finished(struct parallel_processes *pp)
+static int pp_collect_finished(struct parallel_processes *pp,
+			       const struct run_process_parallel_opts *opts)
 {
 	int code;
 	size_t i, n = pp->max_processes;
@@ -1720,9 +1700,12 @@ static int pp_collect_finished(struct parallel_processes *pp)
 
 		code = finish_command(&pp->children[i].process);
 
-		code = pp->task_finished(code, pp->ungroup ? NULL :
-					 &pp->children[i].err, pp->data,
-					 pp->children[i].data);
+		if (opts->task_finished)
+			code = opts->task_finished(code, pp->ungroup ? NULL :
+						   &pp->children[i].err, pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
 
 		if (code)
 			result = code;
@@ -1791,7 +1774,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		    i < spawn_cap && !pp.shutdown &&
 		    pp.nr_processes < pp.max_processes;
 		    i++) {
-			code = pp_start_one(&pp);
+			code = pp_start_one(&pp, opts);
 			if (!code)
 				continue;
 			if (code < 0) {
@@ -1809,7 +1792,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 			pp_buffer_stderr(&pp, output_timeout);
 			pp_output(&pp);
 		}
-		code = pp_collect_finished(&pp);
+		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
 			if (code < 0)
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 18/22] run-command.c: don't copy "ungroup" to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in the preceding commit, let's not
copy the "ungroup" member of the "struct run_process_parallel_opts"
over to the "struct parallel_processes". Now that we're passing the
"opts" down there's no reason to do so.

This makes the code easier to follow, as we have a "const" attribute
on the "struct run_process_parallel_opts", but not "struct
parallel_processes". We do not alter the "ungroup" argument, so
storing it in the non-const structure would make this control flow
less obvious.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index 4fce1193b66..2b1cfbc996f 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1513,7 +1513,6 @@ struct parallel_processes {
 	struct pollfd *pfd;
 
 	unsigned shutdown : 1;
-	const unsigned ungroup : 1;
 
 	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
@@ -1550,7 +1549,7 @@ static void pp_init(struct parallel_processes *pp,
 		BUG("you need to specify a get_next_task function");
 
 	CALLOC_ARRAY(pp->children, n);
-	if (!pp->ungroup)
+	if (!opts->ungroup)
 		CALLOC_ARRAY(pp->pfd, n);
 
 	for (size_t i = 0; i < n; i++) {
@@ -1607,17 +1606,17 @@ static int pp_start_one(struct parallel_processes *pp,
 		BUG("bookkeeping is hard");
 
 	code = opts->get_next_task(&pp->children[i].process,
-				   pp->ungroup ? NULL : &pp->children[i].err,
+				   opts->ungroup ? NULL : &pp->children[i].err,
 				   pp->data,
 				   &pp->children[i].data);
 	if (!code) {
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
 		return 1;
 	}
-	if (!pp->ungroup) {
+	if (!opts->ungroup) {
 		pp->children[i].process.err = -1;
 		pp->children[i].process.stdout_to_stderr = 1;
 	}
@@ -1625,14 +1624,14 @@ static int pp_start_one(struct parallel_processes *pp,
 
 	if (start_command(&pp->children[i].process)) {
 		if (opts->start_failure)
-			code = opts->start_failure(pp->ungroup ? NULL :
+			code = opts->start_failure(opts->ungroup ? NULL :
 						   &pp->children[i].err,
 						   pp->data,
 						   pp->children[i].data);
 		else
 			code = 0;
 
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
@@ -1701,7 +1700,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		code = finish_command(&pp->children[i].process);
 
 		if (opts->task_finished)
-			code = opts->task_finished(code, pp->ungroup ? NULL :
+			code = opts->task_finished(code, opts->ungroup ? NULL :
 						   &pp->children[i].err, pp->data,
 						   pp->children[i].data);
 		else
@@ -1718,7 +1717,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			pp->pfd[i].fd = -1;
 		child_process_init(&pp->children[i].process);
 
-		if (pp->ungroup) {
+		if (opts->ungroup) {
 			; /* no strbuf_*() work to do here */
 		} else if (i != pp->output_owner) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
@@ -1757,7 +1756,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		.max_processes = opts->processes,
 		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
-		.ungroup = opts->ungroup,
 	};
 	/* options */
 	const char *tr2_category = opts->tr2_category;
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 19/22] run-command.c: don't copy "data" to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in a preceding commit, let's not copy
the "data" member of the "struct run_process_parallel_opts" over to
the "struct parallel_processes". Now that we're passing the "opts"
down there's no reason to do so.

This makes the code easier to follow, as we have a "const" attribute
on the "struct run_process_parallel_opts", but not "struct
parallel_processes". We do not alter the "ungroup" argument, so
storing it in the non-const structure would make this control flow
less obvious.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/run-command.c b/run-command.c
index 2b1cfbc996f..d07ee759b34 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1495,8 +1495,6 @@ enum child_state {
 };
 
 struct parallel_processes {
-	void *const data;
-
 	const size_t max_processes;
 	size_t nr_processes;
 
@@ -1607,7 +1605,7 @@ static int pp_start_one(struct parallel_processes *pp,
 
 	code = opts->get_next_task(&pp->children[i].process,
 				   opts->ungroup ? NULL : &pp->children[i].err,
-				   pp->data,
+				   opts->data,
 				   &pp->children[i].data);
 	if (!code) {
 		if (!opts->ungroup) {
@@ -1626,7 +1624,7 @@ static int pp_start_one(struct parallel_processes *pp,
 		if (opts->start_failure)
 			code = opts->start_failure(opts->ungroup ? NULL :
 						   &pp->children[i].err,
-						   pp->data,
+						   opts->data,
 						   pp->children[i].data);
 		else
 			code = 0;
@@ -1701,7 +1699,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 
 		if (opts->task_finished)
 			code = opts->task_finished(code, opts->ungroup ? NULL :
-						   &pp->children[i].err, pp->data,
+						   &pp->children[i].err, opts->data,
 						   pp->children[i].data);
 		else
 			code = 0;
@@ -1754,7 +1752,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	int spawn_cap = 4;
 	struct parallel_processes pp = {
 		.max_processes = opts->processes,
-		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
 	};
 	/* options */
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 20/22] run-command.c: use "opts->processes", not "pp->max_processes"

[Ævar Arnfjörð Bjarmason]

Neither the "processes" nor "max_processes" members ever change after
their initialization, and they're always equivalent, but some existing
code used "pp->max_processes" when we were already passing the "opts"
to the function, let's use the "opts" directly instead.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/run-command.c b/run-command.c
index d07ee759b34..b19b62749e0 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1597,10 +1597,10 @@ static int pp_start_one(struct parallel_processes *pp,
 	size_t i;
 	int code;
 
-	for (i = 0; i < pp->max_processes; i++)
+	for (i = 0; i < opts->processes; i++)
 		if (pp->children[i].state == GIT_CP_FREE)
 			break;
-	if (i == pp->max_processes)
+	if (i == opts->processes)
 		BUG("bookkeeping is hard");
 
 	code = opts->get_next_task(&pp->children[i].process,
@@ -1685,14 +1685,14 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			       const struct run_process_parallel_opts *opts)
 {
 	int code;
-	size_t i, n = pp->max_processes;
+	size_t i;
 	int result = 0;
 
 	while (pp->nr_processes > 0) {
-		for (i = 0; i < pp->max_processes; i++)
+		for (i = 0; i < opts->processes; i++)
 			if (pp->children[i].state == GIT_CP_WAIT_CLEANUP)
 				break;
-		if (i == pp->max_processes)
+		if (i == opts->processes)
 			break;
 
 		code = finish_command(&pp->children[i].process);
@@ -1721,6 +1721,8 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		} else {
+			const size_t n = opts->processes;
+
 			strbuf_write(&pp->children[i].err, stderr);
 			strbuf_reset(&pp->children[i].err);
 
@@ -1767,7 +1769,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	while (1) {
 		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
-		    pp.nr_processes < pp.max_processes;
+		    pp.nr_processes < opts->processes;
 		    i++) {
 			code = pp_start_one(&pp, opts);
 			if (!code)
@@ -1781,7 +1783,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		if (!pp.nr_processes)
 			break;
 		if (opts->ungroup) {
-			for (size_t i = 0; i < pp.max_processes; i++)
+			for (size_t i = 0; i < opts->processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
 			pp_buffer_stderr(&pp, output_timeout);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 21/22] run-command.c: pass "opts" further down, and use "opts->processes"

[Ævar Arnfjörð Bjarmason]

Continue the migration away from the "max_processes" member of "struct
parallel_processes" to the "processes" member of the "struct
run_process_parallel_opts", in this case we needed to pass the "opts"
further down into pp_cleanup() and pp_buffer_stderr().

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index b19b62749e0..6b91235ff2d 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1563,10 +1563,11 @@ static void pp_init(struct parallel_processes *pp,
 	sigchain_push_common(handle_children_on_signal);
 }
 
-static void pp_cleanup(struct parallel_processes *pp)
+static void pp_cleanup(struct parallel_processes *pp,
+		       const struct run_process_parallel_opts *opts)
 {
 	trace_printf("run_processes_parallel: done");
-	for (size_t i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < opts->processes; i++) {
 		strbuf_release(&pp->children[i].err);
 		child_process_clear(&pp->children[i].process);
 	}
@@ -1645,17 +1646,19 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
-static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
+static void pp_buffer_stderr(struct parallel_processes *pp,
+			     const struct run_process_parallel_opts *opts,
+			     int output_timeout)
 {
-	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
+	while (poll(pp->pfd, opts->processes, output_timeout) < 0) {
 		if (errno == EINTR)
 			continue;
-		pp_cleanup(pp);
+		pp_cleanup(pp, opts);
 		die_errno("poll");
 	}
 
 	/* Buffer output from all pipes. */
-	for (size_t i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < opts->processes; i++) {
 		if (pp->children[i].state == GIT_CP_WORKING &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
@@ -1786,7 +1789,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 			for (size_t i = 0; i < opts->processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
-			pp_buffer_stderr(&pp, output_timeout);
+			pp_buffer_stderr(&pp, opts, output_timeout);
 			pp_output(&pp);
 		}
 		code = pp_collect_finished(&pp, opts);
@@ -1797,7 +1800,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 	}
 
-	pp_cleanup(&pp);
+	pp_cleanup(&pp, opts);
 
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v2 22/22] run-command.c: remove "pp->max_processes", add "const" to signal() handler

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in a preceding commit, let's not copy
the "processes" member of the "struct run_process_parallel_opts" over
to the "struct parallel_processes".

In this case we need the number of processes for the kill_children()
function, which will be called from a signal handler. To do that
adjust this code added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) so that we use a
dedicated "struct parallel_processes_for_signal" for passing data to
the signal handler, in addition to the "struct parallel_process" it'll
now have access to our "opts" variable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)

diff --git a/run-command.c b/run-command.c
index 6b91235ff2d..a6c123ade1a 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1495,7 +1495,6 @@ enum child_state {
 };
 
 struct parallel_processes {
-	const size_t max_processes;
 	size_t nr_processes;
 
 	struct {
@@ -1516,24 +1515,38 @@ struct parallel_processes {
 	struct strbuf buffered_output; /* of finished children */
 };
 
-static void kill_children(const struct parallel_processes *pp, int signo)
+struct parallel_processes_for_signal {
+	const struct run_process_parallel_opts *opts;
+	const struct parallel_processes *pp;
+};
+
+static void kill_children(const struct parallel_processes *pp,
+			  const struct run_process_parallel_opts *opts,
+			  int signo)
 {
-	for (size_t i = 0; i < pp->max_processes; i++)
+	for (size_t i = 0; i < opts->processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
 			kill(pp->children[i].process.pid, signo);
 }
 
-static struct parallel_processes *pp_for_signal;
+static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,
+				 int signo)
+{
+	kill_children(pp_sig->pp, pp_sig->opts, signo);
+}
+
+static struct parallel_processes_for_signal *pp_for_signal;
 
 static void handle_children_on_signal(int signo)
 {
-	kill_children(pp_for_signal, signo);
+	kill_children_signal(pp_for_signal, signo);
 	sigchain_pop(signo);
 	raise(signo);
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    const struct run_process_parallel_opts *opts)
+		    const struct run_process_parallel_opts *opts,
+		    struct parallel_processes_for_signal *pp_sig)
 {
 	const size_t n = opts->processes;
 
@@ -1559,7 +1572,8 @@ static void pp_init(struct parallel_processes *pp,
 		}
 	}
 
-	pp_for_signal = pp;
+	pp_sig->pp = pp;
+	pp_sig->opts = opts;
 	sigchain_push_common(handle_children_on_signal);
 }
 
@@ -1755,8 +1769,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	int code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
+	struct parallel_processes_for_signal pp_sig;
 	struct parallel_processes pp = {
-		.max_processes = opts->processes,
 		.buffered_output = STRBUF_INIT,
 	};
 	/* options */
@@ -1768,7 +1782,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
 					   "max:%d", opts->processes);
 
-	pp_init(&pp, opts);
+	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (int i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1779,7 +1793,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 				continue;
 			if (code < 0) {
 				pp.shutdown = 1;
-				kill_children(&pp, -code);
+				kill_children(&pp, opts, -code);
 			}
 			break;
 		}
@@ -1796,7 +1810,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		if (code) {
 			pp.shutdown = 1;
 			if (code < 0)
-				kill_children(&pp, -code);
+				kill_children(&pp, opts,-code);
 		}
 	}
 
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v2 22/22] run-command.c: remove "pp->max_processes", add "const" to signal() handler

[Ævar Arnfjörð Bjarmason]

On Wed, Oct 12 2022, Ævar Arnfjörð Bjarmason wrote:

There's a really stupid logic error here where...

> -static struct parallel_processes *pp_for_signal;
> +static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,
> +				 int signo)
> +{
> +	kill_children(pp_sig->pp, pp_sig->opts, signo);

...this pp_sig is always NULL...

> +}
> +
> +static struct parallel_processes_for_signal *pp_for_signal;
>  
>  static void handle_children_on_signal(int signo)
>  {
> -	kill_children(pp_for_signal, signo);
> +	kill_children_signal(pp_for_signal, signo);
>  	sigchain_pop(signo);
>  	raise(signo);
>  }
>  
>  static void pp_init(struct parallel_processes *pp,
> -		    const struct run_process_parallel_opts *opts)
> +		    const struct run_process_parallel_opts *opts,
> +		    struct parallel_processes_for_signal *pp_sig)
>  {
>  	const size_t n = opts->processes;
>  
> @@ -1559,7 +1572,8 @@ static void pp_init(struct parallel_processes *pp,
>  		}
>  	}
>  
> -	pp_for_signal = pp;

...because this removal has no corresponding addition...

> +	pp_sig->pp = pp;
> +	pp_sig->opts = opts;

...there's a missing:

	pp_for_signal = pp_sig;

Here, sorry!

Junio: If you do pick up this re-roll please just peel off this 22/22.

Re: [PATCH v2 00/22] run-command API: pass functions & opts via struct

[Phillip Wood]

Hi Ævar

On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:
> This series changes the run-command API so that we pass options via a
> struct instead of via the argument list, the end result is that an API
> user looks like e.g.:
> 
>       +       const struct run_process_parallel_opts opts = {
>       +               .tr2_category = "submodule",
>       +               .tr2_label = "parallel/update",
>       +
>       +               .processes = update_data->max_jobs,
>       +
>       +               .get_next_task = update_clone_get_next_task,
>       +               .start_failure = update_clone_start_failure,
>       +               .task_finished = update_clone_task_finished,
>       +               .data = &suc,
>       +       };
>       [...]
>       -       run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
>       -                                  update_clone_start_failure,
>       -                                  update_clone_task_finished, &suc, "submodule",
>       -                                  "parallel/update");
>       +       run_processes_parallel(&opts);

I thought the first version was longer than it needed to be and now it 
has grown by 50%. I think there are a number of changes here that are 
not related to converting run_processes_parallel() to take a struct of 
options. My feeling is that the test cleanups are worthwhile but the 
changes in patches 8-10 are needed and patches 12-14 would be better 
squashed together.

I'm afraid that I'm think this fits a pattern of submitting series that 
incorporate unrelated changes and so end up longer than they need to be. 
I think this leads to poorer reviews and a greater change of regressions 
because reviewers are overwhelmed by the sheer volume of changes they're 
expected to review. I find it particularly fustrating to review code in 
one patch only to find it is deleted in the next patch.

I think the overall goal here is worthwhile, but I'd really like to see 
a much shorter more focused series. In general two shorter series are 
much more likely to receive good quality reviews than one longer series 
making the same changes.

Best Wishes

Phillip

[PATCH v3 00/15] run-command API: pass functions & opts via struct

[Ævar Arnfjörð Bjarmason]

For a general overview see the v2 CL:
https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/

Changes since v2:

 * Ejected various not-to-the-point of converting to the "opts struct"
   per feedback about attempting to make this leaner.
 * I kept the size_t change & the online_cpus() fallback, and updated
   a commit message for the latter. For "int" v.s. "size_t" once we're
   not handling "-1" to mean "use the default" it makes sense to be
   unsigned, and if we're re-doing that at this point we'd need
   rewrites for "!processes" assumptions.
 * Squashed the multi-step introduction of the "opts" struct, per
   Phillip's suggestion.
 * Fixed a segfault in the v2's 22/22 (this 15/15).
 * Got rid of superfluous unsigned conversions of code related to the
   "env" member.

Ævar Arnfjörð Bjarmason (15):
  run-command test helper: use "else if" pattern
  run-command API: have "run_processes_parallel{,_tr2}()" return void
  run-command tests: use "return", not "exit"
  run-command API: make "n" parameter a "size_t"
  run-command API: don't fall back on online_cpus()
  run-command.c: use designated init for pp_init(), add "const"
  run-command API: have run_process_parallel() take an "opts" struct
  run-command API: move *_tr2() users to "run_processes_parallel()"
  run-command.c: make "struct parallel_processes" const if possible
  run-command.c: don't copy *_fn to "struct parallel_processes"
  run-command.c: don't copy "ungroup" to "struct parallel_processes"
  run-command.c: don't copy "data" to "struct parallel_processes"
  run-command.c: use "opts->processes", not "pp->max_processes"
  run-command.c: pass "opts" further down, and use "opts->processes"
  run-command.c: remove "max_processes", add "const" to signal() handler

 builtin/fetch.c             |  25 ++--
 builtin/submodule--helper.c |  16 ++-
 hook.c                      |  23 ++--
 run-command.c               | 236 ++++++++++++++++--------------------
 run-command.h               |  71 ++++++++---
 submodule-config.c          |   2 +
 submodule.c                 |  18 ++-
 t/helper/test-run-command.c |  77 +++++++-----
 t/t5526-fetch-submodules.sh |   8 +-
 9 files changed, 268 insertions(+), 208 deletions(-)

Range-diff against v2:
 1:  bc51dfcb1be <  -:  ----------- hook tests: fix redirection logic error in 96e7225b310
 2:  3027f5587a7 <  -:  ----------- submodule tests: reset "trace.out" between "grep" invocations
 3:  c4923358bbd <  -:  ----------- run-command tests: test stdout of run_command_parallel()
 4:  26e28086252 =  1:  d3a2489d9b2 run-command test helper: use "else if" pattern
 5:  5e09dc68fd9 =  2:  a2e4fd652c1 run-command API: have "run_processes_parallel{,_tr2}()" return void
 6:  e4e91dbbf9e =  3:  4a19de65783 run-command tests: use "return", not "exit"
 7:  b90961ae76d <  -:  ----------- run-command.c: remove dead assignment in while-loop
 8:  279b0430c5d <  -:  ----------- run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful
 9:  a900711270c !  4:  58018a79b2f run-command API: make "n" parameter a "size_t"
    @@ Commit message
     
                 make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
     
    -    Only has one (and new) -Wsigned-compare warning, about a comparison of
    -    "i" to online_cpus(), a subsequent commit will adjust & deal with
    -    online_cpus() and that warning.
    +    Only has one (and new) -Wsigned-compare warning relevant to a
    +    comparison about our "n" or "{nr,max}_processes": About using our
    +    "n" (size_t) in the same expression as online_cpus() (int). A
    +    subsequent commit will adjust & deal with online_cpus() and that
    +    warning.
     
         The only users of the "n" parameter are:
     
    @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
      
      	for (i = 0; i < pp->max_processes; i++)
      		if (pp->children[i].state == GIT_CP_FREE)
    -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
    - 
    - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
    - {
    --	int i;
    --
    - 	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
    - 		if (errno == EINTR)
    - 			continue;
     @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
      	}
      
10:  eb9d672b0d8 !  5:  e230701dff6 run-command API: don't fall back on online_cpus()
    @@ Commit message
         child processor, 2015-12-15).
     
         Most of our code in-tree that scales up to "online_cpus()" by default
    -    calls that function by itself. By having these callers of the
    -    "run_processes_parallel()" API do the same we can in subsequent
    -    commits pass all arguments down as a "const struct".
    +    calls that function by itself. Keeping this default behavior just for
    +    the sake of two callers means that we'd need to maintain this one spot
    +    where we're second-guessing the config passed down into pp_init().
     
         The preceding commit has an overview of the API callers that passed
         "jobs = 0". There were only two of them (actually three, but they
    @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v
     
      ## t/t5526-fetch-submodules.sh ##
     @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '
    + 		GIT_TRACE=$(pwd)/trace.out git fetch &&
    + 		grep "8 tasks" trace.out &&
      		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
    - 		grep "9 tasks" trace.out &&
    - 		>trace.out &&
    +-		grep "9 tasks" trace.out
    ++		grep "9 tasks" trace.out &&
    ++		>trace.out &&
     +
     +		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
     +		grep "preparing to run up to [0-9]* tasks" trace.out &&
11:  aedda10d8e1 !  6:  df2ca5dd097 run-command.c: use designated init for pp_init(), add "const"
    @@ run-command.c: void run_processes_parallel(size_t n,
     -		ungroup);
     +	pp_init(&pp, get_next_task, start_failure, task_finished);
      	while (1) {
    - 		for (int i = 0;
    + 		for (i = 0;
      		    i < spawn_cap && !pp.shutdown &&
12:  fde2af11579 <  -:  ----------- run-command API: add nascent "struct run_process_parallel_opts"
13:  01e894bed90 <  -:  ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers
14:  41c2886b44b !  7:  eaed3d8838d run-command API: have run_process_parallel() take an "opts" struct
    @@ Metadata
      ## Commit message ##
         run-command API: have run_process_parallel() take an "opts" struct
     
    -    Have the "run_process_parallel()" function take an "opts" struct,
    -    which allows us to eliminate it as a wrapper for
    -    "run_processes_parallel_1()".
    +    As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
    +    run_process_parallel(), 2022-06-07) which added the "ungroup" passing
    +    it to "run_process_parallel()" via the global
    +    "run_processes_parallel_ungroup" variable was a compromise to get the
    +    smallest possible regression fix for "maint" at the time.
    +
    +    This follow-up to that is a start at passing that parameter and others
    +    via a new "struct run_process_parallel_opts", as the earlier
    +    version[1] of what became fd3aaf53f71 did.
    +
    +    Since we need to change all of the occurrences of "n" to
    +    "opt->SOMETHING" let's take the opportunity and rename the terse "n"
    +    to "processes". We could also have picked "max_processes", "jobs",
    +    "threads" etc., but as the API is named "run_processes_parallel()"
    +    let's go with "processes".
     
         Since the new "run_processes_parallel()" function is able to take an
         optional "tr2_category" and "tr2_label" via the struct we can at this
         point migrate all of the users of "run_processes_parallel_tr2()" over
         to it.
     
    -    But let's not migrate all the API users, only the two users that
    +    But let's not migrate all the API users yet, only the two users that
         passed the "ungroup" parameter via the
    -    "run_processes_parallel_ungroup" global, allowing us to delete that
    -    global in favor of passing "ungroup" via the "opts" struct. As noted
    -    in fd3aaf53f71 (run-command: add an "ungroup" option to
    -    run_process_parallel(), 2022-06-07) which added
    -    "run_processes_parallel_ungroup" passing this as a global was a hack
    -    to produce a small regression fix for "maint".
    +    "run_processes_parallel_ungroup" global
    +
    +    1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ run-command.c: enum child_state {
      struct parallel_processes {
      	void *const data;
      
    +@@ run-command.c: static void handle_children_on_signal(int signo)
    + }
    + 
    + static void pp_init(struct parallel_processes *pp,
    +-		    get_next_task_fn get_next_task,
    +-		    start_failure_fn start_failure,
    +-		    task_finished_fn task_finished)
    ++		    const struct run_process_parallel_opts *opts)
    + {
    +-	const size_t n = pp->max_processes;
    ++	const size_t n = opts->processes;
    ++	get_next_task_fn get_next_task = opts->get_next_task;
    ++	start_failure_fn start_failure = opts->start_failure;
    ++	task_finished_fn task_finished = opts->task_finished;
    + 
    + 	if (!n)
    + 		BUG("you must provide a non-zero number of processes!");
     @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
      	return result;
      }
      
    --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
    +-void run_processes_parallel(size_t n,
    +-			    get_next_task_fn get_next_task,
    +-			    start_failure_fn start_failure,
    +-			    task_finished_fn task_finished,
    +-			    void *pp_cb)
     +void run_processes_parallel(const struct run_process_parallel_opts *opts)
      {
    - 	int code;
    + 	int i, code;
      	int output_timeout = 100;
      	int spawn_cap = 4;
    +-	int ungroup = run_processes_parallel_ungroup;
      	struct parallel_processes pp = {
    - 		.max_processes = opts->processes,
    +-		.max_processes = n,
     -		.data = pp_cb,
    ++		.max_processes = opts->processes,
     +		.data = opts->data,
      		.buffered_output = STRBUF_INIT,
    --		.ungroup = run_processes_parallel_ungroup,
    +-		.ungroup = ungroup,
     +		.ungroup = opts->ungroup,
      	};
    - 	/* options */
    - 	const char *tr2_category = opts->tr2_category;
    - 	const char *tr2_label = opts->tr2_label;
    - 	const int do_trace2 = tr2_category && tr2_label;
    ++	/* options */
    ++	const char *tr2_category = opts->tr2_category;
    ++	const char *tr2_label = opts->tr2_label;
    ++	const int do_trace2 = tr2_category && tr2_label;
      
     -	/* unset for the next API user */
     -	run_processes_parallel_ungroup = 0;
    --
    - 	if (do_trace2)
    - 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
    - 					   "max:%d", opts->processes);
    -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
    - 		trace2_region_leave(tr2_category, tr2_label, NULL);
    ++	if (do_trace2)
    ++		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
    ++					   "max:%d", opts->processes);
    + 
    +-	pp_init(&pp, get_next_task, start_failure, task_finished);
    ++	pp_init(&pp, opts);
    + 	while (1) {
    + 		for (i = 0;
    + 		    i < spawn_cap && !pp.shutdown &&
    +@@ run-command.c: void run_processes_parallel(size_t n,
    + 		}
    + 		if (!pp.nr_processes)
    + 			break;
    +-		if (ungroup) {
    ++		if (opts->ungroup) {
    + 			for (size_t i = 0; i < pp.max_processes; i++)
    + 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
    + 		} else {
    +@@ run-command.c: void run_processes_parallel(size_t n,
    + 	}
    + 
    + 	pp_cleanup(&pp);
    ++
    ++	if (do_trace2)
    ++		trace2_region_leave(tr2_category, tr2_label, NULL);
      }
      
    --void run_processes_parallel(size_t processes,
    --			    get_next_task_fn get_next_task,
    --			    start_failure_fn start_failure,
    --			    task_finished_fn task_finished,
    --			    void *pp_cb)
    --{
    --	const struct run_process_parallel_opts opts = {
    --		.processes = processes,
    --		.ungroup = run_processes_parallel_ungroup,
    --
    --		.get_next_task = get_next_task,
    --		.start_failure = start_failure,
    --		.task_finished = task_finished,
    --	};
    --
    --	run_processes_parallel_1(&opts, pp_cb);
    --}
    --
    - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
    +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
    ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
      				start_failure_fn start_failure,
      				task_finished_fn task_finished, void *pp_cb,
    -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task
    - 	const struct run_process_parallel_opts opts = {
    - 		.tr2_category = tr2_category,
    - 		.tr2_label = tr2_label,
    --
    - 		.processes = processes,
    --		.ungroup = run_processes_parallel_ungroup,
    + 				const char *tr2_category, const char *tr2_label)
    + {
    +-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
    ++	const struct run_process_parallel_opts opts = {
    ++		.tr2_category = tr2_category,
    ++		.tr2_label = tr2_label,
    ++		.processes = processes,
      
    - 		.get_next_task = get_next_task,
    - 		.start_failure = start_failure,
    - 		.task_finished = task_finished,
    - 	};
    +-	run_processes_parallel(n, get_next_task, start_failure,
    +-			       task_finished, pp_cb);
    ++		.get_next_task = get_next_task,
    ++		.start_failure = start_failure,
    ++		.task_finished = task_finished,
    ++	};
      
    --	run_processes_parallel_1(&opts, pp_cb);
    +-	trace2_region_leave(tr2_category, tr2_label, NULL);
     +	run_processes_parallel(&opts);
      }
      
      int run_auto_maintenance(int quiet)
     
      ## run-command.h ##
    -@@ run-command.h: struct run_process_parallel_opts
    - 	 * NULL to omit any special handling.
    - 	 */
    - 	task_finished_fn task_finished;
    +@@ run-command.h: typedef int (*task_finished_fn)(int result,
    + 				void *pp_task_cb);
    + 
    + /**
    +- * Runs up to n processes at the same time. Whenever a process can be
    +- * started, the callback get_next_task_fn is called to obtain the data
    ++ * Option used by run_processes_parallel(), { 0 }-initialized means no
    ++ * options.
    ++ */
    ++struct run_process_parallel_opts
    ++{
    ++	/**
    ++	 * tr2_category & tr2_label: sets the trace2 category and label for
    ++	 * logging. These must either be unset, or both of them must be set.
    ++	 */
    ++	const char *tr2_category;
    ++	const char *tr2_label;
    ++
    ++	/**
    ++	 * processes: see 'processes' in run_processes_parallel() below.
    ++	 */
    ++	size_t processes;
    ++
    ++	/**
    ++	 * ungroup: see 'ungroup' in run_processes_parallel() below.
    ++	 */
    ++	unsigned int ungroup:1;
    ++
    ++	/**
    ++	 * get_next_task: See get_next_task_fn() above. This must be
    ++	 * specified.
    ++	 */
    ++	get_next_task_fn get_next_task;
    ++
    ++	/**
    ++	 * start_failure: See start_failure_fn() above. This can be
    ++	 * NULL to omit any special handling.
    ++	 */
    ++	start_failure_fn start_failure;
    ++
    ++	/**
    ++	 * task_finished: See task_finished_fn() above. This can be
    ++	 * NULL to omit any special handling.
    ++	 */
    ++	task_finished_fn task_finished;
     +
     +	/**
     +	 * data: user data, will be passed as "pp_cb" to the callback
     +	 * parameters.
     +	 */
     +	void *data;
    - };
    - 
    - /**
    ++};
    ++
    ++/**
     + * Options are passed via the "struct run_process_parallel_opts" above.
     + *
    -  * Runs N 'processes' at the same time. Whenever a process can be
    -- * started, the callback get_next_task_fn is called to obtain the data
    ++ * Runs N 'processes' at the same time. Whenever a process can be
     + * started, the callback opts.get_next_task is called to obtain the data
       * required to start another child process.
       *
       * The children started via this function run in parallel. Their output
    -@@ run-command.h: struct run_process_parallel_opts
    +  * (both stdout and stderr) is routed to stderr in a manner that output
    +  * from different tasks does not interleave (but see "ungroup" below).
    +  *
    +- * start_failure_fn and task_finished_fn can be NULL to omit any
    +- * special handling.
    +- *
    +  * If the "ungroup" option isn't specified, the API will set the
    +  * "stdout_to_stderr" parameter in "struct child_process" and provide
    +  * the callbacks with a "struct strbuf *out" parameter to write output
    +@@ run-command.h: typedef int (*task_finished_fn)(int result,
       * NULL "struct strbuf *out" parameter, and are responsible for
       * emitting their own output, including dealing with any race
       * conditions due to writing in parallel to stdout and stderr.
    @@ run-command.h: struct run_process_parallel_opts
     - * API reads that setting.
       */
     -extern int run_processes_parallel_ungroup;
    --void run_processes_parallel(size_t processes,
    +-void run_processes_parallel(size_t n,
     -			    get_next_task_fn,
     -			    start_failure_fn,
     -			    task_finished_fn,
     -			    void *pp_cb);
    +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
    +-				task_finished_fn, void *pp_cb,
     +void run_processes_parallel(const struct run_process_parallel_opts *opts);
    - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
    - 				start_failure_fn, task_finished_fn, void *pp_cb,
    ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
    ++				start_failure_fn, task_finished_fn, void *pp_cb,
      				const char *tr2_category, const char *tr2_label);
    + 
    + /**
     
      ## t/helper/test-run-command.c ##
     @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {
15:  391d1d99d91 =  8:  c19c60b2e95 run-command API: move *_tr2() users to "run_processes_parallel()"
16:  acac50cc1a5 =  9:  99a4f4f6b9c run-command.c: make "struct parallel_processes" const if possible
17:  fdd64236985 = 10:  bf67e24bcc5 run-command.c: don't copy *_fn to "struct parallel_processes"
18:  17f34d81ecd = 11:  06de42adc2e run-command.c: don't copy "ungroup" to "struct parallel_processes"
19:  9cbee2dfe76 = 12:  3081dfc49d1 run-command.c: don't copy "data" to "struct parallel_processes"
20:  2dabed9e155 ! 13:  b7c10f6a23f run-command.c: use "opts->processes", not "pp->max_processes"
    @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
      
     @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
      	while (1) {
    - 		for (int i = 0;
    + 		for (i = 0;
      		    i < spawn_cap && !pp.shutdown &&
     -		    pp.nr_processes < pp.max_processes;
     +		    pp.nr_processes < opts->processes;
21:  c1a286a8ebb ! 14:  4856d6a4674 run-command.c: pass "opts" further down, and use "opts->processes"
    @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
     +			     const struct run_process_parallel_opts *opts,
     +			     int output_timeout)
      {
    --	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
    -+	while (poll(pp->pfd, opts->processes, output_timeout) < 0) {
    + 	int i;
    + 
    +-	while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
    ++	while ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {
      		if (errno == EINTR)
      			continue;
     -		pp_cleanup(pp);
22:  541f41566e7 ! 15:  39a20be0cbb run-command.c: remove "pp->max_processes", add "const" to signal() handler
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    run-command.c: remove "pp->max_processes", add "const" to signal() handler
    +    run-command.c: remove "max_processes", add "const" to signal() handler
     
         As with the *_fn members removed in a preceding commit, let's not copy
         the "processes" member of the "struct run_process_parallel_opts" over
    @@ run-command.c: static void pp_init(struct parallel_processes *pp,
     -	pp_for_signal = pp;
     +	pp_sig->pp = pp;
     +	pp_sig->opts = opts;
    ++	pp_for_signal = pp_sig;
      	sigchain_push_common(handle_children_on_signal);
      }
      
     @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
    - 	int code;
    + 	int i, code;
      	int output_timeout = 100;
      	int spawn_cap = 4;
     +	struct parallel_processes_for_signal pp_sig;
    @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
     -	pp_init(&pp, opts);
     +	pp_init(&pp, opts, &pp_sig);
      	while (1) {
    - 		for (int i = 0;
    + 		for (i = 0;
      		    i < spawn_cap && !pp.shutdown &&
     @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
      				continue;
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 01/15] run-command test helper: use "else if" pattern

[Ævar Arnfjörð Bjarmason]

Adjust the cmd__run_command() to use an "if/else if" chain rather than
mutually exclusive "if" statements. This non-functional change makes a
subsequent commit smaller.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index c9283b47afa..390fa4fb724 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_clear(&proc.args);
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
-	if (!strcmp(argv[1], "run-command-parallel"))
+	if (!strcmp(argv[1], "run-command-parallel")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, NULL, &proc));
-
-	if (!strcmp(argv[1], "run-command-abort"))
+	} else if (!strcmp(argv[1], "run-command-abort")) {
 		exit(run_processes_parallel(jobs, parallel_next,
 					    NULL, task_finished, &proc));
-
-	if (!strcmp(argv[1], "run-command-no-jobs"))
+	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
 		exit(run_processes_parallel(jobs, no_job,
 					    NULL, task_finished, &proc));
-
-	fprintf(stderr, "check usage\n");
-	return 1;
+	} else {
+		fprintf(stderr, "check usage\n");
+		return 1;
+	}
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 02/15] run-command API: have "run_processes_parallel{,_tr2}()" return void

[Ævar Arnfjörð Bjarmason]

Change the "run_processes_parallel{,_tr2}()" functions to return void,
instead of int. Ever since c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) they have
unconditionally returned 0.

To get a "real" return value out of this function the caller needs to
get it via the "task_finished_fn" callback, see the example in hook.c
added in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).

So the "result = " and "if (!result)" code added to "builtin/fetch.c"
d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
2019-10-05) has always been redundant, we always took that "if"
path. Likewise the "ret =" in "t/helper/test-run-command.c" added in
be5d88e1128 (test-tool run-command: learn to run (parts of) the
testsuite, 2019-10-04) wasn't used, instead we got the return value
from the "if (suite.failed.nr > 0)" block seen in the context.

Subsequent commits will alter this API interface, getting rid of this
always-zero return value makes it easier to understand those changes.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 17 ++++++++---------
 run-command.c               | 27 +++++++++++----------------
 run-command.h               | 16 ++++++++--------
 t/helper/test-run-command.c | 16 ++++++++--------
 4 files changed, 35 insertions(+), 41 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index a0fca93bb6a..78043fb67ef 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
 
 		strvec_push(&argv, "--end-of-options");
-		result = run_processes_parallel_tr2(max_children,
-						    &fetch_next_remote,
-						    &fetch_failed_to_start,
-						    &fetch_finished,
-						    &state,
-						    "fetch", "parallel/fetch");
-
-		if (!result)
-			result = state.result;
+		run_processes_parallel_tr2(max_children,
+					   &fetch_next_remote,
+					   &fetch_failed_to_start,
+					   &fetch_finished,
+					   &state,
+					   "fetch", "parallel/fetch");
+
+		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
 			const char *name = list->items[i].string;
diff --git a/run-command.c b/run-command.c
index 5ec3a46dccf..642e6b6e057 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-int run_processes_parallel(int n,
-			   get_next_task_fn get_next_task,
-			   start_failure_fn start_failure,
-			   task_finished_fn task_finished,
-			   void *pp_cb)
+void run_processes_parallel(int n,
+			    get_next_task_fn get_next_task,
+			    start_failure_fn start_failure,
+			    task_finished_fn task_finished,
+			    void *pp_cb)
 {
 	int i, code;
 	int output_timeout = 100;
@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,
 	}
 
 	pp_cleanup(&pp);
-	return 0;
 }
 
-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
-			       start_failure_fn start_failure,
-			       task_finished_fn task_finished, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label)
+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+				start_failure_fn start_failure,
+				task_finished_fn task_finished, void *pp_cb,
+				const char *tr2_category, const char *tr2_label)
 {
-	int result;
-
 	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
 				   ((n < 1) ? online_cpus() : n));
 
-	result = run_processes_parallel(n, get_next_task, start_failure,
-					task_finished, pp_cb);
+	run_processes_parallel(n, get_next_task, start_failure,
+			       task_finished, pp_cb);
 
 	trace2_region_leave(tr2_category, tr2_label, NULL);
-
-	return result;
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 0e85e5846a5..e76a1b6b5b3 100644
--- a/run-command.h
+++ b/run-command.h
@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-int run_processes_parallel(int n,
-			   get_next_task_fn,
-			   start_failure_fn,
-			   task_finished_fn,
-			   void *pp_cb);
-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
-			       task_finished_fn, void *pp_cb,
-			       const char *tr2_category, const char *tr2_label);
+void run_processes_parallel(int n,
+			    get_next_task_fn,
+			    start_failure_fn,
+			    task_finished_fn,
+			    void *pp_cb);
+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+				task_finished_fn, void *pp_cb,
+				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 390fa4fb724..30c474f3243 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	ret = run_processes_parallel(max_jobs, next_test, test_failed,
-				     test_finished, &suite);
+	run_processes_parallel(max_jobs, next_test, test_failed,
+			       test_finished, &suite);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
@@ -428,16 +428,16 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, NULL, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		exit(run_processes_parallel(jobs, parallel_next,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, parallel_next, NULL,
+				       task_finished, &proc);
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		exit(run_processes_parallel(jobs, no_job,
-					    NULL, task_finished, &proc));
+		run_processes_parallel(jobs, no_job, NULL, task_finished,
+				       &proc);
 	} else {
 		fprintf(stderr, "check usage\n");
 		return 1;
 	}
+	exit(0);
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 03/15] run-command tests: use "return", not "exit"

[Ævar Arnfjörð Bjarmason]

Change the "run-command" test helper to "return" instead of calling
"exit", see 338abb0f045 (builtins + test helpers: use return instead
of exit() in cmd_*, 2021-06-08)

Because we'd previously gotten past the SANITIZE=leak check by using
exit() here we need to move to "goto cleanup" pattern.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/helper/test-run-command.c | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 30c474f3243..ee509aefa2f 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)
 {
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
+	int ret;
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
-		exit(testsuite(argc - 1, argv + 1));
+		return testsuite(argc - 1, argv + 1);
 	if (!strcmp(argv[1], "inherited-handle"))
-		exit(inherit_handle(argv[0]));
+		return inherit_handle(argv[0]);
 	if (!strcmp(argv[1], "inherited-handle-child"))
-		exit(inherit_handle_child());
+		return inherit_handle_child();
 
 	if (argc >= 2 && !strcmp(argv[1], "quote-stress-test"))
 		return !!quote_stress_test(argc - 1, argv + 1);
@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)
 		argv += 2;
 		argc -= 2;
 	}
-	if (argc < 3)
-		return 1;
+	if (argc < 3) {
+		ret = 1;
+		goto cleanup;
+	}
 	strvec_pushv(&proc.args, (const char **)argv + 2);
 
 	if (!strcmp(argv[1], "start-command-ENOENT")) {
-		if (start_command(&proc) < 0 && errno == ENOENT)
-			return 0;
+		if (start_command(&proc) < 0 && errno == ENOENT) {
+			ret = 0;
+			goto cleanup;
+		}
 		fprintf(stderr, "FAIL %s\n", argv[1]);
 		return 1;
 	}
-	if (!strcmp(argv[1], "run-command"))
-		exit(run_command(&proc));
+	if (!strcmp(argv[1], "run-command")) {
+		ret = run_command(&proc);
+		goto cleanup;
+	}
 
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
@@ -436,8 +443,12 @@ int cmd__run_command(int argc, const char **argv)
 		run_processes_parallel(jobs, no_job, NULL, task_finished,
 				       &proc);
 	} else {
+		ret = 1;
 		fprintf(stderr, "check usage\n");
-		return 1;
+		goto cleanup;
 	}
-	exit(0);
+	ret = 0;
+cleanup:
+	child_process_clear(&proc);
+	return ret;
 }
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 04/15] run-command API: make "n" parameter a "size_t"

[Ævar Arnfjörð Bjarmason]

Make the "n" variable added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) a "size_t". As
we'll see in a subsequent commit we do pass "0" here, but never "jobs
< 0".

We could have made it an "unsigned int", but as we're having to change
this let's not leave another case in the codebase where a size_t and
"unsigned int" size differ on some platforms. In this case it's likely
to never matter, but it's easier to not need to worry about it.

After this and preceding changes:

	make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter

Only has one (and new) -Wsigned-compare warning relevant to a
comparison about our "n" or "{nr,max}_processes": About using our
"n" (size_t) in the same expression as online_cpus() (int). A
subsequent commit will adjust & deal with online_cpus() and that
warning.

The only users of the "n" parameter are:

 * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
   config. As seen in the code that parses the config added in
   d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
   2019-10-05) will die if the git_config_int() return value is < 0.

   It will however pass us n = 0, as we'll see in a subsequent commit.

 * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
   config. Read via code originally added in a028a1930c6 (fetching
   submodules: respect `submodule.fetchJobs` config option, 2016-02-29).

   It now piggy-backs on the the submodule.fetchJobs code and
   validation added in f20e7c1ea24 (submodule: remove
   submodule.fetchjobs from submodule-config parsing, 2017-08-02).

   Like builtin/fetch.c it will die if the git_config_int() return
   value is < 0, but like builtin/fetch.c it will pass us n = 0.

 * builtin/submodule--helper.c: defaults to 1. Read via code
   originally added in 2335b870fa7 (submodule update: expose parallelism
   to the user, 2016-02-29).

   Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
   submodule-config parsing, 2017-08-02) it shares a config parser and
   semantics with the submodule.c caller.

 * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
   subcommand, 2021-12-22).

 * t/helper/test-run-command.c: can be -1 after parsing the arguments,
   but will then be overridden to online_cpus() before passing it to
   this API. See be5d88e1128 (test-tool run-command: learn to run (parts
   of) the testsuite, 2019-10-04).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 42 ++++++++++++++++++------------------------
 run-command.h |  4 ++--
 2 files changed, 20 insertions(+), 26 deletions(-)

diff --git a/run-command.c b/run-command.c
index 642e6b6e057..14a6e38e804 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *data;
 
-	int max_processes;
-	int nr_processes;
+	size_t max_processes;
+	size_t nr_processes;
 
 	get_next_task_fn get_next_task;
 	start_failure_fn start_failure;
@@ -1522,7 +1522,7 @@ struct parallel_processes {
 	unsigned shutdown : 1;
 	unsigned ungroup : 1;
 
-	int output_owner;
+	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
 };
 
@@ -1543,9 +1543,7 @@ static int default_task_finished(int result,
 
 static void kill_children(struct parallel_processes *pp, int signo)
 {
-	int i, n = pp->max_processes;
-
-	for (i = 0; i < n; i++)
+	for (size_t i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
 			kill(pp->children[i].process.pid, signo);
 }
@@ -1560,20 +1558,19 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    int n,
+		    size_t n,
 		    get_next_task_fn get_next_task,
 		    start_failure_fn start_failure,
 		    task_finished_fn task_finished,
 		    void *data, int ungroup)
 {
-	int i;
-
 	if (n < 1)
 		n = online_cpus();
 
 	pp->max_processes = n;
 
-	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
+	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
+		     (uintmax_t)n);
 
 	pp->data = data;
 	if (!get_next_task)
@@ -1594,7 +1591,7 @@ static void pp_init(struct parallel_processes *pp,
 		CALLOC_ARRAY(pp->pfd, n);
 	strbuf_init(&pp->buffered_output, 0);
 
-	for (i = 0; i < n; i++) {
+	for (size_t i = 0; i < n; i++) {
 		strbuf_init(&pp->children[i].err, 0);
 		child_process_init(&pp->children[i].process);
 		if (pp->pfd) {
@@ -1609,10 +1606,8 @@ static void pp_init(struct parallel_processes *pp,
 
 static void pp_cleanup(struct parallel_processes *pp)
 {
-	int i;
-
 	trace_printf("run_processes_parallel: done");
-	for (i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < pp->max_processes; i++) {
 		strbuf_release(&pp->children[i].err);
 		child_process_clear(&pp->children[i].process);
 	}
@@ -1639,7 +1634,8 @@ static void pp_cleanup(struct parallel_processes *pp)
  */
 static int pp_start_one(struct parallel_processes *pp)
 {
-	int i, code;
+	size_t i;
+	int code;
 
 	for (i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_FREE)
@@ -1697,7 +1693,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 	}
 
 	/* Buffer output from all pipes. */
-	for (i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < pp->max_processes; i++) {
 		if (pp->children[i].state == GIT_CP_WORKING &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
@@ -1714,7 +1710,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 
 static void pp_output(struct parallel_processes *pp)
 {
-	int i = pp->output_owner;
+	size_t i = pp->output_owner;
 
 	if (pp->children[i].state == GIT_CP_WORKING &&
 	    pp->children[i].err.len) {
@@ -1725,8 +1721,8 @@ static void pp_output(struct parallel_processes *pp)
 
 static int pp_collect_finished(struct parallel_processes *pp)
 {
-	int i, code;
-	int n = pp->max_processes;
+	int code;
+	size_t i, n = pp->max_processes;
 	int result = 0;
 
 	while (pp->nr_processes > 0) {
@@ -1783,7 +1779,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(int n,
+void run_processes_parallel(size_t n,
 			    get_next_task_fn get_next_task,
 			    start_failure_fn start_failure,
 			    task_finished_fn task_finished,
@@ -1817,9 +1813,7 @@ void run_processes_parallel(int n,
 		if (!pp.nr_processes)
 			break;
 		if (ungroup) {
-			int i;
-
-			for (i = 0; i < pp.max_processes; i++)
+			for (size_t i = 0; i < pp.max_processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
 			pp_buffer_stderr(&pp, output_timeout);
@@ -1836,7 +1830,7 @@ void run_processes_parallel(int n,
 	pp_cleanup(&pp);
 }
 
-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
+void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
diff --git a/run-command.h b/run-command.h
index e76a1b6b5b3..6f7604e1146 100644
--- a/run-command.h
+++ b/run-command.h
@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
  * API reads that setting.
  */
 extern int run_processes_parallel_ungroup;
-void run_processes_parallel(int n,
+void run_processes_parallel(size_t n,
 			    get_next_task_fn,
 			    start_failure_fn,
 			    task_finished_fn,
 			    void *pp_cb);
-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
+void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
 				task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
 
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v3 04/15] run-command API: make "n" parameter a "size_t"

[Phillip Wood]

On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:
> Make the "n" variable added in c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) a "size_t". As
> we'll see in a subsequent commit we do pass "0" here, but never "jobs
> < 0".
> 
> We could have made it an "unsigned int", but as we're having to change
> this let's not leave another case in the codebase where a size_t and
> "unsigned int" size differ on some platforms. In this case it's likely
> to never matter, but it's easier to not need to worry about it.
> 
> After this and preceding changes:
> 
> 	make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
> 
> Only has one (and new) -Wsigned-compare warning relevant to a
> comparison about our "n" or "{nr,max}_processes": About using our
> "n" (size_t) in the same expression as online_cpus() (int). A
> subsequent commit will adjust & deal with online_cpus() and that
> warning.

I've changed my mind a bit on this patch. Previously I thought it was 
just unnecessary, now I think it is actually going in the wrong 
direction. You have not changed any of the call sites which are all 
still passing an int, so you are introducing a bunch of signed->unsigned 
conversions. Before this patch we could have caught a negative value of 
n with an assertion. After this patch a negative value passed will 
become a large positive value (you have audited the current callers but 
that does not protect us from future mistakes). To be clear I think 
dropping this patch is the right way forward, not changing all the 
callers. Using an int for the number of processes seems perfectly 
reasonable and matches what all the callers are using.

Best Wishes

Phillip

> The only users of the "n" parameter are:
> 
>   * builtin/fetch.c: defaults to 1, reads from the "fetch.parallel"
>     config. As seen in the code that parses the config added in
>     d54dea77dba (fetch: let --jobs=<n> parallelize --multiple, too,
>     2019-10-05) will die if the git_config_int() return value is < 0.
> 
>     It will however pass us n = 0, as we'll see in a subsequent commit.
> 
>   * submodule.c: defaults to 1, reads from "submodule.fetchJobs"
>     config. Read via code originally added in a028a1930c6 (fetching
>     submodules: respect `submodule.fetchJobs` config option, 2016-02-29).
> 
>     It now piggy-backs on the the submodule.fetchJobs code and
>     validation added in f20e7c1ea24 (submodule: remove
>     submodule.fetchjobs from submodule-config parsing, 2017-08-02).
> 
>     Like builtin/fetch.c it will die if the git_config_int() return
>     value is < 0, but like builtin/fetch.c it will pass us n = 0.
> 
>   * builtin/submodule--helper.c: defaults to 1. Read via code
>     originally added in 2335b870fa7 (submodule update: expose parallelism
>     to the user, 2016-02-29).
> 
>     Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from
>     submodule-config parsing, 2017-08-02) it shares a config parser and
>     semantics with the submodule.c caller.
> 
>   * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'
>     subcommand, 2021-12-22).
> 
>   * t/helper/test-run-command.c: can be -1 after parsing the arguments,
>     but will then be overridden to online_cpus() before passing it to
>     this API. See be5d88e1128 (test-tool run-command: learn to run (parts
>     of) the testsuite, 2019-10-04).
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 42 ++++++++++++++++++------------------------
>   run-command.h |  4 ++--
>   2 files changed, 20 insertions(+), 26 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index 642e6b6e057..14a6e38e804 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;
>   struct parallel_processes {
>   	void *data;
>   
> -	int max_processes;
> -	int nr_processes;
> +	size_t max_processes;
> +	size_t nr_processes;
>   
>   	get_next_task_fn get_next_task;
>   	start_failure_fn start_failure;
> @@ -1522,7 +1522,7 @@ struct parallel_processes {
>   	unsigned shutdown : 1;
>   	unsigned ungroup : 1;
>   
> -	int output_owner;
> +	size_t output_owner;
>   	struct strbuf buffered_output; /* of finished children */
>   };
>   
> @@ -1543,9 +1543,7 @@ static int default_task_finished(int result,
>   
>   static void kill_children(struct parallel_processes *pp, int signo)
>   {
> -	int i, n = pp->max_processes;
> -
> -	for (i = 0; i < n; i++)
> +	for (size_t i = 0; i < pp->max_processes; i++)
>   		if (pp->children[i].state == GIT_CP_WORKING)
>   			kill(pp->children[i].process.pid, signo);
>   }
> @@ -1560,20 +1558,19 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    int n,
> +		    size_t n,
>   		    get_next_task_fn get_next_task,
>   		    start_failure_fn start_failure,
>   		    task_finished_fn task_finished,
>   		    void *data, int ungroup)
>   {
> -	int i;
> -
>   	if (n < 1)
>   		n = online_cpus();
>   
>   	pp->max_processes = n;
>   
> -	trace_printf("run_processes_parallel: preparing to run up to %d tasks", n);
> +	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
> +		     (uintmax_t)n);
>   
>   	pp->data = data;
>   	if (!get_next_task)
> @@ -1594,7 +1591,7 @@ static void pp_init(struct parallel_processes *pp,
>   		CALLOC_ARRAY(pp->pfd, n);
>   	strbuf_init(&pp->buffered_output, 0);
>   
> -	for (i = 0; i < n; i++) {
> +	for (size_t i = 0; i < n; i++) {
>   		strbuf_init(&pp->children[i].err, 0);
>   		child_process_init(&pp->children[i].process);
>   		if (pp->pfd) {
> @@ -1609,10 +1606,8 @@ static void pp_init(struct parallel_processes *pp,
>   
>   static void pp_cleanup(struct parallel_processes *pp)
>   {
> -	int i;
> -
>   	trace_printf("run_processes_parallel: done");
> -	for (i = 0; i < pp->max_processes; i++) {
> +	for (size_t i = 0; i < pp->max_processes; i++) {
>   		strbuf_release(&pp->children[i].err);
>   		child_process_clear(&pp->children[i].process);
>   	}
> @@ -1639,7 +1634,8 @@ static void pp_cleanup(struct parallel_processes *pp)
>    */
>   static int pp_start_one(struct parallel_processes *pp)
>   {
> -	int i, code;
> +	size_t i;
> +	int code;
>   
>   	for (i = 0; i < pp->max_processes; i++)
>   		if (pp->children[i].state == GIT_CP_FREE)
> @@ -1697,7 +1693,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>   	}
>   
>   	/* Buffer output from all pipes. */
> -	for (i = 0; i < pp->max_processes; i++) {
> +	for (size_t i = 0; i < pp->max_processes; i++) {
>   		if (pp->children[i].state == GIT_CP_WORKING &&
>   		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
>   			int n = strbuf_read_once(&pp->children[i].err,
> @@ -1714,7 +1710,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>   
>   static void pp_output(struct parallel_processes *pp)
>   {
> -	int i = pp->output_owner;
> +	size_t i = pp->output_owner;
>   
>   	if (pp->children[i].state == GIT_CP_WORKING &&
>   	    pp->children[i].err.len) {
> @@ -1725,8 +1721,8 @@ static void pp_output(struct parallel_processes *pp)
>   
>   static int pp_collect_finished(struct parallel_processes *pp)
>   {
> -	int i, code;
> -	int n = pp->max_processes;
> +	int code;
> +	size_t i, n = pp->max_processes;
>   	int result = 0;
>   
>   	while (pp->nr_processes > 0) {
> @@ -1783,7 +1779,7 @@ static int pp_collect_finished(struct parallel_processes *pp)
>   	return result;
>   }
>   
> -void run_processes_parallel(int n,
> +void run_processes_parallel(size_t n,
>   			    get_next_task_fn get_next_task,
>   			    start_failure_fn start_failure,
>   			    task_finished_fn task_finished,
> @@ -1817,9 +1813,7 @@ void run_processes_parallel(int n,
>   		if (!pp.nr_processes)
>   			break;
>   		if (ungroup) {
> -			int i;
> -
> -			for (i = 0; i < pp.max_processes; i++)
> +			for (size_t i = 0; i < pp.max_processes; i++)
>   				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
>   		} else {
>   			pp_buffer_stderr(&pp, output_timeout);
> @@ -1836,7 +1830,7 @@ void run_processes_parallel(int n,
>   	pp_cleanup(&pp);
>   }
>   
> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,
> +void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
>   				start_failure_fn start_failure,
>   				task_finished_fn task_finished, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label)
> diff --git a/run-command.h b/run-command.h
> index e76a1b6b5b3..6f7604e1146 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,
>    * API reads that setting.
>    */
>   extern int run_processes_parallel_ungroup;
> -void run_processes_parallel(int n,
> +void run_processes_parallel(size_t n,
>   			    get_next_task_fn,
>   			    start_failure_fn,
>   			    task_finished_fn,
>   			    void *pp_cb);
> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,
> +void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
>   				task_finished_fn, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label);
>   

[PATCH v3 05/15] run-command API: don't fall back on online_cpus()

[Ævar Arnfjörð Bjarmason]

When a "jobs = 0" is passed let's BUG() out rather than fall back on
online_cpus(). The default behavior was added when this API was
implemented in c553c72eed6 (run-command: add an asynchronous parallel
child processor, 2015-12-15).

Most of our code in-tree that scales up to "online_cpus()" by default
calls that function by itself. Keeping this default behavior just for
the sake of two callers means that we'd need to maintain this one spot
where we're second-guessing the config passed down into pp_init().

The preceding commit has an overview of the API callers that passed
"jobs = 0". There were only two of them (actually three, but they
resolved to these two config parsing codepaths).

The "fetch.parallel" caller already had a test for the
"fetch.parallel=0" case added in 0353c688189 (fetch: do not run a
redundant fetch from submodule, 2022-05-16), but there was no such
test for "submodule.fetchJobs". Let's add one here.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 2 ++
 run-command.c               | 7 +++----
 submodule-config.c          | 2 ++
 t/t5526-fetch-submodules.sh | 8 +++++++-
 4 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 78043fb67ef..82f1da14ec1 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
 		fetch_parallel_config = git_config_int(k, v);
 		if (fetch_parallel_config < 0)
 			die(_("fetch.parallel cannot be negative"));
+		if (!fetch_parallel_config)
+			fetch_parallel_config = online_cpus();
 		return 0;
 	}
 
diff --git a/run-command.c b/run-command.c
index 14a6e38e804..14ea409375a 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1564,8 +1564,8 @@ static void pp_init(struct parallel_processes *pp,
 		    task_finished_fn task_finished,
 		    void *data, int ungroup)
 {
-	if (n < 1)
-		n = online_cpus();
+	if (!n)
+		BUG("you must provide a non-zero number of processes!");
 
 	pp->max_processes = n;
 
@@ -1835,8 +1835,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d",
-				   ((n < 1) ? online_cpus() : n));
+	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
 
 	run_processes_parallel(n, get_next_task, start_failure,
 			       task_finished, pp_cb);
diff --git a/submodule-config.c b/submodule-config.c
index cd7ee236a12..4dc61b3a78a 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)
 	int fetchjobs = git_config_int(var, value);
 	if (fetchjobs < 0)
 		die(_("negative values not allowed for submodule.fetchJobs"));
+	if (!fetchjobs)
+		fetchjobs = online_cpus();
 	return fetchjobs;
 }
 
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index a301b56db89..a5f494dfcfb 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -714,7 +714,13 @@ test_expect_success 'fetching submodules respects parallel settings' '
 		GIT_TRACE=$(pwd)/trace.out git fetch &&
 		grep "8 tasks" trace.out &&
 		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
-		grep "9 tasks" trace.out
+		grep "9 tasks" trace.out &&
+		>trace.out &&
+
+		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
+		grep "preparing to run up to [0-9]* tasks" trace.out &&
+		! grep "up to 0 tasks" trace.out &&
+		>trace.out
 	)
 '
 
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 06/15] run-command.c: use designated init for pp_init(), add "const"

[Ævar Arnfjörð Bjarmason]

Use a designated initializer to initialize those parts of pp_init()
that don't need any conditionals for their initialization, this sets
us on a path to pp_init() itself into mostly a validation and
allocation function.

Since we're doing that we can add "const" to some of the members of
the "struct parallel_processes", which helps to clarify and
self-document this code. E.g. we never alter the "data" pointer we
pass t user callbacks, nor (after the preceding change to stop
invoking online_cpus()) do we change "max_processes", the same goes
for the "ungroup" option.

We can also do away with a call to strbuf_init() in favor of macro
initialization, and to rely on other fields being NULL'd or zero'd.

Making members of a struct "const" rather that the pointer to the
struct itself is usually painful, as e.g. it precludes us from
incrementally setting up the structure. In this case we only set it up
with the assignment in run_process_parallel() and pp_init(), and don't
pass the struct pointer around as "const", so making individual
members "const" is worth the potential hassle for extra safety.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 34 ++++++++++++++--------------------
 1 file changed, 14 insertions(+), 20 deletions(-)

diff --git a/run-command.c b/run-command.c
index 14ea409375a..b69deb1cc53 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1498,9 +1498,9 @@ enum child_state {
 
 int run_processes_parallel_ungroup;
 struct parallel_processes {
-	void *data;
+	void *const data;
 
-	size_t max_processes;
+	const size_t max_processes;
 	size_t nr_processes;
 
 	get_next_task_fn get_next_task;
@@ -1520,7 +1520,7 @@ struct parallel_processes {
 	struct pollfd *pfd;
 
 	unsigned shutdown : 1;
-	unsigned ungroup : 1;
+	const unsigned ungroup : 1;
 
 	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
@@ -1558,21 +1558,18 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    size_t n,
 		    get_next_task_fn get_next_task,
 		    start_failure_fn start_failure,
-		    task_finished_fn task_finished,
-		    void *data, int ungroup)
+		    task_finished_fn task_finished)
 {
+	const size_t n = pp->max_processes;
+
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
 
-	pp->max_processes = n;
-
 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
 		     (uintmax_t)n);
 
-	pp->data = data;
 	if (!get_next_task)
 		BUG("you need to specify a get_next_task function");
 	pp->get_next_task = get_next_task;
@@ -1580,16 +1577,9 @@ static void pp_init(struct parallel_processes *pp,
 	pp->start_failure = start_failure ? start_failure : default_start_failure;
 	pp->task_finished = task_finished ? task_finished : default_task_finished;
 
-	pp->nr_processes = 0;
-	pp->output_owner = 0;
-	pp->shutdown = 0;
-	pp->ungroup = ungroup;
 	CALLOC_ARRAY(pp->children, n);
-	if (pp->ungroup)
-		pp->pfd = NULL;
-	else
+	if (!pp->ungroup)
 		CALLOC_ARRAY(pp->pfd, n);
-	strbuf_init(&pp->buffered_output, 0);
 
 	for (size_t i = 0; i < n; i++) {
 		strbuf_init(&pp->children[i].err, 0);
@@ -1789,13 +1779,17 @@ void run_processes_parallel(size_t n,
 	int output_timeout = 100;
 	int spawn_cap = 4;
 	int ungroup = run_processes_parallel_ungroup;
-	struct parallel_processes pp;
+	struct parallel_processes pp = {
+		.max_processes = n,
+		.data = pp_cb,
+		.buffered_output = STRBUF_INIT,
+		.ungroup = ungroup,
+	};
 
 	/* unset for the next API user */
 	run_processes_parallel_ungroup = 0;
 
-	pp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,
-		ungroup);
+	pp_init(&pp, get_next_task, start_failure, task_finished);
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 07/15] run-command API: have run_process_parallel() take an "opts" struct

[Ævar Arnfjörð Bjarmason]

As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
run_process_parallel(), 2022-06-07) which added the "ungroup" passing
it to "run_process_parallel()" via the global
"run_processes_parallel_ungroup" variable was a compromise to get the
smallest possible regression fix for "maint" at the time.

This follow-up to that is a start at passing that parameter and others
via a new "struct run_process_parallel_opts", as the earlier
version[1] of what became fd3aaf53f71 did.

Since we need to change all of the occurrences of "n" to
"opt->SOMETHING" let's take the opportunity and rename the terse "n"
to "processes". We could also have picked "max_processes", "jobs",
"threads" etc., but as the API is named "run_processes_parallel()"
let's go with "processes".

Since the new "run_processes_parallel()" function is able to take an
optional "tr2_category" and "tr2_label" via the struct we can at this
point migrate all of the users of "run_processes_parallel_tr2()" over
to it.

But let's not migrate all the API users yet, only the two users that
passed the "ungroup" parameter via the
"run_processes_parallel_ungroup" global

1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 hook.c                      | 23 +++++++-----
 run-command.c               | 54 ++++++++++++++++------------
 run-command.h               | 72 ++++++++++++++++++++++++++++---------
 t/helper/test-run-command.c | 31 ++++++++++------
 4 files changed, 121 insertions(+), 59 deletions(-)

diff --git a/hook.c b/hook.c
index a493939a4fc..a4fa1031f28 100644
--- a/hook.c
+++ b/hook.c
@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		.options = options,
 	};
 	const char *const hook_path = find_hook(hook_name);
-	int jobs = 1;
 	int ret = 0;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "hook",
+		.tr2_label = hook_name,
+
+		.processes = 1,
+		.ungroup = 1,
+
+		.get_next_task = pick_next_hook,
+		.start_failure = notify_start_failure,
+		.task_finished = notify_hook_finished,
+
+		.data = &cb_data,
+	};
 
 	if (!options)
 		BUG("a struct run_hooks_opt must be provided to run_hooks");
@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
 		cb_data.hook_path = abs_path.buf;
 	}
 
-	run_processes_parallel_ungroup = 1;
-	run_processes_parallel_tr2(jobs,
-				   pick_next_hook,
-				   notify_start_failure,
-				   notify_hook_finished,
-				   &cb_data,
-				   "hook",
-				   hook_name);
+	run_processes_parallel(&opts);
 	ret = cb_data.rc;
 cleanup:
 	strbuf_release(&abs_path);
diff --git a/run-command.c b/run-command.c
index b69deb1cc53..2858ec7bef5 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1496,7 +1496,6 @@ enum child_state {
 	GIT_CP_WAIT_CLEANUP,
 };
 
-int run_processes_parallel_ungroup;
 struct parallel_processes {
 	void *const data;
 
@@ -1558,11 +1557,12 @@ static void handle_children_on_signal(int signo)
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    get_next_task_fn get_next_task,
-		    start_failure_fn start_failure,
-		    task_finished_fn task_finished)
+		    const struct run_process_parallel_opts *opts)
 {
-	const size_t n = pp->max_processes;
+	const size_t n = opts->processes;
+	get_next_task_fn get_next_task = opts->get_next_task;
+	start_failure_fn start_failure = opts->start_failure;
+	task_finished_fn task_finished = opts->task_finished;
 
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
@@ -1769,27 +1769,27 @@ static int pp_collect_finished(struct parallel_processes *pp)
 	return result;
 }
 
-void run_processes_parallel(size_t n,
-			    get_next_task_fn get_next_task,
-			    start_failure_fn start_failure,
-			    task_finished_fn task_finished,
-			    void *pp_cb)
+void run_processes_parallel(const struct run_process_parallel_opts *opts)
 {
 	int i, code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
-	int ungroup = run_processes_parallel_ungroup;
 	struct parallel_processes pp = {
-		.max_processes = n,
-		.data = pp_cb,
+		.max_processes = opts->processes,
+		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
-		.ungroup = ungroup,
+		.ungroup = opts->ungroup,
 	};
+	/* options */
+	const char *tr2_category = opts->tr2_category;
+	const char *tr2_label = opts->tr2_label;
+	const int do_trace2 = tr2_category && tr2_label;
 
-	/* unset for the next API user */
-	run_processes_parallel_ungroup = 0;
+	if (do_trace2)
+		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
+					   "max:%d", opts->processes);
 
-	pp_init(&pp, get_next_task, start_failure, task_finished);
+	pp_init(&pp, opts);
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1806,7 +1806,7 @@ void run_processes_parallel(size_t n,
 		}
 		if (!pp.nr_processes)
 			break;
-		if (ungroup) {
+		if (opts->ungroup) {
 			for (size_t i = 0; i < pp.max_processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
@@ -1822,19 +1822,27 @@ void run_processes_parallel(size_t n,
 	}
 
 	pp_cleanup(&pp);
+
+	if (do_trace2)
+		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
+void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
 				start_failure_fn start_failure,
 				task_finished_fn task_finished, void *pp_cb,
 				const char *tr2_category, const char *tr2_label)
 {
-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = tr2_category,
+		.tr2_label = tr2_label,
+		.processes = processes,
 
-	run_processes_parallel(n, get_next_task, start_failure,
-			       task_finished, pp_cb);
+		.get_next_task = get_next_task,
+		.start_failure = start_failure,
+		.task_finished = task_finished,
+	};
 
-	trace2_region_leave(tr2_category, tr2_label, NULL);
+	run_processes_parallel(&opts);
 }
 
 int run_auto_maintenance(int quiet)
diff --git a/run-command.h b/run-command.h
index 6f7604e1146..aabdaf684db 100644
--- a/run-command.h
+++ b/run-command.h
@@ -459,17 +459,64 @@ typedef int (*task_finished_fn)(int result,
 				void *pp_task_cb);
 
 /**
- * Runs up to n processes at the same time. Whenever a process can be
- * started, the callback get_next_task_fn is called to obtain the data
+ * Option used by run_processes_parallel(), { 0 }-initialized means no
+ * options.
+ */
+struct run_process_parallel_opts
+{
+	/**
+	 * tr2_category & tr2_label: sets the trace2 category and label for
+	 * logging. These must either be unset, or both of them must be set.
+	 */
+	const char *tr2_category;
+	const char *tr2_label;
+
+	/**
+	 * processes: see 'processes' in run_processes_parallel() below.
+	 */
+	size_t processes;
+
+	/**
+	 * ungroup: see 'ungroup' in run_processes_parallel() below.
+	 */
+	unsigned int ungroup:1;
+
+	/**
+	 * get_next_task: See get_next_task_fn() above. This must be
+	 * specified.
+	 */
+	get_next_task_fn get_next_task;
+
+	/**
+	 * start_failure: See start_failure_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	start_failure_fn start_failure;
+
+	/**
+	 * task_finished: See task_finished_fn() above. This can be
+	 * NULL to omit any special handling.
+	 */
+	task_finished_fn task_finished;
+
+	/**
+	 * data: user data, will be passed as "pp_cb" to the callback
+	 * parameters.
+	 */
+	void *data;
+};
+
+/**
+ * Options are passed via the "struct run_process_parallel_opts" above.
+ *
+ * Runs N 'processes' at the same time. Whenever a process can be
+ * started, the callback opts.get_next_task is called to obtain the data
  * required to start another child process.
  *
  * The children started via this function run in parallel. Their output
  * (both stdout and stderr) is routed to stderr in a manner that output
  * from different tasks does not interleave (but see "ungroup" below).
  *
- * start_failure_fn and task_finished_fn can be NULL to omit any
- * special handling.
- *
  * If the "ungroup" option isn't specified, the API will set the
  * "stdout_to_stderr" parameter in "struct child_process" and provide
  * the callbacks with a "struct strbuf *out" parameter to write output
@@ -479,19 +526,10 @@ typedef int (*task_finished_fn)(int result,
  * NULL "struct strbuf *out" parameter, and are responsible for
  * emitting their own output, including dealing with any race
  * conditions due to writing in parallel to stdout and stderr.
- * The "ungroup" option can be enabled by setting the global
- * "run_processes_parallel_ungroup" to "1" before invoking
- * run_processes_parallel(), it will be set back to "0" as soon as the
- * API reads that setting.
  */
-extern int run_processes_parallel_ungroup;
-void run_processes_parallel(size_t n,
-			    get_next_task_fn,
-			    start_failure_fn,
-			    task_finished_fn,
-			    void *pp_cb);
-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
-				task_finished_fn, void *pp_cb,
+void run_processes_parallel(const struct run_process_parallel_opts *opts);
+void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
+				start_failure_fn, task_finished_fn, void *pp_cb,
 				const char *tr2_category, const char *tr2_label);
 
 /**
diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index ee509aefa2f..3ecb830f4a8 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {
 static int testsuite(int argc, const char **argv)
 {
 	struct testsuite suite = TESTSUITE_INIT;
-	int max_jobs = 1, i, ret;
+	int max_jobs = 1, i, ret = 0;
 	DIR *dir;
 	struct dirent *d;
 	struct option options[] = {
@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)
 			 "write JUnit-style XML files"),
 		OPT_END()
 	};
+	struct run_process_parallel_opts opts = {
+		.get_next_task = next_test,
+		.start_failure = test_failed,
+		.task_finished = test_finished,
+		.data = &suite,
+	};
 
 	argc = parse_options(argc, argv, NULL, options,
 			testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)
 	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
 		(uintmax_t)suite.tests.nr, max_jobs);
 
-	run_processes_parallel(max_jobs, next_test, test_failed,
-			       test_finished, &suite);
+	opts.processes = max_jobs;
+	run_processes_parallel(&opts);
 
 	if (suite.failed.nr > 0) {
 		ret = 1;
@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)
 	string_list_clear(&suite.tests, 0);
 	string_list_clear(&suite.failed, 0);
 
-	return !!ret;
+	return ret;
 }
 
 static uint64_t my_random_next = 1234;
@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
 	int ret;
+	struct run_process_parallel_opts opts = {
+		.data = &proc,
+	};
 
 	if (argc > 1 && !strcmp(argv[1], "testsuite"))
 		return testsuite(argc - 1, argv + 1);
@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)
 	if (!strcmp(argv[1], "--ungroup")) {
 		argv += 1;
 		argc -= 1;
-		run_processes_parallel_ungroup = 1;
+		opts.ungroup = 1;
 	}
 
 	jobs = atoi(argv[2]);
@@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)
 	strvec_pushv(&proc.args, (const char **)argv + 3);
 
 	if (!strcmp(argv[1], "run-command-parallel")) {
-		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
+		opts.get_next_task = parallel_next;
 	} else if (!strcmp(argv[1], "run-command-abort")) {
-		run_processes_parallel(jobs, parallel_next, NULL,
-				       task_finished, &proc);
+		opts.get_next_task = parallel_next;
+		opts.task_finished = task_finished;
 	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
-		run_processes_parallel(jobs, no_job, NULL, task_finished,
-				       &proc);
+		opts.get_next_task = no_job;
+		opts.task_finished = task_finished;
 	} else {
 		ret = 1;
 		fprintf(stderr, "check usage\n");
 		goto cleanup;
 	}
+	opts.processes = jobs;
+	run_processes_parallel(&opts);
 	ret = 0;
 cleanup:
 	child_process_clear(&proc);
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v3 07/15] run-command API: have run_process_parallel() take an "opts" struct

[Phillip Wood]

On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:
> As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
> run_process_parallel(), 2022-06-07) which added the "ungroup" passing
> it to "run_process_parallel()" via the global
> "run_processes_parallel_ungroup" variable was a compromise to get the
> smallest possible regression fix for "maint" at the time.
> 
> This follow-up to that is a start at passing that parameter and others
> via a new "struct run_process_parallel_opts", as the earlier
> version[1] of what became fd3aaf53f71 did.
> 
> Since we need to change all of the occurrences of "n" to
> "opt->SOMETHING" let's take the opportunity and rename the terse "n"
> to "processes". We could also have picked "max_processes", "jobs",
> "threads" etc., but as the API is named "run_processes_parallel()"
> let's go with "processes".
> 
> Since the new "run_processes_parallel()" function is able to take an
> optional "tr2_category" and "tr2_label" via the struct we can at this
> point migrate all of the users of "run_processes_parallel_tr2()" over
> to it.
> 
> But let's not migrate all the API users yet, only the two users that
> passed the "ungroup" parameter via the
> "run_processes_parallel_ungroup" global

This all sounds sensible, thanks for squashing the patches together.

Best Wishes

Phillip

> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   hook.c                      | 23 +++++++-----
>   run-command.c               | 54 ++++++++++++++++------------
>   run-command.h               | 72 ++++++++++++++++++++++++++++---------
>   t/helper/test-run-command.c | 31 ++++++++++------
>   4 files changed, 121 insertions(+), 59 deletions(-)
> 
> diff --git a/hook.c b/hook.c
> index a493939a4fc..a4fa1031f28 100644
> --- a/hook.c
> +++ b/hook.c
> @@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
>   		.options = options,
>   	};
>   	const char *const hook_path = find_hook(hook_name);
> -	int jobs = 1;
>   	int ret = 0;
> +	const struct run_process_parallel_opts opts = {
> +		.tr2_category = "hook",
> +		.tr2_label = hook_name,
> +
> +		.processes = 1,
> +		.ungroup = 1,
> +
> +		.get_next_task = pick_next_hook,
> +		.start_failure = notify_start_failure,
> +		.task_finished = notify_hook_finished,
> +
> +		.data = &cb_data,
> +	};
>   
>   	if (!options)
>   		BUG("a struct run_hooks_opt must be provided to run_hooks");
> @@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)
>   		cb_data.hook_path = abs_path.buf;
>   	}
>   
> -	run_processes_parallel_ungroup = 1;
> -	run_processes_parallel_tr2(jobs,
> -				   pick_next_hook,
> -				   notify_start_failure,
> -				   notify_hook_finished,
> -				   &cb_data,
> -				   "hook",
> -				   hook_name);
> +	run_processes_parallel(&opts);
>   	ret = cb_data.rc;
>   cleanup:
>   	strbuf_release(&abs_path);
> diff --git a/run-command.c b/run-command.c
> index b69deb1cc53..2858ec7bef5 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1496,7 +1496,6 @@ enum child_state {
>   	GIT_CP_WAIT_CLEANUP,
>   };
>   
> -int run_processes_parallel_ungroup;
>   struct parallel_processes {
>   	void *const data;
>   
> @@ -1558,11 +1557,12 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    get_next_task_fn get_next_task,
> -		    start_failure_fn start_failure,
> -		    task_finished_fn task_finished)
> +		    const struct run_process_parallel_opts *opts)
>   {
> -	const size_t n = pp->max_processes;
> +	const size_t n = opts->processes;
> +	get_next_task_fn get_next_task = opts->get_next_task;
> +	start_failure_fn start_failure = opts->start_failure;
> +	task_finished_fn task_finished = opts->task_finished;
>   
>   	if (!n)
>   		BUG("you must provide a non-zero number of processes!");
> @@ -1769,27 +1769,27 @@ static int pp_collect_finished(struct parallel_processes *pp)
>   	return result;
>   }
>   
> -void run_processes_parallel(size_t n,
> -			    get_next_task_fn get_next_task,
> -			    start_failure_fn start_failure,
> -			    task_finished_fn task_finished,
> -			    void *pp_cb)
> +void run_processes_parallel(const struct run_process_parallel_opts *opts)
>   {
>   	int i, code;
>   	int output_timeout = 100;
>   	int spawn_cap = 4;
> -	int ungroup = run_processes_parallel_ungroup;
>   	struct parallel_processes pp = {
> -		.max_processes = n,
> -		.data = pp_cb,
> +		.max_processes = opts->processes,
> +		.data = opts->data,
>   		.buffered_output = STRBUF_INIT,
> -		.ungroup = ungroup,
> +		.ungroup = opts->ungroup,
>   	};
> +	/* options */
> +	const char *tr2_category = opts->tr2_category;
> +	const char *tr2_label = opts->tr2_label;
> +	const int do_trace2 = tr2_category && tr2_label;
>   
> -	/* unset for the next API user */
> -	run_processes_parallel_ungroup = 0;
> +	if (do_trace2)
> +		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
> +					   "max:%d", opts->processes);
>   
> -	pp_init(&pp, get_next_task, start_failure, task_finished);
> +	pp_init(&pp, opts);
>   	while (1) {
>   		for (i = 0;
>   		    i < spawn_cap && !pp.shutdown &&
> @@ -1806,7 +1806,7 @@ void run_processes_parallel(size_t n,
>   		}
>   		if (!pp.nr_processes)
>   			break;
> -		if (ungroup) {
> +		if (opts->ungroup) {
>   			for (size_t i = 0; i < pp.max_processes; i++)
>   				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
>   		} else {
> @@ -1822,19 +1822,27 @@ void run_processes_parallel(size_t n,
>   	}
>   
>   	pp_cleanup(&pp);
> +
> +	if (do_trace2)
> +		trace2_region_leave(tr2_category, tr2_label, NULL);
>   }
>   
> -void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
> +void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
>   				start_failure_fn start_failure,
>   				task_finished_fn task_finished, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label)
>   {
> -	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
> +	const struct run_process_parallel_opts opts = {
> +		.tr2_category = tr2_category,
> +		.tr2_label = tr2_label,
> +		.processes = processes,
>   
> -	run_processes_parallel(n, get_next_task, start_failure,
> -			       task_finished, pp_cb);
> +		.get_next_task = get_next_task,
> +		.start_failure = start_failure,
> +		.task_finished = task_finished,
> +	};
>   
> -	trace2_region_leave(tr2_category, tr2_label, NULL);
> +	run_processes_parallel(&opts);
>   }
>   
>   int run_auto_maintenance(int quiet)
> diff --git a/run-command.h b/run-command.h
> index 6f7604e1146..aabdaf684db 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -459,17 +459,64 @@ typedef int (*task_finished_fn)(int result,
>   				void *pp_task_cb);
>   
>   /**
> - * Runs up to n processes at the same time. Whenever a process can be
> - * started, the callback get_next_task_fn is called to obtain the data
> + * Option used by run_processes_parallel(), { 0 }-initialized means no
> + * options.
> + */
> +struct run_process_parallel_opts
> +{
> +	/**
> +	 * tr2_category & tr2_label: sets the trace2 category and label for
> +	 * logging. These must either be unset, or both of them must be set.
> +	 */
> +	const char *tr2_category;
> +	const char *tr2_label;
> +
> +	/**
> +	 * processes: see 'processes' in run_processes_parallel() below.
> +	 */
> +	size_t processes;
> +
> +	/**
> +	 * ungroup: see 'ungroup' in run_processes_parallel() below.
> +	 */
> +	unsigned int ungroup:1;
> +
> +	/**
> +	 * get_next_task: See get_next_task_fn() above. This must be
> +	 * specified.
> +	 */
> +	get_next_task_fn get_next_task;
> +
> +	/**
> +	 * start_failure: See start_failure_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	start_failure_fn start_failure;
> +
> +	/**
> +	 * task_finished: See task_finished_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	task_finished_fn task_finished;
> +
> +	/**
> +	 * data: user data, will be passed as "pp_cb" to the callback
> +	 * parameters.
> +	 */
> +	void *data;
> +};
> +
> +/**
> + * Options are passed via the "struct run_process_parallel_opts" above.
> + *
> + * Runs N 'processes' at the same time. Whenever a process can be
> + * started, the callback opts.get_next_task is called to obtain the data
>    * required to start another child process.
>    *
>    * The children started via this function run in parallel. Their output
>    * (both stdout and stderr) is routed to stderr in a manner that output
>    * from different tasks does not interleave (but see "ungroup" below).
>    *
> - * start_failure_fn and task_finished_fn can be NULL to omit any
> - * special handling.
> - *
>    * If the "ungroup" option isn't specified, the API will set the
>    * "stdout_to_stderr" parameter in "struct child_process" and provide
>    * the callbacks with a "struct strbuf *out" parameter to write output
> @@ -479,19 +526,10 @@ typedef int (*task_finished_fn)(int result,
>    * NULL "struct strbuf *out" parameter, and are responsible for
>    * emitting their own output, including dealing with any race
>    * conditions due to writing in parallel to stdout and stderr.
> - * The "ungroup" option can be enabled by setting the global
> - * "run_processes_parallel_ungroup" to "1" before invoking
> - * run_processes_parallel(), it will be set back to "0" as soon as the
> - * API reads that setting.
>    */
> -extern int run_processes_parallel_ungroup;
> -void run_processes_parallel(size_t n,
> -			    get_next_task_fn,
> -			    start_failure_fn,
> -			    task_finished_fn,
> -			    void *pp_cb);
> -void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
> -				task_finished_fn, void *pp_cb,
> +void run_processes_parallel(const struct run_process_parallel_opts *opts);
> +void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
> +				start_failure_fn, task_finished_fn, void *pp_cb,
>   				const char *tr2_category, const char *tr2_label);
>   
>   /**
> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
> index ee509aefa2f..3ecb830f4a8 100644
> --- a/t/helper/test-run-command.c
> +++ b/t/helper/test-run-command.c
> @@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {
>   static int testsuite(int argc, const char **argv)
>   {
>   	struct testsuite suite = TESTSUITE_INIT;
> -	int max_jobs = 1, i, ret;
> +	int max_jobs = 1, i, ret = 0;
>   	DIR *dir;
>   	struct dirent *d;
>   	struct option options[] = {
> @@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)
>   			 "write JUnit-style XML files"),
>   		OPT_END()
>   	};
> +	struct run_process_parallel_opts opts = {
> +		.get_next_task = next_test,
> +		.start_failure = test_failed,
> +		.task_finished = test_finished,
> +		.data = &suite,
> +	};
>   
>   	argc = parse_options(argc, argv, NULL, options,
>   			testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
> @@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)
>   	fprintf(stderr, "Running %"PRIuMAX" tests (%d at a time)\n",
>   		(uintmax_t)suite.tests.nr, max_jobs);
>   
> -	run_processes_parallel(max_jobs, next_test, test_failed,
> -			       test_finished, &suite);
> +	opts.processes = max_jobs;
> +	run_processes_parallel(&opts);
>   
>   	if (suite.failed.nr > 0) {
>   		ret = 1;
> @@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)
>   	string_list_clear(&suite.tests, 0);
>   	string_list_clear(&suite.failed, 0);
>   
> -	return !!ret;
> +	return ret;
>   }
>   
>   static uint64_t my_random_next = 1234;
> @@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)
>   	struct child_process proc = CHILD_PROCESS_INIT;
>   	int jobs;
>   	int ret;
> +	struct run_process_parallel_opts opts = {
> +		.data = &proc,
> +	};
>   
>   	if (argc > 1 && !strcmp(argv[1], "testsuite"))
>   		return testsuite(argc - 1, argv + 1);
> @@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)
>   	if (!strcmp(argv[1], "--ungroup")) {
>   		argv += 1;
>   		argc -= 1;
> -		run_processes_parallel_ungroup = 1;
> +		opts.ungroup = 1;
>   	}
>   
>   	jobs = atoi(argv[2]);
> @@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)
>   	strvec_pushv(&proc.args, (const char **)argv + 3);
>   
>   	if (!strcmp(argv[1], "run-command-parallel")) {
> -		run_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);
> +		opts.get_next_task = parallel_next;
>   	} else if (!strcmp(argv[1], "run-command-abort")) {
> -		run_processes_parallel(jobs, parallel_next, NULL,
> -				       task_finished, &proc);
> +		opts.get_next_task = parallel_next;
> +		opts.task_finished = task_finished;
>   	} else if (!strcmp(argv[1], "run-command-no-jobs")) {
> -		run_processes_parallel(jobs, no_job, NULL, task_finished,
> -				       &proc);
> +		opts.get_next_task = no_job;
> +		opts.task_finished = task_finished;
>   	} else {
>   		ret = 1;
>   		fprintf(stderr, "check usage\n");
>   		goto cleanup;
>   	}
> +	opts.processes = jobs;
> +	run_processes_parallel(&opts);
>   	ret = 0;
>   cleanup:
>   	child_process_clear(&proc);

[PATCH v3 08/15] run-command API: move *_tr2() users to "run_processes_parallel()"

[Ævar Arnfjörð Bjarmason]

Have the users of the "run_processes_parallel_tr2()" function use
"run_processes_parallel()" instead. In preceding commits the latter
was refactored to take a "struct run_process_parallel_opts" argument,
since the only reason for "run_processes_parallel_tr2()" to exist was
to take arguments that are now a part of that struct we can do away
with it.

See ee4512ed481 (trace2: create new combined trace facility,
2019-02-22) for the addition of the "*_tr2()" variant of the function,
it was used by every caller except "t/helper/test-run-command.c"..

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/fetch.c             | 18 ++++++++++++------
 builtin/submodule--helper.c | 16 ++++++++++++----
 run-command.c               | 18 ------------------
 run-command.h               |  3 ---
 submodule.c                 | 18 ++++++++++++------
 5 files changed, 36 insertions(+), 37 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 82f1da14ec1..b06e454cbdd 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	if (max_children != 1 && list->nr != 1) {
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		const struct run_process_parallel_opts opts = {
+			.tr2_category = "fetch",
+			.tr2_label = "parallel/fetch",
+
+			.processes = max_children,
+
+			.get_next_task = &fetch_next_remote,
+			.start_failure = &fetch_failed_to_start,
+			.task_finished = &fetch_finished,
+			.data = &state,
+		};
 
 		strvec_push(&argv, "--end-of-options");
-		run_processes_parallel_tr2(max_children,
-					   &fetch_next_remote,
-					   &fetch_failed_to_start,
-					   &fetch_finished,
-					   &state,
-					   "fetch", "parallel/fetch");
 
+		run_processes_parallel(&opts);
 		result = state.result;
 	} else
 		for (i = 0; i < list->nr; i++) {
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 0b4acb442b2..df526525c1e 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)
 {
 	int i, ret = 0;
 	struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/update",
+
+		.processes = update_data->max_jobs,
+
+		.get_next_task = update_clone_get_next_task,
+		.start_failure = update_clone_start_failure,
+		.task_finished = update_clone_task_finished,
+		.data = &suc,
+	};
 
 	suc.update_data = update_data;
-	run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,
-				   update_clone_start_failure,
-				   update_clone_task_finished, &suc, "submodule",
-				   "parallel/update");
+	run_processes_parallel(&opts);
 
 	/*
 	 * We saved the output and put it out all at once now.
diff --git a/run-command.c b/run-command.c
index 2858ec7bef5..646ea22e399 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1827,24 +1827,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
 }
 
-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
-				start_failure_fn start_failure,
-				task_finished_fn task_finished, void *pp_cb,
-				const char *tr2_category, const char *tr2_label)
-{
-	const struct run_process_parallel_opts opts = {
-		.tr2_category = tr2_category,
-		.tr2_label = tr2_label,
-		.processes = processes,
-
-		.get_next_task = get_next_task,
-		.start_failure = start_failure,
-		.task_finished = task_finished,
-	};
-
-	run_processes_parallel(&opts);
-}
-
 int run_auto_maintenance(int quiet)
 {
 	int enabled;
diff --git a/run-command.h b/run-command.h
index aabdaf684db..e3e1ea01ad9 100644
--- a/run-command.h
+++ b/run-command.h
@@ -528,9 +528,6 @@ struct run_process_parallel_opts
  * conditions due to writing in parallel to stdout and stderr.
  */
 void run_processes_parallel(const struct run_process_parallel_opts *opts);
-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
-				start_failure_fn, task_finished_fn, void *pp_cb,
-				const char *tr2_category, const char *tr2_label);
 
 /**
  * Convenience function which prepares env for a command to be run in a
diff --git a/submodule.c b/submodule.c
index bf7a2c79183..f7c71f1f4b1 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,
 {
 	int i;
 	struct submodule_parallel_fetch spf = SPF_INIT;
+	const struct run_process_parallel_opts opts = {
+		.tr2_category = "submodule",
+		.tr2_label = "parallel/fetch",
+
+		.processes = max_parallel_jobs,
+
+		.get_next_task = get_next_submodule,
+		.start_failure = fetch_start_failure,
+		.task_finished = fetch_finish,
+		.data = &spf,
+	};
 
 	spf.r = r;
 	spf.command_line_option = command_line_option;
@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,
 
 	calculate_changed_submodule_paths(r, &spf.changed_submodule_names);
 	string_list_sort(&spf.changed_submodule_names);
-	run_processes_parallel_tr2(max_parallel_jobs,
-				   get_next_submodule,
-				   fetch_start_failure,
-				   fetch_finish,
-				   &spf,
-				   "submodule", "parallel/fetch");
+	run_processes_parallel(&opts);
 
 	if (spf.submodules_with_errors.len > 0)
 		fprintf(stderr, _("Errors during submodule fetch:\n%s"),
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 09/15] run-command.c: make "struct parallel_processes" const if possible

[Ævar Arnfjörð Bjarmason]

Add a "const" to two "struct parallel_processes" parameters where
we're not modifying anything in "pp". For kill_children() we'll call
it from both the signal handler, and from run_processes_parallel()
itself. Adding a "const" there makes it clear that we don't need to
modify any state when killing our children.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/run-command.c b/run-command.c
index 646ea22e399..a72c4295adc 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1540,7 +1540,7 @@ static int default_task_finished(int result,
 	return 0;
 }
 
-static void kill_children(struct parallel_processes *pp, int signo)
+static void kill_children(const struct parallel_processes *pp, int signo)
 {
 	for (size_t i = 0; i < pp->max_processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
@@ -1698,7 +1698,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
 	}
 }
 
-static void pp_output(struct parallel_processes *pp)
+static void pp_output(const struct parallel_processes *pp)
 {
 	size_t i = pp->output_owner;
 
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 10/15] run-command.c: don't copy *_fn to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

The only remaining reason for copying the callbacks in the "struct
run_process_parallel_opts" over to the "struct parallel_processes" was
to avoid two if/else statements in case the "start_failure" and
"task_finished" callbacks were NULL.

Let's handle those cases in pp_start_one() and pp_collect_finished()
instead, and avoid the default_* stub functions, and the need to copy
this data around.

Organizing the code like this made more sense before the "struct
run_parallel_parallel_opts" existed, as we'd have needed to pass each
of these as a separate parameter.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 67 +++++++++++++++++++--------------------------------
 1 file changed, 25 insertions(+), 42 deletions(-)

diff --git a/run-command.c b/run-command.c
index a72c4295adc..e10b1b97390 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1502,10 +1502,6 @@ struct parallel_processes {
 	const size_t max_processes;
 	size_t nr_processes;
 
-	get_next_task_fn get_next_task;
-	start_failure_fn start_failure;
-	task_finished_fn task_finished;
-
 	struct {
 		enum child_state state;
 		struct child_process process;
@@ -1525,21 +1521,6 @@ struct parallel_processes {
 	struct strbuf buffered_output; /* of finished children */
 };
 
-static int default_start_failure(struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
-static int default_task_finished(int result,
-				 struct strbuf *out,
-				 void *pp_cb,
-				 void *pp_task_cb)
-{
-	return 0;
-}
-
 static void kill_children(const struct parallel_processes *pp, int signo)
 {
 	for (size_t i = 0; i < pp->max_processes; i++)
@@ -1560,9 +1541,6 @@ static void pp_init(struct parallel_processes *pp,
 		    const struct run_process_parallel_opts *opts)
 {
 	const size_t n = opts->processes;
-	get_next_task_fn get_next_task = opts->get_next_task;
-	start_failure_fn start_failure = opts->start_failure;
-	task_finished_fn task_finished = opts->task_finished;
 
 	if (!n)
 		BUG("you must provide a non-zero number of processes!");
@@ -1570,12 +1548,8 @@ static void pp_init(struct parallel_processes *pp,
 	trace_printf("run_processes_parallel: preparing to run up to %"PRIuMAX" tasks",
 		     (uintmax_t)n);
 
-	if (!get_next_task)
+	if (!opts->get_next_task)
 		BUG("you need to specify a get_next_task function");
-	pp->get_next_task = get_next_task;
-
-	pp->start_failure = start_failure ? start_failure : default_start_failure;
-	pp->task_finished = task_finished ? task_finished : default_task_finished;
 
 	CALLOC_ARRAY(pp->children, n);
 	if (!pp->ungroup)
@@ -1622,7 +1596,8 @@ static void pp_cleanup(struct parallel_processes *pp)
  * <0 no new job was started, user wishes to shutdown early. Use negative code
  *    to signal the children.
  */
-static int pp_start_one(struct parallel_processes *pp)
+static int pp_start_one(struct parallel_processes *pp,
+			const struct run_process_parallel_opts *opts)
 {
 	size_t i;
 	int code;
@@ -1633,10 +1608,10 @@ static int pp_start_one(struct parallel_processes *pp)
 	if (i == pp->max_processes)
 		BUG("bookkeeping is hard");
 
-	code = pp->get_next_task(&pp->children[i].process,
-				 pp->ungroup ? NULL : &pp->children[i].err,
-				 pp->data,
-				 &pp->children[i].data);
+	code = opts->get_next_task(&pp->children[i].process,
+				   pp->ungroup ? NULL : &pp->children[i].err,
+				   pp->data,
+				   &pp->children[i].data);
 	if (!code) {
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
@@ -1651,10 +1626,14 @@ static int pp_start_one(struct parallel_processes *pp)
 	pp->children[i].process.no_stdin = 1;
 
 	if (start_command(&pp->children[i].process)) {
-		code = pp->start_failure(pp->ungroup ? NULL :
-					 &pp->children[i].err,
-					 pp->data,
-					 pp->children[i].data);
+		if (opts->start_failure)
+			code = opts->start_failure(pp->ungroup ? NULL :
+						   &pp->children[i].err,
+						   pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
+
 		if (!pp->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
@@ -1709,7 +1688,8 @@ static void pp_output(const struct parallel_processes *pp)
 	}
 }
 
-static int pp_collect_finished(struct parallel_processes *pp)
+static int pp_collect_finished(struct parallel_processes *pp,
+			       const struct run_process_parallel_opts *opts)
 {
 	int code;
 	size_t i, n = pp->max_processes;
@@ -1724,9 +1704,12 @@ static int pp_collect_finished(struct parallel_processes *pp)
 
 		code = finish_command(&pp->children[i].process);
 
-		code = pp->task_finished(code, pp->ungroup ? NULL :
-					 &pp->children[i].err, pp->data,
-					 pp->children[i].data);
+		if (opts->task_finished)
+			code = opts->task_finished(code, pp->ungroup ? NULL :
+						   &pp->children[i].err, pp->data,
+						   pp->children[i].data);
+		else
+			code = 0;
 
 		if (code)
 			result = code;
@@ -1795,7 +1778,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		    i < spawn_cap && !pp.shutdown &&
 		    pp.nr_processes < pp.max_processes;
 		    i++) {
-			code = pp_start_one(&pp);
+			code = pp_start_one(&pp, opts);
 			if (!code)
 				continue;
 			if (code < 0) {
@@ -1813,7 +1796,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 			pp_buffer_stderr(&pp, output_timeout);
 			pp_output(&pp);
 		}
-		code = pp_collect_finished(&pp);
+		code = pp_collect_finished(&pp, opts);
 		if (code) {
 			pp.shutdown = 1;
 			if (code < 0)
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 11/15] run-command.c: don't copy "ungroup" to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in the preceding commit, let's not
copy the "ungroup" member of the "struct run_process_parallel_opts"
over to the "struct parallel_processes". Now that we're passing the
"opts" down there's no reason to do so.

This makes the code easier to follow, as we have a "const" attribute
on the "struct run_process_parallel_opts", but not "struct
parallel_processes". We do not alter the "ungroup" argument, so
storing it in the non-const structure would make this control flow
less obvious.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/run-command.c b/run-command.c
index e10b1b97390..19d5cff8c65 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1515,7 +1515,6 @@ struct parallel_processes {
 	struct pollfd *pfd;
 
 	unsigned shutdown : 1;
-	const unsigned ungroup : 1;
 
 	size_t output_owner;
 	struct strbuf buffered_output; /* of finished children */
@@ -1552,7 +1551,7 @@ static void pp_init(struct parallel_processes *pp,
 		BUG("you need to specify a get_next_task function");
 
 	CALLOC_ARRAY(pp->children, n);
-	if (!pp->ungroup)
+	if (!opts->ungroup)
 		CALLOC_ARRAY(pp->pfd, n);
 
 	for (size_t i = 0; i < n; i++) {
@@ -1609,17 +1608,17 @@ static int pp_start_one(struct parallel_processes *pp,
 		BUG("bookkeeping is hard");
 
 	code = opts->get_next_task(&pp->children[i].process,
-				   pp->ungroup ? NULL : &pp->children[i].err,
+				   opts->ungroup ? NULL : &pp->children[i].err,
 				   pp->data,
 				   &pp->children[i].data);
 	if (!code) {
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
 		return 1;
 	}
-	if (!pp->ungroup) {
+	if (!opts->ungroup) {
 		pp->children[i].process.err = -1;
 		pp->children[i].process.stdout_to_stderr = 1;
 	}
@@ -1627,14 +1626,14 @@ static int pp_start_one(struct parallel_processes *pp,
 
 	if (start_command(&pp->children[i].process)) {
 		if (opts->start_failure)
-			code = opts->start_failure(pp->ungroup ? NULL :
+			code = opts->start_failure(opts->ungroup ? NULL :
 						   &pp->children[i].err,
 						   pp->data,
 						   pp->children[i].data);
 		else
 			code = 0;
 
-		if (!pp->ungroup) {
+		if (!opts->ungroup) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		}
@@ -1705,7 +1704,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 		code = finish_command(&pp->children[i].process);
 
 		if (opts->task_finished)
-			code = opts->task_finished(code, pp->ungroup ? NULL :
+			code = opts->task_finished(code, opts->ungroup ? NULL :
 						   &pp->children[i].err, pp->data,
 						   pp->children[i].data);
 		else
@@ -1722,7 +1721,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			pp->pfd[i].fd = -1;
 		child_process_init(&pp->children[i].process);
 
-		if (pp->ungroup) {
+		if (opts->ungroup) {
 			; /* no strbuf_*() work to do here */
 		} else if (i != pp->output_owner) {
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
@@ -1761,7 +1760,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		.max_processes = opts->processes,
 		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
-		.ungroup = opts->ungroup,
 	};
 	/* options */
 	const char *tr2_category = opts->tr2_category;
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 12/15] run-command.c: don't copy "data" to "struct parallel_processes"

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in a preceding commit, let's not copy
the "data" member of the "struct run_process_parallel_opts" over to
the "struct parallel_processes". Now that we're passing the "opts"
down there's no reason to do so.

This makes the code easier to follow, as we have a "const" attribute
on the "struct run_process_parallel_opts", but not "struct
parallel_processes". We do not alter the "ungroup" argument, so
storing it in the non-const structure would make this control flow
less obvious.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/run-command.c b/run-command.c
index 19d5cff8c65..d382c42f82d 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1497,8 +1497,6 @@ enum child_state {
 };
 
 struct parallel_processes {
-	void *const data;
-
 	const size_t max_processes;
 	size_t nr_processes;
 
@@ -1609,7 +1607,7 @@ static int pp_start_one(struct parallel_processes *pp,
 
 	code = opts->get_next_task(&pp->children[i].process,
 				   opts->ungroup ? NULL : &pp->children[i].err,
-				   pp->data,
+				   opts->data,
 				   &pp->children[i].data);
 	if (!code) {
 		if (!opts->ungroup) {
@@ -1628,7 +1626,7 @@ static int pp_start_one(struct parallel_processes *pp,
 		if (opts->start_failure)
 			code = opts->start_failure(opts->ungroup ? NULL :
 						   &pp->children[i].err,
-						   pp->data,
+						   opts->data,
 						   pp->children[i].data);
 		else
 			code = 0;
@@ -1705,7 +1703,7 @@ static int pp_collect_finished(struct parallel_processes *pp,
 
 		if (opts->task_finished)
 			code = opts->task_finished(code, opts->ungroup ? NULL :
-						   &pp->children[i].err, pp->data,
+						   &pp->children[i].err, opts->data,
 						   pp->children[i].data);
 		else
 			code = 0;
@@ -1758,7 +1756,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	int spawn_cap = 4;
 	struct parallel_processes pp = {
 		.max_processes = opts->processes,
-		.data = opts->data,
 		.buffered_output = STRBUF_INIT,
 	};
 	/* options */
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 13/15] run-command.c: use "opts->processes", not "pp->max_processes"

[Ævar Arnfjörð Bjarmason]

Neither the "processes" nor "max_processes" members ever change after
their initialization, and they're always equivalent, but some existing
code used "pp->max_processes" when we were already passing the "opts"
to the function, let's use the "opts" directly instead.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/run-command.c b/run-command.c
index d382c42f82d..719a5b58e29 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1599,10 +1599,10 @@ static int pp_start_one(struct parallel_processes *pp,
 	size_t i;
 	int code;
 
-	for (i = 0; i < pp->max_processes; i++)
+	for (i = 0; i < opts->processes; i++)
 		if (pp->children[i].state == GIT_CP_FREE)
 			break;
-	if (i == pp->max_processes)
+	if (i == opts->processes)
 		BUG("bookkeeping is hard");
 
 	code = opts->get_next_task(&pp->children[i].process,
@@ -1689,14 +1689,14 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			       const struct run_process_parallel_opts *opts)
 {
 	int code;
-	size_t i, n = pp->max_processes;
+	size_t i;
 	int result = 0;
 
 	while (pp->nr_processes > 0) {
-		for (i = 0; i < pp->max_processes; i++)
+		for (i = 0; i < opts->processes; i++)
 			if (pp->children[i].state == GIT_CP_WAIT_CLEANUP)
 				break;
-		if (i == pp->max_processes)
+		if (i == opts->processes)
 			break;
 
 		code = finish_command(&pp->children[i].process);
@@ -1725,6 +1725,8 @@ static int pp_collect_finished(struct parallel_processes *pp,
 			strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
 			strbuf_reset(&pp->children[i].err);
 		} else {
+			const size_t n = opts->processes;
+
 			strbuf_write(&pp->children[i].err, stderr);
 			strbuf_reset(&pp->children[i].err);
 
@@ -1771,7 +1773,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
-		    pp.nr_processes < pp.max_processes;
+		    pp.nr_processes < opts->processes;
 		    i++) {
 			code = pp_start_one(&pp, opts);
 			if (!code)
@@ -1785,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		if (!pp.nr_processes)
 			break;
 		if (opts->ungroup) {
-			for (size_t i = 0; i < pp.max_processes; i++)
+			for (size_t i = 0; i < opts->processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
 			pp_buffer_stderr(&pp, output_timeout);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 14/15] run-command.c: pass "opts" further down, and use "opts->processes"

[Ævar Arnfjörð Bjarmason]

Continue the migration away from the "max_processes" member of "struct
parallel_processes" to the "processes" member of the "struct
run_process_parallel_opts", in this case we needed to pass the "opts"
further down into pp_cleanup() and pp_buffer_stderr().

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/run-command.c b/run-command.c
index 719a5b58e29..ce7966394d4 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1565,10 +1565,11 @@ static void pp_init(struct parallel_processes *pp,
 	sigchain_push_common(handle_children_on_signal);
 }
 
-static void pp_cleanup(struct parallel_processes *pp)
+static void pp_cleanup(struct parallel_processes *pp,
+		       const struct run_process_parallel_opts *opts)
 {
 	trace_printf("run_processes_parallel: done");
-	for (size_t i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < opts->processes; i++) {
 		strbuf_release(&pp->children[i].err);
 		child_process_clear(&pp->children[i].process);
 	}
@@ -1647,19 +1648,21 @@ static int pp_start_one(struct parallel_processes *pp,
 	return 0;
 }
 
-static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
+static void pp_buffer_stderr(struct parallel_processes *pp,
+			     const struct run_process_parallel_opts *opts,
+			     int output_timeout)
 {
 	int i;
 
-	while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
+	while ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {
 		if (errno == EINTR)
 			continue;
-		pp_cleanup(pp);
+		pp_cleanup(pp, opts);
 		die_errno("poll");
 	}
 
 	/* Buffer output from all pipes. */
-	for (size_t i = 0; i < pp->max_processes; i++) {
+	for (size_t i = 0; i < opts->processes; i++) {
 		if (pp->children[i].state == GIT_CP_WORKING &&
 		    pp->pfd[i].revents & (POLLIN | POLLHUP)) {
 			int n = strbuf_read_once(&pp->children[i].err,
@@ -1790,7 +1793,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 			for (size_t i = 0; i < opts->processes; i++)
 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
 		} else {
-			pp_buffer_stderr(&pp, output_timeout);
+			pp_buffer_stderr(&pp, opts, output_timeout);
 			pp_output(&pp);
 		}
 		code = pp_collect_finished(&pp, opts);
@@ -1801,7 +1804,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		}
 	}
 
-	pp_cleanup(&pp);
+	pp_cleanup(&pp, opts);
 
 	if (do_trace2)
 		trace2_region_leave(tr2_category, tr2_label, NULL);
-- 
2.38.0.971.ge79ff6d20e7

[PATCH v3 15/15] run-command.c: remove "max_processes", add "const" to signal() handler

[Ævar Arnfjörð Bjarmason]

As with the *_fn members removed in a preceding commit, let's not copy
the "processes" member of the "struct run_process_parallel_opts" over
to the "struct parallel_processes".

In this case we need the number of processes for the kill_children()
function, which will be called from a signal handler. To do that
adjust this code added in c553c72eed6 (run-command: add an
asynchronous parallel child processor, 2015-12-15) so that we use a
dedicated "struct parallel_processes_for_signal" for passing data to
the signal handler, in addition to the "struct parallel_process" it'll
now have access to our "opts" variable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 run-command.c | 37 ++++++++++++++++++++++++++-----------
 1 file changed, 26 insertions(+), 11 deletions(-)

diff --git a/run-command.c b/run-command.c
index ce7966394d4..c772acd7431 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1497,7 +1497,6 @@ enum child_state {
 };
 
 struct parallel_processes {
-	const size_t max_processes;
 	size_t nr_processes;
 
 	struct {
@@ -1518,24 +1517,38 @@ struct parallel_processes {
 	struct strbuf buffered_output; /* of finished children */
 };
 
-static void kill_children(const struct parallel_processes *pp, int signo)
+struct parallel_processes_for_signal {
+	const struct run_process_parallel_opts *opts;
+	const struct parallel_processes *pp;
+};
+
+static void kill_children(const struct parallel_processes *pp,
+			  const struct run_process_parallel_opts *opts,
+			  int signo)
 {
-	for (size_t i = 0; i < pp->max_processes; i++)
+	for (size_t i = 0; i < opts->processes; i++)
 		if (pp->children[i].state == GIT_CP_WORKING)
 			kill(pp->children[i].process.pid, signo);
 }
 
-static struct parallel_processes *pp_for_signal;
+static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,
+				 int signo)
+{
+	kill_children(pp_sig->pp, pp_sig->opts, signo);
+}
+
+static struct parallel_processes_for_signal *pp_for_signal;
 
 static void handle_children_on_signal(int signo)
 {
-	kill_children(pp_for_signal, signo);
+	kill_children_signal(pp_for_signal, signo);
 	sigchain_pop(signo);
 	raise(signo);
 }
 
 static void pp_init(struct parallel_processes *pp,
-		    const struct run_process_parallel_opts *opts)
+		    const struct run_process_parallel_opts *opts,
+		    struct parallel_processes_for_signal *pp_sig)
 {
 	const size_t n = opts->processes;
 
@@ -1561,7 +1574,9 @@ static void pp_init(struct parallel_processes *pp,
 		}
 	}
 
-	pp_for_signal = pp;
+	pp_sig->pp = pp;
+	pp_sig->opts = opts;
+	pp_for_signal = pp_sig;
 	sigchain_push_common(handle_children_on_signal);
 }
 
@@ -1759,8 +1774,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	int i, code;
 	int output_timeout = 100;
 	int spawn_cap = 4;
+	struct parallel_processes_for_signal pp_sig;
 	struct parallel_processes pp = {
-		.max_processes = opts->processes,
 		.buffered_output = STRBUF_INIT,
 	};
 	/* options */
@@ -1772,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
 					   "max:%d", opts->processes);
 
-	pp_init(&pp, opts);
+	pp_init(&pp, opts, &pp_sig);
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
@@ -1783,7 +1798,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 				continue;
 			if (code < 0) {
 				pp.shutdown = 1;
-				kill_children(&pp, -code);
+				kill_children(&pp, opts, -code);
 			}
 			break;
 		}
@@ -1800,7 +1815,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 		if (code) {
 			pp.shutdown = 1;
 			if (code < 0)
-				kill_children(&pp, -code);
+				kill_children(&pp, opts,-code);
 		}
 	}
 
-- 
2.38.0.971.ge79ff6d20e7

Re: [PATCH v3 15/15] run-command.c: remove "max_processes", add "const" to signal() handler

[Glen Choo]

Ævar Arnfjörð Bjarmason         <avarab@gmail.com> writes:

> In this case we need the number of processes for the kill_children()
> function, which will be called from a signal handler. To do that
> adjust this code added in c553c72eed6 (run-command: add an
> asynchronous parallel child processor, 2015-12-15) so that we use a
> dedicated "struct parallel_processes_for_signal" for passing data to
> the signal handler, in addition to the "struct parallel_process" it'll
> now have access to our "opts" variable.

[...]

> -static void kill_children(const struct parallel_processes *pp, int signo)
> +struct parallel_processes_for_signal {
> +	const struct run_process_parallel_opts *opts;
> +	const struct parallel_processes *pp;
> +};

(Treat this feedback as non-blocking)

I find this struct a little odd because it isn't specific to
kill_children_signal(), rather, this seems like we are saying that
"run_process_parallel_opts" and "parallel_processes" belong together.
This also seems confirmed by the fact that at the end of the series, we
pass the opts almost everywhere - only pp_output() takes
"parallel_processes" without "run_process_parallel_opts".

I assume you've already considered this, but I wonder if the code is
simpler if we drop 10-15/15, i.e. use options just for end users
(allowing us to get rid of the *_tr2() variant too, which is great), but
only use "parallel_processes" inside of run-command.c. There's also a
tiny benefit of passing one fewer pointer per function, with the tiny
cost of one-time copying.

Thoughts?

> +
> +static void kill_children(const struct parallel_processes *pp,
> +			  const struct run_process_parallel_opts *opts,
> +			  int signo)
>  {
> -	for (size_t i = 0; i < pp->max_processes; i++)
> +	for (size_t i = 0; i < opts->processes; i++)
>  		if (pp->children[i].state == GIT_CP_WORKING)
>  			kill(pp->children[i].process.pid, signo);
>  }
>  
> -static struct parallel_processes *pp_for_signal;
> +static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,
> +				 int signo)
> +{
> +	kill_children(pp_sig->pp, pp_sig->opts, signo);
> +}
> +
> +static struct parallel_processes_for_signal *pp_for_signal;
>  
>  static void handle_children_on_signal(int signo)
>  {
> -	kill_children(pp_for_signal, signo);
> +	kill_children_signal(pp_for_signal, signo);
>  	sigchain_pop(signo);
>  	raise(signo);
>  }
>  
>  static void pp_init(struct parallel_processes *pp,
> -		    const struct run_process_parallel_opts *opts)
> +		    const struct run_process_parallel_opts *opts,
> +		    struct parallel_processes_for_signal *pp_sig)
>  {
>  	const size_t n = opts->processes;
>  
> @@ -1561,7 +1574,9 @@ static void pp_init(struct parallel_processes *pp,
>  		}
>  	}
>  
> -	pp_for_signal = pp;
> +	pp_sig->pp = pp;
> +	pp_sig->opts = opts;
> +	pp_for_signal = pp_sig;
>  	sigchain_push_common(handle_children_on_signal);
>  }
>  
> @@ -1759,8 +1774,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  	int i, code;
>  	int output_timeout = 100;
>  	int spawn_cap = 4;
> +	struct parallel_processes_for_signal pp_sig;
>  	struct parallel_processes pp = {
> -		.max_processes = opts->processes,
>  		.buffered_output = STRBUF_INIT,
>  	};
>  	/* options */
> @@ -1772,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
>  					   "max:%d", opts->processes);
>  
> -	pp_init(&pp, opts);
> +	pp_init(&pp, opts, &pp_sig);
>  	while (1) {
>  		for (i = 0;
>  		    i < spawn_cap && !pp.shutdown &&
> @@ -1783,7 +1798,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  				continue;
>  			if (code < 0) {
>  				pp.shutdown = 1;
> -				kill_children(&pp, -code);
> +				kill_children(&pp, opts, -code);
>  			}
>  			break;
>  		}
> @@ -1800,7 +1815,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
>  		if (code) {
>  			pp.shutdown = 1;
>  			if (code < 0)
> -				kill_children(&pp, -code);
> +				kill_children(&pp, opts,-code);
>  		}
>  	}
>  
> -- 
> 2.38.0.971.ge79ff6d20e7

Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct

[Calvin Wan]

Hi Ævar

Thank you for condensing the patch series into something more palatable
for reviewers. The general consensus from the review club yesterday (we
looked at v2) was that it was difficult to follow what patches were
relevant to your original intention and what patches were auxiliary QOL
changes. Also having too many intermediary patches where you add
variables/functions that were later deleted made it hard to visualize
which parts of the patch would end up making it into the final state. I
appreciate the "show your work" approach you take, but I think that
approach is better suited for more difficult patch series where there
are significant/difficult to understand functionality changes. In this
case, the end state of a refactor is clear and there are no
functionality changes so I believe a more condensed series would be
easier to review. That being said, I don't believe there is a need to
spend more time trying to condense this series so future reviewers have
an easier time -- the end result and intentions are enough.

Reviewed-by: Calvin Wan <calvinwan@google.com>

On Wed, Oct 12, 2022 at 2:03 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> For a general overview see the v2 CL:
> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/
>
> Changes since v2:
>
>  * Ejected various not-to-the-point of converting to the "opts struct"
>    per feedback about attempting to make this leaner.
>  * I kept the size_t change & the online_cpus() fallback, and updated
>    a commit message for the latter. For "int" v.s. "size_t" once we're
>    not handling "-1" to mean "use the default" it makes sense to be
>    unsigned, and if we're re-doing that at this point we'd need
>    rewrites for "!processes" assumptions.
>  * Squashed the multi-step introduction of the "opts" struct, per
>    Phillip's suggestion.
>  * Fixed a segfault in the v2's 22/22 (this 15/15).
>  * Got rid of superfluous unsigned conversions of code related to the
>    "env" member.
>
> Ævar Arnfjörð Bjarmason (15):
>   run-command test helper: use "else if" pattern
>   run-command API: have "run_processes_parallel{,_tr2}()" return void
>   run-command tests: use "return", not "exit"
>   run-command API: make "n" parameter a "size_t"
>   run-command API: don't fall back on online_cpus()
>   run-command.c: use designated init for pp_init(), add "const"
>   run-command API: have run_process_parallel() take an "opts" struct
>   run-command API: move *_tr2() users to "run_processes_parallel()"
>   run-command.c: make "struct parallel_processes" const if possible
>   run-command.c: don't copy *_fn to "struct parallel_processes"
>   run-command.c: don't copy "ungroup" to "struct parallel_processes"
>   run-command.c: don't copy "data" to "struct parallel_processes"
>   run-command.c: use "opts->processes", not "pp->max_processes"
>   run-command.c: pass "opts" further down, and use "opts->processes"
>   run-command.c: remove "max_processes", add "const" to signal() handler
>
>  builtin/fetch.c             |  25 ++--
>  builtin/submodule--helper.c |  16 ++-
>  hook.c                      |  23 ++--
>  run-command.c               | 236 ++++++++++++++++--------------------
>  run-command.h               |  71 ++++++++---
>  submodule-config.c          |   2 +
>  submodule.c                 |  18 ++-
>  t/helper/test-run-command.c |  77 +++++++-----
>  t/t5526-fetch-submodules.sh |   8 +-
>  9 files changed, 268 insertions(+), 208 deletions(-)
>
> Range-diff against v2:
>  1:  bc51dfcb1be <  -:  ----------- hook tests: fix redirection logic error in 96e7225b310
>  2:  3027f5587a7 <  -:  ----------- submodule tests: reset "trace.out" between "grep" invocations
>  3:  c4923358bbd <  -:  ----------- run-command tests: test stdout of run_command_parallel()
>  4:  26e28086252 =  1:  d3a2489d9b2 run-command test helper: use "else if" pattern
>  5:  5e09dc68fd9 =  2:  a2e4fd652c1 run-command API: have "run_processes_parallel{,_tr2}()" return void
>  6:  e4e91dbbf9e =  3:  4a19de65783 run-command tests: use "return", not "exit"
>  7:  b90961ae76d <  -:  ----------- run-command.c: remove dead assignment in while-loop
>  8:  279b0430c5d <  -:  ----------- run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful
>  9:  a900711270c !  4:  58018a79b2f run-command API: make "n" parameter a "size_t"
>     @@ Commit message
>
>                  make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
>
>     -    Only has one (and new) -Wsigned-compare warning, about a comparison of
>     -    "i" to online_cpus(), a subsequent commit will adjust & deal with
>     -    online_cpus() and that warning.
>     +    Only has one (and new) -Wsigned-compare warning relevant to a
>     +    comparison about our "n" or "{nr,max}_processes": About using our
>     +    "n" (size_t) in the same expression as online_cpus() (int). A
>     +    subsequent commit will adjust & deal with online_cpus() and that
>     +    warning.
>
>          The only users of the "n" parameter are:
>
>     @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
>
>         for (i = 0; i < pp->max_processes; i++)
>                 if (pp->children[i].state == GIT_CP_FREE)
>     -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
>     -
>     - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>     - {
>     --  int i;
>     --
>     -   while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
>     -           if (errno == EINTR)
>     -                   continue;
>      @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>         }
>
> 10:  eb9d672b0d8 !  5:  e230701dff6 run-command API: don't fall back on online_cpus()
>     @@ Commit message
>          child processor, 2015-12-15).
>
>          Most of our code in-tree that scales up to "online_cpus()" by default
>     -    calls that function by itself. By having these callers of the
>     -    "run_processes_parallel()" API do the same we can in subsequent
>     -    commits pass all arguments down as a "const struct".
>     +    calls that function by itself. Keeping this default behavior just for
>     +    the sake of two callers means that we'd need to maintain this one spot
>     +    where we're second-guessing the config passed down into pp_init().
>
>          The preceding commit has an overview of the API callers that passed
>          "jobs = 0". There were only two of them (actually three, but they
>     @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v
>
>       ## t/t5526-fetch-submodules.sh ##
>      @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '
>     +           GIT_TRACE=$(pwd)/trace.out git fetch &&
>     +           grep "8 tasks" trace.out &&
>                 GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
>     -           grep "9 tasks" trace.out &&
>     -           >trace.out &&
>     +-          grep "9 tasks" trace.out
>     ++          grep "9 tasks" trace.out &&
>     ++          >trace.out &&
>      +
>      +          GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
>      +          grep "preparing to run up to [0-9]* tasks" trace.out &&
> 11:  aedda10d8e1 !  6:  df2ca5dd097 run-command.c: use designated init for pp_init(), add "const"
>     @@ run-command.c: void run_processes_parallel(size_t n,
>      -          ungroup);
>      +  pp_init(&pp, get_next_task, start_failure, task_finished);
>         while (1) {
>     -           for (int i = 0;
>     +           for (i = 0;
>                     i < spawn_cap && !pp.shutdown &&
> 12:  fde2af11579 <  -:  ----------- run-command API: add nascent "struct run_process_parallel_opts"
> 13:  01e894bed90 <  -:  ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers
> 14:  41c2886b44b !  7:  eaed3d8838d run-command API: have run_process_parallel() take an "opts" struct
>     @@ Metadata
>       ## Commit message ##
>          run-command API: have run_process_parallel() take an "opts" struct
>
>     -    Have the "run_process_parallel()" function take an "opts" struct,
>     -    which allows us to eliminate it as a wrapper for
>     -    "run_processes_parallel_1()".
>     +    As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
>     +    run_process_parallel(), 2022-06-07) which added the "ungroup" passing
>     +    it to "run_process_parallel()" via the global
>     +    "run_processes_parallel_ungroup" variable was a compromise to get the
>     +    smallest possible regression fix for "maint" at the time.
>     +
>     +    This follow-up to that is a start at passing that parameter and others
>     +    via a new "struct run_process_parallel_opts", as the earlier
>     +    version[1] of what became fd3aaf53f71 did.
>     +
>     +    Since we need to change all of the occurrences of "n" to
>     +    "opt->SOMETHING" let's take the opportunity and rename the terse "n"
>     +    to "processes". We could also have picked "max_processes", "jobs",
>     +    "threads" etc., but as the API is named "run_processes_parallel()"
>     +    let's go with "processes".
>
>          Since the new "run_processes_parallel()" function is able to take an
>          optional "tr2_category" and "tr2_label" via the struct we can at this
>          point migrate all of the users of "run_processes_parallel_tr2()" over
>          to it.
>
>     -    But let's not migrate all the API users, only the two users that
>     +    But let's not migrate all the API users yet, only the two users that
>          passed the "ungroup" parameter via the
>     -    "run_processes_parallel_ungroup" global, allowing us to delete that
>     -    global in favor of passing "ungroup" via the "opts" struct. As noted
>     -    in fd3aaf53f71 (run-command: add an "ungroup" option to
>     -    run_process_parallel(), 2022-06-07) which added
>     -    "run_processes_parallel_ungroup" passing this as a global was a hack
>     -    to produce a small regression fix for "maint".
>     +    "run_processes_parallel_ungroup" global
>     +
>     +    1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
>
>          Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>
>     @@ run-command.c: enum child_state {
>       struct parallel_processes {
>         void *const data;
>
>     +@@ run-command.c: static void handle_children_on_signal(int signo)
>     + }
>     +
>     + static void pp_init(struct parallel_processes *pp,
>     +-              get_next_task_fn get_next_task,
>     +-              start_failure_fn start_failure,
>     +-              task_finished_fn task_finished)
>     ++              const struct run_process_parallel_opts *opts)
>     + {
>     +-  const size_t n = pp->max_processes;
>     ++  const size_t n = opts->processes;
>     ++  get_next_task_fn get_next_task = opts->get_next_task;
>     ++  start_failure_fn start_failure = opts->start_failure;
>     ++  task_finished_fn task_finished = opts->task_finished;
>     +
>     +   if (!n)
>     +           BUG("you must provide a non-zero number of processes!");
>      @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
>         return result;
>       }
>
>     --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
>     +-void run_processes_parallel(size_t n,
>     +-                      get_next_task_fn get_next_task,
>     +-                      start_failure_fn start_failure,
>     +-                      task_finished_fn task_finished,
>     +-                      void *pp_cb)
>      +void run_processes_parallel(const struct run_process_parallel_opts *opts)
>       {
>     -   int code;
>     +   int i, code;
>         int output_timeout = 100;
>         int spawn_cap = 4;
>     +-  int ungroup = run_processes_parallel_ungroup;
>         struct parallel_processes pp = {
>     -           .max_processes = opts->processes,
>     +-          .max_processes = n,
>      -          .data = pp_cb,
>     ++          .max_processes = opts->processes,
>      +          .data = opts->data,
>                 .buffered_output = STRBUF_INIT,
>     --          .ungroup = run_processes_parallel_ungroup,
>     +-          .ungroup = ungroup,
>      +          .ungroup = opts->ungroup,
>         };
>     -   /* options */
>     -   const char *tr2_category = opts->tr2_category;
>     -   const char *tr2_label = opts->tr2_label;
>     -   const int do_trace2 = tr2_category && tr2_label;
>     ++  /* options */
>     ++  const char *tr2_category = opts->tr2_category;
>     ++  const char *tr2_label = opts->tr2_label;
>     ++  const int do_trace2 = tr2_category && tr2_label;
>
>      -  /* unset for the next API user */
>      -  run_processes_parallel_ungroup = 0;
>     --
>     -   if (do_trace2)
>     -           trace2_region_enter_printf(tr2_category, tr2_label, NULL,
>     -                                      "max:%d", opts->processes);
>     -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
>     -           trace2_region_leave(tr2_category, tr2_label, NULL);
>     ++  if (do_trace2)
>     ++          trace2_region_enter_printf(tr2_category, tr2_label, NULL,
>     ++                                     "max:%d", opts->processes);
>     +
>     +-  pp_init(&pp, get_next_task, start_failure, task_finished);
>     ++  pp_init(&pp, opts);
>     +   while (1) {
>     +           for (i = 0;
>     +               i < spawn_cap && !pp.shutdown &&
>     +@@ run-command.c: void run_processes_parallel(size_t n,
>     +           }
>     +           if (!pp.nr_processes)
>     +                   break;
>     +-          if (ungroup) {
>     ++          if (opts->ungroup) {
>     +                   for (size_t i = 0; i < pp.max_processes; i++)
>     +                           pp.children[i].state = GIT_CP_WAIT_CLEANUP;
>     +           } else {
>     +@@ run-command.c: void run_processes_parallel(size_t n,
>     +   }
>     +
>     +   pp_cleanup(&pp);
>     ++
>     ++  if (do_trace2)
>     ++          trace2_region_leave(tr2_category, tr2_label, NULL);
>       }
>
>     --void run_processes_parallel(size_t processes,
>     --                      get_next_task_fn get_next_task,
>     --                      start_failure_fn start_failure,
>     --                      task_finished_fn task_finished,
>     --                      void *pp_cb)
>     --{
>     --  const struct run_process_parallel_opts opts = {
>     --          .processes = processes,
>     --          .ungroup = run_processes_parallel_ungroup,
>     --
>     --          .get_next_task = get_next_task,
>     --          .start_failure = start_failure,
>     --          .task_finished = task_finished,
>     --  };
>     --
>     --  run_processes_parallel_1(&opts, pp_cb);
>     --}
>     --
>     - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
>     +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
>     ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
>                                 start_failure_fn start_failure,
>                                 task_finished_fn task_finished, void *pp_cb,
>     -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task
>     -   const struct run_process_parallel_opts opts = {
>     -           .tr2_category = tr2_category,
>     -           .tr2_label = tr2_label,
>     --
>     -           .processes = processes,
>     --          .ungroup = run_processes_parallel_ungroup,
>     +                           const char *tr2_category, const char *tr2_label)
>     + {
>     +-  trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
>     ++  const struct run_process_parallel_opts opts = {
>     ++          .tr2_category = tr2_category,
>     ++          .tr2_label = tr2_label,
>     ++          .processes = processes,
>
>     -           .get_next_task = get_next_task,
>     -           .start_failure = start_failure,
>     -           .task_finished = task_finished,
>     -   };
>     +-  run_processes_parallel(n, get_next_task, start_failure,
>     +-                         task_finished, pp_cb);
>     ++          .get_next_task = get_next_task,
>     ++          .start_failure = start_failure,
>     ++          .task_finished = task_finished,
>     ++  };
>
>     --  run_processes_parallel_1(&opts, pp_cb);
>     +-  trace2_region_leave(tr2_category, tr2_label, NULL);
>      +  run_processes_parallel(&opts);
>       }
>
>       int run_auto_maintenance(int quiet)
>
>       ## run-command.h ##
>     -@@ run-command.h: struct run_process_parallel_opts
>     -    * NULL to omit any special handling.
>     -    */
>     -   task_finished_fn task_finished;
>     +@@ run-command.h: typedef int (*task_finished_fn)(int result,
>     +                           void *pp_task_cb);
>     +
>     + /**
>     +- * Runs up to n processes at the same time. Whenever a process can be
>     +- * started, the callback get_next_task_fn is called to obtain the data
>     ++ * Option used by run_processes_parallel(), { 0 }-initialized means no
>     ++ * options.
>     ++ */
>     ++struct run_process_parallel_opts
>     ++{
>     ++  /**
>     ++   * tr2_category & tr2_label: sets the trace2 category and label for
>     ++   * logging. These must either be unset, or both of them must be set.
>     ++   */
>     ++  const char *tr2_category;
>     ++  const char *tr2_label;
>     ++
>     ++  /**
>     ++   * processes: see 'processes' in run_processes_parallel() below.
>     ++   */
>     ++  size_t processes;
>     ++
>     ++  /**
>     ++   * ungroup: see 'ungroup' in run_processes_parallel() below.
>     ++   */
>     ++  unsigned int ungroup:1;
>     ++
>     ++  /**
>     ++   * get_next_task: See get_next_task_fn() above. This must be
>     ++   * specified.
>     ++   */
>     ++  get_next_task_fn get_next_task;
>     ++
>     ++  /**
>     ++   * start_failure: See start_failure_fn() above. This can be
>     ++   * NULL to omit any special handling.
>     ++   */
>     ++  start_failure_fn start_failure;
>     ++
>     ++  /**
>     ++   * task_finished: See task_finished_fn() above. This can be
>     ++   * NULL to omit any special handling.
>     ++   */
>     ++  task_finished_fn task_finished;
>      +
>      +  /**
>      +   * data: user data, will be passed as "pp_cb" to the callback
>      +   * parameters.
>      +   */
>      +  void *data;
>     - };
>     -
>     - /**
>     ++};
>     ++
>     ++/**
>      + * Options are passed via the "struct run_process_parallel_opts" above.
>      + *
>     -  * Runs N 'processes' at the same time. Whenever a process can be
>     -- * started, the callback get_next_task_fn is called to obtain the data
>     ++ * Runs N 'processes' at the same time. Whenever a process can be
>      + * started, the callback opts.get_next_task is called to obtain the data
>        * required to start another child process.
>        *
>        * The children started via this function run in parallel. Their output
>     -@@ run-command.h: struct run_process_parallel_opts
>     +  * (both stdout and stderr) is routed to stderr in a manner that output
>     +  * from different tasks does not interleave (but see "ungroup" below).
>     +  *
>     +- * start_failure_fn and task_finished_fn can be NULL to omit any
>     +- * special handling.
>     +- *
>     +  * If the "ungroup" option isn't specified, the API will set the
>     +  * "stdout_to_stderr" parameter in "struct child_process" and provide
>     +  * the callbacks with a "struct strbuf *out" parameter to write output
>     +@@ run-command.h: typedef int (*task_finished_fn)(int result,
>        * NULL "struct strbuf *out" parameter, and are responsible for
>        * emitting their own output, including dealing with any race
>        * conditions due to writing in parallel to stdout and stderr.
>     @@ run-command.h: struct run_process_parallel_opts
>      - * API reads that setting.
>        */
>      -extern int run_processes_parallel_ungroup;
>     --void run_processes_parallel(size_t processes,
>     +-void run_processes_parallel(size_t n,
>      -                      get_next_task_fn,
>      -                      start_failure_fn,
>      -                      task_finished_fn,
>      -                      void *pp_cb);
>     +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
>     +-                          task_finished_fn, void *pp_cb,
>      +void run_processes_parallel(const struct run_process_parallel_opts *opts);
>     - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
>     -                           start_failure_fn, task_finished_fn, void *pp_cb,
>     ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
>     ++                          start_failure_fn, task_finished_fn, void *pp_cb,
>                                 const char *tr2_category, const char *tr2_label);
>     +
>     + /**
>
>       ## t/helper/test-run-command.c ##
>      @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {
> 15:  391d1d99d91 =  8:  c19c60b2e95 run-command API: move *_tr2() users to "run_processes_parallel()"
> 16:  acac50cc1a5 =  9:  99a4f4f6b9c run-command.c: make "struct parallel_processes" const if possible
> 17:  fdd64236985 = 10:  bf67e24bcc5 run-command.c: don't copy *_fn to "struct parallel_processes"
> 18:  17f34d81ecd = 11:  06de42adc2e run-command.c: don't copy "ungroup" to "struct parallel_processes"
> 19:  9cbee2dfe76 = 12:  3081dfc49d1 run-command.c: don't copy "data" to "struct parallel_processes"
> 20:  2dabed9e155 ! 13:  b7c10f6a23f run-command.c: use "opts->processes", not "pp->max_processes"
>     @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
>
>      @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>         while (1) {
>     -           for (int i = 0;
>     +           for (i = 0;
>                     i < spawn_cap && !pp.shutdown &&
>      -              pp.nr_processes < pp.max_processes;
>      +              pp.nr_processes < opts->processes;
> 21:  c1a286a8ebb ! 14:  4856d6a4674 run-command.c: pass "opts" further down, and use "opts->processes"
>     @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
>      +                       const struct run_process_parallel_opts *opts,
>      +                       int output_timeout)
>       {
>     --  while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
>     -+  while (poll(pp->pfd, opts->processes, output_timeout) < 0) {
>     +   int i;
>     +
>     +-  while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
>     ++  while ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {
>                 if (errno == EINTR)
>                         continue;
>      -          pp_cleanup(pp);
> 22:  541f41566e7 ! 15:  39a20be0cbb run-command.c: remove "pp->max_processes", add "const" to signal() handler
>     @@ Metadata
>      Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>
>       ## Commit message ##
>     -    run-command.c: remove "pp->max_processes", add "const" to signal() handler
>     +    run-command.c: remove "max_processes", add "const" to signal() handler
>
>          As with the *_fn members removed in a preceding commit, let's not copy
>          the "processes" member of the "struct run_process_parallel_opts" over
>     @@ run-command.c: static void pp_init(struct parallel_processes *pp,
>      -  pp_for_signal = pp;
>      +  pp_sig->pp = pp;
>      +  pp_sig->opts = opts;
>     ++  pp_for_signal = pp_sig;
>         sigchain_push_common(handle_children_on_signal);
>       }
>
>      @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>     -   int code;
>     +   int i, code;
>         int output_timeout = 100;
>         int spawn_cap = 4;
>      +  struct parallel_processes_for_signal pp_sig;
>     @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
>      -  pp_init(&pp, opts);
>      +  pp_init(&pp, opts, &pp_sig);
>         while (1) {
>     -           for (int i = 0;
>     +           for (i = 0;
>                     i < spawn_cap && !pp.shutdown &&
>      @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>                                 continue;
> --
> 2.38.0.971.ge79ff6d20e7
>

Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct

[Junio C Hamano]

Calvin Wan <calvinwan@google.com> writes:

> Hi Ævar
>
> Thank you for condensing the patch series into something more palatable
> for reviewers. The general consensus from the review club yesterday (we
> looked at v2) was that it was difficult to follow what patches were
> relevant to your original intention and what patches were auxiliary QOL
> changes. Also having too many intermediary patches where you add
> variables/functions that were later deleted made it hard to visualize
> which parts of the patch would end up making it into the final state. I
> appreciate the "show your work" approach you take, but I think that
> approach is better suited for more difficult patch series where there
> are significant/difficult to understand functionality changes. In this
> case, the end state of a refactor is clear and there are no
> functionality changes so I believe a more condensed series would be
> easier to review. That being said, I don't believe there is a need to
> spend more time trying to condense this series so future reviewers have
> an easier time -- the end result and intentions are enough.
>
> Reviewed-by: Calvin Wan <calvinwan@google.com>

Thanks, both.

Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct

[Phillip Wood]

Hi Ævar

On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:
> For a general overview see the v2 CL:
> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/
> 
> Changes since v2:
> 
>   * Ejected various not-to-the-point of converting to the "opts struct"
>     per feedback about attempting to make this leaner.

We're back to the same number of patches as v1 but you've removed three 
test cleanups and squashed three patches together which means there are 
five new patches in this version - what are they doing?

>   * I kept the size_t change & the online_cpus() fallback, and updated
>     a commit message for the latter. For "int" v.s. "size_t" once we're
>     not handling "-1" to mean "use the default" it makes sense to be
>     unsigned, and if we're re-doing that at this point we'd need
>     rewrites for "!processes" assumptions.

I left some comments about this, I think the size_t change is taking us 
in the wrong direction as it is introducing a number of new implicit 
signed->unsigned conversions. I'm still not sure why you need the 
online_cups() changes c.f. 
https://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com/ 
which has never had a reply

>   * Squashed the multi-step introduction of the "opts" struct, per
>     Phillip's suggestion.
That's most welcome, thanks

>   * Fixed a segfault in the v2's 22/22 (this 15/15).
>   * Got rid of superfluous unsigned conversions of code related to the
>     "env" member.

I've read through to 'API: move *_tr2() users to 
"run_processes_parallel()"' and apart from my comments above they seem 
reasonable (I'm not that taken with the const struct members but it is a 
fairly small change)

Best Wishes

Phillip

> Ævar Arnfjörð Bjarmason (15):
>    run-command test helper: use "else if" pattern
>    run-command API: have "run_processes_parallel{,_tr2}()" return void
>    run-command tests: use "return", not "exit"
>    run-command API: make "n" parameter a "size_t"
>    run-command API: don't fall back on online_cpus()
>    run-command.c: use designated init for pp_init(), add "const"
>    run-command API: have run_process_parallel() take an "opts" struct
>    run-command API: move *_tr2() users to "run_processes_parallel()"
>    run-command.c: make "struct parallel_processes" const if possible
>    run-command.c: don't copy *_fn to "struct parallel_processes"
>    run-command.c: don't copy "ungroup" to "struct parallel_processes"
>    run-command.c: don't copy "data" to "struct parallel_processes"
>    run-command.c: use "opts->processes", not "pp->max_processes"
>    run-command.c: pass "opts" further down, and use "opts->processes"
>    run-command.c: remove "max_processes", add "const" to signal() handler
> 
>   builtin/fetch.c             |  25 ++--
>   builtin/submodule--helper.c |  16 ++-
>   hook.c                      |  23 ++--
>   run-command.c               | 236 ++++++++++++++++--------------------
>   run-command.h               |  71 ++++++++---
>   submodule-config.c          |   2 +
>   submodule.c                 |  18 ++-
>   t/helper/test-run-command.c |  77 +++++++-----
>   t/t5526-fetch-submodules.sh |   8 +-
>   9 files changed, 268 insertions(+), 208 deletions(-)
> 
> Range-diff against v2:
>   1:  bc51dfcb1be <  -:  ----------- hook tests: fix redirection logic error in 96e7225b310
>   2:  3027f5587a7 <  -:  ----------- submodule tests: reset "trace.out" between "grep" invocations
>   3:  c4923358bbd <  -:  ----------- run-command tests: test stdout of run_command_parallel()
>   4:  26e28086252 =  1:  d3a2489d9b2 run-command test helper: use "else if" pattern
>   5:  5e09dc68fd9 =  2:  a2e4fd652c1 run-command API: have "run_processes_parallel{,_tr2}()" return void
>   6:  e4e91dbbf9e =  3:  4a19de65783 run-command tests: use "return", not "exit"
>   7:  b90961ae76d <  -:  ----------- run-command.c: remove dead assignment in while-loop
>   8:  279b0430c5d <  -:  ----------- run-command.c: use C99 "for (TYPE VAR = ..." syntax where useful
>   9:  a900711270c !  4:  58018a79b2f run-command API: make "n" parameter a "size_t"
>      @@ Commit message
>       
>                   make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter
>       
>      -    Only has one (and new) -Wsigned-compare warning, about a comparison of
>      -    "i" to online_cpus(), a subsequent commit will adjust & deal with
>      -    online_cpus() and that warning.
>      +    Only has one (and new) -Wsigned-compare warning relevant to a
>      +    comparison about our "n" or "{nr,max}_processes": About using our
>      +    "n" (size_t) in the same expression as online_cpus() (int). A
>      +    subsequent commit will adjust & deal with online_cpus() and that
>      +    warning.
>       
>           The only users of the "n" parameter are:
>       
>      @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)
>        
>        	for (i = 0; i < pp->max_processes; i++)
>        		if (pp->children[i].state == GIT_CP_FREE)
>      -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)
>      -
>      - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>      - {
>      --	int i;
>      --
>      - 	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
>      - 		if (errno == EINTR)
>      - 			continue;
>       @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>        	}
>        
> 10:  eb9d672b0d8 !  5:  e230701dff6 run-command API: don't fall back on online_cpus()
>      @@ Commit message
>           child processor, 2015-12-15).
>       
>           Most of our code in-tree that scales up to "online_cpus()" by default
>      -    calls that function by itself. By having these callers of the
>      -    "run_processes_parallel()" API do the same we can in subsequent
>      -    commits pass all arguments down as a "const struct".
>      +    calls that function by itself. Keeping this default behavior just for
>      +    the sake of two callers means that we'd need to maintain this one spot
>      +    where we're second-guessing the config passed down into pp_init().
>       
>           The preceding commit has an overview of the API callers that passed
>           "jobs = 0". There were only two of them (actually three, but they
>      @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v
>       
>        ## t/t5526-fetch-submodules.sh ##
>       @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '
>      + 		GIT_TRACE=$(pwd)/trace.out git fetch &&
>      + 		grep "8 tasks" trace.out &&
>        		GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
>      - 		grep "9 tasks" trace.out &&
>      - 		>trace.out &&
>      +-		grep "9 tasks" trace.out
>      ++		grep "9 tasks" trace.out &&
>      ++		>trace.out &&
>       +
>       +		GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&
>       +		grep "preparing to run up to [0-9]* tasks" trace.out &&
> 11:  aedda10d8e1 !  6:  df2ca5dd097 run-command.c: use designated init for pp_init(), add "const"
>      @@ run-command.c: void run_processes_parallel(size_t n,
>       -		ungroup);
>       +	pp_init(&pp, get_next_task, start_failure, task_finished);
>        	while (1) {
>      - 		for (int i = 0;
>      + 		for (i = 0;
>        		    i < spawn_cap && !pp.shutdown &&
> 12:  fde2af11579 <  -:  ----------- run-command API: add nascent "struct run_process_parallel_opts"
> 13:  01e894bed90 <  -:  ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers
> 14:  41c2886b44b !  7:  eaed3d8838d run-command API: have run_process_parallel() take an "opts" struct
>      @@ Metadata
>        ## Commit message ##
>           run-command API: have run_process_parallel() take an "opts" struct
>       
>      -    Have the "run_process_parallel()" function take an "opts" struct,
>      -    which allows us to eliminate it as a wrapper for
>      -    "run_processes_parallel_1()".
>      +    As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
>      +    run_process_parallel(), 2022-06-07) which added the "ungroup" passing
>      +    it to "run_process_parallel()" via the global
>      +    "run_processes_parallel_ungroup" variable was a compromise to get the
>      +    smallest possible regression fix for "maint" at the time.
>      +
>      +    This follow-up to that is a start at passing that parameter and others
>      +    via a new "struct run_process_parallel_opts", as the earlier
>      +    version[1] of what became fd3aaf53f71 did.
>      +
>      +    Since we need to change all of the occurrences of "n" to
>      +    "opt->SOMETHING" let's take the opportunity and rename the terse "n"
>      +    to "processes". We could also have picked "max_processes", "jobs",
>      +    "threads" etc., but as the API is named "run_processes_parallel()"
>      +    let's go with "processes".
>       
>           Since the new "run_processes_parallel()" function is able to take an
>           optional "tr2_category" and "tr2_label" via the struct we can at this
>           point migrate all of the users of "run_processes_parallel_tr2()" over
>           to it.
>       
>      -    But let's not migrate all the API users, only the two users that
>      +    But let's not migrate all the API users yet, only the two users that
>           passed the "ungroup" parameter via the
>      -    "run_processes_parallel_ungroup" global, allowing us to delete that
>      -    global in favor of passing "ungroup" via the "opts" struct. As noted
>      -    in fd3aaf53f71 (run-command: add an "ungroup" option to
>      -    run_process_parallel(), 2022-06-07) which added
>      -    "run_processes_parallel_ungroup" passing this as a global was a hack
>      -    to produce a small regression fix for "maint".
>      +    "run_processes_parallel_ungroup" global
>      +
>      +    1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
>       
>           Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>       
>      @@ run-command.c: enum child_state {
>        struct parallel_processes {
>        	void *const data;
>        
>      +@@ run-command.c: static void handle_children_on_signal(int signo)
>      + }
>      +
>      + static void pp_init(struct parallel_processes *pp,
>      +-		    get_next_task_fn get_next_task,
>      +-		    start_failure_fn start_failure,
>      +-		    task_finished_fn task_finished)
>      ++		    const struct run_process_parallel_opts *opts)
>      + {
>      +-	const size_t n = pp->max_processes;
>      ++	const size_t n = opts->processes;
>      ++	get_next_task_fn get_next_task = opts->get_next_task;
>      ++	start_failure_fn start_failure = opts->start_failure;
>      ++	task_finished_fn task_finished = opts->task_finished;
>      +
>      + 	if (!n)
>      + 		BUG("you must provide a non-zero number of processes!");
>       @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)
>        	return result;
>        }
>        
>      --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)
>      +-void run_processes_parallel(size_t n,
>      +-			    get_next_task_fn get_next_task,
>      +-			    start_failure_fn start_failure,
>      +-			    task_finished_fn task_finished,
>      +-			    void *pp_cb)
>       +void run_processes_parallel(const struct run_process_parallel_opts *opts)
>        {
>      - 	int code;
>      + 	int i, code;
>        	int output_timeout = 100;
>        	int spawn_cap = 4;
>      +-	int ungroup = run_processes_parallel_ungroup;
>        	struct parallel_processes pp = {
>      - 		.max_processes = opts->processes,
>      +-		.max_processes = n,
>       -		.data = pp_cb,
>      ++		.max_processes = opts->processes,
>       +		.data = opts->data,
>        		.buffered_output = STRBUF_INIT,
>      --		.ungroup = run_processes_parallel_ungroup,
>      +-		.ungroup = ungroup,
>       +		.ungroup = opts->ungroup,
>        	};
>      - 	/* options */
>      - 	const char *tr2_category = opts->tr2_category;
>      - 	const char *tr2_label = opts->tr2_label;
>      - 	const int do_trace2 = tr2_category && tr2_label;
>      ++	/* options */
>      ++	const char *tr2_category = opts->tr2_category;
>      ++	const char *tr2_label = opts->tr2_label;
>      ++	const int do_trace2 = tr2_category && tr2_label;
>        
>       -	/* unset for the next API user */
>       -	run_processes_parallel_ungroup = 0;
>      --
>      - 	if (do_trace2)
>      - 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
>      - 					   "max:%d", opts->processes);
>      -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt
>      - 		trace2_region_leave(tr2_category, tr2_label, NULL);
>      ++	if (do_trace2)
>      ++		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
>      ++					   "max:%d", opts->processes);
>      +
>      +-	pp_init(&pp, get_next_task, start_failure, task_finished);
>      ++	pp_init(&pp, opts);
>      + 	while (1) {
>      + 		for (i = 0;
>      + 		    i < spawn_cap && !pp.shutdown &&
>      +@@ run-command.c: void run_processes_parallel(size_t n,
>      + 		}
>      + 		if (!pp.nr_processes)
>      + 			break;
>      +-		if (ungroup) {
>      ++		if (opts->ungroup) {
>      + 			for (size_t i = 0; i < pp.max_processes; i++)
>      + 				pp.children[i].state = GIT_CP_WAIT_CLEANUP;
>      + 		} else {
>      +@@ run-command.c: void run_processes_parallel(size_t n,
>      + 	}
>      +
>      + 	pp_cleanup(&pp);
>      ++
>      ++	if (do_trace2)
>      ++		trace2_region_leave(tr2_category, tr2_label, NULL);
>        }
>        
>      --void run_processes_parallel(size_t processes,
>      --			    get_next_task_fn get_next_task,
>      --			    start_failure_fn start_failure,
>      --			    task_finished_fn task_finished,
>      --			    void *pp_cb)
>      --{
>      --	const struct run_process_parallel_opts opts = {
>      --		.processes = processes,
>      --		.ungroup = run_processes_parallel_ungroup,
>      --
>      --		.get_next_task = get_next_task,
>      --		.start_failure = start_failure,
>      --		.task_finished = task_finished,
>      --	};
>      --
>      --	run_processes_parallel_1(&opts, pp_cb);
>      --}
>      --
>      - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
>      +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,
>      ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,
>        				start_failure_fn start_failure,
>        				task_finished_fn task_finished, void *pp_cb,
>      -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task
>      - 	const struct run_process_parallel_opts opts = {
>      - 		.tr2_category = tr2_category,
>      - 		.tr2_label = tr2_label,
>      --
>      - 		.processes = processes,
>      --		.ungroup = run_processes_parallel_ungroup,
>      + 				const char *tr2_category, const char *tr2_label)
>      + {
>      +-	trace2_region_enter_printf(tr2_category, tr2_label, NULL, "max:%d", n);
>      ++	const struct run_process_parallel_opts opts = {
>      ++		.tr2_category = tr2_category,
>      ++		.tr2_label = tr2_label,
>      ++		.processes = processes,
>        
>      - 		.get_next_task = get_next_task,
>      - 		.start_failure = start_failure,
>      - 		.task_finished = task_finished,
>      - 	};
>      +-	run_processes_parallel(n, get_next_task, start_failure,
>      +-			       task_finished, pp_cb);
>      ++		.get_next_task = get_next_task,
>      ++		.start_failure = start_failure,
>      ++		.task_finished = task_finished,
>      ++	};
>        
>      --	run_processes_parallel_1(&opts, pp_cb);
>      +-	trace2_region_leave(tr2_category, tr2_label, NULL);
>       +	run_processes_parallel(&opts);
>        }
>        
>        int run_auto_maintenance(int quiet)
>       
>        ## run-command.h ##
>      -@@ run-command.h: struct run_process_parallel_opts
>      - 	 * NULL to omit any special handling.
>      - 	 */
>      - 	task_finished_fn task_finished;
>      +@@ run-command.h: typedef int (*task_finished_fn)(int result,
>      + 				void *pp_task_cb);
>      +
>      + /**
>      +- * Runs up to n processes at the same time. Whenever a process can be
>      +- * started, the callback get_next_task_fn is called to obtain the data
>      ++ * Option used by run_processes_parallel(), { 0 }-initialized means no
>      ++ * options.
>      ++ */
>      ++struct run_process_parallel_opts
>      ++{
>      ++	/**
>      ++	 * tr2_category & tr2_label: sets the trace2 category and label for
>      ++	 * logging. These must either be unset, or both of them must be set.
>      ++	 */
>      ++	const char *tr2_category;
>      ++	const char *tr2_label;
>      ++
>      ++	/**
>      ++	 * processes: see 'processes' in run_processes_parallel() below.
>      ++	 */
>      ++	size_t processes;
>      ++
>      ++	/**
>      ++	 * ungroup: see 'ungroup' in run_processes_parallel() below.
>      ++	 */
>      ++	unsigned int ungroup:1;
>      ++
>      ++	/**
>      ++	 * get_next_task: See get_next_task_fn() above. This must be
>      ++	 * specified.
>      ++	 */
>      ++	get_next_task_fn get_next_task;
>      ++
>      ++	/**
>      ++	 * start_failure: See start_failure_fn() above. This can be
>      ++	 * NULL to omit any special handling.
>      ++	 */
>      ++	start_failure_fn start_failure;
>      ++
>      ++	/**
>      ++	 * task_finished: See task_finished_fn() above. This can be
>      ++	 * NULL to omit any special handling.
>      ++	 */
>      ++	task_finished_fn task_finished;
>       +
>       +	/**
>       +	 * data: user data, will be passed as "pp_cb" to the callback
>       +	 * parameters.
>       +	 */
>       +	void *data;
>      - };
>      -
>      - /**
>      ++};
>      ++
>      ++/**
>       + * Options are passed via the "struct run_process_parallel_opts" above.
>       + *
>      -  * Runs N 'processes' at the same time. Whenever a process can be
>      -- * started, the callback get_next_task_fn is called to obtain the data
>      ++ * Runs N 'processes' at the same time. Whenever a process can be
>       + * started, the callback opts.get_next_task is called to obtain the data
>         * required to start another child process.
>         *
>         * The children started via this function run in parallel. Their output
>      -@@ run-command.h: struct run_process_parallel_opts
>      +  * (both stdout and stderr) is routed to stderr in a manner that output
>      +  * from different tasks does not interleave (but see "ungroup" below).
>      +  *
>      +- * start_failure_fn and task_finished_fn can be NULL to omit any
>      +- * special handling.
>      +- *
>      +  * If the "ungroup" option isn't specified, the API will set the
>      +  * "stdout_to_stderr" parameter in "struct child_process" and provide
>      +  * the callbacks with a "struct strbuf *out" parameter to write output
>      +@@ run-command.h: typedef int (*task_finished_fn)(int result,
>         * NULL "struct strbuf *out" parameter, and are responsible for
>         * emitting their own output, including dealing with any race
>         * conditions due to writing in parallel to stdout and stderr.
>      @@ run-command.h: struct run_process_parallel_opts
>       - * API reads that setting.
>         */
>       -extern int run_processes_parallel_ungroup;
>      --void run_processes_parallel(size_t processes,
>      +-void run_processes_parallel(size_t n,
>       -			    get_next_task_fn,
>       -			    start_failure_fn,
>       -			    task_finished_fn,
>       -			    void *pp_cb);
>      +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,
>      +-				task_finished_fn, void *pp_cb,
>       +void run_processes_parallel(const struct run_process_parallel_opts *opts);
>      - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
>      - 				start_failure_fn, task_finished_fn, void *pp_cb,
>      ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,
>      ++				start_failure_fn, task_finished_fn, void *pp_cb,
>        				const char *tr2_category, const char *tr2_label);
>      +
>      + /**
>       
>        ## t/helper/test-run-command.c ##
>       @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {
> 15:  391d1d99d91 =  8:  c19c60b2e95 run-command API: move *_tr2() users to "run_processes_parallel()"
> 16:  acac50cc1a5 =  9:  99a4f4f6b9c run-command.c: make "struct parallel_processes" const if possible
> 17:  fdd64236985 = 10:  bf67e24bcc5 run-command.c: don't copy *_fn to "struct parallel_processes"
> 18:  17f34d81ecd = 11:  06de42adc2e run-command.c: don't copy "ungroup" to "struct parallel_processes"
> 19:  9cbee2dfe76 = 12:  3081dfc49d1 run-command.c: don't copy "data" to "struct parallel_processes"
> 20:  2dabed9e155 ! 13:  b7c10f6a23f run-command.c: use "opts->processes", not "pp->max_processes"
>      @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,
>        
>       @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>        	while (1) {
>      - 		for (int i = 0;
>      + 		for (i = 0;
>        		    i < spawn_cap && !pp.shutdown &&
>       -		    pp.nr_processes < pp.max_processes;
>       +		    pp.nr_processes < opts->processes;
> 21:  c1a286a8ebb ! 14:  4856d6a4674 run-command.c: pass "opts" further down, and use "opts->processes"
>      @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,
>       +			     const struct run_process_parallel_opts *opts,
>       +			     int output_timeout)
>        {
>      --	while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {
>      -+	while (poll(pp->pfd, opts->processes, output_timeout) < 0) {
>      + 	int i;
>      +
>      +-	while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
>      ++	while ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {
>        		if (errno == EINTR)
>        			continue;
>       -		pp_cleanup(pp);
> 22:  541f41566e7 ! 15:  39a20be0cbb run-command.c: remove "pp->max_processes", add "const" to signal() handler
>      @@ Metadata
>       Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>       
>        ## Commit message ##
>      -    run-command.c: remove "pp->max_processes", add "const" to signal() handler
>      +    run-command.c: remove "max_processes", add "const" to signal() handler
>       
>           As with the *_fn members removed in a preceding commit, let's not copy
>           the "processes" member of the "struct run_process_parallel_opts" over
>      @@ run-command.c: static void pp_init(struct parallel_processes *pp,
>       -	pp_for_signal = pp;
>       +	pp_sig->pp = pp;
>       +	pp_sig->opts = opts;
>      ++	pp_for_signal = pp_sig;
>        	sigchain_push_common(handle_children_on_signal);
>        }
>        
>       @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>      - 	int code;
>      + 	int i, code;
>        	int output_timeout = 100;
>        	int spawn_cap = 4;
>       +	struct parallel_processes_for_signal pp_sig;
>      @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt
>       -	pp_init(&pp, opts);
>       +	pp_init(&pp, opts, &pp_sig);
>        	while (1) {
>      - 		for (int i = 0;
>      + 		for (i = 0;
>        		    i < spawn_cap && !pp.shutdown &&
>       @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)
>        				continue;

Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct

[Ævar Arnfjörð Bjarmason]

On Fri, Oct 14 2022, Phillip Wood wrote:

> Hi Ævar

Hi, thanks for taking a look again.

> On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:
>> For a general overview see the v2 CL:
>> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/
>> Changes since v2:
>>   * Ejected various not-to-the-point of converting to the "opts
>> struct"
>>     per feedback about attempting to make this leaner.
>
> We're back to the same number of patches as v1 but you've removed
> three test cleanups and squashed three patches together which means
> there are five new patches in this version - what are they doing?

There's nothing *new* in this version, but per the range-diff of v2 some
of the new ones in that version were kept here. I think that's what
you're taling about.

Mainly it's the 'Only the "ungroup" was here in v1[...]' part of the v2
CL:

1. https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/

>>   * I kept the size_t change & the online_cpus() fallback, and updated
>>     a commit message for the latter. For "int" v.s. "size_t" once we're
>>     not handling "-1" to mean "use the default" it makes sense to be
>>     unsigned, and if we're re-doing that at this point we'd need
>>     rewrites for "!processes" assumptions.
>
> I left some comments about this, I think the size_t change is taking
> us in the wrong direction as it is introducing a number of new
> implicit signed->unsigned conversions. I'm still not sure why you need
> the online_cups() changes c.f. 
> https://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com/
> which has never had a reply

Sorry about the non-reply there, between that & later discussion I tried
to address that in the "Keeping this default behavior just for[...]"
commit message update in this v3 (see range-diff).

But no, those changes are not strictly needed. But it's a trade-off I
decided to take.

In the v1 (linked above) you pointed out that we could simply copy this
field to the "struct pp" (a shorthand for "struct parallel_processes", I
assume).

That's true, but for maintaining & understanding this API I think it's
much easier to reason about it when all of our user options are "const",
and we don't second guess any of those, and the "struct
parallel_processes"

For the v1 I can see what that was easy to miss, as we still kept the
copy of the number of processes in the "struct parallel_processes". In
the v2 and this v3 we get to the point where we can remove that, and
"ungroup", the copies of the callbacks etc.

So, leaving out the "provide a default" seemed worth it in the end, it's
just 4 additional lines in the callers per the 05/15 (most of them had
those already).

You also had a related concern in 04/05 (which I'm taking the liberty of
replying ot here):

	https://lore.kernel.org/git/a7463bc5-9a92-8f0f-c0ee-e72fbbeedc09@dunelm.org.uk/

So, first I disagree with it "going in the wrong direction". We've been
converting more things to size_t. For e.g. an "int nr_processes" we can
expect that we'll want to e.g. have a corresponding "struct string_list"
whose "nr" is a "size_t" (or similar aggregate types).

By mixing the two we're mixing types forever with associated warnings (&
suppressions). We've been changing these sort of things to a "size_t"
elsewhere, e.g. dd38e9e510c (trace2: use size_t alloc,nr_open_regions in
tr2tls_thread_ctx, 2022-10-12) is one recent example..

But yes, we do incur warnings under DEVOPTS=extra-all now because things
outside of the API are still "int" in some cases, just as we do with
e.g. "struct string_list" and its users.

As to your:

	Before this patch we could have caught a negative value of n
	with an assertion. After this patch a negative value passed will
	become a large positive value (you have audited the current
	callers but that does not protect us from future mistakes).

I that's a good point. In this case I thought the likelihood that anyone
would accidentally pass ".processes = -1" or whatever wasn't worth
worrying about. If you think it's worth worrying about I think that
concern would be easily addressed e.g. with:

	if (n == SIZE_MAX)
        	BUG("overflow?");

Or whatever.

A much likelier edge case IMO is that you'd have some pattern where you
init a "processes" with "{ 0 }" or whatever, so it's zero, and then we'd
interpret that zero as online_cpus(), which e.g. for the in-flight
paralellizing of certain "git submodule" operations is probably too high
a number.

Since we don't interpret !n or n < 1 as online_cpus() anymore we can
BUG() out on it, which I think is an improvement. The 4 lines of
additions to the relevant callers to call online_cpus() themselves are
better than having those or a new caller potentially have this DWYMery
from the low-level API.

>>   * Squashed the multi-step introduction of the "opts" struct, per
>>     Phillip's suggestion.
> That's most welcome, thanks

Glad you like it!

Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> You also had a related concern in 04/05 (which I'm taking the liberty of
> replying ot here):
>
> 	https://lore.kernel.org/git/a7463bc5-9a92-8f0f-c0ee-e72fbbeedc09@dunelm.org.uk/
>
> So, first I disagree with it "going in the wrong direction". We've been
> converting more things to size_t. For e.g. an "int nr_processes" we can
> expect that we'll want to e.g. have a corresponding "struct string_list"
> whose "nr" is a "size_t" (or similar aggregate types).

I do not quite see why that is relevant.  We may create list of
textual descriptions from list of processes, so we expect to be able
to loop "for (int i = 0; i < nr_processes; i++)" and access i-th
element of the corresponding string_list.  As long as int is
narrower than size_t (and it is wide enough to count the processes
we are going to ever spawn) there is no issue, no?  Also using
signed type is so much more convenient to signal an error (imagine
asking for a process's i that is between 0..nr_processes with some
other key in your database, and having to answer "no such process
known to me").

> By mixing the two we're mixing types forever with associated warnings (&
> suppressions).

That is a complaint about shortcomings of the tool that gives
irrelevant warning, no?  It is not a good reason to make the code
worse.  Compilers, editors, and linters are to serve the code, not
the other way around.