Re: [PATCHv19 00/11] Expose submodule parallelism to the user

[Stefan Beller <sbeller@google.com>]

On Mon, Feb 29, 2016 at 12:48 PM, Johannes Sixt <j6t@kdbg.org> wrote:
> Hi folks,
>
> we have a major breakage in the parallel tasks infrastructure, and I'm
> afraid it is already in master.
>
> Instrument the code in sb/submodule-parallel-update like this and enjoy
> the fireworks of './t7400-submodule-basic.sh -v -i -x --debug':
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 0322282..482c7f6 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -690,8 +690,9 @@ cmd_update()
>                 cmd_init "--" "$@" || return
>         fi
>
> +       set -x
>         {
> -       git submodule--helper update-clone ${GIT_QUIET:+--quiet} \
> +       valgrind git submodule--helper update-clone ${GIT_QUIET:+--quiet} \
>                 ${wt_prefix:+--prefix "$wt_prefix"} \
>                 ${prefix:+--recursive-prefix "$prefix"} \
>                 ${update:+--update "$update"} \
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index 5572327..717e491 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -337,6 +337,7 @@ test_expect_success 'update should fail when path is used by a file' '
>
>         echo "hello" >init &&
>         test_must_fail git submodule update &&
> +       false &&
>
>         test_cmp expect init
>  '
>
> The culprit seems to be default_task_finished(), which accesses argv[]
> of the struct child_process after finish_command has released it,
> provided the child exited with an error, for example:
>
> ==3395== Invalid read of size 8
> ==3395==    at 0x54F991: default_task_finished (run-command.c:932)
> ==3395==    by 0x49158F: update_clone_task_finished (submodule--helper.c:421)
> ==3395==    by 0x5504A2: pp_collect_finished (run-command.c:1122)
> ==3395==    by 0x5507C7: run_processes_parallel (run-command.c:1194)
> ==3395==    by 0x4918EB: update_clone (submodule--helper.c:483)
> ==3395==    by 0x4919D8: cmd_submodule__helper (submodule--helper.c:527)
> ==3395==    by 0x405CBE: run_builtin (git.c:353)
> ==3395==    by 0x405EAA: handle_builtin (git.c:540)
> ==3395==    by 0x405FCC: run_argv (git.c:594)
> ==3395==    by 0x4061BF: main (git.c:701)
> ==3395==  Address 0x5e49370 is 0 bytes inside a block of size 192 free'd
> ==3395==    at 0x4C2A37C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==3395==    by 0x4A26EE: argv_array_clear (argv-array.c:73)
> ==3395==    by 0x54DFC4: child_process_clear (run-command.c:18)
> ==3395==    by 0x54EFA7: finish_command (run-command.c:539)
> ==3395==    by 0x550413: pp_collect_finished (run-command.c:1120)
> ==3395==    by 0x5507C7: run_processes_parallel (run-command.c:1194)
> ==3395==    by 0x4918EB: update_clone (submodule--helper.c:483)
> ==3395==    by 0x4919D8: cmd_submodule__helper (submodule--helper.c:527)
> ==3395==    by 0x405CBE: run_builtin (git.c:353)
> ==3395==    by 0x405EAA: handle_builtin (git.c:540)
> ==3395==    by 0x405FCC: run_argv (git.c:594)
> ==3395==    by 0x4061BF: main (git.c:701)
>
> I haven't thought about a solution, yet. Perhaps you have ideas.
>
> -- Hannes
>

What about unfolding finish_command like so:

diff --git a/run-command.c b/run-command.c
index 863dad5..659abd9 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1115,11 +1115,13 @@ static int pp_collect_finished(struct
parallel_processes *pp)
                if (i == pp->max_processes)
                        break;

-               code = finish_command(&pp->children[i].process);
+               code = wait_or_whine(pp->children[i].process.pid,
+                                    pp->children[i].process.argv[0], 0);

                code = pp->task_finished(code, &pp->children[i].process,
                                         &pp->children[i].err, pp->data,
                                         &pp->children[i].data);
+               child_process_clear(&pp->children[i].process);

                if (code)
                        result = code;