Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel

[Calvin Wan <calvinwan@google.com>]

> I wonder if you can arrange not to let them make unbounded progress?
>
> In order to run diff-files with path A B C D E ... where B and D are
> submodules and others are not submodules, you do not have to run and
> finish comparison for B and D before you can do the comparison for
> other paths, in order to preserve the proper output order.  You can
> start child task for B and D and arrange so that they will run for
> any other submodules, and then you

There is no need to preserve proper output order, as the output is
sorted at the end.

>  - run comparison for A.  The child task for B and D may be running
>    and starting to talk back to you, in which case their write may
>    get stuck waiting for you to read from them, but that is OK, as
>    you will read from them shortly.
>
>  - wait for the child task for B.  This is done by reading from the
>    pipe connected to it and waiting for its death synchronously.
>    The child task for D is still running and may be making progress,
>    but you are not obligated to read its output to the end.  You can
>    postpone reading to conserve memory and it will fill the pipe and
>    stall automatically.  Then accept the result for B.
>
>  - run comparison for C.
>
>  - wait for the child task for D.  You may have some data read
>    already while dealing with B, but you may still have some reading
>    to do at this point.  Let it finish synchronously.
>
>  - run comparison for E.
>
> etc., perhaps?

I understand the idea you're suggesting and I think it would work,
but I'm worried about the overhead this would create. I would rather
implement a separate "git status --porcelain" output for just this
submodule case so 1. we would not have to worry about unbounded
output and 2. both the output parsing and the command could be
optimized. In parse_status_porcelain, the function returns early if
the submodule is found to have untracked and modified changes.
This early termination can happen on the command side, rather
than the parsing side.