From: Junio C Hamano <gitster@pobox.com>
To: Calvin Wan <calvinwan@google.com>
Cc: git@vger.kernel.org, emilyshaffer@google.com
Subject: Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel
Date: Mon, 26 Sep 2022 21:45:31 -0700 [thread overview]
Message-ID: <xmqqpmfh8lxw.fsf@gitster.g> (raw)
In-Reply-To: <CAFySSZA=tThoHdTY7+bMStvC=xeeyMiv4aVDYt-eNW2mQE10qg@mail.gmail.com> (Calvin Wan's message of "Mon, 26 Sep 2022 10:31:05 -0700")
Calvin Wan <calvinwan@google.com> writes:
> You are correct that storing unbounded output doesn't seem like a good
> idea. One idea I have is to parse output during the periodic collection rather
> than waiting till the end. The other idea I had was to add another
> "git status --porcelain" option that would only output the necessary
> pieces of information so we wouldn't have to bother with worrying about
> unbounded output.
>
> Any other thoughts as to how I can workaround this?
I wonder if you can arrange not to let them make unbounded progress?
In order to run diff-files with path A B C D E ... where B and D are
submodules and others are not submodules, you do not have to run and
finish comparison for B and D before you can do the comparison for
other paths, in order to preserve the proper output order. You can
start child task for B and D and arrange so that they will run for
any other submodules, and then you
- run comparison for A. The child task for B and D may be running
and starting to talk back to you, in which case their write may
get stuck waiting for you to read from them, but that is OK, as
you will read from them shortly.
- wait for the child task for B. This is done by reading from the
pipe connected to it and waiting for its death synchronously.
The child task for D is still running and may be making progress,
but you are not obligated to read its output to the end. You can
postpone reading to conserve memory and it will fill the pipe and
stall automatically. Then accept the result for B.
- run comparison for C.
- wait for the child task for D. You may have some data read
already while dealing with B, but you may still have some reading
to do at this point. Let it finish synchronously.
- run comparison for E.
etc., perhaps?
next prev parent reply other threads:[~2022-09-27 4:45 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-22 23:29 [PATCH 0/4] submodule: parallelize status Calvin Wan
2022-09-22 23:29 ` [PATCH 1/4] run-command: add pipe_output to run_processes_parallel Calvin Wan
2022-09-23 7:52 ` Ævar Arnfjörð Bjarmason
2022-09-26 16:59 ` Calvin Wan
2022-09-27 10:52 ` Ævar Arnfjörð Bjarmason
2022-09-23 18:58 ` Junio C Hamano
2022-09-26 17:31 ` Calvin Wan
2022-09-27 4:45 ` Junio C Hamano [this message]
2022-09-27 18:10 ` Calvin Wan
2022-09-27 21:40 ` Junio C Hamano
2022-09-27 9:05 ` Ævar Arnfjörð Bjarmason
2022-09-27 17:55 ` Calvin Wan
2022-09-27 19:34 ` Ævar Arnfjörð Bjarmason
2022-09-27 20:45 ` Calvin Wan
2022-09-28 5:40 ` Ævar Arnfjörð Bjarmason
2022-09-29 20:52 ` Calvin Wan
2022-09-22 23:29 ` [PATCH 2/4] submodule: move status parsing into function Calvin Wan
2022-09-22 23:29 ` [PATCH 3/4] diff-lib: refactor functions Calvin Wan
2022-09-23 20:36 ` Junio C Hamano
2022-09-26 17:35 ` Calvin Wan
2022-09-22 23:29 ` [PATCH 4/4] diff-lib: parallelize run_diff_files for submodules Calvin Wan
2022-09-23 8:06 ` Ævar Arnfjörð Bjarmason
2022-09-24 20:17 ` Junio C Hamano
2022-09-26 17:50 ` Calvin Wan
2022-09-23 21:44 ` Junio C Hamano
2022-09-26 19:12 ` Calvin Wan
2022-09-25 13:59 ` Phillip Wood
2022-09-26 17:11 ` Junio C Hamano
2022-09-26 19:22 ` Calvin Wan
2022-09-27 18:40 ` Emily Shaffer
2022-09-23 22:56 ` [PATCH 0/4] submodule: parallelize status Junio C Hamano
2022-09-26 16:33 ` Calvin Wan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqpmfh8lxw.fsf@gitster.g \
--to=gitster@pobox.com \
--cc=calvinwan@google.com \
--cc=emilyshaffer@google.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).