git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: Calvin Wan <calvinwan@google.com>
Cc: git@vger.kernel.org, emilyshaffer@google.com
Subject: Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel
Date: Mon, 26 Sep 2022 21:45:31 -0700	[thread overview]
Message-ID: <xmqqpmfh8lxw.fsf@gitster.g> (raw)
In-Reply-To: <CAFySSZA=tThoHdTY7+bMStvC=xeeyMiv4aVDYt-eNW2mQE10qg@mail.gmail.com> (Calvin Wan's message of "Mon, 26 Sep 2022 10:31:05 -0700")

Calvin Wan <calvinwan@google.com> writes:

> You are correct that storing unbounded output doesn't seem like a good
> idea. One idea I have is to parse output during the periodic collection rather
> than waiting till the end. The other idea I had was to add another
> "git status --porcelain" option that would only output the necessary
> pieces of information so we wouldn't have to bother with worrying about
> unbounded output.
>
> Any other thoughts as to how I can workaround this?

I wonder if you can arrange not to let them make unbounded progress?

In order to run diff-files with path A B C D E ... where B and D are
submodules and others are not submodules, you do not have to run and
finish comparison for B and D before you can do the comparison for
other paths, in order to preserve the proper output order.  You can
start child task for B and D and arrange so that they will run for
any other submodules, and then you

 - run comparison for A.  The child task for B and D may be running
   and starting to talk back to you, in which case their write may
   get stuck waiting for you to read from them, but that is OK, as
   you will read from them shortly.

 - wait for the child task for B.  This is done by reading from the
   pipe connected to it and waiting for its death synchronously.
   The child task for D is still running and may be making progress,
   but you are not obligated to read its output to the end.  You can
   postpone reading to conserve memory and it will fill the pipe and
   stall automatically.  Then accept the result for B.

 - run comparison for C.

 - wait for the child task for D.  You may have some data read
   already while dealing with B, but you may still have some reading
   to do at this point.  Let it finish synchronously.

 - run comparison for E.

etc., perhaps?


  reply	other threads:[~2022-09-27  4:45 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-22 23:29 [PATCH 0/4] submodule: parallelize status Calvin Wan
2022-09-22 23:29 ` [PATCH 1/4] run-command: add pipe_output to run_processes_parallel Calvin Wan
2022-09-23  7:52   ` Ævar Arnfjörð Bjarmason
2022-09-26 16:59     ` Calvin Wan
2022-09-27 10:52       ` Ævar Arnfjörð Bjarmason
2022-09-23 18:58   ` Junio C Hamano
2022-09-26 17:31     ` Calvin Wan
2022-09-27  4:45       ` Junio C Hamano [this message]
2022-09-27 18:10         ` Calvin Wan
2022-09-27 21:40           ` Junio C Hamano
2022-09-27  9:05       ` Ævar Arnfjörð Bjarmason
2022-09-27 17:55         ` Calvin Wan
2022-09-27 19:34           ` Ævar Arnfjörð Bjarmason
2022-09-27 20:45             ` Calvin Wan
2022-09-28  5:40               ` Ævar Arnfjörð Bjarmason
2022-09-29 20:52                 ` Calvin Wan
2022-09-22 23:29 ` [PATCH 2/4] submodule: move status parsing into function Calvin Wan
2022-09-22 23:29 ` [PATCH 3/4] diff-lib: refactor functions Calvin Wan
2022-09-23 20:36   ` Junio C Hamano
2022-09-26 17:35     ` Calvin Wan
2022-09-22 23:29 ` [PATCH 4/4] diff-lib: parallelize run_diff_files for submodules Calvin Wan
2022-09-23  8:06   ` Ævar Arnfjörð Bjarmason
2022-09-24 20:17     ` Junio C Hamano
2022-09-26 17:50     ` Calvin Wan
2022-09-23 21:44   ` Junio C Hamano
2022-09-26 19:12     ` Calvin Wan
2022-09-25 13:59   ` Phillip Wood
2022-09-26 17:11     ` Junio C Hamano
2022-09-26 19:22     ` Calvin Wan
2022-09-27 18:40   ` Emily Shaffer
2022-09-23 22:56 ` [PATCH 0/4] submodule: parallelize status Junio C Hamano
2022-09-26 16:33   ` Calvin Wan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqqpmfh8lxw.fsf@gitster.g \
    --to=gitster@pobox.com \
    --cc=calvinwan@google.com \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).