git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Calvin Wan <calvinwan@google.com>
To: Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org, emilyshaffer@google.com
Subject: Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel
Date: Tue, 27 Sep 2022 11:10:49 -0700	[thread overview]
Message-ID: <CAFySSZC1DMdnCpi9enDG0=NDb+OU8jufHsq4WkjoN3mp1SZT8w@mail.gmail.com> (raw)
In-Reply-To: <xmqqpmfh8lxw.fsf@gitster.g>

> I wonder if you can arrange not to let them make unbounded progress?
>
> In order to run diff-files with path A B C D E ... where B and D are
> submodules and others are not submodules, you do not have to run and
> finish comparison for B and D before you can do the comparison for
> other paths, in order to preserve the proper output order.  You can
> start child task for B and D and arrange so that they will run for
> any other submodules, and then you

There is no need to preserve proper output order, as the output is
sorted at the end.

>  - run comparison for A.  The child task for B and D may be running
>    and starting to talk back to you, in which case their write may
>    get stuck waiting for you to read from them, but that is OK, as
>    you will read from them shortly.
>
>  - wait for the child task for B.  This is done by reading from the
>    pipe connected to it and waiting for its death synchronously.
>    The child task for D is still running and may be making progress,
>    but you are not obligated to read its output to the end.  You can
>    postpone reading to conserve memory and it will fill the pipe and
>    stall automatically.  Then accept the result for B.
>
>  - run comparison for C.
>
>  - wait for the child task for D.  You may have some data read
>    already while dealing with B, but you may still have some reading
>    to do at this point.  Let it finish synchronously.
>
>  - run comparison for E.
>
> etc., perhaps?

I understand the idea you're suggesting and I think it would work,
but I'm worried about the overhead this would create. I would rather
implement a separate "git status --porcelain" output for just this
submodule case so 1. we would not have to worry about unbounded
output and 2. both the output parsing and the command could be
optimized. In parse_status_porcelain, the function returns early if
the submodule is found to have untracked and modified changes.
This early termination can happen on the command side, rather
than the parsing side.

  reply	other threads:[~2022-09-27 18:11 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-22 23:29 [PATCH 0/4] submodule: parallelize status Calvin Wan
2022-09-22 23:29 ` [PATCH 1/4] run-command: add pipe_output to run_processes_parallel Calvin Wan
2022-09-23  7:52   ` Ævar Arnfjörð Bjarmason
2022-09-26 16:59     ` Calvin Wan
2022-09-27 10:52       ` Ævar Arnfjörð Bjarmason
2022-09-23 18:58   ` Junio C Hamano
2022-09-26 17:31     ` Calvin Wan
2022-09-27  4:45       ` Junio C Hamano
2022-09-27 18:10         ` Calvin Wan [this message]
2022-09-27 21:40           ` Junio C Hamano
2022-09-27  9:05       ` Ævar Arnfjörð Bjarmason
2022-09-27 17:55         ` Calvin Wan
2022-09-27 19:34           ` Ævar Arnfjörð Bjarmason
2022-09-27 20:45             ` Calvin Wan
2022-09-28  5:40               ` Ævar Arnfjörð Bjarmason
2022-09-29 20:52                 ` Calvin Wan
2022-09-22 23:29 ` [PATCH 2/4] submodule: move status parsing into function Calvin Wan
2022-09-22 23:29 ` [PATCH 3/4] diff-lib: refactor functions Calvin Wan
2022-09-23 20:36   ` Junio C Hamano
2022-09-26 17:35     ` Calvin Wan
2022-09-22 23:29 ` [PATCH 4/4] diff-lib: parallelize run_diff_files for submodules Calvin Wan
2022-09-23  8:06   ` Ævar Arnfjörð Bjarmason
2022-09-24 20:17     ` Junio C Hamano
2022-09-26 17:50     ` Calvin Wan
2022-09-23 21:44   ` Junio C Hamano
2022-09-26 19:12     ` Calvin Wan
2022-09-25 13:59   ` Phillip Wood
2022-09-26 17:11     ` Junio C Hamano
2022-09-26 19:22     ` Calvin Wan
2022-09-27 18:40   ` Emily Shaffer
2022-09-23 22:56 ` [PATCH 0/4] submodule: parallelize status Junio C Hamano
2022-09-26 16:33   ` Calvin Wan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAFySSZC1DMdnCpi9enDG0=NDb+OU8jufHsq4WkjoN3mp1SZT8w@mail.gmail.com' \
    --to=calvinwan@google.com \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).