From: Junio C Hamano <gitster@pobox.com>
To: Stefan Beller <sbeller@google.com>
Cc: git@vger.kernel.org, ramsay@ramsayjones.plus.com,
jacob.keller@gmail.com, peff@peff.net, jrnieder@gmail.com,
johannes.schindelin@gmail.com, Jens.Lehmann@web.de,
vlovich@gmail.com, ericsunshine@gmail.com
Subject: Re: [PATCH 1/2] SQUASH???
Date: Thu, 24 Sep 2015 18:09:44 -0700 [thread overview]
Message-ID: <xmqqwpvfticn.fsf@gitster.mtv.corp.google.com> (raw)
In-Reply-To: <xmqqa8sbuxu0.fsf@gitster.mtv.corp.google.com> (Junio C. Hamano's message of "Thu, 24 Sep 2015 17:49:59 -0700")
Junio C Hamano <gitster@pobox.com> writes:
> I think that is a sensible change. Don't we want the same for the
> other failure handler, though. Capture any message from it and
> append it to the output of the process that just finished, or
> something?
Ah, that is already done. Scratch that "don't we want" part.
>
> By the way, I understand that these two are solely for early review
> and I'll be getting them as either new patches or part of updated
> patches in the next reroll (i.e. you are not expecting me to split
> these apart and do "rebase -i" for you to the last-posted version).
> Asking only to make sure we are on the same wavelength.
>
> Thanks.
>
>>
>> Signed-off-by: Junio C Hamano <gitster@pobox.com>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>> run-command.c | 43 ++++++++++++++++++++++++++++++-------------
>> run-command.h | 1 +
>> 2 files changed, 31 insertions(+), 13 deletions(-)
>>
>> diff --git a/run-command.c b/run-command.c
>> index 494e1f8..0d22291 100644
>> --- a/run-command.c
>> +++ b/run-command.c
>> @@ -907,6 +907,7 @@ void default_start_failure(void *data,
>>
>> void default_return_value(void *data,
>> struct child_process *cp,
>> + struct strbuf *err,
>> int result)
>> {
>> int i;
>> @@ -977,7 +978,7 @@ static void set_nonblocking(int fd)
>> "output will be degraded");
>> }
>>
>> -/* returns 1 if a process was started, 0 otherwise */
>> +/* return 0 if get_next_task() ran out of things to do, non-zero otherwise */
>> static int pp_start_one(struct parallel_processes *pp)
>> {
>> int i;
>> @@ -991,26 +992,30 @@ static int pp_start_one(struct parallel_processes *pp)
>> if (!pp->get_next_task(pp->data,
>> &pp->children[i].process,
>> &pp->children[i].err))
>> - return 1;
>> + return 0;
>>
>> - if (start_command(&pp->children[i].process))
>> + if (start_command(&pp->children[i].process)) {
>> pp->start_failure(pp->data,
>> &pp->children[i].process,
>> &pp->children[i].err);
>> + strbuf_addbuf(&pp->buffered_output, &pp->children[i].err);
>> + strbuf_reset(&pp->children[i].err);
>> + return -1;
>> + }
>>
>> set_nonblocking(pp->children[i].process.err);
>>
>> pp->nr_processes++;
>> pp->children[i].in_use = 1;
>> pp->pfd[i].fd = pp->children[i].process.err;
>> - return 0;
>> + return 1;
>> }
>>
>> -static void pp_buffer_stderr(struct parallel_processes *pp)
>> +static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)
>> {
>> int i;
>>
>> - while ((i = poll(pp->pfd, pp->max_processes, 100)) < 0) {
>> + while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {
>> if (errno == EINTR)
>> continue;
>> pp_cleanup(pp);
>> @@ -1069,7 +1074,8 @@ static void pp_collect_finished(struct parallel_processes *pp)
>> error("waitpid is confused (%s)",
>> pp->children[i].process.argv[0]);
>>
>> - pp->return_value(pp->data, &pp->children[i].process, code);
>> + pp->return_value(pp->data, &pp->children[i].process,
>> + &pp->children[i].err, code);
>>
>> argv_array_clear(&pp->children[i].process.args);
>> argv_array_clear(&pp->children[i].process.env_array);
>> @@ -1111,15 +1117,26 @@ int run_processes_parallel(int n, void *data,
>> return_value_fn return_value)
>> {
>> struct parallel_processes pp;
>> - pp_init(&pp, n, data, get_next_task, start_failure, return_value);
>>
>> + pp_init(&pp, n, data, get_next_task, start_failure, return_value);
>> while (1) {
>> - while (pp.nr_processes < pp.max_processes &&
>> - !pp_start_one(&pp))
>> - ; /* nothing */
>> - if (!pp.nr_processes)
>> + int no_more_task, cnt;
>> + int output_timeout = 100;
>> + int spawn_cap = 4;
>> +
>> + for (cnt = spawn_cap, no_more_task = 0;
>> + cnt && pp.nr_processes < pp.max_processes;
>> + cnt--) {
>> + if (!pp_start_one(&pp)) {
>> + no_more_task = 1;
>> + break;
>> + }
>> + }
>> +
>> + if (no_more_task && !pp.nr_processes)
>> break;
>> - pp_buffer_stderr(&pp);
>> + pp_buffer_stderr(&pp, output_timeout);
>> +
>> pp_output(&pp);
>> pp_collect_finished(&pp);
>> }
>> diff --git a/run-command.h b/run-command.h
>> index 3807fd1..f7035cb 100644
>> --- a/run-command.h
>> +++ b/run-command.h
>> @@ -138,6 +138,7 @@ typedef void (*start_failure_fn)(void *data,
>>
>> typedef void (*return_value_fn)(void *data,
>> struct child_process *cp,
>> + struct strbuf *err,
>> int result);
>>
>> /**
next prev parent reply other threads:[~2015-09-25 1:09 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-23 1:45 [PATCHv4 00/14] fetch submodules in parallel and a preview on parallel "submodule update" Stefan Beller
2015-09-23 1:45 ` [PATCHv4 01/14] submodule: Send "Fetching submodule <foo>" to standard error Stefan Beller
2015-09-23 1:45 ` [PATCHv4 02/14] xread: poll on non blocking fds Stefan Beller
2015-09-23 1:45 ` [PATCHv4 03/14] xread_nonblock: add functionality to read from fds without blocking Stefan Beller
2015-09-23 1:45 ` [PATCHv4 04/14] strbuf: add strbuf_read_once to read " Stefan Beller
2015-09-23 1:45 ` [PATCHv4 05/14] run-command: factor out return value computation Stefan Beller
2015-09-23 1:45 ` [PATCHv4 06/14] run-command: add an asynchronous parallel child processor Stefan Beller
2015-09-23 6:29 ` Junio C Hamano
2015-09-23 17:53 ` Stefan Beller
2015-09-23 18:04 ` Junio C Hamano
2015-09-23 19:34 ` Junio C Hamano
2015-09-23 19:39 ` Stefan Beller
2015-09-23 19:47 ` Junio C Hamano
2015-09-23 6:47 ` Junio C Hamano
2015-09-23 14:59 ` Junio C Hamano
2015-09-23 17:54 ` Junio C Hamano
2015-09-23 23:41 ` [PATCHv5] Another squash on " Stefan Beller
2015-09-24 2:17 ` Junio C Hamano
2015-09-24 21:13 ` [PATCH 0/2] " Stefan Beller
2015-09-24 21:13 ` [PATCH 2/2] SQUASH for "fetch_populated_submodules: use new parallel job processing" Stefan Beller
2015-09-24 21:13 ` [PATCH 1/2] SQUASH??? Stefan Beller
2015-09-25 0:49 ` Junio C Hamano
2015-09-25 1:09 ` Junio C Hamano [this message]
2015-09-25 17:52 ` Stefan Beller
2015-09-25 17:56 ` Junio C Hamano
2015-09-25 1:08 ` [PATCH 0/2] Another squash on run-command: add an asynchronous parallel child processor Junio C Hamano
2015-09-25 18:56 ` Stefan Beller
2015-09-25 19:04 ` Junio C Hamano
2015-09-25 19:19 ` Stefan Beller
2015-09-25 19:32 ` Junio C Hamano
2015-09-23 1:45 ` [PATCHv4 07/14] fetch_populated_submodules: use new parallel job processing Stefan Beller
2015-09-23 1:45 ` [PATCHv4 08/14] submodules: allow parallel fetching, add tests and documentation Stefan Beller
2015-09-23 1:45 ` [PATCHv4 09/14] submodule-config: Untangle logic in parse_config Stefan Beller
2015-09-23 1:45 ` [PATCHv4 10/14] submodule config: keep update strategy around Stefan Beller
2015-09-23 1:45 ` [PATCHv4 11/14] git submodule update: cmd_update_recursive Stefan Beller
2015-09-23 1:45 ` [PATCHv4 12/14] git submodule update: cmd_update_clone Stefan Beller
2015-09-23 20:13 ` Junio C Hamano
2015-09-23 1:45 ` [PATCHv4 13/14] git submodule update: cmd_update_fetch Stefan Beller
2015-09-23 1:45 ` [PATCHv4 14/14] Rewrite submodule update in C Stefan Beller
-- strict thread matches above, loose matches on Subject: below --
2016-09-13 16:31 [PATCH v2] ls-files: adding support for submodules Junio C Hamano
2016-09-15 20:51 ` Junio C Hamano
2016-09-15 20:51 ` [PATCH 1/2] SQUASH??? Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqwpvfticn.fsf@gitster.mtv.corp.google.com \
--to=gitster@pobox.com \
--cc=Jens.Lehmann@web.de \
--cc=ericsunshine@gmail.com \
--cc=git@vger.kernel.org \
--cc=jacob.keller@gmail.com \
--cc=johannes.schindelin@gmail.com \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=ramsay@ramsayjones.plus.com \
--cc=sbeller@google.com \
--cc=vlovich@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).