From: Junio C Hamano <gitster@pobox.com>
To: Jeff King <peff@peff.net>
Cc: Stefan Beller <sbeller@google.com>,
git@vger.kernel.org, jrnieder@gmail.com
Subject: Re: [PATCH 4/5] index-pack: Use the new worker pool
Date: Tue, 25 Aug 2015 13:41:27 -0700 [thread overview]
Message-ID: <xmqqfv37xfrc.fsf@gitster.dls.corp.google.com> (raw)
In-Reply-To: <20150825190329.GA11271@sigill.intra.peff.net> (Jeff King's message of "Tue, 25 Aug 2015 15:03:30 -0400")
Jeff King <peff@peff.net> writes:
> On Tue, Aug 25, 2015 at 10:28:25AM -0700, Stefan Beller wrote:
>
>> By treating each object as its own task the workflow is easier to follow
>> as the function used in the worker threads doesn't need any control logic
>> any more.
>
> Have you tried running t/perf/p5302 on this?
>
> I seem to get a pretty consistent 2%-ish slowdown, both against git.git
> and linux.git. That's not a lot, but I'm wondering if there is some
> low-hanging fruit in the locking, or in the pattern of work being
> dispatched. Or it may just be noise, but it seems fairly consistent.
The pattern of work dispatch hopefully is the same, no? add_task()
does the "append at the end" thing and next_task() picks from the
front of the queue. The original is "we have globally N things,
so far M things have been handled, and we want a new one, so we pick
the M+1th one and do it".
The amount of memory that is used to represent a single task may be
much larger than the original, with overhead coming from job_list
structure and the doubly-linked list. We may not be able to spin up
30 threads and throw a million tasks at them using this, because of
the overhead. It would be more suited to handle a pattern in which
an overlord actively creates new tasks while worker threads chew
them, using the add_task/dispatch as the medium for communication
between them.
next prev parent reply other threads:[~2015-08-25 20:41 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-08-25 17:28 [RFC PATCH 0/5] Demonstrate new parallel threading API Stefan Beller
2015-08-25 17:28 ` [PATCH 1/5] FIXUP submodule: implement `module_clone` as a builtin helper Stefan Beller
2015-08-25 17:28 ` [PATCH 2/5] thread-utils: add a threaded task queue Stefan Beller
2015-08-25 17:28 ` [PATCH 3/5] submodule: helper to run foreach in parallel Stefan Beller
2015-08-25 21:09 ` Junio C Hamano
2015-08-25 21:42 ` Stefan Beller
2015-08-25 22:23 ` Junio C Hamano
2015-08-25 22:44 ` Junio C Hamano
2015-08-26 17:06 ` Jeff King
2015-08-26 17:21 ` Stefan Beller
2015-08-25 17:28 ` [PATCH 4/5] index-pack: Use the new worker pool Stefan Beller
2015-08-25 19:03 ` Jeff King
2015-08-25 19:23 ` Stefan Beller
2015-08-25 20:41 ` Junio C Hamano [this message]
2015-08-25 20:59 ` Stefan Beller
2015-08-25 21:12 ` Junio C Hamano
2015-08-25 22:39 ` Stefan Beller
2015-08-25 22:50 ` Junio C Hamano
2015-08-25 17:28 ` [PATCH 5/5] pack-objects: Use " Stefan Beller
-- strict thread matches above, loose matches on Subject: below --
2015-08-27 0:52 [RFC PATCH 0/5] Progressing with `git submodule foreach_parallel` Stefan Beller
2015-08-27 0:52 ` [PATCH 4/5] index-pack: Use the new worker pool Stefan Beller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqfv37xfrc.fsf@gitster.dls.corp.google.com \
--to=gitster@pobox.com \
--cc=git@vger.kernel.org \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=sbeller@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).