Re: [RFC/PATCH 0/4] parallel fetch for submodules

[Stefan Beller <sbeller@google.com>]

On Thu, Aug 6, 2015 at 1:08 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 06.08.2015 um 19:35 schrieb Stefan Beller:
>>
>> When I was looking at the branches of Jens for work done on submodules
>> not yet upstream I found a commit "WIP threaded submodule fetching[1],
>> and I was side tracked wanting to present a different approach to that.
>
>
> Cool. I didn't follow that route further than building a proof of
> concept because I ran into a nasty DNS-timeout on my router at home
> and at work we host all repos on a not-so-beefy server making parallel
> fetch rather pointless. But I suspect this approach will bring down
> fetch times for some users.

The difference between your proof of concept and mine is to have the number
of threads easier configurable. (Think of "git fetch --recurse-submodules=yes
 -j4", similar to "make -j4" splitting work up to 4 different programs
at the same
time. And that thing would be possible to add in this series)

If you fetch lots of submodules, both the client and server load should come
in an ping-pong on-off pattern as the client waits for the server to
prepare stuff
and get it sent to it and then the client needs time to resolve deltas
and write to
disk. Depending on the duty cycle of each, a different number of
parallel threads
make sense (I would expected that they shift their phases against each other
by pure randomness, i.e. one thread is currently resolving deltas
while the other
thread is telling the server to get some work done, so both the client
and server
get utilized at the same time).

>
> Maybe we could also re-use parallel fetch for multiple upstreams in
> the superproject when doing a "git fetch --all" without too much
> extra work?

That's why I tried advertising RFC patch 2/4 as I believe it could be
easily reused for such things like "git fetch --all", but maybe other
people see problems with it (over/under engineered, wrong things
added, but other critical things missing) so I'd like to hear opinions
on that. :)