Are concurrent git operations on the same repo safe?

Are concurrent git operations on the same repo safe?

[Ian Norton]

Hi all,

I'm generally used to the idea that if a thing is not marked as
"thread-safe" then it isn't thread safe, but I thought I'd ask anyway
to be sure.

Is it safe for me to do several operations with git concurrently on
the same local repo? Specifically I'm trying to speed up "git
submodule update" by doing several at the same time.  I've noticed
some odd side effects afterwards though when trying to commit changes
to my super project.

Apologies if this is answered elsewhere, my google-foo is weak today.

Many thanks

Ian

Re: Are concurrent git operations on the same repo safe?

[Duy Nguyen]

On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton <inorton@gmail.com> wrote:
> Hi all,
>
> I'm generally used to the idea that if a thing is not marked as
> "thread-safe" then it isn't thread safe, but I thought I'd ask anyway
> to be sure.
>
> Is it safe for me to do several operations with git concurrently on
> the same local repo?

Off the top of my head, object database access (e.g. things in
.git/objects), refs updates (.git/refs) and .git/index should handle
concurrent operations fine (or in the worst case you get "some
operation is ongoing, aborted" and you need to try again but not
corruption or anything else). I think we generally try to make it safe
concurrently.

> Specifically I'm trying to speed up "git
> submodule update" by doing several at the same time.  I've noticed
> some odd side effects afterwards though when trying to commit changes
> to my super project.

submodule is a hot area with lots of development lately I think,
perhaps you're seeing some bugs... CCing at least one submodule
person...

> Apologies if this is answered elsewhere, my google-foo is weak today.
>
> Many thanks
>
> Ian



-- 
Duy

Re: Are concurrent git operations on the same repo safe?

[Stefan Beller]

On Tue, Feb 6, 2018 at 2:16 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton <inorton@gmail.com> wrote:
>> Hi all,
>>
>> I'm generally used to the idea that if a thing is not marked as
>> "thread-safe" then it isn't thread safe, but I thought I'd ask anyway
>> to be sure.
>>
>> Is it safe for me to do several operations with git concurrently on
>> the same local repo?
>
> Off the top of my head, object database access (e.g. things in
> .git/objects), refs updates (.git/refs) and .git/index should handle
> concurrent operations fine (or in the worst case you get "some
> operation is ongoing, aborted" and you need to try again but not
> corruption or anything else). I think we generally try to make it safe
> concurrently.
>
>> Specifically I'm trying to speed up "git
>> submodule update" by doing several at the same time.  I've noticed
>> some odd side effects afterwards though when trying to commit changes
>> to my super project.
>
> submodule is a hot area with lots of development lately I think,
> perhaps you're seeing some bugs... CCing at least one submodule
> person...
>
>> Apologies if this is answered elsewhere, my google-foo is weak today.
>>
>> Many thanks
>>
>> Ian

"git submodule update" has the network part parallelized,
but not the local part IIRC. (That is cloning/fetching submodules
can be done with "-j <n>", but the local checkout is still serial for
UX purposes, i.e. it wants to stop at the first conflict and only
have one conflict at a time)

Which odd side effects do you see?
I'd be curious to see if that is a bug in the code or documentation.

Thanks,
Stefan

Re: Are concurrent git operations on the same repo safe?

[Jonathan Nieder]

Ian Norton wrote:

>                      Specifically I'm trying to speed up "git
> submodule update" by doing several at the same time.

Can you say more about this?  E.g. how can I reproduce your experience?
Is there a script I can run?

Thanks,
Jonathan

Re: Are concurrent git operations on the same repo safe?

[Ian Norton]

Sure, my office is still devoid of power, i'll have to get back to you
to be more precise but I was using a morally equivalent process to
https://gitlab.com/inorton/git-multi-sync

cd repo
python -m gitmultisync --update-submodules

where I had a superproject containing 5-6 submodules, some of which
were quite large (1-2Gb history)

Afterwards, I had trouble doing "git pull -r" in my submodules to pull
in newer changes.

On 6 February 2018 at 20:10, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ian Norton wrote:
>
>>                      Specifically I'm trying to speed up "git
>> submodule update" by doing several at the same time.
>
> Can you say more about this?  E.g. how can I reproduce your experience?
> Is there a script I can run?
>
> Thanks,
> Jonathan