Possible race condition with git-rebase + .git/index.lock

Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Hi there,

I've been bit by this bug many times, and always forget to report it
here, but this time I finally remembered.

The scenario is this:

- I have a local repository which tracks the upstream of the project.

- I have 120+ local commits.

- Every week or so, I have to rebase my local commits on top of
  upstream's origin/master.

The workflow is simple:

  # git fetch origin && git rebase origin/master

This works without problems most of the time (well, usually there are
conflicts and all, but that's a burden I have to carry).  However,
sometimes I notice that git fails with:

  # git rebase origin/master
  ...
  Applying: commitX
  Applying: commitY
  Applying: commitZ
  fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.

The first thing I did was to check whether the index.lock file existed,
but it doesn't.

I also I thought it might have something to do with "commitZ", but after
I ran "git rebase --abort" and tried rebasing again, I noticed that it
would stop at another commit.

This seems like a race condition, but after inspecting builtin/rebase.c
it doesn't seem like the "git am" commands are run in parallel (and it
wouldn't make sense if they did, anyway).

I tried using GIT_TRACE, but haven't seen any useful information in the
logs.  I tried using the "next" branch, and found that the problem
exists there as well.  I wonder if anyone else has seen this problem.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Possible race condition with git-rebase + .git/index.lock

[Eric Sunshine]

On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior
<sergiodj@sergiodj.net> wrote:
>   # git rebase origin/master
>   ...
>   Applying: commitX
>   Applying: commitY
>   Applying: commitZ
>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>
> The first thing I did was to check whether the index.lock file existed,
> but it doesn't.

What platform is this on? If Windows, I'm wondering if something (such
as a virus scanner) is holding the lock file open long enough to
prevent Git from actually deleting it (and perhaps Git doesn't notice
the failed deletion -- until the next time it tries to take the lock).

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Monday, March 11 2019, Eric Sunshine wrote:

> On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>>   # git rebase origin/master
>>   ...
>>   Applying: commitX
>>   Applying: commitY
>>   Applying: commitZ
>>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>
>> The first thing I did was to check whether the index.lock file existed,
>> but it doesn't.
>
> What platform is this on? If Windows, I'm wondering if something (such
> as a virus scanner) is holding the lock file open long enough to
> prevent Git from actually deleting it (and perhaps Git doesn't notice
> the failed deletion -- until the next time it tries to take the lock).

This is on GNU/Linux (Fedora 28, to be specific).  I haven't had the
chance to test this on other distros, unfortunately.  And since it seems
to be a race condition, it's kind of hard to perform tests that trigger
this.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock

[Duy Nguyen]

On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
<sergiodj@sergiodj.net> wrote:
> This works without problems most of the time (well, usually there are
> conflicts and all, but that's a burden I have to carry).  However,
> sometimes I notice that git fails with:
>
>   # git rebase origin/master
>   ...
>   Applying: commitX
>   Applying: commitY
>   Applying: commitZ
>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>
> The first thing I did was to check whether the index.lock file existed,
> but it doesn't.

Is the output this clean? What I'm looking for is signs of automatic
garbage collection kicking in the middle of the rebase. Something like
"Auto packing the repository blah blah for optimum performance".
-- 
Duy

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Tuesday, March 12 2019, Duy Nguyen wrote:

> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>> This works without problems most of the time (well, usually there are
>> conflicts and all, but that's a burden I have to carry).  However,
>> sometimes I notice that git fails with:
>>
>>   # git rebase origin/master
>>   ...
>>   Applying: commitX
>>   Applying: commitY
>>   Applying: commitZ
>>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>
>> The first thing I did was to check whether the index.lock file existed,
>> but it doesn't.
>
> Is the output this clean? What I'm looking for is signs of automatic
> garbage collection kicking in the middle of the rebase. Something like
> "Auto packing the repository blah blah for optimum performance".

Yeah, this is the exact output.  I also thought about "git gc", but I
don't see any output related to it.  Is it possible that it's being
executed in the background and not printing anything?

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock

[Elijah Newren]

On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
<sergiodj@sergiodj.net> wrote:
> On Tuesday, March 12 2019, Duy Nguyen wrote:
>
> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> > <sergiodj@sergiodj.net> wrote:
> >> This works without problems most of the time (well, usually there are
> >> conflicts and all, but that's a burden I have to carry).  However,
> >> sometimes I notice that git fails with:
> >>
> >>   # git rebase origin/master
> >>   ...
> >>   Applying: commitX
> >>   Applying: commitY
> >>   Applying: commitZ
> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
> >>
> >> The first thing I did was to check whether the index.lock file existed,
> >> but it doesn't.
> >
> > Is the output this clean? What I'm looking for is signs of automatic
> > garbage collection kicking in the middle of the rebase. Something like
> > "Auto packing the repository blah blah for optimum performance".
>
> Yeah, this is the exact output.  I also thought about "git gc", but I
> don't see any output related to it.  Is it possible that it's being
> executed in the background and not printing anything?

Any chance you have a cronjob or other task execution mechanism that
is running git commands in various directories (even simple commands
like 'git status' since it's not read-only contrary to what some
expect, or 'git fetch' which might trigger a background gc)?

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Tuesday, March 12 2019, Elijah Newren wrote:

> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>
>> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>> > <sergiodj@sergiodj.net> wrote:
>> >> This works without problems most of the time (well, usually there are
>> >> conflicts and all, but that's a burden I have to carry).  However,
>> >> sometimes I notice that git fails with:
>> >>
>> >>   # git rebase origin/master
>> >>   ...
>> >>   Applying: commitX
>> >>   Applying: commitY
>> >>   Applying: commitZ
>> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>> >>
>> >> The first thing I did was to check whether the index.lock file existed,
>> >> but it doesn't.
>> >
>> > Is the output this clean? What I'm looking for is signs of automatic
>> > garbage collection kicking in the middle of the rebase. Something like
>> > "Auto packing the repository blah blah for optimum performance".
>>
>> Yeah, this is the exact output.  I also thought about "git gc", but I
>> don't see any output related to it.  Is it possible that it's being
>> executed in the background and not printing anything?
>
> Any chance you have a cronjob or other task execution mechanism that
> is running git commands in various directories (even simple commands
> like 'git status' since it's not read-only contrary to what some
> expect, or 'git fetch' which might trigger a background gc)?

Nope, nothing like this.  AFAIK, nothing is touching that repository at
the same time that I am.  Besides, even if I wait some minutes before
trying again, the bug manifests again.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock

[Elijah Newren]

On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior
<sergiodj@sergiodj.net> wrote:
>
> On Tuesday, March 12 2019, Elijah Newren wrote:
>
> > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
> > <sergiodj@sergiodj.net> wrote:
> >> On Tuesday, March 12 2019, Duy Nguyen wrote:
> >>
> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> >> > <sergiodj@sergiodj.net> wrote:
> >> >> This works without problems most of the time (well, usually there are
> >> >> conflicts and all, but that's a burden I have to carry).  However,
> >> >> sometimes I notice that git fails with:
> >> >>
> >> >>   # git rebase origin/master
> >> >>   ...
> >> >>   Applying: commitX
> >> >>   Applying: commitY
> >> >>   Applying: commitZ
> >> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
> >> >>
> >> >> The first thing I did was to check whether the index.lock file existed,
> >> >> but it doesn't.
> >> >
> >> > Is the output this clean? What I'm looking for is signs of automatic
> >> > garbage collection kicking in the middle of the rebase. Something like
> >> > "Auto packing the repository blah blah for optimum performance".
> >>
> >> Yeah, this is the exact output.  I also thought about "git gc", but I
> >> don't see any output related to it.  Is it possible that it's being
> >> executed in the background and not printing anything?
> >
> > Any chance you have a cronjob or other task execution mechanism that
> > is running git commands in various directories (even simple commands
> > like 'git status' since it's not read-only contrary to what some
> > expect, or 'git fetch' which might trigger a background gc)?
>
> Nope, nothing like this.  AFAIK, nothing is touching that repository at
> the same time that I am.  Besides, even if I wait some minutes before
> trying again, the bug manifests again.

Well, even though you didn't see the output Duy was looking for, if
you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe
gc.autoDetach to false just for good measure), does the problem still
occur?

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Tuesday, March 12 2019, Elijah Newren wrote:

> On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>>
>> On Tuesday, March 12 2019, Elijah Newren wrote:
>>
>> > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>> > <sergiodj@sergiodj.net> wrote:
>> >> On Tuesday, March 12 2019, Duy Nguyen wrote:
>> >>
>> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>> >> > <sergiodj@sergiodj.net> wrote:
>> >> >> This works without problems most of the time (well, usually there are
>> >> >> conflicts and all, but that's a burden I have to carry).  However,
>> >> >> sometimes I notice that git fails with:
>> >> >>
>> >> >>   # git rebase origin/master
>> >> >>   ...
>> >> >>   Applying: commitX
>> >> >>   Applying: commitY
>> >> >>   Applying: commitZ
>> >> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>> >> >>
>> >> >> The first thing I did was to check whether the index.lock file existed,
>> >> >> but it doesn't.
>> >> >
>> >> > Is the output this clean? What I'm looking for is signs of automatic
>> >> > garbage collection kicking in the middle of the rebase. Something like
>> >> > "Auto packing the repository blah blah for optimum performance".
>> >>
>> >> Yeah, this is the exact output.  I also thought about "git gc", but I
>> >> don't see any output related to it.  Is it possible that it's being
>> >> executed in the background and not printing anything?
>> >
>> > Any chance you have a cronjob or other task execution mechanism that
>> > is running git commands in various directories (even simple commands
>> > like 'git status' since it's not read-only contrary to what some
>> > expect, or 'git fetch' which might trigger a background gc)?
>>
>> Nope, nothing like this.  AFAIK, nothing is touching that repository at
>> the same time that I am.  Besides, even if I wait some minutes before
>> trying again, the bug manifests again.
>
> Well, even though you didn't see the output Duy was looking for, if
> you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe
> gc.autoDetach to false just for good measure), does the problem still
> occur?

Thanks, I was indeed going to try this.  I'll test when I have time, and
will let you know.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock

[Phillip Wood]

Hi Sergio

On 12/03/2019 19:32, Sergio Durigan Junior wrote:
> On Tuesday, March 12 2019, Elijah Newren wrote:
> 
>> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>> <sergiodj@sergiodj.net> wrote:
>>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>>
>>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>>>> <sergiodj@sergiodj.net> wrote:
>>>>> This works without problems most of the time (well, usually there are
>>>>> conflicts and all, but that's a burden I have to carry).  However,
>>>>> sometimes I notice that git fails with:
>>>>>
>>>>>    # git rebase origin/master
>>>>>    ...
>>>>>    Applying: commitX
>>>>>    Applying: commitY
>>>>>    Applying: commitZ
>>>>>    fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>>>>
>>>>> The first thing I did was to check whether the index.lock file existed,
>>>>> but it doesn't.
>>>>
>>>> Is the output this clean? What I'm looking for is signs of automatic
>>>> garbage collection kicking in the middle of the rebase. Something like
>>>> "Auto packing the repository blah blah for optimum performance".
>>>
>>> Yeah, this is the exact output.  I also thought about "git gc", but I
>>> don't see any output related to it.  Is it possible that it's being
>>> executed in the background and not printing anything?
>>
>> Any chance you have a cronjob or other task execution mechanism that
>> is running git commands in various directories (even simple commands
>> like 'git status' since it's not read-only contrary to what some
>> expect, or 'git fetch' which might trigger a background gc)?
> 
> Nope, nothing like this.  AFAIK, nothing is touching that repository at
> the same time that I am.  Besides, even if I wait some minutes before
> trying again, the bug manifests again.
> 

Do you have any of the files in the repository open in an editor or ide 
while you are rebasing? I have seen this message when rebasing and one 
of the files is open in emacs which runs git status when it sees a file 
has changed which locks the index while the rebase is running.

Best Wishes

Phillip

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Wednesday, March 13 2019, Phillip Wood wrote:

> Hi Sergio
>
> On 12/03/2019 19:32, Sergio Durigan Junior wrote:
>> On Tuesday, March 12 2019, Elijah Newren wrote:
>>
>>> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>>> <sergiodj@sergiodj.net> wrote:
>>>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>>>
>>>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>>>>> <sergiodj@sergiodj.net> wrote:
>>>>>> This works without problems most of the time (well, usually there are
>>>>>> conflicts and all, but that's a burden I have to carry).  However,
>>>>>> sometimes I notice that git fails with:
>>>>>>
>>>>>>    # git rebase origin/master
>>>>>>    ...
>>>>>>    Applying: commitX
>>>>>>    Applying: commitY
>>>>>>    Applying: commitZ
>>>>>>    fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>>>>>
>>>>>> The first thing I did was to check whether the index.lock file existed,
>>>>>> but it doesn't.
>>>>>
>>>>> Is the output this clean? What I'm looking for is signs of automatic
>>>>> garbage collection kicking in the middle of the rebase. Something like
>>>>> "Auto packing the repository blah blah for optimum performance".
>>>>
>>>> Yeah, this is the exact output.  I also thought about "git gc", but I
>>>> don't see any output related to it.  Is it possible that it's being
>>>> executed in the background and not printing anything?
>>>
>>> Any chance you have a cronjob or other task execution mechanism that
>>> is running git commands in various directories (even simple commands
>>> like 'git status' since it's not read-only contrary to what some
>>> expect, or 'git fetch' which might trigger a background gc)?
>>
>> Nope, nothing like this.  AFAIK, nothing is touching that repository at
>> the same time that I am.  Besides, even if I wait some minutes before
>> trying again, the bug manifests again.
>>
>
> Do you have any of the files in the repository open in an editor or
> ide while you are rebasing? I have seen this message when rebasing and
> one of the files is open in emacs which runs git status when it sees a
> file has changed which locks the index while the rebase is running.

Hey Phillip,

Huh, I do have some of the files opened in Emacs!  They're in
background, but indeed, now that you mentioned I remember that Emacs
keeps track of changes and invokes "git status" sometimes.  Next time I
see the bug, I'll try closing the files and see if it happens again.

Thanks for the insight!

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock

[Jeff King]

On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote:

> Huh, I do have some of the files opened in Emacs!  They're in
> background, but indeed, now that you mentioned I remember that Emacs
> keeps track of changes and invokes "git status" sometimes.  Next time I
> see the bug, I'll try closing the files and see if it happens again.

If Emacs (I guess maybe magit?) is running "git status" behind the
scenes, you might benefit from teaching it to use "git
--no-optional-locks status" instead. See the section "BACKGROUND
REFRESH" in "git help status" for more discussion.

-Peff

Re: Possible race condition with git-rebase + .git/index.lock

[Sergio Durigan Junior]

On Wednesday, March 13 2019, Jeff King wrote:

> On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote:
>
>> Huh, I do have some of the files opened in Emacs!  They're in
>> background, but indeed, now that you mentioned I remember that Emacs
>> keeps track of changes and invokes "git status" sometimes.  Next time I
>> see the bug, I'll try closing the files and see if it happens again.
>
> If Emacs (I guess maybe magit?) is running "git status" behind the
> scenes, you might benefit from teaching it to use "git
> --no-optional-locks status" instead. See the section "BACKGROUND
> REFRESH" in "git help status" for more discussion.

Thanks Jeff.  That's going to be helpful.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: Possible race condition with git-rebase + .git/index.lock - probably a gettext issue

[Mingun]

---
It seems that this bug is related to localization library, similar to this diesel issue: https://github.com/diesel-rs/diesel/discussions/2947#discussioncomment-2025857.

Also, this bug is highly depend on the free space capacity on a drive where working copy is located. I encounter this bug each time when my ssd is nearly full (about 6 GB of free space on 120 GB drive).

About 90% of time, when I try to use `git svn rebase` in such conditions, the operation failed:

```
$ git svn rebase
warning: пропущен уже применённый коммит ef0998895a6
warning: пропущен уже применённый коммит d1dbbd6ef8b
подсказка: use --reapply-cherry-picks to include skipped commits
подсказка: Disable this message with "git config advice.skippedCherryPicks false"
fatal: Не удалось создать «/path/to/my/git/repo/.git/index.lock»: Файл существует.

Похоже, что другой процесс git запущен в этом репозитории,
например редактор открыт из «git commit». Пожалуйста, убедитесь,
что все процессы были завершены и потом попробуйте снова.
Если это не поможет, то возможно процесс git был ранее завершен
с ошибкой в этом репозитории:
удалите файл вручную для продолжения.
rebase refs/remotes/origin/trunk: command returned error: 128

Ветка push удалена (была 5c2212d372e).
$ git rebase --continue
fatal: не удалось прочитать файл журнала «.git/rebase-merge/message»: Нет такого файла или каталога
```

At the same time, when after I free space on my drive or set "C" locale, the operation is completed from first attempt.

I hope this will help to localize and fix the issue.