fatal: cannot rebase with locally recorded submodule modifications

fatal: cannot rebase with locally recorded submodule modifications

[Brice Goglin]

Hello

I don't understand why git pull --rebase --recurse-submodules fails after a git reset --hard
to the previous commit. I didn't touch the submodule, it's supposed to be clean:

$ git clone --recursive git@github.com:open-mpi/ompi.git
[...]
$ git reset --hard HEAD^
HEAD is now at 395e2c9d8f Merge pull request #7364 from awlauria/fix_compile
$ git pull --rebase --recurse-submodules
Fetching submodule opal/mca/hwloc/hwloc2/hwloc
fatal: cannot rebase with locally recorded submodule modifications

Why is failing here?

$ git submodule status
 38433c0f5fae0b761bd20e7b928c77f3ff2e76dc opal/mca/hwloc/hwloc2/hwloc (hwloc-2.1.0rc2-33-g38433c0f)
(that's expected)

$ git status
HEAD detached at 38433c0f
nothing to commit, working tree clean

Thanks

Brice

Re: fatal: cannot rebase with locally recorded submodule modifications

[Shourya Shukla]

Hello Brice,

The error you are facing here is actually a bug in the
'git pull' command[1]. When one tries to use the
'--recurse-submodules' tag with the command, it tends to fail.

> Why is failing here?

Going into a bit detail of 'git pull', the command actually
is a combination of 'git fetch' and 'git merge'(unless any
other tag is specified i.e. 'rebase' here which would mean
a fetch followed by a rebase). Quoting from the documentation:

> Using --recurse-submodules can only fetch new commits in already
checked out submodules right now

Using the '--recurse-submodules' tag will try merging/rebasing any
changes(updations, additions, deletions, etc.) made to the fetched
submodule(s) from upstream, i.e. the _un-checked out_ submodules into
our local repository which will cause the error to be thrown as it is
not supported right now. This will therefore create the detached HEAD
just like the one you obtained.

The "workaround" is to do a 'git fetch' and then merge again after the
'git pull <tag> --recurse-submodules'.

The presence/absence of 'git reset --hard' before the pull  won't create
any difference to this error in my opinion.

Regards,
Shourya Shukla

[1]: https://git-scm.com/docs/git-pull#_bugs

Re: fatal: cannot rebase with locally recorded submodule modifications

[Philippe Blain]

Hi Shourya,

> Le 7 févr. 2020 à 17:04, Shourya Shukla <shouryashukla.oo@gmail.com> a écrit :
> 
> Hello Brice,
> 
> The error you are facing here is actually a bug in the
> 'git pull' command[1]. When one tries to use the
> '--recurse-submodules' tag with the command, it tends to fail.

Let's not discourage our users ;) It is untrue that it tends to fail.

>> Why is failing here?
> 
> Going into a bit detail of 'git pull', the command actually
> is a combination of 'git fetch' and 'git merge'(unless any
> other tag is specified i.e. 'rebase' here which would mean
> a fetch followed by a rebase). Quoting from the documentation:
> 
>> Using --recurse-submodules can only fetch new commits in already
> checked out submodules right now
> 
> Using the '--recurse-submodules' tag will try merging/rebasing any
> changes(updations, additions, deletions, etc.) made to the fetched
> submodule(s) from upstream, i.e. the _un-checked out_ submodules into
> our local repository which will cause the error to be thrown as it is
> not supported right now.

That part of the doc talks about the fact that when using '--recurse-submodules', only commits
in already checked out submodules ("populated submodules" is the wording used in the docs 
for that option) will be *fetched*. If you take a look at, for example,

    git blame -L 240,+10 Documentation/git-pull.txt

you'll see that this section was added in 794a3592ad (fetch/pull: Describe 
--recurse-submodule restrictions in the BUGS section, 2011-03-06), when 'git fetch'
learned the '--recurse-submodules' flag. At the time 'git pull --recurse-submodules' 
did *not* update the working tree of the submodules, it just passed '--recurse-submodules' 
to the underlying 'git fetch'. 'git pull' learned to update the working tree of the submodules
(and also rebase the submodules themselves under very specific circumstances) in 
a6d7eb2c7a (pull: optionally rebase submodules (remote submodule changes only), 
2017-06-23). 

So this "bug" you mention is not what is happening here. Since Brice clone with 
'--recurse-submodules', all submodules will be cloned, initialized and populated as 
part of the superproject clone, so they are all already checked out when 'git pull' is invoked.

Also, the "Bugs" section in the docs that you refer to does not talk about the command
erroring; it just describes a limitation of 'git fetch --recurse-submodules'. And as a side note,
there is no test in the test suite that checks this behaviour (looking at t5526-fetch-submodules.sh
and t5572-pull-submodule.sh). It would be good to fix that by adding a couple 'test_expect_failure' 
tests, or even to fix the behaviour itself. 

Note also that when this "Bugs" section was added, the submodule repositories 
were cloned directly in the superproject working tree instead of being cloned to 
.git/modules/<submodule-name>. This explains why this limitation existed at the time:
you can't fetch the submodules if there is no filesystem location to put their Git repository!
A few months later, 501770e1bb (Move git-dir for submodules, 2011-08-15), implemented 
the "new-style" submodule setup, i.e. cloning the submodules repositories inside the Git repository
of the superproject, as is standard these days. So there is no good reason today to not fix this 
bug, in my opinion.

Going back to Brice's bug:

> The "workaround" is to do a 'git fetch' and then merge again after the
> 'git pull <tag> --recurse-submodules'.

I don't think there is a workaround until my fixes get merged, apart from
not using '--recurse-submodules' with 'git pull --rebase'. Of course then you 
have to remember to *always* do 'git submodule update --recursive' after 
each pull if you want your submodule checkouts to stay in sync with the commits
recorded in the superproject.

> The presence/absence of 'git reset --hard' before the pull  won't create
> any difference to this error in my opinion.

On the contrary: as I wrote in my answer to Brice, it is exactly because 'git reset --hard' is used 
here that the code wrongly errors.

Cheers, and thank you for all the work you put in the rewrite of the submodule code
from shell to C this summer.

Philippe.

Re: fatal: cannot rebase with locally recorded submodule modifications

[Philippe Blain]

Hi Brice,

> Le 7 févr. 2020 à 15:50, Brice Goglin <bgoglin@free.fr> a écrit :
> 
> Hello
> 
> I don't understand why git pull --rebase --recurse-submodules fails after a git reset --hard
> to the previous commit. I didn't touch the submodule, it's supposed to be clean:
> 
> $ git clone --recursive git@github.com:open-mpi/ompi.git
> [...]
> $ git reset --hard HEAD^
> HEAD is now at 395e2c9d8f Merge pull request #7364 from awlauria/fix_compile
> $ git pull --rebase --recurse-submodules
> Fetching submodule opal/mca/hwloc/hwloc2/hwloc
> fatal: cannot rebase with locally recorded submodule modifications
> 
> Why is failing here?

It's failing because there is a bug in the code for 'git pull --rebase --recurse-submodules'
when the current branch and its remote-tracking counterpart have no fork-point,
which seem to be the situation happening here.

I've just finished a patch series that fixes this bug. You can take a look at it here:
https://github.com/gitgitgadget/git/pull/789. I will send it to the list shortly, and will
CC you when I do.

Cheers,

Philippe.