git pull --recurse-submodules wipes uncommitted submodule changes without warning

git pull --recurse-submodules wipes uncommitted submodule changes without warning

[Guillaume Girol]

What did you do before the bug happened? (Steps to reproduce your
issue)

The repository has a submodule bar.
In the current checkout, bar has new commits, but this is not commited:

----------------------------

$ git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-
forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   bar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

no changes added to commit (use "git add" and/or "git commit -a")

----------------------------

There is one commit to pull. It affects a file in the superproject, but
not the submodule.

The issue arises with:

$ git pull --recurse-submodule=on-demand --no-rebase

What did you expect to happen? (Expected behavior)

The file affected by the pulled commit is modified, but the submodule
is left untouched.
The output of git status should look like:

----------------------------

$ git status
On branch master
Your branch up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   bar (new commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

no changes added to commit (use "git add" and/or "git commit -a")

----------------------------

What happened instead? (Actual behavior)

--------------------------

$  git pull --recurse-submodule=on-demand --no-rebase
Updating 67627dd..80f5c51
Fast-forward
 foo | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Submodule path 'bar': checked out
'ef3c0711fedca48b0b43aadfd01b7bb94b519a13'

$  git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	git-bugreport-2022-05-31-2142.txt

nothing added to commit but untracked files present (use "git add" to
track)

-----------------------

my uncommitted changes to the submodule have been wiped, and the commit
of the pulled revision was checked out.

What's different between what you expected and what actually happened?

When there are uncommitted changes to a file foo in a repo, and that I
`git pull` a commit that does not affect this file foo, git pull does
not wipe my changes to `foo`. I expect the same for submodules. If
there is a conflict, I am told about it and nothing is wiped either.


Anything else you want to add:

git pull is able to detect conflicts on submodules, so why not non-
conflicts on submodules?

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.36.0
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /nix/store/0d3wgx8x6dxdb2cpnq105z23hah07z7l-bash-5.1-
p16/bin/bash
uname: Linux 5.15.39 #1-NixOS SMP Thu May 12 10:30:34 UTC 2022 x86_64
compiler info: gnuc: 11.3
libc info: glibc: 2.34
$SHELL (typically, interactive shell): /run/current-system/sw/bin/zsh


[Enabled Hooks]

Re: git pull --recurse-submodules wipes uncommitted submodule changes without warning

[Glen Choo]

Thanks for the report!

Guillaume Girol <symphorien@xlumurb.eu> writes:

> What did you do before the bug happened? (Steps to reproduce your
> issue)
>
> The repository has a submodule bar.
> In the current checkout, bar has new commits, but this is not commited:
>
> ----------------------------
>
> $ git status
> On branch master
> Your branch is behind 'origin/master' by 1 commit, and can be fast-
> forwarded.
>   (use "git pull" to update your local branch)
>
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git restore <file>..." to discard changes in working directory)
> 	modified:   bar (new commits)
>
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
> 	git-bugreport-2022-05-31-2142.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
>
> ----------------------------
>
> There is one commit to pull. It affects a file in the superproject, but
> not the submodule.
>
> The issue arises with:
>
> $ git pull --recurse-submodule=on-demand --no-rebase
>
> What did you expect to happen? (Expected behavior)
>
> The file affected by the pulled commit is modified, but the submodule
> is left untouched.
> The output of git status should look like:
>
> ----------------------------
>
> $ git status
> On branch master
> Your branch up to date with 'origin/master'.
>
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git restore <file>..." to discard changes in working directory)
> 	modified:   bar (new commits)
>
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
> 	git-bugreport-2022-05-31-2142.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
>
> ----------------------------
>
> What happened instead? (Actual behavior)
>
> --------------------------
>
> $  git pull --recurse-submodule=on-demand --no-rebase
> Updating 67627dd..80f5c51
> Fast-forward
>  foo | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> Submodule path 'bar': checked out
> 'ef3c0711fedca48b0b43aadfd01b7bb94b519a13'
>
> $  git status
> On branch master
> Your branch is up to date with 'origin/master'.
>
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
> 	git-bugreport-2022-05-31-2142.txt
>
> nothing added to commit but untracked files present (use "git add" to
> track)
>
> -----------------------
>
> my uncommitted changes to the submodule have been wiped, and the commit
> of the pulled revision was checked out.

This is very clear, thank you.

So basically, because you made a commit in your submodule but did not
commit it back to the superproject, you have 'uncommitted changes'. When
doing "git pull --recurse-submodules", you expect your 'uncommitted
changes' to be left alone, but instead, your submodule has now checked
out an older version.

i.e. your starting state and desired end state is:

  superproject                submodule 
  B (origin/master)           D (HEAD)
  | \                         |
  |  \                        |
  |   ---------               |
  |            \              |
  A (master) ---------------- C

Where you had submodule commit D checked out and it stays checked out.

However your actual end state is:

  superproject                submodule 
  B (origin/master)           D
  | \                         |
  |  \                        |
  |   ---------               |
  |            \              |
  A (master) ---------------- C (HEAD)

where you've checked out submodule commit C, and submodule commit D has
been 'abandoned'.

"checkout --recurse-submodules" is known to abandon commits in a similar
way; the same thing can happen if you had done "git fetch && git
checkout --recurse-submodules origin/master". This is because we only
check for uncommitted changes in the _submodule_, even though the
submodule itself is not committed to the superproject.

But oddly enough, "pull --recurse-submodules" actually abandons the
commits in a _different_ way. After fetching (recursively) and merging
(non-recursively), "git pull" updates the submodules using "git
submodule update --recursive --checkout". This makes each submodule do a
"git checkout <version in merged superproject commit>" (in the example
above, this is commit C). This obviously doesn't actually merge
anything, and presumably we only do this because we want to update the
submodule working tree at the end of "git pull" (see
t/t5572-pull-submodule.sh for examples of how this works).

One way we _might_ be able fix this is to teach "git submodule update"
to notice if the submodule has changed and to abort the checkout if so
(maybe the option could be named --no-abandon?)

> Anything else you want to add:
>
> git pull is able to detect conflicts on submodules, so why not non-
> conflicts on submodules?

IIUC, part of the problem is that we do a very basic job of merging
submodules:

- we check to if the two superproject commits point to the same
  submodule commit
- if they are the same, there is no conflict
- if they are different, there is a merge conflict

In your case, the submodule is not changed (both superproject commits
point to submodule commit C), so there is no merge conflict.

I don't see why we couldn't tighten up our submodule conflict detection
though. This is like a fast-forward/two-way merge, but when we do a
two-way merge, we also check the index to make sure we don't abandon any
local changes. We could do the same for submodules to make sure we don't
abandon any new submodule commits.

What's nice about this idea is that we might be able to reuse this
two-way merge logic to teach "git checkout" not to abandon submodule
commits.