Unexpected conflict during cherry-pick after moving submodule

Unexpected conflict during cherry-pick after moving submodule

[Tamas Peregi]

Hi all,

I ran into something that seems like a bug to me. I'm fairly new to
git, especially git submodules, so it's possible this is expected
behaviour, but if it is, please explain where I went wrong.

Thanks in advance: Tamás

What did you do before the bug happened? (Steps to reproduce your issue)
========================================================================

- I modified a file on one branch
- Then switched to another
- Moved a submodule to another directory
- Tried to cherry-pick my modification from the other branch

This script can be used to reproduce my steps (any submodule works, I
chose gsl-lite because I like it):
    git init

    echo hello > hello.txt
    git add hello.txt
    git commit -m "Add hello.txt"

    git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl
    git commit -a -m "Add submodule"

    git branch world
    git branch move

    git checkout world
    echo world >> hello.txt
    git commit -a -m "Expand hello.txt"

    git checkout move
    mkdir thirdparty
    git mv vendor/gsl thirdparty/gsl
    git commit -a -m "Move submodule"

    git cherry-pick world


What did you expect to happen? (Expected behavior)
==================================================

I expected the cherry-pick to happen without conflicts.


What happened instead? (Actual behavior)
========================================
I ran into a conflict during the cherry-pick:
    Adding as thirdparty/gsl~HEAD instead
    error: could not apply 516d94f... Expand hello.txt
    hint: after resolving the conflicts, mark the corrected paths
    hint: with 'git add <paths>' or 'git rm <paths>'
    hint: and commit the result with 'git commit'


What's different between what you expected and what actually happened?
======================================================================

As hello.txt has nothing to do with the submodule, I assumed the
cherry-pick would go through smoothly.


Anything else you want to add:
==============================

- "git add thirdparty/gsl && git cherry-pick --continue" resolves the
issue as expected.
- If I change which commit in the submodule the superproject points
to, the problem disappears, but I don't necessarily want to do that.
- I'm using "Git for Windows", but the issue was reproduced by my
Mac-based colleagues too with the exact same results.

[System Info]
git version:
git version 2.31.1.windows.1
cpu: x86_64
built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Windows 10.0 19042
compiler info: gnuc: 10.2
libc info: no libc information available
$SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

Re: Unexpected conflict during cherry-pick after moving submodule

[Elijah Newren]

Hi,

On Mon, Apr 12, 2021 at 7:59 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
>
> Hi all,
>
> I ran into something that seems like a bug to me. I'm fairly new to
> git, especially git submodules, so it's possible this is expected
> behaviour, but if it is, please explain where I went wrong.
>
> Thanks in advance: Tamás
>
> What did you do before the bug happened? (Steps to reproduce your issue)
> ========================================================================
>
> - I modified a file on one branch
> - Then switched to another
> - Moved a submodule to another directory
> - Tried to cherry-pick my modification from the other branch
>
> This script can be used to reproduce my steps (any submodule works, I
> chose gsl-lite because I like it):
>     git init
>
>     echo hello > hello.txt
>     git add hello.txt
>     git commit -m "Add hello.txt"
>
>     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl
>     git commit -a -m "Add submodule"
>
>     git branch world
>     git branch move
>
>     git checkout world
>     echo world >> hello.txt
>     git commit -a -m "Expand hello.txt"
>
>     git checkout move
>     mkdir thirdparty
>     git mv vendor/gsl thirdparty/gsl
>     git commit -a -m "Move submodule"
>
>     git cherry-pick world
>
>
> What did you expect to happen? (Expected behavior)
> ==================================================
>
> I expected the cherry-pick to happen without conflicts.
>
>
> What happened instead? (Actual behavior)
> ========================================
> I ran into a conflict during the cherry-pick:
>     Adding as thirdparty/gsl~HEAD instead
>     error: could not apply 516d94f... Expand hello.txt
>     hint: after resolving the conflicts, mark the corrected paths
>     hint: with 'git add <paths>' or 'git rm <paths>'
>     hint: and commit the result with 'git commit'
>
>
> What's different between what you expected and what actually happened?
> ======================================================================
>
> As hello.txt has nothing to do with the submodule, I assumed the
> cherry-pick would go through smoothly.
>
>
> Anything else you want to add:
> ==============================
>
> - "git add thirdparty/gsl && git cherry-pick --continue" resolves the
> issue as expected.
> - If I change which commit in the submodule the superproject points
> to, the problem disappears, but I don't necessarily want to do that.
> - I'm using "Git for Windows", but the issue was reproduced by my
> Mac-based colleagues too with the exact same results.
>
> [System Info]
> git version:
> git version 2.31.1.windows.1
> cpu: x86_64
> built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
> sizeof-long: 4
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Windows 10.0 19042
> compiler info: gnuc: 10.2
> libc info: no libc information available
> $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

Thanks for the report.

We can change the final command to
    git cherry-pick --strategy recursive world

to be more explicit, though that doesn't change things and still
triggers the bug.  However, if we change the above to
    git cherry-pick --strategy ort world

then that fixes the bug, including even in git-2.31.1.  The version of
the 'ort' merge backend in 2.31.1 isn't ready for general usage (it
can't handle file renormalization, or subtree merges, or
sparse-checkouts with conflicts in files outside of the requested
sparsity paths, and also has a platform specific bug where it'll die
on corner case rename/rename conflicts due to assumptions about libc
qsort() stability), but the version in 2.32 will be.  And 'ort' is
going to replace 'recursive' once the project is comfortable enough
with the new backend.

All this is to say that the fix is on the way, even if it doesn't help
you quite yet.

Re: Unexpected conflict during cherry-pick after moving submodule

[Tamas Peregi]

Hi Elijah,

thanks for the quick reply and the useful information about the ort strategy!

Do I understand correctly that the problem is in the recursive
strategy, i.e. inside the cherry-pick step, not the submodule
movement? That sounds a bit unfortunate in my case, as I'm the one
moving the submodule (then merging it back to master), and others in
my company want to cherry-pick over it, so I'm breaking their
workflows if I go ahead with moving. (Unless I tell all of them to use
the experimental ort strategy instead, which may introduce its own set
of problems.) Is there any way of moving the submodule that doesn't
break cherry-picking with the default (recursive) strategy? I'm
willing to do some extra work to spare confusion by others down the
line.

Also: up until now, I thought cherry-pick simply exports the source
commit as a patch, then applies it to the target commit, but
"recursive" is a merge strategy, correct? So is cherry-pick doing
something vastly more complex than I thought, involving merging in the
background?

Thanks in advance: Tamás


On Mon, Apr 12, 2021 at 5:29 PM Elijah Newren <newren@gmail.com> wrote:
>
> Hi,
>
> On Mon, Apr 12, 2021 at 7:59 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
> >
> > Hi all,
> >
> > I ran into something that seems like a bug to me. I'm fairly new to
> > git, especially git submodules, so it's possible this is expected
> > behaviour, but if it is, please explain where I went wrong.
> >
> > Thanks in advance: Tamás
> >
> > What did you do before the bug happened? (Steps to reproduce your issue)
> > ========================================================================
> >
> > - I modified a file on one branch
> > - Then switched to another
> > - Moved a submodule to another directory
> > - Tried to cherry-pick my modification from the other branch
> >
> > This script can be used to reproduce my steps (any submodule works, I
> > chose gsl-lite because I like it):
> >     git init
> >
> >     echo hello > hello.txt
> >     git add hello.txt
> >     git commit -m "Add hello.txt"
> >
> >     git submodule add https://github.com/gsl-lite/gsl-lite/ vendor/gsl
> >     git commit -a -m "Add submodule"
> >
> >     git branch world
> >     git branch move
> >
> >     git checkout world
> >     echo world >> hello.txt
> >     git commit -a -m "Expand hello.txt"
> >
> >     git checkout move
> >     mkdir thirdparty
> >     git mv vendor/gsl thirdparty/gsl
> >     git commit -a -m "Move submodule"
> >
> >     git cherry-pick world
> >
> >
> > What did you expect to happen? (Expected behavior)
> > ==================================================
> >
> > I expected the cherry-pick to happen without conflicts.
> >
> >
> > What happened instead? (Actual behavior)
> > ========================================
> > I ran into a conflict during the cherry-pick:
> >     Adding as thirdparty/gsl~HEAD instead
> >     error: could not apply 516d94f... Expand hello.txt
> >     hint: after resolving the conflicts, mark the corrected paths
> >     hint: with 'git add <paths>' or 'git rm <paths>'
> >     hint: and commit the result with 'git commit'
> >
> >
> > What's different between what you expected and what actually happened?
> > ======================================================================
> >
> > As hello.txt has nothing to do with the submodule, I assumed the
> > cherry-pick would go through smoothly.
> >
> >
> > Anything else you want to add:
> > ==============================
> >
> > - "git add thirdparty/gsl && git cherry-pick --continue" resolves the
> > issue as expected.
> > - If I change which commit in the submodule the superproject points
> > to, the problem disappears, but I don't necessarily want to do that.
> > - I'm using "Git for Windows", but the issue was reproduced by my
> > Mac-based colleagues too with the exact same results.
> >
> > [System Info]
> > git version:
> > git version 2.31.1.windows.1
> > cpu: x86_64
> > built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
> > sizeof-long: 4
> > sizeof-size_t: 8
> > shell-path: /bin/sh
> > feature: fsmonitor--daemon
> > uname: Windows 10.0 19042
> > compiler info: gnuc: 10.2
> > libc info: no libc information available
> > $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe
>
> Thanks for the report.
>
> We can change the final command to
>     git cherry-pick --strategy recursive world
>
> to be more explicit, though that doesn't change things and still
> triggers the bug.  However, if we change the above to
>     git cherry-pick --strategy ort world
>
> then that fixes the bug, including even in git-2.31.1.  The version of
> the 'ort' merge backend in 2.31.1 isn't ready for general usage (it
> can't handle file renormalization, or subtree merges, or
> sparse-checkouts with conflicts in files outside of the requested
> sparsity paths, and also has a platform specific bug where it'll die
> on corner case rename/rename conflicts due to assumptions about libc
> qsort() stability), but the version in 2.32 will be.  And 'ort' is
> going to replace 'recursive' once the project is comfortable enough
> with the new backend.
>
> All this is to say that the fix is on the way, even if it doesn't help
> you quite yet.

Re: Unexpected conflict during cherry-pick after moving submodule

[Elijah Newren]

Hi Tamas,

On Mon, Apr 12, 2021 at 9:08 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
>
> Hi Elijah,
>
> thanks for the quick reply and the useful information about the ort strategy!
>
> Do I understand correctly that the problem is in the recursive
> strategy, i.e. inside the cherry-pick step, not the submodule
> movement? That sounds a bit unfortunate in my case, as I'm the one
> moving the submodule (then merging it back to master), and others in
> my company want to cherry-pick over it, so I'm breaking their
> workflows if I go ahead with moving. (Unless I tell all of them to use
> the experimental ort strategy instead, which may introduce its own set
> of problems.) Is there any way of moving the submodule that doesn't
> break cherry-picking with the default (recursive) strategy? I'm
> willing to do some extra work to spare confusion by others down the
> line.

Unfortunately, the recursive merge machinery had a variety of issues
with submodules (see e.g.
https://git.kernel.org/pub/scm/git/git.git/commit/?id=aa2faac03ad646873ebac2b230581d1d26dd1b99)
and yes, the merge machinery is intrinsic to cherry-pick's
functionality.  I don't have a good workaround for you, short of "wait
until git-2.32 is released".

> Also: up until now, I thought cherry-pick simply exports the source
> commit as a patch, then applies it to the target commit, but
> "recursive" is a merge strategy, correct? So is cherry-pick doing
> something vastly more complex than I thought, involving merging in the
> background?

Right, cherry-pick makes use of the merge machinery using a particular
specially crafted merge while recording it as a regular commit.
Interestingly, the rebase command has multiple backends, one of which
used the merge machinery and another that behaved as you thought by
creating and applying patches.  While the use of an appropriate
special merge (as done by cherry-pick and rebase) is roughly
semantically equivalent to what creating and applying patches
provides, the create-and-apply-patches procedure actually discards
relevant information and results in some shortcomings that simply
cannot be fixed.  As such, we switched the default rebase backend from
"apply" to "merge" as of git-2.26.  If you're curious, read up on the
"BEHAVIORAL DIFFERENCES" section of the git-rebase(1) manual,
especially the "Context" section.

Hope that helps,
Elijah

Re: Unexpected conflict during cherry-pick after moving submodule

[Tamas Peregi]

Hi Elijah,

thanks for the expanded reply, I think it cleared everything up.

Regards,
Tamás

On Mon, Apr 12, 2021 at 6:53 PM Elijah Newren <newren@gmail.com> wrote:
>
> Hi Tamas,
>
> On Mon, Apr 12, 2021 at 9:08 AM Tamas Peregi <tamas.peregi@formlabs.com> wrote:
> >
> > Hi Elijah,
> >
> > thanks for the quick reply and the useful information about the ort strategy!
> >
> > Do I understand correctly that the problem is in the recursive
> > strategy, i.e. inside the cherry-pick step, not the submodule
> > movement? That sounds a bit unfortunate in my case, as I'm the one
> > moving the submodule (then merging it back to master), and others in
> > my company want to cherry-pick over it, so I'm breaking their
> > workflows if I go ahead with moving. (Unless I tell all of them to use
> > the experimental ort strategy instead, which may introduce its own set
> > of problems.) Is there any way of moving the submodule that doesn't
> > break cherry-picking with the default (recursive) strategy? I'm
> > willing to do some extra work to spare confusion by others down the
> > line.
>
> Unfortunately, the recursive merge machinery had a variety of issues
> with submodules (see e.g.
> https://git.kernel.org/pub/scm/git/git.git/commit/?id=aa2faac03ad646873ebac2b230581d1d26dd1b99)
> and yes, the merge machinery is intrinsic to cherry-pick's
> functionality.  I don't have a good workaround for you, short of "wait
> until git-2.32 is released".
>
> > Also: up until now, I thought cherry-pick simply exports the source
> > commit as a patch, then applies it to the target commit, but
> > "recursive" is a merge strategy, correct? So is cherry-pick doing
> > something vastly more complex than I thought, involving merging in the
> > background?
>
> Right, cherry-pick makes use of the merge machinery using a particular
> specially crafted merge while recording it as a regular commit.
> Interestingly, the rebase command has multiple backends, one of which
> used the merge machinery and another that behaved as you thought by
> creating and applying patches.  While the use of an appropriate
> special merge (as done by cherry-pick and rebase) is roughly
> semantically equivalent to what creating and applying patches
> provides, the create-and-apply-patches procedure actually discards
> relevant information and results in some shortcomings that simply
> cannot be fixed.  As such, we switched the default rebase backend from
> "apply" to "merge" as of git-2.26.  If you're curious, read up on the
> "BEHAVIORAL DIFFERENCES" section of the git-rebase(1) manual,
> especially the "Context" section.
>
> Hope that helps,
> Elijah