How to debug a "git merge"?

How to debug a "git merge"?

[Lars Schneider]

Hi,

I am investigating a Git merge (a86dd40fe) in which an older version of 
a file won over the newer version. I try to understand why this is the 
case. I can reproduce the merge with the following commands:
$ git checkout -b test a02fa3303
$ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c

The merge actually generates a merge conflict but not for my
problematic file. The common ancestor of the two parents (merge base) 
is b91161554.

The merge graph is not pretty (the committers don't have a clean 
branching scheme) but I cannot spot a problem between the merge commit
and the common ancestor:
$ git log --graph --oneline a86dd40fe

Can you give me a hint how to debug this merge further? How can I 
understand why Git picked a certain version of a file in a merge?

I am using Git 2.16.2 on Linux.

Thanks,
Lars

Re: How to debug a "git merge"?

[Derrick Stolee]

On 3/14/2018 12:56 PM, Lars Schneider wrote:
> Hi,
>
> I am investigating a Git merge (a86dd40fe) in which an older version of
> a file won over the newer version. I try to understand why this is the
> case. I can reproduce the merge with the following commands:
> $ git checkout -b test a02fa3303
> $ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c
>
> The merge actually generates a merge conflict but not for my
> problematic file. The common ancestor of the two parents (merge base)
> is b91161554.
>
> The merge graph is not pretty (the committers don't have a clean
> branching scheme) but I cannot spot a problem between the merge commit
> and the common ancestor:
> $ git log --graph --oneline a86dd40fe

Have you tried `git log --graph --oneline --simplify-merges -- path` to 
see what changes and merges involved the file? I find that view to be 
very helpful (while the default history simplification can hide things). 
In particular, if there was a change that was reverted in one side and 
not another, we could find out.

You could also use the "A...B" to check your two commits for merging, 
and maybe add "--boundary".

>
> Can you give me a hint how to debug this merge further? How can I
> understand why Git picked a certain version of a file in a merge?
>
> I am using Git 2.16.2 on Linux.
>
> Thanks,
> Lars

Re: How to debug a "git merge"?

[Lars Schneider]

> On 14 Mar 2018, at 18:02, Derrick Stolee <stolee@gmail.com> wrote:
> 
> On 3/14/2018 12:56 PM, Lars Schneider wrote:
>> Hi,
>> 
>> I am investigating a Git merge (a86dd40fe) in which an older version of
>> a file won over the newer version. I try to understand why this is the
>> case. I can reproduce the merge with the following commands:
>> $ git checkout -b test a02fa3303
>> $ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c
>> 
>> The merge actually generates a merge conflict but not for my
>> problematic file. The common ancestor of the two parents (merge base)
>> is b91161554.
>> 
>> The merge graph is not pretty (the committers don't have a clean
>> branching scheme) but I cannot spot a problem between the merge commit
>> and the common ancestor:
>> $ git log --graph --oneline a86dd40fe
> 
> Have you tried `git log --graph --oneline --simplify-merges -- path` to see what changes and merges involved the file? I find that view to be very helpful (while the default history simplification can hide things). In particular, if there was a change that was reverted in one side and not another, we could find out.

Thanks for this tip! Unfortunately, this only confirms my current view:

### First parent
$ git log --graph --oneline --simplify-merges a02fa3303 -- path/to/problem
* 4e47a10c7 <-- old version
* 01f01f61c 

### Second parent
$ git log --graph --oneline --simplify-merges c1b82995c -- path/to/problem
* 590e52ed1 <-- new version
* 8e598828d 
* ad4e9034b 
* 4e47a10c7 
* 01f01f61c 

### Merge
$ git log --graph --oneline --simplify-merges a86dd40fe -- path/to/problem
*   a86dd40fe <-- old version ?!?! That's the problem!
|\
| * 590e52ed1 <-- new version
| * 8e598828d
| * ad4e9034b
|/
* 4e47a10c7 <-- old version
* 01f01f61c


> You could also use the "A...B" to check your two commits for merging, and maybe add "--boundary".

$ git diff --boundary a02fa3303...c1b82995c -- path/to/problem

This looks like the correct diff. The "new version" is mark as +/add/green in the diff.

Does this make any sense to you?

Thank you,
Lars

Re: How to debug a "git merge"?

[Derrick Stolee]

On 3/14/2018 4:53 PM, Lars Schneider wrote:
>> On 14 Mar 2018, at 18:02, Derrick Stolee <stolee@gmail.com> wrote:
>>
>> On 3/14/2018 12:56 PM, Lars Schneider wrote:
>>> Hi,
>>>
>>> I am investigating a Git merge (a86dd40fe) in which an older version of
>>> a file won over the newer version. I try to understand why this is the
>>> case. I can reproduce the merge with the following commands:
>>> $ git checkout -b test a02fa3303
>>> $ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c
>>>
>>> The merge actually generates a merge conflict but not for my
>>> problematic file. The common ancestor of the two parents (merge base)
>>> is b91161554.
>>>
>>> The merge graph is not pretty (the committers don't have a clean
>>> branching scheme) but I cannot spot a problem between the merge commit
>>> and the common ancestor:
>>> $ git log --graph --oneline a86dd40fe
>> Have you tried `git log --graph --oneline --simplify-merges -- path` to see what changes and merges involved the file? I find that view to be very helpful (while the default history simplification can hide things). In particular, if there was a change that was reverted in one side and not another, we could find out.
> Thanks for this tip! Unfortunately, this only confirms my current view:
>
> ### First parent
> $ git log --graph --oneline --simplify-merges a02fa3303 -- path/to/problem
> * 4e47a10c7 <-- old version
> * 01f01f61c
>
> ### Second parent
> $ git log --graph --oneline --simplify-merges c1b82995c -- path/to/problem
> * 590e52ed1 <-- new version
> * 8e598828d
> * ad4e9034b
> * 4e47a10c7
> * 01f01f61c
>
> ### Merge
> $ git log --graph --oneline --simplify-merges a86dd40fe -- path/to/problem
> *   a86dd40fe <-- old version ?!?! That's the problem!
> |\
> | * 590e52ed1 <-- new version
> | * 8e598828d
> | * ad4e9034b
> |/
> * 4e47a10c7 <-- old version
> * 01f01f61c
>
>
>> You could also use the "A...B" to check your two commits for merging, and maybe add "--boundary".
> $ git diff --boundary a02fa3303...c1b82995c -- path/to/problem
>
> This looks like the correct diff. The "new version" is mark as +/add/green in the diff.
>
> Does this make any sense to you?
>
> Thank you,
> Lars

I'm sorry for the delay on this, but in my experience in helping 
customers saying "why doesn't my commit/change appear in the history of 
a file?" is because someone resolved merge conflicts by just taking one 
side instead of doing a real merge (such as using -Xours). The only 
solution is to re-apply the original change again and talk to the user 
who incorrectly merged, to prevent future occurrences. These things 
rarely happen to teams who use pull requests that require review, since 
the diff would be problematic. There are still a lot of teams that push 
directly to master, though.

Thanks,
-Stolee

Re: How to debug a "git merge"?

[Jeff King]

On Wed, Mar 14, 2018 at 05:56:04PM +0100, Lars Schneider wrote:

> I am investigating a Git merge (a86dd40fe) in which an older version of 
> a file won over the newer version. I try to understand why this is the 
> case. I can reproduce the merge with the following commands:
> $ git checkout -b test a02fa3303
> $ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c
> 
> The merge actually generates a merge conflict but not for my
> problematic file. The common ancestor of the two parents (merge base) 
> is b91161554.
> 
> The merge graph is not pretty (the committers don't have a clean 
> branching scheme) but I cannot spot a problem between the merge commit
> and the common ancestor:
> $ git log --graph --oneline a86dd40fe
> 
> Can you give me a hint how to debug this merge further? How can I 
> understand why Git picked a certain version of a file in a merge?

Maybe a stupid question, but: did you make sure that the merge does
indeed pick the wrong version of the file? The other option is that
somebody mistakenly did a "checkout --ours" or similar while resolving
the conflict.

If the wrong file is indeed picked by the merge, then you may want to
try switching merge drivers. E.g., "-s resolve" is a bit simpler and
stupider than the default merge-recursive. If the problem goes away,
then we know it's a possible bug in merge-recursive (or maybe a
confusing implication of its strategy). If the problem is still there
with "resolve", then at least it may be easier to debug. ;)

-Peff

Re: How to debug a "git merge"?

[Lars Schneider]

> On 14 Mar 2018, at 23:20, Jeff King <peff@peff.net> wrote:
> 
> On Wed, Mar 14, 2018 at 05:56:04PM +0100, Lars Schneider wrote:
> 
>> I am investigating a Git merge (a86dd40fe) in which an older version of 
>> a file won over the newer version. I try to understand why this is the 
>> case. I can reproduce the merge with the following commands:
>> $ git checkout -b test a02fa3303
>> $ GIT_MERGE_VERBOSITY=5 git merge --verbose c1b82995c
>> 
>> The merge actually generates a merge conflict but not for my
>> problematic file. The common ancestor of the two parents (merge base) 
>> is b91161554.
>> 
>> The merge graph is not pretty (the committers don't have a clean 
>> branching scheme) but I cannot spot a problem between the merge commit
>> and the common ancestor:
>> $ git log --graph --oneline a86dd40fe
>> 
>> Can you give me a hint how to debug this merge further? How can I 
>> understand why Git picked a certain version of a file in a merge?
> 
> Maybe a stupid question, but: did you make sure that the merge does
> indeed pick the wrong version of the file? The other option is that
> somebody mistakenly did a "checkout --ours" or similar while resolving
> the conflict.

No stupid question at all. That's exactly what they did and I did not
realize it! Thank you!

Next time I won't stumble over this. I wonder if this is a common enough
problem to do something about it? For instance what if `git log` (or just
`git show`) has an option `--verify-merges` or `--reenact-merges` or 
something? This option would perform a "default recursive merge" and 
show the diff between the actual merge and the default merge?

In the most common case there is no diff. If there are merge conflicts
then we would just show the conflicting files. If there is no merge
conflict for a file *but* a difference then we would show it. I think
this would have helped me to realize this kind of problem earlier.

Would that option make sense to you?

Thanks,
Lars

Re: How to debug a "git merge"?

[Jeff King]

On Thu, Mar 15, 2018 at 10:51:27AM +0100, Lars Schneider wrote:

> > Maybe a stupid question, but: did you make sure that the merge does
> > indeed pick the wrong version of the file? The other option is that
> > somebody mistakenly did a "checkout --ours" or similar while resolving
> > the conflict.
> 
> No stupid question at all. That's exactly what they did and I did not
> realize it! Thank you!

Oh good. :) I have run across this same situation before myself, which
is why I thought to ask about it.

> Next time I won't stumble over this. I wonder if this is a common enough
> problem to do something about it? For instance what if `git log` (or just
> `git show`) has an option `--verify-merges` or `--reenact-merges` or 
> something? This option would perform a "default recursive merge" and 
> show the diff between the actual merge and the default merge?
> 
> In the most common case there is no diff. If there are merge conflicts
> then we would just show the conflicting files. If there is no merge
> conflict for a file *but* a difference then we would show it. I think
> this would have helped me to realize this kind of problem earlier.
> 
> Would that option make sense to you?

Yes, it's absolutely a good idea, and a frequent wish-list item. The
problem is that it's tricky to implement. The only working patches I
know of were Thomas Rast's "--remerge-diff" from 2014:

  https://public-inbox.org/git/cover.1409860234.git.tr@thomasrast.ch/

The tricky thing is that you have to actually re-run the diff, and we
don't know what options were used for the original diff (e.g., what
strategy). And also, merge-recursive really wants to have a valid
working tree (though I think maybe that has gotten better over the
years).

But even with those warts, I still found it useful in many cases.

-Peff

Re: How to debug a "git merge"?

[Elijah Newren]

On Thu, Mar 15, 2018 at 7:09 AM, Jeff King <peff@peff.net> wrote:
> On Thu, Mar 15, 2018 at 10:51:27AM +0100, Lars Schneider wrote:
>> Next time I won't stumble over this. I wonder if this is a common enough
>> problem to do something about it? For instance what if `git log` (or just
>> `git show`) has an option `--verify-merges` or `--reenact-merges` or
>> something? This option would perform a "default recursive merge" and
>> show the diff between the actual merge and the default merge?
>>
>> In the most common case there is no diff. If there are merge conflicts
>> then we would just show the conflicting files. If there is no merge
>> conflict for a file *but* a difference then we would show it. I think
>> this would have helped me to realize this kind of problem earlier.
>>
>> Would that option make sense to you?
>
> Yes, it's absolutely a good idea, and a frequent wish-list item. The
> problem is that it's tricky to implement. The only working patches I
> know of were Thomas Rast's "--remerge-diff" from 2014:
>
>   https://public-inbox.org/git/cover.1409860234.git.tr@thomasrast.ch/
>
> The tricky thing is that you have to actually re-run the diff, and we
> don't know what options were used for the original diff (e.g., what
> strategy). And also, merge-recursive really wants to have a valid
> working tree (though I think maybe that has gotten better over the
> years).

merge-recursive's working tree reliance has gotten better, though work
remains.  I'll get back to it, eventually.  :-)

There's also a third issue which I think makes this feature difficult
(though not intractable): some conflict types might be difficult to
represent in such a diff.  For those interested in the details

In particular, at the basic level I'm thinking of conflict types such
as modify/delete, rename/rename(1to2), and directory/file conflicts.
It appears Thomas Rast's series would resolve all of these by doing a
two-way diff against /dev/null for each un-paired but conflicted path.
Perhaps that's good enough, but it does lose a lot of semantic content
(is this conflict because this was an add/add conflict where one side
was an empty file, or did one side of history delete the path, or did
the other side of history rename the path elsewhere, or did one side
have a directory in the way?).  This semantic content isn't stored
anywhere in merge-recursive; it's simply printed with a "CONFLICT"
message to the screen when it runs across the path, so there'd be a
little work involved in re-plumbing that.  Also, directory rename
detection adds implicit renames that add a few more conflict types
that could fall in this set ("Existing file/dir in the way of implicit
rename for path", and "Cannot map more than one path to %s from
implicit renames").  There's also one it adds that maybe could map to
this type of solution but it'd take some work ("directory rename
split").

To make matters worse, there are also multiply-conflicted types that
are possible that may make matters even more difficult for a --remerge
diff.  For example, the files involved in a rename/rename(2to1) could
also be involved in a rename/delete.  Another that came up on the list
yesterday is that both rename/add and rename/delete conflicts can
happen on the same pair.  I think merge-recursive doesn't even detect
these and mis-handles both in the normal case, the --remerge-diff
would just be even more tricky.  There are also a couple new
multiple-conflict-types that are handled fine for the normal merge
case but might make a remerge-diff more complex.  Two examples I can
think of are (1) if a path is involved in a rename/delete, and the
renamed side should get a transitive renaming due to the directory it
getting renamed into having been renamed on the other side of history,
but there's a path in the way of the implicit rename, or (2) if a path
is involved in a modify/delete and should be implicitly renamed due to
a directory rename but there are several paths that need to be
implicitly renamed to the same location.

However, to end on a bit of good news, this week's decision to make
rename/rename(2to1) and rename/add both behave more like add/add will
simplify matters for implementing --remerge-diff.