[PATCH 00/48] Handling more corner cases in merge-recursive.c

[PATCH 00/48] Handling more corner cases in merge-recursive.c

[Elijah Newren]

Well, this took me a LOT longer than I expected, but I now have a
(monster) patch series designed to fix various corner cases in
merge-recursive.c.  For those waiting for the 1.7.6 freeze to end,
reading this patch series makes for some great mental gymnastics.  :-)

You might be surprised by just how corner cases we have and the types of
bugs that can be triggered (including deleting untracked files).  This
series includes testcases for the unnecessary file update issues[1]
triggered by my series from last fall[2] and also a testcase that had
been regressed by my earlier (and now reverted) bandaids designed to fix
the unnecessary file update issue[3] (to make sure we fix both those
issues).  The current series aims to fix these problems using the
strategy Junio outlined[4].  However, this series is about a lot more
than just those issues.

[1] http://thread.gmane.org/gmane.comp.version-control.git/167800
[2] http://thread.gmane.org/gmane.comp.version-control.git/156559
[3] http://thread.gmane.org/gmane.comp.version-control.git/172617
[4] http://thread.gmane.org/gmane.comp.version-control.git/169209

You can note that I did not fix all the testcases I added.  I consider
some cases either unfixable or not worth fixing (see [5] for mailing
list discussion about the main such case).

[5] http://thread.gmane.org/gmane.comp.version-control.git/157591

However, there is one large class of problems that I think is fixable,
I'm just not sure whether it is worth fixing.  git's rename detection
optimization of only considering files that exist on one side of the
diff but not the other causes issues with merges (undetected conflicts,
spurious conflicts, or merged cleanly but wrongly due to deleting files
that should be present).  To fix these cases, we would need some way of
including rewritten files as potential rename candidates, meaning we
would also need to somehow check for rewritten files.  (I refer to these
as rename/add-source situations).  The potential problem with applying
such a fix is that it could make merges significantly more costly in the
common case, in order to fix some relatively rare problems that might
occur.  I don't know whether it is justified, especially since git's
merging can't be expected to know how to merge everything; at some point
the user is responsible for checking the result.  Patch 32 does add some
comments to the code if anyone does decide they want to look into this,
though.

Please test!  I apologize for the earlier regressions I caused in
merge-recursive.  I've tried to be MUCH more careful this time, but the
changes are numerous and much more invasive too.  I'm trying to put this
series through the testing wringer and will continue to do so, but extra
testing from others would be great.

Anyway, if you've read this far...thanks!  



Elijah Newren (48):

In patches 1-12, I add a bunch of corner case tests.  Since there are so
many I want to add and I think putting them all in t6036 would be a bit
unwieldy, I split the tests.  Since t6036 currently only contains
testcases that involve criss-cross merges, I mark it for such usage, and
create t6039 for dealing with corner cases involving renames without
criss-cross merges.

  [01] t6039: Add a testcase where git deletes an untracked file
  [02] t6039: Add failing testcase for rename/modify/add-source conflict
  [03] t6039: Add a pair of cases where undetected renames cause issues
  [04] t6039: Add a testcase where undetected rename causes silent file deletion
  [05] t6039: Add tests for content issues with modify/rename/directory conflicts
  [06] t6039: Add failing testcases for rename/rename/add-{source,dest} conflicts
  [07] t6039: Ensure rename/rename conflicts leave index and workdir in sane state
  [08] t6036: Add differently resolved modify/delete conflict in criss-cross test
  [09] t6036: criss-cross with weird content can fool git into clean merge
  [10] t6036: tests for criss-cross merges with various directory/file conflicts
  [11] t6036: criss-cross w/ rename/rename(1to2)/modify+rename/rename(2to1)/modify
  [12] t6036: criss-cross + rename/rename(1to2)/add-source + modify/modify

In patches 13-14, I add two tests that Junio reverted from master
(6db41050) which we would still like fixed, and a test that my previous
band-aids had broken to make sure we don't break it again.

  [13] t6022: Remove unnecessary untracked files to make test cleaner
  [14] t6022: New tests checking for unnecessary updates of files
  [15] t6022: Add testcase for merging a renamed file with a simple change

In patches 16-19, I'm just doing minor cleanups.  None of these are
critical; I'll be happy to drop any of them.

  [16] merge-recursive: Make BUG message more legible by adding a newline
  [17] merge-recursive: Correct a comment
  [18] merge-recursive: Mark some diff_filespec struct arguments const
  [19] merge-recursive: Remember to free generated unique path names

Patches 20-22 are small but important bug fixes.  Some of these might
make sense to be pulled to maint independent of the other changes.

  [20] merge-recursive: Avoid working directory changes during recursive case
  [21] merge-recursive: Fix recursive case with D/F conflict via add/add conflict
  [22] merge-recursive: Fix sorting order and directory change assumptions

Patches 23-29 are where I try to implement what I understood Junio
wanted merge-recursive to do to avoid so many lstat() and other
unnecessary working directory changes (see link [4] above).

  [23] merge-recursive: Fix code checking for D/F conflicts still being present
  [24] merge-recursive: Save D/F conflict filenames instead of unlinking them
  [25] merge-recursive: Split was_tracked() out of would_lose_untracked()
  [26] merge-recursive: Allow make_room_for_path() to remove D/F entries
  [27] merge-recursive: Consolidate different update_stages functions
  [28] merge-recursive: Split update_stages_and_entry; only update stages at end
  [29] merge-recursive: When we detect we can skip an update, actually skip it

Patch 30 fixes the deletion of an untracked file; I believe it has the
highest ratio of commit message length to patch size, since it has to
explain why we have modified a certain line of code several times and
still didn't have it quite right.

  [30] merge-recursive: Fix deletion of untracked file in rename/delete conflicts

The commit message for patch 31 seems like there's some good zombie joke
in there that someone cleverer than myself ought to be able to come up
with.  Interestingly, our code for rename/rename (2to1) conflicts was
never executed, and would trigger segfaults had there been a way to
execute it.

  [31] merge-recursive: Make dead code for rename/rename(2to1) conflicts undead

There are a few cases where git's rename detection optimization of only
considering files that exist on one side of the diff but not the other
causes issues with merges (undetected conflicts, spurious conflicts, or
merged cleanly but wrongly due to deleting files that should be
present).  To fix these cases, we would need some way of including
rewritten files as potential rename candidates, meaning we would also
need to somehow check for rewritten files.  My series does not address
these cases (I'm unsure whether the performance hit would be worth it).
Instead, I simply have a half-dozen relevant test cases and add some
comments to the code about some things that would need to be considered
if we ever want to tackle these cases.

  [32] merge-recursive: Add comments about handling rename/add-source cases

Patch 33 was specifically written because of a rename/directory conflict
testcase, but tries to worry about three-way content merges in
combination with renames and D/F conflicts and possible criss-cross
merges and getting the appropriate index and working directory contents.

  [33] merge-recursive: Improve handling of rename target vs. directory addition

Patch 34 is an attempted simplification of merge-recursive.c.  We really
don't need process_df_entry(), since we can simply iterate through the
unmerged entries in the order we need.  That can make it easier to keep
track of things, which is particularly important for the rename/rename
fixes below...

  [34] merge-recursive: Consolidate process_entry() and process_df_entry()

Patches 35-48 try to fix our handling of rename/rename conflicts (both
one file being renamed to two or two files being renamed to one).  In
such cases, we need to consider that the renames may not be the only
change; there could also be modifications requiring a three-way content
merge.  Further, as with all other rename cases, we need to consider the
recursive case (o->call_depth > 0) and potential D/F conflicts.  There
is also the possibility of rename/add-source issues for each side of the
rename, though we currently can't detect those anyway.  The
rename_rename_1to2 case is especially interesting, as we also have to
worry about each rename potentially also being involved in rename/delete
and rename/add-dest conflicts.  This all makes handling rename/rename
conflicts a bit hairy.

  [35] merge-recursive: Cleanup and consolidation of rename_conflict_info
  [36] merge-recursive: Provide more info in conflict markers with file renames
  [37] merge-recursive: Fix modify/delete resolution in the recursive case
  [38] merge-recursive: Introduce a merge_file convenience function
  [39] merge-recursive: Fix rename/rename(1to2) resolution for virtual merge base
  [40] merge-recursive: Small cleanups for conflict_rename_rename_1to2
  [41] merge-recursive: Defer rename/rename(2to1) handling until process_entry
  [42] merge-recursive: Record more data needed for merging with dual renames
  [43] merge-recursive: Create function for merging with branchname:file markers
  [44] merge-recursive: Consider modifications in rename/rename(2to1) conflicts
  [45] merge-recursive: Make modify/delete handling code reusable
  [46] merge-recursive: Have conflict_rename_delete reuse modify/delete code
  [47] merge-recursive: add handling for rename/rename/add-dest/add-dest
  [48] merge-recursive: Fix working copy handling for rename/rename/add/add

 merge-recursive.c                    |  985 +++++++++++++++++++++-------------
 merge-recursive.h                    |    1 +
 t/t6020-merge-df.sh                  |   26 +-
 t/t6022-merge-rename.sh              |  102 ++++-
 t/t6036-recursive-corner-cases.sh    |  502 ++++++++++++++++-
 t/t6039-merge-rename-corner-cases.sh |  556 +++++++++++++++++++
 6 files changed, 1752 insertions(+), 420 deletions(-)
 create mode 100755 t/t6039-merge-rename-corner-cases.sh

-- 
1.7.6.rc0.62.g2d69f

[PATCH 01/48] t6039: Add a testcase where git deletes an untracked file

[Elijah Newren]

Current git will nuke an untracked file during a rename/delete conflict if
(a) there is an untracked file whose name matches the source of a rename
and (b) the merge is done in a certain direction.  Add a simple testcase
demonstrating this bug.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh    |    2 +-
 t/t6039-merge-rename-corner-cases.sh |   36 ++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 1 deletions(-)
 create mode 100755 t/t6039-merge-rename-corner-cases.sh

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 871577d..319b6fa 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-test_description='recursive merge corner cases'
+test_description='recursive merge corner cases involving criss-cross merges'
 
 . ./test-lib.sh
 
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
new file mode 100755
index 0000000..5054459
--- /dev/null
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description="recursive merge corner cases w/ renames but not criss-crosses"
+# t6036 has corner cases that involve both criss-cross merges and renames
+
+. ./test-lib.sh
+
+test_expect_success 'setup rename/delete + untracked file' '
+	echo "A pretty inscription" >ring &&
+	git add ring &&
+	test_tick &&
+	git commit -m beginning &&
+
+	git branch people &&
+	git checkout -b rename-the-ring &&
+	git mv ring one-ring-to-rule-them-all &&
+	test_tick &&
+	git commit -m fullname &&
+
+	git checkout people &&
+	git rm ring &&
+	echo gollum >owner &&
+	git add owner &&
+	test_tick &&
+	git commit -m track-people-instead-of-objects &&
+	echo "Myyy PRECIOUSSS" >ring
+'
+
+test_expect_failure "Does git preserve Gollum's precious artifact?" '
+	test_must_fail git merge -s recursive rename-the-ring &&
+
+	# Make sure git did not delete an untracked file
+	test -f ring
+'
+
+test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 02/48] t6039: Add failing testcase for rename/modify/add-source conflict

[Elijah Newren]

If there is a cleanly resolvable rename/modify conflict AND there is a new
file introduced on the renamed side of the merge whose name happens to
match that of the source of the rename (but is otherwise unrelated to the
rename), then git fails to cleanly resolve the merge despite the fact that
the new file should not cause any problems.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |   39 ++++++++++++++++++++++++++++++++++
 1 files changed, 39 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 5054459..276d7dd 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -33,4 +33,43 @@ test_expect_failure "Does git preserve Gollum's precious artifact?" '
 	test -f ring
 '
 
+# Testcase setup for rename/modify/add-source:
+#   Commit A: new file: a
+#   Commit B: modify a slightly
+#   Commit C: rename a->b, add completely different a
+#
+# We should be able to merge B & C cleanly
+
+test_expect_success 'setup rename/modify/add-source conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n6\n7\n" >a &&
+	git add a &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	echo 8 >>a &&
+	git add a &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a b &&
+	echo something completely different >a &&
+	git add a &&
+	git commit -m C
+'
+
+test_expect_failure 'rename/modify/add-source conflict resolvable' '
+	git checkout B^0 &&
+
+	git merge -s recursive C^0 &&
+
+	test $(git rev-parse B:a) = $(git rev-parse b) &&
+	test $(git rev-parse C:a) = $(git rev-parse a)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 03/48] t6039: Add a pair of cases where undetected renames cause issues

[Elijah Newren]

An undetected rename can cause a silent success where a conflict should
have been detected, or can cause an erroneous conflict state where the
merge should have been resolvable.  Add testcases for both.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |   61 ++++++++++++++++++++++++++++++++++
 1 files changed, 61 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 276d7dd..f338fb4 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -72,4 +72,65 @@ test_expect_failure 'rename/modify/add-source conflict resolvable' '
 	test $(git rev-parse C:a) = $(git rev-parse a)
 '
 
+test_expect_success 'setup resolvable conflict missed if rename missed' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n" >a &&
+	echo foo >b &&
+	git add a b &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a c &&
+	echo "Completely different content" >a &&
+	git add a &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	echo 6 >>a &&
+	git add a &&
+	git commit -m C
+'
+
+test_expect_failure 'conflict caused if rename not detected' '
+	git checkout -q C^0 &&
+	git merge -s recursive B^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 0 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test 6 -eq $(wc -l < c) &&
+	test $(git rev-parse HEAD:a) = $(git rev-parse B:a) &&
+	test $(git rev-parse HEAD:b) = $(git rev-parse A:b)
+'
+
+test_expect_success 'setup conflict resolved wrong if rename missed' '
+	git reset --hard &&
+	git clean -f &&
+
+	git checkout -b D A &&
+	echo 7 >>a &&
+	git add a &&
+	git mv a c &&
+	echo "Completely different content" >a &&
+	git add a &&
+	git commit -m D &&
+
+	git checkout -b E A &&
+	git rm a &&
+	echo "Completely different content" >>a &&
+	git add a &&
+	git commit -m E
+'
+
+test_expect_failure 'missed conflict if rename not detected' '
+	git checkout -q E^0 &&
+	test_must_fail git merge -s recursive D^0
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 04/48] t6039: Add a testcase where undetected rename causes silent file deletion

[Elijah Newren]

There are cases where history should merge cleanly, and which current git
does merge cleanly despite not detecting a rename; however the merge
currently nukes files that should not be removed.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |   65 ++++++++++++++++++++++++++++++++++
 1 files changed, 65 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index f338fb4..db5560c 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -133,4 +133,69 @@ test_expect_failure 'missed conflict if rename not detected' '
 	test_must_fail git merge -s recursive D^0
 '
 
+# Tests for undetected rename/add-source causing a file to erroneously be
+# deleted (and for mishandled rename/rename(1to1) causing the same issue).
+#
+# This test uses a rename/rename(1to1)+add-source conflict (1to1 means the
+# same file is renamed on both sides to the same thing; it should trigger
+# the 1to2 logic, which it would do if the add-source didn't cause issues
+# for git's rename detection):
+#   Commit A: new file: a
+#   Commit B: rename a->b
+#   Commit C: rename a->b, add unrelated a
+
+test_expect_success 'setup undetected rename/add-source causes data loss' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n" >a &&
+	git add a &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a b &&
+	echo foobar >a &&
+	git add a &&
+	git commit -m C
+'
+
+test_expect_failure 'detect rename/add-source and preserve all data' '
+	git checkout B^0 &&
+
+	git merge -s recursive C^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test -f a &&
+	test -f b &&
+
+	test $(git rev-parse HEAD:b) = $(git rev-parse A:a) &&
+	test $(git rev-parse HEAD:a) = $(git rev-parse C:a)
+'
+
+test_expect_failure 'detect rename/add-source and preserve all data, merge other way' '
+	git checkout C^0 &&
+
+	git merge -s recursive B^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test -f a &&
+	test -f b &&
+
+	test $(git rev-parse HEAD:b) = $(git rev-parse A:a) &&
+	test $(git rev-parse HEAD:a) = $(git rev-parse C:a)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 05/48] t6039: Add tests for content issues with modify/rename/directory conflicts

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |  141 ++++++++++++++++++++++++++++++++++
 1 files changed, 141 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index db5560c..b465667 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -198,4 +198,145 @@ test_expect_failure 'detect rename/add-source and preserve all data, merge other
 	test $(git rev-parse HEAD:a) = $(git rev-parse C:a)
 '
 
+test_expect_success 'setup content merge + rename/directory conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n6\n" >file &&
+	git add file &&
+	test_tick &&
+	git commit -m base &&
+	git tag base &&
+
+	git checkout -b right &&
+	echo 7 >>file &&
+	mkdir newfile &&
+	echo junk >newfile/realfile &&
+	git add file newfile/realfile &&
+	test_tick &&
+	git commit -m right &&
+
+	git checkout -b left-conflict base &&
+	echo 8 >>file &&
+	git add file &&
+	git mv file newfile &&
+	test_tick &&
+	git commit -m left &&
+
+	git checkout -b left-clean base &&
+	echo 0 >newfile &&
+	cat file >>newfile &&
+	git add newfile &&
+	git rm file &&
+	test_tick &&
+	git commit -m left
+'
+
+test_expect_failure 'rename/directory conflict + clean content merge' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+
+	git checkout left-clean^0 &&
+
+	test_must_fail git merge -s recursive right^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 1 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
+
+	echo 0 >expect &&
+	git cat-file -p base:file >>expect &&
+	echo 7 >>expect &&
+	test_cmp expect newfile~HEAD &&
+
+	test $(git rev-parse :2:newfile) = $(git hash-object expect) &&
+
+	test -f newfile/realfile &&
+	test -f newfile~HEAD
+'
+
+test_expect_failure 'rename/directory conflict + content merge conflict' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+
+	git checkout left-conflict^0 &&
+
+	test_must_fail git merge -s recursive right^0 &&
+
+	test 4 -eq $(git ls-files -s | wc -l) &&
+	test 3 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
+
+	git cat-file -p left-conflict:newfile >left &&
+	git cat-file -p base:file    >base &&
+	git cat-file -p right:file   >right &&
+	test_must_fail git merge-file \
+		-L "HEAD:newfile" \
+		-L "" \
+		-L "right^0:file" \
+		left base right &&
+	test_cmp left newfile~HEAD &&
+
+	test $(git rev-parse :1:newfile) = $(git rev-parse base:file) &&
+	test $(git rev-parse :2:newfile) = $(git rev-parse left-conflict:newfile) &&
+	test $(git rev-parse :3:newfile) = $(git rev-parse right:file) &&
+
+	test -f newfile/realfile &&
+	test -f newfile~HEAD
+'
+
+test_expect_success 'setup content merge + rename/directory conflict w/ disappearing dir' '
+	git reset --hard &&
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	mkdir sub &&
+	printf "1\n2\n3\n4\n5\n6\n" >sub/file &&
+	git add sub/file &&
+	test_tick &&
+	git commit -m base &&
+	git tag base &&
+
+	git checkout -b right &&
+	echo 7 >>sub/file &&
+	git add sub/file &&
+	test_tick &&
+	git commit -m right &&
+
+	git checkout -b left base &&
+	echo 0 >newfile &&
+	cat sub/file >>newfile &&
+	git rm sub/file &&
+	mv newfile sub &&
+	git add sub &&
+	test_tick &&
+	git commit -m left
+'
+
+test_expect_success 'disappearing dir in rename/directory conflict handled' '
+	git reset --hard &&
+	git clean -fdqx &&
+
+	git checkout left^0 &&
+
+	git merge -s recursive right^0 &&
+
+	test 1 -eq $(git ls-files -s | wc -l) &&
+	test 0 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	echo 0 >expect &&
+	git cat-file -p base:sub/file >>expect &&
+	echo 7 >>expect &&
+	test_cmp expect sub &&
+
+	test -f sub
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 06/48] t6039: Add failing testcases for rename/rename/add-{source,dest} conflicts

[Elijah Newren]

Add testcases that cover three failures with current git merge, all
involving renaming one file on both sides of history:

Case 1:
If a single file is renamed to two different filenames on different sides
of history, there should be a conflict.  Adding a new file on one of those
sides of history whose name happens to match the rename source should not
cause the merge to suddenly succeed.

Case 2:
If a single file is renamed on both sides of history but renamed
identically, there should not be a conflict.  This works fine.  However,
if one of those sides also added a new file that happened to match the
rename source, then that file should be left alone.  Currently, the
rename/rename conflict handling causes that new file to become untracked.

Case 3:
If a single file is renamed to two different filenames on different sides
of history, there should be a conflict.  This works currently.  However,
if those renames also involve rename/add conflicts (i.e. there are new
files on one side of history that match the destination of the rename of
the other side of history), then the resulting conflict should be recorded
in the index, showing that there were multiple files with a given filename.
Currently, git silently discards one of file versions.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |  112 ++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index b465667..fd8337f 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -339,4 +339,116 @@ test_expect_success 'disappearing dir in rename/directory conflict handled' '
 	test -f sub
 '
 
+# Testcase setup for rename-rename-add-1:
+#   Commit A: new file: a
+#   Commit B: rename a->b
+#   Commit C: rename a->c, add completely different a
+#
+# Merging of B & C should NOT be clean; there's a rename/rename conflict
+
+test_expect_success 'setup rename/rename(1to2)/add-source conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n6\n7\n" >a &&
+	git add a &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a c &&
+	echo something completely different >a &&
+	git add a &&
+	git commit -m C
+'
+
+test_expect_failure 'detect conflict with rename/rename(1to2)/add-source merge' '
+	git checkout B^0 &&
+
+	test_must_fail git merge -s recursive C^0 &&
+
+	test -f a &&
+	test -f b &&
+	test -f c
+'
+
+test_expect_success 'setup rename/rename(1to2)/add-source resolvable conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	>a &&
+	git add a &&
+	test_tick &&
+	git commit -m base &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	test_tick &&
+	git commit -m one &&
+
+	git checkout -b C A &&
+	git mv a b &&
+	echo important-info >a &&
+	test_tick &&
+	git commit -m two
+'
+
+test_expect_failure 'rename/rename/add-source still tracks new a file' '
+	git checkout C^0 &&
+	git merge -s recursive B^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l)
+'
+
+test_expect_success 'setup rename/rename(1to2)/add-dest conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	echo stuff >a &&
+	git add a &&
+	test_tick &&
+	git commit -m base &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	echo precious-data >c &&
+	git add c &&
+	test_tick &&
+	git commit -m one &&
+
+	git checkout -b C A &&
+	git mv a c &&
+	echo important-info >b &&
+	git add b &&
+	test_tick &&
+	git commit -m two
+'
+
+test_expect_failure 'rename/rename/add-dest merge still knows about conflicting file versions' '
+	git checkout C^0 &&
+	test_must_fail git merge -s recursive B^0 &&
+
+	test 5 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u b | wc -l) &&
+	test 2 -eq $(git ls-files -u c | wc -l) &&
+
+	test $(git rev-parse :2:b) = $(git rev-parse C:b) &&
+	test $(git rev-parse :3:b) = $(git rev-parse B:b) &&
+	test $(git rev-parse :2:c) = $(git rev-parse C:c) &&
+	test $(git rev-parse :3:c) = $(git rev-parse B:c)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 07/48] t6039: Ensure rename/rename conflicts leave index and workdir in sane state

[Elijah Newren]

rename/rename conflicts, both with one file being renamed to two different
files and with two files being renamed to the same file, should leave the
index and the working copy in a sane state with appropriate conflict
recording, auxiliary files, etc.  Git seems to handle one of the two cases
alright, but has some problems with the two files being renamed to one
case.  Add tests for both cases.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6039-merge-rename-corner-cases.sh |   93 ++++++++++++++++++++++++++++++++++
 1 files changed, 93 insertions(+), 0 deletions(-)

diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index fd8337f..06c7ea5 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -451,4 +451,97 @@ test_expect_failure 'rename/rename/add-dest merge still knows about conflicting
 	test $(git rev-parse :3:c) = $(git rev-parse B:c)
 '
 
+test_expect_success 'setup simple rename/rename (1to2) conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	echo stuff >a &&
+	git add a &&
+	test_tick &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	test_tick &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a c &&
+	test_tick &&
+	git commit -m C
+'
+
+test_expect_success 'merge has correct working tree contents' '
+	git checkout C^0 &&
+
+	test_must_fail git merge -s recursive B^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 3 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test -f b &&
+	test -f c
+'
+
+# Test for all kinds of things that can go wrong with rename/rename (2to1):
+#   Commit A: new files: a & b
+#   Commit B: rename a->c, modify b
+#   Commit C: rename b->c, modify a
+#
+# Merging of B & C should NOT be clean.  Questions:
+#   * Both a & b should be removed by the merge; are they?
+#   * The two c's should contain modifications to a & b; do they?
+#   * The index should contain two files, both for c; does it?
+#   * The working copy should have two files, both of form c~<unique>; does it?
+#   * Nothing else should be present.  Is anything?
+
+test_expect_success 'setup rename/rename (2to1) + modify/modify' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n" >a &&
+	printf "5\n4\n3\n2\n1\n" >b &&
+	git add a b &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a c &&
+	echo 0 >>b &&
+	git add b &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv b c &&
+	echo 6 >>a &&
+	git add a &&
+	git commit -m C
+'
+
+test_expect_failure 'handle rename/rename (2to1) conflict correctly' '
+	git checkout B^0 &&
+
+	test_must_fail git merge -s recursive C^0 >out &&
+	grep "CONFLICT (rename/rename)" out &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+	test 2 -eq $(git ls-files -u c | wc -l) &&
+	test 3 -eq $(git ls-files -o | wc -l) &&
+
+	test ! -f a &&
+	test ! -f b &&
+	test -f c~HEAD &&
+	test -f c~C^0 &&
+
+	test $(git hash-object c~HEAD) = $(git rev-parse C:a) &&
+	test $(git hash-object c~C^0) = $(git rev-parse B:b)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 08/48] t6036: Add differently resolved modify/delete conflict in criss-cross test

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh |   83 +++++++++++++++++++++++++++++++++++++
 1 files changed, 83 insertions(+), 0 deletions(-)

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 319b6fa..52d2ecf 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -231,4 +231,87 @@ test_expect_success 'git detects differently handled merges conflict' '
 	test $(git rev-parse :1:new_a) = $(git hash-object merged)
 '
 
+#
+# criss-cross + modify/delete:
+#
+#      B   D
+#      o---o
+#     / \ / \
+#  A o   X   ? F
+#     \ / \ /
+#      o---o
+#      C   E
+#
+#   Commit A: file with contents 'A\n'
+#   Commit B: file with contents 'B\n'
+#   Commit C: file not present
+#   Commit D: file with contents 'B\n'
+#   Commit E: file not present
+#
+# Now, when we merge commits D & E, does git detect the conflict?
+
+test_expect_success 'setup criss-cross + modify/delete resolved differently' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	echo A >file &&
+	git add file &&
+	test_tick &&
+	git commit -m A &&
+
+	git branch B &&
+	git checkout -b C &&
+	git rm file &&
+	test_tick &&
+	git commit -m C &&
+
+	git checkout B &&
+	echo B >file &&
+	git add file &&
+	test_tick &&
+	git commit -m B &&
+
+	git checkout B^0 &&
+	test_must_fail git merge C &&
+	echo B >file &&
+	git add file &&
+	test_tick &&
+	git commit -m D &&
+	git tag D &&
+
+	git checkout C^0 &&
+	test_must_fail git merge B &&
+	git rm file &&
+	test_tick &&
+	git commit -m E &&
+	git tag E
+'
+
+test_expect_failure 'git detects conflict merging criss-cross+modify/delete' '
+	git checkout D^0 &&
+
+	test_must_fail git merge -s recursive E^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+
+	test $(git rev-parse :1:file) = $(git rev-parse master:file) &&
+	test $(git rev-parse :2:file) = $(git rev-parse B:file)
+'
+
+test_expect_failure 'git detects conflict merging criss-cross+modify/delete, reverse direction' '
+	git reset --hard &&
+	git checkout E^0 &&
+
+	test_must_fail git merge -s recursive D^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+
+	test $(git rev-parse :1:file) = $(git rev-parse master:file) &&
+	test $(git rev-parse :3:file) = $(git rev-parse B:file)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 09/48] t6036: criss-cross with weird content can fool git into clean merge

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh |   83 +++++++++++++++++++++++++++++++++++++
 1 files changed, 83 insertions(+), 0 deletions(-)

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 52d2ecf..dab52a4 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -314,4 +314,87 @@ test_expect_failure 'git detects conflict merging criss-cross+modify/delete, rev
 	test $(git rev-parse :3:file) = $(git rev-parse B:file)
 '
 
+#
+# criss-cross + modify/modify with very contrived file contents:
+#
+#      B   D
+#      o---o
+#     / \ / \
+#  A o   X   ? F
+#     \ / \ /
+#      o---o
+#      C   E
+#
+#   Commit A: file with contents 'A\n'
+#   Commit B: file with contents 'B\n'
+#   Commit C: file with contents 'C\n'
+#   Commit D: file with contents 'D\n'
+#   Commit E: file with contents:
+#      <<<<<<< Temporary merge branch 1
+#      C
+#      =======
+#      B
+#      >>>>>>> Temporary merge branch 2
+#
+# Now, when we merge commits D & E, does git detect the conflict?
+
+test_expect_success 'setup differently handled merges of content conflict' '
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	echo A >file &&
+	git add file &&
+	test_tick &&
+	git commit -m A &&
+
+	git branch B &&
+	git checkout -b C &&
+	echo C >file &&
+	git add file &&
+	test_tick &&
+	git commit -m C &&
+
+	git checkout B &&
+	echo B >file &&
+	git add file &&
+	test_tick &&
+	git commit -m B &&
+
+	git checkout B^0 &&
+	test_must_fail git merge C &&
+	echo D >file &&
+	git add file &&
+	test_tick &&
+	git commit -m D &&
+	git tag D &&
+
+	git checkout C^0 &&
+	test_must_fail git merge B &&
+	cat <<EOF >file &&
+<<<<<<< Temporary merge branch 1
+C
+=======
+B
+>>>>>>> Temporary merge branch 2
+EOF
+	git add file &&
+	test_tick &&
+	git commit -m E &&
+	git tag E
+'
+
+test_expect_failure 'git detects conflict w/ criss-cross+contrived resolution' '
+	git checkout D^0 &&
+
+	test_must_fail git merge -s recursive E^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 3 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test $(git rev-parse :2:file) = $(git rev-parse D:file) &&
+	test $(git rev-parse :3:file) = $(git rev-parse E:file)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 10/48] t6036: tests for criss-cross merges with various directory/file conflicts

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh |  149 +++++++++++++++++++++++++++++++++++++
 1 files changed, 149 insertions(+), 0 deletions(-)

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index dab52a4..4993f67 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -397,4 +397,153 @@ test_expect_failure 'git detects conflict w/ criss-cross+contrived resolution' '
 	test $(git rev-parse :3:file) = $(git rev-parse E:file)
 '
 
+#
+# criss-cross + d/f conflict via add/add:
+#   Commit A: Neither file 'a' nor directory 'a/' exist.
+#   Commit B: Introduce 'a'
+#   Commit C: Introduce 'a/file'
+# Two different later cases:
+#   Commit D1: Merge B & C, keeping 'a' and deleting 'a/'
+#   Commit E1: Merge B & C, deleting 'a' but keeping 'a/file'
+#
+#   Commit D2: Merge B & C, keeping a modified 'a' and deleting 'a/'
+#   Commit E2: Merge B & C, deleting 'a' but keeping a modified 'a/file'
+#
+#   Note: D == D1.
+# Finally, someone goes to merge D1&E1 or D1&E2 or D2&E1.  What happens?
+#
+#      B   D1 or D2
+#      o---o
+#     / \ / \
+#  A o   X   ? F
+#     \ / \ /
+#      o---o
+#      C   E1 or E2
+#
+
+test_expect_success 'setup differently handled merges of directory/file conflict' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	>irrelevant-file &&
+	git add irrelevant-file &&
+	test_tick &&
+	git commit -m A &&
+
+	git branch B &&
+	git checkout -b C &&
+	mkdir a &&
+	echo 10 >a/file &&
+	git add a/file &&
+	test_tick &&
+	git commit -m C &&
+
+	git checkout B &&
+	echo 5 >a &&
+	git add a &&
+	test_tick &&
+	git commit -m B &&
+
+	git checkout B^0 &&
+	test_must_fail git merge C &&
+	git clean -f &&
+	rm -rf a/ &&
+	echo 5 >a &&
+	git add a &&
+	test_tick &&
+	git commit -m D &&
+	git tag D &&
+
+	git checkout C^0 &&
+	test_must_fail git merge B &&
+	git clean -f &&
+	git rm --cached a &&
+	echo 10 >a/file &&
+	git add a/file &&
+	test_tick &&
+	git commit -m E1 &&
+	git tag E1 &&
+
+	git checkout C^0 &&
+	test_must_fail git merge B &&
+	git clean -f &&
+	git rm --cached a &&
+	printf "10\n11\n" >a/file &&
+	git add a/file &&
+	test_tick &&
+	git commit -m E2 &&
+	git tag E2
+'
+
+test_expect_failure 'git detects conflict and handles merge of D & E1 correctly' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+	git checkout D^0 &&
+
+	# FIXME: If merge-base could keep both a and a/file in its tree, then
+	# we could this merge would actually be able to succeed.
+	test_must_fail git merge -s recursive E1^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 1 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test $(git rev-parse :2:a) = $(git rev-parse B:a)
+'
+
+test_expect_failure 'git detects conflict and handles merge of E1 & D correctly' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+	git checkout E1^0 &&
+
+	# FIXME: If merge-base could keep both a and a/file in its tree, then
+	# we could this merge would actually be able to succeed.
+	test_must_fail git merge -s recursive D^0 &&
+
+	test 2 -eq $(git ls-files -s | wc -l) &&
+	test 1 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test $(git rev-parse :3:a) = $(git rev-parse B:a)
+'
+
+test_expect_success 'git detects conflict and handles merge of D & E2 correctly' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+	git checkout D^0 &&
+
+	test_must_fail git merge -s recursive E2^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
+
+	test $(git rev-parse :2:a) = $(git rev-parse B:a) &&
+	test $(git rev-parse :3:a/file) = $(git rev-parse E1:a/file)
+	test $(git rev-parse :1:a/file) = $(git rev-parse C:a/file)
+'
+
+test_expect_failure 'git detects conflict and handles merge of E2 & D correctly' '
+	git reset --hard &&
+	git reset --hard &&
+	git clean -fdqx &&
+	git checkout E2^0 &&
+
+	test_must_fail git merge -s recursive D^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 2 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
+
+	test $(git rev-parse :3:a) = $(git rev-parse B:a) &&
+	test $(git rev-parse :2:a/file) = $(git rev-parse E1:a/file)
+	test $(git rev-parse :1:a/file) = $(git rev-parse C:a/file)
+'
+
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 11/48] t6036: criss-cross w/ rename/rename(1to2)/modify+rename/rename(2to1)/modify

[Elijah Newren]

This test is mostly just designed for testing optimality of the virtual
merge base in the event of a rename/rename(1to2) conflict.  The current
choice for resolving this in git seems somewhat confusing and suboptimal.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh |   76 +++++++++++++++++++++++++++++++++++++
 1 files changed, 76 insertions(+), 0 deletions(-)

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 4993f67..eee183e 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -545,5 +545,81 @@ test_expect_failure 'git detects conflict and handles merge of E2 & D correctly'
 	test $(git rev-parse :1:a/file) = $(git rev-parse C:a/file)
 '
 
+#
+# criss-cross with rename/rename(1to2)/modify followed by
+# rename/rename(2to1)/modify:
+#
+#      B   D
+#      o---o
+#     / \ / \
+#  A o   X   ? F
+#     \ / \ /
+#      o---o
+#      C   E
+#
+#   Commit A: new file: a
+#   Commit B: rename a->b, modifying by adding a line
+#   Commit C: rename a->c
+#   Commit D: merge B&C, resolving conflict by keeping contents in newname
+#   Commit E: merge B&C, resolving conflict similar to D but adding another line
+#
+# There is a conflict merging B & C, but one of filename not of file
+# content.  Whoever created D and E chose specific resolutions for that
+# conflict resolution.  Now, since: (1) there is no content conflict
+# merging B & C, (2) D does not modify that merged content further, and (3)
+# both D & E resolve the name conflict in the same way, the modification to
+# newname in E should not cause any conflicts when it is merged with D.
+# (Note that this can be accomplished by having the virtual merge base have
+# the merged contents of b and c stored in a file named a, which seems like
+# the most logical choice anyway.)
+
+test_expect_success 'setup rename/rename(1to2)/modify followed by what looks like rename/rename(2to1)/modify' '
+	git reset --hard &&
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "1\n2\n3\n4\n5\n6\n" >a &&
+	git add a &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	echo 7 >>b &&
+	git add -u &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a c &&
+	git commit -m C &&
+
+	git checkout -q B^0 &&
+	git merge --no-commit -s ours C^0 &&
+	git mv b newname &&
+	git commit -m "Merge commit C^0 into HEAD" &&
+	git tag D &&
+
+	git checkout -q C^0 &&
+	git merge --no-commit -s ours B^0 &&
+	git mv c newname &&
+	printf "7\n8\n" >>newname &&
+	git add -u &&
+	git commit -m "Merge commit B^0 into HEAD" &&
+	git tag E
+'
+
+test_expect_failure 'handle rename/rename(1to2)/modify followed by what looks like rename/rename(2to1)/modify' '
+	git checkout D^0 &&
+
+	git merge -s recursive E^0 &&
+
+	test 1 -eq $(git ls-files -s | wc -l) &&
+	test 0 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test 8 -eq $(wc -l < newname)
+'
 
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 12/48] t6036: criss-cross + rename/rename(1to2)/add-source + modify/modify

[Elijah Newren]

Now THAT's a corner case.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6036-recursive-corner-cases.sh |   72 +++++++++++++++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index eee183e..2fa7c3e 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -622,4 +622,76 @@ test_expect_failure 'handle rename/rename(1to2)/modify followed by what looks li
 	test 8 -eq $(wc -l < newname)
 '
 
+#
+# criss-cross with rename/rename(1to2)/add-source + resolvable modify/modify:
+#
+#      B   D
+#      o---o
+#     / \ / \
+#  A o   X   ? F
+#     \ / \ /
+#      o---o
+#      C   E
+#
+#   Commit A: new file: a
+#   Commit B: rename a->b
+#   Commit C: rename a->c, add different a
+#   Commit D: merge B&C, keeping b&c and (new) a modified at beginning
+#   Commit E: merge B&C, keeping b&c and (new) a modified at end
+#
+# Now, when we merge commits D & E, there should be no conflict...
+
+test_expect_success 'setup criss-cross + rename/rename/add + modify/modify' '
+	git rm -rf . &&
+	git clean -fdqx &&
+	rm -rf .git &&
+	git init &&
+
+	printf "lots\nof\nwords\nand\ncontent\n" >a &&
+	git add a &&
+	git commit -m A &&
+	git tag A &&
+
+	git checkout -b B A &&
+	git mv a b &&
+	git commit -m B &&
+
+	git checkout -b C A &&
+	git mv a c &&
+	printf "2\n3\n4\n5\n6\n7\n" >a &&
+	git add a &&
+	git commit -m C &&
+
+	git checkout B^0 &&
+	git merge --no-commit -s ours C^0 &&
+	git checkout C -- a c &&
+	mv a old_a &&
+	echo 1 >a &&
+	cat old_a >>a &&
+	rm old_a &&
+	git add -u &&
+	git commit -m "Merge commit C^0 into HEAD" &&
+	git tag D &&
+
+	git checkout C^0 &&
+	git merge --no-commit -s ours B^0 &&
+	git checkout B -- b &&
+	echo 8 >> a &&
+	git add -u &&
+	git commit -m "Merge commit B^0 into HEAD" &&
+	git tag E
+'
+
+test_expect_failure 'correctly resolves criss-cross with rename/rename/add and modify/modify conflict' '
+	git checkout D^0 &&
+
+	git merge -s recursive E^0 &&
+
+	test 3 -eq $(git ls-files -s | wc -l) &&
+	test 0 -eq $(git ls-files -u | wc -l) &&
+	test 0 -eq $(git ls-files -o | wc -l) &&
+
+	test 6 -eq $(wc -l < a)
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 13/48] t6022: Remove unnecessary untracked files to make test cleaner

[Elijah Newren]

Since this test later does a git add -A, we should clean out unnecessary
untracked files as part of our cleanup.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6022-merge-rename.sh |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index 1ed259d..1d1b32e 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -252,6 +252,7 @@ test_expect_success 'setup for rename + d/f conflicts' '
 	git reset --hard &&
 	git checkout --orphan dir-in-way &&
 	git rm -rf . &&
+	git clean -fdqx &&
 
 	mkdir sub &&
 	mkdir dir &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 14/48] t6022: New tests checking for unnecessary updates of files

[Elijah Newren]

This testcase was part of en/merge-recursive that was reverted in 6db4105
(Revert "Merge branch 'en/merge-recursive'" 2011-05-19).  While the other
changes in that series caused unfortunate breakage, this testcase is still
useful; reinstate it.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6022-merge-rename.sh |   63 +++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index 1d1b32e..11c5c60 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -610,4 +610,67 @@ test_expect_success 'check handling of differently renamed file with D/F conflic
 	! test -f original
 '
 
+test_expect_success 'setup avoid unnecessary update, normal rename' '
+	git reset --hard &&
+	git checkout --orphan avoid-unnecessary-update-1 &&
+	git rm -rf . &&
+	git clean -fdqx &&
+
+	printf "1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n" >original &&
+	git add -A &&
+	git commit -m "Common commmit" &&
+
+	git mv original rename &&
+	echo 11 >>rename &&
+	git add -u &&
+	git commit -m "Renamed and modified" &&
+
+	git checkout -b merge-branch-1 HEAD~1 &&
+	echo "random content" >random-file &&
+	git add -A &&
+	git commit -m "Random, unrelated changes"
+'
+
+test_expect_failure 'avoid unnecessary update, normal rename' '
+	git checkout -q avoid-unnecessary-update-1^0 &&
+	test-chmtime =1000000000 rename &&
+	test-chmtime -v +0 rename >expect &&
+	git merge merge-branch-1 &&
+	test-chmtime -v +0 rename >actual &&
+	test_cmp expect actual # "rename" should have stayed intact
+'
+
+test_expect_success 'setup to test avoiding unnecessary update, with D/F conflict' '
+	git reset --hard &&
+	git checkout --orphan avoid-unnecessary-update-2 &&
+	git rm -rf . &&
+	git clean -fdqx &&
+
+	mkdir df &&
+	printf "1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n" >df/file &&
+	git add -A &&
+	git commit -m "Common commmit" &&
+
+	git mv df/file temp &&
+	rm -rf df &&
+	git mv temp df &&
+	echo 11 >>df &&
+	git add -u &&
+	git commit -m "Renamed and modified" &&
+
+	git checkout -b merge-branch-2 HEAD~1 &&
+	>unrelated-change &&
+	git add unrelated-change &&
+	git commit -m "Only unrelated changes"
+'
+
+test_expect_failure 'avoid unnecessary update, with D/F conflict' '
+	git checkout -q avoid-unnecessary-update-2^0 &&
+	test-chmtime =1000000000 df &&
+	test-chmtime -v +0 df >expect &&
+	git merge merge-branch-2 &&
+	test-chmtime -v +0 df >actual &&
+	test_cmp expect actual # "df" should have stayed intact
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 15/48] t6022: Add testcase for merging a renamed file with a simple change

[Elijah Newren]

This is a testcase that was broken by b2c8c0a (merge-recursive: When we
detect we can skip an update, actually skip it 2011-02-28) and fixed by
6db4105 (Revert "Merge branch 'en/merge-recursive'" 2011-05-19).  Include
this testcase to ensure we don't regress it again.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/t6022-merge-rename.sh |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index 11c5c60..52a8f0b 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -673,4 +673,26 @@ test_expect_failure 'avoid unnecessary update, with D/F conflict' '
 	test_cmp expect actual # "df" should have stayed intact
 '
 
+test_expect_success 'setup merge of rename + small change' '
+	git reset --hard &&
+	git checkout --orphan rename-plus-small-change &&
+	git rm -rf . &&
+
+	echo ORIGINAL >file &&
+	git add file &&
+
+	test_tick &&
+	git commit -m Initial &&
+	git checkout -b rename_branch &&
+	git mv file renamed_file &&
+	git commit -m Rename &&
+	git checkout rename-plus-small-change &&
+	echo NEW-VERSION >file &&
+	git commit -a -m Reformat
+'
+
+test_expect_success 'merge rename + small change' '
+	git merge rename_branch
+'
+
 test_done
-- 
1.7.6.rc0.62.g2d69f

[PATCH 16/48] merge-recursive: Make BUG message more legible by adding a newline

[Elijah Newren]

Hopefully no one ever hits this error except when making large changes to
merge-recursive.c and debugging...

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index db9ba19..3fcd0a5 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -230,7 +230,7 @@ struct tree *write_tree_from_memory(struct merge_options *o)
 		for (i = 0; i < active_nr; i++) {
 			struct cache_entry *ce = active_cache[i];
 			if (ce_stage(ce))
-				fprintf(stderr, "BUG: %d %.*s", ce_stage(ce),
+				fprintf(stderr, "BUG: %d %.*s\n", ce_stage(ce),
 					(int)ce_namelen(ce), ce->name);
 		}
 		die("Bug in merge-recursive.c");
-- 
1.7.6.rc0.62.g2d69f

[PATCH 17/48] merge-recursive: Correct a comment

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 3fcd0a5..3d464d9 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1439,7 +1439,7 @@ static int process_df_entry(struct merge_options *o,
 		handle_delete_modify(o, path, new_path,
 				     a_sha, a_mode, b_sha, b_mode);
 	} else if (!o_sha && !!a_sha != !!b_sha) {
-		/* directory -> (directory, file) */
+		/* directory -> (directory, file) or <nothing> -> (directory, file) */
 		const char *add_branch;
 		const char *other_branch;
 		unsigned mode;
-- 
1.7.6.rc0.62.g2d69f

[PATCH 18/48] merge-recursive: Mark some diff_filespec struct arguments const

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   19 ++++++++++---------
 1 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 3d464d9..317bf23 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -461,9 +461,10 @@ static struct string_list *get_renames(struct merge_options *o,
 	return renames;
 }
 
-static int update_stages_options(const char *path, struct diff_filespec *o,
-			 struct diff_filespec *a, struct diff_filespec *b,
-			 int clear, int options)
+static int update_stages_options(const char *path, const struct diff_filespec *o,
+				 const struct diff_filespec *a,
+				 const struct diff_filespec *b,
+				 int clear, int options)
 {
 	if (clear)
 		if (remove_file_from_cache(path))
@@ -712,9 +713,9 @@ struct merge_file_info {
 
 static int merge_3way(struct merge_options *o,
 		      mmbuffer_t *result_buf,
-		      struct diff_filespec *one,
-		      struct diff_filespec *a,
-		      struct diff_filespec *b,
+		      const struct diff_filespec *one,
+		      const struct diff_filespec *a,
+		      const struct diff_filespec *b,
 		      const char *branch1,
 		      const char *branch2)
 {
@@ -772,9 +773,9 @@ static int merge_3way(struct merge_options *o,
 }
 
 static struct merge_file_info merge_file(struct merge_options *o,
-				         struct diff_filespec *one,
-					 struct diff_filespec *a,
-					 struct diff_filespec *b,
+					 const struct diff_filespec *one,
+					 const struct diff_filespec *a,
+					 const struct diff_filespec *b,
 					 const char *branch1,
 					 const char *branch2)
 {
-- 
1.7.6.rc0.62.g2d69f

[PATCH 19/48] merge-recursive: Remember to free generated unique path names

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   21 ++++++++++++++-------
 1 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 317bf23..45675b6 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1072,7 +1072,6 @@ static int process_renames(struct merge_options *o,
 				   renamed: clean merge */
 				update_file(o, 1, ren1->pair->two->sha1, ren1->pair->two->mode, ren1_dst);
 			} else if (!sha_eq(dst_other.sha1, null_sha1)) {
-				const char *new_path;
 				clean_merge = 0;
 				try_merge = 1;
 				output(o, 1, "CONFLICT (rename/add): Rename %s->%s in %s. "
@@ -1101,9 +1100,10 @@ static int process_renames(struct merge_options *o,
 						    ren1_dst);
 					try_merge = 0;
 				} else {
-					new_path = unique_path(o, ren1_dst, branch2);
+					char *new_path = unique_path(o, ren1_dst, branch2);
 					output(o, 1, "Adding as %s instead", new_path);
 					update_file(o, 0, dst_other.sha1, dst_other.mode, new_path);
+					free(new_path);
 				}
 			} else if ((item = string_list_lookup(renames2Dst, ren1_dst))) {
 				ren2 = item->util;
@@ -1271,13 +1271,14 @@ static int merge_content(struct merge_options *o,
 	}
 
 	if (df_conflict_remains) {
-		const char *new_path;
+		char *new_path;
 		update_file_flags(o, mfi.sha, mfi.mode, path,
 				  o->call_depth || mfi.clean, 0);
 		new_path = unique_path(o, path, df_rename_conflict_branch);
 		mfi.clean = 0;
 		output(o, 1, "Adding as %s instead", new_path);
 		update_file_flags(o, mfi.sha, mfi.mode, new_path, 0, 1);
+		free(new_path);
 	} else {
 		update_file(o, mfi.clean, mfi.sha, mfi.mode, path);
 	}
@@ -1433,12 +1434,17 @@ static int process_df_entry(struct merge_options *o,
 		}
 	} else if (o_sha && (!a_sha || !b_sha)) {
 		/* Modify/delete; deleted side may have put a directory in the way */
-		const char *new_path = path;
-		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode))
+		char *new_path;
+		int free_me = 0;
+		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
 			new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
+			free_me = 1;
+		}
 		clean_merge = 0;
-		handle_delete_modify(o, path, new_path,
+		handle_delete_modify(o, path, free_me ? new_path : path,
 				     a_sha, a_mode, b_sha, b_mode);
+		if (free_me)
+			free(new_path);
 	} else if (!o_sha && !!a_sha != !!b_sha) {
 		/* directory -> (directory, file) or <nothing> -> (directory, file) */
 		const char *add_branch;
@@ -1461,12 +1467,13 @@ static int process_df_entry(struct merge_options *o,
 			conf = "directory/file";
 		}
 		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
-			const char *new_path = unique_path(o, path, add_branch);
+			char *new_path = unique_path(o, path, add_branch);
 			clean_merge = 0;
 			output(o, 1, "CONFLICT (%s): There is a directory with name %s in %s. "
 			       "Adding %s as %s",
 			       conf, path, other_branch, path, new_path);
 			update_file(o, 0, sha, mode, new_path);
+			free(new_path);
 		} else {
 			output(o, 2, "Adding %s", path);
 			update_file(o, 1, sha, mode, path);
-- 
1.7.6.rc0.62.g2d69f

[PATCH 20/48] merge-recursive: Avoid working directory changes during recursive case

[Elijah Newren]

make_room_for_directories_of_df_conflicts() is about making sure necessary
working directory changes can succeed.  When o->call_depth > 0 (i.e. the
recursive case), we do not want to make any working directory changes so
this function should be skipped.

Note that make_room_for_directories_of_df_conflicts() is broken as has
been pointed out by Junio; it should NOT be unlinking files.  What it
should do is keep track of files that could be unlinked if a directory
later needs to be written in their place.  However, that work also is only
relevant in the non-recursive case, so this change is helpful either way.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 45675b6..7da6aa0 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -355,6 +355,13 @@ static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 	int last_len = 0;
 	int i;
 
+	/*
+	 * If we're merging merge-bases, we don't want to bother with
+	 * any working directory changes.
+	 */
+	if (o->call_depth)
+		return;
+
 	for (i = 0; i < entries->nr; i++) {
 		const char *path = entries->items[i].string;
 		int len = strlen(path);
-- 
1.7.6.rc0.62.g2d69f

[PATCH 21/48] merge-recursive: Fix recursive case with D/F conflict via add/add conflict

[Elijah Newren]

When a D/F conflict is introduced via an add/add conflict, when
o->call_depth > 0 we need to ensure that the higher stage entry from the
base stage is removed.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                 |    2 ++
 t/t6036-recursive-corner-cases.sh |    4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 7da6aa0..4771fb4 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1480,6 +1480,8 @@ static int process_df_entry(struct merge_options *o,
 			       "Adding %s as %s",
 			       conf, path, other_branch, path, new_path);
 			update_file(o, 0, sha, mode, new_path);
+			if (o->call_depth)
+				remove_file_from_cache(path);
 			free(new_path);
 		} else {
 			output(o, 2, "Adding %s", path);
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 2fa7c3e..423fb62 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -494,7 +494,7 @@ test_expect_failure 'git detects conflict and handles merge of D & E1 correctly'
 	test $(git rev-parse :2:a) = $(git rev-parse B:a)
 '
 
-test_expect_failure 'git detects conflict and handles merge of E1 & D correctly' '
+test_expect_success 'git detects conflict and handles merge of E1 & D correctly' '
 	git reset --hard &&
 	git reset --hard &&
 	git clean -fdqx &&
@@ -528,7 +528,7 @@ test_expect_success 'git detects conflict and handles merge of D & E2 correctly'
 	test $(git rev-parse :1:a/file) = $(git rev-parse C:a/file)
 '
 
-test_expect_failure 'git detects conflict and handles merge of E2 & D correctly' '
+test_expect_success 'git detects conflict and handles merge of E2 & D correctly' '
 	git reset --hard &&
 	git reset --hard &&
 	git clean -fdqx &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Elijah Newren]

We cannot assume that directory/file conflicts will appear in sorted
order; for example, 'letters.txt' comes between 'letters' and
'letters/file'.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c   |   31 ++++++++++++++++++++++++++-----
 t/t6020-merge-df.sh |   26 ++++++++++++++++++--------
 2 files changed, 44 insertions(+), 13 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 4771fb4..ed1fdb2 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -333,6 +333,28 @@ static struct string_list *get_unmerged(void)
 	return unmerged;
 }
 
+static int string_list_df_name_compare(const void *a, const void *b)
+{
+	const struct string_list_item *one = a;
+	const struct string_list_item *two = b;
+	/*
+	 * Here we only care that entries for D/F conflicts are
+	 * adjacent, in particular with the file of the D/F conflict
+	 * appearing before files below the corresponding directory.
+	 * The order of the rest of the list is irrelevant for us.
+	 *
+	 * To achieve this, we sort with df_name_compare and provide
+	 * the mode S_IFDIR so that D/F conflicts will sort correctly.
+	 * We use the mode S_IFDIR for everything else for simplicity,
+	 * since in other cases any changes in their order due to
+	 * sorting cause no problems for us.
+	 */
+	return df_name_compare(one->string, strlen(one->string), S_IFDIR,
+			       two->string, strlen(two->string), S_IFDIR);
+}
+
+
+
 static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 						      struct string_list *entries)
 {
@@ -345,11 +367,6 @@ static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 	 * otherwise, if the file is not supposed to be removed by the
 	 * merge, the contents of the file will be placed in another
 	 * unique filename.
-	 *
-	 * NOTE: This function relies on the fact that entries for a
-	 * D/F conflict will appear adjacent in the index, with the
-	 * entries for the file appearing before entries for paths
-	 * below the corresponding directory.
 	 */
 	const char *last_file = NULL;
 	int last_len = 0;
@@ -362,6 +379,10 @@ static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 	if (o->call_depth)
 		return;
 
+	/* Ensure D/F conflicts are adjacent in the entries list. */
+	qsort(entries->items, entries->nr, sizeof(*entries->items),
+	      string_list_df_name_compare);
+
 	for (i = 0; i < entries->nr; i++) {
 		const char *path = entries->items[i].string;
 		int len = strlen(path);
diff --git a/t/t6020-merge-df.sh b/t/t6020-merge-df.sh
index eec8f4e..27c3d73 100755
--- a/t/t6020-merge-df.sh
+++ b/t/t6020-merge-df.sh
@@ -59,15 +59,19 @@ test_expect_success 'setup modify/delete + directory/file conflict' '
 	git add letters &&
 	git commit -m initial &&
 
+	# Throw in letters.txt for sorting order fun
+	# ("letters.txt" sorts between "letters" and "letters/file")
 	echo i >>letters &&
-	git add letters &&
+	echo "version 2" >letters.txt &&
+	git add letters letters.txt &&
 	git commit -m modified &&
 
 	git checkout -b delete HEAD^ &&
 	git rm letters &&
 	mkdir letters &&
 	>letters/file &&
-	git add letters &&
+	echo "version 1" >letters.txt &&
+	git add letters letters.txt &&
 	git commit -m deleted
 '
 
@@ -75,25 +79,31 @@ test_expect_success 'modify/delete + directory/file conflict' '
 	git checkout delete^0 &&
 	test_must_fail git merge modify &&
 
-	test 3 = $(git ls-files -s | wc -l) &&
-	test 2 = $(git ls-files -u | wc -l) &&
-	test 1 = $(git ls-files -o | wc -l) &&
+	test 5 -eq $(git ls-files -s | wc -l) &&
+	test 4 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
 
 	test -f letters/file &&
+	test -f letters.txt &&
 	test -f letters~modify
 '
 
 test_expect_success 'modify/delete + directory/file conflict; other way' '
+	# Yes, we really need the double reset since "letters" appears as
+	# both a file and a directory.
+	git reset --hard &&
 	git reset --hard &&
 	git clean -f &&
 	git checkout modify^0 &&
+
 	test_must_fail git merge delete &&
 
-	test 3 = $(git ls-files -s | wc -l) &&
-	test 2 = $(git ls-files -u | wc -l) &&
-	test 1 = $(git ls-files -o | wc -l) &&
+	test 5 -eq $(git ls-files -s | wc -l) &&
+	test 4 -eq $(git ls-files -u | wc -l) &&
+	test 1 -eq $(git ls-files -o | wc -l) &&
 
 	test -f letters/file &&
+	test -f letters.txt &&
 	test -f letters~HEAD
 '
 
-- 
1.7.6.rc0.62.g2d69f

[PATCH 23/48] merge-recursive: Fix code checking for D/F conflicts still being present

[Elijah Newren]

Previously, we were using lstat() to determine if a directory was still
present after a merge (and thus in the way of adding a file).  We should
have been using lstat() only to determine if untracked directories were in
the way (and then only when necessary to check for untracked directories);
we should instead using the index to determine if there is a tracked
directory in the way.  Create a new function to do this and use it to
replace the existing checks for directories being in the way.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   49 ++++++++++++++++++++++++++++++++++---------------
 1 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index ed1fdb2..3a8e64e 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -592,6 +592,30 @@ static void flush_buffer(int fd, const char *buf, unsigned long size)
 	}
 }
 
+static int dir_in_way(const char *path, int check_working_copy)
+{
+	int pathlen = strlen(path);
+	char *dirpath = xmalloc(pathlen + 2);
+	struct stat st;
+
+	strcpy(dirpath, path);
+	dirpath[pathlen] = '/';
+	dirpath[pathlen+1] = '\0';
+
+	int pos = cache_name_pos(dirpath, pathlen+1);
+
+	if (pos < 0)
+		pos = -1 - pos;
+	if (pos < active_nr &&
+	    !strncmp(dirpath, active_cache[pos]->name, pathlen+1)) {
+		free(dirpath);
+		return 1;
+	}
+
+	free(dirpath);
+	return check_working_copy && !lstat(path, &st) && S_ISDIR(st.st_mode);
+}
+
 static int would_lose_untracked(const char *path)
 {
 	int pos = cache_name_pos(path, strlen(path));
@@ -881,7 +905,6 @@ static void conflict_rename_delete(struct merge_options *o,
 {
 	char *dest_name = pair->two->path;
 	int df_conflict = 0;
-	struct stat st;
 
 	output(o, 1, "CONFLICT (rename/delete): Rename %s->%s in %s "
 	       "and deleted in %s",
@@ -892,7 +915,7 @@ static void conflict_rename_delete(struct merge_options *o,
 			      rename_branch == o->branch1 ? pair->two : NULL,
 			      rename_branch == o->branch1 ? NULL : pair->two,
 			      1);
-	if (lstat(dest_name, &st) == 0 && S_ISDIR(st.st_mode)) {
+	if (dir_in_way(dest_name, !o->call_depth)) {
 		dest_name = unique_path(o, dest_name, rename_branch);
 		df_conflict = 1;
 	}
@@ -914,13 +937,12 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 	const char *ren2_dst = pair2->two->path;
 	const char *dst_name1 = ren1_dst;
 	const char *dst_name2 = ren2_dst;
-	struct stat st;
-	if (lstat(ren1_dst, &st) == 0 && S_ISDIR(st.st_mode)) {
+	if (dir_in_way(ren1_dst, !o->call_depth)) {
 		dst_name1 = del[delp++] = unique_path(o, ren1_dst, branch1);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
 		       ren1_dst, branch2, dst_name1);
 	}
-	if (lstat(ren2_dst, &st) == 0 && S_ISDIR(st.st_mode)) {
+	if (dir_in_way(ren2_dst, !o->call_depth)) {
 		dst_name2 = del[delp++] = unique_path(o, ren2_dst, branch2);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
 		       ren2_dst, branch1, dst_name2);
@@ -1080,7 +1102,7 @@ static int process_renames(struct merge_options *o,
 			try_merge = 0;
 
 			if (sha_eq(src_other.sha1, null_sha1)) {
-				if (string_list_has_string(&o->current_directory_set, ren1_dst)) {
+				if (dir_in_way(ren1_dst, 0 /*check_wc*/)) {
 					ren1->dst_entry->processed = 0;
 					setup_rename_df_conflict_info(RENAME_DELETE,
 								      ren1->pair,
@@ -1159,7 +1181,7 @@ static int process_renames(struct merge_options *o,
 					a = &src_other;
 				}
 				update_stages_and_entry(ren1_dst, ren1->dst_entry, one, a, b, 1);
-				if (string_list_has_string(&o->current_directory_set, ren1_dst)) {
+				if (dir_in_way(ren1_dst, 0 /*check_wc*/)) {
 					setup_rename_df_conflict_info(RENAME_NORMAL,
 								      ren1->pair,
 								      NULL,
@@ -1264,7 +1286,6 @@ static int merge_content(struct merge_options *o,
 	const char *reason = "content";
 	struct merge_file_info mfi;
 	struct diff_filespec one, a, b;
-	struct stat st;
 	unsigned df_conflict_remains = 0;
 
 	if (!o_sha) {
@@ -1281,7 +1302,7 @@ static int merge_content(struct merge_options *o,
 
 	mfi = merge_file(o, &one, &a, &b, o->branch1, o->branch2);
 	if (df_rename_conflict_branch &&
-	    lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
+	    dir_in_way(path, !o->call_depth)) {
 		df_conflict_remains = 1;
 	}
 
@@ -1346,8 +1367,7 @@ static int process_entry(struct merge_options *o,
 				output(o, 2, "Removing %s", path);
 			/* do not touch working file if it did not exist */
 			remove_file(o, 1, path, !a_sha);
-		} else if (string_list_has_string(&o->current_directory_set,
-						  path)) {
+		} else if (dir_in_way(path, 0 /*check_wc*/)) {
 			entry->processed = 0;
 			return 1; /* Assume clean until processed */
 		} else {
@@ -1370,7 +1390,7 @@ static int process_entry(struct merge_options *o,
 			mode = b_mode;
 			sha = b_sha;
 		}
-		if (string_list_has_string(&o->current_directory_set, path)) {
+		if (dir_in_way(path, 0 /*check_wc*/)) {
 			/* Handle D->F conflicts after all subfiles */
 			entry->processed = 0;
 			return 1; /* Assume clean until processed */
@@ -1418,7 +1438,6 @@ static int process_df_entry(struct merge_options *o,
 	unsigned char *o_sha = stage_sha(entry->stages[1].sha, o_mode);
 	unsigned char *a_sha = stage_sha(entry->stages[2].sha, a_mode);
 	unsigned char *b_sha = stage_sha(entry->stages[3].sha, b_mode);
-	struct stat st;
 
 	entry->processed = 1;
 	if (entry->rename_df_conflict_info) {
@@ -1464,7 +1483,7 @@ static int process_df_entry(struct merge_options *o,
 		/* Modify/delete; deleted side may have put a directory in the way */
 		char *new_path;
 		int free_me = 0;
-		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
+		if (dir_in_way(path, !o->call_depth)) {
 			new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
 			free_me = 1;
 		}
@@ -1494,7 +1513,7 @@ static int process_df_entry(struct merge_options *o,
 			sha = b_sha;
 			conf = "directory/file";
 		}
-		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
+		if (dir_in_way(path, !o->call_depth)) {
 			char *new_path = unique_path(o, path, add_branch);
 			clean_merge = 0;
 			output(o, 1, "CONFLICT (%s): There is a directory with name %s in %s. "
-- 
1.7.6.rc0.62.g2d69f

[PATCH 24/48] merge-recursive: Save D/F conflict filenames instead of unlinking them

[Elijah Newren]

Rename make_room_for_directories_of_df_conflicts() to
record_df_conflict_files() to reflect the change in functionality.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   40 ++++++++++++++++++++++++----------------
 merge-recursive.h |    1 +
 2 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 3a8e64e..e59eec0 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -353,20 +353,24 @@ static int string_list_df_name_compare(const void *a, const void *b)
 			       two->string, strlen(two->string), S_IFDIR);
 }
 
-
-
-static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
-						      struct string_list *entries)
+static void record_df_conflict_files(struct merge_options *o,
+				     struct string_list *entries)
 {
-	/* If there are D/F conflicts, and the paths currently exist
-	 * in the working copy as a file, we want to remove them to
-	 * make room for the corresponding directory.  Such paths will
-	 * later be processed in process_df_entry() at the end.  If
-	 * the corresponding directory ends up being removed by the
-	 * merge, then the file will be reinstated at that time;
-	 * otherwise, if the file is not supposed to be removed by the
-	 * merge, the contents of the file will be placed in another
-	 * unique filename.
+	/* If there is a D/F conflict and the file for such a conflict
+	 * currently exist in the working copy, we want to allow it to
+	 * be removed to make room for the corresponding directory if
+	 * needed.  The files underneath the directories of such D/F
+	 * conflicts will be handled in process_entry(), while the
+	 * files of such D/F conflicts will be processed later in
+	 * process_df_entry().  If the corresponding directory ends up
+	 * being removed by the merge, then no additional work needs
+	 * to be done by process_df_entry() for the conflicting file.
+	 * If the directory needs to be written to the working copy,
+	 * then the conflicting file will simply be removed (e.g. in
+	 * make_room_for_path).  If the directory is written to the
+	 * working copy but the file also has a conflict that needs to
+	 * be resolved, then process_df_entry() will reinstate the
+	 * file with a new unique name.
 	 */
 	const char *last_file = NULL;
 	int last_len = 0;
@@ -383,6 +387,7 @@ static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 	qsort(entries->items, entries->nr, sizeof(*entries->items),
 	      string_list_df_name_compare);
 
+	string_list_clear(&o->df_conflict_file_set, 1);
 	for (i = 0; i < entries->nr; i++) {
 		const char *path = entries->items[i].string;
 		int len = strlen(path);
@@ -391,14 +396,15 @@ static void make_room_for_directories_of_df_conflicts(struct merge_options *o,
 		/*
 		 * Check if last_file & path correspond to a D/F conflict;
 		 * i.e. whether path is last_file+'/'+<something>.
-		 * If so, remove last_file to make room for path and friends.
+		 * If so, record that it's okay to remove last_file to make
+		 * room for path and friends if needed.
 		 */
 		if (last_file &&
 		    len > last_len &&
 		    memcmp(path, last_file, last_len) == 0 &&
 		    path[last_len] == '/') {
 			output(o, 3, "Removing %s to make room for subdirectory; may re-add later.", last_file);
-			unlink(last_file);
+			string_list_insert(&o->df_conflict_file_set, last_file);
 		}
 
 		/*
@@ -1574,7 +1580,7 @@ int merge_trees(struct merge_options *o,
 		get_files_dirs(o, merge);
 
 		entries = get_unmerged();
-		make_room_for_directories_of_df_conflicts(o, entries);
+		record_df_conflict_files(o, entries);
 		re_head  = get_renames(o, head, common, head, merge, entries);
 		re_merge = get_renames(o, merge, common, head, merge, entries);
 		clean = process_renames(o, re_head, re_merge);
@@ -1800,6 +1806,8 @@ void init_merge_options(struct merge_options *o)
 	o->current_file_set.strdup_strings = 1;
 	memset(&o->current_directory_set, 0, sizeof(struct string_list));
 	o->current_directory_set.strdup_strings = 1;
+	memset(&o->df_conflict_file_set, 0, sizeof(struct string_list));
+	o->df_conflict_file_set.strdup_strings = 1;
 }
 
 int parse_merge_opt(struct merge_options *o, const char *s)
diff --git a/merge-recursive.h b/merge-recursive.h
index 7e1e972..58f3435 100644
--- a/merge-recursive.h
+++ b/merge-recursive.h
@@ -26,6 +26,7 @@ struct merge_options {
 	struct strbuf obuf;
 	struct string_list current_file_set;
 	struct string_list current_directory_set;
+	struct string_list df_conflict_file_set;
 };
 
 /* merge_trees() but with recursive ancestor consolidation */
-- 
1.7.6.rc0.62.g2d69f

[PATCH 25/48] merge-recursive: Split was_tracked() out of would_lose_untracked()

[Elijah Newren]

Checking whether a filename was part of stage 0 or stage 2 is code that we
would like to be able to call from a few other places without also
lstat()-ing the file to see if it exists in the working copy.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index e59eec0..d19c519 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -622,7 +622,7 @@ static int dir_in_way(const char *path, int check_working_copy)
 	return check_working_copy && !lstat(path, &st) && S_ISDIR(st.st_mode);
 }
 
-static int would_lose_untracked(const char *path)
+static int was_tracked(const char *path)
 {
 	int pos = cache_name_pos(path, strlen(path));
 
@@ -639,11 +639,16 @@ static int would_lose_untracked(const char *path)
 		switch (ce_stage(active_cache[pos])) {
 		case 0:
 		case 2:
-			return 0;
+			return 1;
 		}
 		pos++;
 	}
-	return file_exists(path);
+	return 0;
+}
+
+static int would_lose_untracked(const char *path)
+{
+	return !was_tracked(path) && file_exists(path);
 }
 
 static int make_room_for_path(const char *path)
-- 
1.7.6.rc0.62.g2d69f

[PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Elijah Newren]

If there were several files conflicting below a directory corresponding
to a D/F conflict, and the file of that D/F conflict is in the way, we
want it to be removed.  Since files of D/F conflicts are handled last,
they can be reinstated later and possibly with a new unique name.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                 |   18 +++++++++++++++---
 t/t6036-recursive-corner-cases.sh |    2 +-
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index d19c519..e3033f2 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -651,11 +651,23 @@ static int would_lose_untracked(const char *path)
 	return !was_tracked(path) && file_exists(path);
 }
 
-static int make_room_for_path(const char *path)
+static int make_room_for_path(const struct merge_options *o, const char *path)
 {
-	int status;
+	int status, i;
 	const char *msg = "failed to create path '%s'%s";
 
+	/* Unlink any D/F conflict files that are in the way */
+	for (i = 0; i < o->df_conflict_file_set.nr; i++) {
+		const char *df_path = o->df_conflict_file_set.items[i].string;
+		size_t pathlen = strlen(path);
+		size_t df_pathlen = strlen(df_path);
+		if (df_pathlen < pathlen && strncmp(path, df_path, df_pathlen) == 0) {
+			unlink(df_path);
+			break;
+		}
+	}
+
+	/* Make sure leading directories are created */
 	status = safe_create_leading_directories_const(path);
 	if (status) {
 		if (status == -3) {
@@ -723,7 +735,7 @@ static void update_file_flags(struct merge_options *o,
 			}
 		}
 
-		if (make_room_for_path(path) < 0) {
+		if (make_room_for_path(o, path) < 0) {
 			update_wd = 0;
 			free(buf);
 			goto update_index;
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 423fb62..dea2a65 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -477,7 +477,7 @@ test_expect_success 'setup differently handled merges of directory/file conflict
 	git tag E2
 '
 
-test_expect_failure 'git detects conflict and handles merge of D & E1 correctly' '
+test_expect_success 'git detects conflict and handles merge of D & E1 correctly' '
 	git reset --hard &&
 	git reset --hard &&
 	git clean -fdqx &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 27/48] merge-recursive: Consolidate different update_stages functions

[Elijah Newren]

We are only calling update_stages_options() one way really, so we can
consolidate the slightly different variants into one and remove some
parameters whose values are always the same.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   27 +++++++++------------------
 1 files changed, 9 insertions(+), 18 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index e3033f2..b4baa35 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -495,11 +495,12 @@ static struct string_list *get_renames(struct merge_options *o,
 	return renames;
 }
 
-static int update_stages_options(const char *path, const struct diff_filespec *o,
-				 const struct diff_filespec *a,
-				 const struct diff_filespec *b,
-				 int clear, int options)
+static int update_stages(const char *path, const struct diff_filespec *o,
+			 const struct diff_filespec *a,
+			 const struct diff_filespec *b)
 {
+	int clear = 1;
+	int options = ADD_CACHE_OK_TO_ADD | ADD_CACHE_SKIP_DFCHECK;
 	if (clear)
 		if (remove_file_from_cache(path))
 			return -1;
@@ -515,14 +516,6 @@ static int update_stages_options(const char *path, const struct diff_filespec *o
 	return 0;
 }
 
-static int update_stages(const char *path, struct diff_filespec *o,
-			 struct diff_filespec *a, struct diff_filespec *b,
-			 int clear)
-{
-	int options = ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE;
-	return update_stages_options(path, o, a, b, clear, options);
-}
-
 static int update_stages_and_entry(const char *path,
 				   struct stage_data *entry,
 				   struct diff_filespec *o,
@@ -539,8 +532,7 @@ static int update_stages_and_entry(const char *path,
 	hashcpy(entry->stages[1].sha, o->sha1);
 	hashcpy(entry->stages[2].sha, a->sha1);
 	hashcpy(entry->stages[3].sha, b->sha1);
-	options = ADD_CACHE_OK_TO_ADD | ADD_CACHE_SKIP_DFCHECK;
-	return update_stages_options(path, o, a, b, clear, options);
+	return update_stages(path, o, a, b);
 }
 
 static int remove_file(struct merge_options *o, int clean,
@@ -936,8 +928,7 @@ static void conflict_rename_delete(struct merge_options *o,
 	if (!o->call_depth)
 		update_stages(dest_name, NULL,
 			      rename_branch == o->branch1 ? pair->two : NULL,
-			      rename_branch == o->branch1 ? NULL : pair->two,
-			      1);
+			      rename_branch == o->branch1 ? NULL : pair->two);
 	if (dir_in_way(dest_name, !o->call_depth)) {
 		dest_name = unique_path(o, dest_name, rename_branch);
 		df_conflict = 1;
@@ -980,8 +971,8 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 		 * update_file(o, 0, pair2->two->sha1, pair2->two->mode, dst_name2);
 		 */
 	} else {
-		update_stages(ren1_dst, NULL, pair1->two, NULL, 1);
-		update_stages(ren2_dst, NULL, NULL, pair2->two, 1);
+		update_stages(ren1_dst, NULL, pair1->two, NULL);
+		update_stages(ren2_dst, NULL, NULL, pair2->two);
 
 		update_file(o, 0, pair1->two->sha1, pair1->two->mode, dst_name1);
 		update_file(o, 0, pair2->two->sha1, pair2->two->mode, dst_name2);
-- 
1.7.6.rc0.62.g2d69f

[PATCH 28/48] merge-recursive: Split update_stages_and_entry; only update stages at end

[Elijah Newren]

Instead of having the process_renames logic update the stages in the index
for the rename destination, have the index updated after process_entry or
process_df_entry.  This will also allow us to have process_entry determine
whether a file was tracked and existed in the working copy before the
merge started.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   35 +++++++++++++++++------------------
 1 files changed, 17 insertions(+), 18 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index b4baa35..7878b30 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -90,6 +90,7 @@ struct stage_data {
 	} stages[4];
 	struct rename_df_conflict_info *rename_df_conflict_info;
 	unsigned processed:1;
+	unsigned involved_in_rename:1;
 };
 
 static inline void setup_rename_df_conflict_info(enum rename_type rename_type,
@@ -516,15 +517,11 @@ static int update_stages(const char *path, const struct diff_filespec *o,
 	return 0;
 }
 
-static int update_stages_and_entry(const char *path,
-				   struct stage_data *entry,
-				   struct diff_filespec *o,
-				   struct diff_filespec *a,
-				   struct diff_filespec *b,
-				   int clear)
+static void update_entry(struct stage_data *entry,
+			 struct diff_filespec *o,
+			 struct diff_filespec *a,
+			 struct diff_filespec *b)
 {
-	int options;
-
 	entry->processed = 0;
 	entry->stages[1].mode = o->mode;
 	entry->stages[2].mode = a->mode;
@@ -532,7 +529,6 @@ static int update_stages_and_entry(const char *path,
 	hashcpy(entry->stages[1].sha, o->sha1);
 	hashcpy(entry->stages[2].sha, a->sha1);
 	hashcpy(entry->stages[3].sha, b->sha1);
-	return update_stages(path, o, a, b);
 }
 
 static int remove_file(struct merge_options *o, int clean,
@@ -1084,12 +1080,11 @@ static int process_renames(struct merge_options *o,
 							      ren2->dst_entry);
 			} else {
 				remove_file(o, 1, ren1_src, 1);
-				update_stages_and_entry(ren1_dst,
-							ren1->dst_entry,
-							ren1->pair->one,
-							ren1->pair->two,
-							ren2->pair->two,
-							1 /* clear */);
+				update_entry(ren1->dst_entry,
+					     ren1->pair->one,
+					     ren1->pair->two,
+					     ren2->pair->two);
+				ren1->dst_entry->involved_in_rename = 1;
 			}
 		} else {
 			/* Renamed in 1, maybe changed in 2 */
@@ -1194,7 +1189,8 @@ static int process_renames(struct merge_options *o,
 					b = ren1->pair->two;
 					a = &src_other;
 				}
-				update_stages_and_entry(ren1_dst, ren1->dst_entry, one, a, b, 1);
+				update_entry(ren1->dst_entry, one, a, b);
+				ren1->dst_entry->involved_in_rename = 1;
 				if (dir_in_way(ren1_dst, 0 /*check_wc*/)) {
 					setup_rename_df_conflict_info(RENAME_NORMAL,
 								      ren1->pair,
@@ -1291,6 +1287,7 @@ static void handle_delete_modify(struct merge_options *o,
 }
 
 static int merge_content(struct merge_options *o,
+			 unsigned involved_in_rename,
 			 const char *path,
 			 unsigned char *o_sha, int o_mode,
 			 unsigned char *a_sha, int a_mode,
@@ -1331,6 +1328,8 @@ static int merge_content(struct merge_options *o,
 			reason = "submodule";
 		output(o, 1, "CONFLICT (%s): Merge conflict in %s",
 				reason, path);
+		if (involved_in_rename)
+			update_stages(path, &one, &a, &b);
 	}
 
 	if (df_conflict_remains) {
@@ -1415,7 +1414,7 @@ static int process_entry(struct merge_options *o,
 	} else if (a_sha && b_sha) {
 		/* Case C: Added in both (check for same permissions) and */
 		/* case D: Modified in both, but differently. */
-		clean_merge = merge_content(o, path,
+		clean_merge = merge_content(o, entry->involved_in_rename, path,
 					    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
 					    NULL);
 	} else if (!o_sha && !a_sha && !b_sha) {
@@ -1459,7 +1458,7 @@ static int process_df_entry(struct merge_options *o,
 		char *src;
 		switch (conflict_info->rename_type) {
 		case RENAME_NORMAL:
-			clean_merge = merge_content(o, path,
+			clean_merge = merge_content(o, entry->involved_in_rename, path,
 						    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
 						    conflict_info->branch1);
 			break;
-- 
1.7.6.rc0.62.g2d69f

[PATCH 29/48] merge-recursive: When we detect we can skip an update, actually skip it

[Elijah Newren]

In 882fd11 (merge-recursive: Delay content merging for renames 2010-09-20),
there was code that checked for whether we could skip updating a file in
the working directory, based on whether the merged version matched the
current working copy.  Due to the desire to handle directory/file conflicts
that were resolvable, that commit deferred content merging by first
updating the index with the unmerged entries and then moving the actual
merging (along with the skip-the-content-update check) to another function
that ran later in the merge process.  As part moving the content merging
code, a bug was introduced such that although the message about skipping
the update would be printed (whenever GIT_MERGE_VERBOSITY was sufficiently
high), the file would be unconditionally updated in the working copy
anyway.

When we detect that the file does not need to be updated in the working
copy, update the index appropriately and then return early before updating
the working copy.

Note that there was a similar change in b2c8c0a (merge-recursive: When we
detect we can skip an update, actually skip it 2011-02-28), but it was
reverted by 6db4105 (Revert "Merge branch 'en/merge-recursive'"
2011-05-19) since it did not fix both of the relevant types of unnecessary
update breakages and, worse, it made use of some band-aids that caused
other problems.  The reason this change works is due to the changes of the
last few patches to (a) record_df_conflict_files instead of just unlinking
them early, (b) allowing make_room_for_path() to remove D/F entries, and
(c) the splitting of update_stages_and_entry() to have its functionality
called at different points.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c       |    8 ++++++--
 t/t6022-merge-rename.sh |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 7878b30..987a985 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1318,9 +1318,13 @@ static int merge_content(struct merge_options *o,
 	}
 
 	if (mfi.clean && !df_conflict_remains &&
-	    sha_eq(mfi.sha, a_sha) && mfi.mode == a.mode)
+	    sha_eq(mfi.sha, a_sha) && mfi.mode == a.mode &&
+	    was_tracked(path)) {
 		output(o, 3, "Skipped %s (merged same as existing)", path);
-	else
+		add_cacheinfo(mfi.mode, mfi.sha, path,
+			      0 /*stage*/, 1 /*refresh*/, 0 /*options*/);
+		return mfi.clean;
+	} else
 		output(o, 2, "Auto-merging %s", path);
 
 	if (!mfi.clean) {
diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index 52a8f0b..5e2a686 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -631,7 +631,7 @@ test_expect_success 'setup avoid unnecessary update, normal rename' '
 	git commit -m "Random, unrelated changes"
 '
 
-test_expect_failure 'avoid unnecessary update, normal rename' '
+test_expect_success 'avoid unnecessary update, normal rename' '
 	git checkout -q avoid-unnecessary-update-1^0 &&
 	test-chmtime =1000000000 rename &&
 	test-chmtime -v +0 rename >expect &&
@@ -664,7 +664,7 @@ test_expect_success 'setup to test avoiding unnecessary update, with D/F conflic
 	git commit -m "Only unrelated changes"
 '
 
-test_expect_failure 'avoid unnecessary update, with D/F conflict' '
+test_expect_success 'avoid unnecessary update, with D/F conflict' '
 	git checkout -q avoid-unnecessary-update-2^0 &&
 	test-chmtime =1000000000 df &&
 	test-chmtime -v +0 df >expect &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 30/48] merge-recursive: Fix deletion of untracked file in rename/delete conflicts

[Elijah Newren]

In the recursive case (o->call_depth > 0), we do not modify the working
directory.  However, when o->call_depth==0, file renames can mean we need
to delete the old filename from the working copy.  Since there have been
lots of changes and mistakes here, let's go through the details.  Let's
start with a simple explanation of what we are trying to achieve:

  Original goal: If a file is renamed on the side of history being merged
  into head, the filename serving as the source of that rename needs to be
  removed from the working directory.

The path to getting the above statement implemented in merge-recursive took
several steps.  The relevant bits of code may be instructive to keep in
mind for the explanation, especially since an English-only description
involves double negatives that are hard to follow.  These bits of code are:
  int remove_file(..., const char *path, int no_wd)
  {
    ...
    int update_working_directory = !o->call_depth && !no_wd;
and
  remove_file(o, 1, ren1_src, <expression>);
Where the choice for <expression> has morphed over time:

65ac6e9 (merge-recursive: adjust to loosened "working file clobbered"
check 2006-10-27), introduced the "no_wd" parameter to remove_file() and
used "1" for <expression>.  This meant ren1_src was never deleted, leaving
it around in the working copy.

In 8371234 (Remove uncontested renamed files during merge. 2006-12-13),
<expression> was changed to "index_only" (where index_only ==
!!o->call_depth; see b7fa51da).   This was equivalent to using "0" for
<expression> (due to the early logic in remove_file), and is orthogonal to
the condition we actually want to check at this point; it resulted in the
source file being removed except when index_only was false.  This was
problematic because the file could have been renamed on the side of history
including head, in which case ren1_src could correspond to an untracked
file that should not be deleted.

In 183d797 (Keep untracked files not involved in a merge. 2007-02-04),
<expression> was changed to "index_only || stage == 3".  While this gives
correct behavior, the "index_only ||" portion of <expression> is
unnecessary and makes the code slightly harder to follow.

There were also two further changes to this expression, though without
any change in behavior.  First in b7fa51d (merge-recursive: get rid of the
index_only global variable 2008-09-02), it was changed to "o->call_depth
|| stage == 3".  (index_only == !!o->call_depth).  Later, in 41d70bd6
(merge-recursive: Small code clarification -- variable name and comments),
this was changed to "o->call_depth || renamed_stage == 2" (where stage was
renamed to other_stage and renamed_stage == other_stage ^ 1).

So we ended with <expression> being "o->call_depth || renamed_stage == 2".
But the "o->call_depth ||" piece was unnecessary.  We can remove it,
leaving us with <expression> being "renamed_stage == 2".  This doesn't
change behavior at all, but it makes the code clearer.  Which is good,
because it's about to get uglier.

  Corrected goal: If a file is renamed on the side of history being merged
  into head, the filename serving as the source of that rename needs to be
  removed from the working directory *IF* that file is tracked in head AND
  the file tracked in head is related to the original file.

Note that the only difference between the original goal and the corrected
goal is the two extra conditions added at the end.  The first condition is
relevant in a rename/delete conflict.  If the file was deleted on the
HEAD side of the merge and an untracked file of the same name was added to
the working copy, then without that extra condition the untracked file
will be erroneously deleted.  This changes <expression> to "renamed_stage
== 2 || !was_tracked(ren1_src)".

The second additional condition is relevant in two cases.

The first case the second condition can occur is when a file is deleted
and a completely different file is added with the same name.  To my
knowledge, merge-recursive has no mechanism for detecting deleted-and-
replaced-by-different-file cases, so I am simply punting on this
possibility.

The second case for the second condition to occur is when there is a
rename/rename/add-source conflict.  That is, when the original file was
renamed on both sides of history AND the original filename is being
re-used by some unrelated (but tracked) content.  This case also presents
some additional difficulties for us since we cannot currently detect these
rename/rename/add-source conflicts; as long as the rename detection logic
"optimizes" by ignoring filenames that are present at both ends of the
diff, these conflicts will go unnoticed.  However, rename/rename conflicts
are handled by an entirely separate codepath not being discussed here, so
this case is not relevant for the line of code under consideration.

In summary:
  Change <expression> from "o->call_depth || renamed_stage == 2" to
  "renamed_stage == 2 || !was_tracked(ren1_src)", in order to
  remove unnecessary code and avoid deleting untracked files.

96 lines of explanation in the changelog to describe a one-line fix...

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |    3 ++-
 t/t6039-merge-rename-corner-cases.sh |    2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 987a985..749e501 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1102,7 +1102,8 @@ static int process_renames(struct merge_options *o,
 			int renamed_stage = a_renames == renames1 ? 2 : 3;
 			int other_stage =   a_renames == renames1 ? 3 : 2;
 
-			remove_file(o, 1, ren1_src, o->call_depth || renamed_stage == 2);
+			remove_file(o, 1, ren1_src,
+				    renamed_stage == 2 || !was_tracked(ren1_src));
 
 			hashcpy(src_other.sha1, ren1->src_entry->stages[other_stage].sha);
 			src_other.mode = ren1->src_entry->stages[other_stage].mode;
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 06c7ea5..4f94528 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -26,7 +26,7 @@ test_expect_success 'setup rename/delete + untracked file' '
 	echo "Myyy PRECIOUSSS" >ring
 '
 
-test_expect_failure "Does git preserve Gollum's precious artifact?" '
+test_expect_success "Does git preserve Gollum's precious artifact?" '
 	test_must_fail git merge -s recursive rename-the-ring &&
 
 	# Make sure git did not delete an untracked file
-- 
1.7.6.rc0.62.g2d69f

[PATCH 31/48] merge-recursive: Make dead code for rename/rename(2to1) conflicts undead

[Elijah Newren]

The code for rename_rename_2to1 conflicts (two files both being renamed to
the same filename) was dead since the rename/add path was always being
independently triggered for each of the renames instead.  Further,
reviving the dead code showed that it was inherently buggy and would
always segfault -- among a few other bugs.

Move the else-if branch for the rename/rename block before the rename/add
block to make sure it is checked first, and fix up the rename/rename(2to1)
code segments to make it handle most cases.  Work is still needed to
handle higher dimensional corner cases such as rename/rename/modify/modify
issues.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                 |   70 +++++++++++++++++++++++++-----------
 t/t6036-recursive-corner-cases.sh |   17 +++++----
 2 files changed, 57 insertions(+), 30 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 749e501..4c42838 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -983,17 +983,36 @@ static void conflict_rename_rename_2to1(struct merge_options *o,
 					struct rename *ren2,
 					const char *branch2)
 {
+	char *path = ren1->pair->two->path; /* same as ren2->pair->two->path */
 	/* Two files were renamed to the same thing. */
-	char *new_path1 = unique_path(o, ren1->pair->two->path, branch1);
-	char *new_path2 = unique_path(o, ren2->pair->two->path, branch2);
-	output(o, 1, "Renaming %s to %s and %s to %s instead",
-	       ren1->pair->one->path, new_path1,
-	       ren2->pair->one->path, new_path2);
-	remove_file(o, 0, ren1->pair->two->path, 0);
-	update_file(o, 0, ren1->pair->two->sha1, ren1->pair->two->mode, new_path1);
-	update_file(o, 0, ren2->pair->two->sha1, ren2->pair->two->mode, new_path2);
-	free(new_path2);
-	free(new_path1);
+	if (o->call_depth) {
+		struct merge_file_info mfi;
+		struct diff_filespec one, a, b;
+
+		one.path = a.path = b.path = path;
+		hashcpy(one.sha1, null_sha1);
+		one.mode = 0;
+		hashcpy(a.sha1, ren1->pair->two->sha1);
+		a.mode = ren1->pair->two->mode;
+		hashcpy(b.sha1, ren2->pair->two->sha1);
+		b.mode = ren2->pair->two->mode;
+		mfi = merge_file(o, &one, &a, &b, branch1, branch2);
+		output(o, 1, "Adding merged %s", path);
+		update_file(o, 0, mfi.sha, mfi.mode, path);
+	} else {
+		char *new_path1 = unique_path(o, path, branch1);
+		char *new_path2 = unique_path(o, path, branch2);
+		output(o, 1, "Renaming %s to %s and %s to %s instead",
+		       ren1->pair->one->path, new_path1,
+		       ren2->pair->one->path, new_path2);
+		remove_file(o, 0, path, 0);
+		update_file(o, 0, ren1->pair->two->sha1, ren1->pair->two->mode,
+			    new_path1);
+		update_file(o, 0, ren2->pair->two->sha1, ren2->pair->two->mode,
+			    new_path2);
+		free(new_path2);
+		free(new_path1);
+	}
 }
 
 static int process_renames(struct merge_options *o,
@@ -1008,12 +1027,12 @@ static int process_renames(struct merge_options *o,
 	for (i = 0; i < a_renames->nr; i++) {
 		sre = a_renames->items[i].util;
 		string_list_insert(&a_by_dst, sre->pair->two->path)->util
-			= sre->dst_entry;
+			= (void*)sre;
 	}
 	for (i = 0; i < b_renames->nr; i++) {
 		sre = b_renames->items[i].util;
 		string_list_insert(&b_by_dst, sre->pair->two->path)->util
-			= sre->dst_entry;
+			= (void*)sre;
 	}
 
 	for (i = 0, j = 0; i < a_renames->nr || j < b_renames->nr;) {
@@ -1125,6 +1144,23 @@ static int process_renames(struct merge_options *o,
 					clean_merge = 0;
 					conflict_rename_delete(o, ren1->pair, branch1, branch2);
 				}
+			} else if ((item = string_list_lookup(renames2Dst, ren1_dst))) {
+				char *ren2_src, *ren2_dst;
+				ren2 = item->util;
+				ren2_src = ren2->pair->one->path;
+				ren2_dst = ren2->pair->two->path;
+
+				clean_merge = 0;
+				ren2->processed = 1;
+				remove_file(o, 1, ren2_src,
+					    renamed_stage == 3 || would_lose_untracked(ren1_src));
+
+				output(o, 1, "CONFLICT (rename/rename): "
+				       "Rename %s->%s in %s. "
+				       "Rename %s->%s in %s",
+				       ren1_src, ren1_dst, branch1,
+				       ren2_src, ren2_dst, branch2);
+				conflict_rename_rename_2to1(o, ren1, branch1, ren2, branch2);
 			} else if ((dst_other.mode == ren1->pair->two->mode) &&
 				   sha_eq(dst_other.sha1, ren1->pair->two->sha1)) {
 				/* Added file on the other side
@@ -1165,16 +1201,6 @@ static int process_renames(struct merge_options *o,
 					update_file(o, 0, dst_other.sha1, dst_other.mode, new_path);
 					free(new_path);
 				}
-			} else if ((item = string_list_lookup(renames2Dst, ren1_dst))) {
-				ren2 = item->util;
-				clean_merge = 0;
-				ren2->processed = 1;
-				output(o, 1, "CONFLICT (rename/rename): "
-				       "Rename %s->%s in %s. "
-				       "Rename %s->%s in %s",
-				       ren1_src, ren1_dst, branch1,
-				       ren2->pair->one->path, ren2->pair->two->path, branch2);
-				conflict_rename_rename_2to1(o, ren1, branch1, ren2, branch2);
 			} else
 				try_merge = 1;
 
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index dea2a65..8d1d303 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -60,13 +60,13 @@ test_expect_success 'merge simple rename+criss-cross with no modifications' '
 	test $(git rev-parse :2:three) = $(git rev-parse L2:three) &&
 	test $(git rev-parse :3:three) = $(git rev-parse R2:three) &&
 
-	cp two merged &&
+	cp one merged &&
 	>empty &&
 	test_must_fail git merge-file \
-		-L "Temporary merge branch 2" \
-		-L "" \
 		-L "Temporary merge branch 1" \
-		merged empty one &&
+		-L "" \
+		-L "Temporary merge branch 2" \
+		merged empty two &&
 	test $(git rev-parse :1:three) = $(git hash-object merged)
 '
 
@@ -139,11 +139,12 @@ test_expect_success 'merge criss-cross + rename merges with basic modification'
 	cp one merge-me &&
 	>empty &&
 	test_must_fail git merge-file \
-		-L "Temporary merge branch 2" \
-		-L "" \
 		-L "Temporary merge branch 1" \
-		merged empty merge-me &&
-	test $(git rev-parse :1:three) = $(git hash-object merged)
+		-L "" \
+		-L "Temporary merge branch 2" \
+		merge-me empty merged &&
+
+	test $(git rev-parse :1:three) = $(git hash-object merge-me)
 '
 
 #
-- 
1.7.6.rc0.62.g2d69f

[PATCH 32/48] merge-recursive: Add comments about handling rename/add-source cases

[Elijah Newren]

There are a couple of places where changes are needed to for situations
involving rename/add-source issues.  Add comments about the needed changes
(and existing bugs) until git has been enabled to detect such cases.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 4c42838..5ccc59c 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1072,6 +1072,9 @@ static int process_renames(struct merge_options *o,
 		}
 
 		ren1->dst_entry->processed = 1;
+		/* BUG: We should only mark src_entry as processed if we
+		 * are not dealing with a rename + add-source case.
+		 */
 		ren1->src_entry->processed = 1;
 
 		if (ren1->processed)
@@ -1098,6 +1101,10 @@ static int process_renames(struct merge_options *o,
 							      ren1->dst_entry,
 							      ren2->dst_entry);
 			} else {
+				/* BUG: We should only remove ren1_src in
+				 * the base stage (think of rename +
+				 * add-source cases).
+				 */
 				remove_file(o, 1, ren1_src, 1);
 				update_entry(ren1->dst_entry,
 					     ren1->pair->one,
@@ -1121,6 +1128,10 @@ static int process_renames(struct merge_options *o,
 			int renamed_stage = a_renames == renames1 ? 2 : 3;
 			int other_stage =   a_renames == renames1 ? 3 : 2;
 
+			/* BUG: We should only remove ren1_src in the base
+			 * stage and in other_stage (think of rename +
+			 * add-source case).
+			 */
 			remove_file(o, 1, ren1_src,
 				    renamed_stage == 2 || !was_tracked(ren1_src));
 
-- 
1.7.6.rc0.62.g2d69f

[PATCH 33/48] merge-recursive: Improve handling of rename target vs. directory addition

[Elijah Newren]

When dealing with file merging and renames and D/F conflicts and possible
criss-cross merges (how's that for a corner case?), we did not do a
thorough job ensuring the index and working directory had the correct
contents.   Fix the logic in merge_content() to handle this.  Also,
correct some erroneous tests in t6022 that were expecting the wrong number
of unmerged index entries.  These changes fix one of the tests in t6039
(and almost fix another one from t6039 as well).

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |   27 ++++++++++++++++++++++-----
 t/t6022-merge-rename.sh              |    4 ++--
 t/t6039-merge-rename-corner-cases.sh |    2 +-
 3 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 5ccc59c..e6a97b2 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1370,19 +1370,34 @@ static int merge_content(struct merge_options *o,
 			reason = "submodule";
 		output(o, 1, "CONFLICT (%s): Merge conflict in %s",
 				reason, path);
-		if (involved_in_rename)
+		if (involved_in_rename && !df_conflict_remains)
 			update_stages(path, &one, &a, &b);
 	}
 
 	if (df_conflict_remains) {
 		char *new_path;
-		update_file_flags(o, mfi.sha, mfi.mode, path,
-				  o->call_depth || mfi.clean, 0);
+		if (o->call_depth) {
+			remove_file_from_cache(path);
+		} else {
+			if (!mfi.clean)
+				update_stages(path, &one, &a, &b);
+			else {
+				int file_from_stage2 = was_tracked(path);
+				struct diff_filespec merged;
+				hashcpy(merged.sha1, mfi.sha);
+				merged.mode = mfi.mode;
+
+				update_stages(path, NULL,
+					      file_from_stage2 ? &merged : NULL,
+					      file_from_stage2 ? NULL : &merged);
+			}
+
+		}
 		new_path = unique_path(o, path, df_rename_conflict_branch);
-		mfi.clean = 0;
 		output(o, 1, "Adding as %s instead", new_path);
-		update_file_flags(o, mfi.sha, mfi.mode, new_path, 0, 1);
+		update_file(o, 0, mfi.sha, mfi.mode, new_path);
 		free(new_path);
+		mfi.clean = 0;
 	} else {
 		update_file(o, mfi.clean, mfi.sha, mfi.mode, path);
 	}
@@ -1574,6 +1589,8 @@ static int process_df_entry(struct merge_options *o,
 			output(o, 1, "CONFLICT (%s): There is a directory with name %s in %s. "
 			       "Adding %s as %s",
 			       conf, path, other_branch, path, new_path);
+			if (o->call_depth)
+				remove_file_from_cache(path);
 			update_file(o, 0, sha, mode, new_path);
 			if (o->call_depth)
 				remove_file_from_cache(path);
diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index 5e2a686..cd1e8fb 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -307,7 +307,7 @@ test_expect_success 'Rename+D/F conflict; renamed file merges but dir in way' '
 	grep "Auto-merging dir" output &&
 	grep "Adding as dir~HEAD instead" output &&
 
-	test 2 -eq "$(git ls-files -u | wc -l)" &&
+	test 3 -eq "$(git ls-files -u | wc -l)" &&
 	test 2 -eq "$(git ls-files -u dir/file-in-the-way | wc -l)" &&
 
 	test_must_fail git diff --quiet &&
@@ -329,7 +329,7 @@ test_expect_success 'Same as previous, but merged other way' '
 	grep "Auto-merging dir" output &&
 	grep "Adding as dir~renamed-file-has-no-conflicts instead" output &&
 
-	test 2 -eq "$(git ls-files -u | wc -l)" &&
+	test 3 -eq "$(git ls-files -u | wc -l)" &&
 	test 2 -eq "$(git ls-files -u dir/file-in-the-way | wc -l)" &&
 
 	test_must_fail git diff --quiet &&
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 4f94528..18e7821 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -234,7 +234,7 @@ test_expect_success 'setup content merge + rename/directory conflict' '
 	git commit -m left
 '
 
-test_expect_failure 'rename/directory conflict + clean content merge' '
+test_expect_success 'rename/directory conflict + clean content merge' '
 	git reset --hard &&
 	git reset --hard &&
 	git clean -fdqx &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 34/48] merge-recursive: Consolidate process_entry() and process_df_entry()

[Elijah Newren]

The whole point of adding process_df_entry() was to ensure that files of
D/F conflicts were processed after paths under the corresponding
directory.  However, given that the entries are in sorted order, all we
need to do is iterate through them in reverse order to achieve the same
effect.  That lets us remove some duplicated code, and lets us keep
track of one less thing as we read the code ("do we need to make sure
this is processed before process_df_entry() or do we need to defer it
until then?").

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |  194 ++++++++++++++++------------------------------------
 1 files changed, 60 insertions(+), 134 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index e6a97b2..5673ccb 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -116,7 +116,6 @@ static inline void setup_rename_df_conflict_info(enum rename_type rename_type,
 		ci->dst_entry2 = dst_entry2;
 		ci->pair2 = pair2;
 		dst_entry2->rename_df_conflict_info = ci;
-		dst_entry2->processed = 0;
 	}
 }
 
@@ -358,20 +357,17 @@ static void record_df_conflict_files(struct merge_options *o,
 				     struct string_list *entries)
 {
 	/* If there is a D/F conflict and the file for such a conflict
-	 * currently exist in the working copy, we want to allow it to
-	 * be removed to make room for the corresponding directory if
-	 * needed.  The files underneath the directories of such D/F
-	 * conflicts will be handled in process_entry(), while the
-	 * files of such D/F conflicts will be processed later in
-	 * process_df_entry().  If the corresponding directory ends up
-	 * being removed by the merge, then no additional work needs
-	 * to be done by process_df_entry() for the conflicting file.
-	 * If the directory needs to be written to the working copy,
-	 * then the conflicting file will simply be removed (e.g. in
-	 * make_room_for_path).  If the directory is written to the
-	 * working copy but the file also has a conflict that needs to
-	 * be resolved, then process_df_entry() will reinstate the
-	 * file with a new unique name.
+	 * currently exist in the working copy, we want to allow it to be
+	 * removed to make room for the corresponding directory if needed.
+	 * The files underneath the directories of such D/F conflicts will
+	 * be processed before the corresponding file involved in the D/F
+	 * conflict.  If the D/F directory ends up being removed by the
+	 * merge, then we won't have to touch the D/F file.  If the D/F
+	 * directory needs to be written to the working copy, then the D/F
+	 * file will simply be removed (in make_room_for_path()) to make
+	 * room for the necessary paths.  Note that if both the directory
+	 * and the file need to be present, then the D/F file will be
+	 * reinstated with a new unique name at the time it is processed.
 	 */
 	const char *last_file = NULL;
 	int last_len = 0;
@@ -1310,17 +1306,17 @@ static void handle_delete_modify(struct merge_options *o,
 		       "and modified in %s. Version %s of %s left in tree%s%s.",
 		       path, o->branch1,
 		       o->branch2, o->branch2, path,
-		       path == new_path ? "" : " at ",
-		       path == new_path ? "" : new_path);
-		update_file(o, 0, b_sha, b_mode, new_path);
+		       NULL == new_path ? "" : " at ",
+		       NULL == new_path ? "" : new_path);
+		update_file(o, 0, b_sha, b_mode, new_path ? new_path : path);
 	} else {
 		output(o, 1, "CONFLICT (delete/modify): %s deleted in %s "
 		       "and modified in %s. Version %s of %s left in tree%s%s.",
 		       path, o->branch2,
 		       o->branch1, o->branch1, path,
-		       path == new_path ? "" : " at ",
-		       path == new_path ? "" : new_path);
-		update_file(o, 0, a_sha, a_mode, new_path);
+		       NULL == new_path ? "" : " at ",
+		       NULL == new_path ? "" : new_path);
+		update_file(o, 0, a_sha, a_mode, new_path ? new_path : path);
 	}
 }
 
@@ -1422,93 +1418,6 @@ static int process_entry(struct merge_options *o,
 	unsigned char *a_sha = stage_sha(entry->stages[2].sha, a_mode);
 	unsigned char *b_sha = stage_sha(entry->stages[3].sha, b_mode);
 
-	if (entry->rename_df_conflict_info)
-		return 1; /* Such cases are handled elsewhere. */
-
-	entry->processed = 1;
-	if (o_sha && (!a_sha || !b_sha)) {
-		/* Case A: Deleted in one */
-		if ((!a_sha && !b_sha) ||
-		    (!b_sha && blob_unchanged(o_sha, a_sha, normalize, path)) ||
-		    (!a_sha && blob_unchanged(o_sha, b_sha, normalize, path))) {
-			/* Deleted in both or deleted in one and
-			 * unchanged in the other */
-			if (a_sha)
-				output(o, 2, "Removing %s", path);
-			/* do not touch working file if it did not exist */
-			remove_file(o, 1, path, !a_sha);
-		} else if (dir_in_way(path, 0 /*check_wc*/)) {
-			entry->processed = 0;
-			return 1; /* Assume clean until processed */
-		} else {
-			/* Deleted in one and changed in the other */
-			clean_merge = 0;
-			handle_delete_modify(o, path, path,
-					     a_sha, a_mode, b_sha, b_mode);
-		}
-
-	} else if ((!o_sha && a_sha && !b_sha) ||
-		   (!o_sha && !a_sha && b_sha)) {
-		/* Case B: Added in one. */
-		unsigned mode;
-		const unsigned char *sha;
-
-		if (a_sha) {
-			mode = a_mode;
-			sha = a_sha;
-		} else {
-			mode = b_mode;
-			sha = b_sha;
-		}
-		if (dir_in_way(path, 0 /*check_wc*/)) {
-			/* Handle D->F conflicts after all subfiles */
-			entry->processed = 0;
-			return 1; /* Assume clean until processed */
-		} else {
-			output(o, 2, "Adding %s", path);
-			update_file(o, 1, sha, mode, path);
-		}
-	} else if (a_sha && b_sha) {
-		/* Case C: Added in both (check for same permissions) and */
-		/* case D: Modified in both, but differently. */
-		clean_merge = merge_content(o, entry->involved_in_rename, path,
-					    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
-					    NULL);
-	} else if (!o_sha && !a_sha && !b_sha) {
-		/*
-		 * this entry was deleted altogether. a_mode == 0 means
-		 * we had that path and want to actively remove it.
-		 */
-		remove_file(o, 1, path, !a_mode);
-	} else
-		die("Fatal merge failure, shouldn't happen.");
-
-	return clean_merge;
-}
-
-/*
- * Per entry merge function for D/F (and/or rename) conflicts.  In the
- * cases we can cleanly resolve D/F conflicts, process_entry() can
- * clean out all the files below the directory for us.  All D/F
- * conflict cases must be handled here at the end to make sure any
- * directories that can be cleaned out, are.
- *
- * Some rename conflicts may also be handled here that don't necessarily
- * involve D/F conflicts, since the code to handle them is generic enough
- * to handle those rename conflicts with or without D/F conflicts also
- * being involved.
- */
-static int process_df_entry(struct merge_options *o,
-			    const char *path, struct stage_data *entry)
-{
-	int clean_merge = 1;
-	unsigned o_mode = entry->stages[1].mode;
-	unsigned a_mode = entry->stages[2].mode;
-	unsigned b_mode = entry->stages[3].mode;
-	unsigned char *o_sha = stage_sha(entry->stages[1].sha, o_mode);
-	unsigned char *a_sha = stage_sha(entry->stages[2].sha, a_mode);
-	unsigned char *b_sha = stage_sha(entry->stages[3].sha, b_mode);
-
 	entry->processed = 1;
 	if (entry->rename_df_conflict_info) {
 		struct rename_df_conflict_info *conflict_info = entry->rename_df_conflict_info;
@@ -1543,27 +1452,41 @@ static int process_df_entry(struct merge_options *o,
 						    conflict_info->branch1,
 						    conflict_info->pair2,
 						    conflict_info->branch2);
-			conflict_info->dst_entry2->processed = 1;
 			break;
 		default:
 			entry->processed = 0;
 			break;
 		}
 	} else if (o_sha && (!a_sha || !b_sha)) {
-		/* Modify/delete; deleted side may have put a directory in the way */
-		char *new_path;
-		int free_me = 0;
-		if (dir_in_way(path, !o->call_depth)) {
-			new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
-			free_me = 1;
+		/* Case A: Deleted in one */
+		if ((!a_sha && !b_sha) ||
+		    (!b_sha && blob_unchanged(o_sha, a_sha, normalize, path)) ||
+		    (!a_sha && blob_unchanged(o_sha, b_sha, normalize, path))) {
+			/* Deleted in both or deleted in one and
+			 * unchanged in the other */
+			if (a_sha)
+				output(o, 2, "Removing %s", path);
+			/* do not touch working file if it did not exist */
+			remove_file(o, 1, path, !a_sha);
+		} else {
+			/* Modify/delete; deleted side may have put a directory in the way */
+			char *new_path = NULL;
+			int free_me = 0;
+			clean_merge = 0;
+			if (dir_in_way(path, !o->call_depth)) {
+				new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
+				free_me = 1;
+			}
+			handle_delete_modify(o, path, new_path,
+					     a_sha, a_mode, b_sha, b_mode);
+			if (free_me)
+				free(new_path);
 		}
-		clean_merge = 0;
-		handle_delete_modify(o, path, free_me ? new_path : path,
-				     a_sha, a_mode, b_sha, b_mode);
-		if (free_me)
-			free(new_path);
-	} else if (!o_sha && !!a_sha != !!b_sha) {
-		/* directory -> (directory, file) or <nothing> -> (directory, file) */
+	} else if ((!o_sha && a_sha && !b_sha) ||
+		   (!o_sha && !a_sha && b_sha)) {
+		/* Case B: Added in one. */
+		/* [nothing|directory] -> ([nothing|directory], file) */
+
 		const char *add_branch;
 		const char *other_branch;
 		unsigned mode;
@@ -1599,10 +1522,20 @@ static int process_df_entry(struct merge_options *o,
 			output(o, 2, "Adding %s", path);
 			update_file(o, 1, sha, mode, path);
 		}
-	} else {
-		entry->processed = 0;
-		return 1; /* not handled; assume clean until processed */
-	}
+	} else if (a_sha && b_sha) {
+		/* Case C: Added in both (check for same permissions) and */
+		/* case D: Modified in both, but differently. */
+		clean_merge = merge_content(o, entry->involved_in_rename, path,
+					    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
+					    NULL);
+	} else if (!o_sha && !a_sha && !b_sha) {
+		/*
+		 * this entry was deleted altogether. a_mode == 0 means
+		 * we had that path and want to actively remove it.
+		 */
+		remove_file(o, 1, path, !a_mode);
+	} else
+		die("Fatal merge failure, shouldn't happen.");
 
 	return clean_merge;
 }
@@ -1650,7 +1583,7 @@ int merge_trees(struct merge_options *o,
 		re_head  = get_renames(o, head, common, head, merge, entries);
 		re_merge = get_renames(o, merge, common, head, merge, entries);
 		clean = process_renames(o, re_head, re_merge);
-		for (i = 0; i < entries->nr; i++) {
+		for (i = entries->nr-1; i >=0; i--) {
 			const char *path = entries->items[i].string;
 			struct stage_data *e = entries->items[i].util;
 			if (!e->processed
@@ -1658,13 +1591,6 @@ int merge_trees(struct merge_options *o,
 				clean = 0;
 		}
 		for (i = 0; i < entries->nr; i++) {
-			const char *path = entries->items[i].string;
-			struct stage_data *e = entries->items[i].util;
-			if (!e->processed
-				&& !process_df_entry(o, path, e))
-				clean = 0;
-		}
-		for (i = 0; i < entries->nr; i++) {
 			struct stage_data *e = entries->items[i].util;
 			if (!e->processed)
 				die("Unprocessed path??? %s",
-- 
1.7.6.rc0.62.g2d69f

[PATCH 35/48] merge-recursive: Cleanup and consolidation of rename_conflict_info

[Elijah Newren]

The consolidation of process_entry() and process_df_entry() allows us to
consolidate more code paths concerning rename conflicts, and to do
a few additional related cleanups.  It also means we are using
rename_df_conflict_info in some cases where there is no D/F conflict;
rename it to rename_conflict_info.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |  134 ++++++++++++++++++++++++++---------------------------
 1 files changed, 66 insertions(+), 68 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 5673ccb..5d0a62c 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -66,10 +66,11 @@ static int sha_eq(const unsigned char *a, const unsigned char *b)
 enum rename_type {
 	RENAME_NORMAL = 0,
 	RENAME_DELETE,
+	RENAME_ONE_FILE_TO_ONE,
 	RENAME_ONE_FILE_TO_TWO
 };
 
-struct rename_df_conflict_info {
+struct rename_conflict_info {
 	enum rename_type rename_type;
 	struct diff_filepair *pair1;
 	struct diff_filepair *pair2;
@@ -88,34 +89,33 @@ struct stage_data {
 		unsigned mode;
 		unsigned char sha[20];
 	} stages[4];
-	struct rename_df_conflict_info *rename_df_conflict_info;
+	struct rename_conflict_info *rename_conflict_info;
 	unsigned processed:1;
-	unsigned involved_in_rename:1;
 };
 
-static inline void setup_rename_df_conflict_info(enum rename_type rename_type,
-						 struct diff_filepair *pair1,
-						 struct diff_filepair *pair2,
-						 const char *branch1,
-						 const char *branch2,
-						 struct stage_data *dst_entry1,
-						 struct stage_data *dst_entry2)
+static inline void setup_rename_conflict_info(enum rename_type rename_type,
+					      struct diff_filepair *pair1,
+					      struct diff_filepair *pair2,
+					      const char *branch1,
+					      const char *branch2,
+					      struct stage_data *dst_entry1,
+					      struct stage_data *dst_entry2)
 {
-	struct rename_df_conflict_info *ci = xcalloc(1, sizeof(struct rename_df_conflict_info));
+	struct rename_conflict_info *ci = xcalloc(1, sizeof(struct rename_conflict_info));
 	ci->rename_type = rename_type;
 	ci->pair1 = pair1;
 	ci->branch1 = branch1;
 	ci->branch2 = branch2;
 
 	ci->dst_entry1 = dst_entry1;
-	dst_entry1->rename_df_conflict_info = ci;
+	dst_entry1->rename_conflict_info = ci;
 	dst_entry1->processed = 0;
 
 	assert(!pair2 == !dst_entry2);
 	if (dst_entry2) {
 		ci->dst_entry2 = dst_entry2;
 		ci->pair2 = pair2;
-		dst_entry2->rename_df_conflict_info = ci;
+		dst_entry2->rename_conflict_info = ci;
 	}
 }
 
@@ -939,10 +939,29 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 	/* One file was renamed in both branches, but to different names. */
 	char *del[2];
 	int delp = 0;
+	const char *src      = pair1->one->path;
 	const char *ren1_dst = pair1->two->path;
 	const char *ren2_dst = pair2->two->path;
 	const char *dst_name1 = ren1_dst;
 	const char *dst_name2 = ren2_dst;
+
+	output(o, 1, "CONFLICT (rename/rename): "
+	       "Rename \"%s\"->\"%s\" in branch \"%s\" "
+	       "rename \"%s\"->\"%s\" in \"%s\"%s",
+	       src, pair1->two->path, branch1,
+	       src, pair2->two->path, branch2,
+	       o->call_depth ? " (left unresolved)" : "");
+	if (o->call_depth) {
+		/*
+		 * FIXME: Why remove file from cache, and then
+		 * immediately readd it?  Why not just overwrite using
+		 * update_file only?  Also...this is buggy for
+		 * rename/add-source situations...
+		 */
+		remove_file_from_cache(src);
+		update_file(o, 0, pair1->one->sha1, pair1->one->mode, src);
+	}
+
 	if (dir_in_way(ren1_dst, !o->call_depth)) {
 		dst_name1 = del[delp++] = unique_path(o, ren1_dst, branch1);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
@@ -1083,20 +1102,16 @@ static int process_renames(struct merge_options *o,
 		if (ren2) {
 			const char *ren2_src = ren2->pair->one->path;
 			const char *ren2_dst = ren2->pair->two->path;
+			enum rename_type rename_type;
 			/* Renamed in 1 and renamed in 2 */
 			if (strcmp(ren1_src, ren2_src) != 0)
 				die("ren1.src != ren2.src");
 			ren2->dst_entry->processed = 1;
 			ren2->processed = 1;
 			if (strcmp(ren1_dst, ren2_dst) != 0) {
-				setup_rename_df_conflict_info(RENAME_ONE_FILE_TO_TWO,
-							      ren1->pair,
-							      ren2->pair,
-							      branch1,
-							      branch2,
-							      ren1->dst_entry,
-							      ren2->dst_entry);
+				rename_type = RENAME_ONE_FILE_TO_TWO;
 			} else {
+				rename_type = RENAME_ONE_FILE_TO_ONE;
 				/* BUG: We should only remove ren1_src in
 				 * the base stage (think of rename +
 				 * add-source cases).
@@ -1106,8 +1121,14 @@ static int process_renames(struct merge_options *o,
 					     ren1->pair->one,
 					     ren1->pair->two,
 					     ren2->pair->two);
-				ren1->dst_entry->involved_in_rename = 1;
 			}
+			setup_rename_conflict_info(rename_type,
+						   ren1->pair,
+						   ren2->pair,
+						   branch1,
+						   branch2,
+						   ren1->dst_entry,
+						   ren2->dst_entry);
 		} else {
 			/* Renamed in 1, maybe changed in 2 */
 			struct string_list_item *item;
@@ -1138,19 +1159,13 @@ static int process_renames(struct merge_options *o,
 			try_merge = 0;
 
 			if (sha_eq(src_other.sha1, null_sha1)) {
-				if (dir_in_way(ren1_dst, 0 /*check_wc*/)) {
-					ren1->dst_entry->processed = 0;
-					setup_rename_df_conflict_info(RENAME_DELETE,
-								      ren1->pair,
-								      NULL,
-								      branch1,
-								      branch2,
-								      ren1->dst_entry,
-								      NULL);
-				} else {
-					clean_merge = 0;
-					conflict_rename_delete(o, ren1->pair, branch1, branch2);
-				}
+				setup_rename_conflict_info(RENAME_DELETE,
+							   ren1->pair,
+							   NULL,
+							   branch1,
+							   branch2,
+							   ren1->dst_entry,
+							   NULL);
 			} else if ((item = string_list_lookup(renames2Dst, ren1_dst))) {
 				char *ren2_src, *ren2_dst;
 				ren2 = item->util;
@@ -1224,16 +1239,13 @@ static int process_renames(struct merge_options *o,
 					a = &src_other;
 				}
 				update_entry(ren1->dst_entry, one, a, b);
-				ren1->dst_entry->involved_in_rename = 1;
-				if (dir_in_way(ren1_dst, 0 /*check_wc*/)) {
-					setup_rename_df_conflict_info(RENAME_NORMAL,
-								      ren1->pair,
-								      NULL,
-								      branch1,
-								      NULL,
-								      ren1->dst_entry,
-								      NULL);
-				}
+				setup_rename_conflict_info(RENAME_NORMAL,
+							   ren1->pair,
+							   NULL,
+							   branch1,
+							   NULL,
+							   ren1->dst_entry,
+							   NULL);
 			}
 		}
 	}
@@ -1321,12 +1333,11 @@ static void handle_delete_modify(struct merge_options *o,
 }
 
 static int merge_content(struct merge_options *o,
-			 unsigned involved_in_rename,
 			 const char *path,
 			 unsigned char *o_sha, int o_mode,
 			 unsigned char *a_sha, int a_mode,
 			 unsigned char *b_sha, int b_mode,
-			 const char *df_rename_conflict_branch)
+			 struct rename_conflict_info *rename_conflict_info)
 {
 	const char *reason = "content";
 	struct merge_file_info mfi;
@@ -1346,8 +1357,7 @@ static int merge_content(struct merge_options *o,
 	b.mode = b_mode;
 
 	mfi = merge_file(o, &one, &a, &b, o->branch1, o->branch2);
-	if (df_rename_conflict_branch &&
-	    dir_in_way(path, !o->call_depth)) {
+	if (rename_conflict_info && dir_in_way(path, !o->call_depth)) {
 		df_conflict_remains = 1;
 	}
 
@@ -1366,7 +1376,7 @@ static int merge_content(struct merge_options *o,
 			reason = "submodule";
 		output(o, 1, "CONFLICT (%s): Merge conflict in %s",
 				reason, path);
-		if (involved_in_rename && !df_conflict_remains)
+		if (rename_conflict_info && !df_conflict_remains)
 			update_stages(path, &one, &a, &b);
 	}
 
@@ -1389,7 +1399,7 @@ static int merge_content(struct merge_options *o,
 			}
 
 		}
-		new_path = unique_path(o, path, df_rename_conflict_branch);
+		new_path = unique_path(o, path, rename_conflict_info->branch1);
 		output(o, 1, "Adding as %s instead", new_path);
 		update_file(o, 0, mfi.sha, mfi.mode, new_path);
 		free(new_path);
@@ -1419,14 +1429,14 @@ static int process_entry(struct merge_options *o,
 	unsigned char *b_sha = stage_sha(entry->stages[3].sha, b_mode);
 
 	entry->processed = 1;
-	if (entry->rename_df_conflict_info) {
-		struct rename_df_conflict_info *conflict_info = entry->rename_df_conflict_info;
-		char *src;
+	if (entry->rename_conflict_info) {
+		struct rename_conflict_info *conflict_info = entry->rename_conflict_info;
 		switch (conflict_info->rename_type) {
 		case RENAME_NORMAL:
-			clean_merge = merge_content(o, entry->involved_in_rename, path,
+		case RENAME_ONE_FILE_TO_ONE:
+			clean_merge = merge_content(o, path,
 						    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
-						    conflict_info->branch1);
+						    conflict_info);
 			break;
 		case RENAME_DELETE:
 			clean_merge = 0;
@@ -1435,19 +1445,7 @@ static int process_entry(struct merge_options *o,
 					       conflict_info->branch2);
 			break;
 		case RENAME_ONE_FILE_TO_TWO:
-			src = conflict_info->pair1->one->path;
 			clean_merge = 0;
-			output(o, 1, "CONFLICT (rename/rename): "
-			       "Rename \"%s\"->\"%s\" in branch \"%s\" "
-			       "rename \"%s\"->\"%s\" in \"%s\"%s",
-			       src, conflict_info->pair1->two->path, conflict_info->branch1,
-			       src, conflict_info->pair2->two->path, conflict_info->branch2,
-			       o->call_depth ? " (left unresolved)" : "");
-			if (o->call_depth) {
-				remove_file_from_cache(src);
-				update_file(o, 0, conflict_info->pair1->one->sha1,
-					    conflict_info->pair1->one->mode, src);
-			}
 			conflict_rename_rename_1to2(o, conflict_info->pair1,
 						    conflict_info->branch1,
 						    conflict_info->pair2,
@@ -1525,7 +1523,7 @@ static int process_entry(struct merge_options *o,
 	} else if (a_sha && b_sha) {
 		/* Case C: Added in both (check for same permissions) and */
 		/* case D: Modified in both, but differently. */
-		clean_merge = merge_content(o, entry->involved_in_rename, path,
+		clean_merge = merge_content(o, path,
 					    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
 					    NULL);
 	} else if (!o_sha && !a_sha && !b_sha) {
-- 
1.7.6.rc0.62.g2d69f

[PATCH 36/48] merge-recursive: Provide more info in conflict markers with file renames

[Elijah Newren]

Whenever there are merge conflicts in file contents, we would mark the
different sides of the conflict with the two branches being merged.
However, when there is a rename involved as well, the branchname is not
sufficient to specify where the conflicting content came from.  In such
cases, mark the two sides of the conflict with branchname:filename rather
than just branchname.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |   20 +++++++++++++++++---
 t/t6022-merge-rename.sh              |    8 ++++----
 t/t6039-merge-rename-corner-cases.sh |    2 +-
 3 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 5d0a62c..da507a3 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1340,6 +1340,7 @@ static int merge_content(struct merge_options *o,
 			 struct rename_conflict_info *rename_conflict_info)
 {
 	const char *reason = "content";
+	char *side1 = NULL, *side2 = NULL;
 	struct merge_file_info mfi;
 	struct diff_filespec one, a, b;
 	unsigned df_conflict_remains = 0;
@@ -1356,10 +1357,23 @@ static int merge_content(struct merge_options *o,
 	hashcpy(b.sha1, b_sha);
 	b.mode = b_mode;
 
-	mfi = merge_file(o, &one, &a, &b, o->branch1, o->branch2);
-	if (rename_conflict_info && dir_in_way(path, !o->call_depth)) {
-		df_conflict_remains = 1;
+	if (rename_conflict_info) {
+		const char *path1 = rename_conflict_info->pair2 ?
+				    rename_conflict_info->pair2->one->path : path;
+		const char *path2 = rename_conflict_info->pair1 ?
+				    rename_conflict_info->pair1->one->path : path;
+		side1 = xmalloc(strlen(o->branch1) + strlen(path1) + 2);
+		side2 = xmalloc(strlen(o->branch2) + strlen(path2) + 2);
+		sprintf(side1, "%s:%s", o->branch1, path1);
+		sprintf(side2, "%s:%s", o->branch2, path2);
+
+		if (dir_in_way(path, !o->call_depth))
+			df_conflict_remains = 1;
 	}
+	mfi = merge_file(o, &one, &a, &b,
+			 side1 ? side1:o->branch1, side2 ? side2:o->branch2);
+	free(side1);
+	free(side2);
 
 	if (mfi.clean && !df_conflict_remains &&
 	    sha_eq(mfi.sha, a_sha) && mfi.mode == a.mode &&
diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index cd1e8fb..cfce3d3 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -351,11 +351,11 @@ cat >expected <<\EOF &&
 8
 9
 10
-<<<<<<< HEAD
+<<<<<<< HEAD:dir
 12
 =======
 11
->>>>>>> dir-not-in-way
+>>>>>>> dir-not-in-way:sub/file
 EOF
 
 test_expect_success 'Rename+D/F conflict; renamed file cannot merge, dir not in way' '
@@ -405,11 +405,11 @@ cat >expected <<\EOF &&
 8
 9
 10
-<<<<<<< HEAD
+<<<<<<< HEAD:dir
 11
 =======
 12
->>>>>>> renamed-file-has-conflicts
+>>>>>>> renamed-file-has-conflicts:sub/file
 EOF
 
 test_expect_success 'Same as previous, but merged other way' '
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 18e7821..4d2fd10 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -258,7 +258,7 @@ test_expect_success 'rename/directory conflict + clean content merge' '
 	test -f newfile~HEAD
 '
 
-test_expect_failure 'rename/directory conflict + content merge conflict' '
+test_expect_success 'rename/directory conflict + content merge conflict' '
 	git reset --hard &&
 	git reset --hard &&
 	git clean -fdqx &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 37/48] merge-recursive: Fix modify/delete resolution in the recursive case

[Elijah Newren]

When o->call_depth>0 and we have conflicts, we try to find "middle ground"
when creating the virtual merge base.  In the case of content conflicts,
this can be done by doing a three-way content merge and using the result.
In all parts where the three-way content merge is clean, it is the correct
middle ground, and in parts where it conflicts there is no middle ground
but the conflict markers provide a good compromise since they are unlikely
to accidentally match any further changes.

In the case of a modify/delete conflict, we cannot do the same thing.
Accepting either endpoint as the resolution for the virtual merge base
runs the risk that when handling the non-recursive case we will silently
accept one person's resolution over another without flagging a conflict.
In this case, the closest "middle ground" we have is actually the merge
base of the candidate merge bases.  (We could alternatively attempt a
three way content merge using an empty file in place of the deleted file,
but that seems to be more work than necessary.)

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                 |   32 +++++++++++++++++++++-----------
 t/t6036-recursive-corner-cases.sh |    4 ++--
 2 files changed, 23 insertions(+), 13 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index da507a3..5a70a87 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1309,11 +1309,26 @@ error_return:
 
 static void handle_delete_modify(struct merge_options *o,
 				 const char *path,
-				 const char *new_path,
+				 unsigned char *o_sha, int o_mode,
 				 unsigned char *a_sha, int a_mode,
 				 unsigned char *b_sha, int b_mode)
 {
-	if (!a_sha) {
+	char *new_path = NULL;
+	int free_me = 0;
+	if (dir_in_way(path, !o->call_depth)) {
+		new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
+		free_me = 1;
+	}
+
+	if (o->call_depth) {
+		/*
+		 * We cannot arbitrarily accept either a_sha or b_sha as
+		 * correct; since there is no true "middle point" between
+		 * them, simply reuse the base version for virtual merge base.
+		 */
+		remove_file_from_cache(path);
+		update_file(o, 0, o_sha, o_mode, new_path ? new_path : path);
+	} else if (!a_sha) {
 		output(o, 1, "CONFLICT (delete/modify): %s deleted in %s "
 		       "and modified in %s. Version %s of %s left in tree%s%s.",
 		       path, o->branch1,
@@ -1330,6 +1345,9 @@ static void handle_delete_modify(struct merge_options *o,
 		       NULL == new_path ? "" : new_path);
 		update_file(o, 0, a_sha, a_mode, new_path ? new_path : path);
 	}
+	if (free_me)
+		free(new_path);
+
 }
 
 static int merge_content(struct merge_options *o,
@@ -1482,17 +1500,9 @@ static int process_entry(struct merge_options *o,
 			remove_file(o, 1, path, !a_sha);
 		} else {
 			/* Modify/delete; deleted side may have put a directory in the way */
-			char *new_path = NULL;
-			int free_me = 0;
 			clean_merge = 0;
-			if (dir_in_way(path, !o->call_depth)) {
-				new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
-				free_me = 1;
-			}
-			handle_delete_modify(o, path, new_path,
+			handle_delete_modify(o, path, o_sha, o_mode,
 					     a_sha, a_mode, b_sha, b_mode);
-			if (free_me)
-				free(new_path);
 		}
 	} else if ((!o_sha && a_sha && !b_sha) ||
 		   (!o_sha && !a_sha && b_sha)) {
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index 8d1d303..d27477b 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -290,7 +290,7 @@ test_expect_success 'setup criss-cross + modify/delete resolved differently' '
 	git tag E
 '
 
-test_expect_failure 'git detects conflict merging criss-cross+modify/delete' '
+test_expect_success 'git detects conflict merging criss-cross+modify/delete' '
 	git checkout D^0 &&
 
 	test_must_fail git merge -s recursive E^0 &&
@@ -302,7 +302,7 @@ test_expect_failure 'git detects conflict merging criss-cross+modify/delete' '
 	test $(git rev-parse :2:file) = $(git rev-parse B:file)
 '
 
-test_expect_failure 'git detects conflict merging criss-cross+modify/delete, reverse direction' '
+test_expect_success 'git detects conflict merging criss-cross+modify/delete, reverse direction' '
 	git reset --hard &&
 	git checkout E^0 &&
 
-- 
1.7.6.rc0.62.g2d69f

[PATCH 38/48] merge-recursive: Introduce a merge_file convenience function

[Elijah Newren]

merge_file previously required diff_filespec arguments, but all callers
only had sha1s and modes.  Rename merge_file to merge_file_1 and introduce
a new merge_file convenience function which takes the sha1s and modes and
creates the temporary diff_filespec variables needed to call merge_file_1.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   74 +++++++++++++++++++++++++++-------------------------
 1 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 5a70a87..3a364fa 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -831,12 +831,12 @@ static int merge_3way(struct merge_options *o,
 	return merge_status;
 }
 
-static struct merge_file_info merge_file(struct merge_options *o,
-					 const struct diff_filespec *one,
-					 const struct diff_filespec *a,
-					 const struct diff_filespec *b,
-					 const char *branch1,
-					 const char *branch2)
+static struct merge_file_info merge_file_1(struct merge_options *o,
+					   const struct diff_filespec *one,
+					   const struct diff_filespec *a,
+					   const struct diff_filespec *b,
+					   const char *branch1,
+					   const char *branch2)
 {
 	struct merge_file_info result;
 	result.merge = 0;
@@ -905,6 +905,26 @@ static struct merge_file_info merge_file(struct merge_options *o,
 	return result;
 }
 
+static struct merge_file_info merge_file(struct merge_options *o,
+					 const char *path,
+					 const unsigned char *o_sha, int o_mode,
+					 const unsigned char *a_sha, int a_mode,
+					 const unsigned char *b_sha, int b_mode,
+					 const char *branch1,
+					 const char *branch2)
+{
+	struct diff_filespec one, a, b;
+
+	one.path = a.path = b.path = (char*)path;
+	hashcpy(one.sha1, o_sha);
+	one.mode = o_mode;
+	hashcpy(a.sha1, a_sha);
+	a.mode = a_mode;
+	hashcpy(b.sha1, b_sha);
+	b.mode = b_mode;
+	return merge_file_1(o, &one, &a, &b, branch1, branch2);
+}
+
 static void conflict_rename_delete(struct merge_options *o,
 				   struct diff_filepair *pair,
 				   const char *rename_branch,
@@ -1002,16 +1022,10 @@ static void conflict_rename_rename_2to1(struct merge_options *o,
 	/* Two files were renamed to the same thing. */
 	if (o->call_depth) {
 		struct merge_file_info mfi;
-		struct diff_filespec one, a, b;
-
-		one.path = a.path = b.path = path;
-		hashcpy(one.sha1, null_sha1);
-		one.mode = 0;
-		hashcpy(a.sha1, ren1->pair->two->sha1);
-		a.mode = ren1->pair->two->mode;
-		hashcpy(b.sha1, ren2->pair->two->sha1);
-		b.mode = ren2->pair->two->mode;
-		mfi = merge_file(o, &one, &a, &b, branch1, branch2);
+		mfi = merge_file(o, path, null_sha1, 0,
+				 ren1->pair->two->sha1, ren1->pair->two->mode,
+				 ren2->pair->two->sha1, ren2->pair->two->mode,
+				 branch1, branch2);
 		output(o, 1, "Adding merged %s", path);
 		update_file(o, 0, mfi.sha, mfi.mode, path);
 	} else {
@@ -1198,24 +1212,12 @@ static int process_renames(struct merge_options *o,
 				       ren1_dst, branch2);
 				if (o->call_depth) {
 					struct merge_file_info mfi;
-					struct diff_filespec one, a, b;
-
-					one.path = a.path = b.path =
-						(char *)ren1_dst;
-					hashcpy(one.sha1, null_sha1);
-					one.mode = 0;
-					hashcpy(a.sha1, ren1->pair->two->sha1);
-					a.mode = ren1->pair->two->mode;
-					hashcpy(b.sha1, dst_other.sha1);
-					b.mode = dst_other.mode;
-					mfi = merge_file(o, &one, &a, &b,
-							 branch1,
-							 branch2);
+					mfi = merge_file(o, ren1_dst, null_sha1, 0,
+							 ren1->pair->two->sha1, ren1->pair->two->mode,
+							 dst_other.sha1, dst_other.mode,
+							 branch1, branch2);
 					output(o, 1, "Adding merged %s", ren1_dst);
-					update_file(o, 0,
-						    mfi.sha,
-						    mfi.mode,
-						    ren1_dst);
+					update_file(o, 0, mfi.sha, mfi.mode, ren1_dst);
 					try_merge = 0;
 				} else {
 					char *new_path = unique_path(o, ren1_dst, branch2);
@@ -1388,13 +1390,13 @@ static int merge_content(struct merge_options *o,
 		if (dir_in_way(path, !o->call_depth))
 			df_conflict_remains = 1;
 	}
-	mfi = merge_file(o, &one, &a, &b,
-			 side1 ? side1:o->branch1, side2 ? side2:o->branch2);
+	mfi = merge_file_1(o, &one, &a, &b,
+			   side1 ? side1:o->branch1, side2 ? side2:o->branch2);
 	free(side1);
 	free(side2);
 
 	if (mfi.clean && !df_conflict_remains &&
-	    sha_eq(mfi.sha, a_sha) && mfi.mode == a.mode &&
+	    sha_eq(mfi.sha, a_sha) && mfi.mode == a_mode &&
 	    was_tracked(path)) {
 		output(o, 3, "Skipped %s (merged same as existing)", path);
 		add_cacheinfo(mfi.mode, mfi.sha, path,
-- 
1.7.6.rc0.62.g2d69f

[PATCH 39/48] merge-recursive: Fix rename/rename(1to2) resolution for virtual merge base

[Elijah Newren]

When renaming one file to two files, we really should be doing a content
merge.  Also, in the recursive case, undoing the renames and recording the
merged file in the index with the source of the rename (while deleting
both destinations) allows the renames to be re-detected in the
non-recursive merge and will result in fewer spurious conflicts.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                 |   30 +++++++++++++-----------------
 t/t6036-recursive-corner-cases.sh |    2 +-
 2 files changed, 14 insertions(+), 18 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 3a364fa..576e02d 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -971,17 +971,6 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 	       src, pair1->two->path, branch1,
 	       src, pair2->two->path, branch2,
 	       o->call_depth ? " (left unresolved)" : "");
-	if (o->call_depth) {
-		/*
-		 * FIXME: Why remove file from cache, and then
-		 * immediately readd it?  Why not just overwrite using
-		 * update_file only?  Also...this is buggy for
-		 * rename/add-source situations...
-		 */
-		remove_file_from_cache(src);
-		update_file(o, 0, pair1->one->sha1, pair1->one->mode, src);
-	}
-
 	if (dir_in_way(ren1_dst, !o->call_depth)) {
 		dst_name1 = del[delp++] = unique_path(o, ren1_dst, branch1);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
@@ -993,14 +982,21 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 		       ren2_dst, branch1, dst_name2);
 	}
 	if (o->call_depth) {
-		remove_file_from_cache(dst_name1);
-		remove_file_from_cache(dst_name2);
+		struct merge_file_info mfi;
+		mfi = merge_file(o, src,
+				 pair1->one->sha1, pair1->one->mode,
+				 pair1->two->sha1, pair1->two->mode,
+				 pair2->two->sha1, pair2->two->mode,
+				 branch1, branch2);
 		/*
-		 * Uncomment to leave the conflicting names in the resulting tree
-		 *
-		 * update_file(o, 0, pair1->two->sha1, pair1->two->mode, dst_name1);
-		 * update_file(o, 0, pair2->two->sha1, pair2->two->mode, dst_name2);
+		 * FIXME: For rename/add-source conflicts (if we could detect
+		 * such), this is wrong.  We should instead find a unique
+		 * pathname and then either rename the add-source file to that
+		 * unique path, or use that unique path instead of src here.
 		 */
+		update_file(o, 0, mfi.sha, mfi.mode, src);
+		remove_file_from_cache(ren1_dst);
+		remove_file_from_cache(ren2_dst);
 	} else {
 		update_stages(ren1_dst, NULL, pair1->two, NULL);
 		update_stages(ren2_dst, NULL, NULL, pair2->two);
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index d27477b..a1b609f 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -611,7 +611,7 @@ test_expect_success 'setup rename/rename(1to2)/modify followed by what looks lik
 	git tag E
 '
 
-test_expect_failure 'handle rename/rename(1to2)/modify followed by what looks like rename/rename(2to1)/modify' '
+test_expect_success 'handle rename/rename(1to2)/modify followed by what looks like rename/rename(2to1)/modify' '
 	git checkout D^0 &&
 
 	git merge -s recursive E^0 &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 40/48] merge-recursive: Small cleanups for conflict_rename_rename_1to2

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   60 +++++++++++++++++++++++-----------------------------
 1 files changed, 27 insertions(+), 33 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 576e02d..9c0daf5 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -951,58 +951,55 @@ static void conflict_rename_delete(struct merge_options *o,
 }
 
 static void conflict_rename_rename_1to2(struct merge_options *o,
-					struct diff_filepair *pair1,
-					const char *branch1,
-					struct diff_filepair *pair2,
-					const char *branch2)
+					struct rename_conflict_info *ci)
 {
 	/* One file was renamed in both branches, but to different names. */
+	struct diff_filespec *one = ci->pair1->one;
+	struct diff_filespec *a = ci->pair1->two;
+	struct diff_filespec *b = ci->pair2->two;
+	const char *dst_name_a = a->path;
+	const char *dst_name_b = b->path;
 	char *del[2];
 	int delp = 0;
-	const char *src      = pair1->one->path;
-	const char *ren1_dst = pair1->two->path;
-	const char *ren2_dst = pair2->two->path;
-	const char *dst_name1 = ren1_dst;
-	const char *dst_name2 = ren2_dst;
 
 	output(o, 1, "CONFLICT (rename/rename): "
 	       "Rename \"%s\"->\"%s\" in branch \"%s\" "
 	       "rename \"%s\"->\"%s\" in \"%s\"%s",
-	       src, pair1->two->path, branch1,
-	       src, pair2->two->path, branch2,
+	       one->path, a->path, ci->branch1,
+	       one->path, b->path, ci->branch2,
 	       o->call_depth ? " (left unresolved)" : "");
-	if (dir_in_way(ren1_dst, !o->call_depth)) {
-		dst_name1 = del[delp++] = unique_path(o, ren1_dst, branch1);
+	if (dir_in_way(a->path, !o->call_depth)) {
+		dst_name_a = del[delp++] = unique_path(o, a->path, ci->branch1);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
-		       ren1_dst, branch2, dst_name1);
+		       a->path, ci->branch2, dst_name_a);
 	}
-	if (dir_in_way(ren2_dst, !o->call_depth)) {
-		dst_name2 = del[delp++] = unique_path(o, ren2_dst, branch2);
+	if (dir_in_way(b->path, !o->call_depth)) {
+		dst_name_b = del[delp++] = unique_path(o, b->path, ci->branch2);
 		output(o, 1, "%s is a directory in %s adding as %s instead",
-		       ren2_dst, branch1, dst_name2);
+		       b->path, ci->branch1, dst_name_b);
 	}
 	if (o->call_depth) {
 		struct merge_file_info mfi;
-		mfi = merge_file(o, src,
-				 pair1->one->sha1, pair1->one->mode,
-				 pair1->two->sha1, pair1->two->mode,
-				 pair2->two->sha1, pair2->two->mode,
-				 branch1, branch2);
+		mfi = merge_file(o, one->path,
+				 one->sha1, one->mode,
+				 a->sha1, a->mode,
+				 b->sha1, b->mode,
+				 ci->branch1, ci->branch2);
 		/*
 		 * FIXME: For rename/add-source conflicts (if we could detect
 		 * such), this is wrong.  We should instead find a unique
 		 * pathname and then either rename the add-source file to that
 		 * unique path, or use that unique path instead of src here.
 		 */
-		update_file(o, 0, mfi.sha, mfi.mode, src);
-		remove_file_from_cache(ren1_dst);
-		remove_file_from_cache(ren2_dst);
+		update_file(o, 0, mfi.sha, mfi.mode, one->path);
+		remove_file_from_cache(a->path);
+		remove_file_from_cache(b->path);
 	} else {
-		update_stages(ren1_dst, NULL, pair1->two, NULL);
-		update_stages(ren2_dst, NULL, NULL, pair2->two);
+		update_stages(a->path, NULL, a, NULL);
+		update_stages(b->path, NULL, NULL, b);
 
-		update_file(o, 0, pair1->two->sha1, pair1->two->mode, dst_name1);
-		update_file(o, 0, pair2->two->sha1, pair2->two->mode, dst_name2);
+		update_file(o, 0, a->sha1, a->mode, dst_name_a);
+		update_file(o, 0, b->sha1, b->mode, dst_name_b);
 	}
 	while (delp--)
 		free(del[delp]);
@@ -1476,10 +1473,7 @@ static int process_entry(struct merge_options *o,
 			break;
 		case RENAME_ONE_FILE_TO_TWO:
 			clean_merge = 0;
-			conflict_rename_rename_1to2(o, conflict_info->pair1,
-						    conflict_info->branch1,
-						    conflict_info->pair2,
-						    conflict_info->branch2);
+			conflict_rename_rename_1to2(o, conflict_info);
 			break;
 		default:
 			entry->processed = 0;
-- 
1.7.6.rc0.62.g2d69f

[PATCH 41/48] merge-recursive: Defer rename/rename(2to1) handling until process_entry

[Elijah Newren]

This puts the code for the different types of double rename conflicts
closer together (fewer lines of other code separating the two paths) and
increases similarity between how they are handled.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |  104 +++++++++++++++++++++++++++++++---------------------
 1 files changed, 62 insertions(+), 42 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 9c0daf5..dbc1dd3 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -67,7 +67,8 @@ enum rename_type {
 	RENAME_NORMAL = 0,
 	RENAME_DELETE,
 	RENAME_ONE_FILE_TO_ONE,
-	RENAME_ONE_FILE_TO_TWO
+	RENAME_ONE_FILE_TO_TWO,
+	RENAME_TWO_FILES_TO_ONE
 };
 
 struct rename_conflict_info {
@@ -1006,32 +1007,40 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 }
 
 static void conflict_rename_rename_2to1(struct merge_options *o,
-					struct rename *ren1,
-					const char *branch1,
-					struct rename *ren2,
-					const char *branch2)
+					struct rename_conflict_info *ci)
 {
-	char *path = ren1->pair->two->path; /* same as ren2->pair->two->path */
-	/* Two files were renamed to the same thing. */
+	/* Two files, a & b, were renamed to the same thing, c. */
+	struct diff_filespec *a = ci->pair1->one;
+	struct diff_filespec *b = ci->pair2->one;
+	struct diff_filespec *c1 = ci->pair1->two;
+	struct diff_filespec *c2 = ci->pair2->two;
+	char *path = c1->path; /* == c2->path */
+
+	output(o, 1, "CONFLICT (rename/rename): "
+	       "Rename %s->%s in %s. "
+	       "Rename %s->%s in %s",
+	       a->path, c1->path, ci->branch1,
+	       b->path, c2->path, ci->branch2);
+
+	remove_file(o, 1, a->path, would_lose_untracked(a->path));
+	remove_file(o, 1, b->path, would_lose_untracked(b->path));
+
 	if (o->call_depth) {
 		struct merge_file_info mfi;
 		mfi = merge_file(o, path, null_sha1, 0,
-				 ren1->pair->two->sha1, ren1->pair->two->mode,
-				 ren2->pair->two->sha1, ren2->pair->two->mode,
-				 branch1, branch2);
+				 c1->sha1, c1->mode,
+				 c2->sha1, c2->mode,
+				 ci->branch1, ci->branch2);
 		output(o, 1, "Adding merged %s", path);
 		update_file(o, 0, mfi.sha, mfi.mode, path);
 	} else {
-		char *new_path1 = unique_path(o, path, branch1);
-		char *new_path2 = unique_path(o, path, branch2);
+		char *new_path1 = unique_path(o, path, ci->branch1);
+		char *new_path2 = unique_path(o, path, ci->branch2);
 		output(o, 1, "Renaming %s to %s and %s to %s instead",
-		       ren1->pair->one->path, new_path1,
-		       ren2->pair->one->path, new_path2);
+		       a->path, new_path1, b->path, new_path2);
 		remove_file(o, 0, path, 0);
-		update_file(o, 0, ren1->pair->two->sha1, ren1->pair->two->mode,
-			    new_path1);
-		update_file(o, 0, ren2->pair->two->sha1, ren2->pair->two->mode,
-			    new_path2);
+		update_file(o, 0, c1->sha1, c1->mode, new_path1);
+		update_file(o, 0, c2->sha1, c2->mode, new_path2);
 		free(new_path2);
 		free(new_path1);
 	}
@@ -1062,6 +1071,7 @@ static int process_renames(struct merge_options *o,
 		struct rename *ren1 = NULL, *ren2 = NULL;
 		const char *branch1, *branch2;
 		const char *ren1_src, *ren1_dst;
+		struct string_list_item *lookup;
 
 		if (i >= a_renames->nr) {
 			ren2 = b_renames->items[j++].util;
@@ -1093,30 +1103,30 @@ static int process_renames(struct merge_options *o,
 			ren1 = tmp;
 		}
 
+		if (ren1->processed)
+			continue;
+		ren1->processed = 1;
 		ren1->dst_entry->processed = 1;
 		/* BUG: We should only mark src_entry as processed if we
 		 * are not dealing with a rename + add-source case.
 		 */
 		ren1->src_entry->processed = 1;
 
-		if (ren1->processed)
-			continue;
-		ren1->processed = 1;
-
 		ren1_src = ren1->pair->one->path;
 		ren1_dst = ren1->pair->two->path;
 
 		if (ren2) {
+			/* One file renamed on both sides */
 			const char *ren2_src = ren2->pair->one->path;
 			const char *ren2_dst = ren2->pair->two->path;
 			enum rename_type rename_type;
-			/* Renamed in 1 and renamed in 2 */
 			if (strcmp(ren1_src, ren2_src) != 0)
-				die("ren1.src != ren2.src");
+				die("ren1_src != ren2_src");
 			ren2->dst_entry->processed = 1;
 			ren2->processed = 1;
 			if (strcmp(ren1_dst, ren2_dst) != 0) {
 				rename_type = RENAME_ONE_FILE_TO_TWO;
+				clean_merge = 0;
 			} else {
 				rename_type = RENAME_ONE_FILE_TO_ONE;
 				/* BUG: We should only remove ren1_src in
@@ -1136,9 +1146,32 @@ static int process_renames(struct merge_options *o,
 						   branch2,
 						   ren1->dst_entry,
 						   ren2->dst_entry);
+		} else if ((lookup = string_list_lookup(renames2Dst, ren1_dst))) {
+			/* Two different files renamed to the same thing */
+			char *ren2_dst;
+			ren2 = lookup->util;
+			ren2_dst = ren2->pair->two->path;
+			if (strcmp(ren1_dst, ren2_dst) != 0)
+				die("ren1_dst != ren2_dst");
+
+			clean_merge = 0;
+			ren2->processed = 1;
+			/*
+			 * BUG: We should only mark src_entry as processed
+			 * if we are not dealing with a rename + add-source
+			 * case.
+			 */
+			ren2->src_entry->processed = 1;
+
+			setup_rename_conflict_info(RENAME_TWO_FILES_TO_ONE,
+						   ren1->pair,
+						   ren2->pair,
+						   branch1,
+						   branch2,
+						   ren1->dst_entry,
+						   ren2->dst_entry);
 		} else {
 			/* Renamed in 1, maybe changed in 2 */
-			struct string_list_item *item;
 			/* we only use sha1 and mode of these */
 			struct diff_filespec src_other, dst_other;
 			int try_merge;
@@ -1173,23 +1206,6 @@ static int process_renames(struct merge_options *o,
 							   branch2,
 							   ren1->dst_entry,
 							   NULL);
-			} else if ((item = string_list_lookup(renames2Dst, ren1_dst))) {
-				char *ren2_src, *ren2_dst;
-				ren2 = item->util;
-				ren2_src = ren2->pair->one->path;
-				ren2_dst = ren2->pair->two->path;
-
-				clean_merge = 0;
-				ren2->processed = 1;
-				remove_file(o, 1, ren2_src,
-					    renamed_stage == 3 || would_lose_untracked(ren1_src));
-
-				output(o, 1, "CONFLICT (rename/rename): "
-				       "Rename %s->%s in %s. "
-				       "Rename %s->%s in %s",
-				       ren1_src, ren1_dst, branch1,
-				       ren2_src, ren2_dst, branch2);
-				conflict_rename_rename_2to1(o, ren1, branch1, ren2, branch2);
 			} else if ((dst_other.mode == ren1->pair->two->mode) &&
 				   sha_eq(dst_other.sha1, ren1->pair->two->sha1)) {
 				/* Added file on the other side
@@ -1475,6 +1491,10 @@ static int process_entry(struct merge_options *o,
 			clean_merge = 0;
 			conflict_rename_rename_1to2(o, conflict_info);
 			break;
+		case RENAME_TWO_FILES_TO_ONE:
+			clean_merge = 0;
+			conflict_rename_rename_2to1(o, conflict_info);
+			break;
 		default:
 			entry->processed = 0;
 			break;
-- 
1.7.6.rc0.62.g2d69f

[PATCH 42/48] merge-recursive: Record more data needed for merging with dual renames

[Elijah Newren]

When two different files are renamed to one, we need to be able to do
three-way merges for both of those files.  To do that, we need to record
the sha1sum of the (possibly modified) file on the unrenamed side.  Modify
setup_rename_conflict_info() to take this extra information and record it
when the rename_type is RENAME_TWO_FILES_TO_ONE.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   42 +++++++++++++++++++++++++++++++++++++++---
 1 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index dbc1dd3..fdf67a3 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -79,6 +79,8 @@ struct rename_conflict_info {
 	const char *branch2;
 	struct stage_data *dst_entry1;
 	struct stage_data *dst_entry2;
+	struct diff_filespec ren1_other;
+	struct diff_filespec ren2_other;
 };
 
 /*
@@ -100,7 +102,10 @@ static inline void setup_rename_conflict_info(enum rename_type rename_type,
 					      const char *branch1,
 					      const char *branch2,
 					      struct stage_data *dst_entry1,
-					      struct stage_data *dst_entry2)
+					      struct stage_data *dst_entry2,
+					      struct merge_options *o,
+					      struct stage_data *src_entry1,
+					      struct stage_data *src_entry2)
 {
 	struct rename_conflict_info *ci = xcalloc(1, sizeof(struct rename_conflict_info));
 	ci->rename_type = rename_type;
@@ -118,6 +123,24 @@ static inline void setup_rename_conflict_info(enum rename_type rename_type,
 		ci->pair2 = pair2;
 		dst_entry2->rename_conflict_info = ci;
 	}
+
+	if (rename_type == RENAME_TWO_FILES_TO_ONE) {
+		/*
+		 * For each rename, there could have been
+		 * modifications on the side of history where that
+		 * file was not renamed.
+		 */
+		int ostage1 = o->branch1 == branch1 ? 3 : 2;
+		int ostage2 = ostage1 ^ 1;
+
+		ci->ren1_other.path = pair1->one->path;
+		hashcpy(ci->ren1_other.sha1, src_entry1->stages[ostage1].sha);
+		ci->ren1_other.mode = src_entry1->stages[ostage1].mode;
+
+		ci->ren2_other.path = pair2->one->path;
+		hashcpy(ci->ren2_other.sha1, src_entry2->stages[ostage2].sha);
+		ci->ren2_other.mode = src_entry2->stages[ostage2].mode;
+	}
 }
 
 static int show(struct merge_options *o, int v)
@@ -1145,7 +1168,10 @@ static int process_renames(struct merge_options *o,
 						   branch1,
 						   branch2,
 						   ren1->dst_entry,
-						   ren2->dst_entry);
+						   ren2->dst_entry,
+						   o,
+						   NULL,
+						   NULL);
 		} else if ((lookup = string_list_lookup(renames2Dst, ren1_dst))) {
 			/* Two different files renamed to the same thing */
 			char *ren2_dst;
@@ -1169,7 +1195,11 @@ static int process_renames(struct merge_options *o,
 						   branch1,
 						   branch2,
 						   ren1->dst_entry,
-						   ren2->dst_entry);
+						   ren2->dst_entry,
+						   o,
+						   ren1->src_entry,
+						   ren2->src_entry);
+
 		} else {
 			/* Renamed in 1, maybe changed in 2 */
 			/* we only use sha1 and mode of these */
@@ -1205,6 +1235,9 @@ static int process_renames(struct merge_options *o,
 							   branch1,
 							   branch2,
 							   ren1->dst_entry,
+							   NULL,
+							   o,
+							   NULL,
 							   NULL);
 			} else if ((dst_other.mode == ren1->pair->two->mode) &&
 				   sha_eq(dst_other.sha1, ren1->pair->two->sha1)) {
@@ -1256,6 +1289,9 @@ static int process_renames(struct merge_options *o,
 							   branch1,
 							   NULL,
 							   ren1->dst_entry,
+							   NULL,
+							   o,
+							   NULL,
 							   NULL);
 			}
 		}
-- 
1.7.6.rc0.62.g2d69f

[PATCH 43/48] merge-recursive: Create function for merging with branchname:file markers

[Elijah Newren]

We want to be able to reuse the code to do a three-way file content merge
and have the conflict markers use both branchname and filename.  Split it
out into a separate function.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   51 ++++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 38 insertions(+), 13 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index fdf67a3..b8fd686 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -929,6 +929,36 @@ static struct merge_file_info merge_file_1(struct merge_options *o,
 	return result;
 }
 
+static struct merge_file_info
+merge_file_special_markers(struct merge_options *o,
+			   const struct diff_filespec *one,
+			   const struct diff_filespec *a,
+			   const struct diff_filespec *b,
+			   const char *branch1,
+			   const char *filename1,
+			   const char *branch2,
+			   const char *filename2)
+{
+	char *side1 = NULL;
+	char *side2 = NULL;
+	struct merge_file_info mfi;
+
+	if (filename1) {
+		side1 = xmalloc(strlen(branch1) + strlen(filename1) + 2);
+		sprintf(side1, "%s:%s", branch1, filename1);
+	}
+	if (filename2) {
+		side2 = xmalloc(strlen(branch2) + strlen(filename2) + 2);
+		sprintf(side2, "%s:%s", branch2, filename2);
+	}
+
+	mfi = merge_file_1(o, one, a, b,
+			   side1 ? side1 : branch1, side2 ? side2 : branch2);
+	free(side1);
+	free(side2);
+	return mfi;
+}
+
 static struct merge_file_info merge_file(struct merge_options *o,
 					 const char *path,
 					 const unsigned char *o_sha, int o_mode,
@@ -1405,7 +1435,7 @@ static int merge_content(struct merge_options *o,
 			 struct rename_conflict_info *rename_conflict_info)
 {
 	const char *reason = "content";
-	char *side1 = NULL, *side2 = NULL;
+	const char *path1 = NULL, *path2 = NULL;
 	struct merge_file_info mfi;
 	struct diff_filespec one, a, b;
 	unsigned df_conflict_remains = 0;
@@ -1423,22 +1453,17 @@ static int merge_content(struct merge_options *o,
 	b.mode = b_mode;
 
 	if (rename_conflict_info) {
-		const char *path1 = rename_conflict_info->pair2 ?
-				    rename_conflict_info->pair2->one->path : path;
-		const char *path2 = rename_conflict_info->pair1 ?
-				    rename_conflict_info->pair1->one->path : path;
-		side1 = xmalloc(strlen(o->branch1) + strlen(path1) + 2);
-		side2 = xmalloc(strlen(o->branch2) + strlen(path2) + 2);
-		sprintf(side1, "%s:%s", o->branch1, path1);
-		sprintf(side2, "%s:%s", o->branch2, path2);
+		path1 = rename_conflict_info->pair2 ?
+			rename_conflict_info->pair2->one->path : path;
+		path2 = rename_conflict_info->pair1 ?
+			rename_conflict_info->pair1->one->path : path;
 
 		if (dir_in_way(path, !o->call_depth))
 			df_conflict_remains = 1;
 	}
-	mfi = merge_file_1(o, &one, &a, &b,
-			   side1 ? side1:o->branch1, side2 ? side2:o->branch2);
-	free(side1);
-	free(side2);
+	mfi = merge_file_special_markers(o, &one, &a, &b,
+					 o->branch1, path1,
+					 o->branch2, path2);
 
 	if (mfi.clean && !df_conflict_remains &&
 	    sha_eq(mfi.sha, a_sha) && mfi.mode == a_mode &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 44/48] merge-recursive: Consider modifications in rename/rename(2to1) conflicts

[Elijah Newren]

Our previous conflict resolution for renaming two different files to the
same name ignored the fact that each of those files may have modifications
from both sides of history to consider.  We need to do a three-way merge
for each of those files, and then handle the conflict of both sets of
merged contents trying to be recorded with the same name.

It is important to note that this changes our strategy in the recursive
case.  After doing a three-way content merge of each of the files
involved, we still are faced with the fact that we are trying to put both
of the results (including conflict markers) into the same path.  We could
do another two-way merge, but I think that becomes confusing.  Also,
taking a hint from the modify/delete and rename/delete cases we handled
earlier, a more useful "common ground" would be to keep the three-way
content merge but record it with the original filename.  The renames can
still be detected, we just allow it to be done in the o->call_depth=0
case.  This seems to result in simpler & easier to understand merge
conflicts as well, as evidenced by some of the changes needed in our
testsuite in t6036.  (However, it should be noted that this change will
cause problems those renames also occur along with a file being added
whose name matches the source of the rename.  Since git currently cannot
detect rename/add-source situations, though, this codepath is not
currently used for those cases anyway.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |   30 ++++++++++++++++++--------
 t/t6036-recursive-corner-cases.sh    |   38 +++++++++-------------------------
 t/t6039-merge-rename-corner-cases.sh |    2 +-
 3 files changed, 32 insertions(+), 38 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index b8fd686..ea538e9 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1068,6 +1068,8 @@ static void conflict_rename_rename_2to1(struct merge_options *o,
 	struct diff_filespec *c1 = ci->pair1->two;
 	struct diff_filespec *c2 = ci->pair2->two;
 	char *path = c1->path; /* == c2->path */
+	struct merge_file_info mfi_c1;
+	struct merge_file_info mfi_c2;
 
 	output(o, 1, "CONFLICT (rename/rename): "
 	       "Rename %s->%s in %s. "
@@ -1078,22 +1080,32 @@ static void conflict_rename_rename_2to1(struct merge_options *o,
 	remove_file(o, 1, a->path, would_lose_untracked(a->path));
 	remove_file(o, 1, b->path, would_lose_untracked(b->path));
 
+	mfi_c1 = merge_file_special_markers(o, a, c1, &ci->ren1_other,
+					    o->branch1, c1->path,
+					    o->branch2, ci->ren1_other.path);
+	mfi_c2 = merge_file_special_markers(o, b, &ci->ren2_other, c2,
+					    o->branch1, ci->ren2_other.path,
+					    o->branch2, c2->path);
+
 	if (o->call_depth) {
-		struct merge_file_info mfi;
-		mfi = merge_file(o, path, null_sha1, 0,
-				 c1->sha1, c1->mode,
-				 c2->sha1, c2->mode,
-				 ci->branch1, ci->branch2);
-		output(o, 1, "Adding merged %s", path);
-		update_file(o, 0, mfi.sha, mfi.mode, path);
+		/*
+		 * If mfi_c1.clean && mfi_c2.clean, then it might make
+		 * sense to do a two-way merge of those results.  But, I
+		 * think in all cases, it makes sense to have the virtual
+		 * merge base just undo the renames; they can be detected
+		 * again later for the non-recursive merge.
+		 */
+		remove_file(o, 0, path, 0);
+		update_file(o, 0, mfi_c1.sha, mfi_c1.mode, a->path);
+		update_file(o, 0, mfi_c2.sha, mfi_c2.mode, b->path);
 	} else {
 		char *new_path1 = unique_path(o, path, ci->branch1);
 		char *new_path2 = unique_path(o, path, ci->branch2);
 		output(o, 1, "Renaming %s to %s and %s to %s instead",
 		       a->path, new_path1, b->path, new_path2);
 		remove_file(o, 0, path, 0);
-		update_file(o, 0, c1->sha1, c1->mode, new_path1);
-		update_file(o, 0, c2->sha1, c2->mode, new_path2);
+		update_file(o, 0, mfi_c1.sha, mfi_c1.mode, new_path1);
+		update_file(o, 0, mfi_c2.sha, mfi_c2.mode, new_path2);
 		free(new_path2);
 		free(new_path1);
 	}
diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
index a1b609f..5c3868a 100755
--- a/t/t6036-recursive-corner-cases.sh
+++ b/t/t6036-recursive-corner-cases.sh
@@ -51,23 +51,15 @@ test_expect_success 'merge simple rename+criss-cross with no modifications' '
 
 	test_must_fail git merge -s recursive R2^0 &&
 
-	test 5 = $(git ls-files -s | wc -l) &&
-	test 3 = $(git ls-files -u | wc -l) &&
-	test 0 = $(git ls-files -o | wc -l) &&
+	test 2 = $(git ls-files -s | wc -l) &&
+	test 2 = $(git ls-files -u | wc -l) &&
+	test 2 = $(git ls-files -o | wc -l) &&
 
-	test $(git rev-parse :0:one) = $(git rev-parse L2:one) &&
-	test $(git rev-parse :0:two) = $(git rev-parse R2:two) &&
 	test $(git rev-parse :2:three) = $(git rev-parse L2:three) &&
 	test $(git rev-parse :3:three) = $(git rev-parse R2:three) &&
 
-	cp one merged &&
-	>empty &&
-	test_must_fail git merge-file \
-		-L "Temporary merge branch 1" \
-		-L "" \
-		-L "Temporary merge branch 2" \
-		merged empty two &&
-	test $(git rev-parse :1:three) = $(git hash-object merged)
+	test $(git rev-parse L2:three) = $(git hash-object three~HEAD) &&
+	test $(git rev-parse R2:three) = $(git hash-object three~R2^0)
 '
 
 #
@@ -126,25 +118,15 @@ test_expect_success 'merge criss-cross + rename merges with basic modification'
 
 	test_must_fail git merge -s recursive R2^0 &&
 
-	test 5 = $(git ls-files -s | wc -l) &&
-	test 3 = $(git ls-files -u | wc -l) &&
-	test 0 = $(git ls-files -o | wc -l) &&
+	test 2 = $(git ls-files -s | wc -l) &&
+	test 2 = $(git ls-files -u | wc -l) &&
+	test 2 = $(git ls-files -o | wc -l) &&
 
-	test $(git rev-parse :0:one) = $(git rev-parse L2:one) &&
-	test $(git rev-parse :0:two) = $(git rev-parse R2:two) &&
 	test $(git rev-parse :2:three) = $(git rev-parse L2:three) &&
 	test $(git rev-parse :3:three) = $(git rev-parse R2:three) &&
 
-	head -n 10 two >merged &&
-	cp one merge-me &&
-	>empty &&
-	test_must_fail git merge-file \
-		-L "Temporary merge branch 1" \
-		-L "" \
-		-L "Temporary merge branch 2" \
-		merge-me empty merged &&
-
-	test $(git rev-parse :1:three) = $(git hash-object merge-me)
+	test $(git rev-parse L2:three) = $(git hash-object three~HEAD) &&
+	test $(git rev-parse R2:three) = $(git hash-object three~R2^0)
 '
 
 #
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 4d2fd10..86e96d0 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -524,7 +524,7 @@ test_expect_success 'setup rename/rename (2to1) + modify/modify' '
 	git commit -m C
 '
 
-test_expect_failure 'handle rename/rename (2to1) conflict correctly' '
+test_expect_success 'handle rename/rename (2to1) conflict correctly' '
 	git checkout B^0 &&
 
 	test_must_fail git merge -s recursive C^0 >out &&
-- 
1.7.6.rc0.62.g2d69f

[PATCH 45/48] merge-recursive: Make modify/delete handling code reusable

[Elijah Newren]

modify/delete and rename/delete share a lot of similarities; we'd like all
the criss-cross and D/F conflict handling specializations to be shared
between the two.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c       |   89 +++++++++++++++++++++++++++-------------------
 t/t6022-merge-rename.sh |    4 +-
 2 files changed, 54 insertions(+), 39 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index ea538e9..e38c5b0 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -979,6 +979,50 @@ static struct merge_file_info merge_file(struct merge_options *o,
 	return merge_file_1(o, &one, &a, &b, branch1, branch2);
 }
 
+static void handle_change_delete(struct merge_options *o,
+				 const char *path,
+				 const unsigned char *o_sha, int o_mode,
+				 const unsigned char *a_sha, int a_mode,
+				 const unsigned char *b_sha, int b_mode,
+				 const char *change, const char *change_past)
+{
+	char *new_path = NULL;
+	int free_me = 0;
+	if (dir_in_way(path, !o->call_depth)) {
+		new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
+		free_me = 1;
+	}
+
+	if (o->call_depth) {
+		/*
+		 * We cannot arbitrarily accept either a_sha or b_sha as
+		 * correct; since there is no true "middle point" between
+		 * them, simply reuse the base version for virtual merge base.
+		 */
+		remove_file_from_cache(path);
+		update_file(o, 0, o_sha, o_mode, new_path ? new_path : path);
+	} else if (!a_sha) {
+		output(o, 1, "CONFLICT (%s/delete): %s deleted in %s "
+		       "and %s in %s. Version %s of %s left in tree%s%s.",
+		       change, path, o->branch1,
+		       change_past, o->branch2, o->branch2, path,
+		       NULL == new_path ? "" : " at ",
+		       NULL == new_path ? "" : new_path);
+		update_file(o, 0, b_sha, b_mode, new_path ? new_path : path);
+	} else {
+		output(o, 1, "CONFLICT (%s/delete): %s deleted in %s "
+		       "and %s in %s. Version %s of %s left in tree%s%s.",
+		       change, path, o->branch2,
+		       change_past, o->branch1, o->branch1, path,
+		       NULL == new_path ? "" : " at ",
+		       NULL == new_path ? "" : new_path);
+		update_file(o, 0, a_sha, a_mode, new_path ? new_path : path);
+	}
+	if (free_me)
+		free(new_path);
+
+}
+
 static void conflict_rename_delete(struct merge_options *o,
 				   struct diff_filepair *pair,
 				   const char *rename_branch,
@@ -1396,47 +1440,18 @@ error_return:
 	return ret;
 }
 
-static void handle_delete_modify(struct merge_options *o,
+static void handle_modify_delete(struct merge_options *o,
 				 const char *path,
 				 unsigned char *o_sha, int o_mode,
 				 unsigned char *a_sha, int a_mode,
 				 unsigned char *b_sha, int b_mode)
 {
-	char *new_path = NULL;
-	int free_me = 0;
-	if (dir_in_way(path, !o->call_depth)) {
-		new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
-		free_me = 1;
-	}
-
-	if (o->call_depth) {
-		/*
-		 * We cannot arbitrarily accept either a_sha or b_sha as
-		 * correct; since there is no true "middle point" between
-		 * them, simply reuse the base version for virtual merge base.
-		 */
-		remove_file_from_cache(path);
-		update_file(o, 0, o_sha, o_mode, new_path ? new_path : path);
-	} else if (!a_sha) {
-		output(o, 1, "CONFLICT (delete/modify): %s deleted in %s "
-		       "and modified in %s. Version %s of %s left in tree%s%s.",
-		       path, o->branch1,
-		       o->branch2, o->branch2, path,
-		       NULL == new_path ? "" : " at ",
-		       NULL == new_path ? "" : new_path);
-		update_file(o, 0, b_sha, b_mode, new_path ? new_path : path);
-	} else {
-		output(o, 1, "CONFLICT (delete/modify): %s deleted in %s "
-		       "and modified in %s. Version %s of %s left in tree%s%s.",
-		       path, o->branch2,
-		       o->branch1, o->branch1, path,
-		       NULL == new_path ? "" : " at ",
-		       NULL == new_path ? "" : new_path);
-		update_file(o, 0, a_sha, a_mode, new_path ? new_path : path);
-	}
-	if (free_me)
-		free(new_path);
-
+	handle_change_delete(o,
+			     path,
+			     o_sha, o_mode,
+			     a_sha, a_mode,
+			     b_sha, b_mode,
+			     "modify", "modified");
 }
 
 static int merge_content(struct merge_options *o,
@@ -1586,7 +1601,7 @@ static int process_entry(struct merge_options *o,
 		} else {
 			/* Modify/delete; deleted side may have put a directory in the way */
 			clean_merge = 0;
-			handle_delete_modify(o, path, o_sha, o_mode,
+			handle_modify_delete(o, path, o_sha, o_mode,
 					     a_sha, a_mode, b_sha, b_mode);
 		}
 	} else if ((!o_sha && a_sha && !b_sha) ||
diff --git a/t/t6022-merge-rename.sh b/t/t6022-merge-rename.sh
index cfce3d3..a1ed00b 100755
--- a/t/t6022-merge-rename.sh
+++ b/t/t6022-merge-rename.sh
@@ -303,7 +303,7 @@ test_expect_success 'Rename+D/F conflict; renamed file merges but dir in way' '
 	git checkout -q renamed-file-has-no-conflicts^0 &&
 	test_must_fail git merge --strategy=recursive dir-in-way >output &&
 
-	grep "CONFLICT (delete/modify): dir/file-in-the-way" output &&
+	grep "CONFLICT (modify/delete): dir/file-in-the-way" output &&
 	grep "Auto-merging dir" output &&
 	grep "Adding as dir~HEAD instead" output &&
 
@@ -325,7 +325,7 @@ test_expect_success 'Same as previous, but merged other way' '
 	test_must_fail git merge --strategy=recursive renamed-file-has-no-conflicts >output 2>errors &&
 
 	! grep "error: refusing to lose untracked file at" errors &&
-	grep "CONFLICT (delete/modify): dir/file-in-the-way" output &&
+	grep "CONFLICT (modify/delete): dir/file-in-the-way" output &&
 	grep "Auto-merging dir" output &&
 	grep "Adding as dir~renamed-file-has-no-conflicts instead" output &&
 
-- 
1.7.6.rc0.62.g2d69f

[PATCH 46/48] merge-recursive: Have conflict_rename_delete reuse modify/delete code

[Elijah Newren]

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c |   48 +++++++++++++++++++++++++++++++-----------------
 1 files changed, 31 insertions(+), 17 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index e38c5b0..93b7da5 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1028,24 +1028,38 @@ static void conflict_rename_delete(struct merge_options *o,
 				   const char *rename_branch,
 				   const char *other_branch)
 {
-	char *dest_name = pair->two->path;
-	int df_conflict = 0;
+	const struct diff_filespec *orig = pair->one;
+	const struct diff_filespec *dest = pair->two;
+	const char *path;
+	const unsigned char *a_sha = NULL;
+	const unsigned char *b_sha = NULL;
+	int a_mode = 0;
+	int b_mode = 0;
+
+	if (rename_branch == o->branch1) {
+		a_sha = dest->sha1;
+		a_mode = dest->mode;
+	} else {
+		b_sha = dest->sha1;
+		b_mode = dest->mode;
+	}
 
-	output(o, 1, "CONFLICT (rename/delete): Rename %s->%s in %s "
-	       "and deleted in %s",
-	       pair->one->path, pair->two->path, rename_branch,
-	       other_branch);
-	if (!o->call_depth)
-		update_stages(dest_name, NULL,
-			      rename_branch == o->branch1 ? pair->two : NULL,
-			      rename_branch == o->branch1 ? NULL : pair->two);
-	if (dir_in_way(dest_name, !o->call_depth)) {
-		dest_name = unique_path(o, dest_name, rename_branch);
-		df_conflict = 1;
+	if (o->call_depth) {
+		remove_file_from_cache(dest->path);
+		path = orig->path;
+	} else {
+		path = dest->path;
+		update_stages(dest->path, NULL,
+			      rename_branch == o->branch1 ? dest : NULL,
+			      rename_branch == o->branch1 ? NULL : dest);
 	}
-	update_file(o, 0, pair->two->sha1, pair->two->mode, dest_name);
-	if (df_conflict)
-		free(dest_name);
+
+	handle_change_delete(o,
+			     path,
+			     orig->sha1, orig->mode,
+			     a_sha, a_mode,
+			     b_sha, b_mode,
+			     "rename", "renamed");
 }
 
 static void conflict_rename_rename_1to2(struct merge_options *o,
-- 
1.7.6.rc0.62.g2d69f

[PATCH 47/48] merge-recursive: add handling for rename/rename/add-dest/add-dest

[Elijah Newren]

Each side of the rename in rename/rename(1to2) could potentially also be
involved in a rename/add conflict.  Ensure stages for such conflicts are
also recorded.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |   21 +++++++++++++++++++--
 t/t6039-merge-rename-corner-cases.sh |    2 +-
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 93b7da5..0cfc215 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1062,6 +1062,19 @@ static void conflict_rename_delete(struct merge_options *o,
 			     "rename", "renamed");
 }
 
+static struct diff_filespec* filespec_from_entry(struct diff_filespec *target,
+						 struct stage_data *entry,
+						 int stage)
+{
+	unsigned char *sha = entry->stages[stage].sha;
+	unsigned mode = entry->stages[stage].mode;
+	if (mode == 0 || is_null_sha1(sha))
+		return NULL;
+	hashcpy(target->sha1, sha);
+	target->mode = mode;
+	return target;
+}
+
 static void conflict_rename_rename_1to2(struct merge_options *o,
 					struct rename_conflict_info *ci)
 {
@@ -1107,8 +1120,12 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 		remove_file_from_cache(a->path);
 		remove_file_from_cache(b->path);
 	} else {
-		update_stages(a->path, NULL, a, NULL);
-		update_stages(b->path, NULL, NULL, b);
+		struct diff_filespec other;
+		update_stages(a->path, NULL,
+			      a, filespec_from_entry(&other, ci->dst_entry1, 3));
+
+		update_stages(b->path, NULL,
+			      filespec_from_entry(&other, ci->dst_entry2, 2), b);
 
 		update_file(o, 0, a->sha1, a->mode, dst_name_a);
 		update_file(o, 0, b->sha1, b->mode, dst_name_b);
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index 86e96d0..d2d93e7 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -437,7 +437,7 @@ test_expect_success 'setup rename/rename(1to2)/add-dest conflict' '
 	git commit -m two
 '
 
-test_expect_failure 'rename/rename/add-dest merge still knows about conflicting file versions' '
+test_expect_success 'rename/rename/add-dest merge still knows about conflicting file versions' '
 	git checkout C^0 &&
 	test_must_fail git merge -s recursive B^0 &&
 
-- 
1.7.6.rc0.62.g2d69f

[PATCH 48/48] merge-recursive: Fix working copy handling for rename/rename/add/add

[Elijah Newren]

If either side of a rename/rename(1to2) conflict is itself also involved
in a rename/add-dest conflict, then we need to make sure both the rename
and the added file appear in the working copy.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 merge-recursive.c                    |   73 ++++++++++++++++++++++------------
 t/t6039-merge-rename-corner-cases.sh |   11 +++++-
 2 files changed, 58 insertions(+), 26 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 0cfc215..612a8cb 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1075,6 +1075,52 @@ static struct diff_filespec* filespec_from_entry(struct diff_filespec *target,
 	return target;
 }
 
+static void handle_file(struct merge_options *o,
+			struct diff_filespec *rename,
+			int stage,
+			struct rename_conflict_info *ci)
+{
+	char *dst_name = rename->path;
+	struct stage_data *dst_entry;
+	const char *cur_branch, *other_branch;
+	struct diff_filespec other;
+	struct diff_filespec *add;
+
+	if (stage == 2) {
+		dst_entry = ci->dst_entry1;
+		cur_branch = ci->branch1;
+		other_branch = ci->branch2;
+	} else {
+		dst_entry = ci->dst_entry2;
+		cur_branch = ci->branch2;
+		other_branch = ci->branch1;
+	}
+
+	add = filespec_from_entry(&other, dst_entry, stage ^ 1);
+	if (stage == 2)
+		update_stages(rename->path, NULL, rename, add);
+	else
+		update_stages(rename->path, NULL, add, rename);
+
+	if (add) {
+		char *add_name = unique_path(o, rename->path, other_branch);
+		update_file(o, 0, add->sha1, add->mode, add_name);
+
+		remove_file(o, 0, rename->path, 0);
+		dst_name = unique_path(o, rename->path, cur_branch);
+	} else {
+		if (dir_in_way(rename->path, !o->call_depth)) {
+			dst_name = unique_path(o, rename->path, cur_branch);
+			output(o, 1, "%s is a directory in %s adding as %s instead",
+			       rename->path, other_branch, dst_name);
+		}
+	}
+	update_file(o, 0, rename->sha1, rename->mode, dst_name);
+
+	if (dst_name != rename->path)
+		free(dst_name);
+}
+
 static void conflict_rename_rename_1to2(struct merge_options *o,
 					struct rename_conflict_info *ci)
 {
@@ -1082,10 +1128,6 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 	struct diff_filespec *one = ci->pair1->one;
 	struct diff_filespec *a = ci->pair1->two;
 	struct diff_filespec *b = ci->pair2->two;
-	const char *dst_name_a = a->path;
-	const char *dst_name_b = b->path;
-	char *del[2];
-	int delp = 0;
 
 	output(o, 1, "CONFLICT (rename/rename): "
 	       "Rename \"%s\"->\"%s\" in branch \"%s\" "
@@ -1093,16 +1135,6 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 	       one->path, a->path, ci->branch1,
 	       one->path, b->path, ci->branch2,
 	       o->call_depth ? " (left unresolved)" : "");
-	if (dir_in_way(a->path, !o->call_depth)) {
-		dst_name_a = del[delp++] = unique_path(o, a->path, ci->branch1);
-		output(o, 1, "%s is a directory in %s adding as %s instead",
-		       a->path, ci->branch2, dst_name_a);
-	}
-	if (dir_in_way(b->path, !o->call_depth)) {
-		dst_name_b = del[delp++] = unique_path(o, b->path, ci->branch2);
-		output(o, 1, "%s is a directory in %s adding as %s instead",
-		       b->path, ci->branch1, dst_name_b);
-	}
 	if (o->call_depth) {
 		struct merge_file_info mfi;
 		mfi = merge_file(o, one->path,
@@ -1120,18 +1152,9 @@ static void conflict_rename_rename_1to2(struct merge_options *o,
 		remove_file_from_cache(a->path);
 		remove_file_from_cache(b->path);
 	} else {
-		struct diff_filespec other;
-		update_stages(a->path, NULL,
-			      a, filespec_from_entry(&other, ci->dst_entry1, 3));
-
-		update_stages(b->path, NULL,
-			      filespec_from_entry(&other, ci->dst_entry2, 2), b);
-
-		update_file(o, 0, a->sha1, a->mode, dst_name_a);
-		update_file(o, 0, b->sha1, b->mode, dst_name_b);
+		handle_file(o, a, 2, ci);
+		handle_file(o, b, 3, ci);
 	}
-	while (delp--)
-		free(del[delp]);
 }
 
 static void conflict_rename_rename_2to1(struct merge_options *o,
diff --git a/t/t6039-merge-rename-corner-cases.sh b/t/t6039-merge-rename-corner-cases.sh
index d2d93e7..9db1663 100755
--- a/t/t6039-merge-rename-corner-cases.sh
+++ b/t/t6039-merge-rename-corner-cases.sh
@@ -444,11 +444,20 @@ test_expect_success 'rename/rename/add-dest merge still knows about conflicting
 	test 5 -eq $(git ls-files -s | wc -l) &&
 	test 2 -eq $(git ls-files -u b | wc -l) &&
 	test 2 -eq $(git ls-files -u c | wc -l) &&
+	test 4 -eq $(git ls-files -o | wc -l) &&
 
 	test $(git rev-parse :2:b) = $(git rev-parse C:b) &&
 	test $(git rev-parse :3:b) = $(git rev-parse B:b) &&
 	test $(git rev-parse :2:c) = $(git rev-parse C:c) &&
-	test $(git rev-parse :3:c) = $(git rev-parse B:c)
+	test $(git rev-parse :3:c) = $(git rev-parse B:c) &&
+
+	test $(git hash-object c~HEAD) = $(git rev-parse C:c) &&
+	test $(git hash-object c~B\^0) = $(git rev-parse B:c) &&
+	test $(git hash-object b~HEAD) = $(git rev-parse C:b) &&
+	test $(git hash-object b~B\^0) = $(git rev-parse B:b) &&
+
+	test ! -f b &&
+	test ! -f c
 '
 
 test_expect_success 'setup simple rename/rename (1to2) conflict' '
-- 
1.7.6.rc0.62.g2d69f

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> You can note that I did not fix all the testcases I added.  I consider
> some cases either unfixable or not worth fixing.

Thanks for working on these.  A quick comment (after reading a handful of
tests in the early part of the series) I have on rename cases is that the
test might be aiming too high to be too strict in "undetected rename"
cases. Consider:

 - We start from commit A that has path file1 where commit B and C fork
   from;
 - One side, commit B, renames file1 to file2 and edits file2;
 - The other side, commit C, modifies file1.

When merging B and C, we will ideally want to see file1 removed and
content level merge done between C's edit to file1 and B's edit to file2
left in file2. The content level merge may or may not conflict, and if
it cleanly merges, then that is Ok.

But if B's edit is too extensive, we may see what B did as "delete file1,
add file2", and we may report conflict at file1 (delete/modify) while
resolving file2 cleanly (one side added). When notifying the user with
delete/modify conflict at file1, we should make him aware that among the
paths added by the merge (i.e. file2) there might be a corresponding
rewrite of it on the other side, so that such a case can be manually
merged. As long as that is done, a test of such a case should consider
both clean merge (rename noticed) and a conflict at the tree structure
level (delete/modify with new) valid expected result, I think.

To put it another way, merge-recursive should expect that one side of
history may rewrite a path it renames beyond recognition, and when
punting, loudly ask the user for help, with enough information to help the
user to help it (e.g. perhaps say "file1 has delete/modify conflict; the
deleted side has added file2---you the user might want to inspect it and
see if it is a rename with extensive rewrite and resolve it accordingly if
that is the case").  We might want to dig deeper at that point by checking
the similarity again between the two blobs ourselves at that stage before
punting.

If we change what C did to "removes file1", instead of "modifies file1",
the story changes.

When merging B and C, that would be rename/delete conflict (B renamed, C
deleted). However, if we see that B's history as "delete file1 and add
file2", it would merge cleanly with C that deletes file1. The end result
would be an addition of file2. This tastes bad, as it will end up with a
clean merge that the user may not even notice.

Ideally we would want the merge to somehow warn the user to see if the
added "file2" still is wanted, and the worst part of this problem is that
this cannot be mechanically inferred. We could warn against every merge
that has a history that adds new paths and removes some other paths, but
then the warning becomes meaningless.

The changes done by B to the other parts of the tree (that wasn't involved
in the conflict) still want the updated content in file2 to work
correctly, and the changes done by C will not want the original content in
file1 (now in file2). The latter may or may not care about the presense of
what was added to "file2" by B. If C's work was to remove file1 that was
nothing but dead code (and other parts of C's work removed all the
callsite of code in original "file1" to make them dead), while B's work
was to add a code with purpose similar to the original code "file1" had
(and other parts of B's work calls that new code), then the right merge
result of "file2" might be to keep the line removal of dead code made by C
while keeping the code added by B there. So (I am thinking aloud here
without thinking things through) a possible approach to this issue might
involve considering a removal of a path and modifying the contents of a
path to make it empty similarly.

> However, there is one large class of problems that I think is fixable,
> I'm just not sure whether it is worth fixing.  git's rename detection
> optimization of only considering files that exist on one side of the
> diff but not the other causes issues with merges (undetected conflicts,
> spurious conflicts, or merged cleanly but wrongly due to deleting files
> that should be present).  To fix these cases, we would need some way of
> including rewritten files as potential rename candidates,...

This may be part of the larger "when punting, loudly ask the user for
help, or we may want to go deeper". If we can come up with a way to avoid
"undetected conflicts" ("merged cleanly but wrongly" is saying the same
thing as "undetected conflicts"), maybe we can attempt a cheap way first,
and then only when we know there is something complicated going on, we can
redo the diff with -B/-M options on. Spurious conflicts can be handled the
same way, as we should be able to come up with a clean merge once we dig
deeper (that's the definition of "spurious", right?).

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> It does sound potentially expensive, though, and might mean a lot
> more work in merge-recursive to handle that extra information.  Is that a
> path we want to take at some point?

Probably you can start with backend specific option (e.g. -Xbreak=yes) to
experiment. We made recursive the default not because it deals with
renames (in a broken way) but primarily because it handles criss-cross
better; at some point we might also want to add another backend specific
option (e.g. -Xrename=off) to allow the users to keep the "recursive"
aspect of the strategy while declining a more expensive (and brittle)
rename handling to take effect.

My gut feeling is that -Xbreak=yes, once the code does work well enough,
would have to become the default. It would make the default mode of merge
possibly quite expensive but it is Ok as long as we give projects with
simple/clean history an easy way to use either "recursive -Xrename=off" or
even "resolve" to avoid cost that is unnecessary to handle their needs.

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Elijah Newren <newren@gmail.com> writes:
>
>> It does sound potentially expensive, though, and might mean a lot
>> more work in merge-recursive to handle that extra information.  Is that a
>> path we want to take at some point?
>
> Probably you can start with backend specific option (e.g. -Xbreak=yes) to
> experiment. We made recursive the default not because it deals with
> renames (in a broken way) but primarily because it handles criss-cross
> better; at some point we might also want to add another backend specific
> option (e.g. -Xrename=off) to allow the users to keep the "recursive"
> aspect of the strategy while declining a more expensive (and brittle)
> rename handling to take effect.
>
> My gut feeling is that -Xbreak=yes, once the code does work well enough,
> would have to become the default. It would make the default mode of merge
> possibly quite expensive but it is Ok as long as we give projects with
> simple/clean history an easy way to use either "recursive -Xrename=off" or
> even "resolve" to avoid cost that is unnecessary to handle their needs.

I forgot to mention that there is jk/maint-merge-rename-create effort by
Peff already queued in 'pu' to use "break", which of course has
unfortunate conflicts with your series.

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Johannes Sixt]

Am 6/8/2011 9:30, schrieb Elijah Newren:
>  test_expect_success 'modify/delete + directory/file conflict; other way' '
> +	# Yes, we really need the double reset since "letters" appears as
> +	# both a file and a directory.
> +	git reset --hard &&
>  	git reset --hard &&
>  	git clean -f &&
>  	git checkout modify^0 &&
> +
>  	test_must_fail git merge delete &&
>  
> -	test 3 = $(git ls-files -s | wc -l) &&
> -	test 2 = $(git ls-files -u | wc -l) &&
> -	test 1 = $(git ls-files -o | wc -l) &&
> +	test 5 -eq $(git ls-files -s | wc -l) &&
> +	test 4 -eq $(git ls-files -u | wc -l) &&
> +	test 1 -eq $(git ls-files -o | wc -l) &&
>  
>  	test -f letters/file &&
> +	test -f letters.txt &&
>  	test -f letters~HEAD
>  '

A heads-up: This test case fails here on Windows. The messages produced are:

Merging:
63e2d2b modified
virtual delete
found 1 common ancestor(s):
e6b31cd initial
CONFLICT (modify/delete): letters deleted in delete and modified in HEAD.
Version HEAD of letters left in tree at letters~HEAD.
Adding letters/file
error: failed to create path 'letters/file': perhaps a D/F conflict?
Auto-merging letters.txt
CONFLICT (add/add): Merge conflict in letters.txt
Automatic merge failed; fix conflicts and then commit the result.

whereas they are on Linux:

Merging:
274ce87 modified
virtual delete
found 1 common ancestor(s):
343abf7 initial
Removing letters to make room for subdirectory; may re-add later.
Adding letters/file
CONFLICT (modify/delete): letters deleted in delete and modified in HEAD.
Version HEAD of letters left in tree at letters~HEAD.
Auto-merging letters.txt
CONFLICT (add/add): Merge conflict in letters.txt
Automatic merge failed; fix conflicts and then commit the result.

As you can see, "Removing letters..." is missing on Windows, and the file
'letters' is indeed left in the working tree. Any quick ideas where to
begin debugging this?

-- Hannes

Re: [PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Johannes Sixt]

Am 6/8/2011 9:30, schrieb Elijah Newren:
> diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
> index 423fb62..dea2a65 100755
> --- a/t/t6036-recursive-corner-cases.sh
> +++ b/t/t6036-recursive-corner-cases.sh
> @@ -477,7 +477,7 @@ test_expect_success 'setup differently handled merges of directory/file conflict
>  	git tag E2
>  '
>  
> -test_expect_failure 'git detects conflict and handles merge of D & E1 correctly' '
> +test_expect_success 'git detects conflict and handles merge of D & E1 correctly' '
>  	git reset --hard &&
>  	git reset --hard &&
>  	git clean -fdqx &&

This fails on Windows, very likely for the same reason as t6020 that I
posted earlier: the message "Removing a... to make room..." is missing as
well.

-- Hannes

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Johannes Sixt]

Am 7/11/2011 9:04, schrieb Johannes Sixt:
> A heads-up: This test case fails here on Windows. The messages produced are:

> As you can see, "Removing letters..." is missing on Windows, and the file
> 'letters' is indeed left in the working tree. Any quick ideas where to
> begin debugging this?

And the reason for this is that the qsort call in record_df_conflict_files
assumes that qsort is a stable sort; but this is not guaranteed. In
particular, the entry "letters" can be moved after "letters/file", which
is not expected by the loop that follows the qsort.

-- Hannes

Re: [PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Johannes Sixt]

Am 6/8/2011 9:30, schrieb Elijah Newren:
> +static int make_room_for_path(const struct merge_options *o, const char *path)
>  {
> -	int status;
> +	int status, i;
>  	const char *msg = "failed to create path '%s'%s";
>  
> +	/* Unlink any D/F conflict files that are in the way */
> +	for (i = 0; i < o->df_conflict_file_set.nr; i++) {
> +		const char *df_path = o->df_conflict_file_set.items[i].string;
> +		size_t pathlen = strlen(path);
> +		size_t df_pathlen = strlen(df_path);
> +		if (df_pathlen < pathlen && strncmp(path, df_path, df_pathlen) == 0) {
> +			unlink(df_path);
> +			break;
> +		}
> +	}

Each time this loop is entered it tries to remove the same path again,
even if it does not exist anymore or was morphed into a directory in the
meantime. I suggest to remove a path from o->df_conflict_file_set after it
was unlinked. Or even better: have a separate "make room" phase somewhere
in the merge process.

-- Hannes

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Johannes Sixt]

Am 7/12/2011 9:27, schrieb Johannes Sixt:
> Am 7/11/2011 9:04, schrieb Johannes Sixt:
>> A heads-up: This test case fails here on Windows. The messages produced are:
> 
>> As you can see, "Removing letters..." is missing on Windows, and the file
>> 'letters' is indeed left in the working tree. Any quick ideas where to
>> begin debugging this?
> 
> And the reason for this is that the qsort call in record_df_conflict_files
> assumes that qsort is a stable sort; but this is not guaranteed. In
> particular, the entry "letters" can be moved after "letters/file", which
> is not expected by the loop that follows the qsort.

Perhaps something like below. It passes the tests that previously failed.

--- >8 ---
From: Johannes Sixt <j6t@kdbg.org>
Subject: [PATCH] fixup! Do not assume that qsort is stable

Signed-off-by: Johannes Sixt <j6t@kdbg.org>
---
 merge-recursive.c |   13 +++++++++++--
 1 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 88167c3..81fac5f 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -361,6 +361,8 @@ static int string_list_df_name_compare(const void *a, const void *b)
 {
 	const struct string_list_item *one = a;
 	const struct string_list_item *two = b;
+	int onelen = strlen(one->string);
+	int twolen = strlen(two->string);
 	/*
 	 * Here we only care that entries for D/F conflicts are
 	 * adjacent, in particular with the file of the D/F conflict
@@ -373,8 +375,15 @@ static int string_list_df_name_compare(const void *a, const void *b)
 	 * since in other cases any changes in their order due to
 	 * sorting cause no problems for us.
 	 */
-	return df_name_compare(one->string, strlen(one->string), S_IFDIR,
-			       two->string, strlen(two->string), S_IFDIR);
+	int cmp = df_name_compare(one->string, onelen, S_IFDIR,
+				  two->string, twolen, S_IFDIR);
+	/*
+	 * Now that 'foo' and 'foo/bar' compare equal, we have to make sure
+	 * that 'foo' comes before 'foo/bar'.
+	 */
+	if (cmp)
+		return cmp;
+	return onelen - twolen;
 }
 
 static void record_df_conflict_files(struct merge_options *o,
-- 
1.7.6.1317.gd8c5a.dirty

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Junio C Hamano]

Both good comments; thanks.

Re: [PATCH 05/48] t6039: Add tests for content issues with modify/rename/directory conflicts

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Signed-off-by: Elijah Newren <newren@gmail.com>

Could you add a description for this? Your description on 06/48 is
beautifly done. Here is my attempt.

    One branch renames a file and creates a directory where the file used to
    be, and the other branch updates the file in place. Merging them should
    resolve it cleanly as long as the content level change on the branches do
    not overlap and rename is detected, or should leave conflict without
    losing information.

    One branch renames a file and makes a file where the directory the renamed
    file used to be in, and the other branch updates the file in
    place. Merging them should resolve it cleanly as long as the content level
    change on the branches do not overlap and rename is detected, or should
    leave conflict without losing information.

I think I said this in my earlier review, and this is not limited to 05/48
but also applies to 03/48 and 04/48 as well, but there won't be perfect
rename detection, and the rename detection logic could change (improve)
over time. Ideally, I think the test should declare either case as a
success: (1) detection succeeds and avoids unnecessary conflict, or (2)
rename is missed and conflict is reported, but otherwise there is no data
loss. If it expects only one but not the other, any time the rename logic
is improved, the behaviour could change between (1) and (2) and it will
cause a false positive "breakage" of these tests.

Thanks.

Re: [PATCH 06/48] t6039: Add failing testcases for rename/rename/add-{source,dest} conflicts

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Add testcases that cover three failures with current git merge, all
> involving renaming one file on both sides of history:
>
> Case 1:
> ...
> Case 3:
> ...
>
> Signed-off-by: Elijah Newren <newren@gmail.com>

Very nicely described.

> +test_expect_failure 'detect conflict with rename/rename(1to2)/add-source merge' '
> +	git checkout B^0 &&
> +
> +	test_must_fail git merge -s recursive C^0 &&
> +
> +	test -f a &&
> +	test -f b &&
> +	test -f c

Please check the contents of the index here as well.  I think:

 - "a" should be at stage#0, with "something completely different" from C;
 - "b" should have stage #1 and #2, both with the original "a" from A; and
 - "c" should have stage #1 and #3, both with the original "a" from A

right?  The last test in this patch is nicely done in this regard.

> +test_expect_success 'setup rename/rename(1to2)/add-source resolvable conflict' '
> +	git rm -rf . &&
> +	git clean -fdqx &&
> +	rm -rf .git &&
> +	git init &&
> +
> +	>a &&
> +	git add a &&
> +	test_tick &&
> +	git commit -m base &&
> +	git tag A &&
> +
> +	git checkout -b B A &&
> +	git mv a b &&
> +	test_tick &&
> +	git commit -m one &&
> +
> +	git checkout -b C A &&
> +	git mv a b &&
> +	echo important-info >a &&
> +	test_tick &&
> +	git commit -m two
> +'
> +
> +test_expect_failure 'rename/rename/add-source still tracks new a file' '
> +	git checkout C^0 &&
> +	git merge -s recursive B^0 &&
> +
> +	test 2 -eq $(git ls-files -s | wc -l) &&

Instead of just checking the numbers, could you make the expected result a
bit more explicit? What do we expect here? Perhaps

 - "a" in the index is at stage #0, with "important-info";
 - "a" in the working tree is clean;
 - "b" in the index is at stage #0, with empty content; and
 - "b" in the working tree is clean.

Again, the last test in this patch is nicely done in this regard.

> +test_expect_success 'setup rename/rename(1to2)/add-dest conflict' '
> +	git rm -rf . &&
> +	git clean -fdqx &&
> +	rm -rf .git &&
> +	git init &&
> +
> +	echo stuff >a &&
> +	git add a &&
> +	test_tick &&
> +	git commit -m base &&
> +	git tag A &&
> +
> +	git checkout -b B A &&
> +	git mv a b &&
> +	echo precious-data >c &&
> +	git add c &&
> +	test_tick &&
> +	git commit -m one &&
> +
> +	git checkout -b C A &&
> +	git mv a c &&
> +	echo important-info >b &&
> +	git add b &&
> +	test_tick &&
> +	git commit -m two
> +'
> +
> +test_expect_failure 'rename/rename/add-dest merge still knows about conflicting file versions' '
> +	git checkout C^0 &&
> +	test_must_fail git merge -s recursive B^0 &&
> +
> +	test 5 -eq $(git ls-files -s | wc -l) &&
> +	test 2 -eq $(git ls-files -u b | wc -l) &&
> +	test 2 -eq $(git ls-files -u c | wc -l) &&
> +	test $(git rev-parse :2:b) = $(git rev-parse C:b) &&
> +	test $(git rev-parse :3:b) = $(git rev-parse B:b) &&
> +	test $(git rev-parse :2:c) = $(git rev-parse C:c) &&
> +	test $(git rev-parse :3:c) = $(git rev-parse B:c)

5 - 2 - 2 = 1 so we know that it is merged cleanly in the index, but we
would want to also test "a" in the working tree is merged and clean.

Thanks.

Re: [PATCH 08/48] t6036: Add differently resolved modify/delete conflict in criss-cross test

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  t/t6036-recursive-corner-cases.sh |   83 +++++++++++++++++++++++++++++++++++++
>  1 files changed, 83 insertions(+), 0 deletions(-)
>
> diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
> index 319b6fa..52d2ecf 100755
> --- a/t/t6036-recursive-corner-cases.sh
> +++ b/t/t6036-recursive-corner-cases.sh
> @@ -231,4 +231,87 @@ test_expect_success 'git detects differently handled merges conflict' '
>  	test $(git rev-parse :1:new_a) = $(git hash-object merged)
>  '
>  
> +#
> +# criss-cross + modify/delete:
> +#
> +#      B   D
> +#      o---o
> +#     / \ / \
> +#  A o   X   ? F
> +#     \ / \ /
> +#      o---o
> +#      C   E
> +#
> +#   Commit A: file with contents 'A\n'
> +#   Commit B: file with contents 'B\n'
> +#   Commit C: file not present
> +#   Commit D: file with contents 'B\n'
> +#   Commit E: file not present
> +#
> +# Now, when we merge commits D & E, does git detect the conflict?

Starting from A, upper branch wants to have B instead of A while the lower
branch wants the path gone, so it should result in a modify/delete conflict.

  s/# Now,.*$/Merging commits D & E should result in modify/delete conflict./

Re: [PATCH 09/48] t6036: criss-cross with weird content can fool git into clean merge

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> +#   Commit E: file with contents:
> +#      <<<<<<< Temporary merge branch 1
> +#      C
> +#      =======
> +#      B
> +#      >>>>>>> Temporary merge branch 2
> +#
> +# Now, when we merge commits D & E, does git detect the conflict?

This is way too artificial that I personally feel not worth worrying
about.

Re: [PATCH 11/48] t6036: criss-cross w/ rename/rename(1to2)/modify+rename/rename(2to1)/modify

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> +# criss-cross with rename/rename(1to2)/modify followed by
> +# rename/rename(2to1)/modify:
> +#
> +#      B   D
> +#      o---o
> +#     / \ / \
> +#  A o   X   ? F
> +#     \ / \ /
> +#      o---o
> +#      C   E
> +#
> +#   Commit A: new file: a
> +#   Commit B: rename a->b, modifying by adding a line
> +#   Commit C: rename a->c
> +#   Commit D: merge B&C, resolving conflict by keeping contents in newname
> +#   Commit E: merge B&C, resolving conflict similar to D but adding another line
> +#
> +# There is a conflict merging B & C, but one of filename not of file
> +# content.  Whoever created D and E chose specific resolutions for that
> +# conflict resolution.  Now, since: (1) there is no content conflict
> +# merging B & C, (2) D does not modify that merged content further, and (3)
> +# both D & E resolve the name conflict in the same way, the modification to
> +# newname in E should not cause any conflicts when it is merged with D.
> +# (Note that this can be accomplished by having the virtual merge base have
> +# the merged contents of b and c stored in a file named a, which seems like
> +# the most logical choice anyway.)

Interesting. I do not necessarily agree with the choice "a", but it feels
sound to say "B and C do not agree what the final pathname should be, but
we know this content was derived from the common A:a so we use one path
whose name is arbitrary in the virtual merge base X between D and E" and
then further let the rename detection to notice that that arbitrary path
gets renamed between X-D to "newname" and X-E also to "newname" to resolve
it as both sides renaming it to the same new name. It is akin to what we
do at the content level, i.e. "B and C do not agree what the final
contents should be, so we leave the conflict marker but that may cancel
out at the final merge stage".

> +test_expect_failure 'handle rename/rename(1to2)/modify followed by what looks like rename/rename(2to1)/modify' '
> +	git checkout D^0 &&
> +
> +	git merge -s recursive E^0 &&
> +
> +	test 1 -eq $(git ls-files -s | wc -l) &&
> +	test 0 -eq $(git ls-files -u | wc -l) &&
> +	test 0 -eq $(git ls-files -o | wc -l) &&
> +
> +	test 8 -eq $(wc -l < newname)

Be a bit more explicit with what you are actually testing. Are you
expecting that the index has a single stage #0 entry for "newname" whose
contents are the same as E?

> +'
>  
>  test_done

Re: [PATCH 12/48] t6036: criss-cross + rename/rename(1to2)/add-source + modify/modify

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Now THAT's a corner case.
> ...
> +# criss-cross with rename/rename(1to2)/add-source + resolvable modify/modify:
> +#
> +#      B   D
> +#      o---o
> +#     / \ / \
> +#  A o   X   ? F
> +#     \ / \ /
> +#      o---o
> +#      C   E
> +#
> +#   Commit A: new file: a
> +#   Commit B: rename a->b
> +#   Commit C: rename a->c, add different a
> +#   Commit D: merge B&C, keeping b&c and (new) a modified at beginning
> +#   Commit E: merge B&C, keeping b&c and (new) a modified at end

THAT may be a corner case, but is it a useful corner case?  What on earth
the person who did D (or E) was thinking to keep both b and c that are
derived from A:a to begin with?

> +test_expect_failure 'correctly resolves criss-cross with rename/rename/add and modify/modify conflict' '

I won't repeat the same two comments here from reviews for the previous
patches.

Re: [PATCH 19/48] merge-recursive: Remember to free generated unique path names

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> @@ -1433,12 +1434,17 @@ static int process_df_entry(struct merge_options *o,
>  		}
>  	} else if (o_sha && (!a_sha || !b_sha)) {
>  		/* Modify/delete; deleted side may have put a directory in the way */
> -		const char *new_path = path;
> -		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode))
> +		char *new_path;
> +		int free_me = 0;
> +		if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
>  			new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
> +			free_me = 1;
> +		}
>  		clean_merge = 0;
> -		handle_delete_modify(o, path, new_path,
> +		handle_delete_modify(o, path, free_me ? new_path : path,
>  				     a_sha, a_mode, b_sha, b_mode);

Here free_me is not about free_me but is used as a substitute for "did I
stuff the path to be used in new_path, or should I use the original path?",
and I find the variable misnamed.  How about doing it this way instead?

	char *renamed = NULL;
        if (lstat(path, &st) ...)
        	renamed = unique_path(...);
	clean_merge = 0;
	handle_delete_modify(o, path, renamed ? renamed : path, ...);

> +		if (free_me)
> +			free(new_path);

and then you can free renamed unconditoinally here.

Other than that looks good to me.

Thanks.

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> We cannot assume that directory/file conflicts will appear in sorted
> order; for example, 'letters.txt' comes between 'letters' and
> 'letters/file'.
>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  merge-recursive.c   |   31 ++++++++++++++++++++++++++-----
>  t/t6020-merge-df.sh |   26 ++++++++++++++++++--------
>  2 files changed, 44 insertions(+), 13 deletions(-)
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> index 4771fb4..ed1fdb2 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -333,6 +333,28 @@ static struct string_list *get_unmerged(void)
>  	return unmerged;
>  }
>  
> +static int string_list_df_name_compare(const void *a, const void *b)
> +{
> +	const struct string_list_item *one = a;
> +	const struct string_list_item *two = b;
> +	/*
> +	 * Here we only care that entries for D/F conflicts are
> +	 * adjacent, in particular with the file of the D/F conflict
> +	 * appearing before files below the corresponding directory.
> +	 * The order of the rest of the list is irrelevant for us.
> +	 *
> +	 * To achieve this, we sort with df_name_compare and provide
> +	 * the mode S_IFDIR so that D/F conflicts will sort correctly.
> +	 * We use the mode S_IFDIR for everything else for simplicity,
> +	 * since in other cases any changes in their order due to
> +	 * sorting cause no problems for us.
> +	 */

I recall there was an issue of this sorting reported earlier...

> diff --git a/t/t6020-merge-df.sh b/t/t6020-merge-df.sh
> index eec8f4e..27c3d73 100755
> --- a/t/t6020-merge-df.sh
> +++ b/t/t6020-merge-df.sh
> @@ -59,15 +59,19 @@ test_expect_success 'setup modify/delete + directory/file conflict' '
>  	git add letters &&
>  	git commit -m initial &&
>  
> +	# Throw in letters.txt for sorting order fun
> +	# ("letters.txt" sorts between "letters" and "letters/file")
>  	echo i >>letters &&
> -	git add letters &&
> +	echo "version 2" >letters.txt &&
> +	git add letters letters.txt &&

Nice.

Re: [PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Junio C Hamano]

What 23/-26/48 tries to do feels quite sane.
Thanks.

Re: [PATCH 27/48] merge-recursive: Consolidate different update_stages functions

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> We are only calling update_stages_options() one way really, so we can
> consolidate the slightly different variants into one and remove some
> parameters whose values are always the same.

Nice; I wonder if this can come much earlier in the series?

Re: [PATCH 28/48] merge-recursive: Split update_stages_and_entry; only update stages at end

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Instead of having the process_renames logic update the stages in the index
> for the rename destination, have the index updated after process_entry or
> process_df_entry.  This will also allow us to have process_entry determine
> whether a file was tracked and existed in the working copy before the
> merge started.
>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  merge-recursive.c |   35 +++++++++++++++++------------------
>  1 files changed, 17 insertions(+), 18 deletions(-)
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> index b4baa35..7878b30 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -90,6 +90,7 @@ struct stage_data {
>  	} stages[4];
>  	struct rename_df_conflict_info *rename_df_conflict_info;
>  	unsigned processed:1;
> +	unsigned involved_in_rename:1;
>  };
>  
>  static inline void setup_rename_df_conflict_info(enum rename_type rename_type,
> @@ -516,15 +517,11 @@ static int update_stages(const char *path, const struct diff_filespec *o,
>  	return 0;
>  }
>  
> -static int update_stages_and_entry(const char *path,
> -				   struct stage_data *entry,
> -				   struct diff_filespec *o,
> -				   struct diff_filespec *a,
> -				   struct diff_filespec *b,
> -				   int clear)
> +static void update_entry(struct stage_data *entry,
> +			 struct diff_filespec *o,
> +			 struct diff_filespec *a,
> +			 struct diff_filespec *b)
>  {
> -	int options;
> -
>  	entry->processed = 0;
>  	entry->stages[1].mode = o->mode;
>  	entry->stages[2].mode = a->mode;
> @@ -532,7 +529,6 @@ static int update_stages_and_entry(const char *path,
>  	hashcpy(entry->stages[1].sha, o->sha1);
>  	hashcpy(entry->stages[2].sha, a->sha1);
>  	hashcpy(entry->stages[3].sha, b->sha1);
> -	return update_stages(path, o, a, b);
>  }
>  
>  static int remove_file(struct merge_options *o, int clean,
> @@ -1084,12 +1080,11 @@ static int process_renames(struct merge_options *o,
>  							      ren2->dst_entry);
>  			} else {
>  				remove_file(o, 1, ren1_src, 1);

Hopefully this unconditional removal from the working tree will be fixed
by the end of the series, at least for the virtual ancestor merges.

> -				update_stages_and_entry(ren1_dst,
> -							ren1->dst_entry,
> -							ren1->pair->one,
> -							ren1->pair->two,
> -							ren2->pair->two,
> -							1 /* clear */);
> +				update_entry(ren1->dst_entry,
> +					     ren1->pair->one,
> +					     ren1->pair->two,
> +					     ren2->pair->two);
> +				ren1->dst_entry->involved_in_rename = 1;

So the idea is to just record what the stage data should be, and delay
calling update_stages() until we run the content level merge?

> @@ -1291,6 +1287,7 @@ static void handle_delete_modify(struct merge_options *o,
>  }
>  
>  static int merge_content(struct merge_options *o,
> +			 unsigned involved_in_rename,
>  			 const char *path,
>  			 unsigned char *o_sha, int o_mode,
>  			 unsigned char *a_sha, int a_mode,
> @@ -1331,6 +1328,8 @@ static int merge_content(struct merge_options *o,
>  			reason = "submodule";
>  		output(o, 1, "CONFLICT (%s): Merge conflict in %s",
>  				reason, path);
> +		if (involved_in_rename)
> +			update_stages(path, &one, &a, &b);
>  	}
>  
>  	if (df_conflict_remains) {
> @@ -1415,7 +1414,7 @@ static int process_entry(struct merge_options *o,
>  	} else if (a_sha && b_sha) {
>  		/* Case C: Added in both (check for same permissions) and */
>  		/* case D: Modified in both, but differently. */
> -		clean_merge = merge_content(o, path,
> +		clean_merge = merge_content(o, entry->involved_in_rename, path,
>  					    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
>  					    NULL);
>  	} else if (!o_sha && !a_sha && !b_sha) {
> @@ -1459,7 +1458,7 @@ static int process_df_entry(struct merge_options *o,
>  		char *src;
>  		switch (conflict_info->rename_type) {
>  		case RENAME_NORMAL:
> -			clean_merge = merge_content(o, path,
> +			clean_merge = merge_content(o, entry->involved_in_rename, path,
>  						    o_sha, o_mode, a_sha, a_mode, b_sha, b_mode,
>  						    conflict_info->branch1);
>  			break;

Feels sane from a cursory look, but are there cases where we do update_entry()
above and end up not calling merge_content() at all?

Re: [PATCH 29/48] merge-recursive: When we detect we can skip an update, actually skip it

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> ...  The reason this change works is due to the changes of the
> last few patches to (a) record_df_conflict_files instead of just unlinking
> them early, (b) allowing make_room_for_path() to remove D/F entries, and
> (c) the splitting of update_stages_and_entry() to have its functionality
> called at different points.

Nice.

Re: [PATCH 21/48] merge-recursive: Fix recursive case with D/F conflict via add/add conflict

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> When a D/F conflict is introduced via an add/add conflict, when
> o->call_depth > 0 we need to ensure that the higher stage entry from the
> base stage is removed.


>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  merge-recursive.c                 |    2 ++
>  t/t6036-recursive-corner-cases.sh |    4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> index 7da6aa0..4771fb4 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -1480,6 +1480,8 @@ static int process_df_entry(struct merge_options *o,
>  			       "Adding %s as %s",
>  			       conf, path, other_branch, path, new_path);
>  			update_file(o, 0, sha, mode, new_path);
> +			if (o->call_depth)
> +				remove_file_from_cache(path);

Puzzling... This codepath is inside

	if (lstat(path, &st) == 0 && S_ISDIR(st.st_mode)) {
        	...

which _is_ all about the working tree. Why should the code even looking at
the working tree if it can be called for recursive virtual ancestor merge?
I'll keep reading the series, hoping that this is something that will be
fixed in a later patch (ah, 23/48 does that).

At the outermost merge, you would keep the made-up name "new_path" as well
as the original name "path"?

When you added "d/f" (file in a directory) and I added "d" (file), merging
your work to my tree with this code would result in

	d at stage #2 (no 0/1/3);
	d~madeupsuffix in the working tree;
        d/f at stage #3 (no 0/1/2); and
        d/f in the working tree.

right? I am not sure if you also have d-madeupsuffix in the index.

As I said in my review of an earlier patch for test suite, having "d" in
the index would be helpful to the user ("git diff" cannot be used to see
what's different) so we _might_ be better off leaving this instead:

	d-madeupsuffix at stage #2 (no 0/1/3);
	d~madeupsuffix in the working tree;
        d/f at stage #3 (no 0/1/2); and
        d/f in the working tree.

and removing "d" from the index even for the outermost merge.

Thanks.

Re: [PATCH 07/48] t6039: Ensure rename/rename conflicts leave index and workdir in sane state

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> +test_expect_success 'setup simple rename/rename (1to2) conflict' '
> +	git rm -rf . &&
> +	git clean -fdqx &&
> +	rm -rf .git &&
> +	git init &&
> +
> +	echo stuff >a &&
> +	git add a &&
> +	test_tick &&
> +	git commit -m A &&
> +	git tag A &&
> +
> +	git checkout -b B A &&
> +	git mv a b &&
> +	test_tick &&
> +	git commit -m B &&
> +
> +	git checkout -b C A &&
> +	git mv a c &&
> +	test_tick &&
> +	git commit -m C
> +'

This looks like a simpler variant of the last test in 06/48; shouldn't it
come before the more complex one?

> +
> +test_expect_success 'merge has correct working tree contents' '
> +	git checkout C^0 &&
> +
> +	test_must_fail git merge -s recursive B^0 &&
> +
> +	test 3 -eq $(git ls-files -s | wc -l) &&
> +	test 3 -eq $(git ls-files -u | wc -l) &&
> +	test 0 -eq $(git ls-files -o | wc -l) &&
> +
> +	test -f b &&
> +	test -f c
> +'

This test is being a bit too lazy, compared to the better ones in 06/48,
no? What do we want to see here? Perhaps

 - "a" at stage #1 with "stuff", not in the working tree;
 - "b" at stage #3 with "stuff", same in the working tree; and
 - "c" at stage #2 with "stuff", same in the working tree

> +# Test for all kinds of things that can go wrong with rename/rename (2to1):
> +#   Commit A: new files: a & b
> +#   Commit B: rename a->c, modify b
> +#   Commit C: rename b->c, modify a
> +#
> +# Merging of B & C should NOT be clean.  Questions:
> +#   * Both a & b should be removed by the merge; are they?
> +#   * The two c's should contain modifications to a & b; do they?
> +#   * The index should contain two files, both for c; does it?
> +#   * The working copy should have two files, both of form c~<unique>; does it?
> +#   * Nothing else should be present.  Is anything?

What is the most useful thing to leave in the index and in the working
tree for the person who needs to resolve such a merge using the working
tree, starting from B and merging C? The above "Questions" lists what the
current code might try to do but I am not sure if it is really useful. For
example, in the index, you would have to stuff two stage #1 entries ("a"
from A and "b" from A) for path "c", with stage #2 ("c" from B) and stage
#3 ("c" from C) entries, and represent what B tried to do to "a" (in the
above you said "rename a->c" but it does not have to be a rename without
content change) and what C tried to do to "b" in the half-conflicted
result that is in a single file "c". Because the result are totally
unrelated files (one side wants a variant of original "a" there, the other
side wants a variant of "b"), such a half-merge result is totally useless
to help the person to come up with anything.

Also renaming "c" to "c~<unique>", if they do not have corresponding
entries in the index to let you check with "git diff", would make the
result _harder_ to use, not easier. So if you are going to rename "c" to
"c-B" and "c-C", at least it would make much more sense to have in the
index:

 - "c-B", with stage #1 ("a" from A), stage #2 ("c" from B) and stage #3
   ("a" from C);
 - "c-C", with stage #1 ("b" from A), stage #2 ("b" from B) and stage #3
   ("c" from C); and
 - No "a" nor "b" in the index nor in the working tree.

no?

That way, you could run "git diff" to get what happened to the two
variants of "a" and "b" at the content level, and decide to clean things
up with:

    $ git diff ;# view content level merge
    $ edit c-B c-C; git add c-B c-C
    $ git mv c-B c-some-saner-name
    $ git mv c-C c-another-saner-name
    $ edit other files that refer to c like Makefile
    $ git commit

To take it one step further to the extreme, it might give us a more
reasonable and useful conflicted state if we deliberately dropped some
information instead in a case like this, e.g.:

 - We may want to have "a" at stage #1 (from A) in the index;
 - No "a" remains in the working tree;
 - "b" at stage #1 (from A), stage #2 (from B) and stage #3 ("c" from C);
 - "b" in the working tree a conflicted-merge of the above three;
 - "c" at stage #1 ("a" from A), stage #2 (from B), and stage #3 ("a" from
   C); and
 - "c" in the working tree a conflicted-merge of the above three.

Note that unlike the current merge-recursive that tries to come up with a
temporary pathname to store both versions of C, this would ignore "mv b c"
on the A->C branch, and make the conflicted tentative merge asymmetric
(merging B into C and merging C into B would give different conflicts),
but I suspect that the asymmetry may not hurt us.

Whether the merger wants to keep "c" that was derived from "a" (in line
with the HEAD) or "c" that was derived from "b" (in line with MERGE_HEAD),
if the result were to keep both files in some shape, the content level
edit, renaming of at least one side, and adjusting other files that refer
to it, are all required anyway, e.g.

    $ git diff ;# view content level merge
    $ edit b c; git add b c
    $ edit other files that refer to c line Makefile (the content C's
      change wants is now in "b").
    $ git commit

would be a way to pick "c" as "c-some-saner-name" and "b" as
"c-another-saner-name" in the previous workflow, but needs much less
typing. The complexity of the workflow would be the same if the final
resolution is to take what one side did and dropping the other's work,
I think.

Re: [PATCH 10/48] t6036: tests for criss-cross merges with various directory/file conflicts

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  t/t6036-recursive-corner-cases.sh |  149 +++++++++++++++++++++++++++++++++++++
>  1 files changed, 149 insertions(+), 0 deletions(-)
>
> diff --git a/t/t6036-recursive-corner-cases.sh b/t/t6036-recursive-corner-cases.sh
> index dab52a4..4993f67 100755
> --- a/t/t6036-recursive-corner-cases.sh
> +++ b/t/t6036-recursive-corner-cases.sh
> @@ -397,4 +397,153 @@ test_expect_failure 'git detects conflict w/ criss-cross+contrived resolution' '
>  	test $(git rev-parse :3:file) = $(git rev-parse E:file)
>  '
>  
> +#
> +# criss-cross + d/f conflict via add/add:
> +#   Commit A: Neither file 'a' nor directory 'a/' exist.
> +#   Commit B: Introduce 'a'
> +#   Commit C: Introduce 'a/file'
> +# Two different later cases:
> +#   Commit D1: Merge B & C, keeping 'a' and deleting 'a/'
> +#   Commit E1: Merge B & C, deleting 'a' but keeping 'a/file'

Where does the content of a/file (or a) go?  Discarded?

> +#   Commit D2: Merge B & C, keeping a modified 'a' and deleting 'a/'
> +#   Commit E2: Merge B & C, deleting 'a' but keeping a modified 'a/file'

What "modification" is there to be made?  Are these evil merges?

> +#   Note: D == D1.

If you are not building nor testing D2, do not bring it up in the above
description. Otherwise, use D1 and D2 throughout the test and description.

> +# Finally, someone goes to merge D1&E1 or D1&E2 or D2&E1.  What happens?

State what _should_ happen here.  We are not wasting brainwave for mental
masturbation but are trying to solve real problems for real projects here.

Also aren't we interested in what should happen when D2 and E2 are merged?
If not why not?

> +#      B   D1 or D2
> +#      o---o			(file)
> +#     / \ / \
> +#  A o   X   ? F
> +#     \ / \ /
> +#      o---o
> +#      C   E1 or E2		(dir)
> +#

> +test_expect_failure 'git detects conflict and handles merge of D & E1 correctly' '
> +	git reset --hard &&
> +	git reset --hard &&
> +	git clean -fdqx &&
> +	git checkout D^0 &&

Why do you need to reset twice?  Superstition (you have a commented one
introduced later in the series --- perhaps this shows a bug in reset)?

"Start with a clean state at this commit (in this case D^0)" needs a
helper shell function to make everything below easier to read.

> +	# FIXME: If merge-base could keep both a and a/file in its tree, then
> +	# we could this merge would actually be able to succeed.

What exactly do you refer to in the above sentence with "merge-base"?

> +	test_must_fail git merge -s recursive E1^0 &&
> +
> +	test 2 -eq $(git ls-files -s | wc -l) &&
> +	test 1 -eq $(git ls-files -u | wc -l) &&
> +	test 0 -eq $(git ls-files -o | wc -l) &&
> +
> +	test $(git rev-parse :2:a) = $(git rev-parse B:a)
> +'

What is expected here? "a" at stage #2 ("a" from B) is being tested, and
what is the other entry?  "a/file" at stage #3 ("a/file" from E1) or
something else? What should the working tree have?

> +test_expect_failure 'git detects conflict and handles merge of E1 & D correctly' '

Instead of saying "correctly", state what you think should happen, in
other words, what your definition of "correct" is, because in these
made-up cases what is expected could be vastly different from people to
people.

This is the same merge in the opposite direction, right?

Re: [PATCH 18/48] merge-recursive: Mark some diff_filespec struct arguments const

[Junio C Hamano]

13/-18/48 all look very sensible.

Thanks.

Re: [PATCH 12/48] t6036: criss-cross + rename/rename(1to2)/add-source + modify/modify

[Phil Hord]

On 07/18/2011 07:38 PM, Junio C Hamano wrote:
> Elijah Newren<newren@gmail.com>  writes:
>
>> Now THAT's a corner case.
>> ...
>> +# criss-cross with rename/rename(1to2)/add-source + resolvable modify/modify:
>> +#
>> +#      B   D
>> +#      o---o
>> +#     / \ / \
>> +#  A o   X   ? F
>> +#     \ / \ /
>> +#      o---o
>> +#      C   E
>> +#
>> +#   Commit A: new file: a
>> +#   Commit B: rename a->b
>> +#   Commit C: rename a->c, add different a
>> +#   Commit D: merge B&C, keeping b&c and (new) a modified at beginning
>> +#   Commit E: merge B&C, keeping b&c and (new) a modified at end
> THAT may be a corner case, but is it a useful corner case?  What on earth
> the person who did D (or E) was thinking to keep both b and c that are
> derived from A:a to begin with?

I think this may happen rather commonly in one of our repos.  I must say 
I was quite surprised when I recognized it in this discussion.

The details are boring, so feel free to ignore the rest of this message 
unless you just want to understand or contradict me.  There is one 
difference in my real workflow, but for now I'll imagine this difference 
is not there.

Ciao.


Begin boring details:

We have an expensive, enterprise, config control system that tracks 
"low-volume" file revisions by unique filenames in a specific format.  
Here's the life story of a file named foo:

    foo_00    # Revision 00 (initial rev)
    foo_01    # Revision 01
    foo_02    # Revision 02
    foo_A     # Revision A (first 'published' version)
    foo_B     # Revision B (all subsequent versions get published)

When we transition to an externally published version (a one-time 
event), the revisions change from numbers to letters.

We often need to refer to a specific file version by 'versioned name', 
so we keep all the revisions around in a pool of files.

If I have pending changes not yet in the "config control" system, I keep 
a source file in git to hold them until I'm ready to label them with the 
next revision number.

I have some tools that promote my current edition into the next 
appropriately named-revision.

   'next' is used to promote the current source to the next appropriate 
revision-named file.

   'publish' does the same thing, but it explicitly promotes the file to 
revision 'A' from the current numeric revision.

A few weeks ago, this is how foo showed up in my repo:

     pending/foo
     deploy/foo_00
     deploy/foo_01

Here's a greatly simplified overview of the tools with this state:

     next foo        # "mv -n pending/foo deploy/foo_02"
     publish foo     # "mv -n pending/foo deploy/foo_A"

[Elijah's "Commit A: new file: a": a=pending/foo]

The story begins.

[Elijah's "Commit B: rename a->b": b=deploy/foo_01]

Changes in pending/foo need to be released, so I prep with 'next foo'.


     $ next foo        # "mv pending/foo deploy/foo_02"
     deploy/foo_00
     deploy/foo_01
     deploy/foo_02

Then I commit and push.
     $ git add -u . && git add . && git commit -m "next foo" && git push

[Elijah's "Commit C: rename a->c, add different a": c=deploy/foo_A]

My boss has been told (but he forgot to tell me) that we need to go to rev A in this release so the package can be published externally.  So, in his repo he preps with 'publish foo'.

     $publish foo     #"mv -n pending/foo deploy/foo_A"
     deploy/foo_00
     deploy/foo_01
     deploy/foo_A


My boss also has some changes for a future version (probably rev B) so 
he adds those, too.
     $ echo Frabnotz > pending/foo

Then he commits and pushes.
     $ git add -u . && git add . && git commit -m "publish foo" && git push

It's easy to embellish the rest.  It's so easy, I won't.  But you can imagine it.

The only deviation in this story from my real workflow is that I never delete pending/foo.  But if it changed radically each time, I probably would.

Phil

Re: [PATCH 30/48] merge-recursive: Fix deletion of untracked file in rename/delete conflicts

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> In summary:
>   Change <expression> from "o->call_depth || renamed_stage == 2" to
>   "renamed_stage == 2 || !was_tracked(ren1_src)", in order to
>   remove unnecessary code and avoid deleting untracked files.
>
> 96 lines of explanation in the changelog to describe a one-line fix...

Beautifully written analysis, though ;-)

Re: [PATCH 34/48] merge-recursive: Consolidate process_entry() and process_df_entry()

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> The whole point of adding process_df_entry() was to ensure that files of
> D/F conflicts were processed after paths under the corresponding
> directory.  However, given that the entries are in sorted order, all we
> need to do is iterate through them in reverse order to achieve the same
> effect.  That lets us remove some duplicated code, and lets us keep
> track of one less thing as we read the code ("do we need to make sure
> this is processed before process_df_entry() or do we need to defer it
> until then?").
>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  merge-recursive.c |  194 ++++++++++++++++------------------------------------
>  1 files changed, 60 insertions(+), 134 deletions(-)

;-).

Re: [PATCH 36/48] merge-recursive: Provide more info in conflict markers with file renames

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Whenever there are merge conflicts in file contents, we would mark the
> different sides of the conflict with the two branches being merged.
> However, when there is a rename involved as well, the branchname is not
> sufficient to specify where the conflicting content came from.  In such
> cases, mark the two sides of the conflict with branchname:filename rather
> than just branchname.
>
> Signed-off-by: Elijah Newren <newren@gmail.com>

Nice.

Re: [PATCH 37/48] merge-recursive: Fix modify/delete resolution in the recursive case

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> When o->call_depth>0 and we have conflicts, we try to find "middle ground"
> when creating the virtual merge base.  In the case of content conflicts,
> this can be done by doing a three-way content merge and using the result.
> In all parts where the three-way content merge is clean, it is the correct
> middle ground, and in parts where it conflicts there is no middle ground
> but the conflict markers provide a good compromise since they are unlikely
> to accidentally match any further changes.
>
> In the case of a modify/delete conflict, we cannot do the same thing.
> Accepting either endpoint as the resolution for the virtual merge base
> runs the risk that when handling the non-recursive case we will silently
> accept one person's resolution over another without flagging a conflict.
> In this case, the closest "middle ground" we have is actually the merge
> base of the candidate merge bases.  (We could alternatively attempt a
> three way content merge using an empty file in place of the deleted file,
> but that seems to be more work than necessary.)

What did we use before this patch as the "middle ground"?

Doesn't the "middle ground" also need to contain information from both
sides?  At the content level, The half-merge result with conflict marker
contains all the necessary information on what both sides agreed to do
outside the marker area, while showing what each side wished to do inside,
so as you describe, it is a good "middle ground".

Using the merge-base would mean that at the tree-structural level of this
merge you ignore the wish of one side that wanted to delete the path, and
then at the content level you also ignore the wish of both sides (the side
that wanted to delete the path wanted to leave zero content, while the
other side wanted to modify from the version in merge-base, all of which
is ignored by the above argument).

When you want to make a criss-corss merge between E and F that looks like
this:

      A---C---E---?
     / \ /       /
    O   .       /
     \ / \     /
      B---D---F

if there is no deletion, we run a content level merge between C and D by
using X that is a potentially conflicting merge between A and B as a
"virtual" ancestor.

      A---C---E---?
     / \ /       /
    O   X       /
     \ / \     /
      B---D---F

X would contain both what both A and B agreed to do on top of O, and what
A and B wanted to do that they do not agree with. The differing opinion of
A and B are recorded inside the conflict markers.  The change that turns X
into E contains C's opinion (i.e. it would likely agree to take what both
A and B agreed to do, and it may agree with A and the result in E would
resemble what A wanted to bring to the result despite B's objection). If C
and D both agree how to resolve the conflict, then B's "objection" will
cancel out from the three-way merge between E and F that uses X as the
ancestor.

In a delete/modify situation (e.g. A modified while B deleted), there are
three possibilities:

 * C and D both agree to delete the path;

 * C and D both agree to keep the path, with modifications that may or may
   not conflict (which can be handled by the usual content-level merge);

 * C decides to delete, while D decides to modify (or vice versa).

And in the last case, the outer merge ? needs to decide if it wants to
keep or delete the path anyway, so a simplest solution is to punt the
whole recursive business and make it the responsibility of the user to
resolve it as a merge between E and F using O as the common ancestor.
This patch does so in all three cases.

I however wonder if we can do better in the second case (I do not think
the first case would come into the picture, as we would not see such a
path when merging E and F as it would have been long gone from both
branches). We wouldn't know which commits C and D are exactly, but we do
have E and F. If A modified the path and B deleted it, and C and D both
decided to keep the path and E and F both inherited that path, wouldn't it
be fair to say that what both branches wanted to do is closer to what A
did than what B did? In other words, instead of using O, wouldn't it give
us a better result if we used A (the side that did not delete) as the
common ancestor for the content level merge when both E and F has the
path?

> diff --git a/merge-recursive.c b/merge-recursive.c
> index da507a3..5a70a87 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -1309,11 +1309,26 @@ error_return:
>  
>  static void handle_delete_modify(struct merge_options *o,
>  				 const char *path,
> -				 const char *new_path,
> +				 unsigned char *o_sha, int o_mode,
>  				 unsigned char *a_sha, int a_mode,
>  				 unsigned char *b_sha, int b_mode)
>  {
> -	if (!a_sha) {
> +	char *new_path = NULL;
> +	int free_me = 0;

No need for free_me anymore, no?

> +	if (dir_in_way(path, !o->call_depth)) {
> +		new_path = unique_path(o, path, a_sha ? o->branch1 : o->branch2);
> +		free_me = 1;
> +	}
> +
> +	if (o->call_depth) {
> +		/*
> +		 * We cannot arbitrarily accept either a_sha or b_sha as
> +		 * correct; since there is no true "middle point" between
> +		 * them, simply reuse the base version for virtual merge base.
> +		 */
> +		remove_file_from_cache(path);
> +		update_file(o, 0, o_sha, o_mode, new_path ? new_path : path);
> +	} else if (!a_sha) {
>  		output(o, 1, "CONFLICT (delete/modify): %s deleted in %s "
>  		       "and modified in %s. Version %s of %s left in tree%s%s.",
>  		       path, o->branch1,
> @@ -1330,6 +1345,9 @@ static void handle_delete_modify(struct merge_options *o,
>  		       NULL == new_path ? "" : new_path);
>  		update_file(o, 0, a_sha, a_mode, new_path ? new_path : path);
>  	}
> +	if (free_me)
> +		free(new_path);
> +

... as new_path is non-NULL only when we allocated it.

Re: [PATCH 39/48] merge-recursive: Fix rename/rename(1to2) resolution for virtual merge base

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> When renaming one file to two files, we really should be doing a content
> merge.  Also, in the recursive case, undoing the renames and recording the
> merged file in the index with the source of the rename (while deleting
> both destinations) allows the renames to be re-detected in the
> non-recursive merge and will result in fewer spurious conflicts.

In other words,...

> +		 * FIXME: For rename/add-source conflicts (if we could detect
> +		 * such), this is wrong.  We should instead find a unique
> +		 * pathname and then either rename the add-source file to that
> +		 * unique path, or use that unique path instead of src here.
>  		 */
> +		update_file(o, 0, mfi.sha, mfi.mode, src);
> +		remove_file_from_cache(ren1_dst);
> +		remove_file_from_cache(ren2_dst);

... the use of "src" here is taking a "middle ground" and punting on
resolving the conflicting opinions of both branches for the outer merge to
resolve. I think that is a sensible thing to do.

The rename destinations of both branches are removed. What happens if
ren1_dst (the path one branch wanted to rename src to) were added by the
other branch (which wanted to rename src to ren2_dst), causing rename/add
conflict between branches (i.e. not the one you worry about in the above
FIXME which is about one branch renaming src to ren1_dst while adding an
unrelated content to src)?

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Unfortunately I seem to have found a regression that manifests in the real
life.

When this series is merged to 'next', it mismerges a trivial renamed path.
A sample commit to reproduce is 2d11f21c3, which is a merge between
28b9264dd6 and 5b42477b59.  Check out the former and try to merge in the
latter.

The merge base of these two commits is 2cfe8a68ccb (no criss-cross).

One branch (current one, 28b9264dd6) does not have that path, but it does
have t/t4037-whitespace-status.sh:

    $ git ls-tree 28b9264dd6 t/t4037-whitespace-status.sh
    100755 blob 3c728a3ebf9ce52e5c24c81525d5cb749cfb2957 t/t4037-whitespace-status.sh

What happened to that path on the current branch is this:

    $ git log -m -M --raw --pretty=short 2cfe8a68ccb..28b9264dd6 -- \
      t/t40{37,40}-whitespace-status.sh
    commit af7b41c923677ff9291bab56ec7069922e37453b
    Author: Jeff King <peff@peff.net>

        diff_tree: disable QUICK optimization with diff filter

    :100755 100755 abc4934... 3c728a3... M  t/t4037-whitespace-status.sh

In other words, we had t4037, and we have t4037 but we updated its
contents from abc4934 to 3c728a3.

Running "log -m -M --raw --pretty=short 2cfe8a68ccb..5b42477b59" shows
that the merged branch (5b42477b59) has renamed t4037 to t4040 at
0e098b6d:

    commit 0e098b6d79fbcab763874f2b6fde5aa82144d150
    Author: Johannes Sixt <j6t@kdbg.org>

        Make test case number unique

    :100755 100755 a30b03b... a30b03b... R100 t/t4037-whitespace-status.sh    t/t4040-wh..

This commit is on a side-branch that has not been merged at the merge base
2cfe8a68ccb; its contents a30b03b is older than what 2cfe8a68ccb has, and
was updated by 2cfe8a68ccb to abc4934.

Another merge after this commit updated it at 7d0cf357:

    commit 7d0cf357a31cc8a442342696788d776265482ce9 (from 98b256bd...)
    Merge: 98b256b 2cfe8a6
    Author: Junio C Hamano <gitster@pobox.com>

        Merge branch 'jc/maint-diff-q-filter'

    :100755 100755 a30b03b... abc4934... M  t/t4040-whitespace-status.sh

This abc4934 is the same contents as the merge base had at t4037.

So this is an "our side kept t4037 and updated its contents from abc4934
to 3c728a3, while the other side renamed t4037 to t4040 and kept its
contents as abc4934". We should merge this as structural change "our side
left it untouched, their side renamed, so take the rename without
conflict" and content level change "our side updated but their side left
it untouched, so take our modification without conflict".

In other words, in the result, we should have 3c728a3 at t4040, and that
indeed is what we got in the recorded history:

    $ git ls-tree 2d11f21c3 -- t/t4040-whitespace-status.sh
    100755 blob 3c728a3ebf9ce52e5c24c81525d5cb749cfb2957 t/t4040-whitespace-status.sh

However, the merge retried with this series cleanly resolves the path, but
with wrong contents:

    $ git checkout 28b9264dd6
    $ git merge 5b42477b59
    $ git ls-files -s t/t4040-whitespace-status.sh
    100755 abc49348b196cf0fec232b6f2399356e4fe324d5 0 t/t4040-whitespace-status.sh

Correct answer should be:

    $ git ls-files -s t/t4040-whitespace-status.sh
    100755 3c728a3ebf9ce52e5c24c81525d5cb749cfb2957 0 t/t4040-whitespace-status.sh

I am rewinding today's integration of 'next' to unmerge this topic
now. :-(...

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

A very simple reproduction recipe.

-- >8 --
#!/bin/sh

mkdir en && cd en || exit

git init

echo 1 >one
git add one
git commit -m 'origin'

git checkout -b side
git mv one two
git commit -m 'side renames one to two'

git checkout master
echo 2 >one
git add one
git commit -m 'master updates one'

git checkout HEAD^0
git merge side
-- 8< --

Tonight's "pu" fails like this:

$ git merge side
error: addinfo_cache failed for path 'two'
Merge made by the 'recursive' strategy.
 one |    1 -
 two |    1 +
 2 files changed, 1 insertions(+), 1 deletions(-)
 delete mode 100644 one
 create mode 100644 two

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Elijah Newren]

On Wed, Aug 3, 2011 at 7:48 PM, Junio C Hamano <gitster@pobox.com> wrote:
> A very simple reproduction recipe.

Thanks, I'll take a look this weekend.  I'm a little surprised that
you mentioned merging this to next already; I was assuming you wanted
me to address the various things you pointed out in your reviews first
(as well as the comments from Johannes).  I've been swamped at work,
particularly after trying to catch up on the pile of work accumulated
during my vacation, but should have time again this weekend...finally.


Elijah

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Elijah Newren]

On Wed, Aug 3, 2011 at 6:20 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Unfortunately I seem to have found a regression that manifests in the real
> life.
>
> When this series is merged to 'next', it mismerges a trivial renamed path.

I know the problem, and can think of a couple potential solutions but
I'm not sure which one of those (if any) should be used.  I may need
advice from an unpack_trees expert.

This breakage was introduced in d487486 (merge-recursive: When we
detect we can skip an update, actually skip it 2011-06-08), and the
reason for the breakage is that was_tracked(path) is lying to us.  Let
me explain...

If you run with GIT_MERGE_VERBOSITY=3, you'll note that you get the message
  "Skipped two (merged same as existing)"
which is NOT the path it should be taking.  The reason for this is
that the call to was_tracked('two') just before that is returning true
for us when the correct answer is false.  Explaining why we get the
wrong answer requires understanding a couple things.

Using your simple reproduction recipe as an example to explain this,
the sha1sums and paths for your merge case are:
  Base commit: d00491f... one
  HEAD commit: 0cfbf08... one
  side commit: d00491f... two
In other words, 'one' is modified in HEAD, and unchanged other than
being renamed (to 'two') on side.

Now, was_tracked() simply checks whether there is a cache entry in the
current index with either stage 0 or stage 2.  Clearly, I was
expecting 'two' to only appear with stage 3, but that's not what
happened.

When unpacking these trees, we got three cache entries: one for path
'one' at stage 1, one for path 'one' at stage 2, and one for path
'two' at stage 0.  Yes, stage 0 and not stage 3.  The reason for this
is that this is case 2ALT from
Documentation/technical/trivial-merge.txt:

case  ancest    head    remote    result
----------------------------------------
...
2ALT  (empty)+  *empty* remote    remote

This means the threeway_merge code considers it a clean merge and
calls merged_entry on it, which clears the CE_STAGEMASK bits from the
cache entry flags.

Since the cache entry for path 'two' has the CE_STAGEMASK bits
cleared, when we call was_tracked on that path, we get back the answer
true.  'two' was not tracked in HEAD, though, so this is the wrong
answer and the source of the bug.


Now, the question is...how do we fix the was_tracked function?  It
would be nice to make use of the original index we had before
unpacking, but that is overwritten at the end of unpack_trees.  Should
we save it somewhere?  Or should we save off all the tracked filenames
before unpacking into a linked list and then have was_tracked() use
that instead of looking it up in the current cache? (I'm slightly
worried that may be slow given the number of lookups that exist).

Thoughts?

Elijah

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> ...  It
> would be nice to make use of the original index we had before
> unpacking, but that is overwritten at the end of unpack_trees.

I somehow thought that we can give separate src and dst index
to the unpack_tree() machinery these days. Aren't you using it?

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Elijah Newren]

On Thu, Aug 4, 2011 at 1:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Elijah Newren <newren@gmail.com> writes:
>
>> ...  It
>> would be nice to make use of the original index we had before
>> unpacking, but that is overwritten at the end of unpack_trees.
>
> I somehow thought that we can give separate src and dst index
> to the unpack_tree() machinery these days. Aren't you using it?

Ah, yes, it appears to be possible.  I have not touched that part of
merge-recursive, so it still reads:
	opts.fn = threeway_merge;
	opts.src_index = &the_index;
	opts.dst_index = &the_index;
	setup_unpack_trees_porcelain(&opts, "merge");
I'll take that route to fix this.

Thanks.

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> On Thu, Aug 4, 2011 at 1:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Elijah Newren <newren@gmail.com> writes:
>>
>>> ... It
>>> would be nice to make use of the original index we had before
>>> unpacking, but that is overwritten at the end of unpack_trees.
>>
>> I somehow thought that we can give separate src and dst index
>> to the unpack_tree() machinery these days. Aren't you using it?
>
> Ah, yes, it appears to be possible.

But why do you care about the original index (i.e. the starting state of
our side) in the first place? If your algorithm depends on the original
index, wouldn't that mean you would screw up the same merge if the user
merged branches in the opposite direction?

I think the fact you have a path "two" at stage 0, combined with the two
diffs you ran for rename detection between the common ancestor and two
branches, should be enough to decide if one branch added the path or moved
it from elsewhere (in which case the common ancestor would not have that
path), or it kept the path at the original place (with or without content
change), no?

Re: [PATCH 00/48] Handling more corner cases in merge-recursive.c

[Elijah Newren]

On Fri, Aug 5, 2011 at 11:22 PM, Junio C Hamano <gitster@pobox.com> wrote:
> But why do you care about the original index (i.e. the starting state of
> our side) in the first place? If your algorithm depends on the original
> index, wouldn't that mean you would screw up the same merge if the user
> merged branches in the opposite direction?

No, it does not mess up the same merge in the opposite direction.  The
code in question (with some local modifications) is

       if (mfi.clean && !df_conflict_remains &&
           sha_eq(mfi.sha, a_sha) && mfi.mode == a_mode) {
               output(o, 3, "Skipped %s (merged same as existing)", path);
               /*
                * The content merge resulted in the same file contents we
                * already had.  We can return early if those file contents
                * are recorded at the correct path (which may not be true
                * if the merge involves a rename).
                */
               if (was_tracked(&o->original_index, path)) {
                       add_cacheinfo(mfi.mode, mfi.sha, path,
                                     0 /*stage*/, 1 /*refresh*/, 0 /*options*/)
                       return mfi.clean;
               }
       } else
                output(o, 2, "Auto-merging %s", path);

And the merge is:
  BASE: original-path: v1
  HEAD: original-path: v2
  SIDE: renamed-path: v1

So, HEAD did have the correct contents that we wanted, as checked in
the first if.  However, we cannot bail early because those contents
were recorded at the wrong path.  Repeating the merge in the other
direction would simply show that we don't have the right contents
(a_sha != mfi.sha).  Only if we have both the right contents and have
it at the right path should we exit early.

If we use "&the_index" instead of "&o->original_index", which is what
my previous series essentially did, then we get the wrong answer about
whether the contents are recorded at the necessary path, i.e.
was_tracked() lies to us.


Now, your gut feel and question here does turn out to be
important...in a different case.  You'll note that I'm passing an
index to was_tracked(); that's because there is one case (the call to
was_tracked() from would_lose_untracked()) where it is important that
we use the index as modified by unpack_trees rather than the original
index and not doing so causes a bug depending on the direction things
are merged.  I added a big comment in the code about that one.

> I think the fact you have a path "two" at stage 0, combined with the two
> diffs you ran for rename detection between the common ancestor and two
> branches, should be enough to decide if one branch added the path or moved
> it from elsewhere (in which case the common ancestor would not have that
> path), or it kept the path at the original place (with or without content
> change), no?

You are correct.  I just need to reorder the patch series somewhat
(other patches added the passing of the relevant diff_filepair
information to merge_content, which is needed to do these checks you
suggest), and then I can change the was_tracked() call to instead
compare pathnames.

Re: [PATCH 05/48] t6039: Add tests for content issues with modify/rename/directory conflicts

[Elijah Newren]

Hi,

Thanks for the many detailed reviews.  I've spent much of the weekend
(and before) trying to go through and clean things up, and am still
working on it.  There are some things to comment on outside of the
series re-roll I am putting together...

On Mon, Jul 18, 2011 at 5:37 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Could you add a description for this? Your description on 06/48 is
> beautifly done. Here is my attempt.
...
Done. :-)  It'll be in the reroll I submit.

> I think I said this in my earlier review, and this is not limited to 05/48
> but also applies to 03/48 and 04/48 as well, but there won't be perfect
> rename detection, and the rename detection logic could change (improve)
> over time. Ideally, I think the test should declare either case as a
> success: (1) detection succeeds and avoids unnecessary conflict, or (2)
> rename is missed and conflict is reported, but otherwise there is no data
> loss. If it expects only one but not the other, any time the rename logic
> is improved, the behaviour could change between (1) and (2) and it will
> cause a false positive "breakage" of these tests.

The same could be said for all the rename tests, but I'm a bit worried
about doing this, particularly for the set of tests I added.  I came
up with my testcases by carefully reviewing the code looking for ways
I could trigger problems (which were almost never covered by an
existing regression test), plus trying a few small modifications of
any tests I did come up with.  The tests I added that involve renames
really are specifically designed to test renaming situations.  If
rename detection logic is modified in a way that affects these tests,
and these tests aren't updated, then we are turning some useful unique
tests into mere duplication of simpler tests we have elsewhere.

If you are worried that these tests lack robustness in the face of
rename-detection changes, perhaps we should modify the tests to use
file contents that are more likely to continue being detected as
renames (e.g. longer lines or maybe more of them)?  Or, maybe add an
earlier check in the test to independently verify that the rename is
detected?

Re: [PATCH 07/48] t6039: Ensure rename/rename conflicts leave index and workdir in sane state

[Elijah Newren]

On Mon, Jul 18, 2011 at 5:40 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> +# Test for all kinds of things that can go wrong with rename/rename (2to1):
>> +#   Commit A: new files: a & b
>> +#   Commit B: rename a->c, modify b
>> +#   Commit C: rename b->c, modify a
>> +#
>> +# Merging of B & C should NOT be clean.  Questions:
>> +#   * Both a & b should be removed by the merge; are they?
>> +#   * The two c's should contain modifications to a & b; do they?
>> +#   * The index should contain two files, both for c; does it?
>> +#   * The working copy should have two files, both of form c~<unique>; does it?
>> +#   * Nothing else should be present.  Is anything?
>
> What is the most useful thing to leave in the index and in the working
> tree for the person who needs to resolve such a merge using the working
> tree, starting from B and merging C? The above "Questions" lists what the
> current code might try to do but I am not sure if it is really useful. For
> example, in the index, you would have to stuff two stage #1 entries ("a"
> from A and "b" from A) for path "c", with stage #2 ("c" from B) and stage
> #3 ("c" from C) entries, and represent what B tried to do to "a" (in the
> above you said "rename a->c" but it does not have to be a rename without
> content change) and what C tried to do to "b" in the half-conflicted
> result that is in a single file "c". Because the result are totally
> unrelated files (one side wants a variant of original "a" there, the other
> side wants a variant of "b"), such a half-merge result is totally useless
> to help the person to come up with anything.
>
> Also renaming "c" to "c~<unique>", if they do not have corresponding
> entries in the index to let you check with "git diff", would make the
> result _harder_ to use, not easier. So if you are going to rename "c" to
> "c-B" and "c-C", at least it would make much more sense to have in the
> index:
>
>  - "c-B", with stage #1 ("a" from A), stage #2 ("c" from B) and stage #3
>   ("a" from C);
>  - "c-C", with stage #1 ("b" from A), stage #2 ("b" from B) and stage #3
>   ("c" from C); and
>  - No "a" nor "b" in the index nor in the working tree.
>
> no?
>
> That way, you could run "git diff" to get what happened to the two
> variants of "a" and "b" at the content level, and decide to clean things
> up with:
>
>    $ git diff ;# view content level merge
>    $ edit c-B c-C; git add c-B c-C
>    $ git mv c-B c-some-saner-name
>    $ git mv c-C c-another-saner-name
>    $ edit other files that refer to c like Makefile
>    $ git commit

That sounds very interesting.  My first thought is that you'd have to
do the same thing in the case of a D/F conflict, but I notice that
later in the patch series you asked for exactly that.  The idea
certainly has potential, though I might need to think it through a
little more.

> To take it one step further to the extreme, it might give us a more
> reasonable and useful conflicted state if we deliberately dropped some
> information instead in a case like this, e.g.:
>
>  - We may want to have "a" at stage #1 (from A) in the index;
>  - No "a" remains in the working tree;
>  - "b" at stage #1 (from A), stage #2 (from B) and stage #3 ("c" from C);
>  - "b" in the working tree a conflicted-merge of the above three;
>  - "c" at stage #1 ("a" from A), stage #2 (from B), and stage #3 ("a" from
>   C); and
>  - "c" in the working tree a conflicted-merge of the above three.
>
> Note that unlike the current merge-recursive that tries to come up with a
> temporary pathname to store both versions of C, this would ignore "mv b c"
> on the A->C branch, and make the conflicted tentative merge asymmetric
> (merging B into C and merging C into B would give different conflicts),
> but I suspect that the asymmetry may not hurt us.
>
> Whether the merger wants to keep "c" that was derived from "a" (in line
> with the HEAD) or "c" that was derived from "b" (in line with MERGE_HEAD),
> if the result were to keep both files in some shape, the content level
> edit, renaming of at least one side, and adjusting other files that refer
> to it, are all required anyway, e.g.
>
>    $ git diff ;# view content level merge
>    $ edit b c; git add b c
>    $ edit other files that refer to c line Makefile (the content C's
>      change wants is now in "b").
>    $ git commit
>
> would be a way to pick "c" as "c-some-saner-name" and "b" as
> "c-another-saner-name" in the previous workflow, but needs much less
> typing. The complexity of the workflow would be the same if the final
> resolution is to take what one side did and dropping the other's work,
> I think.

I think the asymmetry is slightly confusing and could become
problematic.  If we decide to turn on break detection, then we would
hit problems in a scenario such as:

Commit A: files a, b are present
Commit B: rename a->c, add an unrelated a
Commit C: rename b->c, add an unrelated b

In that case, "undoing" the rename as you suggest gives us a conflict
with other content that was added at the path.


Also, as mentioned above, D/F conflicts hit similar cases where we
need to rename the path in the working copy.  If we try to handle them
similarly to how you are suggesting for the rename/rename(2to1) case,
we can do so in some cases but hit problems in others.  For example,
take a rename/delete conflict with D/F conflicts:

Commit A: file a is present
Commit B: rename a -> df, possibly also modifying it
Commit C: delete a, add files a/foo and df/bar

We can't use either the path 'df' or 'a' for recording the content.  I
think the rules become too confusing for "selectively undoing renames"
and it'd be easier to just use <bad-dest-path>~<unique> in all cases.
However, I think your suggestion to move index stage information to
these uniquely renamed paths could probably work and may be useful.

Re: [PATCH 09/48] t6036: criss-cross with weird content can fool git into clean merge

[Elijah Newren]

On Mon, Jul 18, 2011 at 5:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Elijah Newren <newren@gmail.com> writes:
>
>> +#   Commit E: file with contents:
>> +#      <<<<<<< Temporary merge branch 1
>> +#      C
>> +#      =======
>> +#      B
>> +#      >>>>>>> Temporary merge branch 2
>> +#
>> +# Now, when we merge commits D & E, does git detect the conflict?
>
> This is way too artificial that I personally feel not worth worrying
> about.

I wasn't going to try to do anything to fix it.  Should I drop this
patch from the series or is it worth keeping around for completeness?

Re: [PATCH 10/48] t6036: tests for criss-cross merges with various directory/file conflicts

[Elijah Newren]

On Mon, Jul 18, 2011 at 5:40 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> +test_expect_failure 'git detects conflict and handles merge of D & E1 correctly' '
>> +     git reset --hard &&
>> +     git reset --hard &&
>> +     git clean -fdqx &&
>> +     git checkout D^0 &&
>
> Why do you need to reset twice?  Superstition (you have a commented one
> introduced later in the series --- perhaps this shows a bug in reset)?

I could have sworn that when I first created these tests last
September that the double reset was needed to get back to a clean
state.  I can't seem to duplicate the issue now, though I do see some
"error" messages printed on the first reset despite the reset working
successfully.  I'll drop the extra one.  *shrug*


[I'm not ignoring the many other good comments you made on this patch;
they just got incorporated into code and comment changes for the next
version of the series I send out.]

Re: [PATCH 22/48] merge-recursive: Fix sorting order and directory change assumptions

[Elijah Newren]

On Mon, Jul 18, 2011 at 5:39 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> +static int string_list_df_name_compare(const void *a, const void *b)
>> +{
>> +     const struct string_list_item *one = a;
>> +     const struct string_list_item *two = b;
>> +     /*
>> +      * Here we only care that entries for D/F conflicts are
>> +      * adjacent, in particular with the file of the D/F conflict
>> +      * appearing before files below the corresponding directory.
>> +      * The order of the rest of the list is irrelevant for us.
>> +      *
>> +      * To achieve this, we sort with df_name_compare and provide
>> +      * the mode S_IFDIR so that D/F conflicts will sort correctly.
>> +      * We use the mode S_IFDIR for everything else for simplicity,
>> +      * since in other cases any changes in their order due to
>> +      * sorting cause no problems for us.
>> +      */
>
> I recall there was an issue of this sorting reported earlier...

Yes, in git-fast-export, though.  It was 060df62 (fast-export: Fix
output order of D/F changes 2010-07-09), if you're curious.

Re: [PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Elijah Newren]

Hi,

On Wed, Jul 13, 2011 at 1:17 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Am 6/8/2011 9:30, schrieb Elijah Newren:
>> +static int make_room_for_path(const struct merge_options *o, const char *path)
>>  {
>> -     int status;
>> +     int status, i;
>>       const char *msg = "failed to create path '%s'%s";
>>
>> +     /* Unlink any D/F conflict files that are in the way */
>> +     for (i = 0; i < o->df_conflict_file_set.nr; i++) {
>> +             const char *df_path = o->df_conflict_file_set.items[i].string;
>> +             size_t pathlen = strlen(path);
>> +             size_t df_pathlen = strlen(df_path);
>> +             if (df_pathlen < pathlen && strncmp(path, df_path, df_pathlen) == 0) {
>> +                     unlink(df_path);
>> +                     break;
>> +             }
>> +     }
>
> Each time this loop is entered it tries to remove the same path again,
> even if it does not exist anymore or was morphed into a directory in the
> meantime. I suggest to remove a path from o->df_conflict_file_set after it
> was unlinked. Or even better: have a separate "make room" phase somewhere
> in the merge process.

Removing it from o->df_conflict_file_set makes sense.  However, there
appears to be no API in string_list.h for deleting entries.  Are such
operations discouraged?  I'm not sure whether to add such API, just
hack it directly, or wait for someone else to come along and change
this to a better data structure (such as a hash)...

I don't think it's possible to move this "make room" phase anywhere
earlier in the merge process.  When we have D/F conflicts, the files
of those D/F conflicts should only be removed if at least one of the
paths under the corresponding directory are not removed by the merge
process.  We don't know whether those paths will need to be removed
until we call process_entry() on each of them, and from there we go
right to this function when we find one that needs to stick around.
So I simply don't see how to move it any earlier.


However, there is clearly a bug in this code.  If there is a D/F
conflict at 'd' (e.g. paths 'd' and 'd/foo' are present), and there is
a file named 'd_bla', then the need to merge 'd_bla' can cause 'd' to
be deleted.  Oops.  (Granted, it's a bug that would be masked by the
later call to process_entry() on 'd', which would reinstate that file
if necessary, but that doesn't mean this code is right.)  I'll fix
that up.

However, I don't see how any of this would address any failure you're
seeing on windows.  Maybe one of my other changes, including one or
two other bugfixes I've found will help?  I'll have to ping you when I
submit the re-roll.

Re: [PATCH 37/48] merge-recursive: Fix modify/delete resolution in the recursive case

[Elijah Newren]

On Thu, Jul 21, 2011 at 12:43 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Elijah Newren <newren@gmail.com> writes:
>
>> When o->call_depth>0 and we have conflicts, we try to find "middle ground"
>> when creating the virtual merge base.  In the case of content conflicts,
>> this can be done by doing a three-way content merge and using the result.
>> In all parts where the three-way content merge is clean, it is the correct
>> middle ground, and in parts where it conflicts there is no middle ground
>> but the conflict markers provide a good compromise since they are unlikely
>> to accidentally match any further changes.
>>
>> In the case of a modify/delete conflict, we cannot do the same thing.
>> Accepting either endpoint as the resolution for the virtual merge base
>> runs the risk that when handling the non-recursive case we will silently
>> accept one person's resolution over another without flagging a conflict.
>> In this case, the closest "middle ground" we have is actually the merge
>> base of the candidate merge bases.  (We could alternatively attempt a
>> three way content merge using an empty file in place of the deleted file,
>> but that seems to be more work than necessary.)
>
> What did we use before this patch as the "middle ground"?

We used precisely what you seem to suggest at the end of your email.
It's a trade-off, and I agree with you about the cases where my new
choice is suboptimal, but I personally think the original choice has
worse trade-offs.

> Doesn't the "middle ground" also need to contain information from both
> sides?  At the content level, The half-merge result with conflict marker
> contains all the necessary information on what both sides agreed to do
> outside the marker area, while showing what each side wished to do inside,
> so as you describe, it is a good "middle ground".
>
> Using the merge-base would mean that at the tree-structural level of this
> merge you ignore the wish of one side that wanted to delete the path, and
> then at the content level you also ignore the wish of both sides (the side
> that wanted to delete the path wanted to leave zero content, while the
> other side wanted to modify from the version in merge-base, all of which
> is ignored by the above argument).

I think it is optimal if the "middle ground" can contain information
from both sides, but I simply do not see how it is possible in either
a modify/delete or rename/delete case.  That is, at least not without
some kind of significant change to the index data structures or
allowing a "virtual merge base" to be an index with higher stage merge
entries instead of just a tree (though that would introduce several
new complications).

And yes, I am essentially ignoring the intents of both sides, and
intentionally so.  I thought I addressed that in the arguments above,
but perhaps I could have been more clear and explicit about that.
Your explanation below provides a great framework for clarifying why I
made this change, so let me respond below using the framework you have
beautifully illustrated and explained.

> When you want to make a criss-corss merge between E and F that looks like
> this:
>
>      A---C---E---?
>     / \ /       /
>    O   .       /
>     \ / \     /
>      B---D---F
>
> if there is no deletion, we run a content level merge between C and D by
> using X that is a potentially conflicting merge between A and B as a
> "virtual" ancestor.
>
>      A---C---E---?
>     / \ /       /
>    O   X       /
>     \ / \     /
>      B---D---F
>
> X would contain both what both A and B agreed to do on top of O, and what
> A and B wanted to do that they do not agree with. The differing opinion of
> A and B are recorded inside the conflict markers.  The change that turns X
> into E contains C's opinion (i.e. it would likely agree to take what both
> A and B agreed to do, and it may agree with A and the result in E would
> resemble what A wanted to bring to the result despite B's objection). If C
> and D both agree how to resolve the conflict, then B's "objection" will
> cancel out from the three-way merge between E and F that uses X as the
> ancestor.
>
> In a delete/modify situation (e.g. A modified while B deleted), there are
> three possibilities:
>
>  * C and D both agree to delete the path;
>
>  * C and D both agree to keep the path, with modifications that may or may
>   not conflict (which can be handled by the usual content-level merge);
>
>  * C decides to delete, while D decides to modify (or vice versa).

This is a really nicely written explanation.  I particularly like this
list of three cases.  Let me augment this list of situations (which
I'll number (1)-(3)) with a list of four choices for virtual merge
base X:

(J) Make X not have any content at the path (i.e. accept the deletion in B)

(K) Make X have the content from O at path (i.e. what my patch does)

(L) Make X have the content from A at path (which we did before)

(M) Make X have some other content at path

I'll get back to these 12 situations (3 cases each with how any of 4
choices could affect them) in a minute...

> And in the last case, the outer merge ? needs to decide if it wants to
> keep or delete the path anyway, so a simplest solution is to punt the
> whole recursive business and make it the responsibility of the user to
> resolve it as a merge between E and F using O as the common ancestor.
> This patch does so in all three cases.
>
> I however wonder if we can do better in the second case (I do not think
> the first case would come into the picture, as we would not see such a
> path when merging E and F as it would have been long gone from both
> branches). We wouldn't know which commits C and D are exactly, but we do
> have E and F. If A modified the path and B deleted it, and C and D both
> decided to keep the path and E and F both inherited that path, wouldn't it
> be fair to say that what both branches wanted to do is closer to what A
> did than what B did? In other words, instead of using O, wouldn't it give
> us a better result if we used A (the side that did not delete) as the
> common ancestor for the content level merge when both E and F has the
> path?

Yes, absolutely.  In the second case, using the content of the path
from A for the virtual merge ancestor would be a slight improvement
over using the content for path from O.  But let's look at all three
cases you specified and how different choices for the virtual merge
base affect it.  Here are my goals for any merge, in priority order:

1st goal: detect real conflicts and notify the user
2nd goal: avoid conflicts in cases where we should be able to
reasonably confidently merge cleanly
3rd goal: make it easy for the user to figure out how to resolve problems

Obviously, we can't always meet these goals, particularly due to
"semantic" conflicts that may be present.  But they're useful to keep
in mind.  Here is how I broke down the three cases:



Case 1: C and D both agree to delete the path

In this case, regardless of the choice (J)-(M), the merge will be
resolved correctly.


Case 2: C and D both agree to keep the path, with modifications that
may or may not conflict (which can be handled by the usual
content-level merge);

In this case, choice (J) would cause content conflicts whenever C & D
had different content at path and would treat it as a two-way content
merge.  That makes choice (J) particularly problematic as it breaks
the first goal.  In order to get the merge right, we really need to do
a three-way content merge, using content that can serve as a
reasonable merge-base for the contents chosen at C & D.  That makes
choice (M) for X a poor one.  Both choices (K) and (L) are reasonable,
with (L) being a better choice (as you also point out above) to
optimize the 2nd goal.


Case 3: C decides to delete, while D decides to modify (or vice versa)

It is worth noting here that the most likely choice for the content at
path for D is exactly the content that is at A.  git is designed for
trees of slowly changing data after all.

This is a case where git should detect the modify/delete conflict and
alert the user.  In this case, choice (J) is particularly bad.
Regardless of the content that D has for path, the merge conflict
won't even be detected and git would silently resolve this by
recording the content from D at path.  That's a fail for the first
goal.

Another choice that is likely to fail the first goal is (L).  If the
content at path D is the same as at A (the most likely case), then git
would not detect the conflict and would simply silently delete the
path.

In fact, any choice here for the merge base could cause the conflict
to go silently undetected (whenever the content at D matched our
choice for X), meaning that no choice for the merge base will be
perfect.  That would mean of the four choices, (M) would be best as it
would be least likely to match the choice made at D, but (K) would be
pretty reasonable.


So, as I view it: choice (J) is horrible in two of the three cases;
choice (M) is really bad in the second case for goal 2; and choice (L)
is likely to be bad in the third case for goal 1.  Choice (K) is
pretty reasonable in all three cases, and that choice is what this
patch implements.

Re: [PATCH 39/48] merge-recursive: Fix rename/rename(1to2) resolution for virtual merge base

[Elijah Newren]

On Mon, Jul 25, 2011 at 2:55 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Elijah Newren <newren@gmail.com> writes:
>> +             update_file(o, 0, mfi.sha, mfi.mode, src);
>> +             remove_file_from_cache(ren1_dst);
>> +             remove_file_from_cache(ren2_dst);
>
> ... the use of "src" here is taking a "middle ground" and punting on
> resolving the conflicting opinions of both branches for the outer merge to
> resolve. I think that is a sensible thing to do.
>
> The rename destinations of both branches are removed. What happens if
> ren1_dst (the path one branch wanted to rename src to) were added by the
> other branch (which wanted to rename src to ren2_dst), causing rename/add
> conflict between branches (i.e. not the one you worry about in the above
> FIXME which is about one branch renaming src to ren1_dst while adding an
> unrelated content to src)?

You found another bug; this time one that was already present in git
rather than having been introduced by my series (the
remove_file_from_cache() calls were already present, I just moved them
around a little bit).  I have a testcase that exposes this new bug
now.

Good catch.  :-)

Re: [PATCH 26/48] merge-recursive: Allow make_room_for_path() to remove D/F entries

[Johannes Sixt]

Am 8/8/2011 22:56, schrieb Elijah Newren:
> Hi,
> 
> On Wed, Jul 13, 2011 at 1:17 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:
>> Am 6/8/2011 9:30, schrieb Elijah Newren:
>>> +static int make_room_for_path(const struct merge_options *o, const char *path)
>>>  {
>>> -     int status;
>>> +     int status, i;
>>>       const char *msg = "failed to create path '%s'%s";
>>>
>>> +     /* Unlink any D/F conflict files that are in the way */
>>> +     for (i = 0; i < o->df_conflict_file_set.nr; i++) {
>>> +             const char *df_path = o->df_conflict_file_set.items[i].string;
>>> +             size_t pathlen = strlen(path);
>>> +             size_t df_pathlen = strlen(df_path);
>>> +             if (df_pathlen < pathlen && strncmp(path, df_path, df_pathlen) == 0) {
>>> +                     unlink(df_path);
>>> +                     break;
>>> +             }
>>> +     }
>>
>> Each time this loop is entered it tries to remove the same path again,
>> even if it does not exist anymore or was morphed into a directory in the
>> meantime. I suggest to remove a path from o->df_conflict_file_set after it
>> was unlinked. Or even better: have a separate "make room" phase somewhere
>> in the merge process.
> 
> Removing it from o->df_conflict_file_set makes sense.  However, there
> appears to be no API in string_list.h for deleting entries.  Are such
> operations discouraged?  I'm not sure whether to add such API, just
> hack it directly, or wait for someone else to come along and change
> this to a better data structure (such as a hash)...
> 
> I don't think it's possible to move this "make room" phase anywhere
> earlier in the merge process.  When we have D/F conflicts, the files
> of those D/F conflicts should only be removed if at least one of the
> paths under the corresponding directory are not removed by the merge
> process.  We don't know whether those paths will need to be removed
> until we call process_entry() on each of them, and from there we go
> right to this function when we find one that needs to stick around.
> So I simply don't see how to move it any earlier.

I missed that the loop is selective: It removes only a subset of the
paths. Without a deeper understanding of the merge machinery I'll just
accept what you explain. Removing the paths from the list that were
unlinked seems to be the right approach.

Here's an attempt for a delete_item API (note: only compile-tested).
Bike-shed painters welcome: delete_item, remove_item, free_item?

Signed-off-by: Johannes Sixt <j6t@kdbg.org>

diff --git a/Documentation/technical/api-string-list.txt b/Documentation/technical/api-string-list.txt
index 3f575bd..ce24eb9 100644
--- a/Documentation/technical/api-string-list.txt
+++ b/Documentation/technical/api-string-list.txt
@@ -29,6 +29,9 @@ member (you need this if you add things later) and you should set the
 
 . Can sort an unsorted list using `sort_string_list`.
 
+. Can remove individual items of an unsorted list using
+  `unsorted_string_list_delete_item`.
+
 . Finally it should free the list using `string_list_clear`.
 
 Example:
@@ -112,6 +115,13 @@ write `string_list_insert(...)->util = ...;`.
 The above two functions need to look through all items, as opposed to their
 counterpart for sorted lists, which performs a binary search.
 
+`unsorted_string_list_delete_item`::
+
+	Remove an item from a string_list. The `string` pointer of the items
+	will be freed in case the `strdup_strings` member of the string_list
+	is set. The third parameter controls if the `util` pointer of the
+	items should be freed or not.
+
 Data structures
 ---------------
 
diff --git a/string-list.c b/string-list.c
index 5168118..d9810ab 100644
--- a/string-list.c
+++ b/string-list.c
@@ -185,3 +185,12 @@ int unsorted_string_list_has_string(struct string_list *list,
 	return unsorted_string_list_lookup(list, string) != NULL;
 }
 
+void unsorted_string_list_delete_item(struct string_list *list, int i, int free_util)
+{
+	if (list->strdup_strings)
+		free(list->items[i].string);
+	if (free_util)
+		free(list->items[i].util);
+	list->items[i] = list->items[list->nr-1];
+	list->nr--;
+}
diff --git a/string-list.h b/string-list.h
index bda6983..0684cb7 100644
--- a/string-list.h
+++ b/string-list.h
@@ -44,4 +44,5 @@ void sort_string_list(struct string_list *list);
 int unsorted_string_list_has_string(struct string_list *list, const char *string);
 struct string_list_item *unsorted_string_list_lookup(struct string_list *list,
 						     const char *string);
+void unsorted_string_list_delete_item(struct string_list *list, int i, int free_util);
 #endif /* STRING_LIST_H */


> [...]
> 
> However, I don't see how any of this would address any failure you're
> seeing on windows.  Maybe one of my other changes, including one or
> two other bugfixes I've found will help?  I'll have to ping you when I
> submit the re-roll.

I stumbled over this only because on Windows we have an implementation of
unlink() that asks whether to retry a failed unlink attempt. I was using
an old version that asks when a directory was tried to unlink where a
retry is pointless.

-- Hannes