[PATCH 0/2] fix union merge with binary files

[PATCH 0/2] fix union merge with binary files

[Jeff King]

This started as an attempt to silence a "gcc -O3" warning. But I was
curious if we could trigger the problem it complains about in practice
(spoiler: we can), so I wrote a test. And it seems there was an even
bigger bug lurking, where we'd generate bogus merge results. :)

This fixes both bugs.

  [1/2]: ll_binary_merge(): handle XDL_MERGE_FAVOR_UNION
  [2/2]: ll_union_merge(): pass name labels to ll_xdl_merge()

 ll-merge.c            |  6 ++++--
 t/t6406-merge-attr.sh | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+), 2 deletions(-)

-Peff

[PATCH 1/2] ll_binary_merge(): handle XDL_MERGE_FAVOR_UNION

[Jeff King]

Prior to commit a944af1d86 (merge: teach -Xours/-Xtheirs to binary
ll-merge driver, 2012-09-08), we always reported a conflict from
ll_binary_merge() by returning "1" (in the xdl_merge and ll_merge code,
this value is the number of conflict hunks). After that commit, we
report zero conflicts if the "variant" flag is set, under the assumption
that it is one of XDL_MERGE_FAVOR_OURS or XDL_MERGE_FAVOR_THEIRS.

But this gets confused by XDL_MERGE_FAVOR_UNION. We do not know how to
do a binary union merge, but erroneously report no conflicts anyway (and
just blindly use the "ours" content as the result).

Let's tighten our check to just the cases that a944af1d86 meant to
cover. This fixes the union case (which existed already back when that
commit was made), as well as future-proofing us against any other
variants that get added later.

Note that you can't trigger this from "git merge-file --union", as that
bails on binary files before even calling into the ll-merge machinery.
The test here uses the "union" merge attribute, which does erroneously
report a successful merge.

Signed-off-by: Jeff King <peff@peff.net>
---
 ll-merge.c            |  4 +++-
 t/t6406-merge-attr.sh | 17 +++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/ll-merge.c b/ll-merge.c
index 9a8a2c365c..145deb12fa 100644
--- a/ll-merge.c
+++ b/ll-merge.c
@@ -91,7 +91,9 @@ static int ll_binary_merge(const struct ll_merge_driver *drv_unused,
 	 * With -Xtheirs or -Xours, we have cleanly merged;
 	 * otherwise we got a conflict.
 	 */
-	return (opts->variant ? 0 : 1);
+	return opts->variant == XDL_MERGE_FAVOR_OURS ||
+	       opts->variant == XDL_MERGE_FAVOR_THEIRS ?
+	       0 : 1;
 }
 
 static int ll_xdl_merge(const struct ll_merge_driver *drv_unused,
diff --git a/t/t6406-merge-attr.sh b/t/t6406-merge-attr.sh
index d5a4ac2d81..c1c458d933 100755
--- a/t/t6406-merge-attr.sh
+++ b/t/t6406-merge-attr.sh
@@ -207,4 +207,21 @@ test_expect_success 'custom merge does not lock index' '
 	git merge main
 '
 
+test_expect_success 'binary files with union attribute' '
+	git checkout -b bin-main &&
+	printf "base\0" >bin.txt &&
+	echo "bin.txt merge=union" >.gitattributes &&
+	git add bin.txt .gitattributes &&
+	git commit -m base &&
+
+	printf "one\0" >bin.txt &&
+	git commit -am one &&
+
+	git checkout -b bin-side HEAD^ &&
+	printf "two\0" >bin.txt &&
+	git commit -am two &&
+
+	test_must_fail git merge bin-main
+'
+
 test_done
-- 
2.32.0.529.g079a794268

Re: [PATCH 1/2] ll_binary_merge(): handle XDL_MERGE_FAVOR_UNION

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Prior to commit a944af1d86 (merge: teach -Xours/-Xtheirs to binary
> ll-merge driver, 2012-09-08), we always reported a conflict from
> ll_binary_merge() by returning "1" (in the xdl_merge and ll_merge code,
> this value is the number of conflict hunks). After that commit, we
> report zero conflicts if the "variant" flag is set, under the assumption
> that it is one of XDL_MERGE_FAVOR_OURS or XDL_MERGE_FAVOR_THEIRS.
>
> But this gets confused by XDL_MERGE_FAVOR_UNION. We do not know how to
> do a binary union merge, but erroneously report no conflicts anyway (and
> just blindly use the "ours" content as the result).
>
> Let's tighten our check to just the cases that a944af1d86 meant to
> cover. This fixes the union case (which existed already back when that
> commit was made), as well as future-proofing us against any other
> variants that get added later.

Makes sense.

> Note that you can't trigger this from "git merge-file --union", as that
> bails on binary files before even calling into the ll-merge machinery.
> The test here uses the "union" merge attribute, which does erroneously
> report a successful merge.

Nice discovery.

Thanks.

[PATCH 2/2] ll_union_merge(): pass name labels to ll_xdl_merge()

[Jeff King]

Since cd1d61c44f (make union merge an xdl merge favor, 2010-03-01), we
pass NULL to ll_xdl_merge() for the "name" labels of the ancestor, ours
and theirs buffers. We usually use these for annotating conflict markers
left in a file. For a union merge, these shouldn't matter; the point of
it is that we'd never leave conflict markers in the first place.

But there is one code path where we may dereference them: if the file
contents appear to be binary, ll_binary_merge() will give up and pass
them to warning() to generate a message for the user (that was true even
when cd1d61c44f was written, though the warning was in ll_xdl_merge()
back then).

That can result in a segfault, though on many systems (including glibc),
the printf routines will helpfully just say "(null)" instead. We can
extend our binary-union test in t6406 to check stderr, which catches the
problem on all systems.

This also fixes a warning from "gcc -O3". Unlike lower optimization
levels, it inlines enough to see that the NULL can make it to warning()
and complains:

  In function ‘ll_binary_merge’,
      inlined from ‘ll_xdl_merge’ at ll-merge.c:115:10,
      inlined from ‘ll_union_merge’ at ll-merge.c:151:9:
  ll-merge.c:74:4: warning: ‘%s’ directive argument is null [-Wformat-overflow=]
     74 |    warning("Cannot merge binary files: %s (%s vs. %s)",
        |    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     75 |     path, name1, name2);
        |     ~~~~~~~~~~~~~~~~~~~

Signed-off-by: Jeff King <peff@peff.net>
---
 ll-merge.c            | 2 +-
 t/t6406-merge-attr.sh | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/ll-merge.c b/ll-merge.c
index 145deb12fa..0ee34d8a01 100644
--- a/ll-merge.c
+++ b/ll-merge.c
@@ -151,7 +151,7 @@ static int ll_union_merge(const struct ll_merge_driver *drv_unused,
 	o = *opts;
 	o.variant = XDL_MERGE_FAVOR_UNION;
 	return ll_xdl_merge(drv_unused, result, path_unused,
-			    orig, NULL, src1, NULL, src2, NULL,
+			    orig, orig_name, src1, name1, src2, name2,
 			    &o, marker_size);
 }
 
diff --git a/t/t6406-merge-attr.sh b/t/t6406-merge-attr.sh
index c1c458d933..8494645837 100755
--- a/t/t6406-merge-attr.sh
+++ b/t/t6406-merge-attr.sh
@@ -221,7 +221,8 @@ test_expect_success 'binary files with union attribute' '
 	printf "two\0" >bin.txt &&
 	git commit -am two &&
 
-	test_must_fail git merge bin-main
+	test_must_fail git merge bin-main 2>stderr &&
+	grep -i "warning.*cannot merge.*HEAD vs. bin-main" stderr
 '
 
 test_done
-- 
2.32.0.529.g079a794268

Re: [PATCH 2/2] ll_union_merge(): pass name labels to ll_xdl_merge()

[Elijah Newren]

On Thu, Jun 10, 2021 at 6:00 AM Jeff King <peff@peff.net> wrote:
>
> Since cd1d61c44f (make union merge an xdl merge favor, 2010-03-01), we
> pass NULL to ll_xdl_merge() for the "name" labels of the ancestor, ours
> and theirs buffers. We usually use these for annotating conflict markers
> left in a file. For a union merge, these shouldn't matter; the point of
> it is that we'd never leave conflict markers in the first place.
>
> But there is one code path where we may dereference them: if the file
> contents appear to be binary, ll_binary_merge() will give up and pass
> them to warning() to generate a message for the user (that was true even
> when cd1d61c44f was written, though the warning was in ll_xdl_merge()
> back then).
>
> That can result in a segfault, though on many systems (including glibc),
> the printf routines will helpfully just say "(null)" instead. We can
> extend our binary-union test in t6406 to check stderr, which catches the
> problem on all systems.

Nice catch (as is your 1/2 as well).

> This also fixes a warning from "gcc -O3". Unlike lower optimization
> levels, it inlines enough to see that the NULL can make it to warning()
> and complains:
>
>   In function ‘ll_binary_merge’,
>       inlined from ‘ll_xdl_merge’ at ll-merge.c:115:10,
>       inlined from ‘ll_union_merge’ at ll-merge.c:151:9:
>   ll-merge.c:74:4: warning: ‘%s’ directive argument is null [-Wformat-overflow=]
>      74 |    warning("Cannot merge binary files: %s (%s vs. %s)",
>         |    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>      75 |     path, name1, name2);
>         |     ~~~~~~~~~~~~~~~~~~~

So the warning uses path as well as name1 and name2...

>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  ll-merge.c            | 2 +-
>  t/t6406-merge-attr.sh | 3 ++-
>  2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/ll-merge.c b/ll-merge.c
> index 145deb12fa..0ee34d8a01 100644
> --- a/ll-merge.c
> +++ b/ll-merge.c
> @@ -151,7 +151,7 @@ static int ll_union_merge(const struct ll_merge_driver *drv_unused,
>         o = *opts;
>         o.variant = XDL_MERGE_FAVOR_UNION;
>         return ll_xdl_merge(drv_unused, result, path_unused,

Should we also rename 'path_unused' to 'path', since it is actually used?

> -                           orig, NULL, src1, NULL, src2, NULL,
> +                           orig, orig_name, src1, name1, src2, name2,
>                             &o, marker_size);
>  }
>
> diff --git a/t/t6406-merge-attr.sh b/t/t6406-merge-attr.sh
> index c1c458d933..8494645837 100755
> --- a/t/t6406-merge-attr.sh
> +++ b/t/t6406-merge-attr.sh
> @@ -221,7 +221,8 @@ test_expect_success 'binary files with union attribute' '
>         printf "two\0" >bin.txt &&
>         git commit -am two &&
>
> -       test_must_fail git merge bin-main
> +       test_must_fail git merge bin-main 2>stderr &&
> +       grep -i "warning.*cannot merge.*HEAD vs. bin-main" stderr
>  '
>
>  test_done
> --
> 2.32.0.529.g079a794268

This patch has a minor textual conflict with my remerge-diff series,
but since I haven't submitted it yet, that's my problem rather than
yours...and it will be an easy fix anyway.

Anyway, good catches.  Other than maybe considering fixing the name of
'path_unused', this looks good to me.

[PATCH 3/2] ll_union_merge(): rename path_unused parameter

[Jeff King]

On Thu, Jun 10, 2021 at 08:19:02AM -0700, Elijah Newren wrote:

> > Signed-off-by: Jeff King <peff@peff.net>
> > ---
> >  ll-merge.c            | 2 +-
> >  t/t6406-merge-attr.sh | 3 ++-
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/ll-merge.c b/ll-merge.c
> > index 145deb12fa..0ee34d8a01 100644
> > --- a/ll-merge.c
> > +++ b/ll-merge.c
> > @@ -151,7 +151,7 @@ static int ll_union_merge(const struct ll_merge_driver *drv_unused,
> >         o = *opts;
> >         o.variant = XDL_MERGE_FAVOR_UNION;
> >         return ll_xdl_merge(drv_unused, result, path_unused,
> 
> Should we also rename 'path_unused' to 'path', since it is actually used?

That seems reasonable, but I'd prefer to do it as a separate patch. So
here that is.

Ironically, orig_path was unused both before this patch (where we threw
it away and passed NULL instead) and after (where we pass it into the
xdl merge, but the union merge should ignore it completely). But I
prefer not to go too wild with these kind of annotations, as they can
easily become inaccurate or out of date. If we're passing it, then we
shouldn't make too many assumptions about what xdl_merge() is doing
under the hood.

So we could also rename drv_unused mentioned below, but I didn't here.

-- >8 --
Subject: [PATCH] ll_union_merge(): rename path_unused parameter

The "path" parameter to ll_union_merge() is named "path_unused", since
we don't ourselves use it. But we do pass it to ll_xdl_merge(), which
may look at it (it gets passed to ll_binary_merge(), which may pass it
to warning()). Let's rename it to correct this inaccuracy (both of the
other functions correctly do not call this "unused").

Note that we also pass drv_unused, but it truly is unused by the rest of
the stack (it only exists at all to provide a generic interface that
matches what ll_ext_merge() needs).

Reported-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
---
 ll-merge.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ll-merge.c b/ll-merge.c
index 0ee34d8a01..261657578c 100644
--- a/ll-merge.c
+++ b/ll-merge.c
@@ -138,7 +138,7 @@ static int ll_xdl_merge(const struct ll_merge_driver *drv_unused,
 
 static int ll_union_merge(const struct ll_merge_driver *drv_unused,
 			  mmbuffer_t *result,
-			  const char *path_unused,
+			  const char *path,
 			  mmfile_t *orig, const char *orig_name,
 			  mmfile_t *src1, const char *name1,
 			  mmfile_t *src2, const char *name2,
@@ -150,7 +150,7 @@ static int ll_union_merge(const struct ll_merge_driver *drv_unused,
 	assert(opts);
 	o = *opts;
 	o.variant = XDL_MERGE_FAVOR_UNION;
-	return ll_xdl_merge(drv_unused, result, path_unused,
+	return ll_xdl_merge(drv_unused, result, path,
 			    orig, orig_name, src1, name1, src2, name2,
 			    &o, marker_size);
 }
-- 
2.32.0.529.g079a794268

Re: [PATCH 0/2] fix union merge with binary files

[Elijah Newren]

On Thu, Jun 10, 2021 at 5:54 AM Jeff King <peff@peff.net> wrote:
>
> This started as an attempt to silence a "gcc -O3" warning. But I was
> curious if we could trigger the problem it complains about in practice
> (spoiler: we can), so I wrote a test. And it seems there was an even
> bigger bug lurking, where we'd generate bogus merge results. :)
>
> This fixes both bugs.

Nice catches, and fixes.  I had a minor comment on 2/2, but with or
without fixing up the 'path_unused' variable name both patches are:

Reviewed-by: Elijah Newren <newren@gmail.com>