[PATCH 0/4] prevent 'checkout -m' from losing staged changes

[PATCH 0/4] prevent 'checkout -m' from losing staged changes

[Nguyễn Thái Ngọc Duy]

During the 'git switch' discusion, Phillip found out that staged changes
could be silently lost when doing 'git checkout -m'. The first attempt
[2] tries to fix it by warning and moving on, because t7201.10 would
fail if we simply die() when there are staged changes.

This round, I update read-tree to be used in that test instead of
'checkout -m'. Then we are able to die().

I start a new thread to keep spam level down for other people, and to
break free from 'git switch' thread because it's unrelated now.

Jonathan is included, just in case he still remembers something from
6a143aa2b2 (checkout -m: attempt merge when deletion of path was
staged, 2014-08-12)

[1] https://public-inbox.org/git/7d3742d6-73e4-2750-6ecb-9edf761d96dd@gmail.com/
[2] https://public-inbox.org/git/20190319093910.20229-1-pclouds@gmail.com/

Nguyễn Thái Ngọc Duy (4):
  unpack-trees: keep gently check inside add_rejected_path
  unpack-trees: rename "gently" flag to "quiet"
  read-tree: add --quiet
  checkout: prevent losing staged changes with --merge

 Documentation/git-read-tree.txt |  4 ++++
 builtin/checkout.c              | 13 +++++++++++--
 builtin/read-tree.c             |  1 +
 t/t7201-co.sh                   |  9 ++-------
 unpack-trees.c                  | 25 +++++++++++--------------
 unpack-trees.h                  |  2 +-
 6 files changed, 30 insertions(+), 24 deletions(-)

-- 
2.21.0.548.gd3c7d92dc2

[PATCH 1/4] unpack-trees: keep gently check inside add_rejected_path

[Nguyễn Thái Ngọc Duy]

This basically follows the footsteps of 6a143aa2b2 (checkout -m:
attempt merge when deletion of path was staged - 2014-08-12) where
there gently check is moved inside reject_merge() so that callers do
not accidentally forget it.

add_rejected_path() has the same usage pattern. All call sites check
gently first, then decide to call add_rejected_path() if needed. Move
the check inside.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 unpack-trees.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 22c41a3ba8..e6c1cc8302 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -219,6 +219,9 @@ static int add_rejected_path(struct unpack_trees_options *o,
 			     enum unpack_trees_error_types e,
 			     const char *path)
 {
+	if (o->gently)
+		return -1;
+
 	if (!o->show_all_errors)
 		return error(ERRORMSG(o, e), super_prefixed(path));
 
@@ -268,8 +271,7 @@ static int check_submodule_move_head(const struct cache_entry *ce,
 		flags |= SUBMODULE_MOVE_HEAD_FORCE;
 
 	if (submodule_move_head(ce->name, old_id, new_id, flags))
-		return o->gently ? -1 :
-				   add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
+		return add_rejected_path(o, ERROR_WOULD_LOSE_SUBMODULE, ce->name);
 	return 0;
 }
 
@@ -1645,8 +1647,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 static int reject_merge(const struct cache_entry *ce,
 			struct unpack_trees_options *o)
 {
-	return o->gently ? -1 :
-		add_rejected_path(o, ERROR_WOULD_OVERWRITE, ce->name);
+	return add_rejected_path(o, ERROR_WOULD_OVERWRITE, ce->name);
 }
 
 static int same(const struct cache_entry *a, const struct cache_entry *b)
@@ -1693,8 +1694,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 			int r = check_submodule_move_head(ce,
 				"HEAD", oid_to_hex(&ce->oid), o);
 			if (r)
-				return o->gently ? -1 :
-					add_rejected_path(o, error_type, ce->name);
+				return add_rejected_path(o, error_type, ce->name);
 			return 0;
 		}
 
@@ -1712,8 +1712,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 	}
 	if (errno == ENOENT)
 		return 0;
-	return o->gently ? -1 :
-		add_rejected_path(o, error_type, ce->name);
+	return add_rejected_path(o, error_type, ce->name);
 }
 
 int verify_uptodate(const struct cache_entry *ce,
@@ -1835,8 +1834,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,
 		d.exclude_per_dir = o->dir->exclude_per_dir;
 	i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);
 	if (i)
-		return o->gently ? -1 :
-			add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);
+		return add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);
 	free(pathbuf);
 	return cnt;
 }
@@ -1905,8 +1903,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,
 			return 0;
 	}
 
-	return o->gently ? -1 :
-		add_rejected_path(o, error_type, name);
+	return add_rejected_path(o, error_type, name);
 }
 
 /*
-- 
2.21.0.548.gd3c7d92dc2

[PATCH 2/4] unpack-trees: rename "gently" flag to "quiet"

[Nguyễn Thái Ngọc Duy]

The gently flag was added in 17e4642667 (Add flag to make unpack_trees()
not print errors. - 2008-02-07) to suppress error messages. The name
"gently" does not quite express that. Granted, being quiet is gentle but
it could mean not performing some other actions. Rename the flag to
"quiet" to be more on point.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/checkout.c | 2 +-
 unpack-trees.c     | 6 +++---
 unpack-trees.h     | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 0e6037b296..22fb6c0cae 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -700,7 +700,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
 		topts.initial_checkout = is_cache_unborn();
 		topts.update = 1;
 		topts.merge = 1;
-		topts.gently = opts->merge && old_branch_info->commit;
+		topts.quiet = opts->merge && old_branch_info->commit;
 		topts.verbose_update = opts->show_progress;
 		topts.fn = twoway_merge;
 		if (opts->overwrite_ignore) {
diff --git a/unpack-trees.c b/unpack-trees.c
index e6c1cc8302..2e5d7b202e 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -219,7 +219,7 @@ static int add_rejected_path(struct unpack_trees_options *o,
 			     enum unpack_trees_error_types e,
 			     const char *path)
 {
-	if (o->gently)
+	if (o->quiet)
 		return -1;
 
 	if (!o->show_all_errors)
@@ -1042,7 +1042,7 @@ static int unpack_nondirectories(int n, unsigned long mask,
 static int unpack_failed(struct unpack_trees_options *o, const char *message)
 {
 	discard_index(&o->result);
-	if (!o->gently && !o->exiting_early) {
+	if (!o->quiet && !o->exiting_early) {
 		if (message)
 			return error("%s", message);
 		return -1;
@@ -2343,7 +2343,7 @@ int bind_merge(const struct cache_entry * const *src,
 		return error("Cannot do a bind merge of %d trees",
 			     o->merge_size);
 	if (a && old)
-		return o->gently ? -1 :
+		return o->quiet ? -1 :
 			error(ERRORMSG(o, ERROR_BIND_OVERLAP),
 			      super_prefixed(a->name),
 			      super_prefixed(old->name));
diff --git a/unpack-trees.h b/unpack-trees.h
index 0135080a7b..d344d7d296 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -56,7 +56,7 @@ struct unpack_trees_options {
 		     diff_index_cached,
 		     debug_unpack,
 		     skip_sparse_checkout,
-		     gently,
+		     quiet,
 		     exiting_early,
 		     show_all_errors,
 		     dry_run;
-- 
2.21.0.548.gd3c7d92dc2

[PATCH 3/4] read-tree: add --quiet

[Nguyễn Thái Ngọc Duy]

read-tree is basically the front end of unpack-trees code and shoud
expose all of its functionality (unless it's designed for internal
use). This "opts.quiet" (formerly "opts.gently") was added for
builtin/checkout.c but there is no reason why other read-tree users
won't find this useful.

The test that is updated to run 'read-tree --quiet' was added because
unpack-trees was accidentally not being quiet [1] in 6a143aa2b2
(checkout -m: attempt merge when deletion of path was staged -
2014-08-12). Because checkout is the only "opts.quiet" user, there was
no other way to test quiet behavior. But we can now test it directly.

6a143aa2b2 was manually reverted to verify that read-tree --quiet
works correctly (i.e. test_must_be_empty fails).

[1] the commit message there say "errors out instead of performing a
    merge" but I'm pretty sure the "performing a merge" happens anyway
    even before that commit. That line should say "errors out
    _in addition to_ performing a merge"

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-read-tree.txt | 4 ++++
 builtin/read-tree.c             | 1 +
 t/t7201-co.sh                   | 3 +++
 3 files changed, 8 insertions(+)

diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt
index 5c70bc2878..1e81f9c4e6 100644
--- a/Documentation/git-read-tree.txt
+++ b/Documentation/git-read-tree.txt
@@ -128,6 +128,10 @@ OPTIONS
 	Instead of reading tree object(s) into the index, just empty
 	it.
 
+-q::
+--quiet::
+	Quiet, suppress feedback messages.
+
 <tree-ish#>::
 	The id of the tree object(s) to be read/merged.
 
diff --git a/builtin/read-tree.c b/builtin/read-tree.c
index 9083dcfa28..5c9c082595 100644
--- a/builtin/read-tree.c
+++ b/builtin/read-tree.c
@@ -154,6 +154,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 		{ OPTION_CALLBACK, 0, "recurse-submodules", NULL,
 			    "checkout", "control recursive updating of submodules",
 			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
+		OPT__QUIET(&opts.quiet, N_("suppress feedback messages")),
 		OPT_END()
 	};
 
diff --git a/t/t7201-co.sh b/t/t7201-co.sh
index 72b9b375ba..f165582019 100755
--- a/t/t7201-co.sh
+++ b/t/t7201-co.sh
@@ -223,6 +223,9 @@ test_expect_success 'switch to another branch while carrying a deletion' '
 	test_must_fail git checkout simple 2>errs &&
 	test_i18ngrep overwritten errs &&
 
+	test_must_fail git read-tree --quiet -m -u HEAD simple 2>errs &&
+	test_must_be_empty errs &&
+
 	git checkout --merge simple 2>errs &&
 	test_i18ngrep ! overwritten errs &&
 	git ls-files -u &&
-- 
2.21.0.548.gd3c7d92dc2

[PATCH 4/4] checkout: prevent losing staged changes with --merge

[Nguyễn Thái Ngọc Duy]

When --merge is specified, we may need to do a real merge (instead of
three-way tree unpacking), the steps are best seen in git-checkout.sh
version before it's removed:

    # Match the index to the working tree, and do a three-way.
    git diff-files --name-only | git update-index --remove --stdin &&
    work=`git write-tree` &&
    git read-tree $v --reset -u $new || exit

    git merge-recursive $old -- $new $work

    # Do not register the cleanly merged paths in the index yet.
    # this is not a real merge before committing, but just carrying
    # the working tree changes along.
    unmerged=`git ls-files -u`
    git read-tree $v --reset $new
    case "$unmerged" in
    '')     ;;
    *)
            (
                    z40=0000000000000000000000000000000000000000
                    echo "$unmerged" |
                    sed -e 's/^[0-7]* [0-9a-f]* /'"0 $z40 /"
                    echo "$unmerged"
            ) | git update-index --index-info
            ;;
    esac

Notice the last 'read-tree --reset' step. We restore worktree back to
'new' tree after worktree's messed up by merge-recursive. If there are
staged changes before this whole command sequence is executed, they
are lost because they are unlikely part of the 'new' tree to be
restored.

There is no easy way to fix this. Elijah may have something up his
sleeves [1], but until then, check if there are staged changes and
refuse to run and lose them. The user would need to do "git reset" to
continue in this case.

A note about the test update. 'checkout -m' in that test will fail
because a deletion is staged. This 'checkout -m' was previously needed
to verify quietness behavior of unpack-trees. But a different check
has been put in place in the last patch. We can safely drop
'checkout -m' now.

[1] CABPp-BFoL_U=bzON4SEMaQSKU2TKwnOgNqjt5MUaOejTKGUJxw@mail.gmail.com

Reported-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/checkout.c | 11 ++++++++++-
 t/t7201-co.sh      | 10 +---------
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 22fb6c0cae..7cd01f62be 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -725,7 +725,10 @@ static int merge_working_tree(const struct checkout_opts *opts,
 			 */
 			struct tree *result;
 			struct tree *work;
+			struct tree *old_tree;
 			struct merge_options o;
+			struct strbuf sb = STRBUF_INIT;
+
 			if (!opts->merge)
 				return 1;
 
@@ -735,6 +738,12 @@ static int merge_working_tree(const struct checkout_opts *opts,
 			 */
 			if (!old_branch_info->commit)
 				return 1;
+			old_tree = get_commit_tree(old_branch_info->commit);
+
+			if (repo_index_has_changes(the_repository, old_tree, &sb))
+				die(_("cannot continue with staged changes in "
+				      "the following files:\n%s"), sb.buf);
+			strbuf_release(&sb);
 
 			/* Do more real merge */
 
@@ -772,7 +781,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
 			ret = merge_trees(&o,
 					  get_commit_tree(new_branch_info->commit),
 					  work,
-					  get_commit_tree(old_branch_info->commit),
+					  old_tree,
 					  &result);
 			if (ret < 0)
 				exit(128);
diff --git a/t/t7201-co.sh b/t/t7201-co.sh
index f165582019..5990299fc9 100755
--- a/t/t7201-co.sh
+++ b/t/t7201-co.sh
@@ -224,15 +224,7 @@ test_expect_success 'switch to another branch while carrying a deletion' '
 	test_i18ngrep overwritten errs &&
 
 	test_must_fail git read-tree --quiet -m -u HEAD simple 2>errs &&
-	test_must_be_empty errs &&
-
-	git checkout --merge simple 2>errs &&
-	test_i18ngrep ! overwritten errs &&
-	git ls-files -u &&
-	test_must_fail git cat-file -t :0:two &&
-	test "$(git cat-file -t :1:two)" = blob &&
-	test "$(git cat-file -t :2:two)" = blob &&
-	test_must_fail git cat-file -t :3:two
+	test_must_be_empty errs
 '
 
 test_expect_success 'checkout to detach HEAD (with advice declined)' '
-- 
2.21.0.548.gd3c7d92dc2

Re: [PATCH 3/4] read-tree: add --quiet

[Philip Oakley]

On 22/03/2019 09:31, Nguyễn Thái Ngọc Duy wrote:
> read-tree is basically the front end of unpack-trees code and shoud

s/shoud/should/

Philip

> expose all of its functionality (unless it's designed for internal
> use). This "opts.quiet" (formerly "opts.gently") was added for
> builtin/checkout.c but there is no reason why other read-tree users
> won't find this useful.
>
> The test that is updated to run 'read-tree --quiet' was added because
> unpack-trees was accidentally not being quiet [1] in 6a143aa2b2
> (checkout -m: attempt merge when deletion of path was staged -
> 2014-08-12). Because checkout is the only "opts.quiet" user, there was
> no other way to test quiet behavior. But we can now test it directly.
>
> 6a143aa2b2 was manually reverted to verify that read-tree --quiet
> works correctly (i.e. test_must_be_empty fails).
>
> [1] the commit message there say "errors out instead of performing a
>      merge" but I'm pretty sure the "performing a merge" happens anyway
>      even before that commit. That line should say "errors out
>      _in addition to_ performing a merge"
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>   Documentation/git-read-tree.txt | 4 ++++
>   builtin/read-tree.c             | 1 +
>   t/t7201-co.sh                   | 3 +++
>   3 files changed, 8 insertions(+)
>
> diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt
> index 5c70bc2878..1e81f9c4e6 100644
> --- a/Documentation/git-read-tree.txt
> +++ b/Documentation/git-read-tree.txt
> @@ -128,6 +128,10 @@ OPTIONS
>   	Instead of reading tree object(s) into the index, just empty
>   	it.
>   
> +-q::
> +--quiet::
> +	Quiet, suppress feedback messages.
> +
>   <tree-ish#>::
>   	The id of the tree object(s) to be read/merged.
>   
> diff --git a/builtin/read-tree.c b/builtin/read-tree.c
> index 9083dcfa28..5c9c082595 100644
> --- a/builtin/read-tree.c
> +++ b/builtin/read-tree.c
> @@ -154,6 +154,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
>   		{ OPTION_CALLBACK, 0, "recurse-submodules", NULL,
>   			    "checkout", "control recursive updating of submodules",
>   			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
> +		OPT__QUIET(&opts.quiet, N_("suppress feedback messages")),
>   		OPT_END()
>   	};
>   
> diff --git a/t/t7201-co.sh b/t/t7201-co.sh
> index 72b9b375ba..f165582019 100755
> --- a/t/t7201-co.sh
> +++ b/t/t7201-co.sh
> @@ -223,6 +223,9 @@ test_expect_success 'switch to another branch while carrying a deletion' '
>   	test_must_fail git checkout simple 2>errs &&
>   	test_i18ngrep overwritten errs &&
>   
> +	test_must_fail git read-tree --quiet -m -u HEAD simple 2>errs &&
> +	test_must_be_empty errs &&
> +
>   	git checkout --merge simple 2>errs &&
>   	test_i18ngrep ! overwritten errs &&
>   	git ls-files -u &&

Re: [PATCH 4/4] checkout: prevent losing staged changes with --merge

[Elijah Newren]

On Fri, Mar 22, 2019 at 2:32 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>
> When --merge is specified, we may need to do a real merge (instead of
> three-way tree unpacking), the steps are best seen in git-checkout.sh
> version before it's removed:
>
>     # Match the index to the working tree, and do a three-way.
>     git diff-files --name-only | git update-index --remove --stdin &&
>     work=`git write-tree` &&
>     git read-tree $v --reset -u $new || exit
>
>     git merge-recursive $old -- $new $work
>
>     # Do not register the cleanly merged paths in the index yet.
>     # this is not a real merge before committing, but just carrying
>     # the working tree changes along.
>     unmerged=`git ls-files -u`
>     git read-tree $v --reset $new
>     case "$unmerged" in
>     '')     ;;
>     *)
>             (
>                     z40=0000000000000000000000000000000000000000
>                     echo "$unmerged" |
>                     sed -e 's/^[0-7]* [0-9a-f]* /'"0 $z40 /"
>                     echo "$unmerged"
>             ) | git update-index --index-info
>             ;;
>     esac
>
> Notice the last 'read-tree --reset' step. We restore worktree back to
> 'new' tree after worktree's messed up by merge-recursive. If there are
> staged changes before this whole command sequence is executed, they
> are lost because they are unlikely part of the 'new' tree to be
> restored.
>
> There is no easy way to fix this. Elijah may have something up his
> sleeves [1], but until then, check if there are staged changes and
> refuse to run and lose them. The user would need to do "git reset" to
> continue in this case.
>
> A note about the test update. 'checkout -m' in that test will fail
> because a deletion is staged. This 'checkout -m' was previously needed
> to verify quietness behavior of unpack-trees. But a different check
> has been put in place in the last patch. We can safely drop
> 'checkout -m' now.
>
> [1] CABPp-BFoL_U=bzON4SEMaQSKU2TKwnOgNqjt5MUaOejTKGUJxw@mail.gmail.com
>
> Reported-by: Phillip Wood <phillip.wood@dunelm.org.uk>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  builtin/checkout.c | 11 ++++++++++-
>  t/t7201-co.sh      | 10 +---------
>  2 files changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/builtin/checkout.c b/builtin/checkout.c
> index 22fb6c0cae..7cd01f62be 100644
> --- a/builtin/checkout.c
> +++ b/builtin/checkout.c
> @@ -725,7 +725,10 @@ static int merge_working_tree(const struct checkout_opts *opts,
>                          */
>                         struct tree *result;
>                         struct tree *work;
> +                       struct tree *old_tree;
>                         struct merge_options o;
> +                       struct strbuf sb = STRBUF_INIT;
> +
>                         if (!opts->merge)
>                                 return 1;
>
> @@ -735,6 +738,12 @@ static int merge_working_tree(const struct checkout_opts *opts,
>                          */
>                         if (!old_branch_info->commit)
>                                 return 1;
> +                       old_tree = get_commit_tree(old_branch_info->commit);
> +
> +                       if (repo_index_has_changes(the_repository, old_tree, &sb))
> +                               die(_("cannot continue with staged changes in "
> +                                     "the following files:\n%s"), sb.buf);
> +                       strbuf_release(&sb);
>
>                         /* Do more real merge */
>
> @@ -772,7 +781,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
>                         ret = merge_trees(&o,
>                                           get_commit_tree(new_branch_info->commit),
>                                           work,
> -                                         get_commit_tree(old_branch_info->commit),
> +                                         old_tree,
>                                           &result);
>                         if (ret < 0)
>                                 exit(128);
> diff --git a/t/t7201-co.sh b/t/t7201-co.sh
> index f165582019..5990299fc9 100755
> --- a/t/t7201-co.sh
> +++ b/t/t7201-co.sh
> @@ -224,15 +224,7 @@ test_expect_success 'switch to another branch while carrying a deletion' '
>         test_i18ngrep overwritten errs &&
>
>         test_must_fail git read-tree --quiet -m -u HEAD simple 2>errs &&
> -       test_must_be_empty errs &&
> -
> -       git checkout --merge simple 2>errs &&
> -       test_i18ngrep ! overwritten errs &&
> -       git ls-files -u &&
> -       test_must_fail git cat-file -t :0:two &&
> -       test "$(git cat-file -t :1:two)" = blob &&
> -       test "$(git cat-file -t :2:two)" = blob &&
> -       test_must_fail git cat-file -t :3:two
> +       test_must_be_empty errs
>  '

Ah, I see you avoid the other bug in checkout by just removing its
usage.  Seems fair enough; I can add a separate test to demonstrate
that bug when I get some time to work on it.  Hopefully I'll find a
quick fix too.