[PATCH 0/8] introduce no-overlay and cached mode in git checkout

[PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Thomas Gummerer]

Here's the series I mentioned a couple of times on the list already,
introducing a no-overlay mode in 'git checkout'.  The inspiration for
this came from Junios message in [*1*].

Basically the idea is to also delete files when the match <pathspec>
in 'git checkout <tree-ish> -- <pathspec>' in the current tree, but
don't match <pathspec> in <tree-ish>.  The rest of the cases are
already properly taken care of by 'git checkout'.

The final step in the series is to actually make use of this in 'git
stash', which simplifies the code there a bit.  I am however happy to
hold off on this step until the stash-in-C series is merged, so we
don't delay that further.

In addition to the no-overlay mode, we also add a --cached mode, which
works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.

Actually deprecating 'git reset <tree-ish> -- <pathspec>' should come
later, probably not before Duy's restore-files command lands, as 'git
checkout --no-overlay <tree-ish> -- <pathspec>' is a bit cumbersome to
type compared to 'git reset <tree-ish> -- <pathspec>'.

My hope is also that the no-overlay mode could become the new default
in the restore-files command Duy is currently working on.

No documentation yet, as I wanted to get this out for review first.
I'm not familiar with most of the code I touched here, so there may
well be much better ways to implement some of this, that I wasn't able
to figure out.  I'd be very happy with some feedback around that.

Another thing I'm not sure about is how to deal with conflicts.  In
the cached mode this patch series is not dealing with it at all, as
'git checkout -- <pathspec>' when pathspec matches a file with
conflicts doesn't update the index.  For the no-overlay mode, the file
is removed if the corresponding stage is not found in the index.  I'm
however not sure this is the right thing to do in all cases?

*1*: <xmqq4loqplou.fsf@gitster.mtv.corp.google.com>

Thomas Gummerer (8):
  move worktree tests to t24*
  entry: factor out unlink_entry function
  entry: support CE_WT_REMOVE flag in checkout_entry
  read-cache: add invalidate parameter to remove_marked_cache_entries
  checkout: introduce --{,no-}overlay option
  checkout: add --cached option
  checkout: add allow ignoring unmatched pathspec
  stash: use git checkout --index

 builtin/checkout.c                            |  66 +++++++++--
 cache.h                                       |   7 +-
 entry.c                                       |  22 ++++
 git-stash.sh                                  |  12 +-
 read-cache.c                                  |   8 +-
 split-index.c                                 |   2 +-
 t/t2016-checkout-patch.sh                     |   8 ++
 t/t2022-checkout-paths.sh                     |   9 ++
 t/t2025-checkout-no-overlay.sh                |  31 ++++++
 t/t2026-checkout-cached.sh                    | 103 ++++++++++++++++++
 ...-worktree-add.sh => t2400-worktree-add.sh} |   0
 ...ktree-prune.sh => t2401-worktree-prune.sh} |   0
 ...orktree-list.sh => t2402-worktree-list.sh} |   0
 t/t9902-completion.sh                         |   3 +
 unpack-trees.c                                |  21 +---
 15 files changed, 251 insertions(+), 41 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh
 create mode 100755 t/t2026-checkout-cached.sh
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)

-- 
2.20.0.rc2.411.g8f28e744c2

[PATCH 1/8] move worktree tests to t24*

[Thomas Gummerer]

The 'git worktree' command used to be just another mode in 'git
checkout', namely 'git checkout --to'.  When the tests for the latter
were retrofitted for the former, the test name was adjusted, but the
test number was kept, even though the test is testing a different
command now.  t/README states: "Second digit tells the particular
command we are testing.", so 'git worktree' should have a separate
number just for itself.

Move the worktree tests to t24* to adhere to that guideline. We're
going to make use of the free'd up numbers in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 t/{t2025-worktree-add.sh => t2400-worktree-add.sh}     | 0
 t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} | 0
 t/{t2027-worktree-list.sh => t2402-worktree-list.sh}   | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)

diff --git a/t/t2025-worktree-add.sh b/t/t2400-worktree-add.sh
similarity index 100%
rename from t/t2025-worktree-add.sh
rename to t/t2400-worktree-add.sh
diff --git a/t/t2026-worktree-prune.sh b/t/t2401-worktree-prune.sh
similarity index 100%
rename from t/t2026-worktree-prune.sh
rename to t/t2401-worktree-prune.sh
diff --git a/t/t2027-worktree-list.sh b/t/t2402-worktree-list.sh
similarity index 100%
rename from t/t2027-worktree-list.sh
rename to t/t2402-worktree-list.sh
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 2/8] entry: factor out unlink_entry function

[Thomas Gummerer]

Factor out the 'unlink_entry()' function from unpack-trees.c to
entry.c.  It will be used in other places as well in subsequent
steps.

As it's no longer a static function, also move the documentation to
the header file to make it more discoverable.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 cache.h        |  5 +++++
 entry.c        | 15 +++++++++++++++
 unpack-trees.c | 19 -------------------
 3 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/cache.h b/cache.h
index ca36b44ee0..c1c953e810 100644
--- a/cache.h
+++ b/cache.h
@@ -1542,6 +1542,11 @@ struct checkout {
 extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
 extern void enable_delayed_checkout(struct checkout *state);
 extern int finish_delayed_checkout(struct checkout *state);
+/*
+ * Unlink the last component and schedule the leading directories for
+ * removal, such that empty directories get removed.
+ */
+extern void unlink_entry(const struct cache_entry *ce);
 
 struct cache_def {
 	struct strbuf path;
diff --git a/entry.c b/entry.c
index 5d136c5d55..3ec148ceee 100644
--- a/entry.c
+++ b/entry.c
@@ -508,3 +508,18 @@ int checkout_entry(struct cache_entry *ce,
 	create_directories(path.buf, path.len, state);
 	return write_entry(ce, path.buf, state, 0);
 }
+
+void unlink_entry(const struct cache_entry *ce)
+{
+	const struct submodule *sub = submodule_from_ce(ce);
+	if (sub) {
+		/* state.force is set at the caller. */
+		submodule_move_head(ce->name, "HEAD", NULL,
+				    SUBMODULE_MOVE_HEAD_FORCE);
+	}
+	if (!check_leading_path(ce->name, ce_namelen(ce)))
+		return;
+	if (remove_or_warn(ce->ce_mode, ce->name))
+		return;
+	schedule_dir_for_removal(ce->name, ce_namelen(ce));
+}
diff --git a/unpack-trees.c b/unpack-trees.c
index 7570df481b..e8d1a6ac50 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -300,25 +300,6 @@ static void load_gitmodules_file(struct index_state *index,
 	}
 }
 
-/*
- * Unlink the last component and schedule the leading directories for
- * removal, such that empty directories get removed.
- */
-static void unlink_entry(const struct cache_entry *ce)
-{
-	const struct submodule *sub = submodule_from_ce(ce);
-	if (sub) {
-		/* state.force is set at the caller. */
-		submodule_move_head(ce->name, "HEAD", NULL,
-				    SUBMODULE_MOVE_HEAD_FORCE);
-	}
-	if (!check_leading_path(ce->name, ce_namelen(ce)))
-		return;
-	if (remove_or_warn(ce->ce_mode, ce->name))
-		return;
-	schedule_dir_for_removal(ce->name, ce_namelen(ce));
-}
-
 static struct progress *get_progress(struct unpack_trees_options *o)
 {
 	unsigned cnt = 0, total = 0;
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Thomas Gummerer]

'checkout_entry()' currently only supports creating new entries in the
working tree, but not deleting them.  Add the ability to remove
entries at the same time if the entry is marked with the CE_WT_REMOVE
flag.

Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
only used in unpack-tree, however we will make use of this in a
subsequent step in the series.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 entry.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/entry.c b/entry.c
index 3ec148ceee..cd1c6601b6 100644
--- a/entry.c
+++ b/entry.c
@@ -441,6 +441,13 @@ int checkout_entry(struct cache_entry *ce,
 	static struct strbuf path = STRBUF_INIT;
 	struct stat st;
 
+	if (ce->ce_flags & CE_WT_REMOVE) {
+		if (topath)
+			BUG("Can't remove entry to a path");
+		unlink_entry(ce);
+		return 0;
+	}
+
 	if (topath)
 		return write_entry(ce, topath, state, 1);
 
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Thomas Gummerer]

When marking cache entries for removal, and later removing them all at
once using 'remove_marked_cache_entries()', cache entries currently
have to be invalidated manually in the cache tree and in the untracked
cache.

Add an invalidate flag to the function.  With the flag set, the
function will take care of invalidating the path in the cache tree and
in the untracked cache.

This will be useful in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---

For the two current callsites, unpack-trees seems to do this
invalidation itself internally.  I don't quite understand why we don't
need it in split-index mode though.  I assume it's because the cache
tree in the main index would already have been invalidated?  I didn't
have much time to dig, but couldn't produce any failures with it
either, so I assume not invalidating paths is the right thing to do
here.

 cache.h        | 2 +-
 read-cache.c   | 8 +++++++-
 split-index.c  | 2 +-
 unpack-trees.c | 2 +-
 4 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index c1c953e810..1deee48f5b 100644
--- a/cache.h
+++ b/cache.h
@@ -751,7 +751,7 @@ extern void rename_index_entry_at(struct index_state *, int pos, const char *new
 /* Remove entry, return true if there are more entries to go. */
 extern int remove_index_entry_at(struct index_state *, int pos);
 
-extern void remove_marked_cache_entries(struct index_state *istate);
+extern void remove_marked_cache_entries(struct index_state *istate, int invalidate);
 extern int remove_file_from_index(struct index_state *, const char *path);
 #define ADD_CACHE_VERBOSE 1
 #define ADD_CACHE_PRETEND 2
diff --git a/read-cache.c b/read-cache.c
index 4ca81286c0..d86a06acba 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -590,13 +590,19 @@ int remove_index_entry_at(struct index_state *istate, int pos)
  * CE_REMOVE is set in ce_flags.  This is much more effective than
  * calling remove_index_entry_at() for each entry to be removed.
  */
-void remove_marked_cache_entries(struct index_state *istate)
+void remove_marked_cache_entries(struct index_state *istate, int invalidate)
 {
 	struct cache_entry **ce_array = istate->cache;
 	unsigned int i, j;
 
 	for (i = j = 0; i < istate->cache_nr; i++) {
 		if (ce_array[i]->ce_flags & CE_REMOVE) {
+			if (invalidate) {
+				cache_tree_invalidate_path(istate,
+							   ce_array[i]->name);
+				untracked_cache_remove_from_index(istate,
+								  ce_array[i]->name);
+			}
 			remove_name_hash(istate, ce_array[i]);
 			save_or_free_index_entry(istate, ce_array[i]);
 		}
diff --git a/split-index.c b/split-index.c
index 5820412dc5..8aebc3661b 100644
--- a/split-index.c
+++ b/split-index.c
@@ -162,7 +162,7 @@ void merge_base_index(struct index_state *istate)
 	ewah_each_bit(si->replace_bitmap, replace_entry, istate);
 	ewah_each_bit(si->delete_bitmap, mark_entry_for_delete, istate);
 	if (si->nr_deletions)
-		remove_marked_cache_entries(istate);
+		remove_marked_cache_entries(istate, 0);
 
 	for (i = si->nr_replacements; i < si->saved_cache_nr; i++) {
 		if (!ce_namelen(si->saved_cache[i]))
diff --git a/unpack-trees.c b/unpack-trees.c
index e8d1a6ac50..8e6afa924d 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -392,7 +392,7 @@ static int check_updates(struct unpack_trees_options *o)
 				unlink_entry(ce);
 		}
 	}
-	remove_marked_cache_entries(index);
+	remove_marked_cache_entries(index, 0);
 	remove_scheduled_dirs();
 
 	if (should_update_submodules() && o->update && !o->dry_run)
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 5/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

Currently 'git checkout' is defined as an overlay operation, which
means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
entry in the index that matches <pathspec>, but that doesn't exist in
<tree-ish>, that entry will not be removed from the index or the
working tree.

Introduce a new --{,no-}overlay option, which allows using 'git
checkout' in non-overlay mode, thus removing files from the working
tree if they do not exist in <tree-ish> but match <pathspec>.

Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
this way, so no changes are needed for the patch mode.  We disallow
'git checkout --overlay -p' to avoid confusing users who would expect
to be able to force overlay mode in 'git checkout -p' this way.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c             | 64 +++++++++++++++++++++++++++-------
 t/t2025-checkout-no-overlay.sh | 47 +++++++++++++++++++++++++
 t/t9902-completion.sh          |  1 +
 3 files changed, 99 insertions(+), 13 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh

diff --git a/builtin/checkout.c b/builtin/checkout.c
index acdafc6e4c..0aef35bbc4 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -44,6 +44,7 @@ struct checkout_opts {
 	int ignore_skipworktree;
 	int ignore_other_worktrees;
 	int show_progress;
+	int overlay_mode;
 	/*
 	 * If new checkout options are added, skip_merge_working_tree
 	 * should be updated accordingly.
@@ -132,7 +133,8 @@ static int skip_same_name(const struct cache_entry *ce, int pos)
 	return pos;
 }
 
-static int check_stage(int stage, const struct cache_entry *ce, int pos)
+static int check_stage(int stage, const struct cache_entry *ce, int pos,
+		       int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -140,6 +142,8 @@ static int check_stage(int stage, const struct cache_entry *ce, int pos)
 			return 0;
 		pos++;
 	}
+	if (!overlay_mode)
+		return 0;
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -165,7 +169,7 @@ static int check_stages(unsigned stages, const struct cache_entry *ce, int pos)
 }
 
 static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
-			  const struct checkout *state)
+			  const struct checkout *state, int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -173,6 +177,10 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
 			return checkout_entry(active_cache[pos], state, NULL);
 		pos++;
 	}
+	if (!overlay_mode) {
+		unlink_entry(ce);
+		return 0;
+	}
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -302,15 +310,29 @@ static int checkout_paths(const struct checkout_opts *opts,
 		ce->ce_flags &= ~CE_MATCHED;
 		if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
 			continue;
-		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
-			/*
-			 * "git checkout tree-ish -- path", but this entry
-			 * is in the original index; it will not be checked
-			 * out to the working tree and it does not matter
-			 * if pathspec matched this entry.  We will not do
-			 * anything to this entry at all.
-			 */
-			continue;
+		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE)) {
+			if (!opts->overlay_mode &&
+			    ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
+				/*
+				 * "git checkout --no-overlay <tree-ish> -- path",
+				 * and the path is not in tree-ish, but is in
+				 * the current index, which means that it should 
+				 * be removed.
+				 */
+				ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
+				continue;
+			} else {
+				/*
+				 * "git checkout tree-ish -- path", but this
+				 * entry is in the original index; it will not
+				 * be checked out to the working tree and it
+				 * does not matter if pathspec matched this
+				 * entry.  We will not do anything to this entry
+				 * at all.
+				 */
+				continue;
+			}
+		}
 		/*
 		 * Either this entry came from the tree-ish we are
 		 * checking the paths out of, or we are checking out
@@ -348,7 +370,7 @@ static int checkout_paths(const struct checkout_opts *opts,
 			if (opts->force) {
 				warning(_("path '%s' is unmerged"), ce->name);
 			} else if (opts->writeout_stage) {
-				errs |= check_stage(opts->writeout_stage, ce, pos);
+				errs |= check_stage(opts->writeout_stage, ce, pos, opts->overlay_mode);
 			} else if (opts->merge) {
 				errs |= check_stages((1<<2) | (1<<3), ce, pos);
 			} else {
@@ -375,12 +397,14 @@ static int checkout_paths(const struct checkout_opts *opts,
 				continue;
 			}
 			if (opts->writeout_stage)
-				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state);
+				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state, opts->overlay_mode);
 			else if (opts->merge)
 				errs |= checkout_merged(pos, &state);
 			pos = skip_same_name(ce, pos) - 1;
 		}
 	}
+	remove_marked_cache_entries(&the_index, 1);
+	remove_scheduled_dirs();
 	errs |= finish_delayed_checkout(&state);
 
 	if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))
@@ -542,6 +566,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
 	 * opts->show_progress only impacts output so doesn't require a merge
 	 */
 
+	/*
+	 * opts->overlay_mode cannot be used with switching branches so is
+	 * not tested here
+	 */
+
 	/*
 	 * If we aren't creating a new branch any changes or updates will
 	 * happen in the existing branch.  Since that could only be updating
@@ -1178,6 +1207,10 @@ static int checkout_branch(struct checkout_opts *opts,
 		die(_("'%s' cannot be used with switching branches"),
 		    "--patch");
 
+	if (!opts->overlay_mode)
+		die(_("'%s' cannot be used with switching branches"),
+		    "--no-overlay");
+
 	if (opts->writeout_stage)
 		die(_("'%s' cannot be used with switching branches"),
 		    "--ours/--theirs");
@@ -1266,6 +1299,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 			    "checkout", "control recursive updating of submodules",
 			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
+		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
 		OPT_END(),
 	};
 
@@ -1274,6 +1308,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	opts.overwrite_ignore = 1;
 	opts.prefix = prefix;
 	opts.show_progress = -1;
+	opts.overlay_mode = -1;
 
 	git_config(git_checkout_config, &opts);
 
@@ -1297,6 +1332,9 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	if ((!!opts.new_branch + !!opts.new_branch_force + !!opts.new_orphan_branch) > 1)
 		die(_("-b, -B and --orphan are mutually exclusive"));
 
+	if (opts.overlay_mode == 1 && opts.patch_mode)
+		die(_("-p and --overlay are mutually exclusive"));
+
 	/*
 	 * From here on, new_branch will contain the branch to be checked out,
 	 * and new_branch_force and new_orphan_branch will tell us which one of
diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
new file mode 100755
index 0000000000..3575321382
--- /dev/null
+++ b/t/t2025-checkout-no-overlay.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+
+test_description='checkout --no-overlay <tree-ish> -- <pathspec>'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git commit --allow-empty -m "initial"
+'
+
+test_expect_success 'checkout --no-overlay deletes files not in <tree>' '
+	>file &&
+	mkdir dir &&
+	>dir/file1 &&
+	git add file dir/file1 &&
+	git checkout --no-overlay HEAD -- file &&
+	test_path_is_missing file &&
+	test_path_is_file dir/file1
+'
+
+test_expect_success 'checkout --no-overlay removing last file from directory' '
+	git checkout --no-overlay HEAD -- dir/file1 &&
+	test_path_is_missing dir
+'
+
+test_expect_success 'checkout -p --overlay is disallowed' '
+	test_must_fail git checkout -p --overlay HEAD 2>actual &&
+	test_i18ngrep "fatal: -p and --overlay are mutually exclusive" actual
+'
+
+test_expect_success '--no-overlay --theirs with M/D conflict deletes file' '
+	test_commit file1 file1 &&
+	test_commit file2 file2 &&
+	git rm --cached file1 &&
+	echo 1234 >file1 &&
+	F1=$(git rev-parse HEAD:file1) &&
+	F2=$(git rev-parse HEAD:file2) &&
+	{
+		echo "100644 $F1 1	file1" &&
+		echo "100644 $F2 2	file1"
+	} | git update-index --index-info &&
+	test_path_is_file file1 &&
+	git checkout --theirs --no-overlay -- file1 &&
+	test_path_is_missing file1
+'
+
+test_done
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index 175f83d704..a3fd9a9630 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -1436,6 +1436,7 @@ test_expect_success 'double dash "git checkout"' '
 	--progress Z
 	--no-quiet Z
 	--no-... Z
+	--overlay Z
 	EOF
 '
 
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 6/8] checkout: add --cached option

[Thomas Gummerer]

Add a new --cached option to git checkout, which works only on the
index, but not the working tree, similar to what 'git reset <tree-ish>
-- <pathspec>... does.  Indeed the tests are adapted from the 'git
reset' tests.

In the longer term the idea is to potentially deprecate 'git reset
<tree-ish> -- <pathspec>...', so the 'git reset' command becomes only
about re-pointing the HEAD, and not also about copying entries from
<tree-ish> to the index.

Note that 'git checkout' by default works in overlay mode, meaning
files that match the pathspec that don't exist in <tree-ish>, but
exist in the index would not be removed.  'git checkout --no-overlay
--cached' can be used to get the same behaviour as 'git reset
<tree-ish> -- <pathspec>'.

One thing this patch doesn't currently deal with is conflicts.
Currently 'git checkout --{ours,theirs} -- <file-with-conflicts>'
doesn't do anything with the index, so the --cached option just
mirrors that behaviour.  But given it doesn't even deal with
conflicts, the '--cached' option doesn't make much sense when no
<tree-ish> is given.  As it operates only on the index, it's always a
no-op if no tree-ish is given.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---

Maybe we can just disallow --cached without <tree-ish> given for now,
and possibly later allow it with some different behaviour for
conflicts, not sure what the best way forward here is.  We can also
just make it update the index as appropriate, and have it behave
different than 'git checkout' curerntly does when handling conflicts?

 builtin/checkout.c         |  26 ++++++++--
 t/t2016-checkout-patch.sh  |   8 +++
 t/t2026-checkout-cached.sh | 103 +++++++++++++++++++++++++++++++++++++
 t/t9902-completion.sh      |   1 +
 4 files changed, 135 insertions(+), 3 deletions(-)
 create mode 100755 t/t2026-checkout-cached.sh

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 0aef35bbc4..6ba85e9de5 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -45,6 +45,7 @@ struct checkout_opts {
 	int ignore_other_worktrees;
 	int show_progress;
 	int overlay_mode;
+	int cached;
 	/*
 	 * If new checkout options are added, skip_merge_working_tree
 	 * should be updated accordingly.
@@ -288,6 +289,10 @@ static int checkout_paths(const struct checkout_opts *opts,
 		die(_("Cannot update paths and switch to branch '%s' at the same time."),
 		    opts->new_branch);
 
+	if (opts->patch_mode && opts->cached)
+		return run_add_interactive(revision, "--patch=reset",
+					   &opts->pathspec);
+
 	if (opts->patch_mode)
 		return run_add_interactive(revision, "--patch=checkout",
 					   &opts->pathspec);
@@ -319,7 +324,9 @@ static int checkout_paths(const struct checkout_opts *opts,
 				 * the current index, which means that it should 
 				 * be removed.
 				 */
-				ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
+				ce->ce_flags |= CE_MATCHED | CE_REMOVE;
+				if (!opts->cached)
+					ce->ce_flags |= CE_WT_REMOVE;
 				continue;
 			} else {
 				/*
@@ -392,6 +399,9 @@ static int checkout_paths(const struct checkout_opts *opts,
 	for (pos = 0; pos < active_nr; pos++) {
 		struct cache_entry *ce = active_cache[pos];
 		if (ce->ce_flags & CE_MATCHED) {
+			if (opts->cached) {
+				continue;
+			}
 			if (!ce_stage(ce)) {
 				errs |= checkout_entry(ce, &state, NULL);
 				continue;
@@ -571,6 +581,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
 	 * not tested here
 	 */
 
+	/*
+	 * opts->cached cannot be used with switching branches so is
+	 * not tested here
+	 */
+
 	/*
 	 * If we aren't creating a new branch any changes or updates will
 	 * happen in the existing branch.  Since that could only be updating
@@ -1207,9 +1222,13 @@ static int checkout_branch(struct checkout_opts *opts,
 		die(_("'%s' cannot be used with switching branches"),
 		    "--patch");
 
-	if (!opts->overlay_mode)
+	if (opts->overlay_mode != -1)
+		die(_("'%s' cannot be used with switching branches"),
+		    "--overlay/--no-overlay");
+
+	if (opts->cached)
 		die(_("'%s' cannot be used with switching branches"),
-		    "--no-overlay");
+		    "--cached");
 
 	if (opts->writeout_stage)
 		die(_("'%s' cannot be used with switching branches"),
@@ -1300,6 +1319,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
 		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
+		OPT_BOOL(0, "cached", &opts.cached, N_("work on the index only")),
 		OPT_END(),
 	};
 
diff --git a/t/t2016-checkout-patch.sh b/t/t2016-checkout-patch.sh
index 47aeb0b167..e8774046e0 100755
--- a/t/t2016-checkout-patch.sh
+++ b/t/t2016-checkout-patch.sh
@@ -108,6 +108,14 @@ test_expect_success PERL 'path limiting works: foo inside dir' '
 	verify_state dir/foo head head
 '
 
+test_expect_success PERL 'git checkout --cached -p' '
+	set_and_save_state dir/foo work work &&
+	test_write_lines n y | git checkout --cached -p >output &&
+	verify_state dir/foo work head &&
+	verify_saved_state bar &&
+	test_i18ngrep "Unstage" output
+'
+
 test_expect_success PERL 'none of this moved HEAD' '
 	verify_saved_head
 '
diff --git a/t/t2026-checkout-cached.sh b/t/t2026-checkout-cached.sh
new file mode 100755
index 0000000000..1b66192727
--- /dev/null
+++ b/t/t2026-checkout-cached.sh
@@ -0,0 +1,103 @@
+#!/bin/sh
+
+test_description='checkout --cached <pathspec>'
+
+. ./test-lib.sh
+
+test_expect_success 'checkout --cached <pathspec>' '
+	echo 1 >file1 &&
+	echo 2 >file2 &&
+	git add file1 file2 &&
+	test_tick &&
+	git commit -m files &&
+	git rm file2 &&
+	echo 3 >file3 &&
+	echo 4 >file1 &&
+	git add file1 file3 &&
+	git checkout --cached HEAD -- file1 file2 &&
+	test_must_fail git diff --quiet &&
+
+	cat >expect <<-\EOF &&
+	diff --git a/file1 b/file1
+	index d00491f..b8626c4 100644
+	--- a/file1
+	+++ b/file1
+	@@ -1 +1 @@
+	-1
+	+4
+	diff --git a/file2 b/file2
+	deleted file mode 100644
+	index 0cfbf08..0000000
+	--- a/file2
+	+++ /dev/null
+	@@ -1 +0,0 @@
+	-2
+	EOF
+	git diff >actual &&
+	test_cmp expect actual &&
+
+	cat >expect <<-\EOF &&
+	diff --git a/file3 b/file3
+	new file mode 100644
+	index 0000000..00750ed
+	--- /dev/null
+	+++ b/file3
+	@@ -0,0 +1 @@
+	+3
+	EOF
+	git diff --cached >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'checking out an unmodified path is a no-op' '
+	git reset --hard &&
+	git checkout --cached HEAD -- file1 &&
+	git diff-files --exit-code &&
+	git diff-index --cached --exit-code HEAD
+'
+
+test_expect_success 'checking out specific path that is unmerged' '
+	test_commit file3 file3 &&
+	git rm --cached file2 &&
+	echo 1234 >file2 &&
+	F1=$(git rev-parse HEAD:file1) &&
+	F2=$(git rev-parse HEAD:file2) &&
+	F3=$(git rev-parse HEAD:file3) &&
+	{
+		echo "100644 $F1 1	file2" &&
+		echo "100644 $F2 2	file2" &&
+		echo "100644 $F3 3	file2"
+	} | git update-index --index-info &&
+	git ls-files -u &&
+	git checkout --cached HEAD file2 &&
+	test_must_fail git diff --quiet &&
+	git diff-index --exit-code --cached HEAD
+'
+
+test_expect_success '--cached without --no-overlay does not remove entry from index' '
+	test_must_fail git checkout --cached HEAD^ file3 &&
+	git ls-files --error-unmatch -- file3
+'
+
+test_expect_success 'file is removed from the index with --no-overlay' '
+	git checkout --cached --no-overlay HEAD^ file3 &&
+	test_path_is_file file3 &&
+	test_must_fail git ls-files --error-unmatch -- file3
+'
+
+test_expect_success 'test checkout --cached --no-overlay at given paths' '
+	mkdir sub &&
+	>sub/file1 &&
+	>sub/file2 &&
+	git update-index --add sub/file1 sub/file2 &&
+	T=$(git write-tree) &&
+	git checkout --cached --no-overlay HEAD sub/file2 &&
+	test_must_fail git diff --quiet &&
+	U=$(git write-tree) &&
+	echo "$T" &&
+	echo "$U" &&
+	test_must_fail git diff-index --cached --exit-code "$T" &&
+	test "$T" != "$U"
+'
+
+test_done
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index a3fd9a9630..cbc304ace8 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -1437,6 +1437,7 @@ test_expect_success 'double dash "git checkout"' '
 	--no-quiet Z
 	--no-... Z
 	--overlay Z
+	--cached Z
 	EOF
 '
 
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 7/8] checkout: allow ignoring unmatched pathspec

[Thomas Gummerer]

Currently when 'git checkout -- <pathspec>...' is invoked with
multiple pathspecs, where one or more of the pathspecs don't match
anything, checkout errors out.

This can be inconvenient in some cases, such as when using git
checkout from a script.  Introduce a new --ignore-unmatched option,
which which allows us to ignore a non-matching pathspec instead of
erroring out.

In a subsequent commit we're going to start using 'git checkout' in
'git stash' and are going to make use of this feature.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c        | 10 +++++++++-
 t/t2022-checkout-paths.sh |  9 +++++++++
 t/t9902-completion.sh     |  1 +
 3 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 6ba85e9de5..7e7b5cd1d3 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -46,6 +46,7 @@ struct checkout_opts {
 	int show_progress;
 	int overlay_mode;
 	int cached;
+	int ignore_unmatched;
 	/*
 	 * If new checkout options are added, skip_merge_working_tree
 	 * should be updated accordingly.
@@ -358,7 +359,8 @@ static int checkout_paths(const struct checkout_opts *opts,
 			ce->ce_flags |= CE_MATCHED;
 	}
 
-	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
+	if (!opts->ignore_unmatched &&
+	    report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
 		free(ps_matched);
 		return 1;
 	}
@@ -586,6 +588,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
 	 * not tested here
 	 */
 
+	/*
+	 * opts->ignore_unmatched cannot be used with switching branches so is
+	 * not tested here
+	 */
+
 	/*
 	 * If we aren't creating a new branch any changes or updates will
 	 * happen in the existing branch.  Since that could only be updating
@@ -1320,6 +1327,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
 		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
 		OPT_BOOL(0, "cached", &opts.cached, N_("work on the index only")),
+		OPT_BOOL(0, "ignore-unmatched", &opts.ignore_unmatched, N_("don't error on unmatched pathspecs")),
 		OPT_END(),
 	};
 
diff --git a/t/t2022-checkout-paths.sh b/t/t2022-checkout-paths.sh
index fc3eb43b89..b44cdf7b63 100755
--- a/t/t2022-checkout-paths.sh
+++ b/t/t2022-checkout-paths.sh
@@ -78,4 +78,13 @@ test_expect_success 'do not touch files that are already up-to-date' '
 	test_cmp expect actual
 '
 
+test_expect_success 'checkout --ignore-unmatched' '
+	test_commit file1 &&
+	echo changed >file1.t &&
+	git checkout --ignore-unmatched -- file1.t unknown-file &&
+	echo file1 >expect &&
+	test_cmp expect file1.t
+
+'
+
 test_done
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index cbc304ace8..475debcf95 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -1438,6 +1438,7 @@ test_expect_success 'double dash "git checkout"' '
 	--no-... Z
 	--overlay Z
 	--cached Z
+	--ignore-unmatched Z
 	EOF
 '
 
-- 
2.20.0.405.gbc1bbc6f85

[PATCH 8/8] stash: use git checkout --no-overlay

[Thomas Gummerer]

Now that we have 'git checkout --no-overlay', we can use it in git
stash, making the codepaths for 'git stash push' with and without
pathspec more similar, and thus easier to follow.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---

As mentioned in the cover letter, not sure if we want to apply this
now.  There are two reasons I did this:
- Showing the new functionality of git checkout
- Increased test coverage, as we are running the new code with all git
  stash tests for free, which helped look at some cases that I was
  missing initially.

 git-stash.sh | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/git-stash.sh b/git-stash.sh
index 94793c1a91..67be04d996 100755
--- a/git-stash.sh
+++ b/git-stash.sh
@@ -314,19 +314,15 @@ push_stash () {
 
 	if test -z "$patch_mode"
 	then
-		test "$untracked" = "all" && CLEAN_X_OPTION=-x || CLEAN_X_OPTION=
-		if test -n "$untracked" && test $# = 0
+		test "$untracked" = "all" && CLEAN_X_OPTION=-X || CLEAN_X_OPTION=
+		if test -n "$untracked"
 		then
-			git clean --force --quiet -d $CLEAN_X_OPTION
+			git clean --force --quiet -d $CLEAN_X_OPTION -- "$@"
 		fi
 
 		if test $# != 0
 		then
-			test -z "$untracked" && UPDATE_OPTION="-u" || UPDATE_OPTION=
-			test "$untracked" = "all" && FORCE_OPTION="--force" || FORCE_OPTION=
-			git add $UPDATE_OPTION $FORCE_OPTION -- "$@"
-			git diff-index -p --cached --binary HEAD -- "$@" |
-			git apply --index -R
+			git checkout --quiet --no-overlay --ignore-unmatched HEAD -- "$@"
 		else
 			git reset --hard -q
 		fi
-- 
2.20.0.405.gbc1bbc6f85

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> Basically the idea is to also delete files when the match <pathspec>
> in 'git checkout <tree-ish> -- <pathspec>' in the current tree, but
> don't match <pathspec> in <tree-ish>.

I cannot quite parse it, but perhaps.

	"git checkout --no-overlay <tree-ish> -- <pathspec>" can
	remove paths in the index and in the working tree that match
	<pathspec>, if they do not appear in <tree-ish>.

If a new file D/F is in the index and in the working tree but not in
HEAD, "git checkout HEAD D/" or "git checkout HEAD D/F" would not
remove D/F from the index or the working tree.

With the --no-overlay option, it would, and that is often closer to
the wish of the user who wanted to say "restore the working tree
state of D/ (or D/F) from the state recorded in HEAD".

> The final step in the series is to actually make use of this in 'git
> stash', which simplifies the code there a bit.  I am however happy to
> hold off on this step until the stash-in-C series is merged, so we
> don't delay that further.

I think that is probably a good idea, for now.

> In addition to the no-overlay mode, we also add a --cached mode, which
> works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.
>
> Actually deprecating 'git reset <tree-ish> -- <pathspec>' should come
> later,...

Or we may not even need to deprecate it.  IIRC, what "stash" wished
to exist was "git reset --hard <tree-ish> -- <pathspec>", which, if
the command followed "--cached/--index" convention, would have been
called "git reset --index ...".  Did we actually have the need for
"--cached" mode?

> probably not before Duy's restore-files command lands, as 'git
> checkout --no-overlay <tree-ish> -- <pathspec>' is a bit cumbersome to
> type compared to 'git reset <tree-ish> -- <pathspec>'.

Yes, between "checkout --cached" and "checkout --no-overlay", the
latter is much more important, as the latter is what a missing "git
reset --hard <tree-ish> -- <pathspec>" would have been, but the
former can be written with an existing command.

> My hope is also that the no-overlay mode could become the new default
> in the restore-files command Duy is currently working on.

Yup, that is my hope, too ;-).

Re: [PATCH 1/8] move worktree tests to t24*

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> The 'git worktree' command used to be just another mode in 'git
> checkout', namely 'git checkout --to'.  When the tests for the latter
> were retrofitted for the former, the test name was adjusted, but the
> test number was kept, even though the test is testing a different
> command now.  t/README states: "Second digit tells the particular
> command we are testing.", so 'git worktree' should have a separate
> number just for itself.

That probably was written in the old world where there were only 10
commands in each category ;-) Nevertheless I have no problem with
this move (and I do not think there are in-flight topics in these
areas).

Thanks.

>
> Move the worktree tests to t24* to adhere to that guideline. We're
> going to make use of the free'd up numbers in a subsequent commit.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  t/{t2025-worktree-add.sh => t2400-worktree-add.sh}     | 0
>  t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} | 0
>  t/{t2027-worktree-list.sh => t2402-worktree-list.sh}   | 0
>  3 files changed, 0 insertions(+), 0 deletions(-)
>  rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
>  rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
>  rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
>
> diff --git a/t/t2025-worktree-add.sh b/t/t2400-worktree-add.sh
> similarity index 100%
> rename from t/t2025-worktree-add.sh
> rename to t/t2400-worktree-add.sh
> diff --git a/t/t2026-worktree-prune.sh b/t/t2401-worktree-prune.sh
> similarity index 100%
> rename from t/t2026-worktree-prune.sh
> rename to t/t2401-worktree-prune.sh
> diff --git a/t/t2027-worktree-list.sh b/t/t2402-worktree-list.sh
> similarity index 100%
> rename from t/t2027-worktree-list.sh
> rename to t/t2402-worktree-list.sh

Re: [PATCH 1/8] move worktree tests to t24*

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> The 'git worktree' command used to be just another mode in 'git
> checkout', namely 'git checkout --to'.  When the tests for the latter
> were retrofitted for the former, the test name was adjusted, but the
> test number was kept, even though the test is testing a different
> command now.  t/README states: "Second digit tells the particular
> command we are testing.", so 'git worktree' should have a separate
> number just for itself.
>
> Move the worktree tests to t24* to adhere to that guideline. We're
> going to make use of the free'd up numbers in a subsequent commit.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  t/{t2025-worktree-add.sh => t2400-worktree-add.sh}     | 0
>  t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} | 0
>  t/{t2027-worktree-list.sh => t2402-worktree-list.sh}   | 0
>  3 files changed, 0 insertions(+), 0 deletions(-)
>  rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
>  rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
>  rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)

Heh.. I did the same thing (in my unsent switch-branch/restore-files
series) and even used the same 24xx range :D You probably want to move
t2028 and t2029 too (not sure if they have landed on 'master')
-- 
Duy

Re: [PATCH 2/8] entry: factor out unlink_entry function

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Factor out the 'unlink_entry()' function from unpack-trees.c to
> entry.c.  It will be used in other places as well in subsequent
> steps.
>
> As it's no longer a static function, also move the documentation to
> the header file to make it more discoverable.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  cache.h        |  5 +++++
>  entry.c        | 15 +++++++++++++++
>  unpack-trees.c | 19 -------------------
>  3 files changed, 20 insertions(+), 19 deletions(-)
>
> diff --git a/cache.h b/cache.h
> index ca36b44ee0..c1c953e810 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1542,6 +1542,11 @@ struct checkout {
>  extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
>  extern void enable_delayed_checkout(struct checkout *state);
>  extern int finish_delayed_checkout(struct checkout *state);
> +/*
> + * Unlink the last component and schedule the leading directories for
> + * removal, such that empty directories get removed.
> + */
> +extern void unlink_entry(const struct cache_entry *ce);

I'm torn. We try to remove 'extern' but I can see you may want to add
it here to be consistent with others. And removing extern even from
functions from entry.c only would cause some conflicts.

I wonder if we should move the 'removal' variable in symlinks to
'struct checkout' to reduce another global variable. But I guess
that's the problem for another day. It's not the focus of this series.
-- 
Duy

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> 'checkout_entry()' currently only supports creating new entries in the
> working tree, but not deleting them.  Add the ability to remove
> entries at the same time if the entry is marked with the CE_WT_REMOVE
> flag.
>
> Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
> only used in unpack-tree, however we will make use of this in a
> subsequent step in the series.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  entry.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/entry.c b/entry.c
> index 3ec148ceee..cd1c6601b6 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -441,6 +441,13 @@ int checkout_entry(struct cache_entry *ce,
>         static struct strbuf path = STRBUF_INIT;
>         struct stat st;
>
> +       if (ce->ce_flags & CE_WT_REMOVE) {
> +               if (topath)
> +                       BUG("Can't remove entry to a path");
> +               unlink_entry(ce);
> +               return 0;
> +       }

This makes the path counting in nd/checkout-noisy less accurate. But
it's not your fault of course.

Junio, do you still want to merge that series down to 'next' or drop
it? If it will be merged down, I'll keep a note and fix it once this
one lands too.

> +
>         if (topath)
>                 return write_entry(ce, topath, state, 1);
>
> --
> 2.20.0.405.gbc1bbc6f85
>


-- 
Duy

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> When marking cache entries for removal, and later removing them all at
> once using 'remove_marked_cache_entries()', cache entries currently
> have to be invalidated manually in the cache tree and in the untracked
> cache.
>
> Add an invalidate flag to the function.  With the flag set, the
> function will take care of invalidating the path in the cache tree and
> in the untracked cache.
>
> This will be useful in a subsequent commit.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>
> For the two current callsites, unpack-trees seems to do this
> invalidation itself internally.

I'm still a bit scared of this invalidation business in unpack-trees.
The thing is, we handle two separate index_state there, src_index and
result and invalidation has to be done on the right one (because index
extensions are on src_index until the very end of unpack-trees;
invalidating on 'result' would be no-op and wrong).
remove_marked_cache_entries() seems to be called on 'result' while
invalidate_ce_path() is on src_index, hm....

> I don't quite understand why we don't
> need it in split-index mode though.  I assume it's because the cache
> tree in the main index would already have been invalidated?  I didn't
> have much time to dig, but couldn't produce any failures with it
> either, so I assume not invalidating paths is the right thing to do
> here.

Yeah I think it's because cache-tree and untracked cache are already
properly invalidated. This merge base thingy is done when we load the
index files up, not when we write them down. The "front" index may
record that a few paths in the base index are no longer valid and need
to be deleted. But untracked cache and cache-tree both should have
recorded that same info when these paths are marked for delete at
index write time.
-- 
Duy

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> @@ -302,15 +310,29 @@ static int checkout_paths(const struct checkout_opts *opts,
>                 ce->ce_flags &= ~CE_MATCHED;
>                 if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
>                         continue;
> -               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
> -                       /*
> -                        * "git checkout tree-ish -- path", but this entry
> -                        * is in the original index; it will not be checked
> -                        * out to the working tree and it does not matter
> -                        * if pathspec matched this entry.  We will not do
> -                        * anything to this entry at all.
> -                        */
> -                       continue;
> +               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE)) {
> +                       if (!opts->overlay_mode &&
> +                           ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
> +                               /*
> +                                * "git checkout --no-overlay <tree-ish> -- path",
> +                                * and the path is not in tree-ish, but is in
> +                                * the current index, which means that it should
> +                                * be removed.
> +                                */
> +                               ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
> +                               continue;
> +                       } else {

In non-overlay mode but when pathspec does not match, we come here too.

> +                               /*
> +                                * "git checkout tree-ish -- path", but this
> +                                * entry is in the original index; it will not

I think the missing key point in this comment block is "..is in the
original index _and it's not in tree-ish_". In non-overlay mode, if
pathspec does not match then it's safe to ignore too. But this logic
starts too get to complex and hurt my brain.

> +                                * be checked out to the working tree and it
> +                                * does not matter if pathspec matched this
> +                                * entry.  We will not do anything to this entry
> +                                * at all.
> +                                */
> +                               continue;
> +                       }
> +               }
>                 /*
>                  * Either this entry came from the tree-ish we are
>                  * checking the paths out of, or we are checking out

> @@ -1266,6 +1299,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
>                             "checkout", "control recursive updating of submodules",
>                             PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
>                 OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
> +               OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),

maybe add " (default)" to the help string.

>                 OPT_END(),
>         };
>
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Add a new --cached option to git checkout, which works only on the
> index, but not the working tree, similar to what 'git reset <tree-ish>
> -- <pathspec>... does.

Elijah wanted another mode (and I agree) that modifies worktree but
leaves the index alone. This is most useful (or least confusing) when
used with <tree-ish> and would be default in restore-files. I'm not
saying you have to implement it, but how do the new command line
options are designed to make sense?

I guess if --cached is "update index only" then --no-cached goes back
to the default "update both worktree and index" and we need a another
option for "worktree only"? Can we have one option with three possible
values (index-only, index-and-worktree, worktree-only) maybe?
-- 
Duy

Re: [PATCH 7/8] checkout: allow ignoring unmatched pathspec

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Currently when 'git checkout -- <pathspec>...' is invoked with
> multiple pathspecs, where one or more of the pathspecs don't match
> anything, checkout errors out.
>
> This can be inconvenient in some cases, such as when using git
> checkout from a script.

Wait, should scripts go with read-tree, checkout-index or other
plumbing commands instead?
-- 
Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Duy Nguyen]

On Sun, Dec 9, 2018 at 9:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> My hope is also that the no-overlay mode could become the new default
> in the restore-files command Duy is currently working on.

I already wrote something like that in git-restore-files.txt even
though the implementation is not there :D

There will be a hell of conflicts when the two series enter 'pu' (or
even worse, when the third one to update worktree only appears) so I'm
going to send the switch-branch/restore-files series out but mostly to
gather comments and will rebase once the other series land.

(Alternatively I'll split my series and let the switch-branch part
land first, may be simpler)
-- 
Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Here's the series I mentioned a couple of times on the list already,
> introducing a no-overlay mode in 'git checkout'.  The inspiration for
> this came from Junios message in [*1*].
>
> Basically the idea is to also delete files when the match <pathspec>
> in 'git checkout <tree-ish> -- <pathspec>' in the current tree, but
> don't match <pathspec> in <tree-ish>.  The rest of the cases are
> already properly taken care of by 'git checkout'.

Yes, but I'd put it a little differently:

"""
Basically, the idea is when the user run "git checkout --no-overlay
<tree-ish> -- <pathspec>" that the given pathspecs should exactly
match <tree-ish> after the operation completes.  This means that we
also want to delete files that match <pathspec> if those paths are not
found in <tree-ish>.
"""

...and maybe even toss in some comments about the fact that this is
the way git checkout should have always behaved, it just traditionally
hasn't.  (You could also work in comments about how with this new mode
the user can run git diff afterward with the given commit-ish and
pathspecs and get back an empty diff, as expected, which wasn't true
before.  But maybe I'm belaboring the point.)


> The final step in the series is to actually make use of this in 'git
> stash', which simplifies the code there a bit.  I am however happy to
> hold off on this step until the stash-in-C series is merged, so we
> don't delay that further.
>
> In addition to the no-overlay mode, we also add a --cached mode, which
> works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.

If you're adding a --cached mode to make it only work on the index,
should there be a similar mode to allow it to only work on the working
tree?  (I'm not as concerned with that here, but I really think the
new restore-files command by default should only operate on the
working tree, and then have options to affect the index either in
addition or instead of the working tree.)

> Actually deprecating 'git reset <tree-ish> -- <pathspec>' should come
> later, probably not before Duy's restore-files command lands, as 'git
> checkout --no-overlay <tree-ish> -- <pathspec>' is a bit cumbersome to
> type compared to 'git reset <tree-ish> -- <pathspec>'.

Makes sense.

> My hope is also that the no-overlay mode could become the new default
> in the restore-files command Duy is currently working on.

Absolutely, yes.  I don't want another broken command.  :-)


> No documentation yet, as I wanted to get this out for review first.
> I'm not familiar with most of the code I touched here, so there may
> well be much better ways to implement some of this, that I wasn't able
> to figure out.  I'd be very happy with some feedback around that.
>
> Another thing I'm not sure about is how to deal with conflicts.  In
> the cached mode this patch series is not dealing with it at all, as
> 'git checkout -- <pathspec>' when pathspec matches a file with
> conflicts doesn't update the index.  For the no-overlay mode, the file
> is removed if the corresponding stage is not found in the index.  I'm
> however not sure this is the right thing to do in all cases?

Here's how I'd go about analyzing that...

If the user passes a <tree-ish>, then the answer about what to do is
pretty obvious; the <tree-ish> didn't have conflicts, so conflicted
paths in the index that match the pathspec should be overwritten with
whatever version of those paths existed in <tree-ish> (possibly
implying deletion of some paths).

Also, as you point out, --cached means only modify the index and not
the working tree; so if they specify both --cached and provide no
tree, then they've specified a no-op.

So it's only interesting when you have conflicts in the index and
specify --no-overlay without a <tree-ish> or --cached.  This boils
down to "how do we update the working tree to match the index, when
the index is conflicted?"  A couple points to consider:
  * This is somewhat of an edge case
  * In the normal case --no-overlay is only different from --overlay
behavior for directories; it'd be nice if that extended to all cases
  * How does this command behave without a <tree-ish> when
--no-overlay is specified and a directory is given for a <pathspec>
and there aren't any conflicts?  Are we being consistent with that
behavior?


However, I think it turns out that the answer is much simpler than all
that initial analysis or what you say you've implemented.  Here's why:

If <pathspec> is a file which is present in both the working tree and
the index and it has conflicts, then "git checkout -- <pathspec>" will
currently throw an error:

$ git checkout -- subdir/counting
error: path 'subdir/counting' is unmerged

In fact, even if every entry in subdir/ is a path that is in both the
index and the working tree (so that --no-overlay and --overlay ought
to behave the same), if any one of the files in subdir is conflicted,
attempting to checkout the subdir will abort with this same error
message and no paths will be updated at all:

$ git checkout -- subdir
error: path 'subdir/counting' is unmerged

as such, the answer with what to do with --no-overlay mode is pretty
clear: if the <pathspec> matches _any_ path that is conflicted, simply
throw an error and abort the operation without making any changes at
all.

Re: [PATCH 2/8] entry: factor out unlink_entry function

[Elijah Newren]

On Mon, Dec 10, 2018 at 7:50 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Factor out the 'unlink_entry()' function from unpack-trees.c to
> > entry.c.  It will be used in other places as well in subsequent
> > steps.
> >
> > As it's no longer a static function, also move the documentation to
> > the header file to make it more discoverable.

I also started using unlink_entry() in another place in a local patch
series that I haven't submitted yet (and which I need to get back to
at some point).  So this will help me too.  :-)

> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >  cache.h        |  5 +++++
> >  entry.c        | 15 +++++++++++++++
> >  unpack-trees.c | 19 -------------------
> >  3 files changed, 20 insertions(+), 19 deletions(-)
> >
> > diff --git a/cache.h b/cache.h
> > index ca36b44ee0..c1c953e810 100644
> > --- a/cache.h
> > +++ b/cache.h
> > @@ -1542,6 +1542,11 @@ struct checkout {
> >  extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
> >  extern void enable_delayed_checkout(struct checkout *state);
> >  extern int finish_delayed_checkout(struct checkout *state);
> > +/*
> > + * Unlink the last component and schedule the leading directories for
> > + * removal, such that empty directories get removed.
> > + */
> > +extern void unlink_entry(const struct cache_entry *ce);
>
> I'm torn. We try to remove 'extern' but I can see you may want to add
> it here to be consistent with others. And removing extern even from
> functions from entry.c only would cause some conflicts.
>
> I wonder if we should move the 'removal' variable in symlinks to
> 'struct checkout' to reduce another global variable. But I guess
> that's the problem for another day. It's not the focus of this series.

"move the 'removal' variable in symlinks"?  I'm having a really hard
time parsing that phrase and the sentence it's embedded in.  Could you
reword for me Duy?

Re: [PATCH 2/8] entry: factor out unlink_entry function

[Duy Nguyen]

On Mon, Dec 10, 2018 at 6:23 PM Elijah Newren <newren@gmail.com> wrote:
> > I wonder if we should move the 'removal' variable in symlinks to
> > 'struct checkout' to reduce another global variable. But I guess
> > that's the problem for another day. It's not the focus of this series.
>
> "move the 'removal' variable in symlinks"?  I'm having a really hard
> time parsing that phrase and the sentence it's embedded in.  Could you
> reword for me Duy?

Sorry s/in symlinks/&.c/. There's a global variable named 'removal' in
symlinks.c which is used by schedule_dir_for_removal() and this
function in turn is used by unlink_entry().
-- 
Duy

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> 'checkout_entry()' currently only supports creating new entries in the
> working tree, but not deleting them.  Add the ability to remove
> entries at the same time if the entry is marked with the CE_WT_REMOVE
> flag.
>
> Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
> only used in unpack-tree, however we will make use of this in a
> subsequent step in the series.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  entry.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/entry.c b/entry.c
> index 3ec148ceee..cd1c6601b6 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -441,6 +441,13 @@ int checkout_entry(struct cache_entry *ce,
>         static struct strbuf path = STRBUF_INIT;
>         struct stat st;
>
> +       if (ce->ce_flags & CE_WT_REMOVE) {
> +               if (topath)
> +                       BUG("Can't remove entry to a path");

Minor nit: This error message is kinda hard to parse, for someone not
that familiar with all the *_entry functions, like myself.  Maybe add
a comment before this line:
    /* No content and thus no path to create, so we have no pathname
to return */
or reword the error slightly?  Or maybe it's fine and I was just
confused from lack of code familiarity, but I'll throw it out there
since I stumbled on it a bit.

> +               unlink_entry(ce);
> +               return 0;
> +       }
> +
>         if (topath)
>                 return write_entry(ce, topath, state, 1);
>
> --
> 2.20.0.405.gbc1bbc6f85

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Elijah Newren]

On Mon, Dec 10, 2018 at 8:09 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > When marking cache entries for removal, and later removing them all at
> > once using 'remove_marked_cache_entries()', cache entries currently
> > have to be invalidated manually in the cache tree and in the untracked
> > cache.
> >
> > Add an invalidate flag to the function.  With the flag set, the
> > function will take care of invalidating the path in the cache tree and
> > in the untracked cache.
> >
> > This will be useful in a subsequent commit.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >
> > For the two current callsites, unpack-trees seems to do this
> > invalidation itself internally.
>
> I'm still a bit scared of this invalidation business in unpack-trees.
> The thing is, we handle two separate index_state there, src_index and
> result and invalidation has to be done on the right one (because index
> extensions are on src_index until the very end of unpack-trees;
> invalidating on 'result' would be no-op and wrong).
> remove_marked_cache_entries() seems to be called on 'result' while
> invalidate_ce_path() is on src_index, hm....

Is Thomas avoiding problems here simply because merge is the only
caller of unpack_trees with src_index != dst_index?  Or does src_index
== dst_index for checkout not actually help?

If that does help with the checkout case, then allow me to find a
different way to muddy the waters...  I think I might want to make use
of this function in the merge machinery at some point, so I either
need to figure out how to convince you to verify if all this cache
tree invalidation stuff is sane, or somehow figure out all the
cache_tree stuff stuff myself so I can figure out what is right here.
:-)

> > I don't quite understand why we don't
> > need it in split-index mode though.  I assume it's because the cache
> > tree in the main index would already have been invalidated?  I didn't
> > have much time to dig, but couldn't produce any failures with it
> > either, so I assume not invalidating paths is the right thing to do
> > here.
>
> Yeah I think it's because cache-tree and untracked cache are already
> properly invalidated. This merge base thingy is done when we load the
> index files up, not when we write them down. The "front" index may
> record that a few paths in the base index are no longer valid and need
> to be deleted. But untracked cache and cache-tree both should have
> recorded that same info when these paths are marked for delete at
> index write time.

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Duy Nguyen]

On Mon, Dec 10, 2018 at 7:09 PM Elijah Newren <newren@gmail.com> wrote:
> > > For the two current callsites, unpack-trees seems to do this
> > > invalidation itself internally.
> >
> > I'm still a bit scared of this invalidation business in unpack-trees.
> > The thing is, we handle two separate index_state there, src_index and
> > result and invalidation has to be done on the right one (because index
> > extensions are on src_index until the very end of unpack-trees;
> > invalidating on 'result' would be no-op and wrong).
> > remove_marked_cache_entries() seems to be called on 'result' while
> > invalidate_ce_path() is on src_index, hm....
>
> Is Thomas avoiding problems here simply because merge is the only
> caller of unpack_trees with src_index != dst_index?  Or does src_index
> == dst_index for checkout not actually help?

I think it would not help. 'result' is a temporary index where we copy
things to (and it does not have anything from the beginning). If you
invalidate stuff in there, you invalidate nothing, regardless whether
dst_index == src_index.

> If that does help with the checkout case, then allow me to find a
> different way to muddy the waters...  I think I might want to make use
> of this function in the merge machinery at some point, so I either
> need to figure out how to convince you to verify if all this cache
> tree invalidation stuff is sane, or somehow figure out all the
> cache_tree stuff stuff myself so I can figure out what is right here.
> :-)

I'm not the unpack-trees man (I think that would still be Junio). And
I'm not saying it's sane either. I think it's just some leftover
things since Linus split "the index" in unpack-tree operation to
'src', 'result' and 'dst' many years ago and nobody was brave enough
to clean it up (then I piled on with untracked cache and split index,
but I did not see it clearly either). That person could be you ;-)
-- 
Duy

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Currently 'git checkout' is defined as an overlay operation, which
> means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
> entry in the index that matches <pathspec>, but that doesn't exist in
> <tree-ish>, that entry will not be removed from the index or the
> working tree.
>
> Introduce a new --{,no-}overlay option, which allows using 'git
> checkout' in non-overlay mode, thus removing files from the working
> tree if they do not exist in <tree-ish> but match <pathspec>.
>
> Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
> this way, so no changes are needed for the patch mode.  We disallow
> 'git checkout --overlay -p' to avoid confusing users who would expect
> to be able to force overlay mode in 'git checkout -p' this way.

Whoa...that's interesting.  To me, that argues even further that the
traditional checkout behavior was wrong all along and the choice of
--overlay vs. --no-overlay in the original implementation was a total
oversight.  I'm really tempted to say that --no-overlay should just be
the default in checkout too...but maybe that's too high a hill to
climb, at least for now.

Making --overlap and -p incompatible is a reasonable first step.  But
you should probably add a comment to the -p option documentation that
it implies --no-overlay.

> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  builtin/checkout.c             | 64 +++++++++++++++++++++++++++-------
>  t/t2025-checkout-no-overlay.sh | 47 +++++++++++++++++++++++++
>  t/t9902-completion.sh          |  1 +
>  3 files changed, 99 insertions(+), 13 deletions(-)
>  create mode 100755 t/t2025-checkout-no-overlay.sh
>
> diff --git a/builtin/checkout.c b/builtin/checkout.c
> index acdafc6e4c..0aef35bbc4 100644
> --- a/builtin/checkout.c
> +++ b/builtin/checkout.c
> @@ -44,6 +44,7 @@ struct checkout_opts {
>         int ignore_skipworktree;
>         int ignore_other_worktrees;
>         int show_progress;
> +       int overlay_mode;
>         /*
>          * If new checkout options are added, skip_merge_working_tree
>          * should be updated accordingly.
> @@ -132,7 +133,8 @@ static int skip_same_name(const struct cache_entry *ce, int pos)
>         return pos;
>  }
>
> -static int check_stage(int stage, const struct cache_entry *ce, int pos)
> +static int check_stage(int stage, const struct cache_entry *ce, int pos,
> +                      int overlay_mode)
>  {
>         while (pos < active_nr &&
>                !strcmp(active_cache[pos]->name, ce->name)) {
> @@ -140,6 +142,8 @@ static int check_stage(int stage, const struct cache_entry *ce, int pos)
>                         return 0;
>                 pos++;
>         }
> +       if (!overlay_mode)
> +               return 0;
>         if (stage == 2)
>                 return error(_("path '%s' does not have our version"), ce->name);
>         else
> @@ -165,7 +169,7 @@ static int check_stages(unsigned stages, const struct cache_entry *ce, int pos)
>  }
>
>  static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
> -                         const struct checkout *state)
> +                         const struct checkout *state, int overlay_mode)
>  {
>         while (pos < active_nr &&
>                !strcmp(active_cache[pos]->name, ce->name)) {
> @@ -173,6 +177,10 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
>                         return checkout_entry(active_cache[pos], state, NULL);
>                 pos++;
>         }
> +       if (!overlay_mode) {
> +               unlink_entry(ce);
> +               return 0;
> +       }
>         if (stage == 2)
>                 return error(_("path '%s' does not have our version"), ce->name);
>         else
> @@ -302,15 +310,29 @@ static int checkout_paths(const struct checkout_opts *opts,
>                 ce->ce_flags &= ~CE_MATCHED;
>                 if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
>                         continue;
> -               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
> -                       /*
> -                        * "git checkout tree-ish -- path", but this entry
> -                        * is in the original index; it will not be checked
> -                        * out to the working tree and it does not matter
> -                        * if pathspec matched this entry.  We will not do
> -                        * anything to this entry at all.
> -                        */
> -                       continue;
> +               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE)) {
> +                       if (!opts->overlay_mode &&
> +                           ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
> +                               /*
> +                                * "git checkout --no-overlay <tree-ish> -- path",
> +                                * and the path is not in tree-ish, but is in
> +                                * the current index, which means that it should
> +                                * be removed.
> +                                */
> +                               ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
> +                               continue;
> +                       } else {
> +                               /*
> +                                * "git checkout tree-ish -- path", but this
> +                                * entry is in the original index; it will not
> +                                * be checked out to the working tree and it
> +                                * does not matter if pathspec matched this
> +                                * entry.  We will not do anything to this entry
> +                                * at all.
> +                                */
> +                               continue;
> +                       }
> +               }
>                 /*
>                  * Either this entry came from the tree-ish we are
>                  * checking the paths out of, or we are checking out
> @@ -348,7 +370,7 @@ static int checkout_paths(const struct checkout_opts *opts,
>                         if (opts->force) {
>                                 warning(_("path '%s' is unmerged"), ce->name);
>                         } else if (opts->writeout_stage) {
> -                               errs |= check_stage(opts->writeout_stage, ce, pos);
> +                               errs |= check_stage(opts->writeout_stage, ce, pos, opts->overlay_mode);
>                         } else if (opts->merge) {
>                                 errs |= check_stages((1<<2) | (1<<3), ce, pos);
>                         } else {
> @@ -375,12 +397,14 @@ static int checkout_paths(const struct checkout_opts *opts,
>                                 continue;
>                         }
>                         if (opts->writeout_stage)
> -                               errs |= checkout_stage(opts->writeout_stage, ce, pos, &state);
> +                               errs |= checkout_stage(opts->writeout_stage, ce, pos, &state, opts->overlay_mode);
>                         else if (opts->merge)
>                                 errs |= checkout_merged(pos, &state);
>                         pos = skip_same_name(ce, pos) - 1;
>                 }
>         }
> +       remove_marked_cache_entries(&the_index, 1);
> +       remove_scheduled_dirs();
>         errs |= finish_delayed_checkout(&state);
>
>         if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))
> @@ -542,6 +566,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
>          * opts->show_progress only impacts output so doesn't require a merge
>          */
>
> +       /*
> +        * opts->overlay_mode cannot be used with switching branches so is
> +        * not tested here
> +        */
> +
>         /*
>          * If we aren't creating a new branch any changes or updates will
>          * happen in the existing branch.  Since that could only be updating
> @@ -1178,6 +1207,10 @@ static int checkout_branch(struct checkout_opts *opts,
>                 die(_("'%s' cannot be used with switching branches"),
>                     "--patch");
>
> +       if (!opts->overlay_mode)
> +               die(_("'%s' cannot be used with switching branches"),
> +                   "--no-overlay");
> +
>         if (opts->writeout_stage)
>                 die(_("'%s' cannot be used with switching branches"),
>                     "--ours/--theirs");
> @@ -1266,6 +1299,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
>                             "checkout", "control recursive updating of submodules",
>                             PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
>                 OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
> +               OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
>                 OPT_END(),
>         };
>
> @@ -1274,6 +1308,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
>         opts.overwrite_ignore = 1;
>         opts.prefix = prefix;
>         opts.show_progress = -1;
> +       opts.overlay_mode = -1;
>
>         git_config(git_checkout_config, &opts);
>
> @@ -1297,6 +1332,9 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
>         if ((!!opts.new_branch + !!opts.new_branch_force + !!opts.new_orphan_branch) > 1)
>                 die(_("-b, -B and --orphan are mutually exclusive"));
>
> +       if (opts.overlay_mode == 1 && opts.patch_mode)
> +               die(_("-p and --overlay are mutually exclusive"));
> +
>         /*
>          * From here on, new_branch will contain the branch to be checked out,
>          * and new_branch_force and new_orphan_branch will tell us which one of
> diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
> new file mode 100755
> index 0000000000..3575321382
> --- /dev/null
> +++ b/t/t2025-checkout-no-overlay.sh
> @@ -0,0 +1,47 @@
> +#!/bin/sh
> +
> +test_description='checkout --no-overlay <tree-ish> -- <pathspec>'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'setup' '
> +       git commit --allow-empty -m "initial"
> +'
> +
> +test_expect_success 'checkout --no-overlay deletes files not in <tree>' '
> +       >file &&
> +       mkdir dir &&
> +       >dir/file1 &&
> +       git add file dir/file1 &&
> +       git checkout --no-overlay HEAD -- file &&
> +       test_path_is_missing file &&
> +       test_path_is_file dir/file1
> +'
> +
> +test_expect_success 'checkout --no-overlay removing last file from directory' '
> +       git checkout --no-overlay HEAD -- dir/file1 &&
> +       test_path_is_missing dir
> +'
> +
> +test_expect_success 'checkout -p --overlay is disallowed' '
> +       test_must_fail git checkout -p --overlay HEAD 2>actual &&
> +       test_i18ngrep "fatal: -p and --overlay are mutually exclusive" actual
> +'
> +
> +test_expect_success '--no-overlay --theirs with M/D conflict deletes file' '
> +       test_commit file1 file1 &&
> +       test_commit file2 file2 &&
> +       git rm --cached file1 &&
> +       echo 1234 >file1 &&
> +       F1=$(git rev-parse HEAD:file1) &&
> +       F2=$(git rev-parse HEAD:file2) &&
> +       {
> +               echo "100644 $F1 1      file1" &&
> +               echo "100644 $F2 2      file1"
> +       } | git update-index --index-info &&
> +       test_path_is_file file1 &&
> +       git checkout --theirs --no-overlay -- file1 &&
> +       test_path_is_missing file1
> +'
> +
> +test_done
> diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
> index 175f83d704..a3fd9a9630 100755
> --- a/t/t9902-completion.sh
> +++ b/t/t9902-completion.sh
> @@ -1436,6 +1436,7 @@ test_expect_success 'double dash "git checkout"' '
>         --progress Z
>         --no-quiet Z
>         --no-... Z
> +       --overlay Z
>         EOF
>  '
>
> --
> 2.20.0.405.gbc1bbc6f85
>

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Elijah Newren]

On Mon, Dec 10, 2018 at 10:19 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Mon, Dec 10, 2018 at 7:09 PM Elijah Newren <newren@gmail.com> wrote:
> > > > For the two current callsites, unpack-trees seems to do this
> > > > invalidation itself internally.
> > >
> > > I'm still a bit scared of this invalidation business in unpack-trees.
> > > The thing is, we handle two separate index_state there, src_index and
> > > result and invalidation has to be done on the right one (because index
> > > extensions are on src_index until the very end of unpack-trees;
> > > invalidating on 'result' would be no-op and wrong).
> > > remove_marked_cache_entries() seems to be called on 'result' while
> > > invalidate_ce_path() is on src_index, hm....
> >
> > Is Thomas avoiding problems here simply because merge is the only
> > caller of unpack_trees with src_index != dst_index?  Or does src_index
> > == dst_index for checkout not actually help?
>
> I think it would not help. 'result' is a temporary index where we copy
> things to (and it does not have anything from the beginning). If you
> invalidate stuff in there, you invalidate nothing, regardless whether
> dst_index == src_index.
>
> > If that does help with the checkout case, then allow me to find a
> > different way to muddy the waters...  I think I might want to make use
> > of this function in the merge machinery at some point, so I either
> > need to figure out how to convince you to verify if all this cache
> > tree invalidation stuff is sane, or somehow figure out all the
> > cache_tree stuff stuff myself so I can figure out what is right here.
> > :-)
>
> I'm not the unpack-trees man (I think that would still be Junio). And
> I'm not saying it's sane either. I think it's just some leftover
> things since Linus split "the index" in unpack-tree operation to
> 'src', 'result' and 'dst' many years ago and nobody was brave enough
> to clean it up (then I piled on with untracked cache and split index,
> but I did not see it clearly either). That person could be you ;-)

Hmm, might make a good New Year's resolution: Enter the abyss, find
out if one can return from it...  or maybe I could just sanely run
away screaming.  We'll see.

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Duy Nguyen]

On Mon, Dec 10, 2018 at 7:25 PM Elijah Newren <newren@gmail.com> wrote:
> > I'm not the unpack-trees man (I think that would still be Junio). And
> > I'm not saying it's sane either. I think it's just some leftover
> > things since Linus split "the index" in unpack-tree operation to
> > 'src', 'result' and 'dst' many years ago and nobody was brave enough
> > to clean it up (then I piled on with untracked cache and split index,
> > but I did not see it clearly either). That person could be you ;-)
>
> Hmm, might make a good New Year's resolution: Enter the abyss, find
> out if one can return from it...  or maybe I could just sanely run
> away screaming.  We'll see.

I'm getting off topic. But my new years resolution would be optimize
for the case where src_index == dst_index, which is somewhat ironic
because we used to do everything in the same index, but it was a messy
mess and had to be split up.
-- 
Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Duy Nguyen]

On Mon, Dec 10, 2018 at 6:18 PM Elijah Newren <newren@gmail.com> wrote:
> > The final step in the series is to actually make use of this in 'git
> > stash', which simplifies the code there a bit.  I am however happy to
> > hold off on this step until the stash-in-C series is merged, so we
> > don't delay that further.
> >
> > In addition to the no-overlay mode, we also add a --cached mode, which
> > works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.
>
> If you're adding a --cached mode to make it only work on the index,
> should there be a similar mode to allow it to only work on the working
> tree?  (I'm not as concerned with that here, but I really think the
> new restore-files command by default should only operate on the
> working tree, and then have options to affect the index either in
> addition or instead of the working tree.)

In the context of restore-files, --target=<worktree|index|both> is a
very good candidate because "restore-files --from=foo --target=index"
is almost like saying "restore files in the index from "foo"". For
checkout, probably not as good. But then we can have different option
names for the two commands. So if "git checkout" is going to never
have "update worktree only" mode, then --cached is a still good way to
go.
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Add a new --cached option to git checkout, which works only on the
> index, but not the working tree, similar to what 'git reset <tree-ish>
> -- <pathspec>... does.  Indeed the tests are adapted from the 'git
> reset' tests.
>
> In the longer term the idea is to potentially deprecate 'git reset
> <tree-ish> -- <pathspec>...', so the 'git reset' command becomes only
> about re-pointing the HEAD, and not also about copying entries from
> <tree-ish> to the index.
>
> Note that 'git checkout' by default works in overlay mode, meaning
> files that match the pathspec that don't exist in <tree-ish>, but
> exist in the index would not be removed.  'git checkout --no-overlay
> --cached' can be used to get the same behaviour as 'git reset
> <tree-ish> -- <pathspec>'.

I think this argues _even more_ that --no-overlay should be the
default.  Your series is valuable even if we don't push on that, I'm
just being noisy about what I think would be an even better world.

Also, I don't think I've mentioned it yet, but I'm really excited
about this series and what you're doing.  It's super cool.  (Which I
expected when I saw the description of the desired behavior, but I'm
also liking and contemplating re-using some code...)

> One thing this patch doesn't currently deal with is conflicts.
> Currently 'git checkout --{ours,theirs} -- <file-with-conflicts>'
> doesn't do anything with the index, so the --cached option just
> mirrors that behaviour.  But given it doesn't even deal with
> conflicts, the '--cached' option doesn't make much sense when no
> <tree-ish> is given.  As it operates only on the index, it's always a
> no-op if no tree-ish is given.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>
> Maybe we can just disallow --cached without <tree-ish> given for now,
> and possibly later allow it with some different behaviour for
> conflicts, not sure what the best way forward here is.  We can also
> just make it update the index as appropriate, and have it behave
> different than 'git checkout' curerntly does when handling conflicts?

Huh?
  "git checkout -- <path>"
means update <path> from the index, meaning the index is left alone
(it's the source) and only the working tree is touched.

When you add a flag named --cached to only update the index and not
the working tree, then the index becomes the sole destination.

Now we combine: no tree is specified means the index is the source of
the writing, and --cached being specified means the index is the sole
destination of the writing.  Thus, you have a no-op.  If the user
specifies --cached and no tree, you should immediately exit with a
message along the lines of "Nothing to do; no tree given and --cached
specified."  The presence of conflicts seems completely irrelevant to
me here.

>  builtin/checkout.c         |  26 ++++++++--
>  t/t2016-checkout-patch.sh  |   8 +++
>  t/t2026-checkout-cached.sh | 103 +++++++++++++++++++++++++++++++++++++
>  t/t9902-completion.sh      |   1 +
>  4 files changed, 135 insertions(+), 3 deletions(-)
>  create mode 100755 t/t2026-checkout-cached.sh
>
> diff --git a/builtin/checkout.c b/builtin/checkout.c
> index 0aef35bbc4..6ba85e9de5 100644
> --- a/builtin/checkout.c
> +++ b/builtin/checkout.c
> @@ -45,6 +45,7 @@ struct checkout_opts {
>         int ignore_other_worktrees;
>         int show_progress;
>         int overlay_mode;
> +       int cached;
>         /*
>          * If new checkout options are added, skip_merge_working_tree
>          * should be updated accordingly.
> @@ -288,6 +289,10 @@ static int checkout_paths(const struct checkout_opts *opts,
>                 die(_("Cannot update paths and switch to branch '%s' at the same time."),
>                     opts->new_branch);
>
> +       if (opts->patch_mode && opts->cached)
> +               return run_add_interactive(revision, "--patch=reset",
> +                                          &opts->pathspec);
> +
>         if (opts->patch_mode)
>                 return run_add_interactive(revision, "--patch=checkout",
>                                            &opts->pathspec);
> @@ -319,7 +324,9 @@ static int checkout_paths(const struct checkout_opts *opts,
>                                  * the current index, which means that it should
>                                  * be removed.
>                                  */
> -                               ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
> +                               ce->ce_flags |= CE_MATCHED | CE_REMOVE;
> +                               if (!opts->cached)
> +                                       ce->ce_flags |= CE_WT_REMOVE;
>                                 continue;
>                         } else {
>                                 /*
> @@ -392,6 +399,9 @@ static int checkout_paths(const struct checkout_opts *opts,
>         for (pos = 0; pos < active_nr; pos++) {
>                 struct cache_entry *ce = active_cache[pos];
>                 if (ce->ce_flags & CE_MATCHED) {
> +                       if (opts->cached) {
> +                               continue;
> +                       }
>                         if (!ce_stage(ce)) {
>                                 errs |= checkout_entry(ce, &state, NULL);
>                                 continue;
> @@ -571,6 +581,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
>          * not tested here
>          */
>
> +       /*
> +        * opts->cached cannot be used with switching branches so is
> +        * not tested here
> +        */
> +
>         /*
>          * If we aren't creating a new branch any changes or updates will
>          * happen in the existing branch.  Since that could only be updating
> @@ -1207,9 +1222,13 @@ static int checkout_branch(struct checkout_opts *opts,
>                 die(_("'%s' cannot be used with switching branches"),
>                     "--patch");
>
> -       if (!opts->overlay_mode)
> +       if (opts->overlay_mode != -1)
> +               die(_("'%s' cannot be used with switching branches"),
> +                   "--overlay/--no-overlay");
> +
> +       if (opts->cached)
>                 die(_("'%s' cannot be used with switching branches"),
> -                   "--no-overlay");
> +                   "--cached");
>
>         if (opts->writeout_stage)
>                 die(_("'%s' cannot be used with switching branches"),
> @@ -1300,6 +1319,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
>                             PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
>                 OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
>                 OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
> +               OPT_BOOL(0, "cached", &opts.cached, N_("work on the index only")),
>                 OPT_END(),
>         };
>
> diff --git a/t/t2016-checkout-patch.sh b/t/t2016-checkout-patch.sh
> index 47aeb0b167..e8774046e0 100755
> --- a/t/t2016-checkout-patch.sh
> +++ b/t/t2016-checkout-patch.sh
> @@ -108,6 +108,14 @@ test_expect_success PERL 'path limiting works: foo inside dir' '
>         verify_state dir/foo head head
>  '
>
> +test_expect_success PERL 'git checkout --cached -p' '
> +       set_and_save_state dir/foo work work &&
> +       test_write_lines n y | git checkout --cached -p >output &&
> +       verify_state dir/foo work head &&
> +       verify_saved_state bar &&
> +       test_i18ngrep "Unstage" output
> +'
> +
>  test_expect_success PERL 'none of this moved HEAD' '
>         verify_saved_head
>  '
> diff --git a/t/t2026-checkout-cached.sh b/t/t2026-checkout-cached.sh
> new file mode 100755
> index 0000000000..1b66192727
> --- /dev/null
> +++ b/t/t2026-checkout-cached.sh
> @@ -0,0 +1,103 @@
> +#!/bin/sh
> +
> +test_description='checkout --cached <pathspec>'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'checkout --cached <pathspec>' '
> +       echo 1 >file1 &&
> +       echo 2 >file2 &&
> +       git add file1 file2 &&
> +       test_tick &&
> +       git commit -m files &&
> +       git rm file2 &&
> +       echo 3 >file3 &&
> +       echo 4 >file1 &&
> +       git add file1 file3 &&
> +       git checkout --cached HEAD -- file1 file2 &&
> +       test_must_fail git diff --quiet &&
> +
> +       cat >expect <<-\EOF &&
> +       diff --git a/file1 b/file1
> +       index d00491f..b8626c4 100644
> +       --- a/file1
> +       +++ b/file1
> +       @@ -1 +1 @@
> +       -1
> +       +4
> +       diff --git a/file2 b/file2
> +       deleted file mode 100644
> +       index 0cfbf08..0000000
> +       --- a/file2
> +       +++ /dev/null
> +       @@ -1 +0,0 @@
> +       -2
> +       EOF
> +       git diff >actual &&
> +       test_cmp expect actual &&
> +
> +       cat >expect <<-\EOF &&
> +       diff --git a/file3 b/file3
> +       new file mode 100644
> +       index 0000000..00750ed
> +       --- /dev/null
> +       +++ b/file3
> +       @@ -0,0 +1 @@
> +       +3
> +       EOF
> +       git diff --cached >actual &&
> +       test_cmp expect actual
> +'
> +
> +test_expect_success 'checking out an unmodified path is a no-op' '
> +       git reset --hard &&
> +       git checkout --cached HEAD -- file1 &&
> +       git diff-files --exit-code &&
> +       git diff-index --cached --exit-code HEAD
> +'
> +
> +test_expect_success 'checking out specific path that is unmerged' '
> +       test_commit file3 file3 &&
> +       git rm --cached file2 &&
> +       echo 1234 >file2 &&
> +       F1=$(git rev-parse HEAD:file1) &&
> +       F2=$(git rev-parse HEAD:file2) &&
> +       F3=$(git rev-parse HEAD:file3) &&
> +       {
> +               echo "100644 $F1 1      file2" &&
> +               echo "100644 $F2 2      file2" &&
> +               echo "100644 $F3 3      file2"
> +       } | git update-index --index-info &&
> +       git ls-files -u &&
> +       git checkout --cached HEAD file2 &&
> +       test_must_fail git diff --quiet &&
> +       git diff-index --exit-code --cached HEAD
> +'
> +
> +test_expect_success '--cached without --no-overlay does not remove entry from index' '
> +       test_must_fail git checkout --cached HEAD^ file3 &&
> +       git ls-files --error-unmatch -- file3
> +'
> +
> +test_expect_success 'file is removed from the index with --no-overlay' '
> +       git checkout --cached --no-overlay HEAD^ file3 &&
> +       test_path_is_file file3 &&
> +       test_must_fail git ls-files --error-unmatch -- file3
> +'
> +
> +test_expect_success 'test checkout --cached --no-overlay at given paths' '
> +       mkdir sub &&
> +       >sub/file1 &&
> +       >sub/file2 &&
> +       git update-index --add sub/file1 sub/file2 &&
> +       T=$(git write-tree) &&
> +       git checkout --cached --no-overlay HEAD sub/file2 &&
> +       test_must_fail git diff --quiet &&
> +       U=$(git write-tree) &&

Do we need to worry at all about losing the exit status of write-tree
in either invocation?  In particular, if the second one for U fails
somehow, we'd end up with $U being a blank string and we'd still
probably get "$T" != "$U" below.

You also had some rev-parse invocations hidden in a sub-shell in both
this patch and patch 5, but subsequent commands relied on non-empty
output out of those, so I figured those were fine.  This one might be
too, but I thought I'd at least mention it.

> +       echo "$T" &&
> +       echo "$U" &&
> +       test_must_fail git diff-index --cached --exit-code "$T" &&
> +       test "$T" != "$U"
> +'
> +
> +test_done
> diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
> index a3fd9a9630..cbc304ace8 100755
> --- a/t/t9902-completion.sh
> +++ b/t/t9902-completion.sh
> @@ -1437,6 +1437,7 @@ test_expect_success 'double dash "git checkout"' '
>         --no-quiet Z
>         --no-... Z
>         --overlay Z
> +       --cached Z
>         EOF
>  '
>
> --
> 2.20.0.405.gbc1bbc6f85

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Elijah Newren]

On Mon, Dec 10, 2018 at 10:34 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Mon, Dec 10, 2018 at 7:25 PM Elijah Newren <newren@gmail.com> wrote:
> > > I'm not the unpack-trees man (I think that would still be Junio). And
> > > I'm not saying it's sane either. I think it's just some leftover
> > > things since Linus split "the index" in unpack-tree operation to
> > > 'src', 'result' and 'dst' many years ago and nobody was brave enough
> > > to clean it up (then I piled on with untracked cache and split index,
> > > but I did not see it clearly either). That person could be you ;-)
> >
> > Hmm, might make a good New Year's resolution: Enter the abyss, find
> > out if one can return from it...  or maybe I could just sanely run
> > away screaming.  We'll see.
>
> I'm getting off topic. But my new years resolution would be optimize
> for the case where src_index == dst_index, which is somewhat ironic
> because we used to do everything in the same index, but it was a messy
> mess and had to be split up.

Ooh, that sounds cool too.  I look forward to seeing it.

Re: [PATCH 7/8] checkout: allow ignoring unmatched pathspec

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Currently when 'git checkout -- <pathspec>...' is invoked with
> multiple pathspecs, where one or more of the pathspecs don't match
> anything, checkout errors out.
>
> This can be inconvenient in some cases, such as when using git
> checkout from a script.  Introduce a new --ignore-unmatched option,
> which which allows us to ignore a non-matching pathspec instead of
> erroring out.
>
> In a subsequent commit we're going to start using 'git checkout' in
> 'git stash' and are going to make use of this feature.

This makes sense, but seems incomplete.  But to explain it, I think
there's another bug I need to demonstrate first because it's related
on builds on it.  First, the setup:

  $ echo foo >subdir/newfile
  $ git add subdir/newfile
  $ echo bar >>subdir/newfile
  $ git status
  On branch A
  Changes to be committed:
    (use "git reset HEAD <file>..." to unstage)

      new file:   subdir/newfile

  Changes not staged for commit:
    (use "git add <file>..." to update what will be committed)
    (use "git checkout -- <file>..." to discard changes in working directory)

      modified:   subdir/newfile

Now, does it do what we expect?

  $ git checkout HEAD -- subdir/newfile
  error: pathspec 'subdir/newfile' did not match any file(s) known to git

This is the old overlay behavior; kinda lame, but you made no claims
about fixing the default behavior.  What about with your new option?

  $ git checkout --no-overlay HEAD -- subdir
  $ git status
  On branch A
  nothing to commit, working tree clean

Yes, the feature seems to work as advertised.  However, let's try
again with a different variant:

  $ echo foo >subdir/newfile
  $ git checkout --no-overlay HEAD -- subdir
  $ git status
  On branch A
  Untracked files:
    (use "git add <file>..." to include in what will be committed)

      subdir/newfile

Why is the file ignored and left there?  Also:

  $ git checkout --no-overlay HEAD -- subdir/newfile
  error: pathspec 'subdir/newfile' did not match any file(s) known to git

That seems wrong to me.  The point of no-overlay is to make it match
HEAD, and while subdir/newfile doesn't exist in HEAD or the index it
does match in the working tree so the intent is clear. But let's say
that the user did go ahead and specify your new flag:


  $ git checkout --no-overlay --ignore-unmatch HEAD -- subdir/newfile
  $ git status
  On branch A
  Untracked files:
    (use "git add <file>..." to include in what will be committed)

      subdir/newfile

  nothing added to commit but untracked files present (use "git add" to track)

So now it avoids erroring out when the user does more work than
necessary, but it still misses appropriately cleaning up the file.

Re: [PATCH 8/8] stash: use git checkout --no-overlay

[Elijah Newren]

On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Now that we have 'git checkout --no-overlay', we can use it in git
> stash, making the codepaths for 'git stash push' with and without
> pathspec more similar, and thus easier to follow.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>
> As mentioned in the cover letter, not sure if we want to apply this
> now.  There are two reasons I did this:
> - Showing the new functionality of git checkout
> - Increased test coverage, as we are running the new code with all git
>   stash tests for free, which helped look at some cases that I was
>   missing initially.
>
>  git-stash.sh | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)
>
> diff --git a/git-stash.sh b/git-stash.sh
> index 94793c1a91..67be04d996 100755
> --- a/git-stash.sh
> +++ b/git-stash.sh
> @@ -314,19 +314,15 @@ push_stash () {
>
>         if test -z "$patch_mode"
>         then
> -               test "$untracked" = "all" && CLEAN_X_OPTION=-x || CLEAN_X_OPTION=
> -               if test -n "$untracked" && test $# = 0
> +               test "$untracked" = "all" && CLEAN_X_OPTION=-X || CLEAN_X_OPTION=
> +               if test -n "$untracked"
>                 then
> -                       git clean --force --quiet -d $CLEAN_X_OPTION
> +                       git clean --force --quiet -d $CLEAN_X_OPTION -- "$@"
>                 fi
>
>                 if test $# != 0
>                 then
> -                       test -z "$untracked" && UPDATE_OPTION="-u" || UPDATE_OPTION=
> -                       test "$untracked" = "all" && FORCE_OPTION="--force" || FORCE_OPTION=
> -                       git add $UPDATE_OPTION $FORCE_OPTION -- "$@"
> -                       git diff-index -p --cached --binary HEAD -- "$@" |
> -                       git apply --index -R
> +                       git checkout --quiet --no-overlay --ignore-unmatched HEAD -- "$@"

Nice.  :-)

Re: [PATCH 2/8] entry: factor out unlink_entry function

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> I wonder if we should move the 'removal' variable in symlinks to
> 'struct checkout' to reduce another global variable. But I guess
> that's the problem for another day. It's not the focus of this series.

Before any such move, I think it is important to notice that the
thing is not thread friendly and devise a way to deal with it.

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>> +       if (ce->ce_flags & CE_WT_REMOVE) {
>> +               if (topath)
>> +                       BUG("Can't remove entry to a path");
>> +               unlink_entry(ce);
>> +               return 0;
>> +       }
>
> This makes the path counting in nd/checkout-noisy less accurate. But
> it's not your fault of course.

When we check out absense of one path, how do we want to count it?
Do we say "one path checked out?" when we remove one path?

> Junio, do you still want to merge that series down to 'next' or drop
> it? If it will be merged down, I'll keep a note and fix it once this
> one lands too.

Sure, I still agree with you that "git checkout" that reports what
it did when given a "<branch>", but does not report what it did when
given a "<pathspec>", is being inconsistent.  If it makes it easier
to manage, I can kick nd/checkout-noisy out of 'next' to be rebased
on whatever more appropriate when rewinding its tip.

Thanks.

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> I'm still a bit scared of this invalidation business in unpack-trees.

I too was (and I suspect that I would realize that I still am, if I
take another fresh look at the current code) afraid when I did the
cache-tree work and decided to invalidate it as a whole upfront.

> The thing is, we handle two separate index_state there, src_index and
> result and invalidation has to be done on the right one (because index
> extensions are on src_index until the very end of unpack-trees;
> invalidating on 'result' would be no-op and wrong).
> ...
> Yeah I think it's because cache-tree and untracked cache are already
> properly invalidated. ...

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

>> Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
>> this way, so no changes are needed for the patch mode.  We disallow
>> 'git checkout --overlay -p' to avoid confusing users who would expect
>> to be able to force overlay mode in 'git checkout -p' this way.
>
> Whoa...that's interesting.  To me, that argues even further that the
> traditional checkout behavior was wrong all along and the choice of
> --overlay vs. --no-overlay in the original implementation was a total
> oversight.  I'm really tempted to say that --no-overlay should just be
> the default in checkout too...but maybe that's too high a hill to
> climb, at least for now.

You are exhibiting a typical reaction to a shiny new toy.

The original "checkout paths out of the trees and/or the index to
recover what was lost" is modeled after "cp -R .../else/where .",
which is an overlay operation, and you wouldn't imagine complaining
that "cp -R" is wrong not to remove things in the target, to be
equivalent to "rm -fr where && cp -R .../else/where .".

Each has its own uses.  We just didn't have the other half of the
pair.

If anything, I would consider "checkout -p" that does not do overlay
is a bug that needs to be fixed.

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> Elijah wanted another mode (and I agree) that modifies worktree but
> leaves the index alone. This is most useful (or least confusing) when
> used with <tree-ish> and would be default in restore-files. I'm not
> saying you have to implement it, but how do the new command line
> options are designed to make sense?

I'd model it after "git apply", i.e.

	git restore-files [--tree=<treeish>] <pathspec>

would work only on the working tree files,

	git restore-files --tree=<treeish> --cached <pathspec>

would match the entries in the index that match pathspec to the
given treeish without touching the working tree, and

	git restore-files --tree=<treeish> --index <pathspec>

would be both.

I have never been happy with the phraso, the (arbitrary) distinction
between --cached/--index we use, so in the very longer term (like,
introducing synonym at 3.0 boundary and deprecating older ones at
4.0 boundary), it may not be a bad idea to rename "--index" to
"--index-and-working-tree" and "--cached" to "--index-only".  

But until that happens, it would be better to use these two
consistently.

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Elijah Newren]

On Mon, Dec 10, 2018 at 7:07 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> >> Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
> >> this way, so no changes are needed for the patch mode.  We disallow
> >> 'git checkout --overlay -p' to avoid confusing users who would expect
> >> to be able to force overlay mode in 'git checkout -p' this way.
> >
> > Whoa...that's interesting.  To me, that argues even further that the
> > traditional checkout behavior was wrong all along and the choice of
> > --overlay vs. --no-overlay in the original implementation was a total
> > oversight.  I'm really tempted to say that --no-overlay should just be
> > the default in checkout too...but maybe that's too high a hill to
> > climb, at least for now.
>
> You are exhibiting a typical reaction to a shiny new toy.
>
> The original "checkout paths out of the trees and/or the index to
> recover what was lost" is modeled after "cp -R .../else/where .",
> which is an overlay operation, and you wouldn't imagine complaining
> that "cp -R" is wrong not to remove things in the target, to be
> equivalent to "rm -fr where && cp -R .../else/where .".
>
> Each has its own uses.  We just didn't have the other half of the
> pair.

Ah, modelled on cp -R.  I think that rather than reacting "to a shiny
new toy", I just had always had a different mental model AND failed to
figure out what the original model was, leading me to always view it
as buggy.  Thanks for giving me the model I was missing.

> If anything, I would consider "checkout -p" that does not do overlay
> is a bug that needs to be fixed.

Yeah, --no-overlay being the default for -p, and --overlay being the
default otherwise is rather inconsistent.  (Though I'm also fine with
that being fixed by some future patch series.)

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Mon, Dec 10, 2018 at 7:13 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Duy Nguyen <pclouds@gmail.com> writes:
>
> > Elijah wanted another mode (and I agree) that modifies worktree but
> > leaves the index alone. This is most useful (or least confusing) when
> > used with <tree-ish> and would be default in restore-files. I'm not
> > saying you have to implement it, but how do the new command line
> > options are designed to make sense?
>
> I'd model it after "git apply", i.e.
>
>         git restore-files [--tree=<treeish>] <pathspec>
>
> would work only on the working tree files,
>
>         git restore-files --tree=<treeish> --cached <pathspec>
>
> would match the entries in the index that match pathspec to the
> given treeish without touching the working tree, and
>
>         git restore-files --tree=<treeish> --index <pathspec>
>
> would be both.
>
> I have never been happy with the phraso, the (arbitrary) distinction
> between --cached/--index we use, so in the very longer term (like,
> introducing synonym at 3.0 boundary and deprecating older ones at
> 4.0 boundary), it may not be a bad idea to rename "--index" to
> "--index-and-working-tree" and "--cached" to "--index-only".
>
> But until that happens, it would be better to use these two
> consistently.

I agree that consistency is important, but trying to distinguish
between "--cached" and "--index" is _extremely_ painful.  I still
can't keep the distinction straight and have to look it up every time
I want to use either.  I don't know how to possibly teach users the
meaning.  Could we introduce synonyms earlier at least, and make the
synonyms more prominent than the "--cached" and "--index" terms in the
documentation?

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Tue, Dec 11, 2018 at 7:12 AM Elijah Newren <newren@gmail.com> wrote:
>
> On Mon, Dec 10, 2018 at 7:13 PM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > Duy Nguyen <pclouds@gmail.com> writes:
> >
> > > Elijah wanted another mode (and I agree) that modifies worktree but
> > > leaves the index alone. This is most useful (or least confusing) when
> > > used with <tree-ish> and would be default in restore-files. I'm not
> > > saying you have to implement it, but how do the new command line
> > > options are designed to make sense?
> >
> > I'd model it after "git apply", i.e.
> >
> >         git restore-files [--tree=<treeish>] <pathspec>
> >
> > would work only on the working tree files,
> >
> >         git restore-files --tree=<treeish> --cached <pathspec>
> >
> > would match the entries in the index that match pathspec to the
> > given treeish without touching the working tree, and
> >
> >         git restore-files --tree=<treeish> --index <pathspec>
> >
> > would be both.
> >
> > I have never been happy with the phraso, the (arbitrary) distinction
> > between --cached/--index we use, so in the very longer term (like,
> > introducing synonym at 3.0 boundary and deprecating older ones at
> > 4.0 boundary), it may not be a bad idea to rename "--index" to
> > "--index-and-working-tree" and "--cached" to "--index-only".
> >
> > But until that happens, it would be better to use these two
> > consistently.
>
> I agree that consistency is important, but trying to distinguish
> between "--cached" and "--index" is _extremely_ painful.  I still
> can't keep the distinction straight and have to look it up every time
> I want to use either.  I don't know how to possibly teach users the
> meaning.  Could we introduce synonyms earlier at least, and make the
> synonyms more prominent than the "--cached" and "--index" terms in the
> documentation?

For git-checkout I think --index and --cached fit. For restore-files,
if you come up with better names, I'll gladly use that. Otherwise I'll
just use these.
-- 
Duy

Re: [PATCH 1/8] move worktree tests to t24*

[Thomas Gummerer]

On 12/10, Duy Nguyen wrote:
> On Sun, Dec 9, 2018 at 9:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > The 'git worktree' command used to be just another mode in 'git
> > checkout', namely 'git checkout --to'.  When the tests for the latter
> > were retrofitted for the former, the test name was adjusted, but the
> > test number was kept, even though the test is testing a different
> > command now.  t/README states: "Second digit tells the particular
> > command we are testing.", so 'git worktree' should have a separate
> > number just for itself.
> >
> > Move the worktree tests to t24* to adhere to that guideline. We're
> > going to make use of the free'd up numbers in a subsequent commit.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >  t/{t2025-worktree-add.sh => t2400-worktree-add.sh}     | 0
> >  t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} | 0
> >  t/{t2027-worktree-list.sh => t2402-worktree-list.sh}   | 0
> >  3 files changed, 0 insertions(+), 0 deletions(-)
> >  rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
> >  rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
> >  rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
> 
> Heh.. I did the same thing (in my unsent switch-branch/restore-files
> series) and even used the same 24xx range :D You probably want to move
> t2028 and t2029 too (not sure if they have landed on 'master')

:)  I unfortunately didn't have time to read the
switch-branch/restore-files series in detail, but good to know someone
thought the same way.  I started this work before t2028 and t2029
landed on master, so I failed to notice them.  But I'll rebase on
master and move these two tests as well, thanks for noticing.

> -- 
> Duy

Re: [PATCH 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Thomas Gummerer]

On 12/10, Elijah Newren wrote:
> On Mon, Dec 10, 2018 at 8:09 AM Duy Nguyen <pclouds@gmail.com> wrote:
> >
> > On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > >
> > > When marking cache entries for removal, and later removing them all at
> > > once using 'remove_marked_cache_entries()', cache entries currently
> > > have to be invalidated manually in the cache tree and in the untracked
> > > cache.
> > >
> > > Add an invalidate flag to the function.  With the flag set, the
> > > function will take care of invalidating the path in the cache tree and
> > > in the untracked cache.
> > >
> > > This will be useful in a subsequent commit.
> > >
> > > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > > ---
> > >
> > > For the two current callsites, unpack-trees seems to do this
> > > invalidation itself internally.
> >
> > I'm still a bit scared of this invalidation business in unpack-trees.
> > The thing is, we handle two separate index_state there, src_index and
> > result and invalidation has to be done on the right one (because index
> > extensions are on src_index until the very end of unpack-trees;
> > invalidating on 'result' would be no-op and wrong).
> > remove_marked_cache_entries() seems to be called on 'result' while
> > invalidate_ce_path() is on src_index, hm....
> 
> Is Thomas avoiding problems here simply because merge is the only
> caller of unpack_trees with src_index != dst_index?  Or does src_index
> == dst_index for checkout not actually help?

I'm trying to avoid problems in this patch by keeping status quo, and
not changing the cache-tree invalidation in any way.  'git checkout --
<pathspec>' doesn't use unpack-trees, so I don't think I have to worry
about src_index vs. dst_index.

In what I was saying above I was merely trying to explain why we don't
need invalidate the cache-tree in the 'remove_marked_cache_entries()'
function.

> If that does help with the checkout case, then allow me to find a
> different way to muddy the waters...  I think I might want to make use
> of this function in the merge machinery at some point, so I either
> need to figure out how to convince you to verify if all this cache
> tree invalidation stuff is sane, or somehow figure out all the
> cache_tree stuff stuff myself so I can figure out what is right here.
> :-)
> 
> > > I don't quite understand why we don't
> > > need it in split-index mode though.  I assume it's because the cache
> > > tree in the main index would already have been invalidated?  I didn't
> > > have much time to dig, but couldn't produce any failures with it
> > > either, so I assume not invalidating paths is the right thing to do
> > > here.
> >
> > Yeah I think it's because cache-tree and untracked cache are already
> > properly invalidated. This merge base thingy is done when we load the
> > index files up, not when we write them down. The "front" index may
> > record that a few paths in the base index are no longer valid and need
> > to be deleted. But untracked cache and cache-tree both should have
> > recorded that same info when these paths are marked for delete at
> > index write time.

Thanks, that makes sense to me.

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Thomas Gummerer]

On 12/10, Elijah Newren wrote:
> On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > 'checkout_entry()' currently only supports creating new entries in the
> > working tree, but not deleting them.  Add the ability to remove
> > entries at the same time if the entry is marked with the CE_WT_REMOVE
> > flag.
> >
> > Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
> > only used in unpack-tree, however we will make use of this in a
> > subsequent step in the series.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >  entry.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/entry.c b/entry.c
> > index 3ec148ceee..cd1c6601b6 100644
> > --- a/entry.c
> > +++ b/entry.c
> > @@ -441,6 +441,13 @@ int checkout_entry(struct cache_entry *ce,
> >         static struct strbuf path = STRBUF_INIT;
> >         struct stat st;
> >
> > +       if (ce->ce_flags & CE_WT_REMOVE) {
> > +               if (topath)
> > +                       BUG("Can't remove entry to a path");
> 
> Minor nit: This error message is kinda hard to parse, for someone not
> that familiar with all the *_entry functions, like myself.  Maybe add
> a comment before this line:
>     /* No content and thus no path to create, so we have no pathname
> to return */
> or reword the error slightly?  Or maybe it's fine and I was just
> confused from lack of code familiarity, but I'll throw it out there
> since I stumbled on it a bit.

I'll try to make it more clear in the new round, thanks!

> > +               unlink_entry(ce);
> > +               return 0;
> > +       }
> > +
> >         if (topath)
> >                 return write_entry(ce, topath, state, 1);
> >
> > --
> > 2.20.0.405.gbc1bbc6f85

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 12/11, Junio C Hamano wrote:
> Elijah Newren <newren@gmail.com> writes:
> 
> >> Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
> >> this way, so no changes are needed for the patch mode.  We disallow
> >> 'git checkout --overlay -p' to avoid confusing users who would expect
> >> to be able to force overlay mode in 'git checkout -p' this way.
> >
> > Whoa...that's interesting.  To me, that argues even further that the
> > traditional checkout behavior was wrong all along and the choice of
> > --overlay vs. --no-overlay in the original implementation was a total
> > oversight.  I'm really tempted to say that --no-overlay should just be
> > the default in checkout too...but maybe that's too high a hill to
> > climb, at least for now.
> 
> You are exhibiting a typical reaction to a shiny new toy.

I wonder whether it's worth introducing a config option for this, so
people could use this new mode by default if they wanted to, without
having to type the extra argument?

'git checkout' is a plumbing command, but I see that some of the shell
scripts in git.git are using it.  Though they are only using it in the
checkout branch mode, rather than the checkout paths mode.

> The original "checkout paths out of the trees and/or the index to
> recover what was lost" is modeled after "cp -R .../else/where .",
> which is an overlay operation, and you wouldn't imagine complaining
> that "cp -R" is wrong not to remove things in the target, to be
> equivalent to "rm -fr where && cp -R .../else/where .".
> 
> Each has its own uses.  We just didn't have the other half of the
> pair.
> 
> If anything, I would consider "checkout -p" that does not do overlay
> is a bug that needs to be fixed.
> 

Re: [PATCH 6/8] checkout: add --cached option

[Thomas Gummerer]

On 12/10, Elijah Newren wrote:
> On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Add a new --cached option to git checkout, which works only on the
> > index, but not the working tree, similar to what 'git reset <tree-ish>
> > -- <pathspec>... does.  Indeed the tests are adapted from the 'git
> > reset' tests.
> >
> > In the longer term the idea is to potentially deprecate 'git reset
> > <tree-ish> -- <pathspec>...', so the 'git reset' command becomes only
> > about re-pointing the HEAD, and not also about copying entries from
> > <tree-ish> to the index.
> >
> > Note that 'git checkout' by default works in overlay mode, meaning
> > files that match the pathspec that don't exist in <tree-ish>, but
> > exist in the index would not be removed.  'git checkout --no-overlay
> > --cached' can be used to get the same behaviour as 'git reset
> > <tree-ish> -- <pathspec>'.
> 
> I think this argues _even more_ that --no-overlay should be the
> default.  Your series is valuable even if we don't push on that, I'm
> just being noisy about what I think would be an even better world.

I think just having that mode in 'git restore-files' Duy is working on
may have to be enough for now.

> Also, I don't think I've mentioned it yet, but I'm really excited
> about this series and what you're doing.  It's super cool.  (Which I
> expected when I saw the description of the desired behavior, but I'm
> also liking and contemplating re-using some code...)

Thanks :)

> > One thing this patch doesn't currently deal with is conflicts.
> > Currently 'git checkout --{ours,theirs} -- <file-with-conflicts>'
> > doesn't do anything with the index, so the --cached option just
> > mirrors that behaviour.  But given it doesn't even deal with
> > conflicts, the '--cached' option doesn't make much sense when no
> > <tree-ish> is given.  As it operates only on the index, it's always a
> > no-op if no tree-ish is given.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >
> > Maybe we can just disallow --cached without <tree-ish> given for now,
> > and possibly later allow it with some different behaviour for
> > conflicts, not sure what the best way forward here is.  We can also
> > just make it update the index as appropriate, and have it behave
> > different than 'git checkout' curerntly does when handling conflicts?
> 
> Huh?
>   "git checkout -- <path>"
> means update <path> from the index, meaning the index is left alone
> (it's the source) and only the working tree is touched.
> 
> When you add a flag named --cached to only update the index and not
> the working tree, then the index becomes the sole destination.
> 
> Now we combine: no tree is specified means the index is the source of
> the writing, and --cached being specified means the index is the sole
> destination of the writing.  Thus, you have a no-op.  If the user
> specifies --cached and no tree, you should immediately exit with a
> message along the lines of "Nothing to do; no tree given and --cached
> specified."  The presence of conflicts seems completely irrelevant to
> me here.

Ah yeah you're right, thanks for a sanity check.  The command I was
most worried about was 'git checkout --cached --{ours,theirs} -- <pathspec>', 
which I thought should update the index.  But as we don't give any
tree-ish, I'm not sure anymore it should.  Maybe just always exiting
with the message you mention above is the right thing to do.

> >  builtin/checkout.c         |  26 ++++++++--
> >  t/t2016-checkout-patch.sh  |   8 +++
> >  t/t2026-checkout-cached.sh | 103 +++++++++++++++++++++++++++++++++++++
> >  t/t9902-completion.sh      |   1 +
> >  4 files changed, 135 insertions(+), 3 deletions(-)
> >  create mode 100755 t/t2026-checkout-cached.sh
> >
> > diff --git a/builtin/checkout.c b/builtin/checkout.c
> > index 0aef35bbc4..6ba85e9de5 100644
> > --- a/builtin/checkout.c
> > +++ b/builtin/checkout.c
> > @@ -45,6 +45,7 @@ struct checkout_opts {
> >         int ignore_other_worktrees;
> >         int show_progress;
> >         int overlay_mode;
> > +       int cached;
> >         /*
> >          * If new checkout options are added, skip_merge_working_tree
> >          * should be updated accordingly.
> > @@ -288,6 +289,10 @@ static int checkout_paths(const struct checkout_opts *opts,
> >                 die(_("Cannot update paths and switch to branch '%s' at the same time."),
> >                     opts->new_branch);
> >
> > +       if (opts->patch_mode && opts->cached)
> > +               return run_add_interactive(revision, "--patch=reset",
> > +                                          &opts->pathspec);
> > +
> >         if (opts->patch_mode)
> >                 return run_add_interactive(revision, "--patch=checkout",
> >                                            &opts->pathspec);
> > @@ -319,7 +324,9 @@ static int checkout_paths(const struct checkout_opts *opts,
> >                                  * the current index, which means that it should
> >                                  * be removed.
> >                                  */
> > -                               ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
> > +                               ce->ce_flags |= CE_MATCHED | CE_REMOVE;
> > +                               if (!opts->cached)
> > +                                       ce->ce_flags |= CE_WT_REMOVE;
> >                                 continue;
> >                         } else {
> >                                 /*
> > @@ -392,6 +399,9 @@ static int checkout_paths(const struct checkout_opts *opts,
> >         for (pos = 0; pos < active_nr; pos++) {
> >                 struct cache_entry *ce = active_cache[pos];
> >                 if (ce->ce_flags & CE_MATCHED) {
> > +                       if (opts->cached) {
> > +                               continue;
> > +                       }
> >                         if (!ce_stage(ce)) {
> >                                 errs |= checkout_entry(ce, &state, NULL);
> >                                 continue;
> > @@ -571,6 +581,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
> >          * not tested here
> >          */
> >
> > +       /*
> > +        * opts->cached cannot be used with switching branches so is
> > +        * not tested here
> > +        */
> > +
> >         /*
> >          * If we aren't creating a new branch any changes or updates will
> >          * happen in the existing branch.  Since that could only be updating
> > @@ -1207,9 +1222,13 @@ static int checkout_branch(struct checkout_opts *opts,
> >                 die(_("'%s' cannot be used with switching branches"),
> >                     "--patch");
> >
> > -       if (!opts->overlay_mode)
> > +       if (opts->overlay_mode != -1)
> > +               die(_("'%s' cannot be used with switching branches"),
> > +                   "--overlay/--no-overlay");
> > +
> > +       if (opts->cached)
> >                 die(_("'%s' cannot be used with switching branches"),
> > -                   "--no-overlay");
> > +                   "--cached");
> >
> >         if (opts->writeout_stage)
> >                 die(_("'%s' cannot be used with switching branches"),
> > @@ -1300,6 +1319,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
> >                             PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
> >                 OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
> >                 OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
> > +               OPT_BOOL(0, "cached", &opts.cached, N_("work on the index only")),
> >                 OPT_END(),
> >         };
> >
> > diff --git a/t/t2016-checkout-patch.sh b/t/t2016-checkout-patch.sh
> > index 47aeb0b167..e8774046e0 100755
> > --- a/t/t2016-checkout-patch.sh
> > +++ b/t/t2016-checkout-patch.sh
> > @@ -108,6 +108,14 @@ test_expect_success PERL 'path limiting works: foo inside dir' '
> >         verify_state dir/foo head head
> >  '
> >
> > +test_expect_success PERL 'git checkout --cached -p' '
> > +       set_and_save_state dir/foo work work &&
> > +       test_write_lines n y | git checkout --cached -p >output &&
> > +       verify_state dir/foo work head &&
> > +       verify_saved_state bar &&
> > +       test_i18ngrep "Unstage" output
> > +'
> > +
> >  test_expect_success PERL 'none of this moved HEAD' '
> >         verify_saved_head
> >  '
> > diff --git a/t/t2026-checkout-cached.sh b/t/t2026-checkout-cached.sh
> > new file mode 100755
> > index 0000000000..1b66192727
> > --- /dev/null
> > +++ b/t/t2026-checkout-cached.sh
> > @@ -0,0 +1,103 @@
> > +#!/bin/sh
> > +
> > +test_description='checkout --cached <pathspec>'
> > +
> > +. ./test-lib.sh
> > +
> > +test_expect_success 'checkout --cached <pathspec>' '
> > +       echo 1 >file1 &&
> > +       echo 2 >file2 &&
> > +       git add file1 file2 &&
> > +       test_tick &&
> > +       git commit -m files &&
> > +       git rm file2 &&
> > +       echo 3 >file3 &&
> > +       echo 4 >file1 &&
> > +       git add file1 file3 &&
> > +       git checkout --cached HEAD -- file1 file2 &&
> > +       test_must_fail git diff --quiet &&
> > +
> > +       cat >expect <<-\EOF &&
> > +       diff --git a/file1 b/file1
> > +       index d00491f..b8626c4 100644
> > +       --- a/file1
> > +       +++ b/file1
> > +       @@ -1 +1 @@
> > +       -1
> > +       +4
> > +       diff --git a/file2 b/file2
> > +       deleted file mode 100644
> > +       index 0cfbf08..0000000
> > +       --- a/file2
> > +       +++ /dev/null
> > +       @@ -1 +0,0 @@
> > +       -2
> > +       EOF
> > +       git diff >actual &&
> > +       test_cmp expect actual &&
> > +
> > +       cat >expect <<-\EOF &&
> > +       diff --git a/file3 b/file3
> > +       new file mode 100644
> > +       index 0000000..00750ed
> > +       --- /dev/null
> > +       +++ b/file3
> > +       @@ -0,0 +1 @@
> > +       +3
> > +       EOF
> > +       git diff --cached >actual &&
> > +       test_cmp expect actual
> > +'
> > +
> > +test_expect_success 'checking out an unmodified path is a no-op' '
> > +       git reset --hard &&
> > +       git checkout --cached HEAD -- file1 &&
> > +       git diff-files --exit-code &&
> > +       git diff-index --cached --exit-code HEAD
> > +'
> > +
> > +test_expect_success 'checking out specific path that is unmerged' '
> > +       test_commit file3 file3 &&
> > +       git rm --cached file2 &&
> > +       echo 1234 >file2 &&
> > +       F1=$(git rev-parse HEAD:file1) &&
> > +       F2=$(git rev-parse HEAD:file2) &&
> > +       F3=$(git rev-parse HEAD:file3) &&
> > +       {
> > +               echo "100644 $F1 1      file2" &&
> > +               echo "100644 $F2 2      file2" &&
> > +               echo "100644 $F3 3      file2"
> > +       } | git update-index --index-info &&
> > +       git ls-files -u &&
> > +       git checkout --cached HEAD file2 &&
> > +       test_must_fail git diff --quiet &&
> > +       git diff-index --exit-code --cached HEAD
> > +'
> > +
> > +test_expect_success '--cached without --no-overlay does not remove entry from index' '
> > +       test_must_fail git checkout --cached HEAD^ file3 &&
> > +       git ls-files --error-unmatch -- file3
> > +'
> > +
> > +test_expect_success 'file is removed from the index with --no-overlay' '
> > +       git checkout --cached --no-overlay HEAD^ file3 &&
> > +       test_path_is_file file3 &&
> > +       test_must_fail git ls-files --error-unmatch -- file3
> > +'
> > +
> > +test_expect_success 'test checkout --cached --no-overlay at given paths' '
> > +       mkdir sub &&
> > +       >sub/file1 &&
> > +       >sub/file2 &&
> > +       git update-index --add sub/file1 sub/file2 &&
> > +       T=$(git write-tree) &&
> > +       git checkout --cached --no-overlay HEAD sub/file2 &&
> > +       test_must_fail git diff --quiet &&
> > +       U=$(git write-tree) &&
> 
> Do we need to worry at all about losing the exit status of write-tree
> in either invocation?  In particular, if the second one for U fails
> somehow, we'd end up with $U being a blank string and we'd still
> probably get "$T" != "$U" below.

Hmm this seems to be a fairly common pattern in our test suite:

$ git grep -F '$(git write-tree)' t/* | wc -l
112

But maybe it's just something we used to do, but should move away
from.  Just writing the output to a file shouldn't be much harder
either, I'll do that in the next iteration.

> You also had some rev-parse invocations hidden in a sub-shell in both
> this patch and patch 5, but subsequent commands relied on non-empty
> output out of those, so I figured those were fine.  This one might be
> too, but I thought I'd at least mention it.
> 
> > +       echo "$T" &&
> > +       echo "$U" &&
> > +       test_must_fail git diff-index --cached --exit-code "$T" &&
> > +       test "$T" != "$U"
> > +'
> > +
> > +test_done
> > diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
> > index a3fd9a9630..cbc304ace8 100755
> > --- a/t/t9902-completion.sh
> > +++ b/t/t9902-completion.sh
> > @@ -1437,6 +1437,7 @@ test_expect_success 'double dash "git checkout"' '
> >         --no-quiet Z
> >         --no-... Z
> >         --overlay Z
> > +       --cached Z
> >         EOF
> >  '
> >
> > --
> > 2.20.0.405.gbc1bbc6f85

Re: [PATCH 7/8] checkout: allow ignoring unmatched pathspec

[Thomas Gummerer]

On 12/10, Duy Nguyen wrote:
> On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Currently when 'git checkout -- <pathspec>...' is invoked with
> > multiple pathspecs, where one or more of the pathspecs don't match
> > anything, checkout errors out.
> >
> > This can be inconvenient in some cases, such as when using git
> > checkout from a script.
> 
> Wait, should scripts go with read-tree, checkout-index or other
> plumbing commands instead?

Possibly.  As mentioned in an other email, we do seem to have some
scripts in git.git that are using 'git checkout' already, but they are
using it in the checkout branch mode, rather than the checkout paths
mode that I would like to use it in git-stash.

But with the rewrite of 'git stash' in C, maybe this step is moot
anyway, and we can just call the checkout_paths function internally
without using the run_command API at all.  We could then have an
internal mode for ignoring unmatched pathspecs that we wouldn't need
to expose to users.

> -- 
> Duy

Re: [PATCH 7/8] checkout: allow ignoring unmatched pathspec

[Thomas Gummerer]

On 12/10, Elijah Newren wrote:
> On Sun, Dec 9, 2018 at 12:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Currently when 'git checkout -- <pathspec>...' is invoked with
> > multiple pathspecs, where one or more of the pathspecs don't match
> > anything, checkout errors out.
> >
> > This can be inconvenient in some cases, such as when using git
> > checkout from a script.  Introduce a new --ignore-unmatched option,
> > which which allows us to ignore a non-matching pathspec instead of
> > erroring out.
> >
> > In a subsequent commit we're going to start using 'git checkout' in
> > 'git stash' and are going to make use of this feature.
> 
> This makes sense, but seems incomplete.  But to explain it, I think
> there's another bug I need to demonstrate first because it's related
> on builds on it.  First, the setup:
> 
>   $ echo foo >subdir/newfile
>   $ git add subdir/newfile
>   $ echo bar >>subdir/newfile
>   $ git status
>   On branch A
>   Changes to be committed:
>     (use "git reset HEAD <file>..." to unstage)
> 
>       new file:   subdir/newfile
> 
>   Changes not staged for commit:
>     (use "git add <file>..." to update what will be committed)
>     (use "git checkout -- <file>..." to discard changes in working directory)
> 
>       modified:   subdir/newfile
> 
> Now, does it do what we expect?
> 
>   $ git checkout HEAD -- subdir/newfile
>   error: pathspec 'subdir/newfile' did not match any file(s) known to git
> 
> This is the old overlay behavior; kinda lame, but you made no claims
> about fixing the default behavior.  What about with your new option?
> 
>   $ git checkout --no-overlay HEAD -- subdir
>   $ git status
>   On branch A
>   nothing to commit, working tree clean
> 
> Yes, the feature seems to work as advertised.  However, let's try
> again with a different variant:
> 
>   $ echo foo >subdir/newfile
>   $ git checkout --no-overlay HEAD -- subdir
>   $ git status
>   On branch A
>   Untracked files:
>     (use "git add <file>..." to include in what will be committed)
> 
>       subdir/newfile
> 
> Why is the file ignored and left there?  Also:
> 
>   $ git checkout --no-overlay HEAD -- subdir/newfile
>   error: pathspec 'subdir/newfile' did not match any file(s) known to git
> 
> That seems wrong to me.

Ah interesting, this is a case I didn't consider.  I'm a bit torn on
this one.  My intention for the no overlay mode was that it would work
similar to what I'd expect 'git reset --hard -- <pathspec>' to work if
it existed, which means not removing untracked files if they exist.

While I think in the example you have above removing subdir/newfile
may be the right behaviour I'm not so sure in the case of 'git
checkout --no-overlay HEAD -- .' or ''git checkout --no-overlay HEAD
-- t/*' for example.  I don't think that should remove all untracked
files in the repository or in the t/ directory.  Removing untracked
files in that case would probably surprise users more than your case
above would.

I think it's okay to keep considering untracked files as special with
respect to how they are treated by 'git checkout --no-overlay'.

>                          The point of no-overlay is to make it match
> HEAD, and while subdir/newfile doesn't exist in HEAD or the index it
> does match in the working tree so the intent is clear. But let's say
> that the user did go ahead and specify your new flag:
> 
> 
>   $ git checkout --no-overlay --ignore-unmatch HEAD -- subdir/newfile
>   $ git status
>   On branch A
>   Untracked files:
>     (use "git add <file>..." to include in what will be committed)
> 
>       subdir/newfile
> 
>   nothing added to commit but untracked files present (use "git add" to track)
> 
> So now it avoids erroring out when the user does more work than
> necessary, but it still misses appropriately cleaning up the file.

Yeah this is a good point, this could be more confusing to the user
than the previous case in my opinion.  Maybe I'll just drop this patch
for now (and the next one, as it's better to hold of until stash in C
lands anyway), and then try to do all this in-core for 'git stash'.

Re: [PATCH 5/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 12/10, Duy Nguyen wrote:
> On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > @@ -302,15 +310,29 @@ static int checkout_paths(const struct checkout_opts *opts,
> >                 ce->ce_flags &= ~CE_MATCHED;
> >                 if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
> >                         continue;
> > -               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
> > -                       /*
> > -                        * "git checkout tree-ish -- path", but this entry
> > -                        * is in the original index; it will not be checked
> > -                        * out to the working tree and it does not matter
> > -                        * if pathspec matched this entry.  We will not do
> > -                        * anything to this entry at all.
> > -                        */
> > -                       continue;
> > +               if (opts->source_tree && !(ce->ce_flags & CE_UPDATE)) {
> > +                       if (!opts->overlay_mode &&
> > +                           ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
> > +                               /*
> > +                                * "git checkout --no-overlay <tree-ish> -- path",
> > +                                * and the path is not in tree-ish, but is in
> > +                                * the current index, which means that it should
> > +                                * be removed.
> > +                                */
> > +                               ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
> > +                               continue;
> > +                       } else {
> 
> In non-overlay mode but when pathspec does not match, we come here too.
> 
> > +                               /*
> > +                                * "git checkout tree-ish -- path", but this
> > +                                * entry is in the original index; it will not
> 
> I think the missing key point in this comment block is "..is in the
> original index _and it's not in tree-ish_". In non-overlay mode, if
> pathspec does not match then it's safe to ignore too. But this logic
> starts too get to complex and hurt my brain.

Yes, that would make it a bit easier to read. I took a while to try
and refactor this to make it easier to read, but couldn't come up with
anything much better unfortunately.  I'll have another stab at
simplifying the logic a bit for v2.

> > +                                * be checked out to the working tree and it
> > +                                * does not matter if pathspec matched this
> > +                                * entry.  We will not do anything to this entry
> > +                                * at all.
> > +                                */
> > +                               continue;
> > +                       }
> > +               }
> >                 /*
> >                  * Either this entry came from the tree-ish we are
> >                  * checking the paths out of, or we are checking out
> 
> > @@ -1266,6 +1299,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
> >                             "checkout", "control recursive updating of submodules",
> >                             PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
> >                 OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
> > +               OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
> 
> maybe add " (default)" to the help string.

Makes sense, will add.

> >                 OPT_END(),
> >         };
> >
> -- 
> Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Thomas Gummerer]

On 12/10, Elijah Newren wrote:
> On Sun, Dec 9, 2018 at 12:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Here's the series I mentioned a couple of times on the list already,
> > introducing a no-overlay mode in 'git checkout'.  The inspiration for
> > this came from Junios message in [*1*].
> >
> > Basically the idea is to also delete files when the match <pathspec>
> > in 'git checkout <tree-ish> -- <pathspec>' in the current tree, but
> > don't match <pathspec> in <tree-ish>.  The rest of the cases are
> > already properly taken care of by 'git checkout'.
> 
> Yes, but I'd put it a little differently:
> 
> """
> Basically, the idea is when the user run "git checkout --no-overlay
> <tree-ish> -- <pathspec>" that the given pathspecs should exactly
> match <tree-ish> after the operation completes.  This means that we
> also want to delete files that match <pathspec> if those paths are not
> found in <tree-ish>.
> """
> 
> ...and maybe even toss in some comments about the fact that this is
> the way git checkout should have always behaved, it just traditionally
> hasn't.  (You could also work in comments about how with this new mode
> the user can run git diff afterward with the given commit-ish and
> pathspecs and get back an empty diff, as expected, which wasn't true
> before.  But maybe I'm belaboring the point.)
> 
> 
> > The final step in the series is to actually make use of this in 'git
> > stash', which simplifies the code there a bit.  I am however happy to
> > hold off on this step until the stash-in-C series is merged, so we
> > don't delay that further.
> >
> > In addition to the no-overlay mode, we also add a --cached mode, which
> > works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.
> 
> If you're adding a --cached mode to make it only work on the index,
> should there be a similar mode to allow it to only work on the working
> tree?  (I'm not as concerned with that here, but I really think the
> new restore-files command by default should only operate on the
> working tree, and then have options to affect the index either in
> addition or instead of the working tree.)

Yeah I think that would be nice to have, though I'm not sure what we
would name it in 'git checkout'.  Maybe just having it in 'git
restore-files' is good enough?

> > Actually deprecating 'git reset <tree-ish> -- <pathspec>' should come
> > later, probably not before Duy's restore-files command lands, as 'git
> > checkout --no-overlay <tree-ish> -- <pathspec>' is a bit cumbersome to
> > type compared to 'git reset <tree-ish> -- <pathspec>'.
> 
> Makes sense.
> 
> > My hope is also that the no-overlay mode could become the new default
> > in the restore-files command Duy is currently working on.
> 
> Absolutely, yes.  I don't want another broken command.  :-)
> 
> 
> > No documentation yet, as I wanted to get this out for review first.
> > I'm not familiar with most of the code I touched here, so there may
> > well be much better ways to implement some of this, that I wasn't able
> > to figure out.  I'd be very happy with some feedback around that.
> >
> > Another thing I'm not sure about is how to deal with conflicts.  In
> > the cached mode this patch series is not dealing with it at all, as
> > 'git checkout -- <pathspec>' when pathspec matches a file with
> > conflicts doesn't update the index.  For the no-overlay mode, the file
> > is removed if the corresponding stage is not found in the index.  I'm
> > however not sure this is the right thing to do in all cases?
> 
> Here's how I'd go about analyzing that...
> 
> If the user passes a <tree-ish>, then the answer about what to do is
> pretty obvious; the <tree-ish> didn't have conflicts, so conflicted
> paths in the index that match the pathspec should be overwritten with
> whatever version of those paths existed in <tree-ish> (possibly
> implying deletion of some paths).
> 
> Also, as you point out, --cached means only modify the index and not
> the working tree; so if they specify both --cached and provide no
> tree, then they've specified a no-op.
> 
> So it's only interesting when you have conflicts in the index and
> specify --no-overlay without a <tree-ish> or --cached.  This boils
> down to "how do we update the working tree to match the index, when
> the index is conflicted?"  A couple points to consider:
>   * This is somewhat of an edge case
>   * In the normal case --no-overlay is only different from --overlay
> behavior for directories; it'd be nice if that extended to all cases

I'm not sure I follow what you mean here.  How is --no-overlay
different from --overlay with respect to directories?  It's only
different with respect to deletions, no?

>   * How does this command behave without a <tree-ish> when
> --no-overlay is specified and a directory is given for a <pathspec>
> and there aren't any conflicts?  Are we being consistent with that
> behavior?
> 
> 
> However, I think it turns out that the answer is much simpler than all
> that initial analysis or what you say you've implemented.  Here's why:
> 
> If <pathspec> is a file which is present in both the working tree and
> the index and it has conflicts, then "git checkout -- <pathspec>" will
> currently throw an error:

I think what was missing from my original description, that actually
makes it slightly more interesting from what you describe below is the
'--ours' and '--theirs' flags in 'git checkout', with which one can
check out a version of the file in the working tree.  This is where it
gets more interesting.

I think I got the right solution for that in patch 5, with deleting
the file if it's deleted in "their" version and we pass --theirs to
'git checkout', and analogous for --ours.  I was just wondering if
there were any further edge cases that I can't think of right no.

> $ git checkout -- subdir/counting
> error: path 'subdir/counting' is unmerged
> 
> In fact, even if every entry in subdir/ is a path that is in both the
> index and the working tree (so that --no-overlay and --overlay ought
> to behave the same), if any one of the files in subdir is conflicted,
> attempting to checkout the subdir will abort with this same error
> message and no paths will be updated at all:
> 
> $ git checkout -- subdir
> error: path 'subdir/counting' is unmerged
> 
> as such, the answer with what to do with --no-overlay mode is pretty
> clear: if the <pathspec> matches _any_ path that is conflicted, simply
> throw an error and abort the operation without making any changes at
> all.

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Duy Nguyen]

On Tue, Dec 11, 2018 at 3:28 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Duy Nguyen <pclouds@gmail.com> writes:
>
> >> +       if (ce->ce_flags & CE_WT_REMOVE) {
> >> +               if (topath)
> >> +                       BUG("Can't remove entry to a path");
> >> +               unlink_entry(ce);
> >> +               return 0;
> >> +       }
> >
> > This makes the path counting in nd/checkout-noisy less accurate. But
> > it's not your fault of course.
>
> When we check out absense of one path, how do we want to count it?
> Do we say "one path checked out?" when we remove one path?

It is still "checked out" according to this non-overlay concept.
Although we could make it clear by saying "5 paths updated, 2 deleted"
(but that may make us say "3 paths added" as well, hmm). Or maybe just
"%d paths updated" where updates include file creation and deletion.
-- 
Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> I think I got the right solution for that in patch 5, with deleting
> the file if it's deleted in "their" version and we pass --theirs to
> 'git checkout', and analogous for --ours.  I was just wondering if
> there were any further edge cases that I can't think of right no.

That sounds quite sensible.

Re: [PATCH 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> Although we could make it clear by saying "5 paths updated, 2 deleted"
> (but that may make us say "3 paths added" as well, hmm). Or maybe just
> "%d paths updated" where updates include file creation and deletion.

Yeah, the last one is the simplest and good.

Re: [PATCH 1/8] move worktree tests to t24*

[Eric Sunshine]

On Tue, Dec 11, 2018 at 4:50 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> On 12/10, Duy Nguyen wrote:
> > On Sun, Dec 9, 2018 at 9:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > > Move the worktree tests to t24* to adhere to that guideline. We're
> > > going to make use of the free'd up numbers in a subsequent commit.
> >
> > Heh.. I did the same thing (in my unsent switch-branch/restore-files
> > series) and even used the same 24xx range :D You probably want to move
> > t2028 and t2029 too (not sure if they have landed on 'master')
>
> [...] good to know someone
> thought the same way.  I started this work before t2028 and t2029
> landed on master, so I failed to notice them.

The thought of renumbering the test script came up as early as
2015-06-30. See the last bullet point of [1], for instance.

[1]: https://public-inbox.org/git/1435640202-95945-1-git-send-email-sunshine@sunshineco.com/

Re: [PATCH 1/8] move worktree tests to t24*

[Duy Nguyen]

On Wed, Dec 12, 2018 at 2:27 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Tue, Dec 11, 2018 at 4:50 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > On 12/10, Duy Nguyen wrote:
> > > On Sun, Dec 9, 2018 at 9:04 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > > > Move the worktree tests to t24* to adhere to that guideline. We're
> > > > going to make use of the free'd up numbers in a subsequent commit.
> > >
> > > Heh.. I did the same thing (in my unsent switch-branch/restore-files
> > > series) and even used the same 24xx range :D You probably want to move
> > > t2028 and t2029 too (not sure if they have landed on 'master')
> >
> > [...] good to know someone
> > thought the same way.  I started this work before t2028 and t2029
> > landed on master, so I failed to notice them.
>
> The thought of renumbering the test script came up as early as
> 2015-06-30. See the last bullet point of [1], for instance.
>
> [1]: https://public-inbox.org/git/1435640202-95945-1-git-send-email-sunshine@sunshineco.com/

Ah good. I thought I was just being lazy and picked a random range to add tests.
-- 
Duy

Re: [PATCH 0/8] introduce no-overlay and cached mode in git checkout

[Thomas Gummerer]

On 12/10, Junio C Hamano wrote:
> Thomas Gummerer <t.gummerer@gmail.com> writes:
> 
> > Basically the idea is to also delete files when the match <pathspec>
> > in 'git checkout <tree-ish> -- <pathspec>' in the current tree, but
> > don't match <pathspec> in <tree-ish>.
> 
> I cannot quite parse it, but perhaps.
> 
> 	"git checkout --no-overlay <tree-ish> -- <pathspec>" can
> 	remove paths in the index and in the working tree that match
> 	<pathspec>, if they do not appear in <tree-ish>.
> 
> If a new file D/F is in the index and in the working tree but not in
> HEAD, "git checkout HEAD D/" or "git checkout HEAD D/F" would not
> remove D/F from the index or the working tree.
> 
> With the --no-overlay option, it would, and that is often closer to
> the wish of the user who wanted to say "restore the working tree
> state of D/ (or D/F) from the state recorded in HEAD".

Yeah thanks, that reads much better.

> > The final step in the series is to actually make use of this in 'git
> > stash', which simplifies the code there a bit.  I am however happy to
> > hold off on this step until the stash-in-C series is merged, so we
> > don't delay that further.
> 
> I think that is probably a good idea, for now.
> 
> > In addition to the no-overlay mode, we also add a --cached mode, which
> > works only on the index, thus similar to 'git reset <tree-ish> -- <pathspec>'.
> >
> > Actually deprecating 'git reset <tree-ish> -- <pathspec>' should come
> > later,...
> 
> Or we may not even need to deprecate it.  IIRC, what "stash" wished
> to exist was "git reset --hard <tree-ish> -- <pathspec>", which, if
> the command followed "--cached/--index" convention, would have been
> called "git reset --index ...".  Did we actually have the need for
> "--cached" mode?

I don't have a pressing need for "--cached".  I mainly included it
because you described "git reset <tree-ish> -- <pathspec>" as bad UI
in the original thread [*1*], which after reading that message I agree
with.  It also seemed to cause some confusion in [*2*].  Since it was
fairly easy to introduce a "--cached" mode it seemed like a potential
UI improvement in the long term to deprecate 'git reset <tree-ish> --
<pathspec>'.

That said, this series is probably not the right place to introduce
this feature, as it should mainly be focused on the no-overlay mode.
I'll drop the patch from v2.

We can revisit whether we want to introduce a "--cached" mode in 'git
checkout' at some point in the future.  

> > probably not before Duy's restore-files command lands, as 'git
> > checkout --no-overlay <tree-ish> -- <pathspec>' is a bit cumbersome to
> > type compared to 'git reset <tree-ish> -- <pathspec>'.
> 
> Yes, between "checkout --cached" and "checkout --no-overlay", the
> latter is much more important, as the latter is what a missing "git
> reset --hard <tree-ish> -- <pathspec>" would have been, but the
> former can be written with an existing command.
> 
> > My hope is also that the no-overlay mode could become the new default
> > in the restore-files command Duy is currently working on.
> 
> Yup, that is my hope, too ;-).

*1*: <xmqq4loqplou.fsf@gitster.mtv.corp.google.com>
*2*: <alpine.LFD.2.21.1812081103500.29142@localhost.localdomain>

Re: [PATCH 2/8] entry: factor out unlink_entry function

[Thomas Gummerer]

On 12/10, Duy Nguyen wrote:
> On Sun, Dec 9, 2018 at 9:05 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> >
> > Factor out the 'unlink_entry()' function from unpack-trees.c to
> > entry.c.  It will be used in other places as well in subsequent
> > steps.
> >
> > As it's no longer a static function, also move the documentation to
> > the header file to make it more discoverable.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >  cache.h        |  5 +++++
> >  entry.c        | 15 +++++++++++++++
> >  unpack-trees.c | 19 -------------------
> >  3 files changed, 20 insertions(+), 19 deletions(-)
> >
> > diff --git a/cache.h b/cache.h
> > index ca36b44ee0..c1c953e810 100644
> > --- a/cache.h
> > +++ b/cache.h
> > @@ -1542,6 +1542,11 @@ struct checkout {
> >  extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
> >  extern void enable_delayed_checkout(struct checkout *state);
> >  extern int finish_delayed_checkout(struct checkout *state);
> > +/*
> > + * Unlink the last component and schedule the leading directories for
> > + * removal, such that empty directories get removed.
> > + */
> > +extern void unlink_entry(const struct cache_entry *ce);
> 
> I'm torn. We try to remove 'extern' but I can see you may want to add
> it here to be consistent with others. And removing extern even from
> functions from entry.c only would cause some conflicts.

Yeah I felt like favoring consistency here would be better.  Once your
path counting series and my series land, this may get quieter and we
can remove the 'extern' then?

> I wonder if we should move the 'removal' variable in symlinks to
> 'struct checkout' to reduce another global variable. But I guess
> that's the problem for another day. It's not the focus of this
> series.

Yeah, I'd prefer leaving that for another day :)

> -- 
> Duy

[PATCH v2 0/8] introduce no-overlay mode in git checkout

[Thomas Gummerer]

Previous round is at <20181209200449.16342-1-t.gummerer@gmail.com>.

Thanks Junio, Duy and Elijah for your comments and suggestions on the
previous round.

This round drops the last three patches from the previous round,
namely introducing a "--cached" and a "--ignore-unmatched" option, and
using the new no-overlay mode in "git stash".  The --ignore-unmatched
option may not be necessary, while using the new mode in 'git stash'
will be done once the stash-in-C topic landed.

Introducing a --cached and --worktree-only (as suggested by Elijah)
option can come in a future step, they are orthogonal to this topic.

Other changes from v1:
- Rebase onto the current master, so we can also move t2028 and t2029
  to the t24xx range.
- Add a comment clarifying why using the CE_WT_REMOVE flag and topath
  in checkout_entry is a bug.
- clarify a comment in checkout.c
- factor out the function to mark a cache entry as CE_MATCHED, and
  have separate such functions for overlay mode and no-overlay mode.
  This should hopefully make the logic a bit easier to follow.
- Adjust the commit message, justifying why we don't remove untracked
  files even in the new no-overlay mode.
- add documentation for the new feature
- document that -p defaults to no overlay mode, and cannot be used
  with overlay mode.
- add a config option checkout.overlayMode, so overlay mode can be
  turned on by default.

Range-diff can be found after the diffstat.

Thomas Gummerer (8):
  move worktree tests to t24*
  entry: factor out unlink_entry function
  entry: support CE_WT_REMOVE flag in checkout_entry
  read-cache: add invalidate parameter to remove_marked_cache_entries
  checkout: clarify comment
  checkout: factor out mark_cache_entry_for_checkout function
  checkout: introduce --{,no-}overlay option
  checkout: introduce checkout.overlayMode config

 Documentation/config/checkout.txt             |   7 +
 Documentation/git-checkout.txt                |  10 ++
 builtin/checkout.c                            | 133 +++++++++++++-----
 cache.h                                       |   7 +-
 entry.c                                       |  26 ++++
 read-cache.c                                  |   8 +-
 split-index.c                                 |   2 +-
 t/t2025-checkout-no-overlay.sh                |  57 ++++++++
 ...-worktree-add.sh => t2400-worktree-add.sh} |   0
 ...ktree-prune.sh => t2401-worktree-prune.sh} |   0
 ...orktree-list.sh => t2402-worktree-list.sh} |   0
 ...orktree-move.sh => t2403-worktree-move.sh} |   0
 ...ree-config.sh => t2404-worktree-config.sh} |   0
 t/t9902-completion.sh                         |   1 +
 unpack-trees.c                                |  21 +--
 15 files changed, 213 insertions(+), 59 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
 rename t/{t2028-worktree-move.sh => t2403-worktree-move.sh} (100%)
 rename t/{t2029-worktree-config.sh => t2404-worktree-config.sh} (100%)

1:  70bd75b202 ! 1:  fa450cda7c move worktree tests to t24*
    @@ -29,3 +29,13 @@
      similarity index 100%
      rename from t/t2027-worktree-list.sh
      rename to t/t2402-worktree-list.sh
    +
    + diff --git a/t/t2028-worktree-move.sh b/t/t2403-worktree-move.sh
    + similarity index 100%
    + rename from t/t2028-worktree-move.sh
    + rename to t/t2403-worktree-move.sh
    +
    + diff --git a/t/t2029-worktree-config.sh b/t/t2404-worktree-config.sh
    + similarity index 100%
    + rename from t/t2029-worktree-config.sh
    + rename to t/t2404-worktree-config.sh
2:  0fd9be987d = 2:  9ada8d3484 entry: factor out unlink_entry function
3:  4d6112b112 ! 3:  41c0ea4047 entry: support CE_WT_REMOVE flag in checkout_entry
    @@ -22,6 +22,10 @@
      
     +	if (ce->ce_flags & CE_WT_REMOVE) {
     +		if (topath)
    ++			/*
    ++			 * No content and thus no path to create, so we have
    ++			 * no pathname to return.
    ++			 */
     +			BUG("Can't remove entry to a path");
     +		unlink_entry(ce);
     +		return 0;
4:  6e9f68b8f1 ! 4:  afccb0848d read-cache: add invalidate parameter to remove_marked_cache_entries
    @@ -11,6 +11,10 @@
         function will take care of invalidating the path in the cache tree and
         in the untracked cache.
     
    +    Note that the current callsites already do the invalidation properly
    +    in other places, so we're just passing 0 from there to keep the status
    +    quo.
    +
         This will be useful in a subsequent commit.
     
         Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
-:  ---------- > 5:  8a2b5efdad checkout: clarify comment
-:  ---------- > 6:  c405f20471 checkout: factor out mark_cache_entry_for_checkout function
5:  4a7670d34c ! 7:  e5b18bcd02 checkout: introduce --{,no-}overlay option
    @@ -17,8 +17,43 @@
         'git checkout --overlay -p' to avoid confusing users who would expect
         to be able to force overlay mode in 'git checkout -p' this way.
     
    +    Untracked files are not affected by this change, so 'git checkout
    +    --no-overlay HEAD -- untracked' will not remove untracked from the
    +    working tree.  This is so e.g. 'git checkout --no-overlay HEAD -- dir/'
    +    doesn't delete all untracked files in dir/, but rather just resets the
    +    state of files that are known to git.
    +
    +    Suggested-by: Junio C Hamano <gitster@pobox.com>
         Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
     
    + diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
    + --- a/Documentation/git-checkout.txt
    + +++ b/Documentation/git-checkout.txt
    +@@
    + This means that you can use `git checkout -p` to selectively discard
    + edits from your current working tree. See the ``Interactive Mode''
    + section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
    +++
    ++Note that this option uses the no overlay mode by default (see also
    ++-`--[no-]overlay`), and currently doesn't support overlay mode.
    + 
    + --ignore-other-worktrees::
    + 	`git checkout` refuses when the wanted ref is already checked
    +@@
    + 	Just like linkgit:git-submodule[1], this will detach the
    + 	submodules HEAD.
    + 
    ++--[no-]overlay::
    ++	In the default overlay mode files `git checkout` never
    ++	removes files from the index or the working tree.  When
    ++	specifying --no-overlay, files that appear in the index and
    ++	working tree, but not in <tree-ish> are removed, to make them
    ++	match <tree-ish> exactly.
    ++
    + <branch>::
    + 	Branch to checkout; if it refers to a branch (i.e., a name that,
    + 	when prepended with "refs/heads/", is a valid ref), then that
    +
      diff --git a/builtin/checkout.c b/builtin/checkout.c
      --- a/builtin/checkout.c
      +++ b/builtin/checkout.c
    @@ -70,44 +105,60 @@
      		return error(_("path '%s' does not have our version"), ce->name);
      	else
     @@
    - 		ce->ce_flags &= ~CE_MATCHED;
    - 		if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
    - 			continue;
    --		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
    --			/*
    --			 * "git checkout tree-ish -- path", but this entry
    --			 * is in the original index; it will not be checked
    --			 * out to the working tree and it does not matter
    --			 * if pathspec matched this entry.  We will not do
    --			 * anything to this entry at all.
    --			 */
    --			continue;
    -+		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE)) {
    -+			if (!opts->overlay_mode &&
    -+			    ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
    -+				/*
    -+				 * "git checkout --no-overlay <tree-ish> -- path",
    -+				 * and the path is not in tree-ish, but is in
    -+				 * the current index, which means that it should 
    -+				 * be removed.
    -+				 */
    -+				ce->ce_flags |= CE_MATCHED | CE_REMOVE | CE_WT_REMOVE;
    -+				continue;
    -+			} else {
    -+				/*
    -+				 * "git checkout tree-ish -- path", but this
    -+				 * entry is in the original index; it will not
    -+				 * be checked out to the working tree and it
    -+				 * does not matter if pathspec matched this
    -+				 * entry.  We will not do anything to this entry
    -+				 * at all.
    -+				 */
    -+				continue;
    -+			}
    -+		}
    - 		/*
    - 		 * Either this entry came from the tree-ish we are
    - 		 * checking the paths out of, or we are checking out
    + 	return status;
    + }
    + 
    +-static void mark_ce_for_checkout(struct cache_entry *ce,
    +-				 char *ps_matched,
    +-				 const struct checkout_opts *opts)
    ++static void mark_ce_for_checkout_overlay(struct cache_entry *ce,
    ++					 char *ps_matched,
    ++					 const struct checkout_opts *opts)
    + {
    + 	ce->ce_flags &= ~CE_MATCHED;
    + 	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
    +@@
    + 		ce->ce_flags |= CE_MATCHED;
    + }
    + 
    ++static void mark_ce_for_checkout_no_overlay(struct cache_entry *ce,
    ++					    char *ps_matched,
    ++					    const struct checkout_opts *opts)
    ++{
    ++	ce->ce_flags &= ~CE_MATCHED;
    ++	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
    ++		return;
    ++	if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
    ++		ce->ce_flags |= CE_MATCHED;
    ++		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
    ++			/*
    ++			 * In overlay mode, but the path is not in
    ++			 * tree-ish, which means we should remove it
    ++			 * from the index and the working tree.
    ++			 */
    ++			ce->ce_flags |= CE_REMOVE | CE_WT_REMOVE;
    ++	}
    ++}
    ++
    + static int checkout_paths(const struct checkout_opts *opts,
    + 			  const char *revision)
    + {
    +@@
    + 	 * to be checked out.
    + 	 */
    + 	for (pos = 0; pos < active_nr; pos++)
    +-		mark_ce_for_checkout(active_cache[pos], ps_matched, opts);
    ++		if (opts->overlay_mode)
    ++			mark_ce_for_checkout_overlay(active_cache[pos],
    ++						     ps_matched,
    ++						     opts);
    ++		else
    ++			mark_ce_for_checkout_no_overlay(active_cache[pos],
    ++							ps_matched,
    ++							opts);
    + 
    + 	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
    + 		free(ps_matched);
     @@
      			if (opts->force) {
      				warning(_("path '%s' is unmerged"), ce->name);
    @@ -160,7 +211,7 @@
      			    "checkout", "control recursive updating of submodules",
      			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
      		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
    -+		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode")),
    ++		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode (default)")),
      		OPT_END(),
      	};
      
    @@ -198,7 +249,7 @@
     +	git commit --allow-empty -m "initial"
     +'
     +
    -+test_expect_success 'checkout --no-overlay deletes files not in <tree>' '
    ++test_expect_success 'checkout --no-overlay deletes files not in <tree-ish>' '
     +	>file &&
     +	mkdir dir &&
     +	>dir/file1 &&
    @@ -218,7 +269,7 @@
     +	test_i18ngrep "fatal: -p and --overlay are mutually exclusive" actual
     +'
     +
    -+test_expect_success '--no-overlay --theirs with M/D conflict deletes file' '
    ++test_expect_success '--no-overlay --theirs with D/F conflict deletes file' '
     +	test_commit file1 file1 &&
     +	test_commit file2 file2 &&
     +	git rm --cached file1 &&
6:  695b671675 < -:  ---------- checkout: add --cached option
7:  d0b5a356b2 < -:  ---------- checkout: allow ignoring unmatched pathspec
8:  0a4565acc1 < -:  ---------- stash: use git checkout --no-overlay
-:  ---------- > 8:  de24990d57 checkout: introduce checkout.overlayMode config

-- 
2.20.1.415.g653613c723

[PATCH v2 1/8] move worktree tests to t24*

[Thomas Gummerer]

The 'git worktree' command used to be just another mode in 'git
checkout', namely 'git checkout --to'.  When the tests for the latter
were retrofitted for the former, the test name was adjusted, but the
test number was kept, even though the test is testing a different
command now.  t/README states: "Second digit tells the particular
command we are testing.", so 'git worktree' should have a separate
number just for itself.

Move the worktree tests to t24* to adhere to that guideline. We're
going to make use of the free'd up numbers in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 t/{t2025-worktree-add.sh => t2400-worktree-add.sh}       | 0
 t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh}   | 0
 t/{t2027-worktree-list.sh => t2402-worktree-list.sh}     | 0
 t/{t2028-worktree-move.sh => t2403-worktree-move.sh}     | 0
 t/{t2029-worktree-config.sh => t2404-worktree-config.sh} | 0
 5 files changed, 0 insertions(+), 0 deletions(-)
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
 rename t/{t2028-worktree-move.sh => t2403-worktree-move.sh} (100%)
 rename t/{t2029-worktree-config.sh => t2404-worktree-config.sh} (100%)

diff --git a/t/t2025-worktree-add.sh b/t/t2400-worktree-add.sh
similarity index 100%
rename from t/t2025-worktree-add.sh
rename to t/t2400-worktree-add.sh
diff --git a/t/t2026-worktree-prune.sh b/t/t2401-worktree-prune.sh
similarity index 100%
rename from t/t2026-worktree-prune.sh
rename to t/t2401-worktree-prune.sh
diff --git a/t/t2027-worktree-list.sh b/t/t2402-worktree-list.sh
similarity index 100%
rename from t/t2027-worktree-list.sh
rename to t/t2402-worktree-list.sh
diff --git a/t/t2028-worktree-move.sh b/t/t2403-worktree-move.sh
similarity index 100%
rename from t/t2028-worktree-move.sh
rename to t/t2403-worktree-move.sh
diff --git a/t/t2029-worktree-config.sh b/t/t2404-worktree-config.sh
similarity index 100%
rename from t/t2029-worktree-config.sh
rename to t/t2404-worktree-config.sh
-- 
2.20.1.415.g653613c723

[PATCH v2 2/8] entry: factor out unlink_entry function

[Thomas Gummerer]

Factor out the 'unlink_entry()' function from unpack-trees.c to
entry.c.  It will be used in other places as well in subsequent
steps.

As it's no longer a static function, also move the documentation to
the header file to make it more discoverable.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 cache.h        |  5 +++++
 entry.c        | 15 +++++++++++++++
 unpack-trees.c | 19 -------------------
 3 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/cache.h b/cache.h
index ca36b44ee0..c1c953e810 100644
--- a/cache.h
+++ b/cache.h
@@ -1542,6 +1542,11 @@ struct checkout {
 extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
 extern void enable_delayed_checkout(struct checkout *state);
 extern int finish_delayed_checkout(struct checkout *state);
+/*
+ * Unlink the last component and schedule the leading directories for
+ * removal, such that empty directories get removed.
+ */
+extern void unlink_entry(const struct cache_entry *ce);
 
 struct cache_def {
 	struct strbuf path;
diff --git a/entry.c b/entry.c
index 0a3c451f5f..b9eef57117 100644
--- a/entry.c
+++ b/entry.c
@@ -508,3 +508,18 @@ int checkout_entry(struct cache_entry *ce,
 	create_directories(path.buf, path.len, state);
 	return write_entry(ce, path.buf, state, 0);
 }
+
+void unlink_entry(const struct cache_entry *ce)
+{
+	const struct submodule *sub = submodule_from_ce(ce);
+	if (sub) {
+		/* state.force is set at the caller. */
+		submodule_move_head(ce->name, "HEAD", NULL,
+				    SUBMODULE_MOVE_HEAD_FORCE);
+	}
+	if (!check_leading_path(ce->name, ce_namelen(ce)))
+		return;
+	if (remove_or_warn(ce->ce_mode, ce->name))
+		return;
+	schedule_dir_for_removal(ce->name, ce_namelen(ce));
+}
diff --git a/unpack-trees.c b/unpack-trees.c
index 7570df481b..e8d1a6ac50 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -300,25 +300,6 @@ static void load_gitmodules_file(struct index_state *index,
 	}
 }
 
-/*
- * Unlink the last component and schedule the leading directories for
- * removal, such that empty directories get removed.
- */
-static void unlink_entry(const struct cache_entry *ce)
-{
-	const struct submodule *sub = submodule_from_ce(ce);
-	if (sub) {
-		/* state.force is set at the caller. */
-		submodule_move_head(ce->name, "HEAD", NULL,
-				    SUBMODULE_MOVE_HEAD_FORCE);
-	}
-	if (!check_leading_path(ce->name, ce_namelen(ce)))
-		return;
-	if (remove_or_warn(ce->ce_mode, ce->name))
-		return;
-	schedule_dir_for_removal(ce->name, ce_namelen(ce));
-}
-
 static struct progress *get_progress(struct unpack_trees_options *o)
 {
 	unsigned cnt = 0, total = 0;
-- 
2.20.1.415.g653613c723

[PATCH v2 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Thomas Gummerer]

'checkout_entry()' currently only supports creating new entries in the
working tree, but not deleting them.  Add the ability to remove
entries at the same time if the entry is marked with the CE_WT_REMOVE
flag.

Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
only used in unpack-tree, however we will make use of this in a
subsequent step in the series.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 entry.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/entry.c b/entry.c
index b9eef57117..3d3701e7ae 100644
--- a/entry.c
+++ b/entry.c
@@ -441,6 +441,17 @@ int checkout_entry(struct cache_entry *ce,
 	static struct strbuf path = STRBUF_INIT;
 	struct stat st;
 
+	if (ce->ce_flags & CE_WT_REMOVE) {
+		if (topath)
+			/*
+			 * No content and thus no path to create, so we have
+			 * no pathname to return.
+			 */
+			BUG("Can't remove entry to a path");
+		unlink_entry(ce);
+		return 0;
+	}
+
 	if (topath)
 		return write_entry(ce, topath, state, 1);
 
-- 
2.20.1.415.g653613c723

[PATCH v2 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Thomas Gummerer]

When marking cache entries for removal, and later removing them all at
once using 'remove_marked_cache_entries()', cache entries currently
have to be invalidated manually in the cache tree and in the untracked
cache.

Add an invalidate flag to the function.  With the flag set, the
function will take care of invalidating the path in the cache tree and
in the untracked cache.

Note that the current callsites already do the invalidation properly
in other places, so we're just passing 0 from there to keep the status
quo.

This will be useful in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 cache.h        | 2 +-
 read-cache.c   | 8 +++++++-
 split-index.c  | 2 +-
 unpack-trees.c | 2 +-
 4 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index c1c953e810..1deee48f5b 100644
--- a/cache.h
+++ b/cache.h
@@ -751,7 +751,7 @@ extern void rename_index_entry_at(struct index_state *, int pos, const char *new
 /* Remove entry, return true if there are more entries to go. */
 extern int remove_index_entry_at(struct index_state *, int pos);
 
-extern void remove_marked_cache_entries(struct index_state *istate);
+extern void remove_marked_cache_entries(struct index_state *istate, int invalidate);
 extern int remove_file_from_index(struct index_state *, const char *path);
 #define ADD_CACHE_VERBOSE 1
 #define ADD_CACHE_PRETEND 2
diff --git a/read-cache.c b/read-cache.c
index bd45dc3e24..978d43f676 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -590,13 +590,19 @@ int remove_index_entry_at(struct index_state *istate, int pos)
  * CE_REMOVE is set in ce_flags.  This is much more effective than
  * calling remove_index_entry_at() for each entry to be removed.
  */
-void remove_marked_cache_entries(struct index_state *istate)
+void remove_marked_cache_entries(struct index_state *istate, int invalidate)
 {
 	struct cache_entry **ce_array = istate->cache;
 	unsigned int i, j;
 
 	for (i = j = 0; i < istate->cache_nr; i++) {
 		if (ce_array[i]->ce_flags & CE_REMOVE) {
+			if (invalidate) {
+				cache_tree_invalidate_path(istate,
+							   ce_array[i]->name);
+				untracked_cache_remove_from_index(istate,
+								  ce_array[i]->name);
+			}
 			remove_name_hash(istate, ce_array[i]);
 			save_or_free_index_entry(istate, ce_array[i]);
 		}
diff --git a/split-index.c b/split-index.c
index 5820412dc5..8aebc3661b 100644
--- a/split-index.c
+++ b/split-index.c
@@ -162,7 +162,7 @@ void merge_base_index(struct index_state *istate)
 	ewah_each_bit(si->replace_bitmap, replace_entry, istate);
 	ewah_each_bit(si->delete_bitmap, mark_entry_for_delete, istate);
 	if (si->nr_deletions)
-		remove_marked_cache_entries(istate);
+		remove_marked_cache_entries(istate, 0);
 
 	for (i = si->nr_replacements; i < si->saved_cache_nr; i++) {
 		if (!ce_namelen(si->saved_cache[i]))
diff --git a/unpack-trees.c b/unpack-trees.c
index e8d1a6ac50..8e6afa924d 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -392,7 +392,7 @@ static int check_updates(struct unpack_trees_options *o)
 				unlink_entry(ce);
 		}
 	}
-	remove_marked_cache_entries(index);
+	remove_marked_cache_entries(index, 0);
 	remove_scheduled_dirs();
 
 	if (should_update_submodules() && o->update && !o->dry_run)
-- 
2.20.1.415.g653613c723

[PATCH v2 5/8] checkout: clarify comment

[Thomas Gummerer]

The key point for the if statement is that read_tree_some did not
update the entry, because either it doesn't exist in tree-ish or
doesn't match the pathspec.  Clarify that.

Suggested-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index acdafc6e4c..cb166b2e07 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -304,10 +304,10 @@ static int checkout_paths(const struct checkout_opts *opts,
 			continue;
 		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
 			/*
-			 * "git checkout tree-ish -- path", but this entry
-			 * is in the original index; it will not be checked
-			 * out to the working tree and it does not matter
-			 * if pathspec matched this entry.  We will not do
+			 * "git checkout tree-ish -- path" and this entry
+			 * is in the original index, but is not in tree-ish
+			 * or does not match the pathspec; it will not be
+			 * checked out to the working tree.  We will not do
 			 * anything to this entry at all.
 			 */
 			continue;
-- 
2.20.1.415.g653613c723

[PATCH v2 6/8] checkout: factor out mark_cache_entry_for_checkout function

[Thomas Gummerer]

Factor out the code that marks a cache entry as matched for checkout
into a separate function.  We are going to introduce a new mode in
'git checkout' in a subsequent commit, that is going to have a
slightly different logic.  This would make this code unnecessarily
complex.

Moving that complexity into separate functions will make the code in
the subsequent step easier to follow.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c | 67 +++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index cb166b2e07..32c4b7f897 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -247,6 +247,40 @@ static int checkout_merged(int pos, const struct checkout *state)
 	return status;
 }
 
+static void mark_ce_for_checkout(struct cache_entry *ce,
+				 char *ps_matched,
+				 const struct checkout_opts *opts)
+{
+	ce->ce_flags &= ~CE_MATCHED;
+	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
+		return;
+	if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
+		/*
+		 * "git checkout tree-ish -- path", but this entry
+		 * is in the original index but is not in tree-ish
+		 * or does not match the pathspec; it will not be
+		 * checked out to the working tree.  We will not do
+		 * anything to this entry at all.
+		 */
+		return;
+	/*
+	 * Either this entry came from the tree-ish we are
+	 * checking the paths out of, or we are checking out
+	 * of the index.
+	 *
+	 * If it comes from the tree-ish, we already know it
+	 * matches the pathspec and could just stamp
+	 * CE_MATCHED to it from update_some(). But we still
+	 * need ps_matched and read_tree_recursive (and
+	 * eventually tree_entry_interesting) cannot fill
+	 * ps_matched yet. Once it can, we can avoid calling
+	 * match_pathspec() for _all_ entries when
+	 * opts->source_tree != NULL.
+	 */
+	if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched))
+		ce->ce_flags |= CE_MATCHED;
+}
+
 static int checkout_paths(const struct checkout_opts *opts,
 			  const char *revision)
 {
@@ -297,37 +331,8 @@ static int checkout_paths(const struct checkout_opts *opts,
 	 * Make sure all pathspecs participated in locating the paths
 	 * to be checked out.
 	 */
-	for (pos = 0; pos < active_nr; pos++) {
-		struct cache_entry *ce = active_cache[pos];
-		ce->ce_flags &= ~CE_MATCHED;
-		if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
-			continue;
-		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
-			/*
-			 * "git checkout tree-ish -- path" and this entry
-			 * is in the original index, but is not in tree-ish
-			 * or does not match the pathspec; it will not be
-			 * checked out to the working tree.  We will not do
-			 * anything to this entry at all.
-			 */
-			continue;
-		/*
-		 * Either this entry came from the tree-ish we are
-		 * checking the paths out of, or we are checking out
-		 * of the index.
-		 *
-		 * If it comes from the tree-ish, we already know it
-		 * matches the pathspec and could just stamp
-		 * CE_MATCHED to it from update_some(). But we still
-		 * need ps_matched and read_tree_recursive (and
-		 * eventually tree_entry_interesting) cannot fill
-		 * ps_matched yet. Once it can, we can avoid calling
-		 * match_pathspec() for _all_ entries when
-		 * opts->source_tree != NULL.
-		 */
-		if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched))
-			ce->ce_flags |= CE_MATCHED;
-	}
+	for (pos = 0; pos < active_nr; pos++)
+		mark_ce_for_checkout(active_cache[pos], ps_matched, opts);
 
 	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
 		free(ps_matched);
-- 
2.20.1.415.g653613c723

[PATCH v2 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

Currently 'git checkout' is defined as an overlay operation, which
means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
entry in the index that matches <pathspec>, but that doesn't exist in
<tree-ish>, that entry will not be removed from the index or the
working tree.

Introduce a new --{,no-}overlay option, which allows using 'git
checkout' in non-overlay mode, thus removing files from the working
tree if they do not exist in <tree-ish> but match <pathspec>.

Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
this way, so no changes are needed for the patch mode.  We disallow
'git checkout --overlay -p' to avoid confusing users who would expect
to be able to force overlay mode in 'git checkout -p' this way.

Untracked files are not affected by this change, so 'git checkout
--no-overlay HEAD -- untracked' will not remove untracked from the
working tree.  This is so e.g. 'git checkout --no-overlay HEAD -- dir/'
doesn't delete all untracked files in dir/, but rather just resets the
state of files that are known to git.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 Documentation/git-checkout.txt | 10 ++++++
 builtin/checkout.c             | 66 +++++++++++++++++++++++++++++-----
 t/t2025-checkout-no-overlay.sh | 47 ++++++++++++++++++++++++
 t/t9902-completion.sh          |  1 +
 4 files changed, 116 insertions(+), 8 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 801de2f764..4ac8c55865 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -260,6 +260,9 @@ the conflicted merge in the specified paths.
 This means that you can use `git checkout -p` to selectively discard
 edits from your current working tree. See the ``Interactive Mode''
 section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
++
+Note that this option uses the no overlay mode by default (see also
+-`--[no-]overlay`), and currently doesn't support overlay mode.
 
 --ignore-other-worktrees::
 	`git checkout` refuses when the wanted ref is already checked
@@ -276,6 +279,13 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
 	Just like linkgit:git-submodule[1], this will detach the
 	submodules HEAD.
 
+--[no-]overlay::
+	In the default overlay mode files `git checkout` never
+	removes files from the index or the working tree.  When
+	specifying --no-overlay, files that appear in the index and
+	working tree, but not in <tree-ish> are removed, to make them
+	match <tree-ish> exactly.
+
 <branch>::
 	Branch to checkout; if it refers to a branch (i.e., a name that,
 	when prepended with "refs/heads/", is a valid ref), then that
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 32c4b7f897..0c5fe948ef 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -44,6 +44,7 @@ struct checkout_opts {
 	int ignore_skipworktree;
 	int ignore_other_worktrees;
 	int show_progress;
+	int overlay_mode;
 	/*
 	 * If new checkout options are added, skip_merge_working_tree
 	 * should be updated accordingly.
@@ -132,7 +133,8 @@ static int skip_same_name(const struct cache_entry *ce, int pos)
 	return pos;
 }
 
-static int check_stage(int stage, const struct cache_entry *ce, int pos)
+static int check_stage(int stage, const struct cache_entry *ce, int pos,
+		       int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -140,6 +142,8 @@ static int check_stage(int stage, const struct cache_entry *ce, int pos)
 			return 0;
 		pos++;
 	}
+	if (!overlay_mode)
+		return 0;
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -165,7 +169,7 @@ static int check_stages(unsigned stages, const struct cache_entry *ce, int pos)
 }
 
 static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
-			  const struct checkout *state)
+			  const struct checkout *state, int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -173,6 +177,10 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
 			return checkout_entry(active_cache[pos], state, NULL);
 		pos++;
 	}
+	if (!overlay_mode) {
+		unlink_entry(ce);
+		return 0;
+	}
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -247,9 +255,9 @@ static int checkout_merged(int pos, const struct checkout *state)
 	return status;
 }
 
-static void mark_ce_for_checkout(struct cache_entry *ce,
-				 char *ps_matched,
-				 const struct checkout_opts *opts)
+static void mark_ce_for_checkout_overlay(struct cache_entry *ce,
+					 char *ps_matched,
+					 const struct checkout_opts *opts)
 {
 	ce->ce_flags &= ~CE_MATCHED;
 	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
@@ -281,6 +289,25 @@ static void mark_ce_for_checkout(struct cache_entry *ce,
 		ce->ce_flags |= CE_MATCHED;
 }
 
+static void mark_ce_for_checkout_no_overlay(struct cache_entry *ce,
+					    char *ps_matched,
+					    const struct checkout_opts *opts)
+{
+	ce->ce_flags &= ~CE_MATCHED;
+	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
+		return;
+	if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
+		ce->ce_flags |= CE_MATCHED;
+		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
+			/*
+			 * In overlay mode, but the path is not in
+			 * tree-ish, which means we should remove it
+			 * from the index and the working tree.
+			 */
+			ce->ce_flags |= CE_REMOVE | CE_WT_REMOVE;
+	}
+}
+
 static int checkout_paths(const struct checkout_opts *opts,
 			  const char *revision)
 {
@@ -332,7 +359,14 @@ static int checkout_paths(const struct checkout_opts *opts,
 	 * to be checked out.
 	 */
 	for (pos = 0; pos < active_nr; pos++)
-		mark_ce_for_checkout(active_cache[pos], ps_matched, opts);
+		if (opts->overlay_mode)
+			mark_ce_for_checkout_overlay(active_cache[pos],
+						     ps_matched,
+						     opts);
+		else
+			mark_ce_for_checkout_no_overlay(active_cache[pos],
+							ps_matched,
+							opts);
 
 	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
 		free(ps_matched);
@@ -353,7 +387,7 @@ static int checkout_paths(const struct checkout_opts *opts,
 			if (opts->force) {
 				warning(_("path '%s' is unmerged"), ce->name);
 			} else if (opts->writeout_stage) {
-				errs |= check_stage(opts->writeout_stage, ce, pos);
+				errs |= check_stage(opts->writeout_stage, ce, pos, opts->overlay_mode);
 			} else if (opts->merge) {
 				errs |= check_stages((1<<2) | (1<<3), ce, pos);
 			} else {
@@ -380,12 +414,14 @@ static int checkout_paths(const struct checkout_opts *opts,
 				continue;
 			}
 			if (opts->writeout_stage)
-				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state);
+				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state, opts->overlay_mode);
 			else if (opts->merge)
 				errs |= checkout_merged(pos, &state);
 			pos = skip_same_name(ce, pos) - 1;
 		}
 	}
+	remove_marked_cache_entries(&the_index, 1);
+	remove_scheduled_dirs();
 	errs |= finish_delayed_checkout(&state);
 
 	if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))
@@ -547,6 +583,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
 	 * opts->show_progress only impacts output so doesn't require a merge
 	 */
 
+	/*
+	 * opts->overlay_mode cannot be used with switching branches so is
+	 * not tested here
+	 */
+
 	/*
 	 * If we aren't creating a new branch any changes or updates will
 	 * happen in the existing branch.  Since that could only be updating
@@ -1183,6 +1224,10 @@ static int checkout_branch(struct checkout_opts *opts,
 		die(_("'%s' cannot be used with switching branches"),
 		    "--patch");
 
+	if (!opts->overlay_mode)
+		die(_("'%s' cannot be used with switching branches"),
+		    "--no-overlay");
+
 	if (opts->writeout_stage)
 		die(_("'%s' cannot be used with switching branches"),
 		    "--ours/--theirs");
@@ -1271,6 +1316,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 			    "checkout", "control recursive updating of submodules",
 			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
+		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode (default)")),
 		OPT_END(),
 	};
 
@@ -1279,6 +1325,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	opts.overwrite_ignore = 1;
 	opts.prefix = prefix;
 	opts.show_progress = -1;
+	opts.overlay_mode = -1;
 
 	git_config(git_checkout_config, &opts);
 
@@ -1302,6 +1349,9 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	if ((!!opts.new_branch + !!opts.new_branch_force + !!opts.new_orphan_branch) > 1)
 		die(_("-b, -B and --orphan are mutually exclusive"));
 
+	if (opts.overlay_mode == 1 && opts.patch_mode)
+		die(_("-p and --overlay are mutually exclusive"));
+
 	/*
 	 * From here on, new_branch will contain the branch to be checked out,
 	 * and new_branch_force and new_orphan_branch will tell us which one of
diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
new file mode 100755
index 0000000000..76330cb5ab
--- /dev/null
+++ b/t/t2025-checkout-no-overlay.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+
+test_description='checkout --no-overlay <tree-ish> -- <pathspec>'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git commit --allow-empty -m "initial"
+'
+
+test_expect_success 'checkout --no-overlay deletes files not in <tree-ish>' '
+	>file &&
+	mkdir dir &&
+	>dir/file1 &&
+	git add file dir/file1 &&
+	git checkout --no-overlay HEAD -- file &&
+	test_path_is_missing file &&
+	test_path_is_file dir/file1
+'
+
+test_expect_success 'checkout --no-overlay removing last file from directory' '
+	git checkout --no-overlay HEAD -- dir/file1 &&
+	test_path_is_missing dir
+'
+
+test_expect_success 'checkout -p --overlay is disallowed' '
+	test_must_fail git checkout -p --overlay HEAD 2>actual &&
+	test_i18ngrep "fatal: -p and --overlay are mutually exclusive" actual
+'
+
+test_expect_success '--no-overlay --theirs with D/F conflict deletes file' '
+	test_commit file1 file1 &&
+	test_commit file2 file2 &&
+	git rm --cached file1 &&
+	echo 1234 >file1 &&
+	F1=$(git rev-parse HEAD:file1) &&
+	F2=$(git rev-parse HEAD:file2) &&
+	{
+		echo "100644 $F1 1	file1" &&
+		echo "100644 $F2 2	file1"
+	} | git update-index --index-info &&
+	test_path_is_file file1 &&
+	git checkout --theirs --no-overlay -- file1 &&
+	test_path_is_missing file1
+'
+
+test_done
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index d01ad8eb25..5758fffa0d 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -1436,6 +1436,7 @@ test_expect_success 'double dash "git checkout"' '
 	--progress Z
 	--no-quiet Z
 	--no-... Z
+	--overlay Z
 	EOF
 '
 
-- 
2.20.1.415.g653613c723

[PATCH v2 8/8] checkout: introduce checkout.overlayMode config

[Thomas Gummerer]

In the previous patch we introduced a new no-overlay mode for git
checkout.  Some users (such as the author of this commit) may want to
have this mode turned on by default as it matches their mental model
more closely.  Make that possible by introducing a new config option
to that extend.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 Documentation/config/checkout.txt |  7 +++++++
 builtin/checkout.c                |  8 +++++++-
 t/t2025-checkout-no-overlay.sh    | 10 ++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt
index c4118fa196..53f917e15e 100644
--- a/Documentation/config/checkout.txt
+++ b/Documentation/config/checkout.txt
@@ -21,3 +21,10 @@ checkout.optimizeNewBranch::
 	will not update the skip-worktree bit in the index nor add/remove
 	files in the working directory to reflect the current sparse checkout
 	settings nor will it show the local changes.
+
+checkout.overlayMode::
+	In the default overlay mode files `git checkout` never
+	removes files from the index or the working tree.  When
+	setting checkout.overlayMode to false, files that appear in
+	the index and working tree, but not in <tree-ish> are removed,
+	to make them match <tree-ish> exactly.
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 0c5fe948ef..b5dfc45736 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -1019,13 +1019,19 @@ static int switch_branches(const struct checkout_opts *opts,
 
 static int git_checkout_config(const char *var, const char *value, void *cb)
 {
+	struct checkout_opts *opts = cb;
+
 	if (!strcmp(var, "checkout.optimizenewbranch")) {
 		checkout_optimize_new_branch = git_config_bool(var, value);
 		return 0;
 	}
 
+	if (!strcmp(var, "checkout.overlaymode")) {
+		opts->overlay_mode = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "diff.ignoresubmodules")) {
-		struct checkout_opts *opts = cb;
 		handle_ignore_submodules_arg(&opts->diff_options, value);
 		return 0;
 	}
diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
index 76330cb5ab..a4912e35cb 100755
--- a/t/t2025-checkout-no-overlay.sh
+++ b/t/t2025-checkout-no-overlay.sh
@@ -44,4 +44,14 @@ test_expect_success '--no-overlay --theirs with D/F conflict deletes file' '
 	test_path_is_missing file1
 '
 
+test_expect_success 'checkout with checkout.overlayMode=false deletes files not in <tree-ish>' '
+	>file &&
+	mkdir dir &&
+	>dir/file1 &&
+	git add file dir/file1 &&
+	git -c checkout.overlayMode=false checkout HEAD -- file &&
+	test_path_is_missing file &&
+	test_path_is_file dir/file1
+'
+
 test_done
-- 
2.20.1.415.g653613c723

Re: [PATCH v2 7/8] checkout: introduce --{,no-}overlay option

[Duy Nguyen]

On Thu, Dec 20, 2018 at 2:48 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
> index 801de2f764..4ac8c55865 100644
> --- a/Documentation/git-checkout.txt
> +++ b/Documentation/git-checkout.txt
> @@ -260,6 +260,9 @@ the conflicted merge in the specified paths.
>  This means that you can use `git checkout -p` to selectively discard
>  edits from your current working tree. See the ``Interactive Mode''
>  section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
> ++
> +Note that this option uses the no overlay mode by default (see also
> +-`--[no-]overlay`), and currently doesn't support overlay mode.
>
>  --ignore-other-worktrees::
>         `git checkout` refuses when the wanted ref is already checked
> @@ -276,6 +279,13 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
>         Just like linkgit:git-submodule[1], this will detach the
>         submodules HEAD.
>
> +--[no-]overlay::
> +       In the default overlay mode files `git checkout` never

-ECANTPARSE. Maybe "files" should be removed from this line?

> +       removes files from the index or the working tree.  When
> +       specifying --no-overlay, files that appear in the index and
> +       working tree, but not in <tree-ish> are removed, to make them
> +       match <tree-ish> exactly.
> +
>  <branch>::
>         Branch to checkout; if it refers to a branch (i.e., a name that,
>         when prepended with "refs/heads/", is a valid ref), then that
-- 
Duy

Re: [PATCH v2 7/8] checkout: introduce --{,no-}overlay option

[Eric Sunshine]

On Sun, Dec 23, 2018 at 3:05 AM Duy Nguyen <pclouds@gmail.com> wrote:
> On Thu, Dec 20, 2018 at 2:48 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > +--[no-]overlay::
> > +       In the default overlay mode files `git checkout` never
>
> -ECANTPARSE. Maybe "files" should be removed from this line?

Also, add a comma after "mode".

Re: [PATCH v2 8/8] checkout: introduce checkout.overlayMode config

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> In the previous patch we introduced a new no-overlay mode for git
> checkout.  Some users (such as the author of this commit) may want to
> have this mode turned on by default as it matches their mental model
> more closely.  Make that possible by introducing a new config option
> to that extend.
>
> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> ---
>  Documentation/config/checkout.txt |  7 +++++++
>  builtin/checkout.c                |  8 +++++++-
>  t/t2025-checkout-no-overlay.sh    | 10 ++++++++++
>  3 files changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt
> index c4118fa196..53f917e15e 100644
> --- a/Documentation/config/checkout.txt
> +++ b/Documentation/config/checkout.txt
> @@ -21,3 +21,10 @@ checkout.optimizeNewBranch::
>  	will not update the skip-worktree bit in the index nor add/remove
>  	files in the working directory to reflect the current sparse checkout
>  	settings nor will it show the local changes.
> +
> +checkout.overlayMode::
> +	In the default overlay mode files `git checkout` never
> +	removes files from the index or the working tree.

Technically the above "never removes" is incorrect.

	$ mv COPYING 1 && mkdir COPYING && mv 1 COPYING/COPYING
	$ git add COPYING
	$ git checkout HEAD COPYING

would remove COPYING/1 from the index and from the working tree to
make room.

Because I think that a bit of white lie like what you wrote would
help readers understand the key point of "overlay or not overlay"
better than an overly precise description of the reason why the
removal in the above three-liner case is the right thing to do, I
think the text in the patch is good enough at least for now, but I'd
mention it in case somebody else can think of a better phrasing to
covey the same key point without being technically incorrect.

Thanks.

Re: [PATCH v2 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 12/23, Eric Sunshine wrote:
> On Sun, Dec 23, 2018 at 3:05 AM Duy Nguyen <pclouds@gmail.com> wrote:
> > On Thu, Dec 20, 2018 at 2:48 PM Thomas Gummerer <t.gummerer@gmail.com> wrote:
> > > +--[no-]overlay::
> > > +       In the default overlay mode files `git checkout` never
> >
> > -ECANTPARSE. Maybe "files" should be removed from this line?
> 
> Also, add a comma after "mode".

Will do, thanks both.

Re: [PATCH v2 8/8] checkout: introduce checkout.overlayMode config

[Thomas Gummerer]

On 01/02, Junio C Hamano wrote:
> Thomas Gummerer <t.gummerer@gmail.com> writes:
> 
> > In the previous patch we introduced a new no-overlay mode for git
> > checkout.  Some users (such as the author of this commit) may want to
> > have this mode turned on by default as it matches their mental model
> > more closely.  Make that possible by introducing a new config option
> > to that extend.
> >
> > Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
> > ---
> >  Documentation/config/checkout.txt |  7 +++++++
> >  builtin/checkout.c                |  8 +++++++-
> >  t/t2025-checkout-no-overlay.sh    | 10 ++++++++++
> >  3 files changed, 24 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt
> > index c4118fa196..53f917e15e 100644
> > --- a/Documentation/config/checkout.txt
> > +++ b/Documentation/config/checkout.txt
> > @@ -21,3 +21,10 @@ checkout.optimizeNewBranch::
> >  	will not update the skip-worktree bit in the index nor add/remove
> >  	files in the working directory to reflect the current sparse checkout
> >  	settings nor will it show the local changes.
> > +
> > +checkout.overlayMode::
> > +	In the default overlay mode files `git checkout` never
> > +	removes files from the index or the working tree.
> 
> Technically the above "never removes" is incorrect.
> 
> 	$ mv COPYING 1 && mkdir COPYING && mv 1 COPYING/COPYING
> 	$ git add COPYING
> 	$ git checkout HEAD COPYING
> 
> would remove COPYING/1 from the index and from the working tree to
> make room.

Right, that's a case I didn't think about.

> Because I think that a bit of white lie like what you wrote would
> help readers understand the key point of "overlay or not overlay"
> better than an overly precise description of the reason why the
> removal in the above three-liner case is the right thing to do, I
> think the text in the patch is good enough at least for now, but I'd
> mention it in case somebody else can think of a better phrasing to
> covey the same key point without being technically incorrect.

Maybe it would be enough to say "... `git checkout` never removes
files, that are not in the tree being checked out, from the index or
the working tree"?  It is more technically correct, but dunno making
the sentence harder to read is worth it.

> Thanks.

Re: [PATCH v2 8/8] checkout: introduce checkout.overlayMode config

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> Maybe it would be enough to say "... `git checkout` never removes
> files, that are not in the tree being checked out, from the index or
> the working tree"?  It is more technically correct, but dunno making
> the sentence harder to read is worth it.

Yeah, I share the same feeling.  Let's say the text in the posted
patch is good enough and move on.

Thanks.

[PATCH v3 0/8] introduce no-overlay mode in git checkout

[Thomas Gummerer]

Previous rounds are at <20181209200449.16342-1-t.gummerer@gmail.com>
and <20181220134820.21810-1-t.gummerer@gmail.com>.

Thanks Duy, Eric and Junio for comments on the previous round.

This round fixes some inconsistencies and improves the grammar in the
docs.  Range-diff below:

1:  fa450cda7c = 1:  fa450cda7c move worktree tests to t24*
2:  9ada8d3484 = 2:  9ada8d3484 entry: factor out unlink_entry function
3:  41c0ea4047 = 3:  41c0ea4047 entry: support CE_WT_REMOVE flag in checkout_entry
4:  afccb0848d = 4:  afccb0848d read-cache: add invalidate parameter to remove_marked_cache_entries
5:  8a2b5efdad = 5:  8a2b5efdad checkout: clarify comment
6:  c405f20471 = 6:  c405f20471 checkout: factor out mark_cache_entry_for_checkout function
7:  e5b18bcd02 ! 7:  a291dc78fa checkout: introduce --{,no-}overlay option
    @@ -35,7 +35,7 @@
      section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
     ++
     +Note that this option uses the no overlay mode by default (see also
    -+-`--[no-]overlay`), and currently doesn't support overlay mode.
    ++`--[no-]overlay`), and currently doesn't support overlay mode.
      
      --ignore-other-worktrees::
      	`git checkout` refuses when the wanted ref is already checked
    @@ -44,9 +44,9 @@
      	submodules HEAD.
      
     +--[no-]overlay::
    -+	In the default overlay mode files `git checkout` never
    ++	In the default overlay mode, `git checkout` never
     +	removes files from the index or the working tree.  When
    -+	specifying --no-overlay, files that appear in the index and
    ++	specifying `--no-overlay`, files that appear in the index and
     +	working tree, but not in <tree-ish> are removed, to make them
     +	match <tree-ish> exactly.
     +
8:  de24990d57 ! 8:  8d4070f142 checkout: introduce checkout.overlayMode config
    @@ -19,9 +19,9 @@
      	settings nor will it show the local changes.
     +
     +checkout.overlayMode::
    -+	In the default overlay mode files `git checkout` never
    ++	In the default overlay mode, `git checkout` never
     +	removes files from the index or the working tree.  When
    -+	setting checkout.overlayMode to false, files that appear in
    ++	setting `checkout.overlayMode` to false, files that appear in
     +	the index and working tree, but not in <tree-ish> are removed,
     +	to make them match <tree-ish> exactly.

Thomas Gummerer (8):
  move worktree tests to t24*
  entry: factor out unlink_entry function
  entry: support CE_WT_REMOVE flag in checkout_entry
  read-cache: add invalidate parameter to remove_marked_cache_entries
  checkout: clarify comment
  checkout: factor out mark_cache_entry_for_checkout function
  checkout: introduce --{,no-}overlay option
  checkout: introduce checkout.overlayMode config

 Documentation/config/checkout.txt             |   7 +
 Documentation/git-checkout.txt                |  10 ++
 builtin/checkout.c                            | 133 +++++++++++++-----
 cache.h                                       |   7 +-
 entry.c                                       |  26 ++++
 read-cache.c                                  |   8 +-
 split-index.c                                 |   2 +-
 t/t2025-checkout-no-overlay.sh                |  57 ++++++++
 ...-worktree-add.sh => t2400-worktree-add.sh} |   0
 ...ktree-prune.sh => t2401-worktree-prune.sh} |   0
 ...orktree-list.sh => t2402-worktree-list.sh} |   0
 ...orktree-move.sh => t2403-worktree-move.sh} |   0
 ...ree-config.sh => t2404-worktree-config.sh} |   0
 t/t9902-completion.sh                         |   1 +
 unpack-trees.c                                |  21 +--
 15 files changed, 213 insertions(+), 59 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
 rename t/{t2028-worktree-move.sh => t2403-worktree-move.sh} (100%)
 rename t/{t2029-worktree-config.sh => t2404-worktree-config.sh} (100%)

-- 
2.20.1.153.gd81d796ee0

     

[PATCH v3 1/8] move worktree tests to t24*

[Thomas Gummerer]

The 'git worktree' command used to be just another mode in 'git
checkout', namely 'git checkout --to'.  When the tests for the latter
were retrofitted for the former, the test name was adjusted, but the
test number was kept, even though the test is testing a different
command now.  t/README states: "Second digit tells the particular
command we are testing.", so 'git worktree' should have a separate
number just for itself.

Move the worktree tests to t24* to adhere to that guideline. We're
going to make use of the free'd up numbers in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 t/{t2025-worktree-add.sh => t2400-worktree-add.sh}       | 0
 t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh}   | 0
 t/{t2027-worktree-list.sh => t2402-worktree-list.sh}     | 0
 t/{t2028-worktree-move.sh => t2403-worktree-move.sh}     | 0
 t/{t2029-worktree-config.sh => t2404-worktree-config.sh} | 0
 5 files changed, 0 insertions(+), 0 deletions(-)
 rename t/{t2025-worktree-add.sh => t2400-worktree-add.sh} (100%)
 rename t/{t2026-worktree-prune.sh => t2401-worktree-prune.sh} (100%)
 rename t/{t2027-worktree-list.sh => t2402-worktree-list.sh} (100%)
 rename t/{t2028-worktree-move.sh => t2403-worktree-move.sh} (100%)
 rename t/{t2029-worktree-config.sh => t2404-worktree-config.sh} (100%)

diff --git a/t/t2025-worktree-add.sh b/t/t2400-worktree-add.sh
similarity index 100%
rename from t/t2025-worktree-add.sh
rename to t/t2400-worktree-add.sh
diff --git a/t/t2026-worktree-prune.sh b/t/t2401-worktree-prune.sh
similarity index 100%
rename from t/t2026-worktree-prune.sh
rename to t/t2401-worktree-prune.sh
diff --git a/t/t2027-worktree-list.sh b/t/t2402-worktree-list.sh
similarity index 100%
rename from t/t2027-worktree-list.sh
rename to t/t2402-worktree-list.sh
diff --git a/t/t2028-worktree-move.sh b/t/t2403-worktree-move.sh
similarity index 100%
rename from t/t2028-worktree-move.sh
rename to t/t2403-worktree-move.sh
diff --git a/t/t2029-worktree-config.sh b/t/t2404-worktree-config.sh
similarity index 100%
rename from t/t2029-worktree-config.sh
rename to t/t2404-worktree-config.sh
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 2/8] entry: factor out unlink_entry function

[Thomas Gummerer]

Factor out the 'unlink_entry()' function from unpack-trees.c to
entry.c.  It will be used in other places as well in subsequent
steps.

As it's no longer a static function, also move the documentation to
the header file to make it more discoverable.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 cache.h        |  5 +++++
 entry.c        | 15 +++++++++++++++
 unpack-trees.c | 19 -------------------
 3 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/cache.h b/cache.h
index ca36b44ee0..c1c953e810 100644
--- a/cache.h
+++ b/cache.h
@@ -1542,6 +1542,11 @@ struct checkout {
 extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
 extern void enable_delayed_checkout(struct checkout *state);
 extern int finish_delayed_checkout(struct checkout *state);
+/*
+ * Unlink the last component and schedule the leading directories for
+ * removal, such that empty directories get removed.
+ */
+extern void unlink_entry(const struct cache_entry *ce);
 
 struct cache_def {
 	struct strbuf path;
diff --git a/entry.c b/entry.c
index 0a3c451f5f..b9eef57117 100644
--- a/entry.c
+++ b/entry.c
@@ -508,3 +508,18 @@ int checkout_entry(struct cache_entry *ce,
 	create_directories(path.buf, path.len, state);
 	return write_entry(ce, path.buf, state, 0);
 }
+
+void unlink_entry(const struct cache_entry *ce)
+{
+	const struct submodule *sub = submodule_from_ce(ce);
+	if (sub) {
+		/* state.force is set at the caller. */
+		submodule_move_head(ce->name, "HEAD", NULL,
+				    SUBMODULE_MOVE_HEAD_FORCE);
+	}
+	if (!check_leading_path(ce->name, ce_namelen(ce)))
+		return;
+	if (remove_or_warn(ce->ce_mode, ce->name))
+		return;
+	schedule_dir_for_removal(ce->name, ce_namelen(ce));
+}
diff --git a/unpack-trees.c b/unpack-trees.c
index 7570df481b..e8d1a6ac50 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -300,25 +300,6 @@ static void load_gitmodules_file(struct index_state *index,
 	}
 }
 
-/*
- * Unlink the last component and schedule the leading directories for
- * removal, such that empty directories get removed.
- */
-static void unlink_entry(const struct cache_entry *ce)
-{
-	const struct submodule *sub = submodule_from_ce(ce);
-	if (sub) {
-		/* state.force is set at the caller. */
-		submodule_move_head(ce->name, "HEAD", NULL,
-				    SUBMODULE_MOVE_HEAD_FORCE);
-	}
-	if (!check_leading_path(ce->name, ce_namelen(ce)))
-		return;
-	if (remove_or_warn(ce->ce_mode, ce->name))
-		return;
-	schedule_dir_for_removal(ce->name, ce_namelen(ce));
-}
-
 static struct progress *get_progress(struct unpack_trees_options *o)
 {
 	unsigned cnt = 0, total = 0;
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 3/8] entry: support CE_WT_REMOVE flag in checkout_entry

[Thomas Gummerer]

'checkout_entry()' currently only supports creating new entries in the
working tree, but not deleting them.  Add the ability to remove
entries at the same time if the entry is marked with the CE_WT_REMOVE
flag.

Currently this doesn't have any effect, as the CE_WT_REMOVE flag is
only used in unpack-tree, however we will make use of this in a
subsequent step in the series.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 entry.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/entry.c b/entry.c
index b9eef57117..3d3701e7ae 100644
--- a/entry.c
+++ b/entry.c
@@ -441,6 +441,17 @@ int checkout_entry(struct cache_entry *ce,
 	static struct strbuf path = STRBUF_INIT;
 	struct stat st;
 
+	if (ce->ce_flags & CE_WT_REMOVE) {
+		if (topath)
+			/*
+			 * No content and thus no path to create, so we have
+			 * no pathname to return.
+			 */
+			BUG("Can't remove entry to a path");
+		unlink_entry(ce);
+		return 0;
+	}
+
 	if (topath)
 		return write_entry(ce, topath, state, 1);
 
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 4/8] read-cache: add invalidate parameter to remove_marked_cache_entries

[Thomas Gummerer]

When marking cache entries for removal, and later removing them all at
once using 'remove_marked_cache_entries()', cache entries currently
have to be invalidated manually in the cache tree and in the untracked
cache.

Add an invalidate flag to the function.  With the flag set, the
function will take care of invalidating the path in the cache tree and
in the untracked cache.

Note that the current callsites already do the invalidation properly
in other places, so we're just passing 0 from there to keep the status
quo.

This will be useful in a subsequent commit.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 cache.h        | 2 +-
 read-cache.c   | 8 +++++++-
 split-index.c  | 2 +-
 unpack-trees.c | 2 +-
 4 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index c1c953e810..1deee48f5b 100644
--- a/cache.h
+++ b/cache.h
@@ -751,7 +751,7 @@ extern void rename_index_entry_at(struct index_state *, int pos, const char *new
 /* Remove entry, return true if there are more entries to go. */
 extern int remove_index_entry_at(struct index_state *, int pos);
 
-extern void remove_marked_cache_entries(struct index_state *istate);
+extern void remove_marked_cache_entries(struct index_state *istate, int invalidate);
 extern int remove_file_from_index(struct index_state *, const char *path);
 #define ADD_CACHE_VERBOSE 1
 #define ADD_CACHE_PRETEND 2
diff --git a/read-cache.c b/read-cache.c
index bd45dc3e24..978d43f676 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -590,13 +590,19 @@ int remove_index_entry_at(struct index_state *istate, int pos)
  * CE_REMOVE is set in ce_flags.  This is much more effective than
  * calling remove_index_entry_at() for each entry to be removed.
  */
-void remove_marked_cache_entries(struct index_state *istate)
+void remove_marked_cache_entries(struct index_state *istate, int invalidate)
 {
 	struct cache_entry **ce_array = istate->cache;
 	unsigned int i, j;
 
 	for (i = j = 0; i < istate->cache_nr; i++) {
 		if (ce_array[i]->ce_flags & CE_REMOVE) {
+			if (invalidate) {
+				cache_tree_invalidate_path(istate,
+							   ce_array[i]->name);
+				untracked_cache_remove_from_index(istate,
+								  ce_array[i]->name);
+			}
 			remove_name_hash(istate, ce_array[i]);
 			save_or_free_index_entry(istate, ce_array[i]);
 		}
diff --git a/split-index.c b/split-index.c
index 5820412dc5..8aebc3661b 100644
--- a/split-index.c
+++ b/split-index.c
@@ -162,7 +162,7 @@ void merge_base_index(struct index_state *istate)
 	ewah_each_bit(si->replace_bitmap, replace_entry, istate);
 	ewah_each_bit(si->delete_bitmap, mark_entry_for_delete, istate);
 	if (si->nr_deletions)
-		remove_marked_cache_entries(istate);
+		remove_marked_cache_entries(istate, 0);
 
 	for (i = si->nr_replacements; i < si->saved_cache_nr; i++) {
 		if (!ce_namelen(si->saved_cache[i]))
diff --git a/unpack-trees.c b/unpack-trees.c
index e8d1a6ac50..8e6afa924d 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -392,7 +392,7 @@ static int check_updates(struct unpack_trees_options *o)
 				unlink_entry(ce);
 		}
 	}
-	remove_marked_cache_entries(index);
+	remove_marked_cache_entries(index, 0);
 	remove_scheduled_dirs();
 
 	if (should_update_submodules() && o->update && !o->dry_run)
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 5/8] checkout: clarify comment

[Thomas Gummerer]

The key point for the if statement is that read_tree_some did not
update the entry, because either it doesn't exist in tree-ish or
doesn't match the pathspec.  Clarify that.

Suggested-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index acdafc6e4c..cb166b2e07 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -304,10 +304,10 @@ static int checkout_paths(const struct checkout_opts *opts,
 			continue;
 		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
 			/*
-			 * "git checkout tree-ish -- path", but this entry
-			 * is in the original index; it will not be checked
-			 * out to the working tree and it does not matter
-			 * if pathspec matched this entry.  We will not do
+			 * "git checkout tree-ish -- path" and this entry
+			 * is in the original index, but is not in tree-ish
+			 * or does not match the pathspec; it will not be
+			 * checked out to the working tree.  We will not do
 			 * anything to this entry at all.
 			 */
 			continue;
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 6/8] checkout: factor out mark_cache_entry_for_checkout function

[Thomas Gummerer]

Factor out the code that marks a cache entry as matched for checkout
into a separate function.  We are going to introduce a new mode in
'git checkout' in a subsequent commit, that is going to have a
slightly different logic.  This would make this code unnecessarily
complex.

Moving that complexity into separate functions will make the code in
the subsequent step easier to follow.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/checkout.c | 67 +++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index cb166b2e07..32c4b7f897 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -247,6 +247,40 @@ static int checkout_merged(int pos, const struct checkout *state)
 	return status;
 }
 
+static void mark_ce_for_checkout(struct cache_entry *ce,
+				 char *ps_matched,
+				 const struct checkout_opts *opts)
+{
+	ce->ce_flags &= ~CE_MATCHED;
+	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
+		return;
+	if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
+		/*
+		 * "git checkout tree-ish -- path", but this entry
+		 * is in the original index but is not in tree-ish
+		 * or does not match the pathspec; it will not be
+		 * checked out to the working tree.  We will not do
+		 * anything to this entry at all.
+		 */
+		return;
+	/*
+	 * Either this entry came from the tree-ish we are
+	 * checking the paths out of, or we are checking out
+	 * of the index.
+	 *
+	 * If it comes from the tree-ish, we already know it
+	 * matches the pathspec and could just stamp
+	 * CE_MATCHED to it from update_some(). But we still
+	 * need ps_matched and read_tree_recursive (and
+	 * eventually tree_entry_interesting) cannot fill
+	 * ps_matched yet. Once it can, we can avoid calling
+	 * match_pathspec() for _all_ entries when
+	 * opts->source_tree != NULL.
+	 */
+	if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched))
+		ce->ce_flags |= CE_MATCHED;
+}
+
 static int checkout_paths(const struct checkout_opts *opts,
 			  const char *revision)
 {
@@ -297,37 +331,8 @@ static int checkout_paths(const struct checkout_opts *opts,
 	 * Make sure all pathspecs participated in locating the paths
 	 * to be checked out.
 	 */
-	for (pos = 0; pos < active_nr; pos++) {
-		struct cache_entry *ce = active_cache[pos];
-		ce->ce_flags &= ~CE_MATCHED;
-		if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
-			continue;
-		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
-			/*
-			 * "git checkout tree-ish -- path" and this entry
-			 * is in the original index, but is not in tree-ish
-			 * or does not match the pathspec; it will not be
-			 * checked out to the working tree.  We will not do
-			 * anything to this entry at all.
-			 */
-			continue;
-		/*
-		 * Either this entry came from the tree-ish we are
-		 * checking the paths out of, or we are checking out
-		 * of the index.
-		 *
-		 * If it comes from the tree-ish, we already know it
-		 * matches the pathspec and could just stamp
-		 * CE_MATCHED to it from update_some(). But we still
-		 * need ps_matched and read_tree_recursive (and
-		 * eventually tree_entry_interesting) cannot fill
-		 * ps_matched yet. Once it can, we can avoid calling
-		 * match_pathspec() for _all_ entries when
-		 * opts->source_tree != NULL.
-		 */
-		if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched))
-			ce->ce_flags |= CE_MATCHED;
-	}
+	for (pos = 0; pos < active_nr; pos++)
+		mark_ce_for_checkout(active_cache[pos], ps_matched, opts);
 
 	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
 		free(ps_matched);
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

Currently 'git checkout' is defined as an overlay operation, which
means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
entry in the index that matches <pathspec>, but that doesn't exist in
<tree-ish>, that entry will not be removed from the index or the
working tree.

Introduce a new --{,no-}overlay option, which allows using 'git
checkout' in non-overlay mode, thus removing files from the working
tree if they do not exist in <tree-ish> but match <pathspec>.

Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works
this way, so no changes are needed for the patch mode.  We disallow
'git checkout --overlay -p' to avoid confusing users who would expect
to be able to force overlay mode in 'git checkout -p' this way.

Untracked files are not affected by this change, so 'git checkout
--no-overlay HEAD -- untracked' will not remove untracked from the
working tree.  This is so e.g. 'git checkout --no-overlay HEAD -- dir/'
doesn't delete all untracked files in dir/, but rather just resets the
state of files that are known to git.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 Documentation/git-checkout.txt | 10 ++++++
 builtin/checkout.c             | 66 +++++++++++++++++++++++++++++-----
 t/t2025-checkout-no-overlay.sh | 47 ++++++++++++++++++++++++
 t/t9902-completion.sh          |  1 +
 4 files changed, 116 insertions(+), 8 deletions(-)
 create mode 100755 t/t2025-checkout-no-overlay.sh

diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt
index 801de2f764..24e52b01e1 100644
--- a/Documentation/git-checkout.txt
+++ b/Documentation/git-checkout.txt
@@ -260,6 +260,9 @@ the conflicted merge in the specified paths.
 This means that you can use `git checkout -p` to selectively discard
 edits from your current working tree. See the ``Interactive Mode''
 section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
++
+Note that this option uses the no overlay mode by default (see also
+`--[no-]overlay`), and currently doesn't support overlay mode.
 
 --ignore-other-worktrees::
 	`git checkout` refuses when the wanted ref is already checked
@@ -276,6 +279,13 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.
 	Just like linkgit:git-submodule[1], this will detach the
 	submodules HEAD.
 
+--[no-]overlay::
+	In the default overlay mode, `git checkout` never
+	removes files from the index or the working tree.  When
+	specifying `--no-overlay`, files that appear in the index and
+	working tree, but not in <tree-ish> are removed, to make them
+	match <tree-ish> exactly.
+
 <branch>::
 	Branch to checkout; if it refers to a branch (i.e., a name that,
 	when prepended with "refs/heads/", is a valid ref), then that
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 32c4b7f897..0c5fe948ef 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -44,6 +44,7 @@ struct checkout_opts {
 	int ignore_skipworktree;
 	int ignore_other_worktrees;
 	int show_progress;
+	int overlay_mode;
 	/*
 	 * If new checkout options are added, skip_merge_working_tree
 	 * should be updated accordingly.
@@ -132,7 +133,8 @@ static int skip_same_name(const struct cache_entry *ce, int pos)
 	return pos;
 }
 
-static int check_stage(int stage, const struct cache_entry *ce, int pos)
+static int check_stage(int stage, const struct cache_entry *ce, int pos,
+		       int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -140,6 +142,8 @@ static int check_stage(int stage, const struct cache_entry *ce, int pos)
 			return 0;
 		pos++;
 	}
+	if (!overlay_mode)
+		return 0;
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -165,7 +169,7 @@ static int check_stages(unsigned stages, const struct cache_entry *ce, int pos)
 }
 
 static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
-			  const struct checkout *state)
+			  const struct checkout *state, int overlay_mode)
 {
 	while (pos < active_nr &&
 	       !strcmp(active_cache[pos]->name, ce->name)) {
@@ -173,6 +177,10 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
 			return checkout_entry(active_cache[pos], state, NULL);
 		pos++;
 	}
+	if (!overlay_mode) {
+		unlink_entry(ce);
+		return 0;
+	}
 	if (stage == 2)
 		return error(_("path '%s' does not have our version"), ce->name);
 	else
@@ -247,9 +255,9 @@ static int checkout_merged(int pos, const struct checkout *state)
 	return status;
 }
 
-static void mark_ce_for_checkout(struct cache_entry *ce,
-				 char *ps_matched,
-				 const struct checkout_opts *opts)
+static void mark_ce_for_checkout_overlay(struct cache_entry *ce,
+					 char *ps_matched,
+					 const struct checkout_opts *opts)
 {
 	ce->ce_flags &= ~CE_MATCHED;
 	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
@@ -281,6 +289,25 @@ static void mark_ce_for_checkout(struct cache_entry *ce,
 		ce->ce_flags |= CE_MATCHED;
 }
 
+static void mark_ce_for_checkout_no_overlay(struct cache_entry *ce,
+					    char *ps_matched,
+					    const struct checkout_opts *opts)
+{
+	ce->ce_flags &= ~CE_MATCHED;
+	if (!opts->ignore_skipworktree && ce_skip_worktree(ce))
+		return;
+	if (ce_path_match(&the_index, ce, &opts->pathspec, ps_matched)) {
+		ce->ce_flags |= CE_MATCHED;
+		if (opts->source_tree && !(ce->ce_flags & CE_UPDATE))
+			/*
+			 * In overlay mode, but the path is not in
+			 * tree-ish, which means we should remove it
+			 * from the index and the working tree.
+			 */
+			ce->ce_flags |= CE_REMOVE | CE_WT_REMOVE;
+	}
+}
+
 static int checkout_paths(const struct checkout_opts *opts,
 			  const char *revision)
 {
@@ -332,7 +359,14 @@ static int checkout_paths(const struct checkout_opts *opts,
 	 * to be checked out.
 	 */
 	for (pos = 0; pos < active_nr; pos++)
-		mark_ce_for_checkout(active_cache[pos], ps_matched, opts);
+		if (opts->overlay_mode)
+			mark_ce_for_checkout_overlay(active_cache[pos],
+						     ps_matched,
+						     opts);
+		else
+			mark_ce_for_checkout_no_overlay(active_cache[pos],
+							ps_matched,
+							opts);
 
 	if (report_path_error(ps_matched, &opts->pathspec, opts->prefix)) {
 		free(ps_matched);
@@ -353,7 +387,7 @@ static int checkout_paths(const struct checkout_opts *opts,
 			if (opts->force) {
 				warning(_("path '%s' is unmerged"), ce->name);
 			} else if (opts->writeout_stage) {
-				errs |= check_stage(opts->writeout_stage, ce, pos);
+				errs |= check_stage(opts->writeout_stage, ce, pos, opts->overlay_mode);
 			} else if (opts->merge) {
 				errs |= check_stages((1<<2) | (1<<3), ce, pos);
 			} else {
@@ -380,12 +414,14 @@ static int checkout_paths(const struct checkout_opts *opts,
 				continue;
 			}
 			if (opts->writeout_stage)
-				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state);
+				errs |= checkout_stage(opts->writeout_stage, ce, pos, &state, opts->overlay_mode);
 			else if (opts->merge)
 				errs |= checkout_merged(pos, &state);
 			pos = skip_same_name(ce, pos) - 1;
 		}
 	}
+	remove_marked_cache_entries(&the_index, 1);
+	remove_scheduled_dirs();
 	errs |= finish_delayed_checkout(&state);
 
 	if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))
@@ -547,6 +583,11 @@ static int skip_merge_working_tree(const struct checkout_opts *opts,
 	 * opts->show_progress only impacts output so doesn't require a merge
 	 */
 
+	/*
+	 * opts->overlay_mode cannot be used with switching branches so is
+	 * not tested here
+	 */
+
 	/*
 	 * If we aren't creating a new branch any changes or updates will
 	 * happen in the existing branch.  Since that could only be updating
@@ -1183,6 +1224,10 @@ static int checkout_branch(struct checkout_opts *opts,
 		die(_("'%s' cannot be used with switching branches"),
 		    "--patch");
 
+	if (!opts->overlay_mode)
+		die(_("'%s' cannot be used with switching branches"),
+		    "--no-overlay");
+
 	if (opts->writeout_stage)
 		die(_("'%s' cannot be used with switching branches"),
 		    "--ours/--theirs");
@@ -1271,6 +1316,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 			    "checkout", "control recursive updating of submodules",
 			    PARSE_OPT_OPTARG, option_parse_recurse_submodules_worktree_updater },
 		OPT_BOOL(0, "progress", &opts.show_progress, N_("force progress reporting")),
+		OPT_BOOL(0, "overlay", &opts.overlay_mode, N_("use overlay mode (default)")),
 		OPT_END(),
 	};
 
@@ -1279,6 +1325,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	opts.overwrite_ignore = 1;
 	opts.prefix = prefix;
 	opts.show_progress = -1;
+	opts.overlay_mode = -1;
 
 	git_config(git_checkout_config, &opts);
 
@@ -1302,6 +1349,9 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)
 	if ((!!opts.new_branch + !!opts.new_branch_force + !!opts.new_orphan_branch) > 1)
 		die(_("-b, -B and --orphan are mutually exclusive"));
 
+	if (opts.overlay_mode == 1 && opts.patch_mode)
+		die(_("-p and --overlay are mutually exclusive"));
+
 	/*
 	 * From here on, new_branch will contain the branch to be checked out,
 	 * and new_branch_force and new_orphan_branch will tell us which one of
diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
new file mode 100755
index 0000000000..76330cb5ab
--- /dev/null
+++ b/t/t2025-checkout-no-overlay.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+
+test_description='checkout --no-overlay <tree-ish> -- <pathspec>'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git commit --allow-empty -m "initial"
+'
+
+test_expect_success 'checkout --no-overlay deletes files not in <tree-ish>' '
+	>file &&
+	mkdir dir &&
+	>dir/file1 &&
+	git add file dir/file1 &&
+	git checkout --no-overlay HEAD -- file &&
+	test_path_is_missing file &&
+	test_path_is_file dir/file1
+'
+
+test_expect_success 'checkout --no-overlay removing last file from directory' '
+	git checkout --no-overlay HEAD -- dir/file1 &&
+	test_path_is_missing dir
+'
+
+test_expect_success 'checkout -p --overlay is disallowed' '
+	test_must_fail git checkout -p --overlay HEAD 2>actual &&
+	test_i18ngrep "fatal: -p and --overlay are mutually exclusive" actual
+'
+
+test_expect_success '--no-overlay --theirs with D/F conflict deletes file' '
+	test_commit file1 file1 &&
+	test_commit file2 file2 &&
+	git rm --cached file1 &&
+	echo 1234 >file1 &&
+	F1=$(git rev-parse HEAD:file1) &&
+	F2=$(git rev-parse HEAD:file2) &&
+	{
+		echo "100644 $F1 1	file1" &&
+		echo "100644 $F2 2	file1"
+	} | git update-index --index-info &&
+	test_path_is_file file1 &&
+	git checkout --theirs --no-overlay -- file1 &&
+	test_path_is_missing file1
+'
+
+test_done
diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
index d01ad8eb25..5758fffa0d 100755
--- a/t/t9902-completion.sh
+++ b/t/t9902-completion.sh
@@ -1436,6 +1436,7 @@ test_expect_success 'double dash "git checkout"' '
 	--progress Z
 	--no-quiet Z
 	--no-... Z
+	--overlay Z
 	EOF
 '
 
-- 
2.20.1.153.gd81d796ee0

[PATCH v3 8/8] checkout: introduce checkout.overlayMode config

[Thomas Gummerer]

In the previous patch we introduced a new no-overlay mode for git
checkout.  Some users (such as the author of this commit) may want to
have this mode turned on by default as it matches their mental model
more closely.  Make that possible by introducing a new config option
to that extend.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 Documentation/config/checkout.txt |  7 +++++++
 builtin/checkout.c                |  8 +++++++-
 t/t2025-checkout-no-overlay.sh    | 10 ++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt
index c4118fa196..73380a8d86 100644
--- a/Documentation/config/checkout.txt
+++ b/Documentation/config/checkout.txt
@@ -21,3 +21,10 @@ checkout.optimizeNewBranch::
 	will not update the skip-worktree bit in the index nor add/remove
 	files in the working directory to reflect the current sparse checkout
 	settings nor will it show the local changes.
+
+checkout.overlayMode::
+	In the default overlay mode, `git checkout` never
+	removes files from the index or the working tree.  When
+	setting `checkout.overlayMode` to false, files that appear in
+	the index and working tree, but not in <tree-ish> are removed,
+	to make them match <tree-ish> exactly.
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 0c5fe948ef..b5dfc45736 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -1019,13 +1019,19 @@ static int switch_branches(const struct checkout_opts *opts,
 
 static int git_checkout_config(const char *var, const char *value, void *cb)
 {
+	struct checkout_opts *opts = cb;
+
 	if (!strcmp(var, "checkout.optimizenewbranch")) {
 		checkout_optimize_new_branch = git_config_bool(var, value);
 		return 0;
 	}
 
+	if (!strcmp(var, "checkout.overlaymode")) {
+		opts->overlay_mode = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "diff.ignoresubmodules")) {
-		struct checkout_opts *opts = cb;
 		handle_ignore_submodules_arg(&opts->diff_options, value);
 		return 0;
 	}
diff --git a/t/t2025-checkout-no-overlay.sh b/t/t2025-checkout-no-overlay.sh
index 76330cb5ab..a4912e35cb 100755
--- a/t/t2025-checkout-no-overlay.sh
+++ b/t/t2025-checkout-no-overlay.sh
@@ -44,4 +44,14 @@ test_expect_success '--no-overlay --theirs with D/F conflict deletes file' '
 	test_path_is_missing file1
 '
 
+test_expect_success 'checkout with checkout.overlayMode=false deletes files not in <tree-ish>' '
+	>file &&
+	mkdir dir &&
+	>dir/file1 &&
+	git add file dir/file1 &&
+	git -c checkout.overlayMode=false checkout HEAD -- file &&
+	test_path_is_missing file &&
+	test_path_is_file dir/file1
+'
+
 test_done
-- 
2.20.1.153.gd81d796ee0

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Jonathan Nieder]

Hi,

Thomas Gummerer wrote:

> Currently 'git checkout' is defined as an overlay operation, which
> means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
> entry in the index that matches <pathspec>, but that doesn't exist in
> <tree-ish>, that entry will not be removed from the index or the
> working tree.
>
> Introduce a new --{,no-}overlay option, which allows using 'git
> checkout' in non-overlay mode, thus removing files from the working
> tree if they do not exist in <tree-ish> but match <pathspec>.

This patch just hit my workstation.  Some initial thoughts:

I had no idea what --overlay would mean and am still not clear on it.
Is this analogous to "git add --ignore-removal"?  If so, can we just
call it --ignore-removal?

Thank you thank you thank you for working on this.  I run into this
all the time and am super excited about the "default to
--no-ignore-removal" future.

I'm nervous about the config with no associated warning or plan for
phasing it out.  It means that scripts using "git checkout" don't
get a consistent behavior unless they explicitly pass this option,
which didn't exist in older versions of Git --- in other words,
scripts have no real good option.  Can we plan a transition to
making --no-ignore-removal the default, in multiple steps?  For
example:

 1. First introduce the commandline option, as in this series

 2. Next, change the default to warn whenever the difference would
    matter, printing a hint about how to configure to explicitly
    request the old or new behavior.

 3. After a release or two has passed so people get a chance
    to update their scripts, flip the default.

 4. Finally, remove the warning.

 5. Warn whenver the difference would matter when a user has
    requested the old behavior through config, in preparation
    for removing the config.

 6. Remove the config.

Steps 5 and 6 are optional but might be nice.

What do you think?

Thanks,
Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> I'm nervous about the config with no associated warning or plan for
> phasing it out.

This was discussed long ago (in my panda-brain timescale) but my
recollection is to keep "checkout" default to the traditional
"overlay what was read from the tree on top of the current index"
behaviour, while the new "checkout-paths" subcommand (split from the
"checkout" subcommand to produce two subcommands, the other one
being the "checkout-branch" subcommand) would default to the new "no
overlay" behaviour.

So I am not sure if we even need a detailed transition plan.

If we were to make "checkout" pay attention to a local
configuration, that is a different story, as scripts that have
always assumed the overlay behaviour will be broken by such a
configuration variable.  But with the introduction of two new
subcommands in the picture to help interactive end users, I am not
sure if it is even worth considering to allow "checkout" to change
behaviour based on a configuration.  Those who want no-overlay
behaviour can switch to checkout-paths and be done with it, while
scripts can keep relying on the overlay behaviour, no?

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 01/22, Jonathan Nieder wrote:
> Hi,
> 
> Thomas Gummerer wrote:
> 
> > Currently 'git checkout' is defined as an overlay operation, which
> > means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an
> > entry in the index that matches <pathspec>, but that doesn't exist in
> > <tree-ish>, that entry will not be removed from the index or the
> > working tree.
> >
> > Introduce a new --{,no-}overlay option, which allows using 'git
> > checkout' in non-overlay mode, thus removing files from the working
> > tree if they do not exist in <tree-ish> but match <pathspec>.
> 
> This patch just hit my workstation.  Some initial thoughts:
> 
> I had no idea what --overlay would mean and am still not clear on it.
> Is this analogous to "git add --ignore-removal"?  If so, can we just
> call it --ignore-removal?

Yes, it seems like they are very similar.  I'm happy to rename the
option.  The topic seems to have made it to 'next' already, so I'll
submit the patches on top, unless reverting the topic out of next and
replacing it is preferred?

> Thank you thank you thank you for working on this.  I run into this
> all the time and am super excited about the "default to
> --no-ignore-removal" future.

:)

> I'm nervous about the config with no associated warning or plan for
> phasing it out.  It means that scripts using "git checkout" don't
> get a consistent behavior unless they explicitly pass this option,
> which didn't exist in older versions of Git --- in other words,
> scripts have no real good option.  Can we plan a transition to
> making --no-ignore-removal the default, in multiple steps?  For
> example:

As Junio mentioned, the plan was to just have this mode default when
we introduce the new checkout-paths command.

As checkout is a porcelain command, I had hoped it would be okay to
also have this as a configuration option, for the time before
'checkout-paths' exists and while I'm getting used to actually typing
'checkout-paths' instead of 'checkout'.  However I get that there may
be scripts that are using git checkout, and expect the previous
behaviour, so I'm also okay with dropping the config option for now.

If we still want to make this the default even after 'checkout-paths'
exists, the plan you outline below sounds good to me, though maybe we
can make the "flip the default" step once we decide to release git
3.0.

>  1. First introduce the commandline option, as in this series
> 
>  2. Next, change the default to warn whenever the difference would
>     matter, printing a hint about how to configure to explicitly
>     request the old or new behavior.
> 
>  3. After a release or two has passed so people get a chance
>     to update their scripts, flip the default.
> 
>  4. Finally, remove the warning.
> 
>  5. Warn whenver the difference would matter when a user has
>     requested the old behavior through config, in preparation
>     for removing the config.
> 
>  6. Remove the config.
> 
> Steps 5 and 6 are optional but might be nice.
> 
> What do you think?
> 
> Thanks,
> Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Jonathan Nieder]

Thomas Gummerer wrote:
> On 01/22, Jonathan Nieder wrote:

>> I had no idea what --overlay would mean and am still not clear on it.
>> Is this analogous to "git add --ignore-removal"?  If so, can we just
>> call it --ignore-removal?
>
> Yes, it seems like they are very similar.  I'm happy to rename the
> option.  The topic seems to have made it to 'next' already, so I'll
> submit the patches on top, unless reverting the topic out of next and
> replacing it is preferred?

A patch on top sounds good.

[...]
>> I'm nervous about the config with no associated warning or plan for
>> phasing it out.  It means that scripts using "git checkout" don't
>> get a consistent behavior unless they explicitly pass this option,
>> which didn't exist in older versions of Git --- in other words,
>> scripts have no real good option.  Can we plan a transition to
>> making --no-ignore-removal the default, in multiple steps?  For
>> example:
>
> As Junio mentioned, the plan was to just have this mode default when
> we introduce the new checkout-paths command.
>
> As checkout is a porcelain command, I had hoped it would be okay to
> also have this as a configuration option, for the time before
> 'checkout-paths' exists and while I'm getting used to actually typing
> 'checkout-paths' instead of 'checkout'.  However I get that there may
> be scripts that are using git checkout, and expect the previous
> behaviour, so I'm also okay with dropping the config option for now.

Yes, if we have no plan for flipping the default later, then I would
prefer to eliminate the config option.  Scripts very frequently use
human-facing commands like "git checkout" when they want the command
to produce (unparsable) friendly output to show to humans, and I don't
think we've provided a good alternative for that use case.

> If we still want to make this the default even after 'checkout-paths'
> exists, the plan you outline below sounds good to me, though maybe we
> can make the "flip the default" step once we decide to release git
> 3.0.

I would really like this, so I might write a series for it.  Please
don't wait for me, though --- feel free to send any patches you're
thinking about and we can work together or I can just appreciate your
work. ;-)

Sincerely,
Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

>> I had no idea what --overlay would mean and am still not clear on it.
>> Is this analogous to "git add --ignore-removal"?  If so, can we just
>> call it --ignore-removal?
>
> Yes, it seems like they are very similar.

Hmm, I am not sure if the word "removal" makes sense in the context
of "checkout", as "removal" is an _action_ just like "checking out"
itself is, and not a _state_.  You'd check out a state out of a tree
to the index and the working tree, so "checking out absence of a
path" may make sense, though, as "absence of a path" is a state
recorded in that source tree object.

The word "removal" makes little sense in "git add --ignore-removal",
but it and "git add --no-all" outlived their usefulness already, so
it may not be worth _fixing_ it.  But I am mildly opposed to spread
the earlier mistake to a new option.

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Jonathan Nieder]

Hi,

Junio C Hamano wrote:
> Thomas Gummerer <t.gummerer@gmail.com> writes:
>> Jonathan Nieder wrote:

>>> Is this analogous to "git add --ignore-removal"?  If so, can we just
>>> call it --ignore-removal?
>>
>> Yes, it seems like they are very similar.
>
> Hmm, I am not sure if the word "removal" makes sense in the context
> of "checkout", as "removal" is an _action_ just like "checking out"
> itself is, and not a _state_.  You'd check out a state out of a tree
> to the index and the working tree, so "checking out absence of a
> path" may make sense, though, as "absence of a path" is a state
> recorded in that source tree object.

I find --ignore-removal fairly easy to understand, and I had no idea
what --overlay would mean.

I realize this is just one user's experience.  I'd be happy to do a
little informal survey (e.g. taking the description from the manpage
and asking people to name the option) if that's useful.

See also https://dl.acm.org/citation.cfm?id=32212 on this subject.

> The word "removal" makes little sense in "git add --ignore-removal",
> but it and "git add --no-all" outlived their usefulness already, so
> it may not be worth _fixing_ it.  But I am mildly opposed to spread
> the earlier mistake to a new option.

I think that's a good place to end up: once we flip the default for
checkout, then --ignore-removal would be an obscure option in that
command as well.  The consistency with "git add" is just a bonus.

Thanks,
Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 01/23, Jonathan Nieder wrote:
> Hi,
> 
> Junio C Hamano wrote:
> > Thomas Gummerer <t.gummerer@gmail.com> writes:
> >> Jonathan Nieder wrote:
> 
> >>> Is this analogous to "git add --ignore-removal"?  If so, can we just
> >>> call it --ignore-removal?
> >>
> >> Yes, it seems like they are very similar.
> >
> > Hmm, I am not sure if the word "removal" makes sense in the context
> > of "checkout", as "removal" is an _action_ just like "checking out"
> > itself is, and not a _state_.  You'd check out a state out of a tree
> > to the index and the working tree, so "checking out absence of a
> > path" may make sense, though, as "absence of a path" is a state
> > recorded in that source tree object.
> 
> I find --ignore-removal fairly easy to understand, and I had no idea
> what --overlay would mean.

What do you think about --[no-]ignore-removed?  That would not be the same
as we are using in 'git add' though, and the slight difference may be
worse than a different option?  Though I suspect not too many people
are using --ignore-removal in 'git add' in the first place.

> I realize this is just one user's experience.  I'd be happy to do a
> little informal survey (e.g. taking the description from the manpage
> and asking people to name the option) if that's useful.

Sure, that sounds like an option if we can't come to an agreement
here.  What would such a survey look like?

> See also https://dl.acm.org/citation.cfm?id=32212 on this subject.

Sorry I don't have access to this, and unfortunately not the time to
read this either at the moment.

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Thomas Gummerer]

On 01/23, Jonathan Nieder wrote:
> Thomas Gummerer wrote:
> > On 01/22, Jonathan Nieder wrote:
> 
> > As checkout is a porcelain command, I had hoped it would be okay to
> > also have this as a configuration option, for the time before
> > 'checkout-paths' exists and while I'm getting used to actually typing
> > 'checkout-paths' instead of 'checkout'.  However I get that there may
> > be scripts that are using git checkout, and expect the previous
> > behaviour, so I'm also okay with dropping the config option for now.
> 
> Yes, if we have no plan for flipping the default later, then I would
> prefer to eliminate the config option.  Scripts very frequently use
> human-facing commands like "git checkout" when they want the command
> to produce (unparsable) friendly output to show to humans, and I don't
> think we've provided a good alternative for that use case.

Ok, I'm happy to drop that for now, and possibly re-introduce that
with another series to start flipping the default.  I'll probably wait
for Duy's checkout-paths command first though, and possibly send a
series later.

Junio, do you just want to revert the patch (1495ff7da5 ("checkout:
introduce checkout.overlayMode config", 2019-01-08)), or would you
prefer me sending a patch for that?

> > If we still want to make this the default even after 'checkout-paths'
> > exists, the plan you outline below sounds good to me, though maybe we
> > can make the "flip the default" step once we decide to release git
> > 3.0.
> 
> I would really like this, so I might write a series for it.  Please
> don't wait for me, though --- feel free to send any patches you're
> thinking about and we can work together or I can just appreciate your
> work. ;-)
> 
> Sincerely,
> Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> I find --ignore-removal fairly easy to understand, and I had no idea
> what --overlay would mean.
>
> I realize this is just one user's experience.

Exactly.  My impression was the exact opposite from yours.

The phrase "removal" in the context of checkout does not click for
me at all, and neither it does in the context of add, especially
given that Git tracks states (i.e. snapshots), not changes.

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Jonathan Nieder]

Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> I find --ignore-removal fairly easy to understand, and I had no idea
>> what --overlay would mean.
>>
>> I realize this is just one user's experience.
>
> Exactly.  My impression was the exact opposite from yours.
>
> The phrase "removal" in the context of checkout does not click for
> me at all, and neither it does in the context of add, especially
> given that Git tracks states (i.e. snapshots), not changes.

Thanks.  What do you think of --skip-removals (or --skip-deletions)?
The idea is "among the changes that you would be making to the
worktree, skip any unlink() steps".

If that seems sensible, we can use it for "git checkout" and,
optionally, add it as a synonym for --ignore-removal to "git add" as
well.

Jonathan

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Duy Nguyen]

On Fri, Jan 25, 2019 at 9:26 AM Jonathan Nieder <jrnieder@gmail.com> wrote:
>
> Junio C Hamano wrote:
> > Jonathan Nieder <jrnieder@gmail.com> writes:
>
> >> I find --ignore-removal fairly easy to understand, and I had no idea
> >> what --overlay would mean.
> >>
> >> I realize this is just one user's experience.
> >
> > Exactly.  My impression was the exact opposite from yours.
> >
> > The phrase "removal" in the context of checkout does not click for
> > me at all, and neither it does in the context of add, especially
> > given that Git tracks states (i.e. snapshots), not changes.
>
> Thanks.  What do you think of --skip-removals (or --skip-deletions)?
> The idea is "among the changes that you would be making to the
> worktree, skip any unlink() steps".

Another option from rsync: --delete (and --no-delete of course)
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Wed, Dec 12, 2018 at 2:23 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Tue, Dec 11, 2018 at 7:12 AM Elijah Newren <newren@gmail.com> wrote:
> >
> > On Mon, Dec 10, 2018 at 7:13 PM Junio C Hamano <gitster@pobox.com> wrote:
> > >
> > > Duy Nguyen <pclouds@gmail.com> writes:
> > >
> > > > Elijah wanted another mode (and I agree) that modifies worktree but
> > > > leaves the index alone. This is most useful (or least confusing) when
> > > > used with <tree-ish> and would be default in restore-files. I'm not
> > > > saying you have to implement it, but how do the new command line
> > > > options are designed to make sense?
> > >
> > > I'd model it after "git apply", i.e.
> > >
> > >         git restore-files [--tree=<treeish>] <pathspec>
> > >
> > > would work only on the working tree files,
> > >
> > >         git restore-files --tree=<treeish> --cached <pathspec>
> > >
> > > would match the entries in the index that match pathspec to the
> > > given treeish without touching the working tree, and
> > >
> > >         git restore-files --tree=<treeish> --index <pathspec>
> > >
> > > would be both.
> > >
> > > I have never been happy with the phraso, the (arbitrary) distinction
> > > between --cached/--index we use, so in the very longer term (like,
> > > introducing synonym at 3.0 boundary and deprecating older ones at
> > > 4.0 boundary), it may not be a bad idea to rename "--index" to
> > > "--index-and-working-tree" and "--cached" to "--index-only".
> > >
> > > But until that happens, it would be better to use these two
> > > consistently.
> >
> > I agree that consistency is important, but trying to distinguish
> > between "--cached" and "--index" is _extremely_ painful.  I still
> > can't keep the distinction straight and have to look it up every time
> > I want to use either.  I don't know how to possibly teach users the
> > meaning.  Could we introduce synonyms earlier at least, and make the
> > synonyms more prominent than the "--cached" and "--index" terms in the
> > documentation?
>
> For git-checkout I think --index and --cached fit. For restore-files,
> if you come up with better names, I'll gladly use that. Otherwise I'll
> just use these.

I've changed my mind. I'm not using --index and --cached for "git
restore" (formerly "git restore-files"). So how about this?

git restore --from=<tree> <pathspec> will update both the index and worktree.

git restore --from=<tree> --keep-index <pathspec> will not update the index

git restore --from=<tree> --keep-worktree <pathspec> will not update worktree

And I'm thinking of adding these to "git reset" too. It will have the
forth form:

git reset [--keep-index] [--keep-worktree] [--keep-nothing] [<commit>]

which is similar to --soft, --mixed and --hard, but with better names:
--soft is --keep-worktree and --keep-index, --mixed is --keep-worktree
and --hard is --keep-nothing (i.e. the only change is HEAD).

Although, if I work it out, I might just ignore "git reset" and make
sure "git switch" and "git restore" can do whatever "git reset" can
then remove its "common command" status. This part is more thinking
out loud, but "git switch" can have a new form

git switch --reset-branch [--keep-worktree] [--keep-index] <start-point>

which resets "HEAD" (and switch of course) but do not enter detached
mode. This covers a common use case of "git reset [options] <commit>".
Then either "git restore" learns about "--abort-in-progress" to abort
cherry-pick/merge/revert, or those commands have a new --abort and
--quit option...
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> I've changed my mind. I'm not using --index and --cached for "git
> restore" (formerly "git restore-files"). So how about this?
>
> git restore --from=<tree> <pathspec> will update both the index and worktree.
>
> git restore --from=<tree> --keep-index <pathspec> will not update the index
>
> git restore --from=<tree> --keep-worktree <pathspec> will not update worktree

An action to "restore" with an option to "keep" (i.e. "do not
touch") smells strongly of double negation.  We are restoring,
i.e. grabbing something that existed in the past out of another
place (like tree or index) and depositing it to the working tree to
recover its previous state, oh, not but not touching the content of
the working tree (or the index) intact?

It would be great if you can come up with phrasing that avoids
specifying what is *not* done, but instructs the command what is to
be done, perhaps along the lines of "restore --index-only", "restore
--worktree-only" and "restore --index-and-worktree (which would be
the default)".

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Fri, Feb 1, 2019 at 2:05 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Duy Nguyen <pclouds@gmail.com> writes:
>
> > I've changed my mind. I'm not using --index and --cached for "git
> > restore" (formerly "git restore-files"). So how about this?
> >
> > git restore --from=<tree> <pathspec> will update both the index and worktree.
> >
> > git restore --from=<tree> --keep-index <pathspec> will not update the index
> >
> > git restore --from=<tree> --keep-worktree <pathspec> will not update worktree
>
> An action to "restore" with an option to "keep" (i.e. "do not
> touch") smells strongly of double negation.  We are restoring,
> i.e. grabbing something that existed in the past out of another
> place (like tree or index) and depositing it to the working tree to
> recover its previous state, oh, not but not touching the content of
> the working tree (or the index) intact?

It is negation (though not double). The thinking was, since "git
restore" means restore everything, extra options will amend that to
"restore everything except ..." and because the parts to keep are more
important (i.e. data loss), highlighting it on the command sounds a
bit better to me: "ok i'm going to need to restore this thing, but I
want this part and that part to stay the same, let's type the
command..."

> It would be great if you can come up with phrasing that avoids
> specifying what is *not* done, but instructs the command what is to
> be done, perhaps along the lines of "restore --index-only", "restore
> --worktree-only" and "restore --index-and-worktree (which would be
> the default)".

My biggest problem with --*-only is that they cannot be combined.
There must be an option for every valid combination. I'm still
thinking about "git reset" where there are three parts to
reset/restore: HEAD, index and worktree. So at least there's another
valid combination "reset HEAD but not index nor worktree", which would
end becoming another option. This route does not look promising.

Of course we could just do --index and --worktree, each option
restores the respective part. Then it's combinable (and extensible in
the future). But then "git restore" means "git restore --index
--worktree" and typing "git restore --index" effectively removes the
default "--worktree", which seems a bit twisted.
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> Of course we could just do --index and --worktree, each option
> restores the respective part. Then it's combinable (and extensible in
> the future). But then "git restore" means "git restore --index
> --worktree" and typing "git restore --index" effectively removes the
> default "--worktree", which seems a bit twisted.

Or "git restore --no-worktree" (essentially, instead of saying
"keep", say "no" to mean "negation").

Incidentally, "git restore --no-index" does not have a counterpart
in "git checkout", but I think it is probably a good thing to add;
as it has to do far more than "git cat-file blob $tree:$path >$path"
these days.

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Sat, Feb 2, 2019 at 12:57 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Duy Nguyen <pclouds@gmail.com> writes:
>
> > Of course we could just do --index and --worktree, each option
> > restores the respective part. Then it's combinable (and extensible in
> > the future). But then "git restore" means "git restore --index
> > --worktree" and typing "git restore --index" effectively removes the
> > default "--worktree", which seems a bit twisted.
>
> Or "git restore --no-worktree" (essentially, instead of saying
> "keep", say "no" to mean "negation").
>
> Incidentally, "git restore --no-index" does not have a counterpart
> in "git checkout", but I think it is probably a good thing to add;
> as it has to do far more than "git cat-file blob $tree:$path >$path"
> these days.

Oh yes, I occasionally need that too.
-- 
Duy

Re: [PATCH v3 7/8] checkout: introduce --{,no-}overlay option

[Philip Oakley]

Hi,

On 24/01/2019 01:12, Jonathan Nieder wrote:
> Hi,
>
> Junio C Hamano wrote:
>> Thomas Gummerer<t.gummerer@gmail.com>  writes:
>>> Jonathan Nieder wrote:
>>>> Is this analogous to "git add --ignore-removal"?  If so, can we just
>>>> call it --ignore-removal?
>>> Yes, it seems like they are very similar.
>> Hmm, I am not sure if the word "removal" makes sense in the context
>> of "checkout", as "removal" is an_action_  just like "checking out"
>> itself is, and not a_state_.  You'd check out a state out of a tree
>> to the index and the working tree, so "checking out absence of a
>> path" may make sense, though, as "absence of a path" is a state
>> recorded in that source tree object.
> I find --ignore-removal fairly easy to understand, and I had no idea
> what --overlay would mean.

I too had difficulty initially as to what 'overlay' meant, or that there 
were options.


>
> I realize this is just one user's experience.  I'd be happy to do a
> little informal survey (e.g. taking the description from the manpage
> and asking people to name the option) if that's useful.
>
> See alsohttps://dl.acm.org/citation.cfm?id=32212  on this subject.

I did locate a copy at 
http://zhang.ist.psu.edu/teaching/501/readings/Furnas.pdf

The whole word choosing problem does smack a bit of Orwell's Vocabulary 
C (OVC) where:

"The [Newspeak] C vocabulary encompasses words that relate specifically 
to science and to technical fields and disciplines. It is designed to 
ensure that technical knowledge remains segmented among many fields, so 
that no one individual can gain access to too much knowledge. In fact, 
there is no word for “science” " [1]

Most of the DVCS concepts have a newness to them that means that we 
don't have good words yet, hence the difficulties. Just my 2 cents.

>> The word "removal" makes little sense in "git add --ignore-removal",
>> but it and "git add --no-all" outlived their usefulness already, so
>> it may not be worth_fixing_  it.  But I am mildly opposed to spread
>> the earlier mistake to a new option.
> I think that's a good place to end up: once we flip the default for
> checkout, then --ignore-removal would be an obscure option in that
> command as well.  The consistency with "git add" is just a bonus.
>
> Thanks,
> Jonathan

Philip

[1] https://www.sparknotes.com/lit/1984/section11/

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Sat, Feb 2, 2019 at 12:57 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Duy Nguyen <pclouds@gmail.com> writes:
>
> > Of course we could just do --index and --worktree, each option
> > restores the respective part. Then it's combinable (and extensible in
> > the future). But then "git restore" means "git restore --index
> > --worktree" and typing "git restore --index" effectively removes the
> > default "--worktree", which seems a bit twisted.
>
> Or "git restore --no-worktree" (essentially, instead of saying
> "keep", say "no" to mean "negation").
>
> Incidentally, "git restore --no-index" does not have a counterpart
> in "git checkout", but I think it is probably a good thing to add;
> as it has to do far more than "git cat-file blob $tree:$path >$path"
> these days.

OK this hopefully will be the final design

(git restore) "[--worktree] <paths>" restores worktree paths from index

"--index <paths>" restores the index from HEAD (aka "git reset")

"--source <tree> (--index|--worktree) <paths>" restores index or
worktree (or both) from <tree>

I'm a bit reluctant to support "git restore --index --worktree
<paths>" without --source, which should default to HEAD, since it's a
bit unclear/inconsistent ("git restore --worktree <paths>" defaults to
index as the source, but here we use a different default source).
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

Hi Duy,

On Mon, Feb 18, 2019 at 8:21 PM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Sat, Feb 2, 2019 at 12:57 AM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > Duy Nguyen <pclouds@gmail.com> writes:
> >
> > > Of course we could just do --index and --worktree, each option
> > > restores the respective part. Then it's combinable (and extensible in
> > > the future). But then "git restore" means "git restore --index
> > > --worktree" and typing "git restore --index" effectively removes the
> > > default "--worktree", which seems a bit twisted.
> >
> > Or "git restore --no-worktree" (essentially, instead of saying
> > "keep", say "no" to mean "negation").
> >
> > Incidentally, "git restore --no-index" does not have a counterpart
> > in "git checkout", but I think it is probably a good thing to add;
> > as it has to do far more than "git cat-file blob $tree:$path >$path"
> > these days.
>
> OK this hopefully will be the final design
>
> (git restore) "[--worktree] <paths>" restores worktree paths from index
>
> "--index <paths>" restores the index from HEAD (aka "git reset")
>
> "--source <tree> (--index|--worktree) <paths>" restores index or
> worktree (or both) from <tree>
>
> I'm a bit reluctant to support "git restore --index --worktree
> <paths>" without --source, which should default to HEAD, since it's a
> bit unclear/inconsistent ("git restore --worktree <paths>" defaults to
> index as the source, but here we use a different default source).

Sorry for going missing a while from the conversation, and thanks so
much for pushing this forward.

Overall this looks good, but there's just one part that confuses me.
Here you seem to suggest that if you pass --source but neither --index
or --worktree that both the index and working tree will be written to.
Why are "restored" changes considered ready for commit?  That seems
confusing to me (and was one of the bugs of checkout, IMO).  See also
second half of https://public-inbox.org/git/xmqq1s6yezk3.fsf@gitster-ct.c.googlers.com/

Elijah

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Tue, Feb 19, 2019 at 9:42 PM Elijah Newren <newren@gmail.com> wrote:
> > OK this hopefully will be the final design
> >
> > (git restore) "[--worktree] <paths>" restores worktree paths from index
> >
> > "--index <paths>" restores the index from HEAD (aka "git reset")
> >
> > "--source <tree> (--index|--worktree) <paths>" restores index or
> > worktree (or both) from <tree>
> >
> > I'm a bit reluctant to support "git restore --index --worktree
> > <paths>" without --source, which should default to HEAD, since it's a
> > bit unclear/inconsistent ("git restore --worktree <paths>" defaults to
> > index as the source, but here we use a different default source).
>
> Sorry for going missing a while from the conversation, and thanks so
> much for pushing this forward.
>
> Overall this looks good, but there's just one part that confuses me.
> Here you seem to suggest that if you pass --source but neither --index
> or --worktree that both the index and working tree will be written to.

No no. Sorry for not being clear. If neither --index or --worktree is
given, the default will be --worktree. --source changes the "source"
but cannot influence target selection. So "git restore --source=HEAD
foo.c" will restore foo.c on the worktree but not the index. I still
remember that point you and Stefan Xenos raised ;-)

Full git-restore.txt is here if you're interested. I think I'll send
it out soon once git-switch part more or less settles.

https://gitlab.com/pclouds/git/blob/switch-and-restore-forever/Documentation/git-restore.txt

> Why are "restored" changes considered ready for commit?  That seems
> confusing to me (and was one of the bugs of checkout, IMO).  See also
> second half of https://public-inbox.org/git/xmqq1s6yezk3.fsf@gitster-ct.c.googlers.com/
>
> Elijah



-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> On Sat, Feb 2, 2019 at 12:57 AM Junio C Hamano <gitster@pobox.com> wrote:
>>
>> Duy Nguyen <pclouds@gmail.com> writes:
>>
>> > Of course we could just do --index and --worktree, each option
>> > restores the respective part. Then it's combinable (and extensible in
>> > the future). But then "git restore" means "git restore --index
>> > --worktree" and typing "git restore --index" effectively removes the
>> > default "--worktree", which seems a bit twisted.
>>
>> Or "git restore --no-worktree" (essentially, instead of saying
>> "keep", say "no" to mean "negation").
>>
>> Incidentally, "git restore --no-index" does not have a counterpart
>> in "git checkout", but I think it is probably a good thing to add;
>> as it has to do far more than "git cat-file blob $tree:$path >$path"
>> these days.
>
> OK this hopefully will be the final design
>
> (git restore) "[--worktree] <paths>" restores worktree paths from index
>
> "--index <paths>" restores the index from HEAD (aka "git reset")
>
> "--source <tree> (--index|--worktree) <paths>" restores index or
> worktree (or both) from <tree>
>
> I'm a bit reluctant to support "git restore --index --worktree
> <paths>" without --source, which should default to HEAD, since it's a
> bit unclear/inconsistent ("git restore --worktree <paths>" defaults to
> index as the source, but here we use a different default source).

Ok, so we grab things from the index by default, but with --source
<tree>, we can tell the command to grab things from the tree.  When
we are explicitly told to update the index with "--index", it would
be nonsense to grab things from the index, so "--source <tree>"
becomes required in that case.  Makes sense.

To summerize and full enumerate all the allowed variants:

 * --index --worktree --source <tree> <path>...

   Update both from a tree; --source <tree> is required.

 * --index --no-worktree --source <tree> <path>...

   Update only the index but not the working tree; --source <tree>
   is required.


 * --no-index --worktree <path>...

   Update only the working tree files without touching the index
   (new feature that cannot be done with the current Git, although
   "cat-file -p >path" may be close enough at times), from the index

 * --no-index --worktree --source <tree> <path>...

   Update only the working tree files without touching the index
   from a tree-ish.

 * --no-index --no-worktree <path>...

   Update nothing, which is a no-op that is not all that useful.

 * --no-index --no-worktree --source <tree> <path>...

   Update nothing, which is a no-op that is not all that useful.

I am getting the impression that to save typing, you would want to
make "--index --worktree" the default (i.e. among the above, only
--no-index and --no-worktree need to be spelled explicitly), but
there is one glitch.  Updating from the index must be spelled
explicitly with "--no-index --worktree".

So perhaps the defaulting rule for the "--index" option must become
a bit more tricky.  Perhaps the rules are:

 * --worktree is the defeault; --no-worktree can be given from the
   command line to countermand it, and --worktree can be given from
   the command line to be more explicit.

 * when --source <tree> is given from the command line, --index is
   the default, and --no-index can be given to countermand it.

 * when --source <tree> is not given from the command line,
   --no-index is the only sensible choice.  It can be given from the
   command line to be more explicit, but giving --index to
   countermand the --no-index default would be an error, as updating
   the index, whether the same update also goes to the working tree,
   must come from a --source <tree>.

Am I following you correctly?

Thanks.

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Overall this looks good, but there's just one part that confuses me.
> Here you seem to suggest that if you pass --source but neither --index
> or --worktree that both the index and working tree will be written to.
> Why are "restored" changes considered ready for commit?  That seems
> confusing to me (and was one of the bugs of checkout, IMO).  See also
> second half of https://public-inbox.org/git/xmqq1s6yezk3.fsf@gitster-ct.c.googlers.com/

As long as worktree-only mode does not lose track of a
previously-untracked path in the index (perhaps use the i-t-a bit),
I do not have a strong objection against making the worktree-only
mode the default.

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I am getting the impression that to save typing, you would want to
> make "--index --worktree" the default (i.e. among the above, only
> --no-index and --no-worktree need to be spelled explicitly), but
> there is one glitch.  Updating from the index must be spelled
> explicitly with "--no-index --worktree".

And after getting reminded by Elijah, the default pair is
<--no-index, --worktree>.

> So perhaps the defaulting rule for the "--index" option must become
> a bit more tricky.  Perhaps the rules are:
>
>  * --worktree is the default; --no-worktree can be given from the
>    command line to countermand it, and --worktree can be given from
>    the command line to be more explicit.
>
>  * when --source <tree> is given from the command line, --index is
>    the default, and --no-index can be given to countermand it.

Correction.

    * when --source <tree> is given, --no-index is the default, but
      --index can be given to countermand it.

>
>  * when --source <tree> is not given from the command line,
>    --no-index is the only sensible choice.  It can be given from the
>    command line to be more explicit, but giving --index to
>    countermand the --no-index default would be an error, as updating
>    the index, whether the same update also goes to the working tree,
>    must come from a --source <tree>.

This is still correct.

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Tue, Feb 19, 2019 at 11:10 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > I am getting the impression that to save typing, you would want to
> > make "--index --worktree" the default (i.e. among the above, only
> > --no-index and --no-worktree need to be spelled explicitly), but
> > there is one glitch.  Updating from the index must be spelled
> > explicitly with "--no-index --worktree".
>
> And after getting reminded by Elijah, the default pair is
> <--no-index, --worktree>.

Why would you want --no-index or --no-worktree as flags?  That seems
to presume a default of modifying both the index and the working tree,
as these names imply undoing pieces of such a default.

I'd rather have a flag like --worktree which alone only modifies the
working tree and is presumed to be the default (but useful to be
explicit or as mentioned later), have a flag for applying the changes
to the index instead (--index?), and treat applying to both the
working tree and the index as unusual and require either both flags
(--worktree --index ?) or some special flag that likely has a longer
name (--worktree-and-index?).

I _think_ Duy does the latter reading over his manpage that he linked
to, but maybe I'm just reading my own biases into it.

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Tue, Feb 19, 2019 at 11:03 AM Junio C Hamano <gitster@pobox.com> wrote:
>  * --no-index --worktree <path>...
>
>    Update only the working tree files without touching the index
>    (new feature that cannot be done with the current Git, although
>    "cat-file -p >path" may be close enough at times), from the index

I worry slightly that wording like this might paint ourselves into a
similar corner of only correctly handling files as paths, rather than
also handling directories as paths, and/or which presumes the
no-overlay mode that I find problematic.  Maybe your "at times" just
meant "when dealing with a path that is a file that existed in the
given source", but I wonder if you are pushing more than that?

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Tue, Feb 19, 2019 at 11:07 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> > Overall this looks good, but there's just one part that confuses me.
> > Here you seem to suggest that if you pass --source but neither --index
> > or --worktree that both the index and working tree will be written to.
> > Why are "restored" changes considered ready for commit?  That seems
> > confusing to me (and was one of the bugs of checkout, IMO).  See also
> > second half of https://public-inbox.org/git/xmqq1s6yezk3.fsf@gitster-ct.c.googlers.com/
>
> As long as worktree-only mode does not lose track of a
> previously-untracked path in the index (perhaps use the i-t-a bit),
> I do not have a strong objection against making the worktree-only
> mode the default.

Could you unpack that for me a bit?  My assumption is that
worktree-only mode doesn't touch the index (other than maybe reading
from it), and treating the index as read-only means by definition it
can't lose anything from there -- but then you mentioned using the
intent-to-add bit, and I feel like I'm missing an important puzzle
piece somewhere.  Trying to make sense of it, I'm wondering if you are
objecting to using overlay mode in general, or are trying to connect
this to the new "precious" concept being advanced, or if there's
something else you are considering here.

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> On Tue, Feb 19, 2019 at 11:10 AM Junio C Hamano <gitster@pobox.com> wrote:
>>
>> Junio C Hamano <gitster@pobox.com> writes:
>>
>> > I am getting the impression that to save typing, you would want to
>> > make "--index --worktree" the default (i.e. among the above, only
>> > --no-index and --no-worktree need to be spelled explicitly), but
>> > there is one glitch.  Updating from the index must be spelled
>> > explicitly with "--no-index --worktree".
>>
>> And after getting reminded by Elijah, the default pair is
>> <--no-index, --worktree>.
>
> Why would you want --no-index or --no-worktree as flags?  That seems
> to presume a default of modifying both the index and the working tree,
> as these names imply undoing pieces of such a default.

By "flags" I think you mean "treat them as two orthogonal booleans"?

It was just how I read Duy's examples (especially the "both --index
and --worktree given" example where "--source <tree>" becomes
mandatory).  I do not have strong preference either way myself, but
I tend to think that treating these as two independent booleans
would be a way to make it clear that the new design departs from
what we have been doing (i.e. "--index" means "both", "--cached"
means "index only" and if we were to introduce the "cat-file -p >"
variant that would be called "--worktree-only"; in these, there is
no "two orthogonal booleans" that can be mixed---instead they come
premixed).

> I'd rather have a flag like --worktree which alone only modifies the
> working tree and is presumed to be the default (but useful to be
> explicit or as mentioned later), have a flag for applying the changes
> to the index instead (--index?), and treat applying to both the
> working tree and the index as unusual and require either both flags
> (--worktree --index ?) or some special flag that likely has a longer
> name (--worktree-and-index?).

So you are in favor of pre-mixed set of options, all are mutually
exclusive.  Which I am perfectly fine with.


I however do think "both worktree and index" is quite common when
tweaking the tree to prepare for the next commit.  A path with
contents freshly taken out of an existing tree may not match exactly
what you plan to record in your next commit, and you would not be
recording it immediately in a commit as-is.  But having the contents
taken from an existing tree recorded as the baseline in the index
would make "git diff (no tree-ish) <path>" a handy tool to review
your progress toward the next commit since that baseline state, in
addition to "git diff HEAD <path>" that is the usual "how does the
overall change relative to the parent of the commit I am preparing
for look like".

So I'd hesitate to endorse "a deliberately longer and harder to type
option" to populate both the index and the working tree files at the
same time.

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> On Tue, Feb 19, 2019 at 11:03 AM Junio C Hamano <gitster@pobox.com> wrote:
>>  * --no-index --worktree <path>...
>>
>>    Update only the working tree files without touching the index
>>    (new feature that cannot be done with the current Git, although
>>    "cat-file -p >path" may be close enough at times), from the index
>
> I worry slightly that wording like this might paint ourselves into a
> similar corner of only correctly handling files as paths, rather than
> also handling directories as paths, and/or which presumes the
> no-overlay mode that I find problematic.  Maybe your "at times" just
> meant "when dealing with a path that is a file that existed in the
> given source", but I wonder if you are pushing more than that?

No, I meant "when you are not doing smudge filters (including eol
conversion)", aka "'cat-file -p' gives you raw blob contents, which
may not be what you want in your worktree."

Re: [PATCH 6/8] checkout: add --cached option

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

>> As long as worktree-only mode does not lose track of a
>> previously-untracked path in the index (perhaps use the i-t-a bit),
>> I do not have a strong objection against making the worktree-only
>> mode the default.
>
> Could you unpack that for me a bit?

Suppose in an ancient version v1.0 there was a file called
README.txt but these days such a file does not exist (there may be
README.md added though).  By default, the command does not stuff the
contents to the index out of the tree and instead operate only on
the working tree.

  $ git restore-path --source v1.0 README.txt

At this point, you are assuming that README.txt will become
untracked and the user needs to manually add it.  I was asking if it
makes sense to at least make the index "aware of" it with I-T-A bit,
and I am leaning towards answering "yes" to that question.

Re: [PATCH 6/8] checkout: add --cached option

[Elijah Newren]

On Tue, Feb 19, 2019 at 2:36 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> >> As long as worktree-only mode does not lose track of a
> >> previously-untracked path in the index (perhaps use the i-t-a bit),
> >> I do not have a strong objection against making the worktree-only
> >> mode the default.
> >
> > Could you unpack that for me a bit?
>
> Suppose in an ancient version v1.0 there was a file called
> README.txt but these days such a file does not exist (there may be
> README.md added though).  By default, the command does not stuff the
> contents to the index out of the tree and instead operate only on
> the working tree.
>
>   $ git restore-path --source v1.0 README.txt
>
> At this point, you are assuming that README.txt will become
> untracked and the user needs to manually add it.  I was asking if it
> makes sense to at least make the index "aware of" it with I-T-A bit,
> and I am leaning towards answering "yes" to that question.

Ah, thanks for all the clarifications (for this and the other emails).
Your suggestion here would mean that --worktree alone wouldn't
actually treat the index as read-only, but I too am leaning towards
the thought that it makes sense to set the i-t-a bit for such cases.

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Wed, Feb 20, 2019 at 5:36 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> >> As long as worktree-only mode does not lose track of a
> >> previously-untracked path in the index (perhaps use the i-t-a bit),
> >> I do not have a strong objection against making the worktree-only
> >> mode the default.
> >
> > Could you unpack that for me a bit?
>
> Suppose in an ancient version v1.0 there was a file called
> README.txt but these days such a file does not exist (there may be
> README.md added though).  By default, the command does not stuff the
> contents to the index out of the tree and instead operate only on
> the working tree.
>
>   $ git restore-path --source v1.0 README.txt
>
> At this point, you are assuming that README.txt will become
> untracked and the user needs to manually add it.  I was asking if it
> makes sense to at least make the index "aware of" it with I-T-A bit,
> and I am leaning towards answering "yes" to that question.

I completely forgot about i-t-a! Do we want the command to add i-t-a
bit by default, or only when --intent-to-add is specified? So far
i-t-a has been a conscious choice, both "git reset <commit>" and "git
add" require --intent-to-add to use it. The first safe step could be
require --intent-to-add even in git-restore, then we could introduce a
config key that enables --intent-to-add in both "git restore" and "git
reset" (and perhaps "git apply" later, I still need to finish that
part).
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Wed, Feb 20, 2019 at 2:10 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > I am getting the impression that to save typing, you would want to
> > make "--index --worktree" the default (i.e. among the above, only
> > --no-index and --no-worktree need to be spelled explicitly), but
> > there is one glitch.  Updating from the index must be spelled
> > explicitly with "--no-index --worktree".
>
> And after getting reminded by Elijah, the default pair is
> <--no-index, --worktree>.
>
> > So perhaps the defaulting rule for the "--index" option must become
> > a bit more tricky.  Perhaps the rules are:
> >
> >  * --worktree is the default; --no-worktree can be given from the
> >    command line to countermand it, and --worktree can be given from
> >    the command line to be more explicit.

I originally went with that (--no-worktree to negate default
settings), but after updating docs to use "git restore" it's just too
much to write. My current rules are

 - default is --worktree
 - as soon as any of --[no-]worktree or --[no-]index is specified, the
above line is void. There is no default, what you specify is what you
get.

While it's a bit more twisted than spelling out --no-worktree to
countermand the default, i think it's more natural to think and write
it: ok i'm going to need to restore this and that so I write --this
and --that. Going with --no-worktree you need to think "i need to
write this and that and counter the default which is that other
thing". We essentially have two modes, the default mode and explicit
mode. Too bad?

> >  * when --source <tree> is given from the command line, --index is
> >    the default, and --no-index can be given to countermand it.
>
> Correction.
>
>     * when --source <tree> is given, --no-index is the default, but
>       --index can be given to countermand it.
>
> >
> >  * when --source <tree> is not given from the command line,
> >    --no-index is the only sensible choice.  It can be given from the
> >    command line to be more explicit, but giving --index to
> >    countermand the --no-index default would be an error, as updating
> >    the index, whether the same update also goes to the working tree,
> >    must come from a --source <tree>.
>
> This is still correct.

So I guess I scratch the default source for "--index --no-worktree"
(or just --index in my rules above) then? It does make sense to
default restoring the index from HEAD. And it makes "git status"
suggestion to unstage a bit shorter ("git restore --index <paths>"
instead of "git restore --source=HEAD --index <paths>")
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Wed, Feb 20, 2019 at 5:29 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> > On Tue, Feb 19, 2019 at 11:10 AM Junio C Hamano <gitster@pobox.com> wrote:
> >>
> >> Junio C Hamano <gitster@pobox.com> writes:
> >>
> >> > I am getting the impression that to save typing, you would want to
> >> > make "--index --worktree" the default (i.e. among the above, only
> >> > --no-index and --no-worktree need to be spelled explicitly), but
> >> > there is one glitch.  Updating from the index must be spelled
> >> > explicitly with "--no-index --worktree".
> >>
> >> And after getting reminded by Elijah, the default pair is
> >> <--no-index, --worktree>.
> >
> > Why would you want --no-index or --no-worktree as flags?  That seems
> > to presume a default of modifying both the index and the working tree,
> > as these names imply undoing pieces of such a default.
>
> By "flags" I think you mean "treat them as two orthogonal booleans"?
>
> It was just how I read Duy's examples (especially the "both --index
> and --worktree given" example where "--source <tree>" becomes
> mandatory).  I do not have strong preference either way myself, but
> I tend to think that treating these as two independent booleans
> would be a way to make it clear that the new design departs from
> what we have been doing (i.e. "--index" means "both", "--cached"
> means "index only" and if we were to introduce the "cat-file -p >"
> variant that would be called "--worktree-only"; in these, there is
> no "two orthogonal booleans" that can be mixed---instead they come
> premixed).

There is indeed inconsistency in "git status" advice with my patches,
where it suggests both "git restore --index" and "git rm --cached",
both operate only in the index but with different option name.

"--index --no-worktree" is one way to deal with this. Another way may
be avoid --index in git-restore and use another name like
--staging-area or something? That avoids the name clash, and we could
even add --worktree and --staging-area to "git rm" / "git apply"
later, deprecating (but still supporting) --index and --cached there.
-- 
Duy

Re: [PATCH 6/8] checkout: add --cached option

[Duy Nguyen]

On Wed, Feb 20, 2019 at 5:05 AM Elijah Newren <newren@gmail.com> wrote:
>
> On Tue, Feb 19, 2019 at 11:10 AM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > Junio C Hamano <gitster@pobox.com> writes:
> >
> > > I am getting the impression that to save typing, you would want to
> > > make "--index --worktree" the default (i.e. among the above, only
> > > --no-index and --no-worktree need to be spelled explicitly), but
> > > there is one glitch.  Updating from the index must be spelled
> > > explicitly with "--no-index --worktree".
> >
> > And after getting reminded by Elijah, the default pair is
> > <--no-index, --worktree>.
>
> Why would you want --no-index or --no-worktree as flags?  That seems
> to presume a default of modifying both the index and the working tree,
> as these names imply undoing pieces of such a default.
>
> I'd rather have a flag like --worktree which alone only modifies the
> working tree and is presumed to be the default (but useful to be
> explicit or as mentioned later), have a flag for applying the changes
> to the index instead (--index?), and treat applying to both the
> working tree and the index as unusual and require either both flags
> (--worktree --index ?) or some special flag that likely has a longer
> name (--worktree-and-index?).

I'd prefer separate options. I even gave --worktree and --index
shortcuts so we could write "git restore -IW" if "git restore --index
--worktree" is used often (and I think it could be an alternative for
"git reset --hard HEAD" if --force is also specified)

> I _think_ Duy does the latter reading over his manpage that he linked
> to, but maybe I'm just reading my own biases into it.

Nah you read it right. The "examples" section also shows it.
-- 
Duy