[PATCH v2 0/8] Additional metadata for filter processes

[PATCH v2 0/8] Additional metadata for filter processes

[brian m. carlson]

Smudge and clean filters are currently provided with one particular
piece of data: the pathname of the file being smudged.  While this is
helpful, there are a variety of situations where people would like to
have more data.  This series introduces additional data for these
filters.

I decided that there is indeed value to providing the full ref when
using git checkout, so an invocation that uses a tag will now pass that
tag along.  It's easy for filters that don't want this data to simply
ignore it, but significantly harder to recover if it's not provided.

One thing I should mention here that may seem obvious: this metadata
only appears for smudge filters.  We obviously cannot know the blob when
data is being cleaned, and the ref, commit, and tree are already known
at that point (because one is using "git add" or an equivalent), so the
filter can simply do a git rev-parse if it's curious.

Changes from v1:
* Add support for arbitrary refs to checkout.
* Use FREE_AND_NULL where appropriate.
* Improve commit message to explain expected result and data as well as
  rationale.

brian m. carlson (8):
  builtin/checkout: pass branch info down to checkout_worktree
  convert: permit passing additional metadata to filter processes
  convert: provide additional metadata to filters
  builtin/checkout: compute checkout metadata for checkouts
  builtin/clone: compute checkout metadata for clones
  builtin/rebase: compute checkout metadata for rebases
  builtin/reset: compute checkout metadata for reset
  t0021: test filter metadata for additional cases

 apply.c                 |   2 +-
 archive.c               |  13 ++-
 archive.h               |   1 +
 builtin/cat-file.c      |   5 +-
 builtin/checkout.c      |  64 +++++++++----
 builtin/clone.c         |   5 +-
 builtin/rebase.c        |   1 +
 builtin/reset.c         |  16 +++-
 cache.h                 |   1 +
 convert.c               |  66 ++++++++++++--
 convert.h               |  29 +++++-
 diff.c                  |   5 +-
 entry.c                 |   7 +-
 merge-recursive.c       |   2 +-
 merge.c                 |   1 +
 sequencer.c             |   1 +
 t/t0021-conversion.sh   | 198 ++++++++++++++++++++++++++++++++++------
 t/t0021/rot13-filter.pl |   6 ++
 unpack-trees.c          |   1 +
 unpack-trees.h          |   1 +
 20 files changed, 350 insertions(+), 75 deletions(-)

Range-diff against v1:
1:  93111f0c13 = 1:  77b064093c builtin/checkout: pass branch info down to checkout_worktree
2:  7e3675fc33 ! 2:  d910bf619c convert: permit passing additional metadata to filter processes
    @@ Commit message
         pass this information in all cases, such as when renormalizing or when
         we're performing diffs, since it doesn't make sense in those cases.
     
    -    This series wires up the code to print this information, but doesn't
    +    The data we currently get from the filter process looks like the
    +    following:
    +
    +      command=smudge
    +      pathname=git.c
    +      0000
    +
    +    With this change, we'll get data more like this:
    +
    +      command=smudge
    +      pathname=git.c
    +      refname=refs/tags/v2.25.1
    +      treeish=c522f061d551c9bb8684a7c3859b2ece4499b56b
    +      blob=7be7ad34bd053884ec48923706e70c81719a8660
    +      0000
    +
    +    There are a couple things to note about this approach.  For operations
    +    like checkout, treeish will always be a commit, since we cannot check
    +    out individual trees, but for other operations, like archive, we can end
    +    up operating on only a particular tree, so we'll provide only a tree as
    +    the treeish.  Similar comments apply for refname, since there are a
    +    variety of cases in which we won't have a ref.
    +
    +    This commit wires up the code to print this information, but doesn't
         pass any of it at this point.  In a future commit, we'll have various
         code paths pass the actual useful data down.
     
3:  ed056a4516 ! 3:  ab2730d9a3 convert: provide additional metadata to filters
    @@ builtin/cat-file.c: static int filter_object(const char *path, unsigned mode,
      			*buf = strbuf_detach(&strbuf, NULL);
     
      ## builtin/checkout.c ##
    +@@ builtin/checkout.c: struct branch_info {
    + 	const char *name; /* The short name used */
    + 	const char *path; /* The full name of a real branch */
    + 	struct commit *commit; /* The named commit */
    ++	char *refname; /* The full name of the ref being checked out. */
    ++	struct object_id oid; /* The object ID of the commit being checked out. */
    + 	/*
    + 	 * if not null the branch is detached because it's already
    + 	 * checked out in this checkout
     @@ builtin/checkout.c: static int checkout_worktree(const struct checkout_opts *opts,
      	state.refresh_cache = 1;
      	state.istate = &the_index;
      
    -+	init_checkout_metadata(&state.meta, info->path,
    -+			       info->commit ? &info->commit->object.oid : NULL,
    ++	init_checkout_metadata(&state.meta, info->refname,
    ++			       info->commit ? &info->commit->object.oid : &info->oid,
     +			       NULL);
     +
      	enable_delayed_checkout(&state);
      	for (pos = 0; pos < active_nr; pos++) {
      		struct cache_entry *ce = active_cache[pos];
    +@@ builtin/checkout.c: static void setup_branch_path(struct branch_info *branch)
    + {
    + 	struct strbuf buf = STRBUF_INIT;
    + 
    ++	/*
    ++	 * If this is a ref, resolve it; otherwise, look up the OID for our
    ++	 * expression.  Failure here is okay.
    ++	 */
    ++	if (!dwim_ref(branch->name, strlen(branch->name), &branch->oid, &branch->refname))
    ++		repo_get_oid_committish(the_repository, branch->name, &branch->oid);
    ++
    + 	strbuf_branchname(&buf, branch->name, INTERPRET_BRANCH_LOCAL);
    + 	if (strcmp(buf.buf, branch->name))
    + 		branch->name = xstrdup(buf.buf);
     
      ## cache.h ##
     @@ cache.h: struct checkout {
4:  676b08a650 ! 4:  3630b2279b builtin/checkout: compute checkout metadata for checkouts
    @@ builtin/checkout.c: static int reset_tree(struct tree *tree, const struct checko
      	opts.verbose_update = o->show_progress;
      	opts.src_index = &the_index;
      	opts.dst_index = &the_index;
    -+	init_checkout_metadata(&opts.meta, info->path,
    ++	init_checkout_metadata(&opts.meta, info->refname,
     +			       info->commit ? &info->commit->object.oid :
    -+			       &tree->object.oid,
    ++			       is_null_oid(&info->oid) ? &tree->object.oid :
    ++			       &info->oid,
     +			       NULL);
      	parse_tree(tree);
      	init_tree_desc(&tree_desc, tree->buffer, tree->size);
    @@ builtin/checkout.c: static int merge_working_tree(const struct checkout_opts *op
      		topts.quiet = opts->merge && old_branch_info->commit;
      		topts.verbose_update = opts->show_progress;
      		topts.fn = twoway_merge;
    -+		init_checkout_metadata(&topts.meta, new_branch_info->path,
    ++		init_checkout_metadata(&topts.meta, new_branch_info->refname,
     +				       new_branch_info->commit ?
     +				       &new_branch_info->commit->object.oid :
    -+				       NULL, NULL);
    ++				       &new_branch_info->oid, NULL);
      		if (opts->overwrite_ignore) {
      			topts.dir = xcalloc(1, sizeof(*topts.dir));
      			topts.dir->flags |= DIR_SHOW_IGNORED;
5:  853af8380c ! 5:  79168566bc builtin/clone: compute checkout metadata for clones
    @@ builtin/clone.c: static int checkout(int submodule_progress)
      	if (!strcmp(head, "HEAD")) {
      		if (advice_detached_head)
      			detach_advice(oid_to_hex(&oid));
    -+		free(head);
    -+		head = NULL;
    ++		FREE_AND_NULL(head);
      	} else {
      		if (!starts_with(head, "refs/heads/"))
      			die(_("HEAD not found below refs/heads!"));
6:  22f2d41884 = 6:  af8e2af64b builtin/rebase: compute checkout metadata for rebases
7:  b789b6db7e = 7:  c0afbd1da2 builtin/reset: compute checkout metadata for reset
8:  b3aaf17a16 = 8:  32da1229ed t0021: test filter metadata for additional cases

[PATCH v2 1/8] builtin/checkout: pass branch info down to checkout_worktree

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

In the future, we're going to want to use the branch info in
checkout_worktree, so let's pass the whole struct branch_info down, not
just the revision name.  We hoist the definition of struct branch_info
so it's in scope.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 builtin/checkout.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index d6773818b8..8a28f48d67 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -88,6 +88,17 @@ struct checkout_opts {
 	struct tree *source_tree;
 };
 
+struct branch_info {
+	const char *name; /* The short name used */
+	const char *path; /* The full name of a real branch */
+	struct commit *commit; /* The named commit */
+	/*
+	 * if not null the branch is detached because it's already
+	 * checked out in this checkout
+	 */
+	char *checkout;
+};
+
 static int post_checkout_hook(struct commit *old_commit, struct commit *new_commit,
 			      int changed)
 {
@@ -337,7 +348,8 @@ static void mark_ce_for_checkout_no_overlay(struct cache_entry *ce,
 	}
 }
 
-static int checkout_worktree(const struct checkout_opts *opts)
+static int checkout_worktree(const struct checkout_opts *opts,
+			     const struct branch_info *info)
 {
 	struct checkout state = CHECKOUT_INIT;
 	int nr_checkouts = 0, nr_unmerged = 0;
@@ -396,7 +408,7 @@ static int checkout_worktree(const struct checkout_opts *opts)
 }
 
 static int checkout_paths(const struct checkout_opts *opts,
-			  const char *revision)
+			  const struct branch_info *new_branch_info)
 {
 	int pos;
 	static char *ps_matched;
@@ -462,7 +474,7 @@ static int checkout_paths(const struct checkout_opts *opts,
 		else
 			BUG("either flag must have been set, worktree=%d, index=%d",
 			    opts->checkout_worktree, opts->checkout_index);
-		return run_add_interactive(revision, patch_mode, &opts->pathspec);
+		return run_add_interactive(new_branch_info->name, patch_mode, &opts->pathspec);
 	}
 
 	repo_hold_locked_index(the_repository, &lock_file, LOCK_DIE_ON_ERROR);
@@ -523,7 +535,7 @@ static int checkout_paths(const struct checkout_opts *opts,
 
 	/* Now we are committed to check them out */
 	if (opts->checkout_worktree)
-		errs |= checkout_worktree(opts);
+		errs |= checkout_worktree(opts, new_branch_info);
 	else
 		remove_marked_cache_entries(&the_index, 1);
 
@@ -620,17 +632,6 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,
 	}
 }
 
-struct branch_info {
-	const char *name; /* The short name used */
-	const char *path; /* The full name of a real branch */
-	struct commit *commit; /* The named commit */
-	/*
-	 * if not null the branch is detached because it's already
-	 * checked out in this checkout
-	 */
-	char *checkout;
-};
-
 static void setup_branch_path(struct branch_info *branch)
 {
 	struct strbuf buf = STRBUF_INIT;
@@ -1710,7 +1711,7 @@ static int checkout_main(int argc, const char **argv, const char *prefix,
 
 	UNLEAK(opts);
 	if (opts->patch_mode || opts->pathspec.nr)
-		return checkout_paths(opts, new_branch_info.name);
+		return checkout_paths(opts, &new_branch_info);
 	else
 		return checkout_branch(opts, &new_branch_info);
 }

[PATCH v2 2/8] convert: permit passing additional metadata to filter processes

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

There are a variety of situations where a filter process can make use of
some additional metadata.  For example, some people find the ident
filter too limiting and would like to include the commit or the branch
in their smudged files.  This information isn't available during
checkout as HEAD hasn't been updated at that point, and it wouldn't be
available in archives either.

Let's add a way to pass this metadata down to the filter.  We pass the
blob we're operating on, the treeish (preferring the commit over the
tree if one exists), and the ref we're operating on.  Note that we won't
pass this information in all cases, such as when renormalizing or when
we're performing diffs, since it doesn't make sense in those cases.

The data we currently get from the filter process looks like the
following:

  command=smudge
  pathname=git.c
  0000

With this change, we'll get data more like this:

  command=smudge
  pathname=git.c
  refname=refs/tags/v2.25.1
  treeish=c522f061d551c9bb8684a7c3859b2ece4499b56b
  blob=7be7ad34bd053884ec48923706e70c81719a8660
  0000

There are a couple things to note about this approach.  For operations
like checkout, treeish will always be a commit, since we cannot check
out individual trees, but for other operations, like archive, we can end
up operating on only a particular tree, so we'll provide only a tree as
the treeish.  Similar comments apply for refname, since there are a
variety of cases in which we won't have a ref.

This commit wires up the code to print this information, but doesn't
pass any of it at this point.  In a future commit, we'll have various
code paths pass the actual useful data down.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 apply.c            |  2 +-
 archive.c          |  2 +-
 builtin/cat-file.c |  2 +-
 convert.c          | 44 ++++++++++++++++++++++++++++++++++----------
 convert.h          | 12 ++++++++++--
 diff.c             |  2 +-
 entry.c            |  4 ++--
 merge-recursive.c  |  2 +-
 8 files changed, 51 insertions(+), 19 deletions(-)

diff --git a/apply.c b/apply.c
index bdc008fae2..144c19aaca 100644
--- a/apply.c
+++ b/apply.c
@@ -4349,7 +4349,7 @@ static int try_create_file(struct apply_state *state, const char *path,
 	if (fd < 0)
 		return 1;
 
-	if (convert_to_working_tree(state->repo->index, path, buf, size, &nbuf)) {
+	if (convert_to_working_tree(state->repo->index, path, buf, size, &nbuf, NULL)) {
 		size = nbuf.len;
 		buf  = nbuf.buf;
 	}
diff --git a/archive.c b/archive.c
index a8da0fcc4f..d9e92cce58 100644
--- a/archive.c
+++ b/archive.c
@@ -85,7 +85,7 @@ void *object_file_to_archive(const struct archiver_args *args,
 		size_t size = 0;
 
 		strbuf_attach(&buf, buffer, *sizep, *sizep + 1);
-		convert_to_working_tree(args->repo->index, path, buf.buf, buf.len, &buf);
+		convert_to_working_tree(args->repo->index, path, buf.buf, buf.len, &buf, NULL);
 		if (commit)
 			format_subst(commit, buf.buf, buf.len, &buf);
 		buffer = strbuf_detach(&buf, &size);
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 272f9fc6d7..545fed4924 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -42,7 +42,7 @@ static int filter_object(const char *path, unsigned mode,
 			     oid_to_hex(oid), path);
 	if ((type == OBJ_BLOB) && S_ISREG(mode)) {
 		struct strbuf strbuf = STRBUF_INIT;
-		if (convert_to_working_tree(&the_index, path, *buf, *size, &strbuf)) {
+		if (convert_to_working_tree(&the_index, path, *buf, *size, &strbuf, NULL)) {
 			free(*buf);
 			*size = strbuf.len;
 			*buf = strbuf_detach(&strbuf, NULL);
diff --git a/convert.c b/convert.c
index 5ead3ce678..6261921cfb 100644
--- a/convert.c
+++ b/convert.c
@@ -797,6 +797,7 @@ static void handle_filter_error(const struct strbuf *filter_status,
 static int apply_multi_file_filter(const char *path, const char *src, size_t len,
 				   int fd, struct strbuf *dst, const char *cmd,
 				   const unsigned int wanted_capability,
+				   const struct checkout_metadata *meta,
 				   struct delayed_checkout *dco)
 {
 	int err;
@@ -855,6 +856,24 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
 	if (err)
 		goto done;
 
+	if (meta && meta->refname) {
+		err = packet_write_fmt_gently(process->in, "ref=%s\n", meta->refname);
+		if (err)
+			goto done;
+	}
+
+	if (meta && !is_null_oid(&meta->treeish)) {
+		err = packet_write_fmt_gently(process->in, "treeish=%s\n", oid_to_hex(&meta->treeish));
+		if (err)
+			goto done;
+	}
+
+	if (meta && !is_null_oid(&meta->blob)) {
+		err = packet_write_fmt_gently(process->in, "blob=%s\n", oid_to_hex(&meta->blob));
+		if (err)
+			goto done;
+	}
+
 	if ((entry->supported_capabilities & CAP_DELAY) &&
 	    dco && dco->state == CE_CAN_DELAY) {
 		can_delay = 1;
@@ -971,6 +990,7 @@ static struct convert_driver {
 static int apply_filter(const char *path, const char *src, size_t len,
 			int fd, struct strbuf *dst, struct convert_driver *drv,
 			const unsigned int wanted_capability,
+			const struct checkout_metadata *meta,
 			struct delayed_checkout *dco)
 {
 	const char *cmd = NULL;
@@ -990,7 +1010,7 @@ static int apply_filter(const char *path, const char *src, size_t len,
 		return apply_single_file_filter(path, src, len, fd, dst, cmd);
 	else if (drv->process && *drv->process)
 		return apply_multi_file_filter(path, src, len, fd, dst,
-			drv->process, wanted_capability, dco);
+			drv->process, wanted_capability, meta, dco);
 
 	return 0;
 }
@@ -1368,7 +1388,7 @@ int would_convert_to_git_filter_fd(const struct index_state *istate, const char
 	if (!ca.drv->required)
 		return 0;
 
-	return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL);
+	return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL, NULL);
 }
 
 const char *get_convert_attr_ascii(const struct index_state *istate, const char *path)
@@ -1406,7 +1426,7 @@ int convert_to_git(const struct index_state *istate,
 
 	convert_attrs(istate, &ca, path);
 
-	ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL);
+	ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL, NULL);
 	if (!ret && ca.drv && ca.drv->required)
 		die(_("%s: clean filter '%s' failed"), path, ca.drv->name);
 
@@ -1441,7 +1461,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,
 	assert(ca.drv);
 	assert(ca.drv->clean || ca.drv->process);
 
-	if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))
+	if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL, NULL))
 		die(_("%s: clean filter '%s' failed"), path, ca.drv->name);
 
 	encode_to_git(path, dst->buf, dst->len, dst, ca.working_tree_encoding, conv_flags);
@@ -1452,7 +1472,9 @@ void convert_to_git_filter_fd(const struct index_state *istate,
 static int convert_to_working_tree_internal(const struct index_state *istate,
 					    const char *path, const char *src,
 					    size_t len, struct strbuf *dst,
-					    int normalizing, struct delayed_checkout *dco)
+					    int normalizing,
+					    const struct checkout_metadata *meta,
+					    struct delayed_checkout *dco)
 {
 	int ret = 0, ret_filter = 0;
 	struct conv_attrs ca;
@@ -1484,7 +1506,7 @@ static int convert_to_working_tree_internal(const struct index_state *istate,
 	}
 
 	ret_filter = apply_filter(
-		path, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);
+		path, src, len, -1, dst, ca.drv, CAP_SMUDGE, meta, dco);
 	if (!ret_filter && ca.drv && ca.drv->required)
 		die(_("%s: smudge filter %s failed"), path, ca.drv->name);
 
@@ -1494,22 +1516,24 @@ static int convert_to_working_tree_internal(const struct index_state *istate,
 int async_convert_to_working_tree(const struct index_state *istate,
 				  const char *path, const char *src,
 				  size_t len, struct strbuf *dst,
+				  const struct checkout_metadata *meta,
 				  void *dco)
 {
-	return convert_to_working_tree_internal(istate, path, src, len, dst, 0, dco);
+	return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);
 }
 
 int convert_to_working_tree(const struct index_state *istate,
 			    const char *path, const char *src,
-			    size_t len, struct strbuf *dst)
+			    size_t len, struct strbuf *dst,
+			    const struct checkout_metadata *meta)
 {
-	return convert_to_working_tree_internal(istate, path, src, len, dst, 0, NULL);
+	return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);
 }
 
 int renormalize_buffer(const struct index_state *istate, const char *path,
 		       const char *src, size_t len, struct strbuf *dst)
 {
-	int ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL);
+	int ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);
 	if (ret) {
 		src = dst->buf;
 		len = dst->len;
diff --git a/convert.h b/convert.h
index 3710969d43..894e01c38b 100644
--- a/convert.h
+++ b/convert.h
@@ -4,10 +4,10 @@
 #ifndef CONVERT_H
 #define CONVERT_H
 
+#include "hash.h"
 #include "string-list.h"
 
 struct index_state;
-struct object_id;
 struct strbuf;
 
 #define CONV_EOL_RNDTRP_DIE   (1<<0) /* Die if CRLF to LF to CRLF is different */
@@ -57,6 +57,12 @@ struct delayed_checkout {
 	struct string_list paths;
 };
 
+struct checkout_metadata {
+	const char *refname;
+	struct object_id treeish;
+	struct object_id blob;
+};
+
 extern enum eol core_eol;
 extern char *check_roundtrip_encoding;
 const char *get_cached_convert_stats_ascii(const struct index_state *istate,
@@ -71,10 +77,12 @@ int convert_to_git(const struct index_state *istate,
 		   struct strbuf *dst, int conv_flags);
 int convert_to_working_tree(const struct index_state *istate,
 			    const char *path, const char *src,
-			    size_t len, struct strbuf *dst);
+			    size_t len, struct strbuf *dst,
+			    const struct checkout_metadata *meta);
 int async_convert_to_working_tree(const struct index_state *istate,
 				  const char *path, const char *src,
 				  size_t len, struct strbuf *dst,
+				  const struct checkout_metadata *meta,
 				  void *dco);
 int async_query_available_blobs(const char *cmd,
 				struct string_list *available_paths);
diff --git a/diff.c b/diff.c
index f2cfbf2214..12761c8017 100644
--- a/diff.c
+++ b/diff.c
@@ -4071,7 +4071,7 @@ static void prep_temp_blob(struct index_state *istate,
 	if (!temp->tempfile)
 		die_errno("unable to create temp-file");
 	if (convert_to_working_tree(istate, path,
-			(const char *)blob, (size_t)size, &buf)) {
+			(const char *)blob, (size_t)size, &buf, NULL)) {
 		blob = buf.buf;
 		size = buf.len;
 	}
diff --git a/entry.c b/entry.c
index 53380bb614..4b2d9b2dad 100644
--- a/entry.c
+++ b/entry.c
@@ -315,13 +315,13 @@ static int write_entry(struct cache_entry *ce,
 		 */
 		if (dco && dco->state != CE_NO_DELAY) {
 			ret = async_convert_to_working_tree(state->istate, ce->name, new_blob,
-							    size, &buf, dco);
+							    size, &buf, NULL, dco);
 			if (ret && string_list_has_string(&dco->paths, ce->name)) {
 				free(new_blob);
 				goto delayed;
 			}
 		} else
-			ret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf);
+			ret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, NULL);
 
 		if (ret) {
 			free(new_blob);
diff --git a/merge-recursive.c b/merge-recursive.c
index 7a4e6f20fa..d92e2acf1e 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -958,7 +958,7 @@ static int update_file_flags(struct merge_options *opt,
 		if (S_ISREG(contents->mode)) {
 			struct strbuf strbuf = STRBUF_INIT;
 			if (convert_to_working_tree(opt->repo->index,
-						    path, buf, size, &strbuf)) {
+						    path, buf, size, &strbuf, NULL)) {
 				free(buf);
 				size = strbuf.len;
 				buf = strbuf_detach(&strbuf, NULL);

[PATCH v2 3/8] convert: provide additional metadata to filters

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

Now that we have the codebase wired up to pass any additional metadata
to filters, let's collect the additional metadata that we'd like to
pass.

The two main places we pass this metadata are checkouts and archives.
In these two situations, reading HEAD isn't a valid option, since HEAD
isn't updated for checkouts until after the working tree is written and
archives can accept an arbitrary tree.  In other situations, HEAD will
usually reflect the refname of the branch in current use.

We pass a smaller amount of data in other cases, such as git cat-file,
where we can really only logically know about the blob.

This commit updates only the parts of the checkout code where we don't
use unpack_trees.  That function and callers of it will be handled in a
future commit.

In the archive code, we leak a small amount of memory, since nothing we
pass in the archiver argument structure is freed.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 archive.c               | 13 ++++++++++---
 archive.h               |  1 +
 builtin/cat-file.c      |  5 ++++-
 builtin/checkout.c      | 13 +++++++++++++
 cache.h                 |  1 +
 convert.c               | 22 ++++++++++++++++++++++
 convert.h               | 17 +++++++++++++++++
 diff.c                  |  5 ++++-
 entry.c                 |  7 +++++--
 t/t0021/rot13-filter.pl |  6 ++++++
 10 files changed, 83 insertions(+), 7 deletions(-)

diff --git a/archive.c b/archive.c
index d9e92cce58..fb39706120 100644
--- a/archive.c
+++ b/archive.c
@@ -77,6 +77,11 @@ void *object_file_to_archive(const struct archiver_args *args,
 {
 	void *buffer;
 	const struct commit *commit = args->convert ? args->commit : NULL;
+	struct checkout_metadata meta;
+
+	init_checkout_metadata(&meta, args->refname,
+			       args->commit_oid ? args->commit_oid :
+			       (args->tree ? &args->tree->object.oid : NULL), oid);
 
 	path += args->baselen;
 	buffer = read_object_file(oid, type, sizep);
@@ -85,7 +90,7 @@ void *object_file_to_archive(const struct archiver_args *args,
 		size_t size = 0;
 
 		strbuf_attach(&buf, buffer, *sizep, *sizep + 1);
-		convert_to_working_tree(args->repo->index, path, buf.buf, buf.len, &buf, NULL);
+		convert_to_working_tree(args->repo->index, path, buf.buf, buf.len, &buf, &meta);
 		if (commit)
 			format_subst(commit, buf.buf, buf.len, &buf);
 		buffer = strbuf_detach(&buf, &size);
@@ -385,16 +390,17 @@ static void parse_treeish_arg(const char **argv,
 	struct tree *tree;
 	const struct commit *commit;
 	struct object_id oid;
+	char *ref = NULL;
 
 	/* Remotes are only allowed to fetch actual refs */
 	if (remote && !remote_allow_unreachable) {
-		char *ref = NULL;
 		const char *colon = strchrnul(name, ':');
 		int refnamelen = colon - name;
 
 		if (!dwim_ref(name, refnamelen, &oid, &ref))
 			die(_("no such ref: %.*s"), refnamelen, name);
-		free(ref);
+	} else {
+		dwim_ref(name, strlen(name), &oid, &ref);
 	}
 
 	if (get_oid(name, &oid))
@@ -427,6 +433,7 @@ static void parse_treeish_arg(const char **argv,
 
 		tree = parse_tree_indirect(&tree_oid);
 	}
+	ar_args->refname = ref;
 	ar_args->tree = tree;
 	ar_args->commit_oid = commit_oid;
 	ar_args->commit = commit;
diff --git a/archive.h b/archive.h
index e60e3dd31c..3bd96bf6bb 100644
--- a/archive.h
+++ b/archive.h
@@ -8,6 +8,7 @@ struct repository;
 
 struct archiver_args {
 	struct repository *repo;
+	const char *refname;
 	const char *base;
 	size_t baselen;
 	struct tree *tree;
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 545fed4924..6ecc8ee6dc 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -42,7 +42,10 @@ static int filter_object(const char *path, unsigned mode,
 			     oid_to_hex(oid), path);
 	if ((type == OBJ_BLOB) && S_ISREG(mode)) {
 		struct strbuf strbuf = STRBUF_INIT;
-		if (convert_to_working_tree(&the_index, path, *buf, *size, &strbuf, NULL)) {
+		struct checkout_metadata meta;
+
+		init_checkout_metadata(&meta, NULL, NULL, oid);
+		if (convert_to_working_tree(&the_index, path, *buf, *size, &strbuf, &meta)) {
 			free(*buf);
 			*size = strbuf.len;
 			*buf = strbuf_detach(&strbuf, NULL);
diff --git a/builtin/checkout.c b/builtin/checkout.c
index 8a28f48d67..1bdb70d3dd 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -92,6 +92,8 @@ struct branch_info {
 	const char *name; /* The short name used */
 	const char *path; /* The full name of a real branch */
 	struct commit *commit; /* The named commit */
+	char *refname; /* The full name of the ref being checked out. */
+	struct object_id oid; /* The object ID of the commit being checked out. */
 	/*
 	 * if not null the branch is detached because it's already
 	 * checked out in this checkout
@@ -360,6 +362,10 @@ static int checkout_worktree(const struct checkout_opts *opts,
 	state.refresh_cache = 1;
 	state.istate = &the_index;
 
+	init_checkout_metadata(&state.meta, info->refname,
+			       info->commit ? &info->commit->object.oid : &info->oid,
+			       NULL);
+
 	enable_delayed_checkout(&state);
 	for (pos = 0; pos < active_nr; pos++) {
 		struct cache_entry *ce = active_cache[pos];
@@ -636,6 +642,13 @@ static void setup_branch_path(struct branch_info *branch)
 {
 	struct strbuf buf = STRBUF_INIT;
 
+	/*
+	 * If this is a ref, resolve it; otherwise, look up the OID for our
+	 * expression.  Failure here is okay.
+	 */
+	if (!dwim_ref(branch->name, strlen(branch->name), &branch->oid, &branch->refname))
+		repo_get_oid_committish(the_repository, branch->name, &branch->oid);
+
 	strbuf_branchname(&buf, branch->name, INTERPRET_BRANCH_LOCAL);
 	if (strcmp(buf.buf, branch->name))
 		branch->name = xstrdup(buf.buf);
diff --git a/cache.h b/cache.h
index 37c899b53f..9b24e5d61f 100644
--- a/cache.h
+++ b/cache.h
@@ -1679,6 +1679,7 @@ struct checkout {
 	const char *base_dir;
 	int base_dir_len;
 	struct delayed_checkout *delayed_checkout;
+	struct checkout_metadata meta;
 	unsigned force:1,
 		 quiet:1,
 		 not_new:1,
diff --git a/convert.c b/convert.c
index 6261921cfb..5aa87d45e3 100644
--- a/convert.c
+++ b/convert.c
@@ -2006,3 +2006,25 @@ int stream_filter(struct stream_filter *filter,
 {
 	return filter->vtbl->filter(filter, input, isize_p, output, osize_p);
 }
+
+void init_checkout_metadata(struct checkout_metadata *meta, const char *refname,
+			    const struct object_id *treeish,
+			    const struct object_id *blob)
+{
+	memset(meta, 0, sizeof(*meta));
+	if (refname)
+		meta->refname = refname;
+	if (treeish)
+		oidcpy(&meta->treeish, treeish);
+	if (blob)
+		oidcpy(&meta->blob, blob);
+}
+
+void clone_checkout_metadata(struct checkout_metadata *dst,
+			     const struct checkout_metadata *src,
+			     const struct object_id *blob)
+{
+	memcpy(dst, src, sizeof(*dst));
+	if (blob)
+		oidcpy(&dst->blob, blob);
+}
diff --git a/convert.h b/convert.h
index 894e01c38b..e29d1026a6 100644
--- a/convert.h
+++ b/convert.h
@@ -102,6 +102,23 @@ void convert_to_git_filter_fd(const struct index_state *istate,
 int would_convert_to_git_filter_fd(const struct index_state *istate,
 				   const char *path);
 
+/*
+ * Initialize the checkout metadata with the given values.  Any argument may be
+ * NULL if it is not applicable.  The treeish should be a commit if that is
+ * available, and a tree otherwise.
+ *
+ * The refname is not copied and must be valid for the lifetime of the struct.
+ * THe object IDs are copied.
+ */
+void init_checkout_metadata(struct checkout_metadata *meta, const char *refname,
+			    const struct object_id *treeish,
+			    const struct object_id *blob);
+
+/* Copy the metadata from src to dst, updating the blob. */
+void clone_checkout_metadata(struct checkout_metadata *dst,
+			     const struct checkout_metadata *src,
+			     const struct object_id *blob);
+
 /*
  * Reset the internal list of attributes used by convert_to_git and
  * convert_to_working_tree.
diff --git a/diff.c b/diff.c
index 12761c8017..1010d806f5 100644
--- a/diff.c
+++ b/diff.c
@@ -4062,6 +4062,9 @@ static void prep_temp_blob(struct index_state *istate,
 	struct strbuf tempfile = STRBUF_INIT;
 	char *path_dup = xstrdup(path);
 	const char *base = basename(path_dup);
+	struct checkout_metadata meta;
+
+	init_checkout_metadata(&meta, NULL, NULL, oid);
 
 	/* Generate "XXXXXX_basename.ext" */
 	strbuf_addstr(&tempfile, "XXXXXX_");
@@ -4071,7 +4074,7 @@ static void prep_temp_blob(struct index_state *istate,
 	if (!temp->tempfile)
 		die_errno("unable to create temp-file");
 	if (convert_to_working_tree(istate, path,
-			(const char *)blob, (size_t)size, &buf, NULL)) {
+			(const char *)blob, (size_t)size, &buf, &meta)) {
 		blob = buf.buf;
 		size = buf.len;
 	}
diff --git a/entry.c b/entry.c
index 4b2d9b2dad..00b4903366 100644
--- a/entry.c
+++ b/entry.c
@@ -264,6 +264,9 @@ static int write_entry(struct cache_entry *ce,
 	size_t newsize = 0;
 	struct stat st;
 	const struct submodule *sub;
+	struct checkout_metadata meta;
+
+	clone_checkout_metadata(&meta, &state->meta, &ce->oid);
 
 	if (ce_mode_s_ifmt == S_IFREG) {
 		struct stream_filter *filter = get_stream_filter(state->istate, ce->name,
@@ -315,13 +318,13 @@ static int write_entry(struct cache_entry *ce,
 		 */
 		if (dco && dco->state != CE_NO_DELAY) {
 			ret = async_convert_to_working_tree(state->istate, ce->name, new_blob,
-							    size, &buf, NULL, dco);
+							    size, &buf, &meta, dco);
 			if (ret && string_list_has_string(&dco->paths, ce->name)) {
 				free(new_blob);
 				goto delayed;
 			}
 		} else
-			ret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, NULL);
+			ret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, &meta);
 
 		if (ret) {
 			free(new_blob);
diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl
index 470107248e..c43cf433cf 100644
--- a/t/t0021/rot13-filter.pl
+++ b/t/t0021/rot13-filter.pl
@@ -135,7 +135,13 @@ sub rot13 {
 				if ( exists $DELAY{$pathname} and $DELAY{$pathname}{"requested"} == 0 ) {
 					$DELAY{$pathname}{"requested"} = 1;
 				}
+			} elsif ($buffer =~ /^(ref|treeish|blob)=/) {
+				# Do nothing.
 			} else {
+				# In general, filters need to be graceful about
+				# new metadata, since it's documented that we
+				# can pass any key-value pairs, but for tests,
+				# let's be a little stricter.
 				die "Unknown message '$buffer'";
 			}
 

[PATCH v2 4/8] builtin/checkout: compute checkout metadata for checkouts

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

Provide commit metadata for checkout code paths that use unpack_trees
and friends.  When we're checking out a commit, use the commit
information, but don't provide commit information if we're checking out
from the index, since there need not be any particular commit associated
with the index, and even if there is one, we can't know what it is.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 builtin/checkout.c      | 18 +++++++---
 merge.c                 |  1 +
 t/t0021-conversion.sh   | 78 ++++++++++++++++++++++++++---------------
 t/t0021/rot13-filter.pl |  2 +-
 unpack-trees.c          |  1 +
 unpack-trees.h          |  1 +
 6 files changed, 68 insertions(+), 33 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 1bdb70d3dd..8bc94d392b 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -604,7 +604,8 @@ static void describe_detached_head(const char *msg, struct commit *commit)
 }
 
 static int reset_tree(struct tree *tree, const struct checkout_opts *o,
-		      int worktree, int *writeout_error)
+		      int worktree, int *writeout_error,
+		      struct branch_info *info)
 {
 	struct unpack_trees_options opts;
 	struct tree_desc tree_desc;
@@ -619,6 +620,11 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,
 	opts.verbose_update = o->show_progress;
 	opts.src_index = &the_index;
 	opts.dst_index = &the_index;
+	init_checkout_metadata(&opts.meta, info->refname,
+			       info->commit ? &info->commit->object.oid :
+			       is_null_oid(&info->oid) ? &tree->object.oid :
+			       &info->oid,
+			       NULL);
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	switch (unpack_trees(1, &tree_desc, &opts)) {
@@ -677,7 +683,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
 	} else
 		new_tree = get_commit_tree(new_branch_info->commit);
 	if (opts->discard_changes) {
-		ret = reset_tree(new_tree, opts, 1, writeout_error);
+		ret = reset_tree(new_tree, opts, 1, writeout_error, new_branch_info);
 		if (ret)
 			return ret;
 	} else {
@@ -706,6 +712,10 @@ static int merge_working_tree(const struct checkout_opts *opts,
 		topts.quiet = opts->merge && old_branch_info->commit;
 		topts.verbose_update = opts->show_progress;
 		topts.fn = twoway_merge;
+		init_checkout_metadata(&topts.meta, new_branch_info->refname,
+				       new_branch_info->commit ?
+				       &new_branch_info->commit->object.oid :
+				       &new_branch_info->oid, NULL);
 		if (opts->overwrite_ignore) {
 			topts.dir = xcalloc(1, sizeof(*topts.dir));
 			topts.dir->flags |= DIR_SHOW_IGNORED;
@@ -776,7 +786,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
 
 			ret = reset_tree(new_tree,
 					 opts, 1,
-					 writeout_error);
+					 writeout_error, new_branch_info);
 			if (ret)
 				return ret;
 			o.ancestor = old_branch_info->name;
@@ -796,7 +806,7 @@ static int merge_working_tree(const struct checkout_opts *opts,
 				exit(128);
 			ret = reset_tree(new_tree,
 					 opts, 0,
-					 writeout_error);
+					 writeout_error, new_branch_info);
 			strbuf_release(&o.obuf);
 			strbuf_release(&old_commit_shortname);
 			if (ret)
diff --git a/merge.c b/merge.c
index 7c1d756c3f..aa36de2f64 100644
--- a/merge.c
+++ b/merge.c
@@ -94,6 +94,7 @@ int checkout_fast_forward(struct repository *r,
 	opts.verbose_update = 1;
 	opts.merge = 1;
 	opts.fn = twoway_merge;
+	init_checkout_metadata(&opts.meta, NULL, remote, NULL);
 	setup_unpack_trees_porcelain(&opts, "merge");
 
 	if (unpack_trees(nr_trees, t, &opts)) {
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index dc664da551..4b8d6a74a7 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -364,6 +364,10 @@ test_expect_success PERL 'required process filter should filter data' '
 		S=$(file_size test.r) &&
 		S2=$(file_size test2.r) &&
 		S3=$(file_size "testsubdir/test3 '\''sq'\'',\$x=.r") &&
+		M=$(git hash-object test.r) &&
+		M2=$(git hash-object test2.r) &&
+		M3=$(git hash-object "testsubdir/test3 '\''sq'\'',\$x=.r") &&
+		EMPTY=$(git hash-object /dev/null) &&
 
 		filter_git add . &&
 		cat >expected.log <<-EOF &&
@@ -378,14 +382,15 @@ test_expect_success PERL 'required process filter should filter data' '
 		test_cmp_count expected.log debug.log &&
 
 		git commit -m "test commit 2" &&
+		META="ref=refs/heads/master treeish=$(git rev-parse --verify master)" &&
 		rm -f test2.r "testsubdir/test3 '\''sq'\'',\$x=.r" &&
 
 		filter_git checkout --quiet --no-progress . &&
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
-			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK]
+			IN: smudge test2.r blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -406,10 +411,10 @@ test_expect_success PERL 'required process filter should filter data' '
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge test.r $S [OK] -- OUT: $S . [OK]
-			IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
-			IN: smudge test4-empty.r 0 [OK] -- OUT: 0  [OK]
-			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK]
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -519,17 +524,22 @@ test_expect_success PERL 'required process filter should process multiple packet
 		EOF
 		test_cmp_count expected.log debug.log &&
 
-		rm -f *.file &&
+		M1="blob=$(git hash-object 1pkt_1__.file)" &&
+		M2="blob=$(git hash-object 2pkt_1+1.file)" &&
+		M3="blob=$(git hash-object 2pkt_2-1.file)" &&
+		M4="blob=$(git hash-object 2pkt_2__.file)" &&
+		M5="blob=$(git hash-object 3pkt_2+1.file)" &&
+		rm -f *.file debug.log &&
 
 		filter_git checkout --quiet --no-progress -- *.file &&
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge 1pkt_1__.file $(($S    )) [OK] -- OUT: $(($S    )) . [OK]
-			IN: smudge 2pkt_1+1.file $(($S  +1)) [OK] -- OUT: $(($S  +1)) .. [OK]
-			IN: smudge 2pkt_2-1.file $(($S*2-1)) [OK] -- OUT: $(($S*2-1)) .. [OK]
-			IN: smudge 2pkt_2__.file $(($S*2  )) [OK] -- OUT: $(($S*2  )) .. [OK]
-			IN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
+			IN: smudge 1pkt_1__.file $M1 $(($S    )) [OK] -- OUT: $(($S    )) . [OK]
+			IN: smudge 2pkt_1+1.file $M2 $(($S  +1)) [OK] -- OUT: $(($S  +1)) .. [OK]
+			IN: smudge 2pkt_2-1.file $M3 $(($S*2-1)) [OK] -- OUT: $(($S*2-1)) .. [OK]
+			IN: smudge 2pkt_2__.file $M4 $(($S*2  )) [OK] -- OUT: $(($S*2  )) .. [OK]
+			IN: smudge 3pkt_2+1.file $M5 $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -578,6 +588,10 @@ test_expect_success PERL 'process filter should restart after unexpected write f
 		S=$(file_size test.r) &&
 		S2=$(file_size test2.r) &&
 		SF=$(file_size smudge-write-fail.r) &&
+		M=$(git hash-object test.r) &&
+		M2=$(git hash-object test2.r) &&
+		MF=$(git hash-object smudge-write-fail.r) &&
+		rm -f debug.log &&
 
 		git add . &&
 		rm -f *.r &&
@@ -591,11 +605,11 @@ test_expect_success PERL 'process filter should restart after unexpected write f
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge smudge-write-fail.r $SF [OK] -- [WRITE FAIL]
+			IN: smudge smudge-write-fail.r blob=$MF $SF [OK] -- [WRITE FAIL]
 			START
 			init handshake complete
-			IN: smudge test.r $S [OK] -- OUT: $S . [OK]
-			IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test.r blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -629,6 +643,10 @@ test_expect_success PERL 'process filter should not be restarted if it signals a
 		S=$(file_size test.r) &&
 		S2=$(file_size test2.r) &&
 		SE=$(file_size error.r) &&
+		M=$(git hash-object test.r) &&
+		M2=$(git hash-object test2.r) &&
+		ME=$(git hash-object error.r) &&
+		rm -f debug.log &&
 
 		git add . &&
 		rm -f *.r &&
@@ -637,9 +655,9 @@ test_expect_success PERL 'process filter should not be restarted if it signals a
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge error.r $SE [OK] -- [ERROR]
-			IN: smudge test.r $S [OK] -- OUT: $S . [OK]
-			IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge error.r blob=$ME $SE [OK] -- [ERROR]
+			IN: smudge test.r blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -665,18 +683,21 @@ test_expect_success PERL 'process filter abort stops processing of all further f
 		echo "error this blob and all future blobs" >abort.o &&
 		cp abort.o abort.r &&
 
+		M="blob=$(git hash-object abort.r)" &&
+		rm -f debug.log &&
 		SA=$(file_size abort.r) &&
 
 		git add . &&
 		rm -f *.r &&
 
+
 		# Note: This test assumes that Git filters files in alphabetical
 		# order ("abort.r" before "test.r").
 		filter_git checkout --quiet --no-progress . &&
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
-			IN: smudge abort.r $SA [OK] -- [ABORT]
+			IN: smudge abort.r $M $SA [OK] -- [ABORT]
 			STOP
 		EOF
 		test_cmp_exclude_clean expected.log debug.log &&
@@ -727,27 +748,28 @@ test_expect_success PERL 'delayed checkout in process filter' '
 	) &&
 
 	S=$(file_size "$TEST_ROOT/test.o") &&
+	M="blob=$(git -C repo rev-parse --verify master:test.a)" &&
 	cat >a.exp <<-EOF &&
 		START
 		init handshake complete
-		IN: smudge test.a $S [OK] -- OUT: $S . [OK]
-		IN: smudge test-delay10.a $S [OK] -- [DELAYED]
-		IN: smudge test-delay11.a $S [OK] -- [DELAYED]
-		IN: smudge test-delay20.a $S [OK] -- [DELAYED]
+		IN: smudge test.a $M $S [OK] -- OUT: $S . [OK]
+		IN: smudge test-delay10.a $M $S [OK] -- [DELAYED]
+		IN: smudge test-delay11.a $M $S [OK] -- [DELAYED]
+		IN: smudge test-delay20.a $M $S [OK] -- [DELAYED]
 		IN: list_available_blobs test-delay10.a test-delay11.a [OK]
-		IN: smudge test-delay10.a 0 [OK] -- OUT: $S . [OK]
-		IN: smudge test-delay11.a 0 [OK] -- OUT: $S . [OK]
+		IN: smudge test-delay10.a $M 0 [OK] -- OUT: $S . [OK]
+		IN: smudge test-delay11.a $M 0 [OK] -- OUT: $S . [OK]
 		IN: list_available_blobs test-delay20.a [OK]
-		IN: smudge test-delay20.a 0 [OK] -- OUT: $S . [OK]
+		IN: smudge test-delay20.a $M 0 [OK] -- OUT: $S . [OK]
 		IN: list_available_blobs [OK]
 		STOP
 	EOF
 	cat >b.exp <<-EOF &&
 		START
 		init handshake complete
-		IN: smudge test-delay10.b $S [OK] -- [DELAYED]
+		IN: smudge test-delay10.b $M $S [OK] -- [DELAYED]
 		IN: list_available_blobs test-delay10.b [OK]
-		IN: smudge test-delay10.b 0 [OK] -- OUT: $S . [OK]
+		IN: smudge test-delay10.b $M 0 [OK] -- OUT: $S . [OK]
 		IN: list_available_blobs [OK]
 		STOP
 	EOF
diff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl
index c43cf433cf..cd32a82da5 100644
--- a/t/t0021/rot13-filter.pl
+++ b/t/t0021/rot13-filter.pl
@@ -136,7 +136,7 @@ sub rot13 {
 					$DELAY{$pathname}{"requested"} = 1;
 				}
 			} elsif ($buffer =~ /^(ref|treeish|blob)=/) {
-				# Do nothing.
+				print $debug " $buffer";
 			} else {
 				# In general, filters need to be graceful about
 				# new metadata, since it's documented that we
diff --git a/unpack-trees.c b/unpack-trees.c
index 1ecdab3304..3aba5da6b6 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -371,6 +371,7 @@ static int check_updates(struct unpack_trees_options *o)
 	state.quiet = 1;
 	state.refresh_cache = 1;
 	state.istate = index;
+	clone_checkout_metadata(&state.meta, &o->meta, NULL);
 
 	if (!o->update || o->dry_run) {
 		remove_marked_cache_entries(index, 0);
diff --git a/unpack-trees.h b/unpack-trees.h
index ae1557fb80..ad41b45a71 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -85,6 +85,7 @@ struct unpack_trees_options {
 	struct index_state result;
 
 	struct pattern_list *pl; /* for internal use */
+	struct checkout_metadata meta;
 };
 
 int unpack_trees(unsigned n, struct tree_desc *t,

[PATCH v2 5/8] builtin/clone: compute checkout metadata for clones

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

When checking out a commit, provide metadata to the filter process
including the ref we're using.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 builtin/clone.c       |  5 ++++-
 t/t0021-conversion.sh | 10 +++++++---
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 1ad26f4d8c..d28b4d4bc5 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -780,11 +780,11 @@ static int checkout(int submodule_progress)
 	if (!strcmp(head, "HEAD")) {
 		if (advice_detached_head)
 			detach_advice(oid_to_hex(&oid));
+		FREE_AND_NULL(head);
 	} else {
 		if (!starts_with(head, "refs/heads/"))
 			die(_("HEAD not found below refs/heads!"));
 	}
-	free(head);
 
 	/* We need to be in the new work tree for the checkout */
 	setup_work_tree();
@@ -799,6 +799,7 @@ static int checkout(int submodule_progress)
 	opts.verbose_update = (option_verbosity >= 0);
 	opts.src_index = &the_index;
 	opts.dst_index = &the_index;
+	init_checkout_metadata(&opts.meta, head, &oid, NULL);
 
 	tree = parse_tree_indirect(&oid);
 	parse_tree(tree);
@@ -806,6 +807,8 @@ static int checkout(int submodule_progress)
 	if (unpack_trees(1, &t, &opts) < 0)
 		die(_("unable to checkout working tree"));
 
+	free(head);
+
 	if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))
 		die(_("unable to write new index file"));
 
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index 4b8d6a74a7..ad329044a3 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -748,7 +748,8 @@ test_expect_success PERL 'delayed checkout in process filter' '
 	) &&
 
 	S=$(file_size "$TEST_ROOT/test.o") &&
-	M="blob=$(git -C repo rev-parse --verify master:test.a)" &&
+	PM="ref=refs/heads/master treeish=$(git -C repo rev-parse --verify master) " &&
+	M="${PM}blob=$(git -C repo rev-parse --verify master:test.a)" &&
 	cat >a.exp <<-EOF &&
 		START
 		init handshake complete
@@ -789,8 +790,11 @@ test_expect_success PERL 'delayed checkout in process filter' '
 
 		rm *.a *.b &&
 		filter_git checkout . &&
-		test_cmp_count ../a.exp a.log &&
-		test_cmp_count ../b.exp b.log &&
+		# We are not checking out a ref here, so filter out ref metadata.
+		sed -e "s!$PM!!" ../a.exp >a.exp.filtered &&
+		sed -e "s!$PM!!" ../b.exp >b.exp.filtered &&
+		test_cmp_count a.exp.filtered a.log &&
+		test_cmp_count b.exp.filtered b.log &&
 
 		test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.a &&
 		test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.a &&

[PATCH v2 6/8] builtin/rebase: compute checkout metadata for rebases

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 builtin/rebase.c      |  1 +
 sequencer.c           |  1 +
 t/t0021-conversion.sh | 39 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/builtin/rebase.c b/builtin/rebase.c
index bff53d5d16..27a07d4e78 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -868,6 +868,7 @@ static int reset_head(struct object_id *oid, const char *action,
 	unpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;
 	unpack_tree_opts.update = 1;
 	unpack_tree_opts.merge = 1;
+	init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);
 	if (!detach_head)
 		unpack_tree_opts.reset = 1;
 
diff --git a/sequencer.c b/sequencer.c
index e528225e78..0dd34496fd 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -3290,6 +3290,7 @@ static int do_reset(struct repository *r,
 	unpack_tree_opts.fn = oneway_merge;
 	unpack_tree_opts.merge = 1;
 	unpack_tree_opts.update = 1;
+	init_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);
 
 	if (repo_read_index_unmerged(r)) {
 		rollback_lock_file(&lock);
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index ad329044a3..3efb0dad20 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -382,7 +382,8 @@ test_expect_success PERL 'required process filter should filter data' '
 		test_cmp_count expected.log debug.log &&
 
 		git commit -m "test commit 2" &&
-		META="ref=refs/heads/master treeish=$(git rev-parse --verify master)" &&
+		MASTER=$(git rev-parse --verify master) &&
+		META="ref=refs/heads/master treeish=$MASTER" &&
 		rm -f test2.r "testsubdir/test3 '\''sq'\'',\$x=.r" &&
 
 		filter_git checkout --quiet --no-progress . &&
@@ -425,6 +426,42 @@ test_expect_success PERL 'required process filter should filter data' '
 	)
 '
 
+test_expect_success PERL 'required process filter should filter data for various subcommands' '
+	test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
+	test_config_global filter.protocol.required true &&
+	(
+		cd repo &&
+
+		S=$(file_size test.r) &&
+		S2=$(file_size test2.r) &&
+		S3=$(file_size "testsubdir/test3 '\''sq'\'',\$x=.r") &&
+		M=$(git hash-object test.r) &&
+		M2=$(git hash-object test2.r) &&
+		M3=$(git hash-object "testsubdir/test3 '\''sq'\'',\$x=.r") &&
+		EMPTY=$(git hash-object /dev/null) &&
+
+		MASTER=$(git rev-parse --verify master) &&
+
+		cp "$TEST_ROOT/test.o" test5.r &&
+		git add test5.r &&
+		git commit -m "test commit 3" &&
+		git checkout empty-branch &&
+		filter_git rebase --onto empty-branch master^^ master &&
+		META="ref=refs/heads/master treeish=$(git rev-parse --verify master)" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge test5.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
+		test_cmp_exclude_clean expected.log debug.log
+	)
+'
+
 test_expect_success PERL 'required process filter takes precedence' '
 	test_config_global filter.protocol.clean false &&
 	test_config_global filter.protocol.process "rot13-filter.pl debug.log clean" &&

[PATCH v2 7/8] builtin/reset: compute checkout metadata for reset

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

Pass the commit, and if we have it, the ref to the filters when we
perform a checkout.  This should only be the case when we invoke git
reset --hard; the metadata will be unused otherwise.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 builtin/reset.c       | 16 +++++++++++++---
 t/t0021-conversion.sh | 32 +++++++++++++++++++++++++++++++-
 2 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/builtin/reset.c b/builtin/reset.c
index 18228c312e..4c634111bd 100644
--- a/builtin/reset.c
+++ b/builtin/reset.c
@@ -46,7 +46,7 @@ static inline int is_merge(void)
 	return !access(git_path_merge_head(the_repository), F_OK);
 }
 
-static int reset_index(const struct object_id *oid, int reset_type, int quiet)
+static int reset_index(const char *ref, const struct object_id *oid, int reset_type, int quiet)
 {
 	int i, nr = 0;
 	struct tree_desc desc[2];
@@ -60,6 +60,7 @@ static int reset_index(const struct object_id *oid, int reset_type, int quiet)
 	opts.dst_index = &the_index;
 	opts.fn = oneway_merge;
 	opts.merge = 1;
+	init_checkout_metadata(&opts.meta, ref, oid, NULL);
 	if (!quiet)
 		opts.verbose_update = 1;
 	switch (reset_type) {
@@ -418,11 +419,20 @@ int cmd_reset(int argc, const char **argv, const char *prefix)
 				}
 			}
 		} else {
-			int err = reset_index(&oid, reset_type, quiet);
+			struct object_id dummy;
+			char *ref = NULL;
+			int err;
+
+			dwim_ref(rev, strlen(rev), &dummy, &ref);
+			if (ref && !starts_with(ref, "refs/"))
+				ref = NULL;
+
+			err = reset_index(ref, &oid, reset_type, quiet);
 			if (reset_type == KEEP && !err)
-				err = reset_index(&oid, MIXED, quiet);
+				err = reset_index(ref, &oid, MIXED, quiet);
 			if (err)
 				die(_("Could not reset index file to revision '%s'."), rev);
+			free(ref);
 		}
 
 		if (write_locked_index(&the_index, &lock, COMMIT_LOCK))
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index 3efb0dad20..a026fd46b4 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -447,7 +447,8 @@ test_expect_success PERL 'required process filter should filter data for various
 		git commit -m "test commit 3" &&
 		git checkout empty-branch &&
 		filter_git rebase --onto empty-branch master^^ master &&
-		META="ref=refs/heads/master treeish=$(git rev-parse --verify master)" &&
+		MASTER2=$(git rev-parse --verify master) &&
+		META="ref=refs/heads/master treeish=$MASTER2" &&
 		cat >expected.log <<-EOF &&
 			START
 			init handshake complete
@@ -458,6 +459,35 @@ test_expect_success PERL 'required process filter should filter data for various
 			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
 			STOP
 		EOF
+		test_cmp_exclude_clean expected.log debug.log &&
+
+		git reset --hard empty-branch &&
+		filter_git reset --hard $MASTER &&
+		META="treeish=$MASTER" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
+		test_cmp_exclude_clean expected.log debug.log &&
+
+		git branch old-master $MASTER &&
+		git reset --hard empty-branch &&
+		filter_git reset --hard old-master &&
+		META="ref=refs/heads/old-master treeish=$MASTER" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
 		test_cmp_exclude_clean expected.log debug.log
 	)
 '

[PATCH v2 8/8] t0021: test filter metadata for additional cases

[brian m. carlson]

From: "brian m. carlson" <bk2204@github.com>

Check that we get the expected data when performing a merges or
generating archives.  Note that we don't expect a ref for merges,
because we won't be checking out any particular ref, but instead a tree
of the merged data.  For archives, however, we expect a ref as normal if
we have one.

Signed-off-by: brian m. carlson <bk2204@github.com>
---
 t/t0021-conversion.sh | 45 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index a026fd46b4..4bfffa9c31 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -488,6 +488,51 @@ test_expect_success PERL 'required process filter should filter data for various
 			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
 			STOP
 		EOF
+		test_cmp_exclude_clean expected.log debug.log &&
+
+		git checkout -b merge empty-branch &&
+		git branch -f master $MASTER2 &&
+		filter_git merge master &&
+		META="treeish=$MASTER2" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge test5.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
+		test_cmp_exclude_clean expected.log debug.log &&
+
+		filter_git archive master >/dev/null &&
+		META="ref=refs/heads/master treeish=$MASTER2" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge test5.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
+		test_cmp_exclude_clean expected.log debug.log &&
+
+		TREE="$(git rev-parse $MASTER2^{tree})" &&
+		filter_git archive $TREE >/dev/null &&
+		META="treeish=$TREE" &&
+		cat >expected.log <<-EOF &&
+			START
+			init handshake complete
+			IN: smudge test.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge test2.r $META blob=$M2 $S2 [OK] -- OUT: $S2 . [OK]
+			IN: smudge test4-empty.r $META blob=$EMPTY 0 [OK] -- OUT: 0  [OK]
+			IN: smudge test5.r $META blob=$M $S [OK] -- OUT: $S . [OK]
+			IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $META blob=$M3 $S3 [OK] -- OUT: $S3 . [OK]
+			STOP
+		EOF
 		test_cmp_exclude_clean expected.log debug.log
 	)
 '