[PATCH 0/4] rebase: update branches in multi-part topic

[PATCH 0/4] rebase: update branches in multi-part topic

[Derrick Stolee via GitGitGadget]

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'git update-ref' commands into
the TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
exec git update-ref refs/heads/bundle-redo/clone HEAD 9e4a6fe9b68a8455b427c9ac8cdbff30c96653b4

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
exec git update-ref refs/heads/bundle-redo/list HEAD 287a732b54c4d95e7f410b3b36ef90d8a19cd346

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

exec git update-ref refs/heads/bundle-redo/advertise HEAD c9578391976ab9899c4e4f9b5fa2827650097305


The first two patches are helpers that are needed, but the full logic of the
--update-refs option is introduced in patch 3. The config option is
available in patch 4.

Thanks, -Stolee

Derrick Stolee (4):
  log-tree: create for_each_decoration()
  branch: add branch_checked_out() helper
  rebase: add --update-refs option
  rebase: add rebase.updateRefs config option

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  11 ++++
 branch.c                        |  24 ++++---
 branch.h                        |   8 +++
 builtin/rebase.c                |  10 +++
 log-tree.c                      | 111 ++++++++++++++++++++++----------
 log-tree.h                      |   4 ++
 sequencer.c                     |  99 ++++++++++++++++++++++++++++
 sequencer.h                     |   1 +
 t/t3404-rebase-interactive.sh   |  34 ++++++++++
 10 files changed, 262 insertions(+), 43 deletions(-)


base-commit: 2668e3608e47494f2f10ef2b6e69f08a84816bcb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1247
-- 
gitgitgadget

[PATCH 1/4] log-tree: create for_each_decoration()

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

It can be helpful to iterate through all the decorations on a commit
without necessarily writing them to a stream. Implement
for_each_decoration() and reimplement format_decorations_extended() to
use that iterator.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 log-tree.c | 111 ++++++++++++++++++++++++++++++++++++-----------------
 log-tree.h |   4 ++
 2 files changed, 80 insertions(+), 35 deletions(-)

diff --git a/log-tree.c b/log-tree.c
index d0ac0a6327a..b15a7c9db22 100644
--- a/log-tree.c
+++ b/log-tree.c
@@ -282,6 +282,54 @@ static void show_name(struct strbuf *sb, const struct name_decoration *decoratio
 		strbuf_addstr(sb, decoration->name);
 }
 
+struct format_decorations_context {
+	struct strbuf *sb;
+	int use_color;
+	const char *prefix;
+	const char *separator;
+	const char *suffix;
+	const char *color_commit;
+	const char *color_reset;
+	const struct name_decoration *current_and_HEAD;
+};
+
+static int append_decoration(const struct name_decoration *d,
+			     void *data)
+{
+	struct format_decorations_context *ctx = data;
+	/*
+	 * When both current and HEAD are there, only
+	 * show HEAD->current where HEAD would have
+	 * appeared, skipping the entry for current.
+	 */
+	if (d != ctx->current_and_HEAD) {
+		strbuf_addstr(ctx->sb, ctx->color_commit);
+		strbuf_addstr(ctx->sb, ctx->prefix);
+		strbuf_addstr(ctx->sb, ctx->color_reset);
+		strbuf_addstr(ctx->sb, decorate_get_color(ctx->use_color, d->type));
+		if (d->type == DECORATION_REF_TAG)
+			strbuf_addstr(ctx->sb, "tag: ");
+
+		show_name(ctx->sb, d);
+
+		if (ctx->current_and_HEAD &&
+		    d->type == DECORATION_REF_HEAD) {
+			strbuf_addstr(ctx->sb, " -> ");
+			strbuf_addstr(ctx->sb, ctx->color_reset);
+			strbuf_addstr(ctx->sb,
+				      decorate_get_color(
+						ctx->use_color,
+						ctx->current_and_HEAD->type));
+			show_name(ctx->sb, ctx->current_and_HEAD);
+		}
+		strbuf_addstr(ctx->sb, ctx->color_reset);
+
+		ctx->prefix = ctx->separator;
+	}
+
+	return 0;
+}
+
 /*
  * The caller makes sure there is no funny color before calling.
  * format_decorations_extended makes sure the same after return.
@@ -294,49 +342,42 @@ void format_decorations_extended(struct strbuf *sb,
 			const char *suffix)
 {
 	const struct name_decoration *decoration;
-	const struct name_decoration *current_and_HEAD;
-	const char *color_commit =
-		diff_get_color(use_color, DIFF_COMMIT);
-	const char *color_reset =
-		decorate_get_color(use_color, DECORATION_NONE);
+	struct format_decorations_context ctx = {
+		.sb = sb,
+		.use_color = use_color,
+		.prefix = prefix,
+		.separator = separator,
+		.suffix = suffix,
+		.color_commit =	diff_get_color(use_color, DIFF_COMMIT),
+		.color_reset = decorate_get_color(use_color, DECORATION_NONE),
+	};
 
 	decoration = get_name_decoration(&commit->object);
 	if (!decoration)
 		return;
 
-	current_and_HEAD = current_pointed_by_HEAD(decoration);
-	while (decoration) {
-		/*
-		 * When both current and HEAD are there, only
-		 * show HEAD->current where HEAD would have
-		 * appeared, skipping the entry for current.
-		 */
-		if (decoration != current_and_HEAD) {
-			strbuf_addstr(sb, color_commit);
-			strbuf_addstr(sb, prefix);
-			strbuf_addstr(sb, color_reset);
-			strbuf_addstr(sb, decorate_get_color(use_color, decoration->type));
-			if (decoration->type == DECORATION_REF_TAG)
-				strbuf_addstr(sb, "tag: ");
-
-			show_name(sb, decoration);
-
-			if (current_and_HEAD &&
-			    decoration->type == DECORATION_REF_HEAD) {
-				strbuf_addstr(sb, " -> ");
-				strbuf_addstr(sb, color_reset);
-				strbuf_addstr(sb, decorate_get_color(use_color, current_and_HEAD->type));
-				show_name(sb, current_and_HEAD);
-			}
-			strbuf_addstr(sb, color_reset);
+	ctx.current_and_HEAD = current_pointed_by_HEAD(decoration);
 
-			prefix = separator;
-		}
+	for_each_decoration(commit, append_decoration, &ctx);
+
+	strbuf_addstr(sb, ctx.color_commit);
+	strbuf_addstr(sb, ctx.suffix);
+	strbuf_addstr(sb, ctx.color_reset);
+}
+
+int for_each_decoration(const struct commit *c, decoration_fn fn, void *data)
+{
+	const struct name_decoration *decoration;
+
+	decoration = get_name_decoration(&c->object);
+	while (decoration) {
+		int res;
+		if ((res = fn(decoration, data)))
+			return res;
 		decoration = decoration->next;
 	}
-	strbuf_addstr(sb, color_commit);
-	strbuf_addstr(sb, suffix);
-	strbuf_addstr(sb, color_reset);
+
+	return 0;
 }
 
 void show_decorations(struct rev_info *opt, struct commit *commit)
diff --git a/log-tree.h b/log-tree.h
index e7e4641cf83..ea07da2625b 100644
--- a/log-tree.h
+++ b/log-tree.h
@@ -35,4 +35,8 @@ void fmt_output_commit(struct strbuf *, struct commit *, struct rev_info *);
 void fmt_output_subject(struct strbuf *, const char *subject, struct rev_info *);
 void fmt_output_email_subject(struct strbuf *, struct rev_info *);
 
+typedef int decoration_fn(const struct name_decoration *d,
+			  void *data);
+int for_each_decoration(const struct commit *c, decoration_fn fn, void *data);
+
 #endif
-- 
gitgitgadget

Re: [PATCH 1/4] log-tree: create for_each_decoration()

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

>  	const struct name_decoration *decoration;
> +	struct format_decorations_context ctx = {
> +		.sb = sb,
> +		.use_color = use_color,
> +		.prefix = prefix,
> +		.separator = separator,
> +		.suffix = suffix,
> +		.color_commit =	diff_get_color(use_color, DIFF_COMMIT),
> +		.color_reset = decorate_get_color(use_color, DECORATION_NONE),
> +	};
>  
>  	decoration = get_name_decoration(&commit->object);
>  	if (!decoration)
>  		return;
>  
> +	ctx.current_and_HEAD = current_pointed_by_HEAD(decoration);
>  
> +	for_each_decoration(commit, append_decoration, &ctx);

The function for_each_decoration() that does not take decoration but
a commit felt iffy, especially because we already have called
get_name_decoration() to obtain one for commit, and the API forces
us to do that again at the beginning of for_each_decoration().

> +	strbuf_addstr(sb, ctx.color_commit);
> +	strbuf_addstr(sb, ctx.suffix);
> +	strbuf_addstr(sb, ctx.color_reset);
> +}
> +
> +int for_each_decoration(const struct commit *c, decoration_fn fn, void *data)
> +{
> +	const struct name_decoration *decoration;
> +
> +	decoration = get_name_decoration(&c->object);
> +	while (decoration) {
> +		int res;
> +		if ((res = fn(decoration, data)))
> +			return res;
>  		decoration = decoration->next;
>  	}
> +
> +	return 0;
>  }

We'll know if this small waste is worth it when we see the new
caller(s), I guess, but even if they start from commit, allowing
them the same early return trick would require this piece of code:

	decoration = get_name_decoration(&commit->object);
	if (!decoration)
		return;

in them, and even if they do not need it, it would be trivial for
them to say

	for_each_decoration(get_name_decoration(&commit->object),
        		    do_whatever_they_need_to_do, &ctx);

so, we may want to revisit this after we finish reading the series
through.  Making the iterator take name_decoration does not look
too bad.

>  void show_decorations(struct rev_info *opt, struct commit *commit)
> diff --git a/log-tree.h b/log-tree.h
> index e7e4641cf83..ea07da2625b 100644
> --- a/log-tree.h
> +++ b/log-tree.h
> @@ -35,4 +35,8 @@ void fmt_output_commit(struct strbuf *, struct commit *, struct rev_info *);
>  void fmt_output_subject(struct strbuf *, const char *subject, struct rev_info *);
>  void fmt_output_email_subject(struct strbuf *, struct rev_info *);
>  
> +typedef int decoration_fn(const struct name_decoration *d,
> +			  void *data);
> +int for_each_decoration(const struct commit *c, decoration_fn fn, void *data);
> +
>  #endif

Re: [PATCH 1/4] log-tree: create for_each_decoration()

[Derrick Stolee]

On 6/3/2022 1:39 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>>  	const struct name_decoration *decoration;
>> +	struct format_decorations_context ctx = {
>> +		.sb = sb,
>> +		.use_color = use_color,
>> +		.prefix = prefix,
>> +		.separator = separator,
>> +		.suffix = suffix,
>> +		.color_commit =	diff_get_color(use_color, DIFF_COMMIT),
>> +		.color_reset = decorate_get_color(use_color, DECORATION_NONE),
>> +	};
>>  
>>  	decoration = get_name_decoration(&commit->object);
>>  	if (!decoration)
>>  		return;
>>  
>> +	ctx.current_and_HEAD = current_pointed_by_HEAD(decoration);
>>  
>> +	for_each_decoration(commit, append_decoration, &ctx);
> 
> The function for_each_decoration() that does not take decoration but
> a commit felt iffy, especially because we already have called
> get_name_decoration() to obtain one for commit, and the API forces
> us to do that again at the beginning of for_each_decoration().

This is more an issue with this particular caller since it needs
that current_pointed_by_HEAD() information. I had considered it
as something to include in the prototype of decoration_fn so it
could be incorporated into for_each_decoration(), but it ended up
being overly clunky for this one consumer. I'm open to other ideas,
though.

>> +	strbuf_addstr(sb, ctx.color_commit);
>> +	strbuf_addstr(sb, ctx.suffix);
>> +	strbuf_addstr(sb, ctx.color_reset);
>> +}
>> +
>> +int for_each_decoration(const struct commit *c, decoration_fn fn, void *data)

The goal of this method was to make it super simple to iterate
without doing any prep work: just load the commit and go.

>> +{
>> +	const struct name_decoration *decoration;
>> +
>> +	decoration = get_name_decoration(&c->object);
>> +	while (decoration) {
>> +		int res;
>> +		if ((res = fn(decoration, data)))
>> +			return res;
>>  		decoration = decoration->next;
>>  	}
>> +
>> +	return 0;
>>  }
> 
> We'll know if this small waste is worth it when we see the new
> caller(s), I guess, but even if they start from commit, allowing
> them the same early return trick would require this piece of code:
> 
> 	decoration = get_name_decoration(&commit->object);
> 	if (!decoration)
> 		return;

The iterator returns quickly because the while loop notices a
NULL decoration. This use in format_decorations_extended() needs
that check because of the current_pointed_by_HEAD(decoration)
call based on the result being non-NULL.

> in them, and even if they do not need it, it would be trivial for
> them to say
> 
> 	for_each_decoration(get_name_decoration(&commit->object),
>         		    do_whatever_they_need_to_do, &ctx);
> 
> so, we may want to revisit this after we finish reading the series
> through.  Making the iterator take name_decoration does not look
> too bad.

You are right that this would work, but it's a bit messier. We
could go this route to avoid the duplicate get_name_decoration()
calls in format_decorations_extended().

Thanks,
-Stolee

Re: [PATCH 1/4] log-tree: create for_each_decoration()

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

>>> +	for_each_decoration(commit, append_decoration, &ctx);
>> 
>> The function for_each_decoration() that does not take decoration but
>> a commit felt iffy,...
>> ...
>> 	for_each_decoration(get_name_decoration(&commit->object),
>>         		    do_whatever_they_need_to_do, &ctx);
>> 
>> so, we may want to revisit this after we finish reading the series
>> through.  Making the iterator take name_decoration does not look
>> too bad.
>
> You are right that this would work, but it's a bit messier. We
> could go this route to avoid the duplicate get_name_decoration()
> calls in format_decorations_extended().

Having a convenience function that takes a commit and let you walk
over its decoration is OK, but calling it for_each_decoration()
bothers me somewhat (which is where my comment started from after
all).

Hopefully it will stay unambiguous, and it will stay to be
unnecessary to name it for_each_decoration_for_commit(), as I do not
think we'd add "decoration" to things that are not commits (or if we
added one, we probably will call it differently from "decoration").

So, OK.

[PATCH 2/4] branch: add branch_checked_out() helper

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The validate_new_branchname() method contains a check to see if a branch
is checked out in any non-bare worktree. This is intended to prevent a
force push that will mess up an existing checkout. This helper is not
suitable to performing just that check, because the method will die()
when the branch is checked out instead of returning an error code.

Extract branch_checked_out() and use it within
validate_new_branchname(). Another caller will be added in a coming
change.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 branch.c | 24 ++++++++++++++++--------
 branch.h |  8 ++++++++
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/branch.c b/branch.c
index 2d6569b0c62..2e6419cdfa5 100644
--- a/branch.c
+++ b/branch.c
@@ -369,6 +369,19 @@ int validate_branchname(const char *name, struct strbuf *ref)
 	return ref_exists(ref->buf);
 }
 
+int branch_checked_out(const char *refname, char **path)
+{
+	struct worktree **worktrees = get_worktrees();
+	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
+	int result = wt && !wt->is_bare;
+
+	if (result && path)
+		*path = xstrdup(wt->path);
+
+	free_worktrees(worktrees);
+	return result;
+}
+
 /*
  * Check if a branch 'name' can be created as a new branch; die otherwise.
  * 'force' can be used when it is OK for the named branch already exists.
@@ -377,9 +390,7 @@ int validate_branchname(const char *name, struct strbuf *ref)
  */
 int validate_new_branchname(const char *name, struct strbuf *ref, int force)
 {
-	struct worktree **worktrees;
-	const struct worktree *wt;
-
+	char *path;
 	if (!validate_branchname(name, ref))
 		return 0;
 
@@ -387,13 +398,10 @@ int validate_new_branchname(const char *name, struct strbuf *ref, int force)
 		die(_("a branch named '%s' already exists"),
 		    ref->buf + strlen("refs/heads/"));
 
-	worktrees = get_worktrees();
-	wt = find_shared_symref(worktrees, "HEAD", ref->buf);
-	if (wt && !wt->is_bare)
+	if (branch_checked_out(ref->buf, &path))
 		die(_("cannot force update the branch '%s' "
 		      "checked out at '%s'"),
-		    ref->buf + strlen("refs/heads/"), wt->path);
-	free_worktrees(worktrees);
+		    ref->buf + strlen("refs/heads/"), path);
 
 	return 1;
 }
diff --git a/branch.h b/branch.h
index 560b6b96a8f..5ea93d217b1 100644
--- a/branch.h
+++ b/branch.h
@@ -101,6 +101,14 @@ void create_branches_recursively(struct repository *r, const char *name,
 				 const char *tracking_name, int force,
 				 int reflog, int quiet, enum branch_track track,
 				 int dry_run);
+
+/*
+ * Returns true if the branch at 'refname' is checked out at any
+ * non-bare worktree. The path of the worktree is stored in the
+ * given 'path', if provided.
+ */
+int branch_checked_out(const char *refname, char **path);
+
 /*
  * Check if 'name' can be a valid name for a branch; die otherwise.
  * Return 1 if the named branch already exists; return 0 otherwise.
-- 
gitgitgadget

Re: [PATCH 2/4] branch: add branch_checked_out() helper

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +int branch_checked_out(const char *refname, char **path)
> +{
> +	struct worktree **worktrees = get_worktrees();
> +	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
> +	int result = wt && !wt->is_bare;
> +
> +	if (result && path)
> +		*path = xstrdup(wt->path);
> +
> +	free_worktrees(worktrees);
> +	return result;
> +}

Don't you plan to call this repeatedly from the for_each_deco
iteration?  I am wondering if it should take the result of
get_worktrees() and reuse the result of get_worktrees().

There also was another topic that was triggered by
find_shared_symref() being relatively heavy-weight, which suggests a
more involved refactoring.

I wonder if we rather want to rewrite find_shared_symref() *not* to
take the target parameter at all, and instead introduce a new
function that iterates over worktrees and report the branch that is
checked out (or being operated on via rebase or bisect).  Then we
can

 - create a strset out of its result, i.e. set of branches that
   should not be touched;

 - iterate over refs that point into the history being rebased
   (using for_each_decoration()), and consult that strset to see if
   any of them is being rewritten.  

With the API of find_shared_symref(), we'd need to iterate over all
worktrees for each decoration.  With such a restructuring, we can
iterate over all worktrees just once, and match the result with
decoration, so the problem becomes O(N)+O(M) and not O(N*M) for
number of worktrees N and number of decorations M.

[PATCH 3/4] rebase: add --update-refs option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When working on a large feature, it can be helpful to break that feature
into multiple smaller parts that become reviewed in sequence. During
development or during review, a change to one part of the feature could
affect multiple of these parts. An interactive rebase can help adjust
the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the
feature, then rebasing the entire list of commits can create commits not
reachable from those "sub branches". It can take a manual step to update
those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'git
update-ref' exec steps to the todo file whenever a commit that is being
rebased is decorated with that <ref>. This allows the user to rebase a
long list of commits in a multi-part feature and keep all of their
pointers to those parts.

Use the new for_each_decoration() while iterating over the existing todo
list. Be sure to iterate after any squashing or fixups are placed.
Update the branch only after those squashes and fixups are complete.
This allows a --fixup commit at the tip of the feature to apply
correctly to the sub branch, even if it is fixing up the most-recent
commit in that part.

One potential problem here is that refs decorating commits that are
already marked as "fixup!" or "squash!" will not be included in this
list. Generally, the reordering of the "fixup!" and "squash!" is likely
to change the relative order of these refs, so it is not recommended.
The workflow here is intended to allow these kinds of commits at the tip
of the rebased branch while the other sub branches come along for the
ride without intervention.

Be careful to not attempt updating any branch that is checked out. The
most common example is the branch being rebased is checked out and
decorates the tip commit. If the user is rebasing commits reachable from
a different branch that is checked out in a different worktree, then
they may be surprised to not see that ref update. However, it's probably
best to not optimize for this scenario and do the safest thing that will
result in a successful rebase. A comment is left in the TODO list that
signals that these refs are currently checked out.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-rebase.txt  |  7 +++
 builtin/rebase.c              |  5 ++
 sequencer.c                   | 99 +++++++++++++++++++++++++++++++++++
 sequencer.h                   |  1 +
 t/t3404-rebase-interactive.sh | 29 ++++++++++
 5 files changed, 141 insertions(+)

diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 262fb01aec0..866554fc978 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	or point to a `squash! ...` or `fixup! ...` commit are not updated
+	in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 7ab50cda2ad..56d82a52106 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update local refs that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
diff --git a/sequencer.c b/sequencer.c
index 8c3ed3532ac..d6151af9849 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -35,6 +35,8 @@
 #include "commit-reach.h"
 #include "rebase-interactive.h"
 #include "reset.h"
+#include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -5603,10 +5605,104 @@ static int skip_unnecessary_picks(struct repository *r,
 	return 0;
 }
 
+struct todo_add_branch_context {
+	struct todo_list new_list;
+	struct strbuf *buf;
+	struct commit *commit;
+};
+
+static int add_branch_for_decoration(const struct name_decoration *d, void *data)
+{
+	struct todo_add_branch_context *ctx = data;
+	size_t base_offset = ctx->buf->len;
+	int i = ctx->new_list.nr;
+	struct todo_item *item;
+	char *path;
+
+	ALLOC_GROW(ctx->new_list.items,
+		   ctx->new_list.nr + 1,
+		   ctx->new_list.alloc);
+	item = &ctx->new_list.items[i];
+
+	/* If the branch is checked out, then leave a comment instead. */
+	if (branch_checked_out(d->name, &path)) {
+		item->command = TODO_COMMENT;
+		strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
+			    d->name, path);
+		free(path);
+	} else {
+		item->command = TODO_EXEC;
+		strbuf_addf(ctx->buf, "git update-ref %s HEAD %s\n",
+			    d->name, oid_to_hex(&ctx->commit->object.oid));
+	}
+
+	item->commit = NULL;
+	item->offset_in_buf = base_offset;
+	item->arg_offset = base_offset;
+	item->arg_len = ctx->buf->len - base_offset;
+	ctx->new_list.nr++;
+
+	return 0;
+}
+
+/*
+ * For each 'pick' command, find out if the commit has a decoration in
+ * refs/heads/. If so, then add a 'git branch -f' exec command after
+ * that 'pick' (plus any following 'squash' or 'fixup' commands).
+ */
+static int todo_list_add_branch_commands(struct todo_list *todo_list)
+{
+	int i;
+	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
+	struct decoration_filter decoration_filter = {
+		.include_ref_pattern = &decorate_refs_include,
+		.exclude_ref_pattern = &decorate_refs_exclude,
+		.exclude_ref_config_pattern = &decorate_refs_exclude_config
+	};
+	struct todo_add_branch_context ctx = {
+		.new_list = TODO_LIST_INIT,
+		.buf = &todo_list->buf,
+	};
+
+	string_list_append(&decorate_refs_include, "refs/heads/");
+	load_ref_decorations(&decoration_filter, 0);
+
+	for (i = 0; i < todo_list->nr; ) {
+		struct todo_item *item = &todo_list->items[i];
+
+		do {
+			/* insert ith item into new list */
+			ALLOC_GROW(ctx.new_list.items,
+				   ctx.new_list.nr + 1,
+				   ctx.new_list.alloc);
+
+			memcpy(&ctx.new_list.items[ctx.new_list.nr++],
+			       &todo_list->items[i],
+			       sizeof(struct todo_item));
+
+			i++;
+		} while (i < todo_list->nr &&
+			 todo_list->items[i].command != TODO_PICK);
+
+		ctx.commit = item->commit;
+		for_each_decoration(item->commit, add_branch_for_decoration, &ctx);
+	}
+
+	free(todo_list->items);
+	todo_list->items = ctx.new_list.items;
+	todo_list->nr = ctx.new_list.nr;
+	todo_list->alloc = ctx.new_list.alloc;
+
+	return 0;
+}
+
 int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned keep_decorations,
 		    struct todo_list *todo_list)
 {
 	char shortonto[GIT_MAX_HEXSZ + 1];
@@ -5628,6 +5724,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 	if (autosquash && todo_list_rearrange_squash(todo_list))
 		return -1;
 
+	if (keep_decorations && todo_list_add_branch_commands(todo_list))
+		return -1;
+
 	if (commands->nr)
 		todo_list_add_exec_commands(todo_list, commands);
 
diff --git a/sequencer.h b/sequencer.h
index da64473636b..1add3c7119f 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -166,6 +166,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned keep_decorations,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index f31afd4a547..38c7ef95e0e 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1743,6 +1743,35 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
 	test_cmp_rev ORIG_HEAD test-orig-head@{1}
 '
 
+test_expect_success '--update-refs adds git branch commands' '
+	git checkout -b update-refs no-conflict-branch &&
+	test_commit extra fileX &&
+	git commit --amend --fixup=L &&
+	(
+		set_cat_todo_editor &&
+		git branch -f base A &&
+		git branch -f first J &&
+		git branch -f second J &&
+		git branch -f third L &&
+
+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+
+		cat >expect <<-EOF &&
+		pick $(git log -1 --format=%h J) J
+		exec git update-ref refs/heads/second HEAD $(git rev-parse J)
+		exec git update-ref refs/heads/first HEAD $(git rev-parse  J)
+		pick $(git log -1 --format=%h K) K
+		pick $(git log -1 --format=%h L) L
+		fixup $(git log -1 --format=%h update-refs) fixup! L
+		exec git update-ref refs/heads/third HEAD $(git rev-parse L)
+		pick $(git log -1 --format=%h M) M
+		exec git update-ref refs/heads/no-conflict-branch HEAD $(git rev-parse M)
+		EOF
+
+		test_cmp expect todo
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH 3/4] rebase: add --update-refs option

[Phillip Wood]

Hi Stolee

Just a couple of minor comments.

On 03/06/2022 14:37, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> [...]
> diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
> index 262fb01aec0..866554fc978 100644
> --- a/Documentation/git-rebase.txt
> +++ b/Documentation/git-rebase.txt
> @@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
>   start would be overridden by the presence of
>   `rebase.rescheduleFailedExec=true` configuration.
>   
> +--update-refs::
> +--no-update-refs::
> +	Automatically force-update any branches that point to commits that
> +	are being rebased. Any branches that are checked out in a worktree
> +	or point to a `squash! ...` or `fixup! ...` commit are not updated
> +	in this way.
> +
>   INCOMPATIBLE OPTIONS
>   --------------------

We should add --update-refs to the list of options that are incompatible 
with --apply.

> diff --git a/sequencer.c b/sequencer.c
> index 8c3ed3532ac..d6151af9849 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -35,6 +35,8 @@
>   #include "commit-reach.h"
>   #include "rebase-interactive.h"
>   #include "reset.h"
> +#include "branch.h"
> +#include "log-tree.h"
>   
>   #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>   
> @@ -5603,10 +5605,104 @@ static int skip_unnecessary_picks(struct repository *r,
>   	return 0;
>   }
>   
> +struct todo_add_branch_context {
> +	struct todo_list new_list;

Rather than using a struct todo_list I think it would be simpler overall 
to add
	struct todo_list_item *item;
	size_t nr, alloc;

instead, as I found it confusing that we were (correctly) using the 
strbuf of the old list when adding the update-ref line to the new list.

> +	struct strbuf *buf;
> +	struct commit *commit;
> +};
> [...]
> +	for (i = 0; i < todo_list->nr; ) {
> +		struct todo_item *item = &todo_list->items[i];
> +
> +		do {
> +			/* insert ith item into new list */
> +			ALLOC_GROW(ctx.new_list.items,
> +				   ctx.new_list.nr + 1,
> +				   ctx.new_list.alloc);
> +
> +			memcpy(&ctx.new_list.items[ctx.new_list.nr++],
> +			       &todo_list->items[i],
> +			       sizeof(struct todo_item));

May be
	ctx.new_list.items[ctx.new_list.nr++] = todo_list->items[i++];
would be clearer

Best Wishes

Phillip

[PATCH 4/4] rebase: add rebase.updateRefs config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change added the --update-refs command-line option.  For
users who always want this mode, create the rebase.updateRefs config
option which behaves the same way as rebase.autoSquash does with the
--autosquash option.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config/rebase.txt | 3 +++
 Documentation/git-rebase.txt    | 4 ++++
 builtin/rebase.c                | 5 +++++
 t/t3404-rebase-interactive.sh   | 5 +++++
 4 files changed, 17 insertions(+)

diff --git a/Documentation/config/rebase.txt b/Documentation/config/rebase.txt
index 8c979cb20f2..f19bd0e0407 100644
--- a/Documentation/config/rebase.txt
+++ b/Documentation/config/rebase.txt
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 866554fc978..cb1b498bd99 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -615,6 +615,10 @@ start would be overridden by the presence of
 	are being rebased. Any branches that are checked out in a worktree
 	or point to a `squash! ...` or `fixup! ...` commit are not updated
 	in this way.
++
+If the `--update-refs` option is enabled by default using the
+configuration variable `rebase.updateRefs`, this option can be
+used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 56d82a52106..8ebc98ea505 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 38c7ef95e0e..1fa9f78d40d 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1768,6 +1768,11 @@ test_expect_success '--update-refs adds git branch commands' '
 		exec git update-ref refs/heads/no-conflict-branch HEAD $(git rev-parse M)
 		EOF
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
 		test_cmp expect todo
 	)
 '
-- 
gitgitgadget

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
>
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'git update-ref' commands into
> the TODO list. This is powered by the commit decoration machinery.

;-)

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Taylor Blau]

On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.

This is really exciting. I'm glad that you're working on it, because I
have also wanted this feature a handful of times over the years.

This definitely would have come in handy when writing MIDX bitmaps,
where I was often editing a local branch pointing at the final topic,
and then trying to reconstruct all of the intermediate branches after
each rebase. Not ever having to do that again sounds like a delight ;-).

> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730

But I wonder if we can or should delay these update-refs as long as
possible. In particular: what happens if I get past this "exec" line (so
that I've already updated my bundle-redo/fetch branch to point at the
new thing), but decide at some later point to abort the rebase?

I think users will expect us to restore bundle-redo/fetch to where it
was before if we end up in that case. Recovering from it manually sounds
like kind of a headache.

What if instead we created labels here, and then delayed all ref updates
to the end by replacing this with:

    label bundle-redo/fetch

and then at the end of the todo list we'd add:

    exec git update-ref refs/heads/bundle-redo/fetch refs/rewritten/bundle-redo/fetch

If we do all of those ref updates in a single transaction at the end, it
should be much easier to roll back from if desired, and we'd avoid the
aborted-rebase problem entirely.

Thanks,
Taylor

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> But I wonder if we can or should delay these update-refs as long as
> possible. In particular: what happens if I get past this "exec" line (so
> that I've already updated my bundle-redo/fetch branch to point at the
> new thing), but decide at some later point to abort the rebase?
>
> I think users will expect us to restore bundle-redo/fetch to where it
> was before if we end up in that case. Recovering from it manually sounds
> like kind of a headache.

That is a very good safety and usability concern.  I am glad
somebody thought of it.

> What if instead we created labels here, and then delayed all ref updates
> to the end by replacing this with:
>
>     label bundle-redo/fetch
>
> and then at the end of the todo list we'd add:
>
>     exec git update-ref refs/heads/bundle-redo/fetch refs/rewritten/bundle-redo/fetch
>
> If we do all of those ref updates in a single transaction at the end, it
> should be much easier to roll back from if desired, and we'd avoid the
> aborted-rebase problem entirely.

;-)

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Jeff Hostetler]

On 6/3/22 2:27 PM, Taylor Blau wrote:
> On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
>> This is a feature I've wanted for quite a while. When working on the sparse
>> index topic, I created a long RFC that actually broke into three topics for
>> full review upstream. These topics were sequential, so any feedback on an
>> earlier one required updates to the later ones. I would work on the full
>> feature and use interactive rebase to update the full list of commits.
>> However, I would need to update the branches pointing to those sub-topics.
> 
> This is really exciting. I'm glad that you're working on it, because I
> have also wanted this feature a handful of times over the years.
> 
> This definitely would have come in handy when writing MIDX bitmaps,
> where I was often editing a local branch pointing at the final topic,
> and then trying to reconstruct all of the intermediate branches after
> each rebase. Not ever having to do that again sounds like a delight ;-).
> 
>> pick 2d966282ff3 docs: document bundle URI standard
>> pick 31396e9171a remote-curl: add 'get' capability
>> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
>> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
>> pick 6adaf842684 fetch: add --bundle-uri option
>> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
>> exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730
> 
> But I wonder if we can or should delay these update-refs as long as
> possible. In particular: what happens if I get past this "exec" line (so
> that I've already updated my bundle-redo/fetch branch to point at the
> new thing), but decide at some later point to abort the rebase?
> 
> I think users will expect us to restore bundle-redo/fetch to where it
> was before if we end up in that case. Recovering from it manually sounds
> like kind of a headache.
> 
> What if instead we created labels here, and then delayed all ref updates
> to the end by replacing this with:
> 
>      label bundle-redo/fetch
> 
> and then at the end of the todo list we'd add:
> 
>      exec git update-ref refs/heads/bundle-redo/fetch refs/rewritten/bundle-redo/fetch
> 
> If we do all of those ref updates in a single transaction at the end, it
> should be much easier to roll back from if desired, and we'd avoid the
> aborted-rebase problem entirely.
> 
> Thanks,
> Taylor
> 

I agree. I could have really used this while juggling all of the
parts of FSMonitor recently.  And yes, it should write the updates
at the bottom in case of an abort.

Should this take a branch pattern/regex to limit the set of branches
that are updated (or offered to be updated)?  For example, if I have
an intermediate commit in the series that has 2 branch names pointing
at it, do we want to offer to update both of them or only the one
that matches some pattern related to the tip?  Or is it sufficient to
just enumerate them at the bottom of the todo list and let the user
delete the lines they don't want?

Should we actually do the update-ref's or should we write a script
that lets the user do it later?  The latter would let us also write
out the commands to force update the remote refs if that would be
helpful.

Thanks,
Jeff

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Taylor Blau]

On Fri, Jun 03, 2022 at 03:59:00PM -0400, Jeff Hostetler wrote:
> Should this take a branch pattern/regex to limit the set of branches
> that are updated (or offered to be updated)?  For example, if I have
> an intermediate commit in the series that has 2 branch names pointing
> at it, do we want to offer to update both of them or only the one
> that matches some pattern related to the tip?  Or is it sufficient to
> just enumerate them at the bottom of the todo list and let the user
> delete the lines they don't want?

That could be a useful feature to layer on top. I don't think it's
required here, though, since (as you note) users can remove lines from
the update-ref invocation(s) at the bottom that they don't want to
perform.

> Should we actually do the update-ref's or should we write a script
> that lets the user do it later?  The latter would let us also write
> out the commands to force update the remote refs if that would be
> helpful.

That seems like it would be outside the bounds of what `rebase` should
provide, but others may feel differently.

Thanks,
Taylor

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Junio C Hamano]

Jeff Hostetler <git@jeffhostetler.com> writes:

> I agree. I could have really used this while juggling all of the
> parts of FSMonitor recently.  And yes, it should write the updates
> at the bottom in case of an abort.
>
> Should this take a branch pattern/regex to limit the set of branches
> that are updated (or offered to be updated)?  For example, if I have
> an intermediate commit in the series that has 2 branch names pointing
> at it, do we want to offer to update both of them or only the one
> that matches some pattern related to the tip?  Or is it sufficient to
> just enumerate them at the bottom of the todo list and let the user
> delete the lines they don't want?

The latter sounds sufficient (starting from something simpler should
work well in this case).

> Should we actually do the update-ref's or should we write a script
> that lets the user do it later?  The latter would let us also write
> out the commands to force update the remote refs if that would be
> helpful.

Aren't we writing "a script" already by implementing it as an
additional "exec git update-ref" in the todo list already?  I
found that, combined with the idea to use the "label" Taylor
mentioned, was the most brilliant part of this proposal.

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Phillip Wood]

On 03/06/2022 19:27, Taylor Blau wrote:
> On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
>> This is a feature I've wanted for quite a while. When working on the sparse
>> index topic, I created a long RFC that actually broke into three topics for
>> full review upstream. These topics were sequential, so any feedback on an
>> earlier one required updates to the later ones. I would work on the full
>> feature and use interactive rebase to update the full list of commits.
>> However, I would need to update the branches pointing to those sub-topics.
> 
> This is really exciting. I'm glad that you're working on it, because I
> have also wanted this feature a handful of times over the years.

Yes, thank you Stolee. I agree this will be useful, but I wonder if 
users will be confused that --update-refs only updates the branch heads 
that happen to be in the todo list, rather than updating all the 
branches that contain a rewritten commit. I think the latter is 
something we should try to add in the future and so we should take care 
to design this topic so that is possible.

> This definitely would have come in handy when writing MIDX bitmaps,
> where I was often editing a local branch pointing at the final topic,
> and then trying to reconstruct all of the intermediate branches after
> each rebase. Not ever having to do that again sounds like a delight ;-).
> 
>> pick 2d966282ff3 docs: document bundle URI standard
>> pick 31396e9171a remote-curl: add 'get' capability
>> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
>> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
>> pick 6adaf842684 fetch: add --bundle-uri option
>> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
>> exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730
> 
> But I wonder if we can or should delay these update-refs as long as
> possible. In particular: what happens if I get past this "exec" line (so
> that I've already updated my bundle-redo/fetch branch to point at the
> new thing), but decide at some later point to abort the rebase?

Absolutely! There is also the question of what to do if a user skips a 
commit that is a branch head. It is not obvious if they just want to 
drop that commit from the branch or if they want to skip updating the 
ref as well (I guess they can edit the todo list for the latter case).

> I think users will expect us to restore bundle-redo/fetch to where it
> was before if we end up in that case. Recovering from it manually sounds
> like kind of a headache.
> 
> What if instead we created labels here, and then delayed all ref updates
> to the end by replacing this with:
> 
>      label bundle-redo/fetch

Instead of using 'label' and 'exec' I'd prefer a new todo list command 
('update-ref' or 'update-branch'?) used in place of 'label' that takes a 
branch name and updates the branch ref at the end of the rebase. That 
would make it easy to do all the updates in a single transaction as you 
suggested. Adding exec lines to do this makes the todo list messy and we 
have been trying to stop rebase forking all the time.

Best Wishes

Phillip

> and then at the end of the todo list we'd add:
> 
>      exec git update-ref refs/heads/bundle-redo/fetch refs/rewritten/bundle-redo/fetch
> 
> If we do all of those ref updates in a single transaction at the end, it
> should be much easier to roll back from if desired, and we'd avoid the
> aborted-rebase problem entirely.


> Thanks,
> Taylor

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Derrick Stolee]

On 6/4/2022 11:28 AM, Phillip Wood wrote:
> On 03/06/2022 19:27, Taylor Blau wrote:
>> On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
>>> This is a feature I've wanted for quite a while. When working on the sparse
>>> index topic, I created a long RFC that actually broke into three topics for
>>> full review upstream. These topics were sequential, so any feedback on an
>>> earlier one required updates to the later ones. I would work on the full
>>> feature and use interactive rebase to update the full list of commits.
>>> However, I would need to update the branches pointing to those sub-topics.
>>
>> This is really exciting. I'm glad that you're working on it, because I
>> have also wanted this feature a handful of times over the years.
> 
> Yes, thank you Stolee. I agree this will be useful, but I wonder if users
> will be confused that --update-refs only updates the branch heads that happen
> to be in the todo list, rather than updating all the branches that contain a
> rewritten commit. I think the latter is something we should try to add in the
> future and so we should take care to design this topic so that is possible.

At the moment, the design adds a comment to the TODO list, showing which
branches are not possible to move because they are checked out at another
worktree (or is currently checked out and will be updated by the rebase
itself). That seems like a good place to insert alternative logic in the
future if we see a need for better behavior here.

Unless: am I misunderstanding something about your concern here? Are you
worried about refs outside of refs/heads/*? Are you concerned about it
being _all_ refs/heads/* that we find?

One potential way to extend this (in the future) is to make --update-refs
take an optional string argument containing a refspec. This would replace
the default refspec of refs/heads/*.
 
>> This definitely would have come in handy when writing MIDX bitmaps,
>> where I was often editing a local branch pointing at the final topic,
>> and then trying to reconstruct all of the intermediate branches after
>> each rebase. Not ever having to do that again sounds like a delight ;-).
>>
>>> pick 2d966282ff3 docs: document bundle URI standard
>>> pick 31396e9171a remote-curl: add 'get' capability
>>> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
>>> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
>>> pick 6adaf842684 fetch: add --bundle-uri option
>>> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
>>> exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730
>>
>> But I wonder if we can or should delay these update-refs as long as
>> possible. In particular: what happens if I get past this "exec" line (so
>> that I've already updated my bundle-redo/fetch branch to point at the
>> new thing), but decide at some later point to abort the rebase?
> 
> Absolutely! There is also the question of what to do if a user skips a
> commit that is a branch head. It is not obvious if they just want to drop> that commit from the branch or if they want to skip updating the ref as> well (I guess they can edit the todo list for the latter case).

>> I think users will expect us to restore bundle-redo/fetch to where it
>> was before if we end up in that case. Recovering from it manually sounds
>> like kind of a headache.
>>
>> What if instead we created labels here, and then delayed all ref updates
>> to the end by replacing this with:
>>
>>      label bundle-redo/fetch
> 
> Instead of using 'label' and 'exec' I'd prefer a new todo list command
> ('update-ref' or 'update-branch'?) used in place of 'label' that takes a
> branch name and updates the branch ref at the end of the rebase. That
> would make it easy to do all the updates in a single transaction as you
> suggested. Adding exec lines to do this makes the todo list messy and we
> have been trying to stop rebase forking all the time.

Thanks. Yes, a new 'update-refs' step at the end would be good to make
it clear we want to rewrite the refs in one go without a possible
interruption from the user.

Thanks,
-Stolee

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Phillip Wood]

On 06/06/2022 16:12, Derrick Stolee wrote:
> On 6/4/2022 11:28 AM, Phillip Wood wrote:
>> On 03/06/2022 19:27, Taylor Blau wrote:
>>> On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
>>>> This is a feature I've wanted for quite a while. When working on the sparse
>>>> index topic, I created a long RFC that actually broke into three topics for
>>>> full review upstream. These topics were sequential, so any feedback on an
>>>> earlier one required updates to the later ones. I would work on the full
>>>> feature and use interactive rebase to update the full list of commits.
>>>> However, I would need to update the branches pointing to those sub-topics.
>>>
>>> This is really exciting. I'm glad that you're working on it, because I
>>> have also wanted this feature a handful of times over the years.
>>
>> Yes, thank you Stolee. I agree this will be useful, but I wonder if users
>> will be confused that --update-refs only updates the branch heads that happen
>> to be in the todo list, rather than updating all the branches that contain a
>> rewritten commit. I think the latter is something we should try to add in the
>> future and so we should take care to design this topic so that is possible.
> 
> At the moment, the design adds a comment to the TODO list, showing which
> branches are not possible to move because they are checked out at another
> worktree (or is currently checked out and will be updated by the rebase
> itself). That seems like a good place to insert alternative logic in the
> future if we see a need for better behavior here.

I think the question of whether to update branches that are checked out 
in another worktree is a question of whether it is less inconvenient to 
the user to skip the ref update and leave the user to manually update 
the branch or to update the ref and leave the worktree in a potentially 
awkward state if the user was half way through building a commit. The 
answer probably depends on the preferences of the user.

I've been using a script that updates the refs for all the branches 
being rewritten for a while and have found it preferable to always 
update the ref rather than have to do it manually. My script also 
updates the worktree checkout to the new HEAD if there are no 
uncommitted changes which I have found very convenient. My preference is 
probably because I tend not to have uncommitted changes lying around in 
the worktrees whose branches get updated.

> Unless: am I misunderstanding something about your concern here? Are you
> worried about refs outside of refs/heads/*? Are you concerned about it
> being _all_ refs/heads/* that we find?

My concerns are primarily about being able to extend the --update-ref 
option in a backwards compatible way.

I'd like to be able to add functionality to rebase all refs/heads/* that 
are descended from the commits that a simple rebase would rewrite. Say I 
want to edit $commit then I want the rebase to rewrite all the commits 
and update all the branches in in

     git rev-list $(git for-each-ref --contains $commit refs/heads/) 
^$commit^@

Ideally we'd avoid adding a new commandline option when adding that. I 
think we could use an optional argument as you suggest below (though it 
would not be a refspec).

> One potential way to extend this (in the future) is to make --update-refs
> take an optional string argument containing a refspec. This would replace
> the default refspec of refs/heads/*.
> [...]  
>> Instead of using 'label' and 'exec' I'd prefer a new todo list command
>> ('update-ref' or 'update-branch'?) used in place of 'label' that takes a
>> branch name and updates the branch ref at the end of the rebase. That
>> would make it easy to do all the updates in a single transaction as you
>> suggested. Adding exec lines to do this makes the todo list messy and we
>> have been trying to stop rebase forking all the time.
> 
> Thanks. Yes, a new 'update-refs' step at the end would be good to make
> it clear we want to rewrite the refs in one go without a possible
> interruption from the user.

That's great, it is a bit more work for you but I think it gives a much 
nicer UI.

Best Wishes

Phillip

> Thanks,
> -Stolee

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Derrick Stolee]

On 6/7/2022 6:11 AM, Phillip Wood wrote:
> On 06/06/2022 16:12, Derrick Stolee wrote:
>> On 6/4/2022 11:28 AM, Phillip Wood wrote:
>>> On 03/06/2022 19:27, Taylor Blau wrote:
>>>> On Fri, Jun 03, 2022 at 01:37:48PM +0000, Derrick Stolee via GitGitGadget wrote:
>>>>> This is a feature I've wanted for quite a while. When working on the sparse
>>>>> index topic, I created a long RFC that actually broke into three topics for
>>>>> full review upstream. These topics were sequential, so any feedback on an
>>>>> earlier one required updates to the later ones. I would work on the full
>>>>> feature and use interactive rebase to update the full list of commits.
>>>>> However, I would need to update the branches pointing to those sub-topics.
>>>>
>>>> This is really exciting. I'm glad that you're working on it, because I
>>>> have also wanted this feature a handful of times over the years.
>>>
>>> Yes, thank you Stolee. I agree this will be useful, but I wonder if users
>>> will be confused that --update-refs only updates the branch heads that happen
>>> to be in the todo list, rather than updating all the branches that contain a
>>> rewritten commit. I think the latter is something we should try to add in the
>>> future and so we should take care to design this topic so that is possible.
>>
>> At the moment, the design adds a comment to the TODO list, showing which
>> branches are not possible to move because they are checked out at another
>> worktree (or is currently checked out and will be updated by the rebase
>> itself). That seems like a good place to insert alternative logic in the
>> future if we see a need for better behavior here.
> 
> I think the question of whether to update branches that are checked out in
> another worktree is a question of whether it is less inconvenient to the user
> to skip the ref update and leave the user to manually update the branch or to> update the ref and leave the worktree in a potentially awkward state if the
> user was half way through building a commit. The answer probably depends on
> the preferences of the user.

I think that their 'git status' will look strange no matter what: their
working directory and index could look significantly different from what
the branch at HEAD is reporting. For this situation, I would rather continue
preventing these ref updates from underneath worktrees.
 
> I've been using a script that updates the refs for all the branches being
> rewritten for a while and have found it preferable to always update the ref
> rather than have to do it manually. My script also updates the worktree
> checkout to the new HEAD if there are no uncommitted changes which I have
> found very convenient. My preference is probably because I tend not to have
> uncommitted changes lying around in the worktrees whose branches get updated.

Actually updating the worktree to match seems like an interesting twist, which
we would want to consider if we go this route in the future.
 
>> Unless: am I misunderstanding something about your concern here? Are you
>> worried about refs outside of refs/heads/*? Are you concerned about it
>> being _all_ refs/heads/* that we find?
> 
> My concerns are primarily about being able to extend the --update-ref
> option in a backwards compatible way.
> 
> I'd like to be able to add functionality to rebase all refs/heads/* that
> are descended from the commits that a simple rebase would rewrite. Say I want
> to edit $commit then I want the rebase to rewrite all the commits and update
> all the branches in in
> 
>     git rev-list $(git for-each-ref --contains $commit refs/heads/) ^$commit^@
> 
> Ideally we'd avoid adding a new commandline option when adding that. I think
> we could use an optional argument as you suggest below (though it would not be
> a refspec).

This is sort of the opposite of what my series is doing: you want to find all
refs that contain the current commits and make sure that they are _also_
rebased as we rebase the current set of commits. That seems like an interesting
direction, with a very different set of complexities (ignoring other worktrees
seems like much more of a blocker here).

I would consider this to be _yet another mode_ that is separate from
--update-refs, though it could share some underlying logic. It gets more
complicated if there are merge commits involved (do we have a --rebase-merges
option?).

This makes me think of earlier discussions around a "git evolve" command
(based on Mercurial's evolve command). The idea is to have a different
workflow than I am presenting: instead of creating `fixup!` commits at the
top of a multi-part topic, you could check out a branch on a middle part
and work forward from there. Then, you could "rebase every branch with a
rewritten commit" to get things back into a nice line.

In conclusion: I don't think we should go down this rabbit hole right now.
I think that --update-refs would be something we want to enable by default
if we have such a mode, though.
 
>> One potential way to extend this (in the future) is to make --update-refs
>> take an optional string argument containing a refspec. This would replace
>> the default refspec of refs/heads/*.
>> [...] 
>>> Instead of using 'label' and 'exec' I'd prefer a new todo list command
>>> ('update-ref' or 'update-branch'?) used in place of 'label' that takes a
>>> branch name and updates the branch ref at the end of the rebase. That
>>> would make it easy to do all the updates in a single transaction as you
>>> suggested. Adding exec lines to do this makes the todo list messy and we
>>> have been trying to stop rebase forking all the time.
>>
>> Thanks. Yes, a new 'update-refs' step at the end would be good to make
>> it clear we want to rewrite the refs in one go without a possible
>> interruption from the user.
> 
> That's great, it is a bit more work for you but I think it gives a much
> nicer UI.

Thanks. After some hurdles on my end (and some additional complexities
discovered in the process) I will have v2 ready soon.

Thanks,
-Stolee

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

>> I think the question of whether to update branches that are checked out in
>> another worktree is a question of whether it is less inconvenient to the user
>> to skip the ref update and leave the user to manually update the branch or to
>> update the ref and leave the worktree in a potentially awkward state if the
>> user was half way through building a commit. The answer probably depends on
>> the preferences of the user.

To some degree, it might be true, but I think the recent thinking is
that by default any refs being worked on by the user must be kept
intact.  "switch" does not let you check out a branch that is
already checked out elsewhere, "fetch" does not let you overwrite
the branch that is currently checked out without "--update-head-ok",
etc.

> I think that their 'git status' will look strange no matter what: their
> working directory and index could look significantly different from what
> the branch at HEAD is reporting. For this situation, I would rather continue
> preventing these ref updates from underneath worktrees.
>  
>> I've been using a script that updates the refs for all the branches being
>> rewritten for a while and have found it preferable to always update the ref
>> rather than have to do it manually. My script also updates the worktree
>> checkout to the new HEAD if there are no uncommitted changes which I have
>> found very convenient. My preference is probably because I tend not to have
>> uncommitted changes lying around in the worktrees whose branches get updated.
>
> Actually updating the worktree to match seems like an interesting twist, which
> we would want to consider if we go this route in the future.

Usually I caution against adding "features" that can complicate the
end-user experience like this, but in this case, the potential for
extra conflicts coming from such updates to the working trees may
not be too bad.

It all depends on why the user has these intermediate branches and
how they are used, but in order to see any conflicts happen after
rebasing a branch here, before you start that rebase, you need to
have in different working trees that had these intermediate branches
checked out and had local modifications on top of them.

I would say that you deserve it if conflicts resulting from such a
set-up hurts you ;-)  If you are using these intermediate branches
so that you can work on steps in the middle of the larger whole,
then the proposed "rebase --update-refs" is not for you.  Having
local modification in these separate working trees that check out
the intermediate branches is OK, but once you create a commit on
such a branch, the commit that is in the larger topic is no longer
at the tip, so "rebase --update-refs" would not notice it anyway.

Rather, if you have the intermediate branches, you'd probably want
to work on each of them to make them stable, and rebase them on top
of each other in an appropriate order.  "rebase --update-refs" that
runs on the larger topic does not have enough information to rebuild
it with tips of intermediate branches that are updated elsewhere.

Which means that anybody sane who uses "rebase --update-refs" would
not modify these intermediate branches outside the context of the
larger topic, and those who do not follow this rule can keep both
halves of their history ;-)

FWIW a complementary tool that would work in the other direction, to
expect the user to have worked on smaller branches and rebuild the
larger branch that contains them, is also missing from our tool set.
Those who check out these smaller branches in separate working trees
would want to use such a tool, not "rebase --update-ref".

Thanks.

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> Instead of using 'label' and 'exec' I'd prefer a new todo list command
> ('update-ref' or 'update-branch'?) used in place of 'label' that takes
> a branch name and updates the branch ref at the end of the
> rebase. That would make it easy to do all the updates in a single
> transaction as you suggested.

Sounds like a good approach.

Re: [PATCH 0/4] rebase: update branches in multi-part topic

[Elijah Newren]

On Fri, Jun 3, 2022 at 11:01 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
>
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'git update-ref' commands into
> the TODO list. This is powered by the commit decoration machinery.
>
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
>
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730
>
> pick 1e3f6546632 clone: add --bundle-uri option
> pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
> exec git update-ref refs/heads/bundle-redo/clone HEAD 9e4a6fe9b68a8455b427c9ac8cdbff30c96653b4
>
> pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
> pick 3029c3aca15 bundle-uri: create base key-value pair parsing
> pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
> pick 92625a47673 bundle-uri: unit test "key=value" parsing
> pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
> pick 9d6809a8d53 bundle-uri: parse bundle list in config format
> pick 287a732b54c bundle-uri: fetch a list of bundles
> exec git update-ref refs/heads/bundle-redo/list HEAD 287a732b54c4d95e7f410b3b36ef90d8a19cd346
>
> pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
> pick 520204dcd1c bundle-uri client: add minimal NOOP client
> pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
> pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
> pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
> pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
> pick df255000b7e bundle-uri: download bundles from an advertised list
> pick d71beabf199 clone: unbundle the advertised bundles
> pick c9578391976 t5601: basic bundle URI tests
> # Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'
>
> exec git update-ref refs/heads/bundle-redo/advertise HEAD c9578391976ab9899c4e4f9b5fa2827650097305
>
>
> The first two patches are helpers that are needed, but the full logic of the
> --update-refs option is introduced in patch 3. The config option is
> available in patch 4.

Very interesting; nice to see that others are thinking on a similar
wavelength.  I've got some related work in git-replay, and it heavily
uses update-refs style handling.

Not sure if there are any differences worth affecting your design or
mine, but perhaps it's worth mentioning my current work and plans in
git-replay related to the use of update-refs:



Since git-replay doesn't touch the working tree or index, I had the
natural question of what the output should be and whether ref-updates
should automatically happen.  I decided that like git-merge-tree, ref
updates should not be automatic (and thus it behaves somewhat like a
"dry run" version of rebase/cherry-pick).  Since I also want to
intrinsically handle replaying multiple branches simultaneously, the
update-ref style output seemed like a natural fit to me (meaning users
would pipe the output from git-reply to git-update-ref).  Thus, one
can do:

    $ git replay --onto main startpoint..mytopic

and see git-update-ref style output:

    update refs/heads/mytopic ${NEW_mytopic_HASH} ${OLD_mytopic_HASH}

where ${NEW_mytopic_HASH} is the commit at the tip of a chain of
rebased mytopic commits.  (Sidenote: There's the question of whether
to update 'main' (i.e. cherry-picking) or 'mytopic' (i.e. rebasing);
thus, users have to choose --advance vs. --onto to specify which of
these should happen, except in cases where it can be implicitly
determined.)

Or, if you have several branches to update:

    $ git replay --onto main ^startpoint mytopic1 mytopic2 mytopic3

and see

    update refs/heads/mytopic1 ${NEW_mytopic1_HASH} ${OLD_mytopic1_HASH}
    update refs/heads/mytopic2 ${NEW_mytopic2_HASH} ${OLD_mytopic2_HASH}
    update refs/heads/mytopic3 ${NEW_mytopic3_HASH} ${OLD_mytopic3_HASH}

(and yes, common commits from before will be replayed and then shared
by the new branch versions.)

If the topics are totally contained within each other, you can do this
more simply as

    $ git replay --contained --onto main main..my_tip_topic

and still see all the update commands:

    update refs/heads/mytopic1 ${NEW_mytopic1_HASH} ${OLD_mytopic1_HASH}
    update refs/heads/mytopic2 ${NEW_mytopic2_HASH} ${OLD_mytopic2_HASH}
    update refs/heads/mytopic3 ${NEW_mytopic3_HASH} ${OLD_mytopic3_HASH}
    update refs/heads/my_tip_topic ${NEW_my_tip_topic_HASH}
${OLD_my_tip_topic_HASH}

The option to list several refs provides a natural way of allowing
users to control what gets updated without updating all contained
branches, though rebase's implicit ranges doesn't really have a nice
analogue for you due to its implicit range handling.

Right now, git-replay doesn't handle conflicts (it just die()s).  That
and a few other thing are still TODO.

However, git-replay does already handle basic replaying of merges,
including ones with evil changes (so long as none of the merges
involved have conflicts).  So, I'm hoping git-replay will also useful
for stuff like:

   $ git replay --first-parent --interactive --onto next ^next
special_topic seen

where special_topic is one of the topics previously merged into seen.
This last commit would allow seen to be rebuilt with some changes to
special_topic.  Basically, it makes whatever interactive changes are
specified to special_topic followed by replaying all the merges in
next..seen (including any necessary conflict resolutions or other
semantic changes recorded in the merge commits being replayed), unless
those .  However, I do need to figure out some syntax for keeping
special_topic from being transplanted onto next here...

[PATCH v2 0/7] rebase: update branches in multi-part topic

[Derrick Stolee via GitGitGadget]

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'label' and 'update-refs'
commands into the TODO list. This is powered by the commit decoration
machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
label for-update-refs/refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
label for-update-refs/refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
label for-update-refs/refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

label for-update-refs/refs/heads/bundle-redo/advertise

update-refs


 * Patches 1-2 are basic refactors that we will need for our implementation.
 * Patch 3 is a cleanup of an array with indexes given by the todo_command
   enum.
 * Patches 4-6 implement the feature incrementally. For instance, we can
   test the edits to the TODO file in patch 5, but we don't actually do
   anything when seeing the 'update-refs' command until patch 6.
 * Patch 7 adds the rebase.updateRefs config option similar to
   rebase.autoSquash.


Updates in v2
=============

As recommended by the excellent feedback, I have removed the 'exec' commands
in favor of the 'label' commands and a new 'update-refs' command at the very
end. This way, there is only one step that updates all of the refs at the
end instead of updating refs during the rebase. If a user runs 'git rebase
--abort' in the middle, then their refs are still where they need to be.

Based on some of the discussion, it seemed like one way to do this would be
to have an 'update-ref ' command that would take the place of these 'label'
commands. However, this would require two things that make it a bit awkward:

 1. We would need to replicate the storage of those positions during the
    rebase. 'label' already does this pretty well. I've added the
    "for-update-refs/" label to help here.
 2. If we want to close out all of the refs as the rebase is finishing, then
    that "step" becomes invisible to the user (and a bit more complicated to
    insert). Thus, the 'update-refs' step performs this action. If the user
    wants to do things after that step, then they can do so by editing the
    TODO list.

Other updates:

 * The 'keep_decorations' parameter was renamed to 'update_refs'.
 * I added tests for --rebase-merges=rebase-cousins to show how these labels
   interact with other labels and merge commands.
 * I changed the order of the insertion of these update-refs labels to be
   before the fixups are rearranged. This fixes a bug where the tip commit
   is a fixup! so its decorations are never inspected (and they would be in
   the wrong place even if they were). The fixup! commands are properly
   inserted between a pick and its following label command. Tests
   demonstrate this is correct.
 * Numerous style choices are updated based on feedback.

Thank you for all of the detailed review and ideas in this space. I
appreciate any more ideas that can make this feature as effective as it can
be.

Thanks, -Stolee

Derrick Stolee (7):
  log-tree: create for_each_decoration()
  branch: add branch_checked_out() helper
  sequencer: define array with enum values
  sequencer: add update-refs command
  rebase: add --update-refs option
  sequencer: implement 'update-refs' command
  rebase: add rebase.updateRefs config option

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  12 ++
 branch.c                        |  24 ++--
 branch.h                        |   8 ++
 builtin/rebase.c                |  10 ++
 log-tree.c                      | 111 +++++++++++------
 log-tree.h                      |   4 +
 sequencer.c                     | 206 +++++++++++++++++++++++++++++---
 sequencer.h                     |   2 +
 t/t3404-rebase-interactive.sh   | 110 +++++++++++++++++
 10 files changed, 432 insertions(+), 58 deletions(-)


base-commit: 2668e3608e47494f2f10ef2b6e69f08a84816bcb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1247

Range-diff vs v1:

 1:  4f9f3487641 = 1:  4f9f3487641 log-tree: create for_each_decoration()
 2:  5f54766e103 = 2:  5f54766e103 branch: add branch_checked_out() helper
 -:  ----------- > 3:  9f261c7df2c sequencer: define array with enum values
 -:  ----------- > 4:  842b2186d25 sequencer: add update-refs command
 3:  23fa6638864 ! 5:  0a4c110127b rebase: add --update-refs option
     @@ Commit message
          reachable from those "sub branches". It can take a manual step to update
          those branches.
      
     -    Add a new --update-refs option to 'git rebase -i' that adds 'git
     -    update-ref' exec steps to the todo file whenever a commit that is being
     -    rebased is decorated with that <ref>. This allows the user to rebase a
     -    long list of commits in a multi-part feature and keep all of their
     -    pointers to those parts.
     +    Add a new --update-refs option to 'git rebase -i' that adds 'label
     +    for-update-refs/*' steps to the todo file whenever a commit that is
     +    being rebased is decorated with that <ref>. At the very end, the
     +    'update-refs' step is added to update all of the branches referenced by
     +    the 'label' steps. This allows the user to rebase a long list of commits
     +    in a multi-part feature and keep all of their pointers to those parts.
     +
     +    NOTE: This change only introduce the --update-refs option and implements
     +    the changes to the todo file. It does _not_ yet implement the action
     +    taken by the 'update-refs' todo step, which will be implemented and
     +    tested in a later change.
      
          Use the new for_each_decoration() while iterating over the existing todo
          list. Be sure to iterate after any squashing or fixups are placed.
     @@ Documentation/git-rebase.txt: provided. Otherwise an explicit `--no-reschedule-f
       INCOMPATIBLE OPTIONS
       --------------------
       
     +@@ Documentation/git-rebase.txt: are incompatible with the following options:
     +  * --empty=
     +  * --reapply-cherry-picks
     +  * --edit-todo
     ++ * --update-refs
     +  * --root when used in combination with --onto
     + 
     + In addition, the following pairs of options are incompatible:
      
       ## builtin/rebase.c ##
      @@ builtin/rebase.c: struct rebase_options {
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
       }
       
      +struct todo_add_branch_context {
     -+	struct todo_list new_list;
     ++	struct todo_item *items;
     ++	size_t items_nr;
     ++	size_t items_alloc;
      +	struct strbuf *buf;
      +	struct commit *commit;
      +};
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +{
      +	struct todo_add_branch_context *ctx = data;
      +	size_t base_offset = ctx->buf->len;
     -+	int i = ctx->new_list.nr;
      +	struct todo_item *item;
      +	char *path;
      +
     -+	ALLOC_GROW(ctx->new_list.items,
     -+		   ctx->new_list.nr + 1,
     -+		   ctx->new_list.alloc);
     -+	item = &ctx->new_list.items[i];
     ++	ALLOC_GROW(ctx->items,
     ++		   ctx->items_nr + 1,
     ++		   ctx->items_alloc);
     ++	item = &ctx->items[ctx->items_nr];
     ++	memset(item, 0, sizeof(*item));
      +
      +	/* If the branch is checked out, then leave a comment instead. */
      +	if (branch_checked_out(d->name, &path)) {
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +			    d->name, path);
      +		free(path);
      +	} else {
     -+		item->command = TODO_EXEC;
     -+		strbuf_addf(ctx->buf, "git update-ref %s HEAD %s\n",
     -+			    d->name, oid_to_hex(&ctx->commit->object.oid));
     ++		item->command = TODO_LABEL;
     ++		strbuf_addf(ctx->buf, "for-update-refs/%s\n", d->name);
      +	}
      +
     -+	item->commit = NULL;
      +	item->offset_in_buf = base_offset;
      +	item->arg_offset = base_offset;
      +	item->arg_len = ctx->buf->len - base_offset;
     -+	ctx->new_list.nr++;
     ++	ctx->items_nr++;
      +
      +	return 0;
      +}
      +
      +/*
      + * For each 'pick' command, find out if the commit has a decoration in
     -+ * refs/heads/. If so, then add a 'git branch -f' exec command after
     -+ * that 'pick' (plus any following 'squash' or 'fixup' commands).
     ++ * refs/heads/. If so, then add a 'label for-update-refs/' command.
      + */
     -+static int todo_list_add_branch_commands(struct todo_list *todo_list)
     ++static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
      +{
      +	int i;
      +	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +		.exclude_ref_config_pattern = &decorate_refs_exclude_config
      +	};
      +	struct todo_add_branch_context ctx = {
     -+		.new_list = TODO_LIST_INIT,
      +		.buf = &todo_list->buf,
      +	};
      +
     ++	ctx.items_alloc = 2 * todo_list->nr + 1;
     ++	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
     ++
      +	string_list_append(&decorate_refs_include, "refs/heads/");
      +	load_ref_decorations(&decoration_filter, 0);
      +
      +	for (i = 0; i < todo_list->nr; ) {
      +		struct todo_item *item = &todo_list->items[i];
      +
     -+		do {
     -+			/* insert ith item into new list */
     -+			ALLOC_GROW(ctx.new_list.items,
     -+				   ctx.new_list.nr + 1,
     -+				   ctx.new_list.alloc);
     ++		/* insert ith item into new list */
     ++		ALLOC_GROW(ctx.items,
     ++			   ctx.items_nr + 1,
     ++			   ctx.items_alloc);
      +
     -+			memcpy(&ctx.new_list.items[ctx.new_list.nr++],
     -+			       &todo_list->items[i],
     -+			       sizeof(struct todo_item));
     ++		ctx.items[ctx.items_nr++] = todo_list->items[i++];
      +
     -+			i++;
     -+		} while (i < todo_list->nr &&
     -+			 todo_list->items[i].command != TODO_PICK);
     -+
     -+		ctx.commit = item->commit;
     -+		for_each_decoration(item->commit, add_branch_for_decoration, &ctx);
     ++		if (item->commit) {
     ++			ctx.commit = item->commit;
     ++			for_each_decoration(item->commit,
     ++					    add_branch_for_decoration,
     ++					    &ctx);
     ++		}
      +	}
      +
     ++	/* Add the "update-refs" step. */
     ++	ALLOC_GROW(ctx.items,
     ++		   ctx.items_nr + 1,
     ++		   ctx.items_alloc);
     ++	memset(&ctx.items[ctx.items_nr], 0, sizeof(struct todo_item));
     ++	ctx.items[ctx.items_nr].command = TODO_UPDATE_REFS;
     ++	ctx.items_nr++;
     ++
      +	free(todo_list->items);
     -+	todo_list->items = ctx.new_list.items;
     -+	todo_list->nr = ctx.new_list.nr;
     -+	todo_list->alloc = ctx.new_list.alloc;
     ++	todo_list->items = ctx.items;
     ++	todo_list->nr = ctx.items_nr;
     ++	todo_list->alloc = ctx.items_alloc;
      +
      +	return 0;
      +}
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
       		    const char *shortrevisions, const char *onto_name,
       		    struct commit *onto, const struct object_id *orig_head,
       		    struct string_list *commands, unsigned autosquash,
     -+		    unsigned keep_decorations,
     ++		    unsigned update_refs,
       		    struct todo_list *todo_list)
       {
       	char shortonto[GIT_MAX_HEXSZ + 1];
      @@ sequencer.c: int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
     - 	if (autosquash && todo_list_rearrange_squash(todo_list))
     - 		return -1;
     + 		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
     + 	}
       
     -+	if (keep_decorations && todo_list_add_branch_commands(todo_list))
     ++	if (update_refs && todo_list_add_update_ref_commands(todo_list))
      +		return -1;
      +
     - 	if (commands->nr)
     - 		todo_list_add_exec_commands(todo_list, commands);
     + 	if (autosquash && todo_list_rearrange_squash(todo_list))
     + 		return -1;
       
      
       ## sequencer.h ##
     @@ sequencer.h: int complete_action(struct repository *r, struct replay_opts *opts,
       		    const char *shortrevisions, const char *onto_name,
       		    struct commit *onto, const struct object_id *orig_head,
       		    struct string_list *commands, unsigned autosquash,
     -+		    unsigned keep_decorations,
     ++		    unsigned update_refs,
       		    struct todo_list *todo_list);
       int todo_list_rearrange_squash(struct todo_list *todo_list);
       
     @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
       	test_cmp_rev ORIG_HEAD test-orig-head@{1}
       '
       
     -+test_expect_success '--update-refs adds git branch commands' '
     ++test_expect_success '--update-refs adds label and update-ref commands' '
      +	git checkout -b update-refs no-conflict-branch &&
     -+	test_commit extra fileX &&
     -+	git commit --amend --fixup=L &&
     ++	git branch -f base HEAD~4 &&
     ++	git branch -f first HEAD~3 &&
     ++	git branch -f second HEAD~3 &&
     ++	git branch -f third HEAD~1 &&
     ++	git commit --allow-empty --fixup=third &&
     ++	git branch -f shared-tip &&
      +	(
      +		set_cat_todo_editor &&
     -+		git branch -f base A &&
     -+		git branch -f first J &&
     -+		git branch -f second J &&
     -+		git branch -f third L &&
     -+
     -+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
      +
      +		cat >expect <<-EOF &&
      +		pick $(git log -1 --format=%h J) J
     -+		exec git update-ref refs/heads/second HEAD $(git rev-parse J)
     -+		exec git update-ref refs/heads/first HEAD $(git rev-parse  J)
     ++		label for-update-refs/refs/heads/second
     ++		label for-update-refs/refs/heads/first
      +		pick $(git log -1 --format=%h K) K
      +		pick $(git log -1 --format=%h L) L
     -+		fixup $(git log -1 --format=%h update-refs) fixup! L
     -+		exec git update-ref refs/heads/third HEAD $(git rev-parse L)
     ++		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
     ++		label for-update-refs/refs/heads/third
      +		pick $(git log -1 --format=%h M) M
     -+		exec git update-ref refs/heads/no-conflict-branch HEAD $(git rev-parse M)
     ++		label for-update-refs/refs/heads/no-conflict-branch
     ++		label for-update-refs/refs/heads/shared-tip
     ++		update-refs
      +		EOF
      +
     ++		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
     ++		test_cmp expect todo
     ++	)
     ++'
     ++
     ++test_expect_success '--update-refs adds commands with --rebase-merges' '
     ++	git checkout -b update-refs-with-merge no-conflict-branch &&
     ++	git branch -f base HEAD~4 &&
     ++	git branch -f first HEAD~3 &&
     ++	git branch -f second HEAD~3 &&
     ++	git branch -f third HEAD~1 &&
     ++	git merge -m merge branch2 &&
     ++	git branch -f merge-branch &&
     ++	git commit --fixup=third --allow-empty &&
     ++	(
     ++		set_cat_todo_editor &&
     ++
     ++		cat >expect <<-EOF &&
     ++		label onto
     ++		reset onto
     ++		pick $(git log -1 --format=%h branch2~1) F
     ++		pick $(git log -1 --format=%h branch2) I
     ++		label for-update-refs/refs/heads/branch2
     ++		label merge
     ++		reset onto
     ++		pick $(git log -1 --format=%h refs/heads/second) J
     ++		label for-update-refs/refs/heads/second
     ++		label for-update-refs/refs/heads/first
     ++		pick $(git log -1 --format=%h refs/heads/third~1) K
     ++		pick $(git log -1 --format=%h refs/heads/third) L
     ++		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
     ++		label for-update-refs/refs/heads/third
     ++		pick $(git log -1 --format=%h HEAD~2) M
     ++		label for-update-refs/refs/heads/no-conflict-branch
     ++		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
     ++		label for-update-refs/refs/heads/merge-branch
     ++		update-refs
     ++		EOF
     ++
     ++		test_must_fail git rebase -i --autosquash \
     ++				   --rebase-merges=rebase-cousins \
     ++				   --update-refs primary >todo &&
     ++
      +		test_cmp expect todo
      +	)
      +'
 -:  ----------- > 6:  68f8e51b19c sequencer: implement 'update-refs' command
 4:  b99c5bf34ef ! 7:  3d7d3f656b4 rebase: add rebase.updateRefs config option
     @@ builtin/rebase.c: static int rebase_config(const char *var, const char *value, v
       		return 0;
      
       ## t/t3404-rebase-interactive.sh ##
     -@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds git branch commands' '
     - 		exec git update-ref refs/heads/no-conflict-branch HEAD $(git rev-parse M)
     +@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds label and update-ref commands' '
       		EOF
       
     + 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
      +		test_cmp expect todo &&
      +
      +		test_must_fail git -c rebase.autosquash=true \
      +				   -c rebase.updaterefs=true \
      +				   rebase -i primary >todo &&
     ++
     + 		test_cmp expect todo
     + 	)
     + '
     +@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands with --rebase-merges' '
     + 				   --rebase-merges=rebase-cousins \
     + 				   --update-refs primary >todo &&
     + 
     ++		test_cmp expect todo &&
     ++
     ++		test_must_fail git -c rebase.autosquash=true \
     ++				   -c rebase.updaterefs=true \
     ++				   rebase -i \
     ++				   --rebase-merges=rebase-cousins \
     ++				   primary >todo &&
     ++
       		test_cmp expect todo
       	)
       '

-- 
gitgitgadget

[PATCH v2 1/7] log-tree: create for_each_decoration()

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

It can be helpful to iterate through all the decorations on a commit
without necessarily writing them to a stream. Implement
for_each_decoration() and reimplement format_decorations_extended() to
use that iterator.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 log-tree.c | 111 ++++++++++++++++++++++++++++++++++++-----------------
 log-tree.h |   4 ++
 2 files changed, 80 insertions(+), 35 deletions(-)

diff --git a/log-tree.c b/log-tree.c
index d0ac0a6327a..b15a7c9db22 100644
--- a/log-tree.c
+++ b/log-tree.c
@@ -282,6 +282,54 @@ static void show_name(struct strbuf *sb, const struct name_decoration *decoratio
 		strbuf_addstr(sb, decoration->name);
 }
 
+struct format_decorations_context {
+	struct strbuf *sb;
+	int use_color;
+	const char *prefix;
+	const char *separator;
+	const char *suffix;
+	const char *color_commit;
+	const char *color_reset;
+	const struct name_decoration *current_and_HEAD;
+};
+
+static int append_decoration(const struct name_decoration *d,
+			     void *data)
+{
+	struct format_decorations_context *ctx = data;
+	/*
+	 * When both current and HEAD are there, only
+	 * show HEAD->current where HEAD would have
+	 * appeared, skipping the entry for current.
+	 */
+	if (d != ctx->current_and_HEAD) {
+		strbuf_addstr(ctx->sb, ctx->color_commit);
+		strbuf_addstr(ctx->sb, ctx->prefix);
+		strbuf_addstr(ctx->sb, ctx->color_reset);
+		strbuf_addstr(ctx->sb, decorate_get_color(ctx->use_color, d->type));
+		if (d->type == DECORATION_REF_TAG)
+			strbuf_addstr(ctx->sb, "tag: ");
+
+		show_name(ctx->sb, d);
+
+		if (ctx->current_and_HEAD &&
+		    d->type == DECORATION_REF_HEAD) {
+			strbuf_addstr(ctx->sb, " -> ");
+			strbuf_addstr(ctx->sb, ctx->color_reset);
+			strbuf_addstr(ctx->sb,
+				      decorate_get_color(
+						ctx->use_color,
+						ctx->current_and_HEAD->type));
+			show_name(ctx->sb, ctx->current_and_HEAD);
+		}
+		strbuf_addstr(ctx->sb, ctx->color_reset);
+
+		ctx->prefix = ctx->separator;
+	}
+
+	return 0;
+}
+
 /*
  * The caller makes sure there is no funny color before calling.
  * format_decorations_extended makes sure the same after return.
@@ -294,49 +342,42 @@ void format_decorations_extended(struct strbuf *sb,
 			const char *suffix)
 {
 	const struct name_decoration *decoration;
-	const struct name_decoration *current_and_HEAD;
-	const char *color_commit =
-		diff_get_color(use_color, DIFF_COMMIT);
-	const char *color_reset =
-		decorate_get_color(use_color, DECORATION_NONE);
+	struct format_decorations_context ctx = {
+		.sb = sb,
+		.use_color = use_color,
+		.prefix = prefix,
+		.separator = separator,
+		.suffix = suffix,
+		.color_commit =	diff_get_color(use_color, DIFF_COMMIT),
+		.color_reset = decorate_get_color(use_color, DECORATION_NONE),
+	};
 
 	decoration = get_name_decoration(&commit->object);
 	if (!decoration)
 		return;
 
-	current_and_HEAD = current_pointed_by_HEAD(decoration);
-	while (decoration) {
-		/*
-		 * When both current and HEAD are there, only
-		 * show HEAD->current where HEAD would have
-		 * appeared, skipping the entry for current.
-		 */
-		if (decoration != current_and_HEAD) {
-			strbuf_addstr(sb, color_commit);
-			strbuf_addstr(sb, prefix);
-			strbuf_addstr(sb, color_reset);
-			strbuf_addstr(sb, decorate_get_color(use_color, decoration->type));
-			if (decoration->type == DECORATION_REF_TAG)
-				strbuf_addstr(sb, "tag: ");
-
-			show_name(sb, decoration);
-
-			if (current_and_HEAD &&
-			    decoration->type == DECORATION_REF_HEAD) {
-				strbuf_addstr(sb, " -> ");
-				strbuf_addstr(sb, color_reset);
-				strbuf_addstr(sb, decorate_get_color(use_color, current_and_HEAD->type));
-				show_name(sb, current_and_HEAD);
-			}
-			strbuf_addstr(sb, color_reset);
+	ctx.current_and_HEAD = current_pointed_by_HEAD(decoration);
 
-			prefix = separator;
-		}
+	for_each_decoration(commit, append_decoration, &ctx);
+
+	strbuf_addstr(sb, ctx.color_commit);
+	strbuf_addstr(sb, ctx.suffix);
+	strbuf_addstr(sb, ctx.color_reset);
+}
+
+int for_each_decoration(const struct commit *c, decoration_fn fn, void *data)
+{
+	const struct name_decoration *decoration;
+
+	decoration = get_name_decoration(&c->object);
+	while (decoration) {
+		int res;
+		if ((res = fn(decoration, data)))
+			return res;
 		decoration = decoration->next;
 	}
-	strbuf_addstr(sb, color_commit);
-	strbuf_addstr(sb, suffix);
-	strbuf_addstr(sb, color_reset);
+
+	return 0;
 }
 
 void show_decorations(struct rev_info *opt, struct commit *commit)
diff --git a/log-tree.h b/log-tree.h
index e7e4641cf83..ea07da2625b 100644
--- a/log-tree.h
+++ b/log-tree.h
@@ -35,4 +35,8 @@ void fmt_output_commit(struct strbuf *, struct commit *, struct rev_info *);
 void fmt_output_subject(struct strbuf *, const char *subject, struct rev_info *);
 void fmt_output_email_subject(struct strbuf *, struct rev_info *);
 
+typedef int decoration_fn(const struct name_decoration *d,
+			  void *data);
+int for_each_decoration(const struct commit *c, decoration_fn fn, void *data);
+
 #endif
-- 
gitgitgadget

[PATCH v2 2/7] branch: add branch_checked_out() helper

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The validate_new_branchname() method contains a check to see if a branch
is checked out in any non-bare worktree. This is intended to prevent a
force push that will mess up an existing checkout. This helper is not
suitable to performing just that check, because the method will die()
when the branch is checked out instead of returning an error code.

Extract branch_checked_out() and use it within
validate_new_branchname(). Another caller will be added in a coming
change.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 branch.c | 24 ++++++++++++++++--------
 branch.h |  8 ++++++++
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/branch.c b/branch.c
index 2d6569b0c62..2e6419cdfa5 100644
--- a/branch.c
+++ b/branch.c
@@ -369,6 +369,19 @@ int validate_branchname(const char *name, struct strbuf *ref)
 	return ref_exists(ref->buf);
 }
 
+int branch_checked_out(const char *refname, char **path)
+{
+	struct worktree **worktrees = get_worktrees();
+	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
+	int result = wt && !wt->is_bare;
+
+	if (result && path)
+		*path = xstrdup(wt->path);
+
+	free_worktrees(worktrees);
+	return result;
+}
+
 /*
  * Check if a branch 'name' can be created as a new branch; die otherwise.
  * 'force' can be used when it is OK for the named branch already exists.
@@ -377,9 +390,7 @@ int validate_branchname(const char *name, struct strbuf *ref)
  */
 int validate_new_branchname(const char *name, struct strbuf *ref, int force)
 {
-	struct worktree **worktrees;
-	const struct worktree *wt;
-
+	char *path;
 	if (!validate_branchname(name, ref))
 		return 0;
 
@@ -387,13 +398,10 @@ int validate_new_branchname(const char *name, struct strbuf *ref, int force)
 		die(_("a branch named '%s' already exists"),
 		    ref->buf + strlen("refs/heads/"));
 
-	worktrees = get_worktrees();
-	wt = find_shared_symref(worktrees, "HEAD", ref->buf);
-	if (wt && !wt->is_bare)
+	if (branch_checked_out(ref->buf, &path))
 		die(_("cannot force update the branch '%s' "
 		      "checked out at '%s'"),
-		    ref->buf + strlen("refs/heads/"), wt->path);
-	free_worktrees(worktrees);
+		    ref->buf + strlen("refs/heads/"), path);
 
 	return 1;
 }
diff --git a/branch.h b/branch.h
index 560b6b96a8f..5ea93d217b1 100644
--- a/branch.h
+++ b/branch.h
@@ -101,6 +101,14 @@ void create_branches_recursively(struct repository *r, const char *name,
 				 const char *tracking_name, int force,
 				 int reflog, int quiet, enum branch_track track,
 				 int dry_run);
+
+/*
+ * Returns true if the branch at 'refname' is checked out at any
+ * non-bare worktree. The path of the worktree is stored in the
+ * given 'path', if provided.
+ */
+int branch_checked_out(const char *refname, char **path);
+
 /*
  * Check if 'name' can be a valid name for a branch; die otherwise.
  * Return 1 if the named branch already exists; return 0 otherwise.
-- 
gitgitgadget

Re: [PATCH v2 2/7] branch: add branch_checked_out() helper

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +int branch_checked_out(const char *refname, char **path)
> +{
> +	struct worktree **worktrees = get_worktrees();
> +	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
> +	int result = wt && !wt->is_bare;
> +
> +	if (result && path)
> +		*path = xstrdup(wt->path);
> +
> +	free_worktrees(worktrees);
> +	return result;
> +}

Don't you plan to call this repeatedly from the for_each_deco
iteration?  I am wondering if it should take the result of
get_worktrees() and reuse the result of get_worktrees(), instead of
enumerating the list of worktrees and freeing for each of the
branches you need to inspect.

There also was another topic

https://lore.kernel.org/git/pull.1266.v2.git.git.1652690501963.gitgitgadget@gmail.com/

that was triggered by find_shared_symref() being relatively
heavy-weight, which suggests a more involved refactoring.

I wonder if we rather want to rewrite find_shared_symref() *not* to
take the target parameter at all, and instead introduce a new
function that takes a worktree, and report the branch that is
checked out (or being operated on via rebase or bisect).  Then we
can

 - create a strset out of its result, i.e. set of branches that
   should not be touched;

 - iterate over refs that point into the history being rebased
   (using for_each_decoration()), and consult that strset to see if
   any of them is being rewritten.

With the API of find_shared_symref(), we'd need to iterate over all
worktrees for each decoration.  With such a restructuring, we can
iterate over all worktrees just once, and match the result with
decoration, so the problem becomes O(N)+O(M) and not O(N*M) for
number of worktrees N and number of decorations M.

Re: [PATCH v2 2/7] branch: add branch_checked_out() helper

[Derrick Stolee]

On 6/7/2022 6:09 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> +int branch_checked_out(const char *refname, char **path)
>> +{
>> +	struct worktree **worktrees = get_worktrees();
>> +	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
>> +	int result = wt && !wt->is_bare;
>> +
>> +	if (result && path)
>> +		*path = xstrdup(wt->path);
>> +
>> +	free_worktrees(worktrees);
>> +	return result;
>> +}
> 
> Don't you plan to call this repeatedly from the for_each_deco
> iteration?  I am wondering if it should take the result of
> get_worktrees() and reuse the result of get_worktrees(), instead of
> enumerating the list of worktrees and freeing for each of the
> branches you need to inspect.

Sorry, you had mentioned this in v1 and I just forgot to fix this.

> There also was another topic
> 
> https://lore.kernel.org/git/pull.1266.v2.git.git.1652690501963.gitgitgadget@gmail.com/
> 
> that was triggered by find_shared_symref() being relatively
> heavy-weight, which suggests a more involved refactoring.
> 
> I wonder if we rather want to rewrite find_shared_symref() *not* to
> take the target parameter at all, and instead introduce a new
> function that takes a worktree, and report the branch that is
> checked out (or being operated on via rebase or bisect).  Then we
> can
> 
>  - create a strset out of its result, i.e. set of branches that
>    should not be touched;
> 
>  - iterate over refs that point into the history being rebased
>    (using for_each_decoration()), and consult that strset to see if
>    any of them is being rewritten.
> 
> With the API of find_shared_symref(), we'd need to iterate over all
> worktrees for each decoration.  With such a restructuring, we can
> iterate over all worktrees just once, and match the result with
> decoration, so the problem becomes O(N)+O(M) and not O(N*M) for
> number of worktrees N and number of decorations M.

Yes, that's a good plan. I'll take a look.

Thanks,
-Stolee

Re: [PATCH v2 2/7] branch: add branch_checked_out() helper

[Derrick Stolee]

On 6/7/2022 10:14 PM, Derrick Stolee wrote:
> On 6/7/2022 6:09 PM, Junio C Hamano wrote:
>>
>> I wonder if we rather want to rewrite find_shared_symref() *not* to
>> take the target parameter at all, and instead introduce a new
>> function that takes a worktree, and report the branch that is
>> checked out (or being operated on via rebase or bisect).  Then we
>> can
>>
>>  - create a strset out of its result, i.e. set of branches that
>>    should not be touched;
>>
>>  - iterate over refs that point into the history being rebased
>>    (using for_each_decoration()), and consult that strset to see if
>>    any of them is being rewritten.
>>
>> With the API of find_shared_symref(), we'd need to iterate over all
>> worktrees for each decoration.  With such a restructuring, we can
>> iterate over all worktrees just once, and match the result with
>> decoration, so the problem becomes O(N)+O(M) and not O(N*M) for
>> number of worktrees N and number of decorations M.
> 
> Yes, that's a good plan. I'll take a look.

Here is a fixup for this patch that should work. I'll put it in v3.

diff --git a/branch.c b/branch.c
index 2e6419cdfa5..514212f5619 100644
--- a/branch.c
+++ b/branch.c
@@ -10,6 +10,7 @@
 #include "worktree.h"
 #include "submodule-config.h"
 #include "run-command.h"
+#include "strmap.h"
 
 struct tracking {
 	struct refspec_item spec;
@@ -369,17 +370,44 @@ int validate_branchname(const char *name, struct strbuf *ref)
 	return ref_exists(ref->buf);
 }
 
-int branch_checked_out(const char *refname, char **path)
+static int initialized_checked_out_branches;
+static struct strmap current_checked_out_branches = STRMAP_INIT;
+
+static void prepare_checked_out_branches(void)
 {
-	struct worktree **worktrees = get_worktrees();
-	const struct worktree *wt = find_shared_symref(worktrees, "HEAD", refname);
-	int result = wt && !wt->is_bare;
+	int i = 0;
+	struct worktree **worktrees;
+
+	if (initialized_checked_out_branches)
+		return;
+	initialized_checked_out_branches = 1;
+
+	worktrees = get_worktrees();
+
+	while (worktrees[i]) {
+		struct worktree *wt = worktrees[i];
 
-	if (result && path)
-		*path = xstrdup(wt->path);
+		if (!wt->is_bare && wt->head_ref)
+			strmap_put(&current_checked_out_branches,
+				   wt->head_ref,
+				   wt->path);
+
+		i++;
+	}
 
 	free_worktrees(worktrees);
-	return result;
+}
+
+int branch_checked_out(const char *refname, char **path)
+{
+	const char *path_in_set;
+	prepare_checked_out_branches();
+
+	path_in_set = strmap_get(&current_checked_out_branches, refname);
+	if (path_in_set && path)
+		*path = xstrdup(path_in_set);
+
+	return !!path_in_set;
 }
 
 /*

>> There also was another topic
>>
>> https://lore.kernel.org/git/pull.1266.v2.git.git.1652690501963.gitgitgadget@gmail.com/
>>
>> that was triggered by find_shared_symref() being relatively
>> heavy-weight, which suggests a more involved refactoring.

The patch there was this:

diff --git a/builtin/fetch.c b/builtin/fetch.c
index e3791f09ed5..eeee5ac8f15 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1440,6 +1440,7 @@ static void check_not_current_branch(struct ref *ref_map,
 	const struct worktree *wt;
 	for (; ref_map; ref_map = ref_map->next)
 		if (ref_map->peer_ref &&
+		    starts_with(ref_map->peer_ref->name, "refs/heads/") &&
 		    (wt = find_shared_symref(worktrees, "HEAD",
 					     ref_map->peer_ref->name)) &&
 		    !wt->is_bare)

And there is another use of find_shared_symref() in the same file, allowing
us to do the following:

diff --git a/builtin/fetch.c b/builtin/fetch.c
index ac29c2b1ae3..3933c482839 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -885,7 +885,7 @@ static int update_local_ref(struct ref *ref,
 			    struct worktree **worktrees)
 {
 	struct commit *current = NULL, *updated;
-	const struct worktree *wt;
+	char *path = NULL;
 	const char *pretty_ref = prettify_refname(ref->name);
 	int fast_forward = 0;
 
@@ -900,17 +900,17 @@ static int update_local_ref(struct ref *ref,
 	}
 
 	if (!update_head_ok &&
-	    (wt = find_shared_symref(worktrees, "HEAD", ref->name)) &&
-	    !wt->is_bare && !is_null_oid(&ref->old_oid)) {
+	    !is_null_oid(&ref->old_oid) &&
+	    branch_checked_out(ref->name, &path)) {
 		/*
 		 * If this is the head, and it's not okay to update
 		 * the head, and the old value of the head isn't empty...
 		 */
 		format_display(display, '!', _("[rejected]"),
-			       wt->is_current ?
-				       _("can't fetch in current branch") :
-				       _("checked out in another worktree"),
+			       path ? _("can't fetch in current branch") :
+				      _("checked out in another worktree"),
 			       remote, pretty_ref, summary_width);
+		free(path);
 		return 1;
 	}
 
@@ -1437,16 +1437,14 @@ static int prune_refs(struct refspec *rs,
 static void check_not_current_branch(struct ref *ref_map,
 				     struct worktree **worktrees)
 {
-	const struct worktree *wt;
+	char *path;
 	for (; ref_map; ref_map = ref_map->next)
 		if (ref_map->peer_ref &&
 		    starts_with(ref_map->peer_ref->name, "refs/heads/") &&
-		    (wt = find_shared_symref(worktrees, "HEAD",
-					     ref_map->peer_ref->name)) &&
-		    !wt->is_bare)
+		    branch_checked_out(ref_map->peer_ref->name, &path))
 			die(_("refusing to fetch into branch '%s' "
 			      "checked out at '%s'"),
-			    ref_map->peer_ref->name, wt->path);
+			    ref_map->peer_ref->name, path);
 }
 
 static int truncate_fetch_head(void)

I can extract these as patches in their own series if we think
that's something we worth fast-tracking.

Thanks,
-Stolee

Re: [PATCH v2 2/7] branch: add branch_checked_out() helper

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

>> Yes, that's a good plan. I'll take a look.
>
> Here is a fixup for this patch that should work. I'll put it in v3.
>
> diff --git a/branch.c b/branch.c
> index 2e6419cdfa5..514212f5619 100644
> --- a/branch.c
> +++ b/branch.c
> @@ -10,6 +10,7 @@
>  #include "worktree.h"
>  #include "submodule-config.h"
>  #include "run-command.h"
> +#include "strmap.h"
>  
>  struct tracking {
>  	struct refspec_item spec;
> @@ -369,17 +370,44 @@ int validate_branchname(const char *name, struct strbuf *ref)
>  	return ref_exists(ref->buf);
>  }
>  
> -int branch_checked_out(const char *refname, char **path)
> +static int initialized_checked_out_branches;
> +static struct strmap current_checked_out_branches = STRMAP_INIT;
> +
> +static void prepare_checked_out_branches(void)
>  {
> +	int i = 0;
> +	struct worktree **worktrees;
> +
> +	if (initialized_checked_out_branches)
> +		return;
> +	initialized_checked_out_branches = 1;
> +
> +	worktrees = get_worktrees();
> +
> +	while (worktrees[i]) {
> +		struct worktree *wt = worktrees[i];
>  
> +		if (!wt->is_bare && wt->head_ref)
> +			strmap_put(&current_checked_out_branches,
> +				   wt->head_ref,
> +				   wt->path);
> +
> +		i++;
> +	}
>  	free_worktrees(worktrees);
> -	return result;
> +}

Yeah, the above illustrates what I had in mind pretty closely.  The
above outline only checks where the HEAD symref points at, and it
does not protect a branch that is in the middle of getting bisected
or rebased, which find_shared_symref() tries to do, IIRC, though.

It should not be too hard to add that to the above to allow us to
achieve feature parity with the original.

> +
> +int branch_checked_out(const char *refname, char **path)
> +{
> +	const char *path_in_set;
> +	prepare_checked_out_branches();
> +
> +	path_in_set = strmap_get(&current_checked_out_branches, refname);
> +	if (path_in_set && path)
> +		*path = xstrdup(path_in_set);
> +
> +	return !!path_in_set;
>  }

This one looks quite straight-forward.

> And there is another use of find_shared_symref() in the same file, allowing
> us to do the following:
>
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index ac29c2b1ae3..3933c482839 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -885,7 +885,7 @@ static int update_local_ref(struct ref *ref,
>  			    struct worktree **worktrees)
>  {
>  	struct commit *current = NULL, *updated;
> -	const struct worktree *wt;
> +	char *path = NULL;
>  	const char *pretty_ref = prettify_refname(ref->name);
>  	int fast_forward = 0;
>  
> @@ -900,17 +900,17 @@ static int update_local_ref(struct ref *ref,
>  	}
>  
>  	if (!update_head_ok &&
> -	    (wt = find_shared_symref(worktrees, "HEAD", ref->name)) &&
> -	    !wt->is_bare && !is_null_oid(&ref->old_oid)) {
> +	    !is_null_oid(&ref->old_oid) &&
> +	    branch_checked_out(ref->name, &path)) {

Yes.  It looks like a good direction to go in.

Thanks.

[PATCH v2 3/7] sequencer: define array with enum values

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The todo_command_info array defines which strings match with which
todo_command enum values. The array is defined in the same order as the
enum values, but if one changed without the other, then we would have
unexpected results.

Make it easier to see changes to the enum and this array by using the
enum values as the indices of the array.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 8c3ed3532ac..8e26c9a6261 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1685,20 +1685,20 @@ static struct {
 	char c;
 	const char *str;
 } todo_command_info[] = {
-	{ 'p', "pick" },
-	{ 0,   "revert" },
-	{ 'e', "edit" },
-	{ 'r', "reword" },
-	{ 'f', "fixup" },
-	{ 's', "squash" },
-	{ 'x', "exec" },
-	{ 'b', "break" },
-	{ 'l', "label" },
-	{ 't', "reset" },
-	{ 'm', "merge" },
-	{ 0,   "noop" },
-	{ 'd', "drop" },
-	{ 0,   NULL }
+	[TODO_PICK] = { 'p', "pick" },
+	[TODO_REVERT] = { 0,   "revert" },
+	[TODO_EDIT] = { 'e', "edit" },
+	[TODO_REWORD] = { 'r', "reword" },
+	[TODO_FIXUP] = { 'f', "fixup" },
+	[TODO_SQUASH] = { 's', "squash" },
+	[TODO_EXEC] = { 'x', "exec" },
+	[TODO_BREAK] = { 'b', "break" },
+	[TODO_LABEL] = { 'l', "label" },
+	[TODO_RESET] = { 't', "reset" },
+	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_NOOP] = { 0,   "noop" },
+	[TODO_DROP] = { 'd', "drop" },
+	[TODO_COMMENT] = { 0,   NULL },
 };
 
 static const char *command_to_string(const enum todo_command command)
-- 
gitgitgadget

Re: [PATCH v2 3/7] sequencer: define array with enum values

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The todo_command_info array defines which strings match with which
> todo_command enum values. The array is defined in the same order as the
> enum values, but if one changed without the other, then we would have
> unexpected results.
>
> Make it easier to see changes to the enum and this array by using the
> enum values as the indices of the array.
>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  sequencer.c | 28 ++++++++++++++--------------
>  1 file changed, 14 insertions(+), 14 deletions(-)

Good thinking.

I love seeing a future-proofing change like this.

Thanks.

> diff --git a/sequencer.c b/sequencer.c
> index 8c3ed3532ac..8e26c9a6261 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -1685,20 +1685,20 @@ static struct {
>  	char c;
>  	const char *str;
>  } todo_command_info[] = {
> -	{ 'p', "pick" },
> -	{ 0,   "revert" },
> -	{ 'e', "edit" },
> -	{ 'r', "reword" },
> -	{ 'f', "fixup" },
> -	{ 's', "squash" },
> -	{ 'x', "exec" },
> -	{ 'b', "break" },
> -	{ 'l', "label" },
> -	{ 't', "reset" },
> -	{ 'm', "merge" },
> -	{ 0,   "noop" },
> -	{ 'd', "drop" },
> -	{ 0,   NULL }
> +	[TODO_PICK] = { 'p', "pick" },
> +	[TODO_REVERT] = { 0,   "revert" },
> +	[TODO_EDIT] = { 'e', "edit" },
> +	[TODO_REWORD] = { 'r', "reword" },
> +	[TODO_FIXUP] = { 'f', "fixup" },
> +	[TODO_SQUASH] = { 's', "squash" },
> +	[TODO_EXEC] = { 'x', "exec" },
> +	[TODO_BREAK] = { 'b', "break" },
> +	[TODO_LABEL] = { 'l', "label" },
> +	[TODO_RESET] = { 't', "reset" },
> +	[TODO_MERGE] = { 'm', "merge" },
> +	[TODO_NOOP] = { 0,   "noop" },
> +	[TODO_DROP] = { 'd', "drop" },
> +	[TODO_COMMENT] = { 0,   NULL },
>  };
>  
>  static const char *command_to_string(const enum todo_command command)

[PATCH v2 4/7] sequencer: add update-refs command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

Add the boilerplat for an "update-refs" command in the sequencer. This
connects to the current no-op do_update_refs() which will be filled in
after more connections are created.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 13 ++++++++++++-
 sequencer.h |  1 +
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/sequencer.c b/sequencer.c
index 8e26c9a6261..68f7c76e896 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1696,6 +1696,7 @@ static struct {
 	[TODO_LABEL] = { 'l', "label" },
 	[TODO_RESET] = { 't', "reset" },
 	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_UPDATE_REFS] = { 'u', "update-refs" },
 	[TODO_NOOP] = { 0,   "noop" },
 	[TODO_DROP] = { 'd', "drop" },
 	[TODO_COMMENT] = { 0,   NULL },
@@ -2442,7 +2443,9 @@ static int parse_insn_line(struct repository *r, struct todo_item *item,
 	padding = strspn(bol, " \t");
 	bol += padding;
 
-	if (item->command == TODO_NOOP || item->command == TODO_BREAK) {
+	if (item->command == TODO_NOOP ||
+	    item->command == TODO_BREAK ||
+	    item->command == TODO_UPDATE_REFS) {
 		if (bol != eol)
 			return error(_("%s does not accept arguments: '%s'"),
 				     command_to_string(item->command), bol);
@@ -4056,6 +4059,11 @@ leave_merge:
 	return ret;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	return 0;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4431,6 +4439,9 @@ static int pick_commits(struct repository *r,
 				return error_with_patch(r, item->commit,
 							arg, item->arg_len,
 							opts, res, 0);
+		} else if (item->command == TODO_UPDATE_REFS) {
+			if ((res = do_update_refs(r)))
+				reschedule = 1;
 		} else if (!is_noop(item->command))
 			return error(_("unknown command %d"), item->command);
 
diff --git a/sequencer.h b/sequencer.h
index da64473636b..c2b4e148d8f 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -95,6 +95,7 @@ enum todo_command {
 	TODO_LABEL,
 	TODO_RESET,
 	TODO_MERGE,
+	TODO_UPDATE_REFS,
 	/* commands that do nothing but are counted for reporting progress */
 	TODO_NOOP,
 	TODO_DROP,
-- 
gitgitgadget

[PATCH v2 5/7] rebase: add --update-refs option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When working on a large feature, it can be helpful to break that feature
into multiple smaller parts that become reviewed in sequence. During
development or during review, a change to one part of the feature could
affect multiple of these parts. An interactive rebase can help adjust
the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the
feature, then rebasing the entire list of commits can create commits not
reachable from those "sub branches". It can take a manual step to update
those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'label
for-update-refs/*' steps to the todo file whenever a commit that is
being rebased is decorated with that <ref>. At the very end, the
'update-refs' step is added to update all of the branches referenced by
the 'label' steps. This allows the user to rebase a long list of commits
in a multi-part feature and keep all of their pointers to those parts.

NOTE: This change only introduce the --update-refs option and implements
the changes to the todo file. It does _not_ yet implement the action
taken by the 'update-refs' todo step, which will be implemented and
tested in a later change.

Use the new for_each_decoration() while iterating over the existing todo
list. Be sure to iterate after any squashing or fixups are placed.
Update the branch only after those squashes and fixups are complete.
This allows a --fixup commit at the tip of the feature to apply
correctly to the sub branch, even if it is fixing up the most-recent
commit in that part.

One potential problem here is that refs decorating commits that are
already marked as "fixup!" or "squash!" will not be included in this
list. Generally, the reordering of the "fixup!" and "squash!" is likely
to change the relative order of these refs, so it is not recommended.
The workflow here is intended to allow these kinds of commits at the tip
of the rebased branch while the other sub branches come along for the
ride without intervention.

Be careful to not attempt updating any branch that is checked out. The
most common example is the branch being rebased is checked out and
decorates the tip commit. If the user is rebasing commits reachable from
a different branch that is checked out in a different worktree, then
they may be surprised to not see that ref update. However, it's probably
best to not optimize for this scenario and do the safest thing that will
result in a successful rebase. A comment is left in the TODO list that
signals that these refs are currently checked out.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-rebase.txt  |   8 +++
 builtin/rebase.c              |   5 ++
 sequencer.c                   | 105 ++++++++++++++++++++++++++++++++++
 sequencer.h                   |   1 +
 t/t3404-rebase-interactive.sh |  72 +++++++++++++++++++++++
 5 files changed, 191 insertions(+)

diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 262fb01aec0..e7611b4089c 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	or point to a `squash! ...` or `fixup! ...` commit are not updated
+	in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
@@ -632,6 +639,7 @@ are incompatible with the following options:
  * --empty=
  * --reapply-cherry-picks
  * --edit-todo
+ * --update-refs
  * --root when used in combination with --onto
 
 In addition, the following pairs of options are incompatible:
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 7ab50cda2ad..56d82a52106 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update local refs that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
diff --git a/sequencer.c b/sequencer.c
index 68f7c76e896..94f8d52e041 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -35,6 +35,8 @@
 #include "commit-reach.h"
 #include "rebase-interactive.h"
 #include "reset.h"
+#include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -5614,10 +5616,110 @@ static int skip_unnecessary_picks(struct repository *r,
 	return 0;
 }
 
+struct todo_add_branch_context {
+	struct todo_item *items;
+	size_t items_nr;
+	size_t items_alloc;
+	struct strbuf *buf;
+	struct commit *commit;
+};
+
+static int add_branch_for_decoration(const struct name_decoration *d, void *data)
+{
+	struct todo_add_branch_context *ctx = data;
+	size_t base_offset = ctx->buf->len;
+	struct todo_item *item;
+	char *path;
+
+	ALLOC_GROW(ctx->items,
+		   ctx->items_nr + 1,
+		   ctx->items_alloc);
+	item = &ctx->items[ctx->items_nr];
+	memset(item, 0, sizeof(*item));
+
+	/* If the branch is checked out, then leave a comment instead. */
+	if (branch_checked_out(d->name, &path)) {
+		item->command = TODO_COMMENT;
+		strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
+			    d->name, path);
+		free(path);
+	} else {
+		item->command = TODO_LABEL;
+		strbuf_addf(ctx->buf, "for-update-refs/%s\n", d->name);
+	}
+
+	item->offset_in_buf = base_offset;
+	item->arg_offset = base_offset;
+	item->arg_len = ctx->buf->len - base_offset;
+	ctx->items_nr++;
+
+	return 0;
+}
+
+/*
+ * For each 'pick' command, find out if the commit has a decoration in
+ * refs/heads/. If so, then add a 'label for-update-refs/' command.
+ */
+static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
+{
+	int i;
+	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
+	struct decoration_filter decoration_filter = {
+		.include_ref_pattern = &decorate_refs_include,
+		.exclude_ref_pattern = &decorate_refs_exclude,
+		.exclude_ref_config_pattern = &decorate_refs_exclude_config
+	};
+	struct todo_add_branch_context ctx = {
+		.buf = &todo_list->buf,
+	};
+
+	ctx.items_alloc = 2 * todo_list->nr + 1;
+	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
+
+	string_list_append(&decorate_refs_include, "refs/heads/");
+	load_ref_decorations(&decoration_filter, 0);
+
+	for (i = 0; i < todo_list->nr; ) {
+		struct todo_item *item = &todo_list->items[i];
+
+		/* insert ith item into new list */
+		ALLOC_GROW(ctx.items,
+			   ctx.items_nr + 1,
+			   ctx.items_alloc);
+
+		ctx.items[ctx.items_nr++] = todo_list->items[i++];
+
+		if (item->commit) {
+			ctx.commit = item->commit;
+			for_each_decoration(item->commit,
+					    add_branch_for_decoration,
+					    &ctx);
+		}
+	}
+
+	/* Add the "update-refs" step. */
+	ALLOC_GROW(ctx.items,
+		   ctx.items_nr + 1,
+		   ctx.items_alloc);
+	memset(&ctx.items[ctx.items_nr], 0, sizeof(struct todo_item));
+	ctx.items[ctx.items_nr].command = TODO_UPDATE_REFS;
+	ctx.items_nr++;
+
+	free(todo_list->items);
+	todo_list->items = ctx.items;
+	todo_list->nr = ctx.items_nr;
+	todo_list->alloc = ctx.items_alloc;
+
+	return 0;
+}
+
 int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list)
 {
 	char shortonto[GIT_MAX_HEXSZ + 1];
@@ -5636,6 +5738,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
 	}
 
+	if (update_refs && todo_list_add_update_ref_commands(todo_list))
+		return -1;
+
 	if (autosquash && todo_list_rearrange_squash(todo_list))
 		return -1;
 
diff --git a/sequencer.h b/sequencer.h
index c2b4e148d8f..e268208b315 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index f31afd4a547..5e99ad7f3b6 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1743,6 +1743,78 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
 	test_cmp_rev ORIG_HEAD test-orig-head@{1}
 '
 
+test_expect_success '--update-refs adds label and update-ref commands' '
+	git checkout -b update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git commit --allow-empty --fixup=third &&
+	git branch -f shared-tip &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		pick $(git log -1 --format=%h J) J
+		label for-update-refs/refs/heads/second
+		label for-update-refs/refs/heads/first
+		pick $(git log -1 --format=%h K) K
+		pick $(git log -1 --format=%h L) L
+		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
+		label for-update-refs/refs/heads/third
+		pick $(git log -1 --format=%h M) M
+		label for-update-refs/refs/heads/no-conflict-branch
+		label for-update-refs/refs/heads/shared-tip
+		update-refs
+		EOF
+
+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo
+	)
+'
+
+test_expect_success '--update-refs adds commands with --rebase-merges' '
+	git checkout -b update-refs-with-merge no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git merge -m merge branch2 &&
+	git branch -f merge-branch &&
+	git commit --fixup=third --allow-empty &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		label onto
+		reset onto
+		pick $(git log -1 --format=%h branch2~1) F
+		pick $(git log -1 --format=%h branch2) I
+		label for-update-refs/refs/heads/branch2
+		label merge
+		reset onto
+		pick $(git log -1 --format=%h refs/heads/second) J
+		label for-update-refs/refs/heads/second
+		label for-update-refs/refs/heads/first
+		pick $(git log -1 --format=%h refs/heads/third~1) K
+		pick $(git log -1 --format=%h refs/heads/third) L
+		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
+		label for-update-refs/refs/heads/third
+		pick $(git log -1 --format=%h HEAD~2) M
+		label for-update-refs/refs/heads/no-conflict-branch
+		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
+		label for-update-refs/refs/heads/merge-branch
+		update-refs
+		EOF
+
+		test_must_fail git rebase -i --autosquash \
+				   --rebase-merges=rebase-cousins \
+				   --update-refs primary >todo &&
+
+		test_cmp expect todo
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

[PATCH v2 6/7] sequencer: implement 'update-refs' command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change allowed 'git rebase --update-refs' to create 'label'
commands for each branch  among the commits being rewritten and add an
'update-refs' command at the end of the todo list. Now, teach Git to
update the refs during that final 'update-refs' command.

We need to create an array of new and old OIDs for each ref by iterating
over the refs/rewritten/for-update-refs/ namespace. We cannot update the
refs in-place since this will confuse the refs iterator.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 62 ++++++++++++++++++++++++++++++++++-
 t/t3404-rebase-interactive.sh | 24 ++++++++++++++
 2 files changed, 85 insertions(+), 1 deletion(-)

diff --git a/sequencer.c b/sequencer.c
index 94f8d52e041..a8f62ce8e5f 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -4061,11 +4061,71 @@ leave_merge:
 	return ret;
 }
 
-static int do_update_refs(struct repository *r)
+struct update_refs_context {
+	struct ref_store *refs;
+	char **ref_names;
+	struct object_id *old;
+	struct object_id *new;
+	size_t nr;
+	size_t alloc;
+};
+
+static int add_ref_to_context(const char *refname,
+			      const struct object_id *oid,
+			      int flags,
+			      void *data)
 {
+	int f = 0;
+	const char *name;
+	struct update_refs_context *ctx = data;
+
+	ALLOC_GROW(ctx->ref_names, ctx->nr + 1, ctx->alloc);
+	ALLOC_GROW(ctx->old, ctx->nr + 1, ctx->alloc);
+	ALLOC_GROW(ctx->new, ctx->nr + 1, ctx->alloc);
+
+	if (!skip_prefix(refname, "refs/rewritten/for-update-refs/", &name))
+		return 1;
+
+	ctx->ref_names[ctx->nr] = xstrdup(name);
+	oidcpy(&ctx->new[ctx->nr], oid);
+	if (!refs_resolve_ref_unsafe(ctx->refs, name, 0,
+				     &ctx->old[ctx->nr], &f))
+		return 1;
+
+	ctx->nr++;
 	return 0;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	int i, res;
+	struct update_refs_context ctx = {
+		.refs = get_main_ref_store(r),
+		.alloc = 16,
+	};
+	ALLOC_ARRAY(ctx.ref_names, ctx.alloc);
+	ALLOC_ARRAY(ctx.old, ctx.alloc);
+	ALLOC_ARRAY(ctx.new, ctx.alloc);
+
+	res = refs_for_each_fullref_in(ctx.refs,
+				       "refs/rewritten/for-update-refs/",
+				       add_ref_to_context,
+				       &ctx);
+
+	for (i = 0; !res && i < ctx.nr; i++)
+		res = refs_update_ref(ctx.refs, "rewritten during rebase",
+				ctx.ref_names[i],
+				&ctx.new[i], &ctx.old[i],
+				0, UPDATE_REFS_MSG_ON_ERR);
+
+	for (i = 0; i < ctx.nr; i++)
+		free(ctx.ref_names[i]);
+	free(ctx.ref_names);
+	free(ctx.old);
+	free(ctx.new);
+	return res;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 5e99ad7f3b6..72711efec28 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1815,6 +1815,30 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 	)
 '
 
+compare_two_refs () {
+	git rev-parse $1 >expect &&
+	git rev-parse $2 >actual &&
+	test_cmp expect actual
+}
+
+test_expect_success '--update-refs updates refs correctly' '
+	git checkout -B update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	test_commit extra2 fileX &&
+	git commit --amend --fixup=L &&
+	(
+		git rebase -i --autosquash --update-refs primary &&
+
+		compare_two_refs HEAD~3 refs/heads/first &&
+		compare_two_refs HEAD~3 refs/heads/second &&
+		compare_two_refs HEAD~1 refs/heads/third &&
+		compare_two_refs HEAD refs/heads/no-conflict-branch
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v2 6/7] sequencer: implement 'update-refs' command

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The previous change allowed 'git rebase --update-refs' to create 'label'
> commands for each branch  among the commits being rewritten and add an
> 'update-refs' command at the end of the todo list. Now, teach Git to
> update the refs during that final 'update-refs' command.
>
> We need to create an array of new and old OIDs for each ref by iterating
> over the refs/rewritten/for-update-refs/ namespace. We cannot update the
> refs in-place since this will confuse the refs iterator.

In other words, grab everything we need to do and then use that
in-core information to tell refs API what refs to change to what
values?  That dounds like a quite reasonable thing to do.

Looking at the patch text, the only thing that stands out at me is
that this does not seem to perform the updates in a single
transaction, which may often not matter in practice, but may be a
prudent thing to do anyway at philosophical level.

Thanks, will queue.

[PATCH v2 7/7] rebase: add rebase.updateRefs config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change added the --update-refs command-line option.  For
users who always want this mode, create the rebase.updateRefs config
option which behaves the same way as rebase.autoSquash does with the
--autosquash option.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config/rebase.txt |  3 +++
 Documentation/git-rebase.txt    |  4 ++++
 builtin/rebase.c                |  5 +++++
 t/t3404-rebase-interactive.sh   | 14 ++++++++++++++
 4 files changed, 26 insertions(+)

diff --git a/Documentation/config/rebase.txt b/Documentation/config/rebase.txt
index 8c979cb20f2..f19bd0e0407 100644
--- a/Documentation/config/rebase.txt
+++ b/Documentation/config/rebase.txt
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index e7611b4089c..7a56dbb9bec 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -615,6 +615,10 @@ start would be overridden by the presence of
 	are being rebased. Any branches that are checked out in a worktree
 	or point to a `squash! ...` or `fixup! ...` commit are not updated
 	in this way.
++
+If the `--update-refs` option is enabled by default using the
+configuration variable `rebase.updateRefs`, this option can be
+used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 56d82a52106..8ebc98ea505 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 72711efec28..f580afd8723 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1769,6 +1769,12 @@ test_expect_success '--update-refs adds label and update-ref commands' '
 		EOF
 
 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
+
 		test_cmp expect todo
 	)
 '
@@ -1811,6 +1817,14 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 				   --rebase-merges=rebase-cousins \
 				   --update-refs primary >todo &&
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i \
+				   --rebase-merges=rebase-cousins \
+				   primary >todo &&
+
 		test_cmp expect todo
 	)
 '
-- 
gitgitgadget

Re: [PATCH v2 0/7] rebase: update branches in multi-part topic

[Phillip Wood]

Hi Stolee

On 07/06/2022 21:42, Derrick Stolee via GitGitGadget wrote:
> [...]
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
> 
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> label for-update-refs/refs/heads/bundle-redo/fetch
> [...] 
> update-refs
> [...]
> Based on some of the discussion, it seemed like one way to do this would be
> to have an 'update-ref ' command that would take the place of these 'label'
> commands. However, this would require two things that make it a bit awkward:
> 
>   1. We would need to replicate the storage of those positions during the
>      rebase. 'label' already does this pretty well. I've added the
>      "for-update-refs/" label to help here.

I'm afraid I don't think that using a label with a name constructed from 
a magic prefix and the full refname is a good user interface.

(i) Having labels behave differently based on their name is confusing.

(ii) The length of the label string is cumbersome for users. Rebase 
already has a reputation for being unfriendly and hard to use, this will 
not improve that. "update-ref bundle-redo/fetch" is much simpler.

(iii) It means that we no longer store the oid of each branch when 
creating the todo list and so cannot check if it is safe to update it at 
the end of the rebase (just using the current value of the branch ref at 
the end of a long running operation like rebase is not sufficient to be 
safe).

>   2. If we want to close out all of the refs as the rebase is finishing, then
>      that "step" becomes invisible to the user (and a bit more complicated to
>      insert). Thus, the 'update-refs' step performs this action. If the user
>      wants to do things after that step, then they can do so by editing the
>      TODO list.

I'm not sure what the use case is for making the update-refs step 
visible to the user - it seems to be added complexity for them to deal 
with. We don't do that for the branch that is being rebased so why do we 
need to do it for the other branches? As for the implementation we could 
just update the branches at the end of the loop in pick_commits() where 
we update the branch and run the post-rewrite hook etc. there is no need 
for an entry in the todo list.

Best Wishes

Phillip

Re: [PATCH v2 0/7] rebase: update branches in multi-part topic

[Derrick Stolee]

On 6/8/2022 10:32 AM, Phillip Wood wrote:
> Hi Stolee
> 
> On 07/06/2022 21:42, Derrick Stolee via GitGitGadget wrote:
>> [...]
>> As an example, here is my in-progress bundle URI RFC split into subtopics as
>> they appear during the TODO list of a git rebase -i --update-refs:
>>
>> pick 2d966282ff3 docs: document bundle URI standard
>> pick 31396e9171a remote-curl: add 'get' capability
>> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
>> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
>> pick 6adaf842684 fetch: add --bundle-uri option
>> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
>> label for-update-refs/refs/heads/bundle-redo/fetch
>> [...] update-refs
>> [...]
>> Based on some of the discussion, it seemed like one way to do this would be
>> to have an 'update-ref ' command that would take the place of these 'label'
>> commands. However, this would require two things that make it a bit awkward:
>>
>>   1. We would need to replicate the storage of those positions during the
>>      rebase. 'label' already does this pretty well. I've added the
>>      "for-update-refs/" label to help here.
> 
> I'm afraid I don't think that using a label with a name constructed from
> a magic prefix and the full refname is a good user interface.
> 
> (i) Having labels behave differently based on their name is confusing.

The label commands do as advertised, but the 'update-refs' command does
something with the set of refs based on their names, yes. We would need to
store this information during the rebase _somewhere_, and refs/rewritten/
seems natural.
 
> (ii) The length of the label string is cumbersome for users. Rebase already
> has a reputation for being unfriendly and hard to use, this will not improve
> that. "update-ref bundle-redo/fetch" is much simpler.

I agree that your proposed replacement for "label for-update-refs/..." would
look cleaner. I don't think that that is enough to justify hiding information
from the user.

> (iii) It means that we no longer store the oid of each branch when creating> the todo list and so cannot check if it is safe to update it at the end of the
> rebase (just using the current value of the branch ref at the end of a long
> running operation like rebase is not sufficient to be safe).

The operation we are doing necessarily requires a force-update, since we are
modifying history. The safety you are caring about here is about the case where
the user modifies one of these refs between the initial 'git rebase --update-refs'
and the final 'git rebase --continue' that actually writes the refs. You want to
prevent that final update from succeeding because another change to those refs
could be lost.

I'm less concerned about this case, because the user is requesting that we
update the refs pointing to this set of commits. But, I'm glad you brought it
up.

One way to prevent this situation would be to extend the idea of "what branch
is being rebased?" (REBASE_HEAD) to "which branches are being rewritten?"
(REBASE_HEADS?). If the worktree kept the full list somewhere in the $GIT_DIR,
then other Git commands could notice that those refs are currently being
overwritten and those updates should fail (similarly to how we currently fail
to update a branch being rebased). This ties into your later point:

>>   2. If we want to close out all of the refs as the rebase is finishing, then
>>      that "step" becomes invisible to the user (and a bit more complicated to
>>      insert). Thus, the 'update-refs' step performs this action. If the user
>>      wants to do things after that step, then they can do so by editing the
>>      TODO list.
> 
> I'm not sure what the use case is for making the update-refs step visible to
> the user - it seems to be added complexity for them to deal with. We don't do
> that for the branch that is being rebased so why do we need to do it for the
> other branches? As for the implementation we could just update the branches
> at the end of the loop in pick_commits() where we update the branch and run the
> post-rewrite hook etc. there is no need for an entry in the todo list.

I concede that allowing the user to move the 'update-refs' command around is
not super important.

The biggest concern I originally had with this idea was that there was no
connection between the "--update-refs" option in the first "git rebase"
command and the final "git rebase --continue" command. That is, other than
which refs are in refs/rewritten/*.

HOWEVER, using refs/rewritten/* is likely buggy: if we had two rebases
happening at the same time (operating on a disjoint set of refs), then how
do we differentiate which refs make sense for us to update as we complete
this rebase?

So, I'm coming to the conclusion that using refs/rewritten/* is problematic
and I should use something closer to REBASE_HEAD as a way to describe which
refs are being updated. In that context, your 'update-ref <ref>' command
makes a lot more sense. The user can decide to move those around _or_ remove
them; it won't change their protection under "REBASE_HEADS" but would
prevent them from being rewritten.

While I think on this, I'll send my branch_checked_out() patches in a
separate thread, since those have grown in complexity since this version.

Thanks,
-Stolee

Re: [PATCH v2 0/7] rebase: update branches in multi-part topic

[Phillip Wood]

Hi Stolee

On 08/06/2022 19:09, Derrick Stolee wrote:
> On 6/8/2022 10:32 AM, Phillip Wood wrote:
>> Hi Stolee
>>
>> On 07/06/2022 21:42, Derrick Stolee via GitGitGadget wrote:
>>> [...]
>>> As an example, here is my in-progress bundle URI RFC split into subtopics as
>>> they appear during the TODO list of a git rebase -i --update-refs:
>>>
>>> pick 2d966282ff3 docs: document bundle URI standard
>>> pick 31396e9171a remote-curl: add 'get' capability
>>> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
>>> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
>>> pick 6adaf842684 fetch: add --bundle-uri option
>>> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
>>> label for-update-refs/refs/heads/bundle-redo/fetch
>>> [...] update-refs
>>> [...]
>>> Based on some of the discussion, it seemed like one way to do this would be
>>> to have an 'update-ref ' command that would take the place of these 'label'
>>> commands. However, this would require two things that make it a bit awkward:
>>>
>>>    1. We would need to replicate the storage of those positions during the
>>>       rebase. 'label' already does this pretty well. I've added the
>>>       "for-update-refs/" label to help here.
>>
>> I'm afraid I don't think that using a label with a name constructed from
>> a magic prefix and the full refname is a good user interface.
>>
>> (i) Having labels behave differently based on their name is confusing.
> 
> The label commands do as advertised, but the 'update-refs' command does
> something with the set of refs based on their names, yes. We would need to
> store this information during the rebase _somewhere_, and refs/rewritten/
> seems natural.
>   
>> (ii) The length of the label string is cumbersome for users. Rebase already
>> has a reputation for being unfriendly and hard to use, this will not improve
>> that. "update-ref bundle-redo/fetch" is much simpler.
> 
> I agree that your proposed replacement for "label for-update-refs/..." would
> look cleaner. I don't think that that is enough to justify hiding information
> from the user.

In general I think it is better not to burden the user with unnecessary 
information - it makes the ui more complicated and exposes 
implementation details which makes it harder to change the implementation.

>> (iii) It means that we no longer store the oid of each branch when creating> the todo list and so cannot check if it is safe to update it at the end of the
>> rebase (just using the current value of the branch ref at the end of a long
>> running operation like rebase is not sufficient to be safe).
> 
> The operation we are doing necessarily requires a force-update, since we are
> modifying history. The safety you are caring about here is about the case where
> the user modifies one of these refs between the initial 'git rebase --update-refs'
> and the final 'git rebase --continue' that actually writes the refs. You want to
> prevent that final update from succeeding because another change to those refs
> could be lost.
> 
> I'm less concerned about this case, because the user is requesting that we
> update the refs pointing to this set of commits. But, I'm glad you brought it
> up.

At the end of the rebase we pass the oid of the branch that we store at 
the start of the rebase when updating the ref to avoid nasty surprises. 
I think it makes sense to do the same for the other branches being 
rewritten - it is easy to get stuck on a conflict resolution and go and 
do something else for a while and forget a branch is being rebased. If 
we cannot update the ref at the end of the rebase we should print the 
new oid with instructions to run "git reset --hard" or "git update-ref" 
if the user wants to update the branch.

> One way to prevent this situation would be to extend the idea of "what branch
> is being rebased?" (REBASE_HEAD) to "which branches are being rewritten?"
> (REBASE_HEADS?). 

Just to clarify REBASE_HEAD points to the commit we're currently trying 
to pick, the branch ref is stored it .git/rebase-merge/head-name and its 
oid in .git/rebase-merge/orig-head

> If the worktree kept the full list somewhere in the $GIT_DIR,
> then other Git commands could notice that those refs are currently being
> overwritten and those updates should fail (similarly to how we currently fail
> to update a branch being rebased).

That's a good point and one that I should have remembered as I 
implemented that check in my script. I agree that we should stop the 
user starting a rebase in a worktree whose branch is being updated 
elsewhere. If we write a list of the refs we want to update under 
.git/rebase-merge before walking the other worktrees to see what's being 
rebased elsewhere that avoids a race where two processes start that try 
to rebase the same branch in different worktrees at the same time. It 
would be easy to store the original oids with the ref names.

An added complexity is that we would have to check the todo list for any 
new update-ref commands and check those refs are not being rebased 
elsewhere each time the list is edited. In the future we should update 
"git status" to read that file but I don't think that needs to be part 
of the initial implementation.

> This ties into your later point:
> 
>>>    2. If we want to close out all of the refs as the rebase is finishing, then
>>>       that "step" becomes invisible to the user (and a bit more complicated to
>>>       insert). Thus, the 'update-refs' step performs this action. If the user
>>>       wants to do things after that step, then they can do so by editing the
>>>       TODO list.
>>
>> I'm not sure what the use case is for making the update-refs step visible to
>> the user - it seems to be added complexity for them to deal with. We don't do
>> that for the branch that is being rebased so why do we need to do it for the
>> other branches? As for the implementation we could just update the branches
>> at the end of the loop in pick_commits() where we update the branch and run the
>> post-rewrite hook etc. there is no need for an entry in the todo list.
> 
> I concede that allowing the user to move the 'update-refs' command around is
> not super important.
> 
> The biggest concern I originally had with this idea was that there was no
> connection between the "--update-refs" option in the first "git rebase"
> command and the final "git rebase --continue" command. That is, other than
> which refs are in refs/rewritten/*.
> 
> HOWEVER, using refs/rewritten/* is likely buggy: if we had two rebases
> happening at the same time (operating on a disjoint set of refs), then how
> do we differentiate which refs make sense for us to update as we complete
> this rebase?
 >> So, I'm coming to the conclusion that using refs/rewritten/* is 
problematic
> and I should use something closer to REBASE_HEAD as a way to describe which
> refs are being updated. In that context, your 'update-ref <ref>' command
> makes a lot more sense. The user can decide to move those around _or_ remove
> them; it won't change their protection under "REBASE_HEADS" but would
> prevent them from being rewritten.

Whenever one adds a new option to rebase it almost always involves 
writing more state to .git/rebase-merge, in this case I think we want to 
store the branches from the update-ref commands and their original oids 
there. I'm not sure that we want to expose the file to the user though 
so we can change the implementation in the future if we need to (e.g. we 
may decide it is better to store this information for all worktrees 
under a single file in $GIT_COMMON_DIR).

> While I think on this, I'll send my branch_checked_out() patches in a
> separate thread, since those have grown in complexity since this version.

If you want to discuss any ideas face-to-face drop me a email and we can 
arrange a time to chat if that would be useful.

Best Wishes

Phillip

> -Stolee

[PATCH v3 0/8] rebase: update branches in multi-part topic

[Derrick Stolee via GitGitGadget]

(Update: This is now based on ds/branch-checked-out.)

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'update-ref' commands into the
TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
update-ref refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
update-ref refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
update-ref refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

update-ref refs/heads/bundle-redo/advertise


Here is an outline of the series:

 * Patch 1 updates some tests for branch_checked_out() for the 'apply'
   backend.
 * Patch 2 updates branch_checked_out() to parse the
   rebase-merge/update-refs file to block concurrent ref updates and
   checkouts on branches selected by --update-refs.
 * Patch 3 updates the todo list documentation to remove some unnecessary
   dots in the 'merge' command. This makes it consistent with the 'fixup'
   command before we document the 'update-ref' command.
 * Patch 4 updates the definition of todo_command_info to use enum values as
   array indices.
 * Patches 5-7 implement the --update-refs logic itself.
 * Patch 8 adds the rebase.updateRefs config option similar to
   rebase.autoSquash.


Updates in v3
=============

 * The branch_checked_out() API was extracted to its own topic and is now
   the ds/branch-checked-out branch. This series is now based on that one.
 * The for_each_decoration() API was removed, since it became trivial once
   it did not take a commit directly.
 * The branch_checked_out() tests did not verify the rebase-apply data (for
   the apply backend), so that is fixed.
 * Instead of using the 'label' command and a final 'update-refs' command in
   the todo list, use a new 'update-ref ' command. This command updates the
   rebase-merge/update-refs file with the OID of HEAD at these steps. At the
   very end of the rebase sequence, those refs are updated to the stored OID
   values (assuming that they were not removed by the user, in which case we
   notice that the OID is the null OID and we do nothing).
 * New tests are added.
 * The todo-list comment documentation has some new formatting updates, but
   also includes a description of 'update-refs' in this version.


Updates in v2
=============

As recommended by the excellent feedback, I have removed the 'exec' commands
in favor of the 'label' commands and a new 'update-refs' command at the very
end. This way, there is only one step that updates all of the refs at the
end instead of updating refs during the rebase. If a user runs 'git rebase
--abort' in the middle, then their refs are still where they need to be.

Based on some of the discussion, it seemed like one way to do this would be
to have an 'update-ref ' command that would take the place of these 'label'
commands. However, this would require two things that make it a bit awkward:

 1. We would need to replicate the storage of those positions during the
    rebase. 'label' already does this pretty well. I've added the
    "for-update-refs/" label to help here.
 2. If we want to close out all of the refs as the rebase is finishing, then
    that "step" becomes invisible to the user (and a bit more complicated to
    insert). Thus, the 'update-refs' step performs this action. If the user
    wants to do things after that step, then they can do so by editing the
    TODO list.

Other updates:

 * The 'keep_decorations' parameter was renamed to 'update_refs'.
 * I added tests for --rebase-merges=rebase-cousins to show how these labels
   interact with other labels and merge commands.
 * I changed the order of the insertion of these update-refs labels to be
   before the fixups are rearranged. This fixes a bug where the tip commit
   is a fixup! so its decorations are never inspected (and they would be in
   the wrong place even if they were). The fixup! commands are properly
   inserted between a pick and its following label command. Tests
   demonstrate this is correct.
 * Numerous style choices are updated based on feedback.

Thank you for all of the detailed review and ideas in this space. I
appreciate any more ideas that can make this feature as effective as it can
be.

Thanks, -Stolee

Derrick Stolee (8):
  t2407: test branches currently using apply backend
  branch: consider refs under 'update-refs'
  rebase-interactive: update 'merge' description
  sequencer: define array with enum values
  sequencer: add update-ref command
  rebase: add --update-refs option
  rebase: update refs from 'update-ref' commands
  rebase: add rebase.updateRefs config option

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  11 ++
 branch.c                        |  13 ++
 builtin/rebase.c                |  10 ++
 rebase-interactive.c            |   9 +-
 sequencer.c                     | 303 ++++++++++++++++++++++++++++++--
 sequencer.h                     |  11 ++
 t/t2407-worktree-heads.sh       |  52 +++++-
 t/t3404-rebase-interactive.sh   | 107 +++++++++++
 9 files changed, 496 insertions(+), 23 deletions(-)


base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1247

Range-diff vs v2:

 1:  4f9f3487641 < -:  ----------- log-tree: create for_each_decoration()
 2:  5f54766e103 < -:  ----------- branch: add branch_checked_out() helper
 -:  ----------- > 1:  fbaedc7f1f0 t2407: test branches currently using apply backend
 -:  ----------- > 2:  2bc647b6fcd branch: consider refs under 'update-refs'
 -:  ----------- > 3:  669f4abd59e rebase-interactive: update 'merge' description
 3:  9f261c7df2c = 4:  6528a50343f sequencer: define array with enum values
 4:  842b2186d25 ! 5:  e95ad41d355 sequencer: add update-refs command
     @@ Metadata
      Author: Derrick Stolee <derrickstolee@github.com>
      
       ## Commit message ##
     -    sequencer: add update-refs command
     +    sequencer: add update-ref command
      
     -    Add the boilerplat for an "update-refs" command in the sequencer. This
     -    connects to the current no-op do_update_refs() which will be filled in
     +    Add the boilerplate for an "update-ref" command in the sequencer. This
     +    connects to the current no-op do_update_ref() which will be filled in
          after more connections are created.
      
     +    The syntax in the todo list will be "update-ref <ref-name>" to signal
     +    that we should store the current commit as the value for updating
     +    <ref-name> at the end of the rebase.
     +
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
     + ## rebase-interactive.c ##
     +@@ rebase-interactive.c: void append_todo_help(int command_count,
     + "        create a merge commit using the original merge commit's\n"
     + "        message (or the oneline, if no original merge commit was\n"
     + "        specified); use -c <commit> to reword the commit message\n"
     ++"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
     ++"                      to this position in the new commits. The <ref> is\n"
     ++"                      updated at the end of the rebase\n"
     + "\n"
     + "These lines can be re-ordered; they are executed from top to bottom.\n");
     + 	unsigned edit_todo = !(shortrevisions && shortonto);
     +
       ## sequencer.c ##
      @@ sequencer.c: static struct {
       	[TODO_LABEL] = { 'l', "label" },
       	[TODO_RESET] = { 't', "reset" },
       	[TODO_MERGE] = { 'm', "merge" },
     -+	[TODO_UPDATE_REFS] = { 'u', "update-refs" },
     ++	[TODO_UPDATE_REF] = { 'u', "update-ref" },
       	[TODO_NOOP] = { 0,   "noop" },
       	[TODO_DROP] = { 'd', "drop" },
       	[TODO_COMMENT] = { 0,   NULL },
      @@ sequencer.c: static int parse_insn_line(struct repository *r, struct todo_item *item,
     - 	padding = strspn(bol, " \t");
     - 	bol += padding;
     + 			     command_to_string(item->command));
       
     --	if (item->command == TODO_NOOP || item->command == TODO_BREAK) {
     -+	if (item->command == TODO_NOOP ||
     -+	    item->command == TODO_BREAK ||
     -+	    item->command == TODO_UPDATE_REFS) {
     - 		if (bol != eol)
     - 			return error(_("%s does not accept arguments: '%s'"),
     - 				     command_to_string(item->command), bol);
     + 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
     +-	    item->command == TODO_RESET) {
     ++	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
     + 		item->commit = NULL;
     + 		item->arg_offset = bol - buf;
     + 		item->arg_len = (int)(eol - bol);
      @@ sequencer.c: leave_merge:
       	return ret;
       }
       
     -+static int do_update_refs(struct repository *r)
     ++static int do_update_ref(struct repository *r, const char *ref_name)
      +{
      +	return 0;
      +}
     @@ sequencer.c: static int pick_commits(struct repository *r,
       				return error_with_patch(r, item->commit,
       							arg, item->arg_len,
       							opts, res, 0);
     -+		} else if (item->command == TODO_UPDATE_REFS) {
     -+			if ((res = do_update_refs(r)))
     ++		} else if (item->command == TODO_UPDATE_REF) {
     ++			struct strbuf ref = STRBUF_INIT;
     ++			strbuf_add(&ref, arg, item->arg_len);
     ++			if ((res = do_update_ref(r, ref.buf)))
      +				reschedule = 1;
     ++			strbuf_release(&ref);
       		} else if (!is_noop(item->command))
       			return error(_("unknown command %d"), item->command);
       
     @@ sequencer.h: enum todo_command {
       	TODO_LABEL,
       	TODO_RESET,
       	TODO_MERGE,
     -+	TODO_UPDATE_REFS,
     ++	TODO_UPDATE_REF,
       	/* commands that do nothing but are counted for reporting progress */
       	TODO_NOOP,
       	TODO_DROP,
 5:  0a4c110127b ! 6:  918b398d6a2 rebase: add --update-refs option
     @@ Commit message
          reachable from those "sub branches". It can take a manual step to update
          those branches.
      
     -    Add a new --update-refs option to 'git rebase -i' that adds 'label
     -    for-update-refs/*' steps to the todo file whenever a commit that is
     -    being rebased is decorated with that <ref>. At the very end, the
     -    'update-refs' step is added to update all of the branches referenced by
     -    the 'label' steps. This allows the user to rebase a long list of commits
     -    in a multi-part feature and keep all of their pointers to those parts.
     +    Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
     +    <ref>' steps to the todo file whenever a commit that is being rebased is
     +    decorated with that <ref>. At the very end, the rebase process updates
     +    all of the listed refs to the values stored during the rebase operation.
      
     -    NOTE: This change only introduce the --update-refs option and implements
     -    the changes to the todo file. It does _not_ yet implement the action
     -    taken by the 'update-refs' todo step, which will be implemented and
     -    tested in a later change.
     -
     -    Use the new for_each_decoration() while iterating over the existing todo
     -    list. Be sure to iterate after any squashing or fixups are placed.
     -    Update the branch only after those squashes and fixups are complete.
     -    This allows a --fixup commit at the tip of the feature to apply
     -    correctly to the sub branch, even if it is fixing up the most-recent
     -    commit in that part.
     +    Be sure to iterate after any squashing or fixups are placed. Update the
     +    branch only after those squashes and fixups are complete. This allows a
     +    --fixup commit at the tip of the feature to apply correctly to the sub
     +    branch, even if it is fixing up the most-recent commit in that part.
      
          One potential problem here is that refs decorating commits that are
          already marked as "fixup!" or "squash!" will not be included in this
     @@ Commit message
          of the rebased branch while the other sub branches come along for the
          ride without intervention.
      
     -    Be careful to not attempt updating any branch that is checked out. The
     -    most common example is the branch being rebased is checked out and
     -    decorates the tip commit. If the user is rebasing commits reachable from
     -    a different branch that is checked out in a different worktree, then
     -    they may be surprised to not see that ref update. However, it's probably
     -    best to not optimize for this scenario and do the safest thing that will
     -    result in a successful rebase. A comment is left in the TODO list that
     -    signals that these refs are currently checked out.
     +    This change update the documentation and builtin to accept the
     +    --update-refs option as well as updating the todo file with the
     +    'update-ref' commands. Tests are added to ensure that these todo
     +    commands are added in the correct locations.
     +
     +    A future change will update the behavior to actually update the refs
     +    at the end of the rebase sequence.
      
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
     @@ sequencer.c
       #include "rebase-interactive.h"
       #include "reset.h"
      +#include "branch.h"
     -+#include "log-tree.h"
       
       #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
       
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +	size_t items_alloc;
      +	struct strbuf *buf;
      +	struct commit *commit;
     ++	struct string_list refs_to_oids;
      +};
      +
     -+static int add_branch_for_decoration(const struct name_decoration *d, void *data)
     ++static int add_decorations_to_list(const struct commit *commit,
     ++				   struct todo_add_branch_context *ctx)
      +{
     -+	struct todo_add_branch_context *ctx = data;
     -+	size_t base_offset = ctx->buf->len;
     -+	struct todo_item *item;
     -+	char *path;
     ++	const struct name_decoration *decoration = get_name_decoration(&commit->object);
      +
     -+	ALLOC_GROW(ctx->items,
     -+		   ctx->items_nr + 1,
     -+		   ctx->items_alloc);
     -+	item = &ctx->items[ctx->items_nr];
     -+	memset(item, 0, sizeof(*item));
     ++	while (decoration) {
     ++		struct todo_item *item;
     ++		const char *path;
     ++		size_t base_offset = ctx->buf->len;
      +
     -+	/* If the branch is checked out, then leave a comment instead. */
     -+	if (branch_checked_out(d->name, &path)) {
     -+		item->command = TODO_COMMENT;
     -+		strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
     -+			    d->name, path);
     -+		free(path);
     -+	} else {
     -+		item->command = TODO_LABEL;
     -+		strbuf_addf(ctx->buf, "for-update-refs/%s\n", d->name);
     -+	}
     ++		ALLOC_GROW(ctx->items,
     ++			ctx->items_nr + 1,
     ++			ctx->items_alloc);
     ++		item = &ctx->items[ctx->items_nr];
     ++		memset(item, 0, sizeof(*item));
     ++
     ++		/* If the branch is checked out, then leave a comment instead. */
     ++		if ((path = branch_checked_out(decoration->name))) {
     ++			item->command = TODO_COMMENT;
     ++			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
     ++				    decoration->name, path);
     ++		} else {
     ++			struct string_list_item *sti;
     ++			item->command = TODO_UPDATE_REF;
     ++			strbuf_addf(ctx->buf, "%s\n", decoration->name);
     ++
     ++			sti = string_list_append(&ctx->refs_to_oids,
     ++						 decoration->name);
     ++			sti->util = oiddup(the_hash_algo->null_oid);
     ++		}
     ++
     ++		item->offset_in_buf = base_offset;
     ++		item->arg_offset = base_offset;
     ++		item->arg_len = ctx->buf->len - base_offset;
     ++		ctx->items_nr++;
      +
     -+	item->offset_in_buf = base_offset;
     -+	item->arg_offset = base_offset;
     -+	item->arg_len = ctx->buf->len - base_offset;
     -+	ctx->items_nr++;
     ++		decoration = decoration->next;
     ++	}
      +
      +	return 0;
      +}
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +	struct decoration_filter decoration_filter = {
      +		.include_ref_pattern = &decorate_refs_include,
      +		.exclude_ref_pattern = &decorate_refs_exclude,
     -+		.exclude_ref_config_pattern = &decorate_refs_exclude_config
     ++		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
      +	};
      +	struct todo_add_branch_context ctx = {
      +		.buf = &todo_list->buf,
     ++		.refs_to_oids = STRING_LIST_INIT_DUP,
      +	};
      +
      +	ctx.items_alloc = 2 * todo_list->nr + 1;
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +
      +		if (item->commit) {
      +			ctx.commit = item->commit;
     -+			for_each_decoration(item->commit,
     -+					    add_branch_for_decoration,
     -+					    &ctx);
     ++			add_decorations_to_list(item->commit, &ctx);
      +		}
      +	}
      +
     -+	/* Add the "update-refs" step. */
     -+	ALLOC_GROW(ctx.items,
     -+		   ctx.items_nr + 1,
     -+		   ctx.items_alloc);
     -+	memset(&ctx.items[ctx.items_nr], 0, sizeof(struct todo_item));
     -+	ctx.items[ctx.items_nr].command = TODO_UPDATE_REFS;
     -+	ctx.items_nr++;
     -+
     ++	string_list_clear(&ctx.refs_to_oids, 1);
      +	free(todo_list->items);
      +	todo_list->items = ctx.items;
      +	todo_list->nr = ctx.items_nr;
     @@ sequencer.h: int complete_action(struct repository *r, struct replay_opts *opts,
       int todo_list_rearrange_squash(struct todo_list *todo_list);
       
      
     + ## t/t2407-worktree-heads.sh ##
     +@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite when in error states' '
     + 	done
     + '
     + 
     ++. "$TEST_DIRECTORY"/lib-rebase.sh
     ++
     ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
     ++	git commit --fixup HEAD~2 --allow-empty &&
     ++	(
     ++		set_cat_todo_editor &&
     ++		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
     ++		! grep "update-refs" todo
     ++	) &&
     ++	git branch -f allow-update HEAD~2 &&
     ++	(
     ++		set_cat_todo_editor &&
     ++		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
     ++		grep "update-ref refs/heads/allow-update" todo
     ++	)
     ++'
     ++
     + test_done
     +
       ## t/t3404-rebase-interactive.sh ##
      @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correctly' '
       	test_cmp_rev ORIG_HEAD test-orig-head@{1}
     @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
      +
      +		cat >expect <<-EOF &&
      +		pick $(git log -1 --format=%h J) J
     -+		label for-update-refs/refs/heads/second
     -+		label for-update-refs/refs/heads/first
     ++		update-ref refs/heads/second
     ++		update-ref refs/heads/first
      +		pick $(git log -1 --format=%h K) K
      +		pick $(git log -1 --format=%h L) L
      +		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
     -+		label for-update-refs/refs/heads/third
     ++		update-ref refs/heads/third
      +		pick $(git log -1 --format=%h M) M
     -+		label for-update-refs/refs/heads/no-conflict-branch
     -+		label for-update-refs/refs/heads/shared-tip
     -+		update-refs
     ++		update-ref refs/heads/no-conflict-branch
     ++		update-ref refs/heads/shared-tip
      +		EOF
      +
      +		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
     @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
      +		reset onto
      +		pick $(git log -1 --format=%h branch2~1) F
      +		pick $(git log -1 --format=%h branch2) I
     -+		label for-update-refs/refs/heads/branch2
     ++		update-ref refs/heads/branch2
      +		label merge
      +		reset onto
      +		pick $(git log -1 --format=%h refs/heads/second) J
     -+		label for-update-refs/refs/heads/second
     -+		label for-update-refs/refs/heads/first
     ++		update-ref refs/heads/second
     ++		update-ref refs/heads/first
      +		pick $(git log -1 --format=%h refs/heads/third~1) K
      +		pick $(git log -1 --format=%h refs/heads/third) L
      +		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
     -+		label for-update-refs/refs/heads/third
     ++		update-ref refs/heads/third
      +		pick $(git log -1 --format=%h HEAD~2) M
     -+		label for-update-refs/refs/heads/no-conflict-branch
     ++		update-ref refs/heads/no-conflict-branch
      +		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
     -+		label for-update-refs/refs/heads/merge-branch
     -+		update-refs
     ++		update-ref refs/heads/merge-branch
      +		EOF
      +
      +		test_must_fail git rebase -i --autosquash \
 6:  68f8e51b19c ! 7:  72e0481b643 sequencer: implement 'update-refs' command
     @@ Metadata
      Author: Derrick Stolee <derrickstolee@github.com>
      
       ## Commit message ##
     -    sequencer: implement 'update-refs' command
     +    rebase: update refs from 'update-ref' commands
      
     -    The previous change allowed 'git rebase --update-refs' to create 'label'
     -    commands for each branch  among the commits being rewritten and add an
     -    'update-refs' command at the end of the todo list. Now, teach Git to
     -    update the refs during that final 'update-refs' command.
     +    The previous change introduced the 'git rebase --update-refs' option
     +    which added 'update-ref <ref>' commands to the todo list of an
     +    interactive rebase.
      
     -    We need to create an array of new and old OIDs for each ref by iterating
     -    over the refs/rewritten/for-update-refs/ namespace. We cannot update the
     -    refs in-place since this will confuse the refs iterator.
     +    Teach Git to record the HEAD position when reaching these 'update-ref'
     +    commands. The ref/OID pair is stored in the
     +    $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
     +    file to avoid having other processes updating the refs in that file
     +    while the rebase is in progress.
     +
     +    Not only do we update the file when the sequencer reaches these
     +    'update-ref' commands, we then update the refs themselves at the end of
     +    the rebase sequence. If the rebase is aborted before this final step,
     +    then the refs are not updated.
      
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
       ## sequencer.c ##
     +@@
     + #include "rebase-interactive.h"
     + #include "reset.h"
     + #include "branch.h"
     ++#include "log-tree.h"
     + 
     + #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
     + 
     +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
     +  */
     + static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
     + 
     ++/*
     ++ * The update-refs file stores a list of refs that will be updated at the end
     ++ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
     ++ * update the OIDs for the refs in this file, but the refs are not updated
     ++ * until the end of the rebase sequence.
     ++ */
     ++static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
     ++
     + /*
     +  * The following files are written by git-rebase just after parsing the
     +  * command-line.
      @@ sequencer.c: leave_merge:
       	return ret;
       }
       
     --static int do_update_refs(struct repository *r)
     -+struct update_refs_context {
     -+	struct ref_store *refs;
     -+	char **ref_names;
     -+	struct object_id *old;
     -+	struct object_id *new;
     -+	size_t nr;
     -+	size_t alloc;
     -+};
     -+
     -+static int add_ref_to_context(const char *refname,
     -+			      const struct object_id *oid,
     -+			      int flags,
     -+			      void *data)
     +-static int do_update_ref(struct repository *r, const char *ref_name)
     ++static int write_update_refs_state(struct string_list *refs_to_oids)
     ++{
     ++	int result = 0;
     ++	FILE *fp = NULL;
     ++	struct string_list_item *item;
     ++	char *path = xstrdup(rebase_path_update_refs());
     ++
     ++	if (safe_create_leading_directories(path)) {
     ++		result = error(_("unable to create leading directories of %s"),
     ++			       path);
     ++		goto cleanup;
     ++	}
     ++
     ++	fp = fopen(path, "w");
     ++	if (!fp) {
     ++		result = error_errno(_("could not open '%s' for writing"), path);
     ++		goto cleanup;
     ++	}
     ++
     ++	for_each_string_list_item(item, refs_to_oids)
     ++		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
     ++
     ++cleanup:
     ++	if (fp)
     ++		fclose(fp);
     ++	return result;
     ++}
     ++
     ++static int do_update_ref(struct repository *r, const char *refname)
       {
     -+	int f = 0;
     -+	const char *name;
     -+	struct update_refs_context *ctx = data;
     ++	struct string_list_item *item;
     ++	struct string_list list = STRING_LIST_INIT_DUP;
     ++	int found = 0;
     ++
     ++	sequencer_get_update_refs_state(r->gitdir, &list);
      +
     -+	ALLOC_GROW(ctx->ref_names, ctx->nr + 1, ctx->alloc);
     -+	ALLOC_GROW(ctx->old, ctx->nr + 1, ctx->alloc);
     -+	ALLOC_GROW(ctx->new, ctx->nr + 1, ctx->alloc);
     ++	for_each_string_list_item(item, &list) {
     ++		if (!strcmp(item->string, refname)) {
     ++			struct object_id oid;
     ++			free(item->util);
     ++			found = 1;
      +
     -+	if (!skip_prefix(refname, "refs/rewritten/for-update-refs/", &name))
     -+		return 1;
     ++			if (!read_ref("HEAD", &oid)) {
     ++				item->util = oiddup(&oid);
     ++				break;
     ++			}
     ++		}
     ++	}
      +
     -+	ctx->ref_names[ctx->nr] = xstrdup(name);
     -+	oidcpy(&ctx->new[ctx->nr], oid);
     -+	if (!refs_resolve_ref_unsafe(ctx->refs, name, 0,
     -+				     &ctx->old[ctx->nr], &f))
     -+		return 1;
     ++	if (!found) {
     ++		struct object_id oid;
     ++		item = string_list_append(&list, refname);
      +
     -+	ctx->nr++;
     ++		if (!read_ref("HEAD", &oid))
     ++			item->util = oiddup(&oid);
     ++		else
     ++			item->util = oiddup(the_hash_algo->null_oid);
     ++	}
     ++
     ++	write_update_refs_state(&list);
     ++	string_list_clear(&list, 1);
       	return 0;
       }
       
      +static int do_update_refs(struct repository *r)
      +{
     -+	int i, res;
     -+	struct update_refs_context ctx = {
     -+		.refs = get_main_ref_store(r),
     -+		.alloc = 16,
     -+	};
     -+	ALLOC_ARRAY(ctx.ref_names, ctx.alloc);
     -+	ALLOC_ARRAY(ctx.old, ctx.alloc);
     -+	ALLOC_ARRAY(ctx.new, ctx.alloc);
     -+
     -+	res = refs_for_each_fullref_in(ctx.refs,
     -+				       "refs/rewritten/for-update-refs/",
     -+				       add_ref_to_context,
     -+				       &ctx);
     -+
     -+	for (i = 0; !res && i < ctx.nr; i++)
     -+		res = refs_update_ref(ctx.refs, "rewritten during rebase",
     -+				ctx.ref_names[i],
     -+				&ctx.new[i], &ctx.old[i],
     ++	int res = 0;
     ++	struct string_list_item *item;
     ++	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
     ++	struct ref_store *refs = get_main_ref_store(r);
     ++
     ++	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
     ++
     ++	for_each_string_list_item(item, &refs_to_oids) {
     ++		struct object_id *oid_to = item->util;
     ++		struct object_id oid_from;
     ++
     ++		if (oideq(oid_to, the_hash_algo->null_oid)) {
     ++			/*
     ++			 * Ref was not updated. User may have deleted the
     ++			 * 'update-ref' step.
     ++			 */
     ++			continue;
     ++		}
     ++
     ++		if (read_ref(item->string, &oid_from)) {
     ++			/*
     ++			 * The ref does not exist. The user probably
     ++			 * inserted a new 'update-ref' step with a new
     ++			 * branch name.
     ++			 */
     ++			oidcpy(&oid_from, the_hash_algo->null_oid);
     ++		}
     ++
     ++		res |= refs_update_ref(refs, "rewritten during rebase",
     ++				item->string,
     ++				oid_to, &oid_from,
      +				0, UPDATE_REFS_MSG_ON_ERR);
     ++	}
      +
     -+	for (i = 0; i < ctx.nr; i++)
     -+		free(ctx.ref_names[i]);
     -+	free(ctx.ref_names);
     -+	free(ctx.old);
     -+	free(ctx.new);
     ++	string_list_clear(&refs_to_oids, 1);
      +	return res;
      +}
      +
       static int is_final_fixup(struct todo_list *todo_list)
       {
       	int i = todo_list->current;
     +@@ sequencer.c: cleanup_head_ref:
     + 		strbuf_release(&head_ref);
     + 	}
     + 
     ++	do_update_refs(r);
     ++
     + 	/*
     + 	 * Sequence of picks finished successfully; cleanup by
     + 	 * removing the .git/sequencer directory
     +@@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
     + 		}
     + 	}
     + 
     ++	write_update_refs_state(&ctx.refs_to_oids);
     ++
     + 	string_list_clear(&ctx.refs_to_oids, 1);
     + 	free(todo_list->items);
     + 	todo_list->items = ctx.items;
      
       ## t/t3404-rebase-interactive.sh ##
      @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands with --rebase-merges' '
     @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands
      +	git branch -f third HEAD~1 &&
      +	test_commit extra2 fileX &&
      +	git commit --amend --fixup=L &&
     -+	(
     -+		git rebase -i --autosquash --update-refs primary &&
     -+
     -+		compare_two_refs HEAD~3 refs/heads/first &&
     -+		compare_two_refs HEAD~3 refs/heads/second &&
     -+		compare_two_refs HEAD~1 refs/heads/third &&
     -+		compare_two_refs HEAD refs/heads/no-conflict-branch
     -+	)
     ++
     ++	git rebase -i --autosquash --update-refs primary &&
     ++
     ++	compare_two_refs HEAD~3 refs/heads/first &&
     ++	compare_two_refs HEAD~3 refs/heads/second &&
     ++	compare_two_refs HEAD~1 refs/heads/third &&
     ++	compare_two_refs HEAD refs/heads/no-conflict-branch
      +'
      +
       # This must be the last test in this file
 7:  3d7d3f656b4 ! 8:  d2cfdbfc431 rebase: add rebase.updateRefs config option
     @@ Documentation/git-rebase.txt: start would be overridden by the presence of
       	or point to a `squash! ...` or `fixup! ...` commit are not updated
       	in this way.
      ++
     -+If the `--update-refs` option is enabled by default using the
     -+configuration variable `rebase.updateRefs`, this option can be
     -+used to override and disable this setting.
     ++If the configuration variable `rebase.updateRefs` is set, then this option
     ++can be used to override and disable this setting.
       
       INCOMPATIBLE OPTIONS
       --------------------

-- 
gitgitgadget

[PATCH v3 1/8] t2407: test branches currently using apply backend

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The tests in t2407 that verify the branch_checked_out() helper in the
case of bisects and rebases were added by 9347303db89 (branch: check for
bisects and rebases, 2022-06-08). However, that commit failed to check
for rebases that are using the 'apply' backend.

Add a test that checks the apply backend. The implementation was already
correct here, but it is good to have regression tests before modifying
the implementation further.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 t/t2407-worktree-heads.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index b6be42f74a2..4f59bc21303 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -54,7 +54,18 @@ test_expect_success 'refuse to overwrite: worktree in bisect' '
 	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase' '
+test_expect_success 'refuse to overwrite: worktree in rebase (apply)' '
+	test_when_finished rm -rf .git/worktrees/wt-*/rebase-apply &&
+
+	mkdir -p .git/worktrees/wt-3/rebase-apply &&
+	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-apply/head-name &&
+	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-apply/onto &&
+
+	test_must_fail git branch -f fake-1 HEAD 2>err &&
+	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
+'
+
+test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
 	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
 
 	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-- 
gitgitgadget

Re: [PATCH v3 1/8] t2407: test branches currently using apply backend

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The tests in t2407 that verify the branch_checked_out() helper in the
> case of bisects and rebases were added by 9347303db89 (branch: check for
> bisects and rebases, 2022-06-08). However, that commit failed to check
> for rebases that are using the 'apply' backend.
>
> Add a test that checks the apply backend. The implementation was already
> correct here, but it is good to have regression tests before modifying
> the implementation further.
>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  t/t2407-worktree-heads.sh | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
> index b6be42f74a2..4f59bc21303 100755
> --- a/t/t2407-worktree-heads.sh
> +++ b/t/t2407-worktree-heads.sh
> @@ -54,7 +54,18 @@ test_expect_success 'refuse to overwrite: worktree in bisect' '
>  	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
>  '
>  
> -test_expect_success 'refuse to overwrite: worktree in rebase' '
> +test_expect_success 'refuse to overwrite: worktree in rebase (apply)' '
> +	test_when_finished rm -rf .git/worktrees/wt-*/rebase-apply &&
> +
> +	mkdir -p .git/worktrees/wt-3/rebase-apply &&
> +	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-apply/head-name &&
> +	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-apply/onto &&
> +
> +	test_must_fail git branch -f fake-1 HEAD 2>err &&
> +	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
> +'
> +
> +test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
>  	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
>  
>  	mkdir -p .git/worktrees/wt-3/rebase-merge &&

This is not the first offender, since the other existing one is
doing the same, but it is a bit sad that this makes it worse to
expose and depend on the details of the way the rebase happens to be
currently implemented.

Perhaps a more kosher way to do this is to find an commit that
surely would not allow fake-1 branch to be cleanly rebased onto and
actually start (and cause it to stop) a rebase.

I notice that the original offence was committed fairly recently, by
d2ba271a (branch: check for bisects and rebases, 2022-06-14) that we
can easily eject out of the 'next' branch when we rewind and rebuild
it, if we wanted to.

Anyway, let's read on.

Thanks.

Re: [PATCH v3 1/8] t2407: test branches currently using apply backend

[Derrick Stolee]

On 6/28/22 4:44 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

>> -test_expect_success 'refuse to overwrite: worktree in rebase' '
>> +test_expect_success 'refuse to overwrite: worktree in rebase (apply)' '
>> +	test_when_finished rm -rf .git/worktrees/wt-*/rebase-apply &&
>> +
>> +	mkdir -p .git/worktrees/wt-3/rebase-apply &&
>> +	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-apply/head-name &&
>> +	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-apply/onto &&
>> +
>> +	test_must_fail git branch -f fake-1 HEAD 2>err &&
>> +	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
>> +'
>> +
>> +test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
>>  	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
>>  
>>  	mkdir -p .git/worktrees/wt-3/rebase-merge &&
> 
> This is not the first offender, since the other existing one is
> doing the same, but it is a bit sad that this makes it worse to
> expose and depend on the details of the way the rebase happens to be
> currently implemented.

This is true that it is based on how  it's currently implemented. I
was confused that none of these rebase storage mechanisms are
documented anywhere (so I couldn't extend that documentation with the
new update-refs file added in this series). But perhaps that's by
design: this is intentionally an internal implementation detail.

I do wonder where that leaves third-party implementations like libgit2
to handle any changes we make in this space, or how much we care about
supporting someone rebasing in a worktree with an older Git client
while also checking out a branch in another worktree using a newer one.

> Perhaps a more kosher way to do this is to find an commit that
> surely would not allow fake-1 branch to be cleanly rebased onto and
> actually start (and cause it to stop) a rebase.

This, and the equivalent "pause in the middle of a bisect" would be
ways to hide these internal details.

While I was thinking that this provides a low-weight mechanism for
testing the implementation of branch_checked_out(), it is only testing
the implementation but not the _intention_. If 'git rebase' changed
its backend, then these tests would not start failing as we need them
to.

> I notice that the original offence was committed fairly recently, by
> d2ba271a (branch: check for bisects and rebases, 2022-06-14) that we
> can easily eject out of the 'next' branch when we rewind and rebuild
> it, if we wanted to.

I can either re-roll that series or create a new forward-fix that
includes the functionality of this test. Both are the same amount of
work for me, so let me know which you prefer.

Thanks,
-Stolee

Re: [PATCH v3 1/8] t2407: test branches currently using apply backend

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> I can either re-roll that series or create a new forward-fix that
> includes the functionality of this test. Both are the same amount of
> work for me, so let me know which you prefer.

Either is fine by me.  Other than this small glitch in the test, the
remainder of the "should we allow this branch to be touched?" topic
looked really cleanly done and ready to be unleashed to the public.

Thanks.

Re: [PATCH v3 1/8] t2407: test branches currently using apply backend

[Derrick Stolee]

On 6/30/2022 12:44 PM, Junio C Hamano wrote:
> Derrick Stolee <derrickstolee@github.com> writes:
> 
>> I can either re-roll that series or create a new forward-fix that
>> includes the functionality of this test. Both are the same amount of
>> work for me, so let me know which you prefer.
> 
> Either is fine by me.  Other than this small glitch in the test, the
> remainder of the "should we allow this branch to be touched?" topic
> looked really cleanly done and ready to be unleashed to the public.

If you don't mind, I've got some forward-fixes prepped for the next
version of this topic. They are test-only changes and do not change
the implementation of branch_checked_out().

One thing I noticed from looking back on the series is that I wrote a
test that specifically tests a case that is impossible to construct
without modifying the .git directory directly (having two worktrees
"reserving" a branch for different reasons). That test will need to
stay as one that knows about the internals of this storage, but the
others can be replaced with more opaque steps.

In patch 2 of this series, I test the 'update-refs' storage using a
similar mechanism. I can replace that with the opaque version that
calls 'git rebase --update-refs', but only as part of patch 7, since
writing the file is not implemented yet.

I was going to ask if I should reorder the patches, but there is a
possibility that we will change the storage of these "reserved" refs
yet again, so I won't focus on this point quite yet.

Thanks,
-Stolee

[PATCH v3 2/8] branch: consider refs under 'update-refs'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The branch_checked_out() helper helps commands like 'git branch' and
'git fetch' from overwriting refs that are currently checked out in
other worktrees.

A future update to 'git rebase' will introduce a new '--update-refs'
option which will update the local refs that point to commits that are
being rebased. To avoid collisions as the rebase completes, we want to
make the future data store for these refs to be considered by
branch_checked_out().

The data store is a plaintext file inside the 'rebase-merge' directory
for that worktree. The file alternates refnames and OIDs. The OIDs will
be used to store the to-be-written values as the rebase progresses, but
can be ignored at the moment.

Create a new sequencer_get_update_refs_state() method that parses this
file and populates a struct string_list with the ref-OID pairs. We can
then use this list to add to the current_checked_out_branches strmap
used by branch_checked_out().

To properly navigate to the rebase directory for a given worktree,
extract the static strbuf_worktree_gitdir() method to a public API
method.

We can test that this works without having Git write this file by
artificially creating one in our test script.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 branch.c                  | 13 ++++++++++++
 sequencer.c               | 42 +++++++++++++++++++++++++++++++++++++++
 sequencer.h               |  9 +++++++++
 t/t2407-worktree-heads.sh | 22 ++++++++++++++++----
 4 files changed, 82 insertions(+), 4 deletions(-)

diff --git a/branch.c b/branch.c
index 526e8237673..f252c4eef6c 100644
--- a/branch.c
+++ b/branch.c
@@ -365,6 +365,7 @@ static void prepare_checked_out_branches(void)
 		char *old;
 		struct wt_status_state state = { 0 };
 		struct worktree *wt = worktrees[i++];
+		struct string_list update_refs = STRING_LIST_INIT_DUP;
 
 		if (wt->is_bare)
 			continue;
@@ -400,6 +401,18 @@ static void prepare_checked_out_branches(void)
 			strbuf_release(&ref);
 		}
 		wt_status_state_free_buffers(&state);
+
+		if (!sequencer_get_update_refs_state(get_worktree_git_dir(wt),
+						     &update_refs)) {
+			struct string_list_item *item;
+			for_each_string_list_item(item, &update_refs) {
+				old = strmap_put(&current_checked_out_branches,
+						 item->string,
+						 xstrdup(wt->path));
+				free(old);
+			}
+			string_list_clear(&update_refs, 1);
+		}
 	}
 
 	free_worktrees(worktrees);
diff --git a/sequencer.c b/sequencer.c
index 8c3ed3532ac..1094e146b99 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -5901,3 +5901,45 @@ int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
 
 	return 0;
 }
+
+int sequencer_get_update_refs_state(const char *wt_dir,
+				    struct string_list *refs)
+{
+	int result = 0;
+	struct stat st;
+	FILE *fp = NULL;
+	struct strbuf ref = STRBUF_INIT;
+	struct strbuf hash = STRBUF_INIT;
+	char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);
+
+	if (stat(path, &st))
+		goto cleanup;
+
+	fp = fopen(path, "r");
+	if (!fp)
+		goto cleanup;
+
+	while (strbuf_getline(&ref, fp) != EOF) {
+		struct object_id oid;
+		struct string_list_item *item;
+
+		if (strbuf_getline(&hash, fp) == EOF ||
+		    get_oid_hex(hash.buf, &oid)) {
+			warning(_("update-refs file at '%s' is invalid"),
+				  path);
+			result = -1;
+			goto cleanup;
+		}
+
+		item = string_list_append(refs, ref.buf);
+		item->util = oiddup(&oid);
+	}
+
+cleanup:
+	if (fp)
+		fclose(fp);
+	free(path);
+	strbuf_release(&ref);
+	strbuf_release(&hash);
+	return result;
+}
diff --git a/sequencer.h b/sequencer.h
index da64473636b..3ae541bb145 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -232,4 +232,13 @@ void sequencer_post_commit_cleanup(struct repository *r, int verbose);
 int sequencer_get_last_command(struct repository* r,
 			       enum replay_action *action);
 int sequencer_determine_whence(struct repository *r, enum commit_whence *whence);
+
+/**
+ * Append the set of ref-OID pairs that are currently stored for the 'git
+ * rebase --update-refs' feature if such a rebase is currently happening.
+ *
+ * Localized to a worktree's git dir.
+ */
+int sequencer_get_update_refs_state(const char *wt_dir, struct string_list *refs);
+
 #endif /* SEQUENCER_H */
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 4f59bc21303..f1e9e172a0c 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -7,8 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true
 
 test_expect_success 'setup' '
 	test_commit init &&
-	git branch -f fake-1 &&
-	git branch -f fake-2 &&
+
+	for i in 1 2 3 4
+	do
+		git branch -f fake-$i || return 1
+	done &&
 
 	for i in 1 2 3 4
 	do
@@ -73,8 +76,19 @@ test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
 	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-merge/head-name &&
 	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-merge/onto &&
 
-	test_must_fail git branch -f fake-1 HEAD 2>err &&
-	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
+	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
+	refs/heads/fake-3
+	$(git rev-parse HEAD~1)
+	refs/heads/fake-4
+	$(git rev-parse HEAD)
+	EOF
+
+	for i in 1 3 4
+	do
+		test_must_fail git branch -f fake-$i HEAD 2>err &&
+		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+			return 1
+	done
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
-- 
gitgitgadget

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The branch_checked_out() helper helps commands like 'git branch' and
> 'git fetch' from overwriting refs that are currently checked out in
> other worktrees.
>
> A future update to 'git rebase' will introduce a new '--update-refs'
> option which will update the local refs that point to commits that are
> being rebased. To avoid collisions as the rebase completes, we want to
> make the future data store for these refs to be considered by
> branch_checked_out().
>
> The data store is a plaintext file inside the 'rebase-merge' directory
> for that worktree. The file alternates refnames and OIDs. The OIDs will
> be used to store the to-be-written values as the rebase progresses, but
> can be ignored at the moment.
>
> Create a new sequencer_get_update_refs_state() method that parses this
> file and populates a struct string_list with the ref-OID pairs. We can
> then use this list to add to the current_checked_out_branches strmap
> used by branch_checked_out().
>
> To properly navigate to the rebase directory for a given worktree,
> extract the static strbuf_worktree_gitdir() method to a public API
> method.
>
> We can test that this works without having Git write this file by
> artificially creating one in our test script.

Hmph, I am not thrilled to see an ad-hoc text file for things like
this.  Do the objects that appear in this file otherwise already
anchored by existing refs, or can some of them be "floating" without
any refs pointing at them, i.e. making the "connected" and "fsck"
machinery also being aware of them to protect them from collected as
garbage and reported as dangling/unreachable?

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Derrick Stolee]

On 6/28/22 4:48 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> The branch_checked_out() helper helps commands like 'git branch' and
>> 'git fetch' from overwriting refs that are currently checked out in
>> other worktrees.
>>
>> A future update to 'git rebase' will introduce a new '--update-refs'
>> option which will update the local refs that point to commits that are
>> being rebased. To avoid collisions as the rebase completes, we want to
>> make the future data store for these refs to be considered by
>> branch_checked_out().
>>
>> The data store is a plaintext file inside the 'rebase-merge' directory
>> for that worktree. The file alternates refnames and OIDs. The OIDs will
>> be used to store the to-be-written values as the rebase progresses, but
>> can be ignored at the moment.
>>
>> Create a new sequencer_get_update_refs_state() method that parses this
>> file and populates a struct string_list with the ref-OID pairs. We can
>> then use this list to add to the current_checked_out_branches strmap
>> used by branch_checked_out().
>>
>> To properly navigate to the rebase directory for a given worktree,
>> extract the static strbuf_worktree_gitdir() method to a public API
>> method.
>>
>> We can test that this works without having Git write this file by
>> artificially creating one in our test script.
> 
> Hmph, I am not thrilled to see an ad-hoc text file for things like
> this.  Do the objects that appear in this file otherwise already
> anchored by existing refs, or can some of them be "floating" without
> any refs pointing at them, i.e. making the "connected" and "fsck"
> machinery also being aware of them to protect them from collected as
> garbage and reported as dangling/unreachable?

You're right that we could have this sequence of events in a todo
file:

  pick deadbeef Here is a commit that will become unreachable
  update-ref will-be-lost

  squash 1234567 make will-be-lost unreachable
  ...

So if a GC runs after this 'update-ref' step but before the rebase
completes, then that commit can be lost. The ref-update at the end
of the rebase process would fail.

I wonder how important such a situation is, though. But I'm willing
to add the extra lookups in 'git gc' and 'git fsck' to make this a
non-issue.

I'll take a look to see where the other refs in rebase-<backend>/*
are handled by these processes.

Thanks,
-Stolee

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Phillip Wood]

On 29/06/2022 13:58, Derrick Stolee wrote:
> On 6/28/22 4:48 PM, Junio C Hamano wrote:
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>>
>>> From: Derrick Stolee <derrickstolee@github.com>
>>>
>>> The branch_checked_out() helper helps commands like 'git branch' and
>>> 'git fetch' from overwriting refs that are currently checked out in
>>> other worktrees.
>>>
>>> A future update to 'git rebase' will introduce a new '--update-refs'
>>> option which will update the local refs that point to commits that are
>>> being rebased. To avoid collisions as the rebase completes, we want to
>>> make the future data store for these refs to be considered by
>>> branch_checked_out().
>>>
>>> The data store is a plaintext file inside the 'rebase-merge' directory
>>> for that worktree. The file alternates refnames and OIDs. The OIDs will
>>> be used to store the to-be-written values as the rebase progresses, but
>>> can be ignored at the moment.
>>>
>>> Create a new sequencer_get_update_refs_state() method that parses this
>>> file and populates a struct string_list with the ref-OID pairs. We can
>>> then use this list to add to the current_checked_out_branches strmap
>>> used by branch_checked_out().
>>>
>>> To properly navigate to the rebase directory for a given worktree,
>>> extract the static strbuf_worktree_gitdir() method to a public API
>>> method.
>>>
>>> We can test that this works without having Git write this file by
>>> artificially creating one in our test script.
>>
>> Hmph, I am not thrilled to see an ad-hoc text file for things like
>> this.  Do the objects that appear in this file otherwise already
>> anchored by existing refs, or can some of them be "floating" without
>> any refs pointing at them, i.e. making the "connected" and "fsck"
>> machinery also being aware of them to protect them from collected as
>> garbage and reported as dangling/unreachable?
> 
> You're right that we could have this sequence of events in a todo
> file:
> 
>    pick deadbeef Here is a commit that will become unreachable
>    update-ref will-be-lost
> 
>    squash 1234567 make will-be-lost unreachable
>    ...
> 
> So if a GC runs after this 'update-ref' step but before the rebase
> completes, then that commit can be lost. The ref-update at the end
> of the rebase process would fail.

The commit is protected by HEAD's reflog for a while because we update 
HEAD after each pick but it is not permanently safe.

I've taken a look at the first few patches and they're looking good, I 
hope to look at the rest tomorrow

Best Wishes

Phillip

> I wonder how important such a situation is, though. But I'm willing
> to add the extra lookups in 'git gc' and 'git fsck' to make this a
> non-issue.
> 
> I'll take a look to see where the other refs in rebase-<backend>/*
> are handled by these processes.
> 
> Thanks,
> -Stolee

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> The commit is protected by HEAD's reflog for a while because we update
> HEAD after each pick but it is not permanently safe.

Ah, I forgot about that one.  Thanks for pointing it out.  It does
sound like HEAD's reflog makes it safe enough.

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> I wonder how important such a situation is, though. But I'm willing
> to add the extra lookups in 'git gc' and 'git fsck' to make this a
> non-issue.

What I was hinting at was if some of this can be done by using a
ref, so that we do not have to touch "gc" or "fsck" at all.

As to the importance, I would say it is about the same importance as
being able to prevent the branches involved in the rebase from
touched in other worktrees.  We expect it to take sufficiently large
wallclock time from the beginning of a rebase and its finish to make
such a protection necessary, so an auto-gc may even be started from
other worktrees without our knowledge.

Thanks.

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Phillip Wood]

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> The branch_checked_out() helper helps commands like 'git branch' and
> 'git fetch' from overwriting refs that are currently checked out in
> other worktrees.
> 
> A future update to 'git rebase' will introduce a new '--update-refs'
> option which will update the local refs that point to commits that are
> being rebased. To avoid collisions as the rebase completes, we want to
> make the future data store for these refs to be considered by
> branch_checked_out().
> 
> The data store is a plaintext file inside the 'rebase-merge' directory
> for that worktree. The file alternates refnames and OIDs. The OIDs will
> be used to store the to-be-written values as the rebase progresses, but
> can be ignored at the moment.
> 
> Create a new sequencer_get_update_refs_state() method that parses this
> file and populates a struct string_list with the ref-OID pairs. We can
> then use this list to add to the current_checked_out_branches strmap
> used by branch_checked_out().
> 
> To properly navigate to the rebase directory for a given worktree,
> extract the static strbuf_worktree_gitdir() method to a public API
> method.
> 
> We can test that this works without having Git write this file by
> artificially creating one in our test script.

This looks good apart from a couple of small nits

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   branch.c                  | 13 ++++++++++++
>   sequencer.c               | 42 +++++++++++++++++++++++++++++++++++++++
>   sequencer.h               |  9 +++++++++
>   t/t2407-worktree-heads.sh | 22 ++++++++++++++++----
>   4 files changed, 82 insertions(+), 4 deletions(-)
> 
> diff --git a/branch.c b/branch.c
> index 526e8237673..f252c4eef6c 100644
> --- a/branch.c
> +++ b/branch.c
> @@ -365,6 +365,7 @@ static void prepare_checked_out_branches(void)
>   		char *old;
>   		struct wt_status_state state = { 0 };
>   		struct worktree *wt = worktrees[i++];
> +		struct string_list update_refs = STRING_LIST_INIT_DUP;
>   
>   		if (wt->is_bare)
>   			continue;
> @@ -400,6 +401,18 @@ static void prepare_checked_out_branches(void)
>   			strbuf_release(&ref);
>   		}
>   		wt_status_state_free_buffers(&state);
> +
> +		if (!sequencer_get_update_refs_state(get_worktree_git_dir(wt),
> +						     &update_refs)) {
> +			struct string_list_item *item;
> +			for_each_string_list_item(item, &update_refs) {
> +				old = strmap_put(&current_checked_out_branches,
> +						 item->string,
> +						 xstrdup(wt->path));
> +				free(old);
> +			}
> +			string_list_clear(&update_refs, 1);
> +		}
>   	}
>   
>   	free_worktrees(worktrees);
> diff --git a/sequencer.c b/sequencer.c
> index 8c3ed3532ac..1094e146b99 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -5901,3 +5901,45 @@ int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
>   
>   	return 0;
>   }
> +
> +int sequencer_get_update_refs_state(const char *wt_dir,
> +				    struct string_list *refs)
> +{
> +	int result = 0;
> +	struct stat st;
> +	FILE *fp = NULL;
> +	struct strbuf ref = STRBUF_INIT;
> +	struct strbuf hash = STRBUF_INIT;
> +	char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);

I think it would make sense to introduce rebase_path_update_refs() in 
this patch rather than later in the series

> +
> +	if (stat(path, &st))
> +		goto cleanup;

What's the reason for stating the file first, rather than just letting 
fopen() fail if it does not exist?

Best Wishes

Phillip

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Derrick Stolee]

On 6/30/2022 5:32 AM, Phillip Wood wrote:
> On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>

>> +int sequencer_get_update_refs_state(const char *wt_dir,
>> +                    struct string_list *refs)
>> +{
>> +    int result = 0;
>> +    struct stat st;
>> +    FILE *fp = NULL;
>> +    struct strbuf ref = STRBUF_INIT;
>> +    struct strbuf hash = STRBUF_INIT;
>> +    char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);
> 
> I think it would make sense to introduce rebase_path_update_refs() in this patch rather than later in the series

The biggest difference is that rebase_path_update_refs() only
gives the path for the current worktree, while this method needs
to read the file for any worktree.

There is likely some opportunity to create rebase_path_update_refs()
in a different way that serves both purposes.

>> +
>> +    if (stat(path, &st))
>> +        goto cleanup;
> 
> What's the reason for stating the file first, rather than just letting fopen() fail if it does not exist?

Not sure what I was looking at that gave this pattern, but you're
right that it isn't necessary.

Thanks,
-Stolee

Re: [PATCH v3 2/8] branch: consider refs under 'update-refs'

[Phillip Wood]

Hi Stolee

On 30/06/2022 14:35, Derrick Stolee wrote:
> On 6/30/2022 5:32 AM, Phillip Wood wrote:
>> On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
>>> From: Derrick Stolee <derrickstolee@github.com>
> 
>>> +int sequencer_get_update_refs_state(const char *wt_dir,
>>> +                    struct string_list *refs)
>>> +{
>>> +    int result = 0;
>>> +    struct stat st;
>>> +    FILE *fp = NULL;
>>> +    struct strbuf ref = STRBUF_INIT;
>>> +    struct strbuf hash = STRBUF_INIT;
>>> +    char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);
>>
>> I think it would make sense to introduce rebase_path_update_refs() in this patch rather than later in the series
> 
> The biggest difference is that rebase_path_update_refs() only
> gives the path for the current worktree, while this method needs
> to read the file for any worktree.

That's an important distinction that I'd failed to notice

> There is likely some opportunity to create rebase_path_update_refs()
> in a different way that serves both purposes.

That would be nice, even just having a #define for 
"rebase-merge/update-refs" and using that here and in the other patch 
would mean we're not hard coding the filename in two different places.

Best Wishes

Phillip

>>> +
>>> +    if (stat(path, &st))
>>> +        goto cleanup;
>>
>> What's the reason for stating the file first, rather than just letting fopen() fail if it does not exist?
> 
> Not sure what I was looking at that gave this pattern, but you're
> right that it isn't necessary.
> 
> Thanks,
> -Stolee

[PATCH v3 3/8] rebase-interactive: update 'merge' description

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The 'merge' command description for the todo list documentation in an
interactive rebase has multiple lines. The lines other than the first
one start with dots ('.') while the similar multi-line documentation for
'fixup' does not.

The 'merge' command was documented when interactive rebase was first
ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
2018-08-10). These dots might have been carried over from the previous
shell implementation.

The 'fixup' command was documented more recently in 9e3cebd97cb (rebase
-i: add fixup [-C | -c] command, 2021-01-29).

Looking at the output in an editor, my personal opinion is that the dots
are unnecessary and noisy. Remove them now before adding more commands
with multi-line documentation.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 87649d0c016..22394224faa 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -54,9 +54,9 @@ void append_todo_help(int command_count,
 "l, label <label> = label current HEAD with a name\n"
 "t, reset <label> = reset HEAD to a label\n"
 "m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n"
-".       create a merge commit using the original merge commit's\n"
-".       message (or the oneline, if no original merge commit was\n"
-".       specified); use -c <commit> to reword the commit message\n"
+"        create a merge commit using the original merge commit's\n"
+"        message (or the oneline, if no original merge commit was\n"
+"        specified); use -c <commit> to reword the commit message\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
-- 
gitgitgadget

Re: [PATCH v3 3/8] rebase-interactive: update 'merge' description

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The 'merge' command description for the todo list documentation in an
> interactive rebase has multiple lines. The lines other than the first
> one start with dots ('.') while the similar multi-line documentation for
> 'fixup' does not.
>
> The 'merge' command was documented when interactive rebase was first
> ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
> 2018-08-10). These dots might have been carried over from the previous
> shell implementation.

The text indeed does come literally from the block removed by that
commit.  I wondered if the shell "gettext" in git-i18n.sh had some
magic in it, but it does not seem to have anything, and po/git.pot
around that time has these lines with leading dots, so I suspect
that they were quite deliberately added, not for the reasons of
formatting machinery (e.g. preventing somebody in the dataflow from
losing leading indentation), but to show them to the end users.

Unfortunately, the offending commit 4c68e7dd (sequencer: introduce
the `merge` command, 2018-04-25) does not justify them X-<.

> Looking at the output in an editor, my personal opinion is that the dots
> are unnecessary and noisy. Remove them now before adding more commands
> with multi-line documentation.

I personally do not mind having them in the UI, but I can also be
happy to see them go.  It is unlikely that any program is consuming
these strings, so I would say this is a fairly safe clean-up.

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  rebase-interactive.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/rebase-interactive.c b/rebase-interactive.c
> index 87649d0c016..22394224faa 100644
> --- a/rebase-interactive.c
> +++ b/rebase-interactive.c
> @@ -54,9 +54,9 @@ void append_todo_help(int command_count,
>  "l, label <label> = label current HEAD with a name\n"
>  "t, reset <label> = reset HEAD to a label\n"
>  "m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n"
> -".       create a merge commit using the original merge commit's\n"
> -".       message (or the oneline, if no original merge commit was\n"
> -".       specified); use -c <commit> to reword the commit message\n"
> +"        create a merge commit using the original merge commit's\n"
> +"        message (or the oneline, if no original merge commit was\n"
> +"        specified); use -c <commit> to reword the commit message\n"
>  "\n"
>  "These lines can be re-ordered; they are executed from top to bottom.\n");
>  	unsigned edit_todo = !(shortrevisions && shortonto);

Re: [PATCH v3 3/8] rebase-interactive: update 'merge' description

[Derrick Stolee]

On 6/28/22 5:00 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> The 'merge' command description for the todo list documentation in an
>> interactive rebase has multiple lines. The lines other than the first
>> one start with dots ('.') while the similar multi-line documentation for
>> 'fixup' does not.
>>
>> The 'merge' command was documented when interactive rebase was first
>> ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
>> 2018-08-10). These dots might have been carried over from the previous
>> shell implementation.
> 
> The text indeed does come literally from the block removed by that
> commit.  I wondered if the shell "gettext" in git-i18n.sh had some
> magic in it, but it does not seem to have anything, and po/git.pot
> around that time has these lines with leading dots, so I suspect
> that they were quite deliberately added, not for the reasons of
> formatting machinery (e.g. preventing somebody in the dataflow from
> losing leading indentation), but to show them to the end users.
> 
> Unfortunately, the offending commit 4c68e7dd (sequencer: introduce
> the `merge` command, 2018-04-25) does not justify them X-<.
> 
>> Looking at the output in an editor, my personal opinion is that the dots
>> are unnecessary and noisy. Remove them now before adding more commands
>> with multi-line documentation.
> 
> I personally do not mind having them in the UI, but I can also be
> happy to see them go.  It is unlikely that any program is consuming
> these strings, so I would say this is a fairly safe clean-up.

Perhaps I should be a bit clearer that these are appearing in the
comment section of the todo-file when presented to the user for editing:

# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
#                    commit's log message, unless -C is used, in which case
#                    keep only this commit's message; -c is same as -C but
#                    opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified); use -c <commit> to reword the commit message
#
# These lines can be re-ordered; they are executed from top to bottom.
#

This does not appear to be used anywhere else.

Thanks,
-Stolee

Re: [PATCH v3 3/8] rebase-interactive: update 'merge' description

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> Perhaps I should be a bit clearer that these are appearing in the
> comment section of the todo-file when presented to the user for editing:

Yup, it is good to spell it out for others.

> This does not appear to be used anywhere else.

Nowhere in our codebase.  I do not think we have a way to know if
users (e.g. hooks and scripts) depend on them, but I am reasonably
sure that nobody would complain ;-)

Re: [PATCH v3 3/8] rebase-interactive: update 'merge' description

[Phillip Wood]

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> The 'merge' command description for the todo list documentation in an
> interactive rebase has multiple lines. The lines other than the first
> one start with dots ('.') while the similar multi-line documentation for
> 'fixup' does not.
> 
> The 'merge' command was documented when interactive rebase was first
> ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
> 2018-08-10). These dots might have been carried over from the previous
> shell implementation.
> 
> The 'fixup' command was documented more recently in 9e3cebd97cb (rebase
> -i: add fixup [-C | -c] command, 2021-01-29).
> 
> Looking at the output in an editor, my personal opinion is that the dots
> are unnecessary and noisy. Remove them now before adding more commands
> with multi-line documentation.

I agree, this is a nice cleanup

Thanks

Phillip

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   rebase-interactive.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/rebase-interactive.c b/rebase-interactive.c
> index 87649d0c016..22394224faa 100644
> --- a/rebase-interactive.c
> +++ b/rebase-interactive.c
> @@ -54,9 +54,9 @@ void append_todo_help(int command_count,
>   "l, label <label> = label current HEAD with a name\n"
>   "t, reset <label> = reset HEAD to a label\n"
>   "m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n"
> -".       create a merge commit using the original merge commit's\n"
> -".       message (or the oneline, if no original merge commit was\n"
> -".       specified); use -c <commit> to reword the commit message\n"
> +"        create a merge commit using the original merge commit's\n"
> +"        message (or the oneline, if no original merge commit was\n"
> +"        specified); use -c <commit> to reword the commit message\n"
>   "\n"
>   "These lines can be re-ordered; they are executed from top to bottom.\n");
>   	unsigned edit_todo = !(shortrevisions && shortonto);

[PATCH v3 4/8] sequencer: define array with enum values

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The todo_command_info array defines which strings match with which
todo_command enum values. The array is defined in the same order as the
enum values, but if one changed without the other, then we would have
unexpected results.

Make it easier to see changes to the enum and this array by using the
enum values as the indices of the array.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 1094e146b99..2ffee79619c 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1685,20 +1685,20 @@ static struct {
 	char c;
 	const char *str;
 } todo_command_info[] = {
-	{ 'p', "pick" },
-	{ 0,   "revert" },
-	{ 'e', "edit" },
-	{ 'r', "reword" },
-	{ 'f', "fixup" },
-	{ 's', "squash" },
-	{ 'x', "exec" },
-	{ 'b', "break" },
-	{ 'l', "label" },
-	{ 't', "reset" },
-	{ 'm', "merge" },
-	{ 0,   "noop" },
-	{ 'd', "drop" },
-	{ 0,   NULL }
+	[TODO_PICK] = { 'p', "pick" },
+	[TODO_REVERT] = { 0,   "revert" },
+	[TODO_EDIT] = { 'e', "edit" },
+	[TODO_REWORD] = { 'r', "reword" },
+	[TODO_FIXUP] = { 'f', "fixup" },
+	[TODO_SQUASH] = { 's', "squash" },
+	[TODO_EXEC] = { 'x', "exec" },
+	[TODO_BREAK] = { 'b', "break" },
+	[TODO_LABEL] = { 'l', "label" },
+	[TODO_RESET] = { 't', "reset" },
+	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_NOOP] = { 0,   "noop" },
+	[TODO_DROP] = { 'd', "drop" },
+	[TODO_COMMENT] = { 0,   NULL },
 };
 
 static const char *command_to_string(const enum todo_command command)
-- 
gitgitgadget

Re: [PATCH v3 4/8] sequencer: define array with enum values

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The todo_command_info array defines which strings match with which
> todo_command enum values. The array is defined in the same order as the
> enum values, but if one changed without the other, then we would have
> unexpected results.
>
> Make it easier to see changes to the enum and this array by using the
> enum values as the indices of the array.

OK.  It is a bit of shame that we cannot define this array and enum
at the same time in a single construct, but this is much safer than
the status quo.  Thanks.

>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  sequencer.c | 28 ++++++++++++++--------------
>  1 file changed, 14 insertions(+), 14 deletions(-)
>
> diff --git a/sequencer.c b/sequencer.c
> index 1094e146b99..2ffee79619c 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -1685,20 +1685,20 @@ static struct {
>  	char c;
>  	const char *str;
>  } todo_command_info[] = {
> -	{ 'p', "pick" },
> -	{ 0,   "revert" },
> -	{ 'e', "edit" },
> -	{ 'r', "reword" },
> -	{ 'f', "fixup" },
> -	{ 's', "squash" },
> -	{ 'x', "exec" },
> -	{ 'b', "break" },
> -	{ 'l', "label" },
> -	{ 't', "reset" },
> -	{ 'm', "merge" },
> -	{ 0,   "noop" },
> -	{ 'd', "drop" },
> -	{ 0,   NULL }
> +	[TODO_PICK] = { 'p', "pick" },
> +	[TODO_REVERT] = { 0,   "revert" },
> +	[TODO_EDIT] = { 'e', "edit" },
> +	[TODO_REWORD] = { 'r', "reword" },
> +	[TODO_FIXUP] = { 'f', "fixup" },
> +	[TODO_SQUASH] = { 's', "squash" },
> +	[TODO_EXEC] = { 'x', "exec" },
> +	[TODO_BREAK] = { 'b', "break" },
> +	[TODO_LABEL] = { 'l', "label" },
> +	[TODO_RESET] = { 't', "reset" },
> +	[TODO_MERGE] = { 'm', "merge" },
> +	[TODO_NOOP] = { 0,   "noop" },
> +	[TODO_DROP] = { 'd', "drop" },
> +	[TODO_COMMENT] = { 0,   NULL },
>  };
>  
>  static const char *command_to_string(const enum todo_command command)

[PATCH v3 5/8] sequencer: add update-ref command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

Add the boilerplate for an "update-ref" command in the sequencer. This
connects to the current no-op do_update_ref() which will be filled in
after more connections are created.

The syntax in the todo list will be "update-ref <ref-name>" to signal
that we should store the current commit as the value for updating
<ref-name> at the end of the rebase.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c |  3 +++
 sequencer.c          | 14 +++++++++++++-
 sequencer.h          |  1 +
 3 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 22394224faa..1ff07647af3 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -57,6 +57,9 @@ void append_todo_help(int command_count,
 "        create a merge commit using the original merge commit's\n"
 "        message (or the oneline, if no original merge commit was\n"
 "        specified); use -c <commit> to reword the commit message\n"
+"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
+"                      to this position in the new commits. The <ref> is\n"
+"                      updated at the end of the rebase\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
diff --git a/sequencer.c b/sequencer.c
index 2ffee79619c..0b61835d441 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1696,6 +1696,7 @@ static struct {
 	[TODO_LABEL] = { 'l', "label" },
 	[TODO_RESET] = { 't', "reset" },
 	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_UPDATE_REF] = { 'u', "update-ref" },
 	[TODO_NOOP] = { 0,   "noop" },
 	[TODO_DROP] = { 'd', "drop" },
 	[TODO_COMMENT] = { 0,   NULL },
@@ -2457,7 +2458,7 @@ static int parse_insn_line(struct repository *r, struct todo_item *item,
 			     command_to_string(item->command));
 
 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
-	    item->command == TODO_RESET) {
+	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
 		item->commit = NULL;
 		item->arg_offset = bol - buf;
 		item->arg_len = (int)(eol - bol);
@@ -4056,6 +4057,11 @@ leave_merge:
 	return ret;
 }
 
+static int do_update_ref(struct repository *r, const char *ref_name)
+{
+	return 0;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4431,6 +4437,12 @@ static int pick_commits(struct repository *r,
 				return error_with_patch(r, item->commit,
 							arg, item->arg_len,
 							opts, res, 0);
+		} else if (item->command == TODO_UPDATE_REF) {
+			struct strbuf ref = STRBUF_INIT;
+			strbuf_add(&ref, arg, item->arg_len);
+			if ((res = do_update_ref(r, ref.buf)))
+				reschedule = 1;
+			strbuf_release(&ref);
 		} else if (!is_noop(item->command))
 			return error(_("unknown command %d"), item->command);
 
diff --git a/sequencer.h b/sequencer.h
index 3ae541bb145..2cf5c1b9a38 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -95,6 +95,7 @@ enum todo_command {
 	TODO_LABEL,
 	TODO_RESET,
 	TODO_MERGE,
+	TODO_UPDATE_REF,
 	/* commands that do nothing but are counted for reporting progress */
 	TODO_NOOP,
 	TODO_DROP,
-- 
gitgitgadget

Re: [PATCH v3 5/8] sequencer: add update-ref command

[Phillip Wood]

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> Add the boilerplate for an "update-ref" command in the sequencer. This
> connects to the current no-op do_update_ref() which will be filled in
> after more connections are created.
> 
> The syntax in the todo list will be "update-ref <ref-name>" to signal
> that we should store the current commit as the value for updating
> <ref-name> at the end of the rebase.
> 
> [...]
> @@ -4431,6 +4437,12 @@ static int pick_commits(struct repository *r,
>   				return error_with_patch(r, item->commit,
>   							arg, item->arg_len,
>   							opts, res, 0);
> +		} else if (item->command == TODO_UPDATE_REF) {
> +			struct strbuf ref = STRBUF_INIT;
> +			strbuf_add(&ref, arg, item->arg_len);

It's a bit of a shame we have to faff here just to get a NULL terminated 
string. It does not matter but using xmemdupz() & free() would be one 
less line.

> +			if ((res = do_update_ref(r, ref.buf)))
> +				reschedule = 1;

Rescheduling here is good

Best Wishes

Phillip

[PATCH v3 6/8] rebase: add --update-refs option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When working on a large feature, it can be helpful to break that feature
into multiple smaller parts that become reviewed in sequence. During
development or during review, a change to one part of the feature could
affect multiple of these parts. An interactive rebase can help adjust
the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the
feature, then rebasing the entire list of commits can create commits not
reachable from those "sub branches". It can take a manual step to update
those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
<ref>' steps to the todo file whenever a commit that is being rebased is
decorated with that <ref>. At the very end, the rebase process updates
all of the listed refs to the values stored during the rebase operation.

Be sure to iterate after any squashing or fixups are placed. Update the
branch only after those squashes and fixups are complete. This allows a
--fixup commit at the tip of the feature to apply correctly to the sub
branch, even if it is fixing up the most-recent commit in that part.

One potential problem here is that refs decorating commits that are
already marked as "fixup!" or "squash!" will not be included in this
list. Generally, the reordering of the "fixup!" and "squash!" is likely
to change the relative order of these refs, so it is not recommended.
The workflow here is intended to allow these kinds of commits at the tip
of the rebased branch while the other sub branches come along for the
ride without intervention.

This change update the documentation and builtin to accept the
--update-refs option as well as updating the todo file with the
'update-ref' commands. Tests are added to ensure that these todo
commands are added in the correct locations.

A future change will update the behavior to actually update the refs
at the end of the rebase sequence.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-rebase.txt  |   8 +++
 builtin/rebase.c              |   5 ++
 sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
 sequencer.h                   |   1 +
 t/t2407-worktree-heads.sh     |  17 ++++++
 t/t3404-rebase-interactive.sh |  70 ++++++++++++++++++++++
 6 files changed, 208 insertions(+)

diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 262fb01aec0..e7611b4089c 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	or point to a `squash! ...` or `fixup! ...` commit are not updated
+	in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
@@ -632,6 +639,7 @@ are incompatible with the following options:
  * --empty=
  * --reapply-cherry-picks
  * --edit-todo
+ * --update-refs
  * --root when used in combination with --onto
 
 In addition, the following pairs of options are incompatible:
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 7ab50cda2ad..56d82a52106 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update local refs that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
diff --git a/sequencer.c b/sequencer.c
index 0b61835d441..915d87a0336 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -35,6 +35,7 @@
 #include "commit-reach.h"
 #include "rebase-interactive.h"
 #include "reset.h"
+#include "branch.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -5615,10 +5616,113 @@ static int skip_unnecessary_picks(struct repository *r,
 	return 0;
 }
 
+struct todo_add_branch_context {
+	struct todo_item *items;
+	size_t items_nr;
+	size_t items_alloc;
+	struct strbuf *buf;
+	struct commit *commit;
+	struct string_list refs_to_oids;
+};
+
+static int add_decorations_to_list(const struct commit *commit,
+				   struct todo_add_branch_context *ctx)
+{
+	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+
+	while (decoration) {
+		struct todo_item *item;
+		const char *path;
+		size_t base_offset = ctx->buf->len;
+
+		ALLOC_GROW(ctx->items,
+			ctx->items_nr + 1,
+			ctx->items_alloc);
+		item = &ctx->items[ctx->items_nr];
+		memset(item, 0, sizeof(*item));
+
+		/* If the branch is checked out, then leave a comment instead. */
+		if ((path = branch_checked_out(decoration->name))) {
+			item->command = TODO_COMMENT;
+			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
+				    decoration->name, path);
+		} else {
+			struct string_list_item *sti;
+			item->command = TODO_UPDATE_REF;
+			strbuf_addf(ctx->buf, "%s\n", decoration->name);
+
+			sti = string_list_append(&ctx->refs_to_oids,
+						 decoration->name);
+			sti->util = oiddup(the_hash_algo->null_oid);
+		}
+
+		item->offset_in_buf = base_offset;
+		item->arg_offset = base_offset;
+		item->arg_len = ctx->buf->len - base_offset;
+		ctx->items_nr++;
+
+		decoration = decoration->next;
+	}
+
+	return 0;
+}
+
+/*
+ * For each 'pick' command, find out if the commit has a decoration in
+ * refs/heads/. If so, then add a 'label for-update-refs/' command.
+ */
+static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
+{
+	int i;
+	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
+	struct decoration_filter decoration_filter = {
+		.include_ref_pattern = &decorate_refs_include,
+		.exclude_ref_pattern = &decorate_refs_exclude,
+		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
+	};
+	struct todo_add_branch_context ctx = {
+		.buf = &todo_list->buf,
+		.refs_to_oids = STRING_LIST_INIT_DUP,
+	};
+
+	ctx.items_alloc = 2 * todo_list->nr + 1;
+	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
+
+	string_list_append(&decorate_refs_include, "refs/heads/");
+	load_ref_decorations(&decoration_filter, 0);
+
+	for (i = 0; i < todo_list->nr; ) {
+		struct todo_item *item = &todo_list->items[i];
+
+		/* insert ith item into new list */
+		ALLOC_GROW(ctx.items,
+			   ctx.items_nr + 1,
+			   ctx.items_alloc);
+
+		ctx.items[ctx.items_nr++] = todo_list->items[i++];
+
+		if (item->commit) {
+			ctx.commit = item->commit;
+			add_decorations_to_list(item->commit, &ctx);
+		}
+	}
+
+	string_list_clear(&ctx.refs_to_oids, 1);
+	free(todo_list->items);
+	todo_list->items = ctx.items;
+	todo_list->nr = ctx.items_nr;
+	todo_list->alloc = ctx.items_alloc;
+
+	return 0;
+}
+
 int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list)
 {
 	char shortonto[GIT_MAX_HEXSZ + 1];
@@ -5637,6 +5741,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
 	}
 
+	if (update_refs && todo_list_add_update_ref_commands(todo_list))
+		return -1;
+
 	if (autosquash && todo_list_rearrange_squash(todo_list))
 		return -1;
 
diff --git a/sequencer.h b/sequencer.h
index 2cf5c1b9a38..e671d7e0743 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index f1e9e172a0c..203997d92c6 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -151,4 +151,21 @@ test_expect_success 'refuse to overwrite when in error states' '
 	done
 '
 
+. "$TEST_DIRECTORY"/lib-rebase.sh
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
+	git commit --fixup HEAD~2 --allow-empty &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		! grep "update-refs" todo
+	) &&
+	git branch -f allow-update HEAD~2 &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		grep "update-ref refs/heads/allow-update" todo
+	)
+'
+
 test_done
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index f31afd4a547..3cd20733bc8 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1743,6 +1743,76 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
 	test_cmp_rev ORIG_HEAD test-orig-head@{1}
 '
 
+test_expect_success '--update-refs adds label and update-ref commands' '
+	git checkout -b update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git commit --allow-empty --fixup=third &&
+	git branch -f shared-tip &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		pick $(git log -1 --format=%h J) J
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h K) K
+		pick $(git log -1 --format=%h L) L
+		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h M) M
+		update-ref refs/heads/no-conflict-branch
+		update-ref refs/heads/shared-tip
+		EOF
+
+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo
+	)
+'
+
+test_expect_success '--update-refs adds commands with --rebase-merges' '
+	git checkout -b update-refs-with-merge no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git merge -m merge branch2 &&
+	git branch -f merge-branch &&
+	git commit --fixup=third --allow-empty &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		label onto
+		reset onto
+		pick $(git log -1 --format=%h branch2~1) F
+		pick $(git log -1 --format=%h branch2) I
+		update-ref refs/heads/branch2
+		label merge
+		reset onto
+		pick $(git log -1 --format=%h refs/heads/second) J
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h refs/heads/third~1) K
+		pick $(git log -1 --format=%h refs/heads/third) L
+		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h HEAD~2) M
+		update-ref refs/heads/no-conflict-branch
+		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
+		update-ref refs/heads/merge-branch
+		EOF
+
+		test_must_fail git rebase -i --autosquash \
+				   --rebase-merges=rebase-cousins \
+				   --update-refs primary >todo &&
+
+		test_cmp expect todo
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> When working on a large feature, it can be helpful to break that feature
> into multiple smaller parts that become reviewed in sequence. During
> development or during review, a change to one part of the feature could
> affect multiple of these parts. An interactive rebase can help adjust
> the multi-part "story" of the branch.
>
> However, if there are branches tracking the different parts of the
> feature, then rebasing the entire list of commits can create commits not
> reachable from those "sub branches". It can take a manual step to update
> those branches.
>
> Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
> <ref>' steps to the todo file whenever a commit that is being rebased is
> decorated with that <ref>. At the very end, the rebase process updates
> all of the listed refs to the values stored during the rebase operation.
>
> Be sure to iterate after any squashing or fixups are placed. Update the
> branch only after those squashes and fixups are complete. This allows a
> --fixup commit at the tip of the feature to apply correctly to the sub
> branch, even if it is fixing up the most-recent commit in that part.
>
> One potential problem here is that refs decorating commits that are
> already marked as "fixup!" or "squash!" will not be included in this
> list. Generally, the reordering of the "fixup!" and "squash!" is likely
> to change the relative order of these refs, so it is not recommended.
> The workflow here is intended to allow these kinds of commits at the tip
> of the rebased branch while the other sub branches come along for the
> ride without intervention.
>
> This change update the documentation and builtin to accept the
> --update-refs option as well as updating the todo file with the
> 'update-ref' commands. Tests are added to ensure that these todo
> commands are added in the correct locations.

OK, so command line option is `--update-refs` because the range
being rebased could contain more than one branch tips, and the
option causes one `update-ref` insn per such ref left in the todo
list?  I had to read the above twice to remind me what was going on.

The design looks OK here but has "rebase-merge/update-refs" already
came into the picture, or will it be in future steps?  I lost track.

Let's read on.

Thanks.

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Derrick Stolee]

On 6/28/22 5:09 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
...
>> This change update the documentation and builtin to accept the
>> --update-refs option as well as updating the todo file with the
>> 'update-ref' commands. Tests are added to ensure that these todo
>> commands are added in the correct locations.
> 
> OK, so command line option is `--update-refs` because the range
> being rebased could contain more than one branch tips, and the
> option causes one `update-ref` insn per such ref left in the todo
> list?  I had to read the above twice to remind me what was going on.
> 
> The design looks OK here but has "rebase-merge/update-refs" already
> came into the picture, or will it be in future steps?  I lost track.

Sorry, we have the ability to read the file (and tests that read an
artificial version of the file for the branch_checked_out() logic) but
have not written it from within Git yet.

I can make it clearer what is _not_ in this patch and coming in a
future change.

Thanks,
-Stolee

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Phillip Wood]

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> When working on a large feature, it can be helpful to break that feature
> into multiple smaller parts that become reviewed in sequence. During
> development or during review, a change to one part of the feature could
> affect multiple of these parts. An interactive rebase can help adjust
> the multi-part "story" of the branch.
> 
> However, if there are branches tracking the different parts of the
> feature, then rebasing the entire list of commits can create commits not
> reachable from those "sub branches". It can take a manual step to update
> those branches.
> 
> Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
> <ref>' steps to the todo file whenever a commit that is being rebased is
> decorated with that <ref>. At the very end, the rebase process updates
> all of the listed refs to the values stored during the rebase operation.
> 
> Be sure to iterate after any squashing or fixups are placed. Update the
> branch only after those squashes and fixups are complete. This allows a
> --fixup commit at the tip of the feature to apply correctly to the sub
> branch, even if it is fixing up the most-recent commit in that part.
> 
> One potential problem here is that refs decorating commits that are
> already marked as "fixup!" or "squash!" will not be included in this
> list. Generally, the reordering of the "fixup!" and "squash!" is likely
> to change the relative order of these refs, so it is not recommended.
> The workflow here is intended to allow these kinds of commits at the tip
> of the rebased branch while the other sub branches come along for the
> ride without intervention.
> 
> This change update the documentation and builtin to accept the
> --update-refs option as well as updating the todo file with the
> 'update-ref' commands. Tests are added to ensure that these todo
> commands are added in the correct locations.
> 
> A future change will update the behavior to actually update the refs
> at the end of the rebase sequence.

This looks quite good to me, my main concern is that we don't check what 
the user did when they edited the todo list which leaves a potentially 
long window when another process could update a branch that is being 
rebased. I've left some comments below, they're mostly me talking to 
myself while reading the code.

> +struct todo_add_branch_context {
> +	struct todo_item *items;
> +	size_t items_nr;
> +	size_t items_alloc;
> +	struct strbuf *buf;
> +	struct commit *commit;
> +	struct string_list refs_to_oids;
> +};
> +
> +static int add_decorations_to_list(const struct commit *commit,
> +				   struct todo_add_branch_context *ctx)
> +{
> +	const struct name_decoration *decoration = get_name_decoration(&commit->object);
> +
> +	while (decoration) {
> +		struct todo_item *item;
> +		const char *path;
> +		size_t base_offset = ctx->buf->len;
> +
> +		ALLOC_GROW(ctx->items,
> +			ctx->items_nr + 1,
> +			ctx->items_alloc);
> +		item = &ctx->items[ctx->items_nr];
> +		memset(item, 0, sizeof(*item));
> +
> +		/* If the branch is checked out, then leave a comment instead. */
> +		if ((path = branch_checked_out(decoration->name))) {
> +			item->command = TODO_COMMENT;
> +			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
> +				    decoration->name, path);
> +		} else {
> +			struct string_list_item *sti;
> +			item->command = TODO_UPDATE_REF;
> +			strbuf_addf(ctx->buf, "%s\n", decoration->name);
> +
> +			sti = string_list_append(&ctx->refs_to_oids,
> +						 decoration->name);
> +			sti->util = oiddup(the_hash_algo->null_oid);

We don't record where the branch is pointing, but we do stop checkout 
from checking out the branch. There is a small race window where another 
process can checkout the branch (or start rebasing a branch with 'git 
rebase <upstream> <branch>') after we've added it to this list but have 
not written it to disk.

> +		}
> +
> +		item->offset_in_buf = base_offset;
> +		item->arg_offset = base_offset;
> +		item->arg_len = ctx->buf->len - base_offset;
> +		ctx->items_nr++;
> +
> +		decoration = decoration->next;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * For each 'pick' command, find out if the commit has a decoration in
> + * refs/heads/. If so, then add a 'label for-update-refs/' command.
> + */
> +static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
> +{
> +	int i;
> +	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
> +	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
> +	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
> +	struct decoration_filter decoration_filter = {
> +		.include_ref_pattern = &decorate_refs_include,
> +		.exclude_ref_pattern = &decorate_refs_exclude,
> +		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
> +	};
> +	struct todo_add_branch_context ctx = {
> +		.buf = &todo_list->buf,
> +		.refs_to_oids = STRING_LIST_INIT_DUP,
> +	};
> +
> +	ctx.items_alloc = 2 * todo_list->nr + 1;

We pre-allocate enough space for each commit to have an update-refs 
command added after it ...

> +	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
> +
> +	string_list_append(&decorate_refs_include, "refs/heads/");
> +	load_ref_decorations(&decoration_filter, 0);
> +
> +	for (i = 0; i < todo_list->nr; ) {
> +		struct todo_item *item = &todo_list->items[i];
> +
> +		/* insert ith item into new list */
> +		ALLOC_GROW(ctx.items,
> +			   ctx.items_nr + 1,
> +			   ctx.items_alloc);

But we are careful to grow the array if that space is not enough.

> +		ctx.items[ctx.items_nr++] = todo_list->items[i++];
> +
> +		if (item->commit) {
> +			ctx.commit = item->commit;
> +			add_decorations_to_list(item->commit, &ctx);
> +		}
> +	}
> +
> +	string_list_clear(&ctx.refs_to_oids, 1);
> +	free(todo_list->items);
> +	todo_list->items = ctx.items;
> +	todo_list->nr = ctx.items_nr;
> +	todo_list->alloc = ctx.items_alloc;
> +
> +	return 0;
> +}
> +
>   int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
>   		    const char *shortrevisions, const char *onto_name,
>   		    struct commit *onto, const struct object_id *orig_head,
>   		    struct string_list *commands, unsigned autosquash,
> +		    unsigned update_refs,
>   		    struct todo_list *todo_list)
>   {
>   	char shortonto[GIT_MAX_HEXSZ + 1];
> @@ -5637,6 +5741,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
>   		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
>   	}
>   
> +	if (update_refs && todo_list_add_update_ref_commands(todo_list))
> +		return -1;
> +

We add the update-ref commands before the user edits the todo list but 
do not check if they have added or removed any update-ref commands when 
they edit it. This means we may lock a branch unnecessarily if they 
deleted an update-ref command or fail to lock a branch in a timely 
manner if they add an update-ref command. It will eventually be locked 
when we process the update-ref command during the rebase but that could 
be quite a bit later if there are conflicts.

If we were to add the update-ref commands before the user edited the 
list and then write the update-refs file after it has been edited that 
would avoid some of these problems but leaves a bigger race window when 
two process think they can safely update a branch. That could be 
addressed by checking the other worktrees again once we have written the 
file which we really need to do in order to check the user hasn't added 
an update-ref command for a branch that is already being rebased in 
another worktree.

Best Wishes

Phillip

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Elijah Newren]

On Tue, Jun 28, 2022 at 6:26 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <derrickstolee@github.com>
>
> When working on a large feature, it can be helpful to break that feature
> into multiple smaller parts that become reviewed in sequence. During
> development or during review, a change to one part of the feature could
> affect multiple of these parts. An interactive rebase can help adjust
> the multi-part "story" of the branch.
>
> However, if there are branches tracking the different parts of the
> feature, then rebasing the entire list of commits can create commits not
> reachable from those "sub branches". It can take a manual step to update
> those branches.
>
> Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
> <ref>' steps to the todo file whenever a commit that is being rebased is
> decorated with that <ref>. At the very end, the rebase process updates
> all of the listed refs to the values stored during the rebase operation.
>
> Be sure to iterate after any squashing or fixups are placed. Update the
> branch only after those squashes and fixups are complete. This allows a
> --fixup commit at the tip of the feature to apply correctly to the sub
> branch, even if it is fixing up the most-recent commit in that part.
>
> One potential problem here is that refs decorating commits that are
> already marked as "fixup!" or "squash!" will not be included in this
> list. Generally, the reordering of the "fixup!" and "squash!" is likely
> to change the relative order of these refs, so it is not recommended.
> The workflow here is intended to allow these kinds of commits at the tip
> of the rebased branch while the other sub branches come along for the
> ride without intervention.
>
> This change update the documentation and builtin to accept the
> --update-refs option as well as updating the todo file with the
> 'update-ref' commands. Tests are added to ensure that these todo
> commands are added in the correct locations.
>
> A future change will update the behavior to actually update the refs
> at the end of the rebase sequence.
>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  Documentation/git-rebase.txt  |   8 +++
>  builtin/rebase.c              |   5 ++
>  sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
>  sequencer.h                   |   1 +
>  t/t2407-worktree-heads.sh     |  17 ++++++
>  t/t3404-rebase-interactive.sh |  70 ++++++++++++++++++++++
>  6 files changed, 208 insertions(+)
>
> diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
> index 262fb01aec0..e7611b4089c 100644
> --- a/Documentation/git-rebase.txt
> +++ b/Documentation/git-rebase.txt
> @@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
>  start would be overridden by the presence of
>  `rebase.rescheduleFailedExec=true` configuration.
>
> +--update-refs::
> +--no-update-refs::
> +       Automatically force-update any branches that point to commits that
> +       are being rebased. Any branches that are checked out in a worktree
> +       or point to a `squash! ...` or `fixup! ...` commit are not updated
> +       in this way.
> +
>  INCOMPATIBLE OPTIONS
>  --------------------
>
> @@ -632,6 +639,7 @@ are incompatible with the following options:
>   * --empty=
>   * --reapply-cherry-picks
>   * --edit-todo
> + * --update-refs
>   * --root when used in combination with --onto
>
>  In addition, the following pairs of options are incompatible:
> diff --git a/builtin/rebase.c b/builtin/rebase.c
> index 7ab50cda2ad..56d82a52106 100644
> --- a/builtin/rebase.c
> +++ b/builtin/rebase.c
> @@ -102,6 +102,7 @@ struct rebase_options {
>         int reschedule_failed_exec;
>         int reapply_cherry_picks;
>         int fork_point;
> +       int update_refs;
>  };
>
>  #define REBASE_OPTIONS_INIT {                          \
> @@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
>                 ret = complete_action(the_repository, &replay, flags,
>                         shortrevisions, opts->onto_name, opts->onto,
>                         &opts->orig_head, &commands, opts->autosquash,
> +                       opts->update_refs,
>                         &todo_list);
>         }
>
> @@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
>                 OPT_BOOL(0, "autosquash", &options.autosquash,
>                          N_("move commits that begin with "
>                             "squash!/fixup! under -i")),
> +               OPT_BOOL(0, "update-refs", &options.update_refs,
> +                        N_("update local refs that point to commits "
> +                           "that are being rebased")),
>                 { OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
>                         N_("GPG-sign commits"),
>                         PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
> diff --git a/sequencer.c b/sequencer.c
> index 0b61835d441..915d87a0336 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -35,6 +35,7 @@
>  #include "commit-reach.h"
>  #include "rebase-interactive.h"
>  #include "reset.h"
> +#include "branch.h"
>
>  #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>
> @@ -5615,10 +5616,113 @@ static int skip_unnecessary_picks(struct repository *r,
>         return 0;
>  }
>
> +struct todo_add_branch_context {
> +       struct todo_item *items;
> +       size_t items_nr;
> +       size_t items_alloc;
> +       struct strbuf *buf;
> +       struct commit *commit;
> +       struct string_list refs_to_oids;
> +};
> +
> +static int add_decorations_to_list(const struct commit *commit,
> +                                  struct todo_add_branch_context *ctx)
> +{
> +       const struct name_decoration *decoration = get_name_decoration(&commit->object);
> +
> +       while (decoration) {
> +               struct todo_item *item;
> +               const char *path;
> +               size_t base_offset = ctx->buf->len;
> +
> +               ALLOC_GROW(ctx->items,
> +                       ctx->items_nr + 1,
> +                       ctx->items_alloc);
> +               item = &ctx->items[ctx->items_nr];
> +               memset(item, 0, sizeof(*item));
> +
> +               /* If the branch is checked out, then leave a comment instead. */
> +               if ((path = branch_checked_out(decoration->name))) {
> +                       item->command = TODO_COMMENT;
> +                       strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
> +                                   decoration->name, path);
> +               } else {
> +                       struct string_list_item *sti;
> +                       item->command = TODO_UPDATE_REF;
> +                       strbuf_addf(ctx->buf, "%s\n", decoration->name);
> +
> +                       sti = string_list_append(&ctx->refs_to_oids,
> +                                                decoration->name);
> +                       sti->util = oiddup(the_hash_algo->null_oid);
> +               }
> +
> +               item->offset_in_buf = base_offset;
> +               item->arg_offset = base_offset;
> +               item->arg_len = ctx->buf->len - base_offset;
> +               ctx->items_nr++;
> +
> +               decoration = decoration->next;
> +       }
> +
> +       return 0;
> +}
> +
> +/*
> + * For each 'pick' command, find out if the commit has a decoration in

Is this really limited to picks?  If someone uses --autosquash and has
a fixup or squash in the list, wouldn't this apply as well, or does
all of this apply before the transformations to fixup/squash?  Also,
what if the user is doing --rebase-merges and there's a merge commit
with a branch pointing at the merge.   Would that be included?

> + * refs/heads/. If so, then add a 'label for-update-refs/' command.
> + */
> +static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
> +{
> +       int i;
> +       static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
> +       static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
> +       static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
> +       struct decoration_filter decoration_filter = {
> +               .include_ref_pattern = &decorate_refs_include,
> +               .exclude_ref_pattern = &decorate_refs_exclude,
> +               .exclude_ref_config_pattern = &decorate_refs_exclude_config,
> +       };
> +       struct todo_add_branch_context ctx = {
> +               .buf = &todo_list->buf,
> +               .refs_to_oids = STRING_LIST_INIT_DUP,
> +       };
> +
> +       ctx.items_alloc = 2 * todo_list->nr + 1;
> +       ALLOC_ARRAY(ctx.items, ctx.items_alloc);
> +
> +       string_list_append(&decorate_refs_include, "refs/heads/");
> +       load_ref_decorations(&decoration_filter, 0);
> +
> +       for (i = 0; i < todo_list->nr; ) {
> +               struct todo_item *item = &todo_list->items[i];
> +
> +               /* insert ith item into new list */
> +               ALLOC_GROW(ctx.items,
> +                          ctx.items_nr + 1,
> +                          ctx.items_alloc);
> +
> +               ctx.items[ctx.items_nr++] = todo_list->items[i++];
> +
> +               if (item->commit) {
> +                       ctx.commit = item->commit;
> +                       add_decorations_to_list(item->commit, &ctx);
> +               }
> +       }
> +
> +       string_list_clear(&ctx.refs_to_oids, 1);
> +       free(todo_list->items);
> +       todo_list->items = ctx.items;
> +       todo_list->nr = ctx.items_nr;
> +       todo_list->alloc = ctx.items_alloc;
> +
> +       return 0;
> +}
> +
>  int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
>                     const char *shortrevisions, const char *onto_name,
>                     struct commit *onto, const struct object_id *orig_head,
>                     struct string_list *commands, unsigned autosquash,
> +                   unsigned update_refs,
>                     struct todo_list *todo_list)
>  {
>         char shortonto[GIT_MAX_HEXSZ + 1];
> @@ -5637,6 +5741,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
>                 item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
>         }
>
> +       if (update_refs && todo_list_add_update_ref_commands(todo_list))
> +               return -1;
> +
>         if (autosquash && todo_list_rearrange_squash(todo_list))
>                 return -1;
>
> diff --git a/sequencer.h b/sequencer.h
> index 2cf5c1b9a38..e671d7e0743 100644
> --- a/sequencer.h
> +++ b/sequencer.h
> @@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
>                     const char *shortrevisions, const char *onto_name,
>                     struct commit *onto, const struct object_id *orig_head,
>                     struct string_list *commands, unsigned autosquash,
> +                   unsigned update_refs,
>                     struct todo_list *todo_list);
>  int todo_list_rearrange_squash(struct todo_list *todo_list);
>
> diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
> index f1e9e172a0c..203997d92c6 100755
> --- a/t/t2407-worktree-heads.sh
> +++ b/t/t2407-worktree-heads.sh
> @@ -151,4 +151,21 @@ test_expect_success 'refuse to overwrite when in error states' '
>         done
>  '
>
> +. "$TEST_DIRECTORY"/lib-rebase.sh
> +
> +test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
> +       git commit --fixup HEAD~2 --allow-empty &&
> +       (
> +               set_cat_todo_editor &&
> +               test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
> +               ! grep "update-refs" todo
> +       ) &&
> +       git branch -f allow-update HEAD~2 &&
> +       (
> +               set_cat_todo_editor &&
> +               test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
> +               grep "update-ref refs/heads/allow-update" todo
> +       )
> +'
> +
>  test_done
> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index f31afd4a547..3cd20733bc8 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1743,6 +1743,76 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
>         test_cmp_rev ORIG_HEAD test-orig-head@{1}
>  '
>
> +test_expect_success '--update-refs adds label and update-ref commands' '
> +       git checkout -b update-refs no-conflict-branch &&
> +       git branch -f base HEAD~4 &&
> +       git branch -f first HEAD~3 &&
> +       git branch -f second HEAD~3 &&
> +       git branch -f third HEAD~1 &&
> +       git commit --allow-empty --fixup=third &&
> +       git branch -f shared-tip &&
> +       (
> +               set_cat_todo_editor &&
> +
> +               cat >expect <<-EOF &&
> +               pick $(git log -1 --format=%h J) J
> +               update-ref refs/heads/second
> +               update-ref refs/heads/first
> +               pick $(git log -1 --format=%h K) K
> +               pick $(git log -1 --format=%h L) L
> +               fixup $(git log -1 --format=%h update-refs) fixup! L # empty
> +               update-ref refs/heads/third
> +               pick $(git log -1 --format=%h M) M
> +               update-ref refs/heads/no-conflict-branch
> +               update-ref refs/heads/shared-tip
> +               EOF
> +
> +               test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
> +               test_cmp expect todo
> +       )
> +'
> +
> +test_expect_success '--update-refs adds commands with --rebase-merges' '
> +       git checkout -b update-refs-with-merge no-conflict-branch &&
> +       git branch -f base HEAD~4 &&
> +       git branch -f first HEAD~3 &&
> +       git branch -f second HEAD~3 &&
> +       git branch -f third HEAD~1 &&
> +       git merge -m merge branch2 &&
> +       git branch -f merge-branch &&
> +       git commit --fixup=third --allow-empty &&
> +       (
> +               set_cat_todo_editor &&
> +
> +               cat >expect <<-EOF &&
> +               label onto
> +               reset onto
> +               pick $(git log -1 --format=%h branch2~1) F
> +               pick $(git log -1 --format=%h branch2) I
> +               update-ref refs/heads/branch2
> +               label merge
> +               reset onto
> +               pick $(git log -1 --format=%h refs/heads/second) J
> +               update-ref refs/heads/second
> +               update-ref refs/heads/first
> +               pick $(git log -1 --format=%h refs/heads/third~1) K
> +               pick $(git log -1 --format=%h refs/heads/third) L
> +               fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
> +               update-ref refs/heads/third
> +               pick $(git log -1 --format=%h HEAD~2) M
> +               update-ref refs/heads/no-conflict-branch
> +               merge -C $(git log -1 --format=%h HEAD~1) merge # merge
> +               update-ref refs/heads/merge-branch
> +               EOF
> +
> +               test_must_fail git rebase -i --autosquash \
> +                                  --rebase-merges=rebase-cousins \
> +                                  --update-refs primary >todo &&
> +
> +               test_cmp expect todo
> +       )
> +'
> +
>  # This must be the last test in this file
>  test_expect_success '$EDITOR and friends are unchanged' '
>         test_editor_unchanged
> --
> gitgitgadget
>

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Derrick Stolee]

On 7/1/22 3:20 PM, Elijah Newren wrote:
>> +/*
>> + * For each 'pick' command, find out if the commit has a decoration in
> Is this really limited to picks?  If someone uses --autosquash and has
> a fixup or squash in the list, wouldn't this apply as well, or does
> all of this apply before the transformations to fixup/squash?  Also,
> what if the user is doing --rebase-merges and there's a merge commit
> with a branch pointing at the merge.   Would that be included?
> 
>> + * refs/heads/. If so, then add a 'label for-update-refs/' command.
>> + */

This is limited to picks (for now) mostly for the reason that the
fixup! and squash! commits are probably getting reordered, even if
they exist at the tip of some refs.

The workflow I am optimizing for is to create fixup! and squash!
commits at the tip of a long multi-topic series, and then those
get reordered into the list. In that case, the only ref that is
pointing at one of those commits is also the HEAD branch, so that
is not updated using this mechanism.

This is a case where I'm very interested in alternatives here, and
maybe I should clearly mark this option as experimental so we can
work out cases like this based on use.

Thanks,
-Stolee

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Elijah Newren]

On Mon, Jul 4, 2022 at 5:56 AM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 7/1/22 3:20 PM, Elijah Newren wrote:
> >> +/*
> >> + * For each 'pick' command, find out if the commit has a decoration in
> > Is this really limited to picks?  If someone uses --autosquash and has
> > a fixup or squash in the list, wouldn't this apply as well, or does
> > all of this apply before the transformations to fixup/squash?  Also,
> > what if the user is doing --rebase-merges and there's a merge commit
> > with a branch pointing at the merge.   Would that be included?
> >
> >> + * refs/heads/. If so, then add a 'label for-update-refs/' command.
> >> + */
>
> This is limited to picks (for now) mostly for the reason that the
> fixup! and squash! commits are probably getting reordered, even if
> they exist at the tip of some refs.
>
> The workflow I am optimizing for is to create fixup! and squash!
> commits at the tip of a long multi-topic series, and then those
> get reordered into the list. In that case, the only ref that is
> pointing at one of those commits is also the HEAD branch, so that
> is not updated using this mechanism.
>
> This is a case where I'm very interested in alternatives here, and
> maybe I should clearly mark this option as experimental so we can
> work out cases like this based on use.

So my question was that if I had the following `git log --oneline
origin/main..` output:

    555555 (HEAD -> branch2) fixup! second commit
    444444 fourth commit
    333333 third commit
    222222 (branch1) second commit
    111111 first commit

and I run `git rebase -i --autosquash --update-refs origin/main`, what
would I get?  Is branch1 updated to the rebase of 222222, or does it
include the amending from squashing 555555 into the rebased 222222?
Does branch2 get "left behind" because it wasn't a pick commit at the
tip of history, leading us to not update branch2 at all?  Or does it
get correctly pointed at the rebased version of 444444?

Actually, I checked out ds/rebase-update-ref just now to try it, and
it seems like it does the right thing:

    pick 111111 first commit
    pick 222222 second commit
    fixup 555555 fixup! second commit
    update-ref refs/heads/branch1

    pick 333333 third commit
    pick 444444 fourth commit
    # Ref refs/heads/branch2 checked out at '...'

The last line was very disorienting to me at first and made me think
we had a bug, but the update-refs stuff is built on top of the normal
rebase mechanism and branch2 will be updated by that logic rather than
by the special update-refs handling.  If I add another branch with a
few commits on top of branch2, then branch2 is indeed updated and
after the pick of 444444 (and the additional branch, say branch3,
would be updated by the normal rebase logic instead of by the
update-refs handling).  So it all works correctly, but users might get
worried or confused along the way wondering whether it will function
correctly.

Another part that users might find disorienting is that at the end,
the rebase reports:
    Successfully rebased and updated refs/heads/branch2.
which is correct but totally ignores the fact that it *also* rebased
and updated other branches.

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Derrick Stolee]

On 7/4/22 11:57 AM, Elijah Newren wrote:
> Actually, I checked out ds/rebase-update-ref just now to try it, and
> it seems like it does the right thing:
> 
>     pick 111111 first commit
>     pick 222222 second commit
>     fixup 555555 fixup! second commit
>     update-ref refs/heads/branch1
> 
>     pick 333333 third commit
>     pick 444444 fourth commit

Thanks for trying it out! This is definitely the main goal of the
feature, although it is also helpful when resolving conflicts or
doing 'edit' steps.

>     # Ref refs/heads/branch2 checked out at '...'
> 
> The last line was very disorienting to me at first and made me think
> we had a bug, but the update-refs stuff is built on top of the normal
> rebase mechanism and branch2 will be updated by that logic rather than
> by the special update-refs handling.  If I add another branch with a
> few commits on top of branch2, then branch2 is indeed updated and
> after the pick of 444444 (and the additional branch, say branch3,
> would be updated by the normal rebase logic instead of by the
> update-refs handling).  So it all works correctly, but users might get
> worried or confused along the way wondering whether it will function
> correctly.

I'll add a patch that removes the comment in the case of the HEAD
ref. Thanks for the idea!

> Another part that users might find disorienting is that at the end,
> the rebase reports:
>     Successfully rebased and updated refs/heads/branch2.
> which is correct but totally ignores the fact that it *also* rebased
> and updated other branches.

Good point. I can add an extra message at the end (as well as a
warning for any refs that did not properly update at the end).

Thanks,
-Stolee

Re: [PATCH v3 6/8] rebase: add --update-refs option

[Elijah Newren]

On Tue, Jul 5, 2022 at 3:22 PM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 7/4/22 11:57 AM, Elijah Newren wrote:
> > Actually, I checked out ds/rebase-update-ref just now to try it, and
> > it seems like it does the right thing:
> >
> >     pick 111111 first commit
> >     pick 222222 second commit
> >     fixup 555555 fixup! second commit
> >     update-ref refs/heads/branch1
> >
> >     pick 333333 third commit
> >     pick 444444 fourth commit
>
> Thanks for trying it out! This is definitely the main goal of the
> feature, although it is also helpful when resolving conflicts or
> doing 'edit' steps.
>
> >     # Ref refs/heads/branch2 checked out at '...'
> >
> > The last line was very disorienting to me at first and made me think
> > we had a bug, but the update-refs stuff is built on top of the normal
> > rebase mechanism and branch2 will be updated by that logic rather than
> > by the special update-refs handling.  If I add another branch with a
> > few commits on top of branch2, then branch2 is indeed updated and
> > after the pick of 444444 (and the additional branch, say branch3,
> > would be updated by the normal rebase logic instead of by the
> > update-refs handling).  So it all works correctly, but users might get
> > worried or confused along the way wondering whether it will function
> > correctly.
>
> I'll add a patch that removes the comment in the case of the HEAD
> ref. Thanks for the idea!

Thanks, that'd improve things.  I'm curious whether it'd be even
better to have an update-ref line for HEAD, so that users don't wonder
whether it's omitted from the updates.  (That would leave an open
question whether you filter out HEAD before actually calling
update-ref, or use some other mechanism to make it all work behind the
scenes.)

> > Another part that users might find disorienting is that at the end,
> > the rebase reports:
> >     Successfully rebased and updated refs/heads/branch2.
> > which is correct but totally ignores the fact that it *also* rebased
> > and updated other branches.
>
> Good point. I can add an extra message at the end (as well as a
> warning for any refs that did not properly update at the end).

Sounds good.

[PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change introduced the 'git rebase --update-refs' option
which added 'update-ref <ref>' commands to the todo list of an
interactive rebase.

Teach Git to record the HEAD position when reaching these 'update-ref'
commands. The ref/OID pair is stored in the
$GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
file to avoid having other processes updating the refs in that file
while the rebase is in progress.

Not only do we update the file when the sequencer reaches these
'update-ref' commands, we then update the refs themselves at the end of
the rebase sequence. If the rebase is aborted before this final step,
then the refs are not updated.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 114 +++++++++++++++++++++++++++++++++-
 t/t3404-rebase-interactive.sh |  23 +++++++
 2 files changed, 136 insertions(+), 1 deletion(-)

diff --git a/sequencer.c b/sequencer.c
index 915d87a0336..4fd1c0b5bce 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -36,6 +36,7 @@
 #include "rebase-interactive.h"
 #include "reset.h"
 #include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -148,6 +149,14 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
  */
 static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
 
+/*
+ * The update-refs file stores a list of refs that will be updated at the end
+ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
+ * update the OIDs for the refs in this file, but the refs are not updated
+ * until the end of the rebase sequence.
+ */
+static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
+
 /*
  * The following files are written by git-rebase just after parsing the
  * command-line.
@@ -4058,11 +4067,110 @@ leave_merge:
 	return ret;
 }
 
-static int do_update_ref(struct repository *r, const char *ref_name)
+static int write_update_refs_state(struct string_list *refs_to_oids)
+{
+	int result = 0;
+	FILE *fp = NULL;
+	struct string_list_item *item;
+	char *path = xstrdup(rebase_path_update_refs());
+
+	if (safe_create_leading_directories(path)) {
+		result = error(_("unable to create leading directories of %s"),
+			       path);
+		goto cleanup;
+	}
+
+	fp = fopen(path, "w");
+	if (!fp) {
+		result = error_errno(_("could not open '%s' for writing"), path);
+		goto cleanup;
+	}
+
+	for_each_string_list_item(item, refs_to_oids)
+		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
+
+cleanup:
+	if (fp)
+		fclose(fp);
+	return result;
+}
+
+static int do_update_ref(struct repository *r, const char *refname)
 {
+	struct string_list_item *item;
+	struct string_list list = STRING_LIST_INIT_DUP;
+	int found = 0;
+
+	sequencer_get_update_refs_state(r->gitdir, &list);
+
+	for_each_string_list_item(item, &list) {
+		if (!strcmp(item->string, refname)) {
+			struct object_id oid;
+			free(item->util);
+			found = 1;
+
+			if (!read_ref("HEAD", &oid)) {
+				item->util = oiddup(&oid);
+				break;
+			}
+		}
+	}
+
+	if (!found) {
+		struct object_id oid;
+		item = string_list_append(&list, refname);
+
+		if (!read_ref("HEAD", &oid))
+			item->util = oiddup(&oid);
+		else
+			item->util = oiddup(the_hash_algo->null_oid);
+	}
+
+	write_update_refs_state(&list);
+	string_list_clear(&list, 1);
 	return 0;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	int res = 0;
+	struct string_list_item *item;
+	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
+	struct ref_store *refs = get_main_ref_store(r);
+
+	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
+
+	for_each_string_list_item(item, &refs_to_oids) {
+		struct object_id *oid_to = item->util;
+		struct object_id oid_from;
+
+		if (oideq(oid_to, the_hash_algo->null_oid)) {
+			/*
+			 * Ref was not updated. User may have deleted the
+			 * 'update-ref' step.
+			 */
+			continue;
+		}
+
+		if (read_ref(item->string, &oid_from)) {
+			/*
+			 * The ref does not exist. The user probably
+			 * inserted a new 'update-ref' step with a new
+			 * branch name.
+			 */
+			oidcpy(&oid_from, the_hash_algo->null_oid);
+		}
+
+		res |= refs_update_ref(refs, "rewritten during rebase",
+				item->string,
+				oid_to, &oid_from,
+				0, UPDATE_REFS_MSG_ON_ERR);
+	}
+
+	string_list_clear(&refs_to_oids, 1);
+	return res;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4580,6 +4688,8 @@ cleanup_head_ref:
 		strbuf_release(&head_ref);
 	}
 
+	do_update_refs(r);
+
 	/*
 	 * Sequence of picks finished successfully; cleanup by
 	 * removing the .git/sequencer directory
@@ -5709,6 +5819,8 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 		}
 	}
 
+	write_update_refs_state(&ctx.refs_to_oids);
+
 	string_list_clear(&ctx.refs_to_oids, 1);
 	free(todo_list->items);
 	todo_list->items = ctx.items;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 3cd20733bc8..63a69bc073e 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1813,6 +1813,29 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 	)
 '
 
+compare_two_refs () {
+	git rev-parse $1 >expect &&
+	git rev-parse $2 >actual &&
+	test_cmp expect actual
+}
+
+test_expect_success '--update-refs updates refs correctly' '
+	git checkout -B update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	test_commit extra2 fileX &&
+	git commit --amend --fixup=L &&
+
+	git rebase -i --autosquash --update-refs primary &&
+
+	compare_two_refs HEAD~3 refs/heads/first &&
+	compare_two_refs HEAD~3 refs/heads/second &&
+	compare_two_refs HEAD~1 refs/heads/third &&
+	compare_two_refs HEAD refs/heads/no-conflict-branch
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <derrickstolee@github.com>
>
> The previous change introduced the 'git rebase --update-refs' option
> which added 'update-ref <ref>' commands to the todo list of an
> interactive rebase.
>
> Teach Git to record the HEAD position when reaching these 'update-ref'
> commands. The ref/OID pair is stored in the
> $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
> file to avoid having other processes updating the refs in that file
> while the rebase is in progress.
>
> Not only do we update the file when the sequencer reaches these
> 'update-ref' commands, we then update the refs themselves at the end of
> the rebase sequence. If the rebase is aborted before this final step,
> then the refs are not updated.

So, in general, we would

 * first scan the range of commits being rebased
 * compute what should happen and write the "todo" thing
 * write also "update-refs" to lick the branches to repel others
 * execute the "todo" insns one by one, possibly giving control back

And this order is important---update-refs file is written fairly
early, and the branches potentially involved in the rebase are all
protected during the time-consuming (due to possibility of manual
conflict resolution) execution step.

Makes sense.

> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index 3cd20733bc8..63a69bc073e 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1813,6 +1813,29 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
>  	)
>  '
>  
> +compare_two_refs () {
> +	git rev-parse $1 >expect &&
> +	git rev-parse $2 >actual &&
> +	test_cmp expect actual
> +}
> +
> +test_expect_success '--update-refs updates refs correctly' '
> +	git checkout -B update-refs no-conflict-branch &&
> +	git branch -f base HEAD~4 &&
> +	git branch -f first HEAD~3 &&
> +	git branch -f second HEAD~3 &&
> +	git branch -f third HEAD~1 &&
> +	test_commit extra2 fileX &&
> +	git commit --amend --fixup=L &&
> +
> +	git rebase -i --autosquash --update-refs primary &&
> +
> +	compare_two_refs HEAD~3 refs/heads/first &&
> +	compare_two_refs HEAD~3 refs/heads/second &&
> +	compare_two_refs HEAD~1 refs/heads/third &&
> +	compare_two_refs HEAD refs/heads/no-conflict-branch
> +'
> +
>  # This must be the last test in this file
>  test_expect_success '$EDITOR and friends are unchanged' '
>  	test_editor_unchanged

It would be nice to also have a test that makes sure that other
people will be prevented from checking out a branch whose tips may
be updated at the end.

Thanks.

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Derrick Stolee]

On 6/28/22 5:15 PM, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
...
> It would be nice to also have a test that makes sure that other
> people will be prevented from checking out a branch whose tips may
> be updated at the end.

Patch 3 adds the tests that 'git branch -f' cannot update these refs,
but I could add 'git checkout' tests, too. They both run the same
branch_checked_out() helper, but they are doing different things once
they have "access" to the branch.

Thanks,
-Stolee

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> On 6/28/22 5:15 PM, Junio C Hamano wrote:
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> ...
>> It would be nice to also have a test that makes sure that other
>> people will be prevented from checking out a branch whose tips may
>> be updated at the end.
>
> Patch 3 adds the tests that 'git branch -f' cannot update these refs,

Ah, I didn't notice.  We are covered then.  Thanks.

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Derrick Stolee]

On 6/28/22 9:25 AM, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>

> +static int write_update_refs_state(struct string_list *refs_to_oids)
> +{
> +	int result = 0;
> +	FILE *fp = NULL;
> +	struct string_list_item *item;
> +	char *path = xstrdup(rebase_path_update_refs());
> +
> +	if (safe_create_leading_directories(path)) {
> +		result = error(_("unable to create leading directories of %s"),
> +			       path);
> +		goto cleanup;
> +	}
> +
> +	fp = fopen(path, "w");

I realized (while trying to go to sleep, of course) that I need
to use a lockfile here. Parallel processes might be reading this
file while we are writing it.

Thanks,
-Stolee

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Phillip Wood]

Hi Stolee

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> The previous change introduced the 'git rebase --update-refs' option
> which added 'update-ref <ref>' commands to the todo list of an
> interactive rebase.
> 
> Teach Git to record the HEAD position when reaching these 'update-ref'
> commands. The ref/OID pair is stored in the
> $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
> file to avoid having other processes updating the refs in that file
> while the rebase is in progress.
> 
> Not only do we update the file when the sequencer reaches these
> 'update-ref' commands, we then update the refs themselves at the end of
> the rebase sequence. If the rebase is aborted before this final step,
> then the refs are not updated.

This looks good, I've left a few comments but it seems basically sound 
to me.

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   sequencer.c                   | 114 +++++++++++++++++++++++++++++++++-
>   t/t3404-rebase-interactive.sh |  23 +++++++
>   2 files changed, 136 insertions(+), 1 deletion(-)
> 
> diff --git a/sequencer.c b/sequencer.c
> index 915d87a0336..4fd1c0b5bce 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -36,6 +36,7 @@
>   #include "rebase-interactive.h"
>   #include "reset.h"
>   #include "branch.h"
> +#include "log-tree.h"
>   
>   #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>   
> @@ -148,6 +149,14 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>    */
>   static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>   
> +/*
> + * The update-refs file stores a list of refs that will be updated at the end
> + * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
> + * update the OIDs for the refs in this file, but the refs are not updated
> + * until the end of the rebase sequence.
> + */
> +static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
> +
>   /*
>    * The following files are written by git-rebase just after parsing the
>    * command-line.
> @@ -4058,11 +4067,110 @@ leave_merge:
>   	return ret;
>   }
>   
> -static int do_update_ref(struct repository *r, const char *ref_name)
> +static int write_update_refs_state(struct string_list *refs_to_oids)
> +{
> +	int result = 0;
> +	FILE *fp = NULL;
> +	struct string_list_item *item;
> +	char *path = xstrdup(rebase_path_update_refs());

This is leaked

> +	if (safe_create_leading_directories(path)) {
> +		result = error(_("unable to create leading directories of %s"),
> +			       path);
> +		goto cleanup;
> +	}
> +
> +	fp = fopen(path, "w");
> +	if (!fp) {
> +		result = error_errno(_("could not open '%s' for writing"), path);
> +		goto cleanup;
> +	}

Can we use fopen_or_warn() here? It ignores ENOENT and ENOTDIR but I 
don't think that should matter here.

> +
> +	for_each_string_list_item(item, refs_to_oids)
> +		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
> +
> +cleanup:
> +	if (fp)
> +		fclose(fp);
> +	return result;
> +}

> +compare_two_refs () {
> +	git rev-parse $1 >expect &&
> +	git rev-parse $2 >actual &&
> +	test_cmp expect actual
> +}

This is just test_cmp_rev

> +test_expect_success '--update-refs updates refs correctly' '
> +	git checkout -B update-refs no-conflict-branch &&
> +	git branch -f base HEAD~4 &&
> +	git branch -f first HEAD~3 &&
> +	git branch -f second HEAD~3 &&
> +	git branch -f third HEAD~1 &&
> +	test_commit extra2 fileX &&
> +	git commit --amend --fixup=L &&
> +
> +	git rebase -i --autosquash --update-refs primary &&
> +
> +	compare_two_refs HEAD~3 refs/heads/first &&
> +	compare_two_refs HEAD~3 refs/heads/second &&
> +	compare_two_refs HEAD~1 refs/heads/third &&
> +	compare_two_refs HEAD refs/heads/no-conflict-branch
> +'

Do we need a new test for this or can we just check the refs at the end 
of one of the tests added in the last patch?

>   # This must be the last test in this file
>   test_expect_success '$EDITOR and friends are unchanged' '
>   	test_editor_unchanged

I forgot to say on the last patch but you could maybe add a 
test_editor_unchanged at the end of t2407 now that there are tests which 
call test_set_editor

Thanks for working on this, it will be a really useful addition to rebase.

Phillip

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

>> +static int write_update_refs_state(struct string_list *refs_to_oids)
>> +{
>> +	int result = 0;
>> +	FILE *fp = NULL;
>> +	struct string_list_item *item;
>> +	char *path = xstrdup(rebase_path_update_refs());
>
> This is leaked

Good eyes.

>> +	if (safe_create_leading_directories(path)) {
>> +		result = error(_("unable to create leading directories of %s"),
>> +			       path);
>> +		goto cleanup;
>> +	}
>> +
>> +	fp = fopen(path, "w");
>> +	if (!fp) {
>> +		result = error_errno(_("could not open '%s' for writing"), path);
>> +		goto cleanup;
>> +	}
>
> Can we use fopen_or_warn() here? It ignores ENOENT and ENOTDIR but I
> don't think that should matter here.

fopen("no-such-directory/file", "w") would fail with ENOENT, and
fopen("README.md/file", "w") would fail with ENOTDIR, I would think,
so "should not matter because we are writing" is not the reason, but
because we know the path is in a subdirectory of ".git/" that we
know should exist, the most likely reason for the fopen to fail is
because (1) the repository is broken (we will get ENOENT, ENOTDIR,
which we want to warn about but fopen_or_warn() ignores, as well as
other errors such as EISDIR), (2) the repository is unwritable (we
will get EACCES), or (3) we are running low on diskspace (ENOSPC).

I think that the fopen_or_warn() helper was primarily invented to
read an optional file (so we deliberately ignore a failure to open
one due to ENOENT and ENOTDIR), and we should be careful of its use
for any other purpose, i.e. write access for any purpose and read
access for files that we know we should be able to.

>> +compare_two_refs () {
>> +	git rev-parse $1 >expect &&
>> +	git rev-parse $2 >actual &&
>> +	test_cmp expect actual
>> +}
>
> This is just test_cmp_rev

I love to see reviewers who know the existing API and helpers very
well (including the fopen-or-warn above).

Very much appreciated.

> Thanks for working on this, it will be a really useful addition to rebase.

Ditto.

Thanks.

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Elijah Newren]

On Tue, Jun 28, 2022 at 6:26 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <derrickstolee@github.com>
>
> The previous change introduced the 'git rebase --update-refs' option
> which added 'update-ref <ref>' commands to the todo list of an
> interactive rebase.
>
> Teach Git to record the HEAD position when reaching these 'update-ref'
> commands. The ref/OID pair is stored in the
> $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
> file to avoid having other processes updating the refs in that file
> while the rebase is in progress.
>
> Not only do we update the file when the sequencer reaches these
> 'update-ref' commands, we then update the refs themselves at the end of
> the rebase sequence. If the rebase is aborted before this final step,
> then the refs are not updated.

Why update with each update-ref command?  Couldn't the values be
stored in memory and only written when we hit a conflict?

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  sequencer.c                   | 114 +++++++++++++++++++++++++++++++++-
>  t/t3404-rebase-interactive.sh |  23 +++++++
>  2 files changed, 136 insertions(+), 1 deletion(-)
>
> diff --git a/sequencer.c b/sequencer.c
> index 915d87a0336..4fd1c0b5bce 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -36,6 +36,7 @@
>  #include "rebase-interactive.h"
>  #include "reset.h"
>  #include "branch.h"
> +#include "log-tree.h"
>
>  #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>
> @@ -148,6 +149,14 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>   */
>  static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>
> +/*
> + * The update-refs file stores a list of refs that will be updated at the end
> + * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
> + * update the OIDs for the refs in this file, but the refs are not updated
> + * until the end of the rebase sequence.
> + */
> +static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
> +
>  /*
>   * The following files are written by git-rebase just after parsing the
>   * command-line.
> @@ -4058,11 +4067,110 @@ leave_merge:
>         return ret;
>  }
>
> -static int do_update_ref(struct repository *r, const char *ref_name)
> +static int write_update_refs_state(struct string_list *refs_to_oids)
> +{
> +       int result = 0;
> +       FILE *fp = NULL;
> +       struct string_list_item *item;
> +       char *path = xstrdup(rebase_path_update_refs());
> +
> +       if (safe_create_leading_directories(path)) {
> +               result = error(_("unable to create leading directories of %s"),
> +                              path);
> +               goto cleanup;
> +       }
> +
> +       fp = fopen(path, "w");
> +       if (!fp) {
> +               result = error_errno(_("could not open '%s' for writing"), path);
> +               goto cleanup;
> +       }
> +
> +       for_each_string_list_item(item, refs_to_oids)
> +               fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));

Here you are storing the ref and the new oid to move it to.  Any
reason you don't store the previous oid for the branch?  In
particular, if someone hits a conflict, needs to resolve, and comes
back some time much later and these intermediate branches have since
moved on, should we actually update those branches?  (And, if we do,
should we at least give a warning?)

> +cleanup:
> +       if (fp)
> +               fclose(fp);
> +       return result;
> +}
> +
> +static int do_update_ref(struct repository *r, const char *refname)
>  {
> +       struct string_list_item *item;
> +       struct string_list list = STRING_LIST_INIT_DUP;
> +       int found = 0;
> +
> +       sequencer_get_update_refs_state(r->gitdir, &list);
> +
> +       for_each_string_list_item(item, &list) {
> +               if (!strcmp(item->string, refname)) {
> +                       struct object_id oid;
> +                       free(item->util);
> +                       found = 1;
> +
> +                       if (!read_ref("HEAD", &oid)) {
> +                               item->util = oiddup(&oid);
> +                               break;
> +                       }
> +               }
> +       }
> +
> +       if (!found) {
> +               struct object_id oid;
> +               item = string_list_append(&list, refname);
> +
> +               if (!read_ref("HEAD", &oid))
> +                       item->util = oiddup(&oid);
> +               else
> +                       item->util = oiddup(the_hash_algo->null_oid);

Seems a little unfortunate to not use a cached value from the latest
commit we picked (and wrote to HEAD) and instead need to re-read HEAD.
But, I guess sequencer is written to round-trip through files, so
maybe optimizing this here doesn't make sense given all the other
places in sequencer where we do lots of file reading and writing.

> +       }
> +
> +       write_update_refs_state(&list);
> +       string_list_clear(&list, 1);
>         return 0;
>  }
>
> +static int do_update_refs(struct repository *r)
> +{
> +       int res = 0;
> +       struct string_list_item *item;
> +       struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
> +       struct ref_store *refs = get_main_ref_store(r);
> +
> +       sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
> +
> +       for_each_string_list_item(item, &refs_to_oids) {
> +               struct object_id *oid_to = item->util;
> +               struct object_id oid_from;
> +
> +               if (oideq(oid_to, the_hash_algo->null_oid)) {
> +                       /*
> +                        * Ref was not updated. User may have deleted the
> +                        * 'update-ref' step.
> +                        */
> +                       continue;
> +               }
> +
> +               if (read_ref(item->string, &oid_from)) {
> +                       /*
> +                        * The ref does not exist. The user probably
> +                        * inserted a new 'update-ref' step with a new
> +                        * branch name.
> +                        */
> +                       oidcpy(&oid_from, the_hash_algo->null_oid);
> +               }
> +
> +               res |= refs_update_ref(refs, "rewritten during rebase",
> +                               item->string,
> +                               oid_to, &oid_from,
> +                               0, UPDATE_REFS_MSG_ON_ERR);
> +       }
> +
> +       string_list_clear(&refs_to_oids, 1);
> +       return res;
> +}
> +
>  static int is_final_fixup(struct todo_list *todo_list)
>  {
>         int i = todo_list->current;
> @@ -4580,6 +4688,8 @@ cleanup_head_ref:
>                 strbuf_release(&head_ref);
>         }
>
> +       do_update_refs(r);
> +
>         /*
>          * Sequence of picks finished successfully; cleanup by
>          * removing the .git/sequencer directory
> @@ -5709,6 +5819,8 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>                 }
>         }
>
> +       write_update_refs_state(&ctx.refs_to_oids);
> +
>         string_list_clear(&ctx.refs_to_oids, 1);
>         free(todo_list->items);
>         todo_list->items = ctx.items;
> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index 3cd20733bc8..63a69bc073e 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1813,6 +1813,29 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
>         )
>  '
>
> +compare_two_refs () {
> +       git rev-parse $1 >expect &&
> +       git rev-parse $2 >actual &&
> +       test_cmp expect actual
> +}
> +
> +test_expect_success '--update-refs updates refs correctly' '
> +       git checkout -B update-refs no-conflict-branch &&
> +       git branch -f base HEAD~4 &&
> +       git branch -f first HEAD~3 &&
> +       git branch -f second HEAD~3 &&
> +       git branch -f third HEAD~1 &&
> +       test_commit extra2 fileX &&
> +       git commit --amend --fixup=L &&
> +
> +       git rebase -i --autosquash --update-refs primary &&
> +
> +       compare_two_refs HEAD~3 refs/heads/first &&
> +       compare_two_refs HEAD~3 refs/heads/second &&
> +       compare_two_refs HEAD~1 refs/heads/third &&
> +       compare_two_refs HEAD refs/heads/no-conflict-branch
> +'
> +
>  # This must be the last test in this file
>  test_expect_success '$EDITOR and friends are unchanged' '
>         test_editor_unchanged
> --
> gitgitgadget
>

Re: [PATCH v3 7/8] rebase: update refs from 'update-ref' commands

[Derrick Stolee]

On 7/1/22 5:18 PM, Elijah Newren wrote:
> On Tue, Jun 28, 2022 at 6:26 AM Derrick Stolee via GitGitGadget

>> Not only do we update the file when the sequencer reaches these
>> 'update-ref' commands, we then update the refs themselves at the end of
>> the rebase sequence. If the rebase is aborted before this final step,
>> then the refs are not updated.
> 
> Why update with each update-ref command?  Couldn't the values be
> stored in memory and only written when we hit a conflict?

I think that is a natural way to optimize the feature. I didn't
look into that option as it seemed more error-prone to me. I'd
be happy to be wrong here if an easy tweak makes this possible.

>> -static int do_update_ref(struct repository *r, const char *ref_name)
>> +static int write_update_refs_state(struct string_list *refs_to_oids)
>> +{
>> +       int result = 0;
>> +       FILE *fp = NULL;
>> +       struct string_list_item *item;
>> +       char *path = xstrdup(rebase_path_update_refs());
>> +
>> +       if (safe_create_leading_directories(path)) {
>> +               result = error(_("unable to create leading directories of %s"),
>> +                              path);
>> +               goto cleanup;
>> +       }
>> +
>> +       fp = fopen(path, "w");
>> +       if (!fp) {
>> +               result = error_errno(_("could not open '%s' for writing"), path);
>> +               goto cleanup;
>> +       }
>> +
>> +       for_each_string_list_item(item, refs_to_oids)
>> +               fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
> 
> Here you are storing the ref and the new oid to move it to.  Any
> reason you don't store the previous oid for the branch?  In
> particular, if someone hits a conflict, needs to resolve, and comes
> back some time much later and these intermediate branches have since
> moved on, should we actually update those branches?  (And, if we do,
> should we at least give a warning?)

The branches don't move forward because other Git processes respect
the update-refs file (this does not take into account third-party
tools that don't know about that file, but we need to start somewhere).

You're right that storing the old value would help us in this case
where another tool updated the ref while we were in the middle of the
rebase. The storage of the null OID is only to prevent writing over a
ref when the user has removed the 'update-ref <ref>' command from the
todo-list. It's probably better to remove the ref from the list when
that happens.

>> +       if (!found) {
>> +               struct object_id oid;
>> +               item = string_list_append(&list, refname);
>> +
>> +               if (!read_ref("HEAD", &oid))
>> +                       item->util = oiddup(&oid);
>> +               else
>> +                       item->util = oiddup(the_hash_algo->null_oid);
> 
> Seems a little unfortunate to not use a cached value from the latest
> commit we picked (and wrote to HEAD) and instead need to re-read HEAD.
> But, I guess sequencer is written to round-trip through files, so
> maybe optimizing this here doesn't make sense given all the other
> places in sequencer where we do lots of file reading and writing.

It's also possible to optimize several things, but I tried to
minimize the amount of change to the existing methods. This is
my first foray into this part of the code, so I don't always know
which information is readily available. I appreciate your attention
here.

Thanks,
-Stolee

[PATCH v3 8/8] rebase: add rebase.updateRefs config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change added the --update-refs command-line option.  For
users who always want this mode, create the rebase.updateRefs config
option which behaves the same way as rebase.autoSquash does with the
--autosquash option.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config/rebase.txt |  3 +++
 Documentation/git-rebase.txt    |  3 +++
 builtin/rebase.c                |  5 +++++
 t/t3404-rebase-interactive.sh   | 14 ++++++++++++++
 4 files changed, 25 insertions(+)

diff --git a/Documentation/config/rebase.txt b/Documentation/config/rebase.txt
index 8c979cb20f2..f19bd0e0407 100644
--- a/Documentation/config/rebase.txt
+++ b/Documentation/config/rebase.txt
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index e7611b4089c..1249f9ed617 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -615,6 +615,9 @@ start would be overridden by the presence of
 	are being rebased. Any branches that are checked out in a worktree
 	or point to a `squash! ...` or `fixup! ...` commit are not updated
 	in this way.
++
+If the configuration variable `rebase.updateRefs` is set, then this option
+can be used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 56d82a52106..8ebc98ea505 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 63a69bc073e..77478d94f3b 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1768,6 +1768,12 @@ test_expect_success '--update-refs adds label and update-ref commands' '
 		EOF
 
 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
+
 		test_cmp expect todo
 	)
 '
@@ -1809,6 +1815,14 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 				   --rebase-merges=rebase-cousins \
 				   --update-refs primary >todo &&
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i \
+				   --rebase-merges=rebase-cousins \
+				   primary >todo &&
+
 		test_cmp expect todo
 	)
 '
-- 
gitgitgadget

Re: [PATCH v3 0/8] rebase: update branches in multi-part topic

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'update-ref' commands into the
> TODO list. This is powered by the commit decoration machinery.

I read through the patches and they made sense to me.  Hopefully I
can push out the integration result including this topic by the end
of the day (today is actually my usual "every other Tuesday gitster
goes offline" day, but I'll go offline tomorrow instead, so I want
to injest as many reasonably-done topics as possible today).

Thanks.

Re: [PATCH v3 0/8] rebase: update branches in multi-part topic

[Phillip Wood]

Hi Stolee

On 28/06/2022 14:25, Derrick Stolee via GitGitGadget wrote:
> (Update: This is now based on ds/branch-checked-out.)
> 
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
> 
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'update-ref' commands into the
> TODO list. This is powered by the commit decoration machinery.
> 
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
> 
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> update-ref refs/heads/bundle-redo/fetch

I like the direction this is moving, my only concern with this version 
is how we handle the user adding and removing update-ref commands when 
the user edits the todo list.

Best Wishes

Phillip

> pick 1e3f6546632 clone: add --bundle-uri option
> pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
> update-ref refs/heads/bundle-redo/clone
> 
> pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
> pick 3029c3aca15 bundle-uri: create base key-value pair parsing
> pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
> pick 92625a47673 bundle-uri: unit test "key=value" parsing
> pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
> pick 9d6809a8d53 bundle-uri: parse bundle list in config format
> pick 287a732b54c bundle-uri: fetch a list of bundles
> update-ref refs/heads/bundle-redo/list
> 
> pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
> pick 520204dcd1c bundle-uri client: add minimal NOOP client
> pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
> pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
> pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
> pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
> pick df255000b7e bundle-uri: download bundles from an advertised list
> pick d71beabf199 clone: unbundle the advertised bundles
> pick c9578391976 t5601: basic bundle URI tests
> # Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'
> 
> update-ref refs/heads/bundle-redo/advertise
> 
> 
> Here is an outline of the series:
> 
>   * Patch 1 updates some tests for branch_checked_out() for the 'apply'
>     backend.
>   * Patch 2 updates branch_checked_out() to parse the
>     rebase-merge/update-refs file to block concurrent ref updates and
>     checkouts on branches selected by --update-refs.
>   * Patch 3 updates the todo list documentation to remove some unnecessary
>     dots in the 'merge' command. This makes it consistent with the 'fixup'
>     command before we document the 'update-ref' command.
>   * Patch 4 updates the definition of todo_command_info to use enum values as
>     array indices.
>   * Patches 5-7 implement the --update-refs logic itself.
>   * Patch 8 adds the rebase.updateRefs config option similar to
>     rebase.autoSquash.
> 
> 
> Updates in v3
> =============
> 
>   * The branch_checked_out() API was extracted to its own topic and is now
>     the ds/branch-checked-out branch. This series is now based on that one.
>   * The for_each_decoration() API was removed, since it became trivial once
>     it did not take a commit directly.
>   * The branch_checked_out() tests did not verify the rebase-apply data (for
>     the apply backend), so that is fixed.
>   * Instead of using the 'label' command and a final 'update-refs' command in
>     the todo list, use a new 'update-ref ' command. This command updates the
>     rebase-merge/update-refs file with the OID of HEAD at these steps. At the
>     very end of the rebase sequence, those refs are updated to the stored OID
>     values (assuming that they were not removed by the user, in which case we
>     notice that the OID is the null OID and we do nothing).
>   * New tests are added.
>   * The todo-list comment documentation has some new formatting updates, but
>     also includes a description of 'update-refs' in this version.
> 
> 
> Updates in v2
> =============
> 
> As recommended by the excellent feedback, I have removed the 'exec' commands
> in favor of the 'label' commands and a new 'update-refs' command at the very
> end. This way, there is only one step that updates all of the refs at the
> end instead of updating refs during the rebase. If a user runs 'git rebase
> --abort' in the middle, then their refs are still where they need to be.
> 
> Based on some of the discussion, it seemed like one way to do this would be
> to have an 'update-ref ' command that would take the place of these 'label'
> commands. However, this would require two things that make it a bit awkward:
> 
>   1. We would need to replicate the storage of those positions during the
>      rebase. 'label' already does this pretty well. I've added the
>      "for-update-refs/" label to help here.
>   2. If we want to close out all of the refs as the rebase is finishing, then
>      that "step" becomes invisible to the user (and a bit more complicated to
>      insert). Thus, the 'update-refs' step performs this action. If the user
>      wants to do things after that step, then they can do so by editing the
>      TODO list.
> 
> Other updates:
> 
>   * The 'keep_decorations' parameter was renamed to 'update_refs'.
>   * I added tests for --rebase-merges=rebase-cousins to show how these labels
>     interact with other labels and merge commands.
>   * I changed the order of the insertion of these update-refs labels to be
>     before the fixups are rearranged. This fixes a bug where the tip commit
>     is a fixup! so its decorations are never inspected (and they would be in
>     the wrong place even if they were). The fixup! commands are properly
>     inserted between a pick and its following label command. Tests
>     demonstrate this is correct.
>   * Numerous style choices are updated based on feedback.
> 
> Thank you for all of the detailed review and ideas in this space. I
> appreciate any more ideas that can make this feature as effective as it can
> be.
> 
> Thanks, -Stolee
> 
> Derrick Stolee (8):
>    t2407: test branches currently using apply backend
>    branch: consider refs under 'update-refs'
>    rebase-interactive: update 'merge' description
>    sequencer: define array with enum values
>    sequencer: add update-ref command
>    rebase: add --update-refs option
>    rebase: update refs from 'update-ref' commands
>    rebase: add rebase.updateRefs config option
> 
>   Documentation/config/rebase.txt |   3 +
>   Documentation/git-rebase.txt    |  11 ++
>   branch.c                        |  13 ++
>   builtin/rebase.c                |  10 ++
>   rebase-interactive.c            |   9 +-
>   sequencer.c                     | 303 ++++++++++++++++++++++++++++++--
>   sequencer.h                     |  11 ++
>   t/t2407-worktree-heads.sh       |  52 +++++-
>   t/t3404-rebase-interactive.sh   | 107 +++++++++++
>   9 files changed, 496 insertions(+), 23 deletions(-)
> 
> 
> base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v3
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v3
> Pull-Request: https://github.com/gitgitgadget/git/pull/1247
> 
> Range-diff vs v2:
> 
>   1:  4f9f3487641 < -:  ----------- log-tree: create for_each_decoration()
>   2:  5f54766e103 < -:  ----------- branch: add branch_checked_out() helper
>   -:  ----------- > 1:  fbaedc7f1f0 t2407: test branches currently using apply backend
>   -:  ----------- > 2:  2bc647b6fcd branch: consider refs under 'update-refs'
>   -:  ----------- > 3:  669f4abd59e rebase-interactive: update 'merge' description
>   3:  9f261c7df2c = 4:  6528a50343f sequencer: define array with enum values
>   4:  842b2186d25 ! 5:  e95ad41d355 sequencer: add update-refs command
>       @@ Metadata
>        Author: Derrick Stolee <derrickstolee@github.com>
>        
>         ## Commit message ##
>       -    sequencer: add update-refs command
>       +    sequencer: add update-ref command
>        
>       -    Add the boilerplat for an "update-refs" command in the sequencer. This
>       -    connects to the current no-op do_update_refs() which will be filled in
>       +    Add the boilerplate for an "update-ref" command in the sequencer. This
>       +    connects to the current no-op do_update_ref() which will be filled in
>            after more connections are created.
>        
>       +    The syntax in the todo list will be "update-ref <ref-name>" to signal
>       +    that we should store the current commit as the value for updating
>       +    <ref-name> at the end of the rebase.
>       +
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>       + ## rebase-interactive.c ##
>       +@@ rebase-interactive.c: void append_todo_help(int command_count,
>       + "        create a merge commit using the original merge commit's\n"
>       + "        message (or the oneline, if no original merge commit was\n"
>       + "        specified); use -c <commit> to reword the commit message\n"
>       ++"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
>       ++"                      to this position in the new commits. The <ref> is\n"
>       ++"                      updated at the end of the rebase\n"
>       + "\n"
>       + "These lines can be re-ordered; they are executed from top to bottom.\n");
>       + 	unsigned edit_todo = !(shortrevisions && shortonto);
>       +
>         ## sequencer.c ##
>        @@ sequencer.c: static struct {
>         	[TODO_LABEL] = { 'l', "label" },
>         	[TODO_RESET] = { 't', "reset" },
>         	[TODO_MERGE] = { 'm', "merge" },
>       -+	[TODO_UPDATE_REFS] = { 'u', "update-refs" },
>       ++	[TODO_UPDATE_REF] = { 'u', "update-ref" },
>         	[TODO_NOOP] = { 0,   "noop" },
>         	[TODO_DROP] = { 'd', "drop" },
>         	[TODO_COMMENT] = { 0,   NULL },
>        @@ sequencer.c: static int parse_insn_line(struct repository *r, struct todo_item *item,
>       - 	padding = strspn(bol, " \t");
>       - 	bol += padding;
>       + 			     command_to_string(item->command));
>         
>       --	if (item->command == TODO_NOOP || item->command == TODO_BREAK) {
>       -+	if (item->command == TODO_NOOP ||
>       -+	    item->command == TODO_BREAK ||
>       -+	    item->command == TODO_UPDATE_REFS) {
>       - 		if (bol != eol)
>       - 			return error(_("%s does not accept arguments: '%s'"),
>       - 				     command_to_string(item->command), bol);
>       + 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
>       +-	    item->command == TODO_RESET) {
>       ++	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
>       + 		item->commit = NULL;
>       + 		item->arg_offset = bol - buf;
>       + 		item->arg_len = (int)(eol - bol);
>        @@ sequencer.c: leave_merge:
>         	return ret;
>         }
>         
>       -+static int do_update_refs(struct repository *r)
>       ++static int do_update_ref(struct repository *r, const char *ref_name)
>        +{
>        +	return 0;
>        +}
>       @@ sequencer.c: static int pick_commits(struct repository *r,
>         				return error_with_patch(r, item->commit,
>         							arg, item->arg_len,
>         							opts, res, 0);
>       -+		} else if (item->command == TODO_UPDATE_REFS) {
>       -+			if ((res = do_update_refs(r)))
>       ++		} else if (item->command == TODO_UPDATE_REF) {
>       ++			struct strbuf ref = STRBUF_INIT;
>       ++			strbuf_add(&ref, arg, item->arg_len);
>       ++			if ((res = do_update_ref(r, ref.buf)))
>        +				reschedule = 1;
>       ++			strbuf_release(&ref);
>         		} else if (!is_noop(item->command))
>         			return error(_("unknown command %d"), item->command);
>         
>       @@ sequencer.h: enum todo_command {
>         	TODO_LABEL,
>         	TODO_RESET,
>         	TODO_MERGE,
>       -+	TODO_UPDATE_REFS,
>       ++	TODO_UPDATE_REF,
>         	/* commands that do nothing but are counted for reporting progress */
>         	TODO_NOOP,
>         	TODO_DROP,
>   5:  0a4c110127b ! 6:  918b398d6a2 rebase: add --update-refs option
>       @@ Commit message
>            reachable from those "sub branches". It can take a manual step to update
>            those branches.
>        
>       -    Add a new --update-refs option to 'git rebase -i' that adds 'label
>       -    for-update-refs/*' steps to the todo file whenever a commit that is
>       -    being rebased is decorated with that <ref>. At the very end, the
>       -    'update-refs' step is added to update all of the branches referenced by
>       -    the 'label' steps. This allows the user to rebase a long list of commits
>       -    in a multi-part feature and keep all of their pointers to those parts.
>       +    Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
>       +    <ref>' steps to the todo file whenever a commit that is being rebased is
>       +    decorated with that <ref>. At the very end, the rebase process updates
>       +    all of the listed refs to the values stored during the rebase operation.
>        
>       -    NOTE: This change only introduce the --update-refs option and implements
>       -    the changes to the todo file. It does _not_ yet implement the action
>       -    taken by the 'update-refs' todo step, which will be implemented and
>       -    tested in a later change.
>       -
>       -    Use the new for_each_decoration() while iterating over the existing todo
>       -    list. Be sure to iterate after any squashing or fixups are placed.
>       -    Update the branch only after those squashes and fixups are complete.
>       -    This allows a --fixup commit at the tip of the feature to apply
>       -    correctly to the sub branch, even if it is fixing up the most-recent
>       -    commit in that part.
>       +    Be sure to iterate after any squashing or fixups are placed. Update the
>       +    branch only after those squashes and fixups are complete. This allows a
>       +    --fixup commit at the tip of the feature to apply correctly to the sub
>       +    branch, even if it is fixing up the most-recent commit in that part.
>        
>            One potential problem here is that refs decorating commits that are
>            already marked as "fixup!" or "squash!" will not be included in this
>       @@ Commit message
>            of the rebased branch while the other sub branches come along for the
>            ride without intervention.
>        
>       -    Be careful to not attempt updating any branch that is checked out. The
>       -    most common example is the branch being rebased is checked out and
>       -    decorates the tip commit. If the user is rebasing commits reachable from
>       -    a different branch that is checked out in a different worktree, then
>       -    they may be surprised to not see that ref update. However, it's probably
>       -    best to not optimize for this scenario and do the safest thing that will
>       -    result in a successful rebase. A comment is left in the TODO list that
>       -    signals that these refs are currently checked out.
>       +    This change update the documentation and builtin to accept the
>       +    --update-refs option as well as updating the todo file with the
>       +    'update-ref' commands. Tests are added to ensure that these todo
>       +    commands are added in the correct locations.
>       +
>       +    A future change will update the behavior to actually update the refs
>       +    at the end of the rebase sequence.
>        
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>       @@ sequencer.c
>         #include "rebase-interactive.h"
>         #include "reset.h"
>        +#include "branch.h"
>       -+#include "log-tree.h"
>         
>         #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>         
>       @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
>        +	size_t items_alloc;
>        +	struct strbuf *buf;
>        +	struct commit *commit;
>       ++	struct string_list refs_to_oids;
>        +};
>        +
>       -+static int add_branch_for_decoration(const struct name_decoration *d, void *data)
>       ++static int add_decorations_to_list(const struct commit *commit,
>       ++				   struct todo_add_branch_context *ctx)
>        +{
>       -+	struct todo_add_branch_context *ctx = data;
>       -+	size_t base_offset = ctx->buf->len;
>       -+	struct todo_item *item;
>       -+	char *path;
>       ++	const struct name_decoration *decoration = get_name_decoration(&commit->object);
>        +
>       -+	ALLOC_GROW(ctx->items,
>       -+		   ctx->items_nr + 1,
>       -+		   ctx->items_alloc);
>       -+	item = &ctx->items[ctx->items_nr];
>       -+	memset(item, 0, sizeof(*item));
>       ++	while (decoration) {
>       ++		struct todo_item *item;
>       ++		const char *path;
>       ++		size_t base_offset = ctx->buf->len;
>        +
>       -+	/* If the branch is checked out, then leave a comment instead. */
>       -+	if (branch_checked_out(d->name, &path)) {
>       -+		item->command = TODO_COMMENT;
>       -+		strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
>       -+			    d->name, path);
>       -+		free(path);
>       -+	} else {
>       -+		item->command = TODO_LABEL;
>       -+		strbuf_addf(ctx->buf, "for-update-refs/%s\n", d->name);
>       -+	}
>       ++		ALLOC_GROW(ctx->items,
>       ++			ctx->items_nr + 1,
>       ++			ctx->items_alloc);
>       ++		item = &ctx->items[ctx->items_nr];
>       ++		memset(item, 0, sizeof(*item));
>       ++
>       ++		/* If the branch is checked out, then leave a comment instead. */
>       ++		if ((path = branch_checked_out(decoration->name))) {
>       ++			item->command = TODO_COMMENT;
>       ++			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
>       ++				    decoration->name, path);
>       ++		} else {
>       ++			struct string_list_item *sti;
>       ++			item->command = TODO_UPDATE_REF;
>       ++			strbuf_addf(ctx->buf, "%s\n", decoration->name);
>       ++
>       ++			sti = string_list_append(&ctx->refs_to_oids,
>       ++						 decoration->name);
>       ++			sti->util = oiddup(the_hash_algo->null_oid);
>       ++		}
>       ++
>       ++		item->offset_in_buf = base_offset;
>       ++		item->arg_offset = base_offset;
>       ++		item->arg_len = ctx->buf->len - base_offset;
>       ++		ctx->items_nr++;
>        +
>       -+	item->offset_in_buf = base_offset;
>       -+	item->arg_offset = base_offset;
>       -+	item->arg_len = ctx->buf->len - base_offset;
>       -+	ctx->items_nr++;
>       ++		decoration = decoration->next;
>       ++	}
>        +
>        +	return 0;
>        +}
>       @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
>        +	struct decoration_filter decoration_filter = {
>        +		.include_ref_pattern = &decorate_refs_include,
>        +		.exclude_ref_pattern = &decorate_refs_exclude,
>       -+		.exclude_ref_config_pattern = &decorate_refs_exclude_config
>       ++		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
>        +	};
>        +	struct todo_add_branch_context ctx = {
>        +		.buf = &todo_list->buf,
>       ++		.refs_to_oids = STRING_LIST_INIT_DUP,
>        +	};
>        +
>        +	ctx.items_alloc = 2 * todo_list->nr + 1;
>       @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
>        +
>        +		if (item->commit) {
>        +			ctx.commit = item->commit;
>       -+			for_each_decoration(item->commit,
>       -+					    add_branch_for_decoration,
>       -+					    &ctx);
>       ++			add_decorations_to_list(item->commit, &ctx);
>        +		}
>        +	}
>        +
>       -+	/* Add the "update-refs" step. */
>       -+	ALLOC_GROW(ctx.items,
>       -+		   ctx.items_nr + 1,
>       -+		   ctx.items_alloc);
>       -+	memset(&ctx.items[ctx.items_nr], 0, sizeof(struct todo_item));
>       -+	ctx.items[ctx.items_nr].command = TODO_UPDATE_REFS;
>       -+	ctx.items_nr++;
>       -+
>       ++	string_list_clear(&ctx.refs_to_oids, 1);
>        +	free(todo_list->items);
>        +	todo_list->items = ctx.items;
>        +	todo_list->nr = ctx.items_nr;
>       @@ sequencer.h: int complete_action(struct repository *r, struct replay_opts *opts,
>         int todo_list_rearrange_squash(struct todo_list *todo_list);
>         
>        
>       + ## t/t2407-worktree-heads.sh ##
>       +@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite when in error states' '
>       + 	done
>       + '
>       +
>       ++. "$TEST_DIRECTORY"/lib-rebase.sh
>       ++
>       ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
>       ++	git commit --fixup HEAD~2 --allow-empty &&
>       ++	(
>       ++		set_cat_todo_editor &&
>       ++		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
>       ++		! grep "update-refs" todo
>       ++	) &&
>       ++	git branch -f allow-update HEAD~2 &&
>       ++	(
>       ++		set_cat_todo_editor &&
>       ++		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
>       ++		grep "update-ref refs/heads/allow-update" todo
>       ++	)
>       ++'
>       ++
>       + test_done
>       +
>         ## t/t3404-rebase-interactive.sh ##
>        @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correctly' '
>         	test_cmp_rev ORIG_HEAD test-orig-head@{1}
>       @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
>        +
>        +		cat >expect <<-EOF &&
>        +		pick $(git log -1 --format=%h J) J
>       -+		label for-update-refs/refs/heads/second
>       -+		label for-update-refs/refs/heads/first
>       ++		update-ref refs/heads/second
>       ++		update-ref refs/heads/first
>        +		pick $(git log -1 --format=%h K) K
>        +		pick $(git log -1 --format=%h L) L
>        +		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
>       -+		label for-update-refs/refs/heads/third
>       ++		update-ref refs/heads/third
>        +		pick $(git log -1 --format=%h M) M
>       -+		label for-update-refs/refs/heads/no-conflict-branch
>       -+		label for-update-refs/refs/heads/shared-tip
>       -+		update-refs
>       ++		update-ref refs/heads/no-conflict-branch
>       ++		update-ref refs/heads/shared-tip
>        +		EOF
>        +
>        +		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
>       @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
>        +		reset onto
>        +		pick $(git log -1 --format=%h branch2~1) F
>        +		pick $(git log -1 --format=%h branch2) I
>       -+		label for-update-refs/refs/heads/branch2
>       ++		update-ref refs/heads/branch2
>        +		label merge
>        +		reset onto
>        +		pick $(git log -1 --format=%h refs/heads/second) J
>       -+		label for-update-refs/refs/heads/second
>       -+		label for-update-refs/refs/heads/first
>       ++		update-ref refs/heads/second
>       ++		update-ref refs/heads/first
>        +		pick $(git log -1 --format=%h refs/heads/third~1) K
>        +		pick $(git log -1 --format=%h refs/heads/third) L
>        +		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
>       -+		label for-update-refs/refs/heads/third
>       ++		update-ref refs/heads/third
>        +		pick $(git log -1 --format=%h HEAD~2) M
>       -+		label for-update-refs/refs/heads/no-conflict-branch
>       ++		update-ref refs/heads/no-conflict-branch
>        +		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
>       -+		label for-update-refs/refs/heads/merge-branch
>       -+		update-refs
>       ++		update-ref refs/heads/merge-branch
>        +		EOF
>        +
>        +		test_must_fail git rebase -i --autosquash \
>   6:  68f8e51b19c ! 7:  72e0481b643 sequencer: implement 'update-refs' command
>       @@ Metadata
>        Author: Derrick Stolee <derrickstolee@github.com>
>        
>         ## Commit message ##
>       -    sequencer: implement 'update-refs' command
>       +    rebase: update refs from 'update-ref' commands
>        
>       -    The previous change allowed 'git rebase --update-refs' to create 'label'
>       -    commands for each branch  among the commits being rewritten and add an
>       -    'update-refs' command at the end of the todo list. Now, teach Git to
>       -    update the refs during that final 'update-refs' command.
>       +    The previous change introduced the 'git rebase --update-refs' option
>       +    which added 'update-ref <ref>' commands to the todo list of an
>       +    interactive rebase.
>        
>       -    We need to create an array of new and old OIDs for each ref by iterating
>       -    over the refs/rewritten/for-update-refs/ namespace. We cannot update the
>       -    refs in-place since this will confuse the refs iterator.
>       +    Teach Git to record the HEAD position when reaching these 'update-ref'
>       +    commands. The ref/OID pair is stored in the
>       +    $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
>       +    file to avoid having other processes updating the refs in that file
>       +    while the rebase is in progress.
>       +
>       +    Not only do we update the file when the sequencer reaches these
>       +    'update-ref' commands, we then update the refs themselves at the end of
>       +    the rebase sequence. If the rebase is aborted before this final step,
>       +    then the refs are not updated.
>        
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>         ## sequencer.c ##
>       +@@
>       + #include "rebase-interactive.h"
>       + #include "reset.h"
>       + #include "branch.h"
>       ++#include "log-tree.h"
>       +
>       + #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>       +
>       +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>       +  */
>       + static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>       +
>       ++/*
>       ++ * The update-refs file stores a list of refs that will be updated at the end
>       ++ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
>       ++ * update the OIDs for the refs in this file, but the refs are not updated
>       ++ * until the end of the rebase sequence.
>       ++ */
>       ++static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
>       ++
>       + /*
>       +  * The following files are written by git-rebase just after parsing the
>       +  * command-line.
>        @@ sequencer.c: leave_merge:
>         	return ret;
>         }
>         
>       --static int do_update_refs(struct repository *r)
>       -+struct update_refs_context {
>       -+	struct ref_store *refs;
>       -+	char **ref_names;
>       -+	struct object_id *old;
>       -+	struct object_id *new;
>       -+	size_t nr;
>       -+	size_t alloc;
>       -+};
>       -+
>       -+static int add_ref_to_context(const char *refname,
>       -+			      const struct object_id *oid,
>       -+			      int flags,
>       -+			      void *data)
>       +-static int do_update_ref(struct repository *r, const char *ref_name)
>       ++static int write_update_refs_state(struct string_list *refs_to_oids)
>       ++{
>       ++	int result = 0;
>       ++	FILE *fp = NULL;
>       ++	struct string_list_item *item;
>       ++	char *path = xstrdup(rebase_path_update_refs());
>       ++
>       ++	if (safe_create_leading_directories(path)) {
>       ++		result = error(_("unable to create leading directories of %s"),
>       ++			       path);
>       ++		goto cleanup;
>       ++	}
>       ++
>       ++	fp = fopen(path, "w");
>       ++	if (!fp) {
>       ++		result = error_errno(_("could not open '%s' for writing"), path);
>       ++		goto cleanup;
>       ++	}
>       ++
>       ++	for_each_string_list_item(item, refs_to_oids)
>       ++		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
>       ++
>       ++cleanup:
>       ++	if (fp)
>       ++		fclose(fp);
>       ++	return result;
>       ++}
>       ++
>       ++static int do_update_ref(struct repository *r, const char *refname)
>         {
>       -+	int f = 0;
>       -+	const char *name;
>       -+	struct update_refs_context *ctx = data;
>       ++	struct string_list_item *item;
>       ++	struct string_list list = STRING_LIST_INIT_DUP;
>       ++	int found = 0;
>       ++
>       ++	sequencer_get_update_refs_state(r->gitdir, &list);
>        +
>       -+	ALLOC_GROW(ctx->ref_names, ctx->nr + 1, ctx->alloc);
>       -+	ALLOC_GROW(ctx->old, ctx->nr + 1, ctx->alloc);
>       -+	ALLOC_GROW(ctx->new, ctx->nr + 1, ctx->alloc);
>       ++	for_each_string_list_item(item, &list) {
>       ++		if (!strcmp(item->string, refname)) {
>       ++			struct object_id oid;
>       ++			free(item->util);
>       ++			found = 1;
>        +
>       -+	if (!skip_prefix(refname, "refs/rewritten/for-update-refs/", &name))
>       -+		return 1;
>       ++			if (!read_ref("HEAD", &oid)) {
>       ++				item->util = oiddup(&oid);
>       ++				break;
>       ++			}
>       ++		}
>       ++	}
>        +
>       -+	ctx->ref_names[ctx->nr] = xstrdup(name);
>       -+	oidcpy(&ctx->new[ctx->nr], oid);
>       -+	if (!refs_resolve_ref_unsafe(ctx->refs, name, 0,
>       -+				     &ctx->old[ctx->nr], &f))
>       -+		return 1;
>       ++	if (!found) {
>       ++		struct object_id oid;
>       ++		item = string_list_append(&list, refname);
>        +
>       -+	ctx->nr++;
>       ++		if (!read_ref("HEAD", &oid))
>       ++			item->util = oiddup(&oid);
>       ++		else
>       ++			item->util = oiddup(the_hash_algo->null_oid);
>       ++	}
>       ++
>       ++	write_update_refs_state(&list);
>       ++	string_list_clear(&list, 1);
>         	return 0;
>         }
>         
>        +static int do_update_refs(struct repository *r)
>        +{
>       -+	int i, res;
>       -+	struct update_refs_context ctx = {
>       -+		.refs = get_main_ref_store(r),
>       -+		.alloc = 16,
>       -+	};
>       -+	ALLOC_ARRAY(ctx.ref_names, ctx.alloc);
>       -+	ALLOC_ARRAY(ctx.old, ctx.alloc);
>       -+	ALLOC_ARRAY(ctx.new, ctx.alloc);
>       -+
>       -+	res = refs_for_each_fullref_in(ctx.refs,
>       -+				       "refs/rewritten/for-update-refs/",
>       -+				       add_ref_to_context,
>       -+				       &ctx);
>       -+
>       -+	for (i = 0; !res && i < ctx.nr; i++)
>       -+		res = refs_update_ref(ctx.refs, "rewritten during rebase",
>       -+				ctx.ref_names[i],
>       -+				&ctx.new[i], &ctx.old[i],
>       ++	int res = 0;
>       ++	struct string_list_item *item;
>       ++	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
>       ++	struct ref_store *refs = get_main_ref_store(r);
>       ++
>       ++	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
>       ++
>       ++	for_each_string_list_item(item, &refs_to_oids) {
>       ++		struct object_id *oid_to = item->util;
>       ++		struct object_id oid_from;
>       ++
>       ++		if (oideq(oid_to, the_hash_algo->null_oid)) {
>       ++			/*
>       ++			 * Ref was not updated. User may have deleted the
>       ++			 * 'update-ref' step.
>       ++			 */
>       ++			continue;
>       ++		}
>       ++
>       ++		if (read_ref(item->string, &oid_from)) {
>       ++			/*
>       ++			 * The ref does not exist. The user probably
>       ++			 * inserted a new 'update-ref' step with a new
>       ++			 * branch name.
>       ++			 */
>       ++			oidcpy(&oid_from, the_hash_algo->null_oid);
>       ++		}
>       ++
>       ++		res |= refs_update_ref(refs, "rewritten during rebase",
>       ++				item->string,
>       ++				oid_to, &oid_from,
>        +				0, UPDATE_REFS_MSG_ON_ERR);
>       ++	}
>        +
>       -+	for (i = 0; i < ctx.nr; i++)
>       -+		free(ctx.ref_names[i]);
>       -+	free(ctx.ref_names);
>       -+	free(ctx.old);
>       -+	free(ctx.new);
>       ++	string_list_clear(&refs_to_oids, 1);
>        +	return res;
>        +}
>        +
>         static int is_final_fixup(struct todo_list *todo_list)
>         {
>         	int i = todo_list->current;
>       +@@ sequencer.c: cleanup_head_ref:
>       + 		strbuf_release(&head_ref);
>       + 	}
>       +
>       ++	do_update_refs(r);
>       ++
>       + 	/*
>       + 	 * Sequence of picks finished successfully; cleanup by
>       + 	 * removing the .git/sequencer directory
>       +@@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>       + 		}
>       + 	}
>       +
>       ++	write_update_refs_state(&ctx.refs_to_oids);
>       ++
>       + 	string_list_clear(&ctx.refs_to_oids, 1);
>       + 	free(todo_list->items);
>       + 	todo_list->items = ctx.items;
>        
>         ## t/t3404-rebase-interactive.sh ##
>        @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands with --rebase-merges' '
>       @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands
>        +	git branch -f third HEAD~1 &&
>        +	test_commit extra2 fileX &&
>        +	git commit --amend --fixup=L &&
>       -+	(
>       -+		git rebase -i --autosquash --update-refs primary &&
>       -+
>       -+		compare_two_refs HEAD~3 refs/heads/first &&
>       -+		compare_two_refs HEAD~3 refs/heads/second &&
>       -+		compare_two_refs HEAD~1 refs/heads/third &&
>       -+		compare_two_refs HEAD refs/heads/no-conflict-branch
>       -+	)
>       ++
>       ++	git rebase -i --autosquash --update-refs primary &&
>       ++
>       ++	compare_two_refs HEAD~3 refs/heads/first &&
>       ++	compare_two_refs HEAD~3 refs/heads/second &&
>       ++	compare_two_refs HEAD~1 refs/heads/third &&
>       ++	compare_two_refs HEAD refs/heads/no-conflict-branch
>        +'
>        +
>         # This must be the last test in this file
>   7:  3d7d3f656b4 ! 8:  d2cfdbfc431 rebase: add rebase.updateRefs config option
>       @@ Documentation/git-rebase.txt: start would be overridden by the presence of
>         	or point to a `squash! ...` or `fixup! ...` commit are not updated
>         	in this way.
>        ++
>       -+If the `--update-refs` option is enabled by default using the
>       -+configuration variable `rebase.updateRefs`, this option can be
>       -+used to override and disable this setting.
>       ++If the configuration variable `rebase.updateRefs` is set, then this option
>       ++can be used to override and disable this setting.
>         
>         INCOMPATIBLE OPTIONS
>         --------------------
> 

[PATCH v4 00/12] rebase: update branches in multi-part topic

[Derrick Stolee via GitGitGadget]

This series is based on ds/branch-checked-out.

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'update-ref' commands into the
TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
update-ref refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
update-ref refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
update-ref refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

update-ref refs/heads/bundle-redo/advertise


Here is an outline of the series:

 * Patch 1 updates some tests for branch_checked_out() to use 'git bisect'
   and 'git rebase' as black-boxes instead of manually editing files inside
   $GIT_DIR. (Thanks, Junio!)
 * Patch 2 updates some tests for branch_checked_out() for the 'apply'
   backend.
 * Patch 3 updates branch_checked_out() to parse the
   rebase-merge/update-refs file to block concurrent ref updates and
   checkouts on branches selected by --update-refs.
 * Patch 4 updates the todo list documentation to remove some unnecessary
   dots in the 'merge' command. This makes it consistent with the 'fixup'
   command before we document the 'update-ref' command.
 * Patch 5 updates the definition of todo_command_info to use enum values as
   array indices.
 * Patches 6-8 implement the --update-refs logic itself.
 * Patch 9 specifically updates the update-refs file every time the user
   edits the todo-list (Thanks Phillip!)
 * Patch 10 adds the rebase.updateRefs config option similar to
   rebase.autoSquash.
 * Patch 11 ignores the HEAD ref when creating the todo list instead of
   making a comment (Thanks Elijah!)
 * Patch 12 adds messaging to the end of the rebase stating which refs were
   updated (Thanks Elijah!)


Updates in v4
=============

This version took longer than I'd hoped (I had less time to work on it than
anticipated) but it also has some major updates. These major updates are
direct responses to the significant review this series has received. Thank
you!

 * The update-refs file now stores "ref/before/after" triples (still
   separated by lines). This allows us to store the "before" OID of a ref in
   addition to the "after" that we will write to that ref at the end of the
   rebase. This allows us to do a "force-with-lease" update. The
   branch_checked_out() updates should prevent Git from updating those refs
   while under the rebase, but older versions and third-party tools don't
   have that protection.
 * The update-refs file is updated with every update to the todo-list file.
   This allows for some advanced changes to the file, including removing,
   adding, and duplicating 'update-ref' commands.
 * The message at the end of the rebase process now lists which refs were
   updated with the update-ref steps. This includes any ref updates that
   fail.
 * The branch_checked_out() tests now use 'git bisect' and 'git rebase' as
   black-boxes instead of testing their internals directly.

Here are the more minor updates:

 * Dropped an unnecessary stat() call.
 * Updated commit messages to include extra details, based on confusion in
   last round.
 * The HEAD branch no longer appears as a comment line in the initial todo
   list.
 * The update-refs file is now written using a lockfile.
 * Tests now use test_cmp_rev.
 * A memory leak ('path' variable) is resolved.


Updates in v3
=============

 * The branch_checked_out() API was extracted to its own topic and is now
   the ds/branch-checked-out branch. This series is now based on that one.
 * The for_each_decoration() API was removed, since it became trivial once
   it did not take a commit directly.
 * The branch_checked_out() tests did not verify the rebase-apply data (for
   the apply backend), so that is fixed.
 * Instead of using the 'label' command and a final 'update-refs' command in
   the todo list, use a new 'update-ref ' command. This command updates the
   rebase-merge/update-refs file with the OID of HEAD at these steps. At the
   very end of the rebase sequence, those refs are updated to the stored OID
   values (assuming that they were not removed by the user, in which case we
   notice that the OID is the null OID and we do nothing).
 * New tests are added.
 * The todo-list comment documentation has some new formatting updates, but
   also includes a description of 'update-refs' in this version.


Updates in v2
=============

As recommended by the excellent feedback, I have removed the 'exec' commands
in favor of the 'label' commands and a new 'update-refs' command at the very
end. This way, there is only one step that updates all of the refs at the
end instead of updating refs during the rebase. If a user runs 'git rebase
--abort' in the middle, then their refs are still where they need to be.

Based on some of the discussion, it seemed like one way to do this would be
to have an 'update-ref ' command that would take the place of these 'label'
commands. However, this would require two things that make it a bit awkward:

 1. We would need to replicate the storage of those positions during the
    rebase. 'label' already does this pretty well. I've added the
    "for-update-refs/" label to help here.
 2. If we want to close out all of the refs as the rebase is finishing, then
    that "step" becomes invisible to the user (and a bit more complicated to
    insert). Thus, the 'update-refs' step performs this action. If the user
    wants to do things after that step, then they can do so by editing the
    TODO list.

Other updates:

 * The 'keep_decorations' parameter was renamed to 'update_refs'.
 * I added tests for --rebase-merges=rebase-cousins to show how these labels
   interact with other labels and merge commands.
 * I changed the order of the insertion of these update-refs labels to be
   before the fixups are rearranged. This fixes a bug where the tip commit
   is a fixup! so its decorations are never inspected (and they would be in
   the wrong place even if they were). The fixup! commands are properly
   inserted between a pick and its following label command. Tests
   demonstrate this is correct.
 * Numerous style choices are updated based on feedback.

Thank you for all of the detailed review and ideas in this space. I
appreciate any more ideas that can make this feature as effective as it can
be.

Thanks, -Stolee

Derrick Stolee (12):
  t2407: test bisect and rebase as black-boxes
  t2407: test branches currently using apply backend
  branch: consider refs under 'update-refs'
  rebase-interactive: update 'merge' description
  sequencer: define array with enum values
  sequencer: add update-ref command
  rebase: add --update-refs option
  rebase: update refs from 'update-ref' commands
  sequencer: rewrite update-refs as user edits todo list
  rebase: add rebase.updateRefs config option
  sequencer: ignore HEAD ref under --update-refs
  sequencer: notify user of --update-refs activity

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  11 +
 branch.c                        |  13 +
 builtin/rebase.c                |  10 +
 rebase-interactive.c            |  15 +-
 sequencer.c                     | 469 +++++++++++++++++++++++++++++++-
 sequencer.h                     |  23 ++
 t/lib-rebase.sh                 |  15 +
 t/t2407-worktree-heads.sh       | 103 +++++--
 t/t3404-rebase-interactive.sh   | 269 ++++++++++++++++++
 10 files changed, 887 insertions(+), 44 deletions(-)


base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1247

Range-diff vs v3:

  -:  ----------- >  1:  9e53a27017a t2407: test bisect and rebase as black-boxes
  1:  fbaedc7f1f0 !  2:  540a3be256f t2407: test branches currently using apply backend
     @@ Commit message
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
       ## t/t2407-worktree-heads.sh ##
     -@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite: worktree in bisect' '
     - 	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
     +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
     + 	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
       '
       
     --test_expect_success 'refuse to overwrite: worktree in rebase' '
     -+test_expect_success 'refuse to overwrite: worktree in rebase (apply)' '
     -+	test_when_finished rm -rf .git/worktrees/wt-*/rebase-apply &&
     +-test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
     ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (apply)' '
     ++	test_when_finished git -C wt-2 rebase --abort &&
      +
     -+	mkdir -p .git/worktrees/wt-3/rebase-apply &&
     -+	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-apply/head-name &&
     -+	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-apply/onto &&
     ++	# This will fail part-way through due to a conflict.
     ++	test_must_fail git -C wt-2 rebase --apply conflict-2 &&
      +
     -+	test_must_fail git branch -f fake-1 HEAD 2>err &&
     -+	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
     ++	test_must_fail git branch -f wt-2 HEAD 2>err &&
     ++	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
      +'
      +
     -+test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
     - 	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
     ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (merge)' '
     + 	test_when_finished git -C wt-2 rebase --abort &&
       
     - 	mkdir -p .git/worktrees/wt-3/rebase-merge &&
     + 	# This will fail part-way through due to a conflict.
  2:  2bc647b6fcd !  3:  bf301a054e3 branch: consider refs under 'update-refs'
     @@ Commit message
          branch_checked_out().
      
          The data store is a plaintext file inside the 'rebase-merge' directory
     -    for that worktree. The file alternates refnames and OIDs. The OIDs will
     -    be used to store the to-be-written values as the rebase progresses, but
     -    can be ignored at the moment.
     +    for that worktree. The file lists refnames followed by two OIDs, each on
     +    separate lines. The OIDs will be used to store the original values of
     +    the refs and the to-be-written values as the rebase progresses, but can
     +    be ignored at the moment.
      
          Create a new sequencer_get_update_refs_state() method that parses this
          file and populates a struct string_list with the ref-OID pairs. We can
     @@ Commit message
          method.
      
          We can test that this works without having Git write this file by
     -    artificially creating one in our test script.
     +    artificially creating one in our test script, at least until 'git rebase
     +    --update-refs' is implemented and we can use it directly.
      
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
     @@ branch.c: static void prepare_checked_out_branches(void)
       	free_worktrees(worktrees);
      
       ## sequencer.c ##
     +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
     +  */
     + static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
     + 
     ++/*
     ++ * The update-refs file stores a list of refs that will be updated at the end
     ++ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
     ++ * update the OIDs for the refs in this file, but the refs are not updated
     ++ * until the end of the rebase sequence.
     ++ *
     ++ * rebase_path_update_refs() returns the path to this file for a given
     ++ * worktree directory. For the current worktree, pass the_repository->gitdir.
     ++ */
     ++static char *rebase_path_update_refs(const char *wt_dir)
     ++{
     ++	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
     ++}
     ++
     + /*
     +  * The following files are written by git-rebase just after parsing the
     +  * command-line.
     +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_no_reschedule_failed_exec, "rebase-merge/no-res
     + static GIT_PATH_FUNC(rebase_path_drop_redundant_commits, "rebase-merge/drop_redundant_commits")
     + static GIT_PATH_FUNC(rebase_path_keep_redundant_commits, "rebase-merge/keep_redundant_commits")
     + 
     ++/**
     ++ * A 'struct update_refs_record' represents a value in the update-refs
     ++ * list. We use a string_list to map refs to these (before, after) pairs.
     ++ */
     ++struct update_ref_record {
     ++	struct object_id before;
     ++	struct object_id after;
     ++};
     ++
     + static int git_sequencer_config(const char *k, const char *v, void *cb)
     + {
     + 	struct replay_opts *opts = cb;
      @@ sequencer.c: int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
       
       	return 0;
     @@ sequencer.c: int sequencer_determine_whence(struct repository *r, enum commit_wh
      +				    struct string_list *refs)
      +{
      +	int result = 0;
     -+	struct stat st;
      +	FILE *fp = NULL;
      +	struct strbuf ref = STRBUF_INIT;
      +	struct strbuf hash = STRBUF_INIT;
     -+	char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);
     ++	struct update_ref_record *rec = NULL;
      +
     -+	if (stat(path, &st))
     -+		goto cleanup;
     ++	char *path = rebase_path_update_refs(wt_dir);
      +
      +	fp = fopen(path, "r");
      +	if (!fp)
      +		goto cleanup;
      +
      +	while (strbuf_getline(&ref, fp) != EOF) {
     -+		struct object_id oid;
      +		struct string_list_item *item;
      +
     ++		CALLOC_ARRAY(rec, 1);
     ++
      +		if (strbuf_getline(&hash, fp) == EOF ||
     -+		    get_oid_hex(hash.buf, &oid)) {
     ++		    get_oid_hex(hash.buf, &rec->before)) {
      +			warning(_("update-refs file at '%s' is invalid"),
      +				  path);
      +			result = -1;
      +			goto cleanup;
      +		}
      +
     -+		item = string_list_append(refs, ref.buf);
     -+		item->util = oiddup(&oid);
     ++		if (strbuf_getline(&hash, fp) == EOF ||
     ++		    get_oid_hex(hash.buf, &rec->after)) {
     ++			warning(_("update-refs file at '%s' is invalid"),
     ++				  path);
     ++			result = -1;
     ++			goto cleanup;
     ++		}
     ++
     ++		item = string_list_insert(refs, ref.buf);
     ++		item->util = rec;
     ++		rec = NULL;
      +	}
      +
      +cleanup:
      +	if (fp)
      +		fclose(fp);
      +	free(path);
     ++	free(rec);
      +	strbuf_release(&ref);
      +	strbuf_release(&hash);
      +	return result;
     @@ sequencer.h: void sequencer_post_commit_cleanup(struct repository *r, int verbos
       #endif /* SEQUENCER_H */
      
       ## t/t2407-worktree-heads.sh ##
     -@@ t/t2407-worktree-heads.sh: TEST_PASSES_SANITIZE_LEAK=true
     +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
     + 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
     + '
       
     - test_expect_success 'setup' '
     - 	test_commit init &&
     --	git branch -f fake-1 &&
     --	git branch -f fake-2 &&
     ++test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
     ++	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
     ++
     ++	mkdir -p .git/worktrees/wt-3/rebase-merge &&
     ++	touch .git/worktrees/wt-3/rebase-merge/interactive &&
      +
     -+	for i in 1 2 3 4
     -+	do
     -+		git branch -f fake-$i || return 1
     -+	done &&
     - 
     - 	for i in 1 2 3 4
     - 	do
     -@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
     - 	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-merge/head-name &&
     - 	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-merge/onto &&
     - 
     --	test_must_fail git branch -f fake-1 HEAD 2>err &&
     --	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
      +	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
      +	refs/heads/fake-3
      +	$(git rev-parse HEAD~1)
     ++	$(git rev-parse HEAD)
      +	refs/heads/fake-4
      +	$(git rev-parse HEAD)
     ++	$(git rev-parse HEAD)
      +	EOF
      +
     -+	for i in 1 3 4
     ++	for i in 3 4
      +	do
      +		test_must_fail git branch -f fake-$i HEAD 2>err &&
      +		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
      +			return 1
      +	done
     - '
     - 
     ++'
     ++
       test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
     + 	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
     + 	grep "refusing to fetch into branch '\''refs/heads/wt-3'\''" err &&
  3:  669f4abd59e !  4:  dec95681d2b rebase-interactive: update 'merge' description
     @@ Commit message
          The 'merge' command description for the todo list documentation in an
          interactive rebase has multiple lines. The lines other than the first
          one start with dots ('.') while the similar multi-line documentation for
     -    'fixup' does not.
     +    'fixup' does not. This description only appears in the comment text of
     +    the todo file during an interactive rebase.
      
          The 'merge' command was documented when interactive rebase was first
          ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
  4:  6528a50343f =  5:  b2c09600918 sequencer: define array with enum values
  5:  e95ad41d355 =  6:  fa7ecb718cf sequencer: add update-ref command
  6:  918b398d6a2 !  7:  3ec2cc922f9 rebase: add --update-refs option
     @@ Commit message
          'update-ref' commands. Tests are added to ensure that these todo
          commands are added in the correct locations.
      
     -    A future change will update the behavior to actually update the refs
     -    at the end of the rebase sequence.
     +    This change does _not_ include the actual behavior of tracking the
     +    updated refs and writing the new ref values at the end of the rebase
     +    process. That is deferred to a later change.
      
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
     @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
      +			item->command = TODO_UPDATE_REF;
      +			strbuf_addf(ctx->buf, "%s\n", decoration->name);
      +
     -+			sti = string_list_append(&ctx->refs_to_oids,
     ++			sti = string_list_insert(&ctx->refs_to_oids,
      +						 decoration->name);
      +			sti->util = oiddup(the_hash_algo->null_oid);
      +		}
     @@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite when in erro
      +		grep "update-ref refs/heads/allow-update" todo
      +	)
      +'
     ++
     ++# This must be the last test in this file
     ++test_expect_success '$EDITOR and friends are unchanged' '
     ++	test_editor_unchanged
     ++'
      +
       test_done
      
  7:  72e0481b643 !  8:  fb5f64c5201 rebase: update refs from 'update-ref' commands
     @@ Commit message
          interactive rebase.
      
          Teach Git to record the HEAD position when reaching these 'update-ref'
     -    commands. The ref/OID pair is stored in the
     +    commands. The ref/before/after triple is stored in the
          $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
          file to avoid having other processes updating the refs in that file
          while the rebase is in progress.
     @@ Commit message
          Not only do we update the file when the sequencer reaches these
          'update-ref' commands, we then update the refs themselves at the end of
          the rebase sequence. If the rebase is aborted before this final step,
     -    then the refs are not updated.
     +    then the refs are not updated. The 'before' value is used to ensure that
     +    we do not accidentally obliterate a ref that was updated concurrently
     +    (say, by an older version of Git or a third-party tool).
     +
     +    Now that the 'git rebase --update-refs' command is implemented to write
     +    to the update-refs file, we can remove the fake construction of the
     +    update-refs file from a test in t2407-worktree-heads.sh.
      
          Signed-off-by: Derrick Stolee <derrickstolee@github.com>
      
     @@ sequencer.c
       
       #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
       
     -@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
     -  */
     - static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
     +@@ sequencer.c: struct update_ref_record {
     + 	struct object_id after;
     + };
       
     -+/*
     -+ * The update-refs file stores a list of refs that will be updated at the end
     -+ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
     -+ * update the OIDs for the refs in this file, but the refs are not updated
     -+ * until the end of the rebase sequence.
     -+ */
     -+static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
     -+
     - /*
     -  * The following files are written by git-rebase just after parsing the
     -  * command-line.
     ++static struct update_ref_record *init_update_ref_record(const char *ref)
     ++{
     ++	struct update_ref_record *rec = xmalloc(sizeof(*rec));
     ++
     ++	oidcpy(&rec->before, null_oid());
     ++	oidcpy(&rec->after, null_oid());
     ++
     ++	/* This may fail, but that's fine, we will keep the null OID. */
     ++	read_ref(ref, &rec->before);
     ++
     ++	return rec;
     ++}
     ++
     + static int git_sequencer_config(const char *k, const char *v, void *cb)
     + {
     + 	struct replay_opts *opts = cb;
      @@ sequencer.c: leave_merge:
       	return ret;
       }
       
      -static int do_update_ref(struct repository *r, const char *ref_name)
      +static int write_update_refs_state(struct string_list *refs_to_oids)
     -+{
     + {
      +	int result = 0;
     ++	struct lock_file lock = LOCK_INIT;
      +	FILE *fp = NULL;
      +	struct string_list_item *item;
     -+	char *path = xstrdup(rebase_path_update_refs());
     ++	char *path;
     ++
     ++	if (!refs_to_oids->nr)
     ++		return 0;
     ++
     ++	path = rebase_path_update_refs(the_repository->gitdir);
      +
      +	if (safe_create_leading_directories(path)) {
      +		result = error(_("unable to create leading directories of %s"),
     @@ sequencer.c: leave_merge:
      +		goto cleanup;
      +	}
      +
     -+	fp = fopen(path, "w");
     ++	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
     ++		result = error(_("another 'rebase' process appears to be running; "
     ++				 "'%s.lock' already exists"),
     ++			       path);
     ++		goto cleanup;
     ++	}
     ++
     ++	fp = fdopen_lock_file(&lock, "w");
      +	if (!fp) {
      +		result = error_errno(_("could not open '%s' for writing"), path);
     ++		rollback_lock_file(&lock);
      +		goto cleanup;
      +	}
      +
     -+	for_each_string_list_item(item, refs_to_oids)
     -+		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
     ++	for_each_string_list_item(item, refs_to_oids) {
     ++		struct update_ref_record *rec = item->util;
     ++		fprintf(fp, "%s\n%s\n%s\n", item->string,
     ++			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
     ++	}
     ++
     ++	result = commit_lock_file(&lock);
      +
      +cleanup:
     -+	if (fp)
     -+		fclose(fp);
     ++	free(path);
      +	return result;
      +}
      +
      +static int do_update_ref(struct repository *r, const char *refname)
     - {
     ++{
      +	struct string_list_item *item;
      +	struct string_list list = STRING_LIST_INIT_DUP;
     -+	int found = 0;
      +
      +	sequencer_get_update_refs_state(r->gitdir, &list);
      +
      +	for_each_string_list_item(item, &list) {
      +		if (!strcmp(item->string, refname)) {
     -+			struct object_id oid;
     -+			free(item->util);
     -+			found = 1;
     -+
     -+			if (!read_ref("HEAD", &oid)) {
     -+				item->util = oiddup(&oid);
     -+				break;
     -+			}
     ++			struct update_ref_record *rec = item->util;
     ++			read_ref("HEAD", &rec->after);
     ++			break;
      +		}
      +	}
      +
     -+	if (!found) {
     -+		struct object_id oid;
     -+		item = string_list_append(&list, refname);
     -+
     -+		if (!read_ref("HEAD", &oid))
     -+			item->util = oiddup(&oid);
     -+		else
     -+			item->util = oiddup(the_hash_algo->null_oid);
     -+	}
     -+
      +	write_update_refs_state(&list);
      +	string_list_clear(&list, 1);
       	return 0;
     @@ sequencer.c: leave_merge:
      +	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
      +
      +	for_each_string_list_item(item, &refs_to_oids) {
     -+		struct object_id *oid_to = item->util;
     -+		struct object_id oid_from;
     ++		struct update_ref_record *rec = item->util;
      +
     -+		if (oideq(oid_to, the_hash_algo->null_oid)) {
     ++		if (oideq(&rec->after, the_hash_algo->null_oid)) {
      +			/*
      +			 * Ref was not updated. User may have deleted the
      +			 * 'update-ref' step.
     @@ sequencer.c: leave_merge:
      +			continue;
      +		}
      +
     -+		if (read_ref(item->string, &oid_from)) {
     -+			/*
     -+			 * The ref does not exist. The user probably
     -+			 * inserted a new 'update-ref' step with a new
     -+			 * branch name.
     -+			 */
     -+			oidcpy(&oid_from, the_hash_algo->null_oid);
     -+		}
     -+
      +		res |= refs_update_ref(refs, "rewritten during rebase",
     -+				item->string,
     -+				oid_to, &oid_from,
     -+				0, UPDATE_REFS_MSG_ON_ERR);
     ++				       item->string,
     ++				       &rec->after, &rec->before,
     ++				       0, UPDATE_REFS_MSG_ON_ERR);
      +	}
      +
      +	string_list_clear(&refs_to_oids, 1);
     @@ sequencer.c: cleanup_head_ref:
       	/*
       	 * Sequence of picks finished successfully; cleanup by
       	 * removing the .git/sequencer directory
     +@@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
     + 
     + 			sti = string_list_insert(&ctx->refs_to_oids,
     + 						 decoration->name);
     +-			sti->util = oiddup(the_hash_algo->null_oid);
     ++			sti->util = init_update_ref_record(decoration->name);
     + 		}
     + 
     + 		item->offset_in_buf = base_offset;
      @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
       		}
       	}
     @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo
       	free(todo_list->items);
       	todo_list->items = ctx.items;
      
     + ## t/t2407-worktree-heads.sh ##
     +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
     + 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
     + '
     + 
     +-test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
     +-	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
     +-
     +-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
     +-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
     ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
     ++	test_when_finished git -C wt-3 rebase --abort &&
     + 
     +-	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
     +-	refs/heads/fake-3
     +-	$(git rev-parse HEAD~1)
     +-	$(git rev-parse HEAD)
     +-	refs/heads/fake-4
     +-	$(git rev-parse HEAD)
     +-	$(git rev-parse HEAD)
     +-	EOF
     ++	git branch -f can-be-updated wt-3 &&
     ++	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
     + 
     + 	for i in 3 4
     + 	do
     +-		test_must_fail git branch -f fake-$i HEAD 2>err &&
     +-		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
     ++		test_must_fail git branch -f can-be-updated HEAD 2>err &&
     ++		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
     + 			return 1
     + 	done
     + '
     +
       ## t/t3404-rebase-interactive.sh ##
      @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands with --rebase-merges' '
       	)
       '
       
     -+compare_two_refs () {
     -+	git rev-parse $1 >expect &&
     -+	git rev-parse $2 >actual &&
     -+	test_cmp expect actual
     -+}
     -+
      +test_expect_success '--update-refs updates refs correctly' '
      +	git checkout -B update-refs no-conflict-branch &&
      +	git branch -f base HEAD~4 &&
     @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands
      +
      +	git rebase -i --autosquash --update-refs primary &&
      +
     -+	compare_two_refs HEAD~3 refs/heads/first &&
     -+	compare_two_refs HEAD~3 refs/heads/second &&
     -+	compare_two_refs HEAD~1 refs/heads/third &&
     -+	compare_two_refs HEAD refs/heads/no-conflict-branch
     ++	test_cmp_rev HEAD~3 refs/heads/first &&
     ++	test_cmp_rev HEAD~3 refs/heads/second &&
     ++	test_cmp_rev HEAD~1 refs/heads/third &&
     ++	test_cmp_rev HEAD refs/heads/no-conflict-branch
      +'
      +
       # This must be the last test in this file
  -:  ----------- >  9:  29c7c76805a sequencer: rewrite update-refs as user edits todo list
  8:  d2cfdbfc431 = 10:  c0022d07579 rebase: add rebase.updateRefs config option
  -:  ----------- > 11:  d53b4ff2cee sequencer: ignore HEAD ref under --update-refs
  -:  ----------- > 12:  d5cd4b49e46 sequencer: notify user of --update-refs activity

-- 
gitgitgadget

[PATCH v4 01/12] t2407: test bisect and rebase as black-boxes

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The tests added by d2ba271aad0 (branch: check for bisects and rebases,
2022-06-14) modified hidden state to verify the branch_checked_out()
helper. While this indeed checks that the method implementation is _as
designed_, it doesn't show that it is _correct_. Specifically, if 'git
bisect' or 'git rebase' change their back-end for preserving refs, then
these tests do not demonstrate that drift as a bug in
branch_checked_out().

Modify the tests in t2407 to actually rely on a paused bisect or rebase.
This requires adding the !SANITIZE_LEAK prereq for tests using those
builtins. The logic is still tested for leaks in the final test which
does set up that back-end directly for an error state that should not be
possible using Git commands.

Reported-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 t/t2407-worktree-heads.sh | 57 +++++++++++++++++++++------------------
 1 file changed, 31 insertions(+), 26 deletions(-)

diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index b6be42f74a2..100ab286d5c 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -7,13 +7,18 @@ TEST_PASSES_SANITIZE_LEAK=true
 
 test_expect_success 'setup' '
 	test_commit init &&
-	git branch -f fake-1 &&
-	git branch -f fake-2 &&
 
 	for i in 1 2 3 4
 	do
+		git checkout -b conflict-$i &&
+		echo "not I" >$i.t &&
+		git add $i.t &&
+		git commit -m "will conflict" &&
+
+		git checkout - &&
 		test_commit $i &&
 		git branch wt-$i &&
+		git branch fake-$i &&
 		git worktree add wt-$i wt-$i || return 1
 	done &&
 
@@ -44,26 +49,26 @@ test_expect_success 'refuse to overwrite: checked out in worktree' '
 	done
 '
 
-test_expect_success 'refuse to overwrite: worktree in bisect' '
-	test_when_finished rm -rf .git/worktrees/wt-*/BISECT_* &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
+	test_when_finished git -C wt-4 bisect reset &&
 
-	touch .git/worktrees/wt-4/BISECT_LOG &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/BISECT_START &&
+	# Set up a bisect so HEAD no longer points to wt-4.
+	git -C wt-4 bisect start &&
+	git -C wt-4 bisect bad wt-4 &&
+	git -C wt-4 bisect good wt-1 &&
 
-	test_must_fail git branch -f fake-2 HEAD 2>err &&
-	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
+	test_must_fail git branch -f wt-4 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase' '
-	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
+	test_when_finished git -C wt-2 rebase --abort &&
 
-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
-	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-merge/head-name &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-merge/onto &&
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-2 rebase conflict-2 &&
 
-	test_must_fail git branch -f fake-1 HEAD 2>err &&
-	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
+	test_must_fail git branch -f wt-2 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
@@ -77,24 +82,24 @@ test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: worktree in bisect' '
-	test_when_finished rm -rf .git/worktrees/wt-*/BISECT_* &&
+	test_when_finished git -C wt-4 bisect reset &&
 
-	touch .git/worktrees/wt-4/BISECT_LOG &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/BISECT_START &&
+	# Set up a bisect so HEAD no longer points to wt-4.
+	git -C wt-4 bisect start &&
+	git -C wt-4 bisect bad wt-4 &&
+	git -C wt-4 bisect good wt-1 &&
 
-	test_must_fail git fetch server +refs/heads/fake-2:refs/heads/fake-2 2>err &&
+	test_must_fail git fetch server +refs/heads/wt-4:refs/heads/wt-4 2>err &&
 	grep "refusing to fetch into branch" err
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: worktree in rebase' '
-	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
+	test_when_finished git -C wt-3 rebase --abort &&
 
-	mkdir -p .git/worktrees/wt-4/rebase-merge &&
-	touch .git/worktrees/wt-4/rebase-merge/interactive &&
-	echo refs/heads/fake-1 >.git/worktrees/wt-4/rebase-merge/head-name &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/rebase-merge/onto &&
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-3 rebase conflict-3 &&
 
-	test_must_fail git fetch server +refs/heads/fake-1:refs/heads/fake-1 2>err &&
+	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
 	grep "refusing to fetch into branch" err
 '
 
-- 
gitgitgadget

[PATCH v4 02/12] t2407: test branches currently using apply backend

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The tests in t2407 that verify the branch_checked_out() helper in the
case of bisects and rebases were added by 9347303db89 (branch: check for
bisects and rebases, 2022-06-08). However, that commit failed to check
for rebases that are using the 'apply' backend.

Add a test that checks the apply backend. The implementation was already
correct here, but it is good to have regression tests before modifying
the implementation further.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 t/t2407-worktree-heads.sh | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 100ab286d5c..a67ce5fb003 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -61,7 +61,17 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
 	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
 '
 
-test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (apply)' '
+	test_when_finished git -C wt-2 rebase --abort &&
+
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-2 rebase --apply conflict-2 &&
+
+	test_must_fail git branch -f wt-2 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
+'
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (merge)' '
 	test_when_finished git -C wt-2 rebase --abort &&
 
 	# This will fail part-way through due to a conflict.
-- 
gitgitgadget

[PATCH v4 03/12] branch: consider refs under 'update-refs'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The branch_checked_out() helper helps commands like 'git branch' and
'git fetch' from overwriting refs that are currently checked out in
other worktrees.

A future update to 'git rebase' will introduce a new '--update-refs'
option which will update the local refs that point to commits that are
being rebased. To avoid collisions as the rebase completes, we want to
make the future data store for these refs to be considered by
branch_checked_out().

The data store is a plaintext file inside the 'rebase-merge' directory
for that worktree. The file lists refnames followed by two OIDs, each on
separate lines. The OIDs will be used to store the original values of
the refs and the to-be-written values as the rebase progresses, but can
be ignored at the moment.

Create a new sequencer_get_update_refs_state() method that parses this
file and populates a struct string_list with the ref-OID pairs. We can
then use this list to add to the current_checked_out_branches strmap
used by branch_checked_out().

To properly navigate to the rebase directory for a given worktree,
extract the static strbuf_worktree_gitdir() method to a public API
method.

We can test that this works without having Git write this file by
artificially creating one in our test script, at least until 'git rebase
--update-refs' is implemented and we can use it directly.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 branch.c                  | 13 +++++++
 sequencer.c               | 74 +++++++++++++++++++++++++++++++++++++++
 sequencer.h               |  9 +++++
 t/t2407-worktree-heads.sh | 23 ++++++++++++
 4 files changed, 119 insertions(+)

diff --git a/branch.c b/branch.c
index 526e8237673..f252c4eef6c 100644
--- a/branch.c
+++ b/branch.c
@@ -365,6 +365,7 @@ static void prepare_checked_out_branches(void)
 		char *old;
 		struct wt_status_state state = { 0 };
 		struct worktree *wt = worktrees[i++];
+		struct string_list update_refs = STRING_LIST_INIT_DUP;
 
 		if (wt->is_bare)
 			continue;
@@ -400,6 +401,18 @@ static void prepare_checked_out_branches(void)
 			strbuf_release(&ref);
 		}
 		wt_status_state_free_buffers(&state);
+
+		if (!sequencer_get_update_refs_state(get_worktree_git_dir(wt),
+						     &update_refs)) {
+			struct string_list_item *item;
+			for_each_string_list_item(item, &update_refs) {
+				old = strmap_put(&current_checked_out_branches,
+						 item->string,
+						 xstrdup(wt->path));
+				free(old);
+			}
+			string_list_clear(&update_refs, 1);
+		}
 	}
 
 	free_worktrees(worktrees);
diff --git a/sequencer.c b/sequencer.c
index 8c3ed3532ac..e93c61cbd25 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -147,6 +147,20 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
  */
 static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
 
+/*
+ * The update-refs file stores a list of refs that will be updated at the end
+ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
+ * update the OIDs for the refs in this file, but the refs are not updated
+ * until the end of the rebase sequence.
+ *
+ * rebase_path_update_refs() returns the path to this file for a given
+ * worktree directory. For the current worktree, pass the_repository->gitdir.
+ */
+static char *rebase_path_update_refs(const char *wt_dir)
+{
+	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
+}
+
 /*
  * The following files are written by git-rebase just after parsing the
  * command-line.
@@ -169,6 +183,15 @@ static GIT_PATH_FUNC(rebase_path_no_reschedule_failed_exec, "rebase-merge/no-res
 static GIT_PATH_FUNC(rebase_path_drop_redundant_commits, "rebase-merge/drop_redundant_commits")
 static GIT_PATH_FUNC(rebase_path_keep_redundant_commits, "rebase-merge/keep_redundant_commits")
 
+/**
+ * A 'struct update_refs_record' represents a value in the update-refs
+ * list. We use a string_list to map refs to these (before, after) pairs.
+ */
+struct update_ref_record {
+	struct object_id before;
+	struct object_id after;
+};
+
 static int git_sequencer_config(const char *k, const char *v, void *cb)
 {
 	struct replay_opts *opts = cb;
@@ -5901,3 +5924,54 @@ int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
 
 	return 0;
 }
+
+int sequencer_get_update_refs_state(const char *wt_dir,
+				    struct string_list *refs)
+{
+	int result = 0;
+	FILE *fp = NULL;
+	struct strbuf ref = STRBUF_INIT;
+	struct strbuf hash = STRBUF_INIT;
+	struct update_ref_record *rec = NULL;
+
+	char *path = rebase_path_update_refs(wt_dir);
+
+	fp = fopen(path, "r");
+	if (!fp)
+		goto cleanup;
+
+	while (strbuf_getline(&ref, fp) != EOF) {
+		struct string_list_item *item;
+
+		CALLOC_ARRAY(rec, 1);
+
+		if (strbuf_getline(&hash, fp) == EOF ||
+		    get_oid_hex(hash.buf, &rec->before)) {
+			warning(_("update-refs file at '%s' is invalid"),
+				  path);
+			result = -1;
+			goto cleanup;
+		}
+
+		if (strbuf_getline(&hash, fp) == EOF ||
+		    get_oid_hex(hash.buf, &rec->after)) {
+			warning(_("update-refs file at '%s' is invalid"),
+				  path);
+			result = -1;
+			goto cleanup;
+		}
+
+		item = string_list_insert(refs, ref.buf);
+		item->util = rec;
+		rec = NULL;
+	}
+
+cleanup:
+	if (fp)
+		fclose(fp);
+	free(path);
+	free(rec);
+	strbuf_release(&ref);
+	strbuf_release(&hash);
+	return result;
+}
diff --git a/sequencer.h b/sequencer.h
index da64473636b..3ae541bb145 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -232,4 +232,13 @@ void sequencer_post_commit_cleanup(struct repository *r, int verbose);
 int sequencer_get_last_command(struct repository* r,
 			       enum replay_action *action);
 int sequencer_determine_whence(struct repository *r, enum commit_whence *whence);
+
+/**
+ * Append the set of ref-OID pairs that are currently stored for the 'git
+ * rebase --update-refs' feature if such a rebase is currently happening.
+ *
+ * Localized to a worktree's git dir.
+ */
+int sequencer_get_update_refs_state(const char *wt_dir, struct string_list *refs);
+
 #endif /* SEQUENCER_H */
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index a67ce5fb003..97f5c87f8c8 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -81,6 +81,29 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
+test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
+	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
+
+	mkdir -p .git/worktrees/wt-3/rebase-merge &&
+	touch .git/worktrees/wt-3/rebase-merge/interactive &&
+
+	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
+	refs/heads/fake-3
+	$(git rev-parse HEAD~1)
+	$(git rev-parse HEAD)
+	refs/heads/fake-4
+	$(git rev-parse HEAD)
+	$(git rev-parse HEAD)
+	EOF
+
+	for i in 3 4
+	do
+		test_must_fail git branch -f fake-$i HEAD 2>err &&
+		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+			return 1
+	done
+'
+
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
 	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
 	grep "refusing to fetch into branch '\''refs/heads/wt-3'\''" err &&
-- 
gitgitgadget

Re: [PATCH v4 03/12] branch: consider refs under 'update-refs'

[Phillip Wood]

On 12/07/2022 14:06, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> The branch_checked_out() helper helps commands like 'git branch' and
> 'git fetch' from overwriting refs that are currently checked out in
> other worktrees.
> 
> A future update to 'git rebase' will introduce a new '--update-refs'
> option which will update the local refs that point to commits that are
> being rebased. To avoid collisions as the rebase completes, we want to
> make the future data store for these refs to be considered by
> branch_checked_out().
> 
> The data store is a plaintext file inside the 'rebase-merge' directory
> for that worktree. The file lists refnames followed by two OIDs, each on
> separate lines. The OIDs will be used to store the original values of
> the refs and the to-be-written values as the rebase progresses, but can
> be ignored at the moment.
> 
> Create a new sequencer_get_update_refs_state() method that parses this
> file and populates a struct string_list with the ref-OID pairs. We can
> then use this list to add to the current_checked_out_branches strmap
> used by branch_checked_out().
> 
> To properly navigate to the rebase directory for a given worktree,
> extract the static strbuf_worktree_gitdir() method to a public API
> method.
> 
> We can test that this works without having Git write this file by
> artificially creating one in our test script, at least until 'git rebase
> --update-refs' is implemented and we can use it directly.

This looks good, I've left a couple of comments below but I'm not sure 
they're even worth reading let alone replying to!

Best Wishes

Phillip

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   branch.c                  | 13 +++++++
>   sequencer.c               | 74 +++++++++++++++++++++++++++++++++++++++
>   sequencer.h               |  9 +++++
>   t/t2407-worktree-heads.sh | 23 ++++++++++++
>   4 files changed, 119 insertions(+)
> 
> diff --git a/branch.c b/branch.c
> index 526e8237673..f252c4eef6c 100644
> --- a/branch.c
> +++ b/branch.c
> @@ -365,6 +365,7 @@ static void prepare_checked_out_branches(void)
>   		char *old;
>   		struct wt_status_state state = { 0 };
>   		struct worktree *wt = worktrees[i++];
> +		struct string_list update_refs = STRING_LIST_INIT_DUP;
>   
>   		if (wt->is_bare)
>   			continue;
> @@ -400,6 +401,18 @@ static void prepare_checked_out_branches(void)
>   			strbuf_release(&ref);
>   		}
>   		wt_status_state_free_buffers(&state);
> +
> +		if (!sequencer_get_update_refs_state(get_worktree_git_dir(wt),
> +						     &update_refs)) {
> +			struct string_list_item *item;
> +			for_each_string_list_item(item, &update_refs) {
> +				old = strmap_put(&current_checked_out_branches,
> +						 item->string,
> +						 xstrdup(wt->path));
> +				free(old);
> +			}
> +			string_list_clear(&update_refs, 1);
> +		}
>   	}
>   
>   	free_worktrees(worktrees);
> diff --git a/sequencer.c b/sequencer.c
> index 8c3ed3532ac..e93c61cbd25 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -147,6 +147,20 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>    */
>   static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>   
> +/*
> + * The update-refs file stores a list of refs that will be updated at the end
> + * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
> + * update the OIDs for the refs in this file, but the refs are not updated
> + * until the end of the rebase sequence.
> + *
> + * rebase_path_update_refs() returns the path to this file for a given
> + * worktree directory. For the current worktree, pass the_repository->gitdir.
> + */
> +static char *rebase_path_update_refs(const char *wt_dir)

It is nice to see this function. The naming  of wt_dir confused me 
slightly, wt_git_dir would make it clear that this function takes the 
git_dir of the worktree and not the worktree directory. Theu 
documentation does a good job of clarifying that though.

> +{
> +	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
> +}
> +
>   /*
>    * The following files are written by git-rebase just after parsing the
>    * command-line.
> @@ -169,6 +183,15 @@ static GIT_PATH_FUNC(rebase_path_no_reschedule_failed_exec, "rebase-merge/no-res
>   static GIT_PATH_FUNC(rebase_path_drop_redundant_commits, "rebase-merge/drop_redundant_commits")
>   static GIT_PATH_FUNC(rebase_path_keep_redundant_commits, "rebase-merge/keep_redundant_commits")
>   
> +/**
> + * A 'struct update_refs_record' represents a value in the update-refs
> + * list. We use a string_list to map refs to these (before, after) pairs.
> + */
> +struct update_ref_record {
> +	struct object_id before;
> +	struct object_id after;
> +};
> +
>   static int git_sequencer_config(const char *k, const char *v, void *cb)
>   {
>   	struct replay_opts *opts = cb;
> @@ -5901,3 +5924,54 @@ int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
>   
>   	return 0;
>   }
> +
> +int sequencer_get_update_refs_state(const char *wt_dir,
> +				    struct string_list *refs)
> +{
> +	int result = 0;
> +	FILE *fp = NULL;
> +	struct strbuf ref = STRBUF_INIT;
> +	struct strbuf hash = STRBUF_INIT;
> +	struct update_ref_record *rec = NULL;
> +
> +	char *path = rebase_path_update_refs(wt_dir);
> +
> +	fp = fopen(path, "r");
> +	if (!fp)
> +		goto cleanup;
> +
> +	while (strbuf_getline(&ref, fp) != EOF) {
> +		struct string_list_item *item;
> +
> +		CALLOC_ARRAY(rec, 1);

It does not matter but later in the series we use rec = 
xmalloc(sizeof(*rec)) to do this. I'm not sure the project has a 
standard way of allocating a single instance of something.

> +
> +		if (strbuf_getline(&hash, fp) == EOF ||
> +		    get_oid_hex(hash.buf, &rec->before)) {
> +			warning(_("update-refs file at '%s' is invalid"),
> +				  path);
> +			result = -1;
> +			goto cleanup;
> +		}
> +
> +		if (strbuf_getline(&hash, fp) == EOF ||
> +		    get_oid_hex(hash.buf, &rec->after)) {
> +			warning(_("update-refs file at '%s' is invalid"),
> +				  path);
> +			result = -1;
> +			goto cleanup;
> +		}
> +
> +		item = string_list_insert(refs, ref.buf);
> +		item->util = rec;
> +		rec = NULL;
> +	}
> +
> +cleanup:
> +	if (fp)
> +		fclose(fp);
> +	free(path);
> +	free(rec);
> +	strbuf_release(&ref);
> +	strbuf_release(&hash);
> +	return result;
> +}
> diff --git a/sequencer.h b/sequencer.h
> index da64473636b..3ae541bb145 100644
> --- a/sequencer.h
> +++ b/sequencer.h
> @@ -232,4 +232,13 @@ void sequencer_post_commit_cleanup(struct repository *r, int verbose);
>   int sequencer_get_last_command(struct repository* r,
>   			       enum replay_action *action);
>   int sequencer_determine_whence(struct repository *r, enum commit_whence *whence);
> +
> +/**
> + * Append the set of ref-OID pairs that are currently stored for the 'git
> + * rebase --update-refs' feature if such a rebase is currently happening.
> + *
> + * Localized to a worktree's git dir.
> + */
> +int sequencer_get_update_refs_state(const char *wt_dir, struct string_list *refs);
> +
>   #endif /* SEQUENCER_H */
> diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
> index a67ce5fb003..97f5c87f8c8 100755
> --- a/t/t2407-worktree-heads.sh
> +++ b/t/t2407-worktree-heads.sh
> @@ -81,6 +81,29 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
>   	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
>   '
>   
> +test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
> +	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
> +
> +	mkdir -p .git/worktrees/wt-3/rebase-merge &&
> +	touch .git/worktrees/wt-3/rebase-merge/interactive &&
> +
> +	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
> +	refs/heads/fake-3
> +	$(git rev-parse HEAD~1)
> +	$(git rev-parse HEAD)
> +	refs/heads/fake-4
> +	$(git rev-parse HEAD)
> +	$(git rev-parse HEAD)
> +	EOF
> +
> +	for i in 3 4
> +	do
> +		test_must_fail git branch -f fake-$i HEAD 2>err &&
> +		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
> +			return 1
> +	done
> +'
> +
>   test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
>   	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
>   	grep "refusing to fetch into branch '\''refs/heads/wt-3'\''" err &&

[PATCH v4 04/12] rebase-interactive: update 'merge' description

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The 'merge' command description for the todo list documentation in an
interactive rebase has multiple lines. The lines other than the first
one start with dots ('.') while the similar multi-line documentation for
'fixup' does not. This description only appears in the comment text of
the todo file during an interactive rebase.

The 'merge' command was documented when interactive rebase was first
ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
2018-08-10). These dots might have been carried over from the previous
shell implementation.

The 'fixup' command was documented more recently in 9e3cebd97cb (rebase
-i: add fixup [-C | -c] command, 2021-01-29).

Looking at the output in an editor, my personal opinion is that the dots
are unnecessary and noisy. Remove them now before adding more commands
with multi-line documentation.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 87649d0c016..22394224faa 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -54,9 +54,9 @@ void append_todo_help(int command_count,
 "l, label <label> = label current HEAD with a name\n"
 "t, reset <label> = reset HEAD to a label\n"
 "m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n"
-".       create a merge commit using the original merge commit's\n"
-".       message (or the oneline, if no original merge commit was\n"
-".       specified); use -c <commit> to reword the commit message\n"
+"        create a merge commit using the original merge commit's\n"
+"        message (or the oneline, if no original merge commit was\n"
+"        specified); use -c <commit> to reword the commit message\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
-- 
gitgitgadget

[PATCH v4 05/12] sequencer: define array with enum values

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The todo_command_info array defines which strings match with which
todo_command enum values. The array is defined in the same order as the
enum values, but if one changed without the other, then we would have
unexpected results.

Make it easier to see changes to the enum and this array by using the
enum values as the indices of the array.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index e93c61cbd25..575f9be5ea0 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1708,20 +1708,20 @@ static struct {
 	char c;
 	const char *str;
 } todo_command_info[] = {
-	{ 'p', "pick" },
-	{ 0,   "revert" },
-	{ 'e', "edit" },
-	{ 'r', "reword" },
-	{ 'f', "fixup" },
-	{ 's', "squash" },
-	{ 'x', "exec" },
-	{ 'b', "break" },
-	{ 'l', "label" },
-	{ 't', "reset" },
-	{ 'm', "merge" },
-	{ 0,   "noop" },
-	{ 'd', "drop" },
-	{ 0,   NULL }
+	[TODO_PICK] = { 'p', "pick" },
+	[TODO_REVERT] = { 0,   "revert" },
+	[TODO_EDIT] = { 'e', "edit" },
+	[TODO_REWORD] = { 'r', "reword" },
+	[TODO_FIXUP] = { 'f', "fixup" },
+	[TODO_SQUASH] = { 's', "squash" },
+	[TODO_EXEC] = { 'x', "exec" },
+	[TODO_BREAK] = { 'b', "break" },
+	[TODO_LABEL] = { 'l', "label" },
+	[TODO_RESET] = { 't', "reset" },
+	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_NOOP] = { 0,   "noop" },
+	[TODO_DROP] = { 'd', "drop" },
+	[TODO_COMMENT] = { 0,   NULL },
 };
 
 static const char *command_to_string(const enum todo_command command)
-- 
gitgitgadget

[PATCH v4 06/12] sequencer: add update-ref command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

Add the boilerplate for an "update-ref" command in the sequencer. This
connects to the current no-op do_update_ref() which will be filled in
after more connections are created.

The syntax in the todo list will be "update-ref <ref-name>" to signal
that we should store the current commit as the value for updating
<ref-name> at the end of the rebase.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c |  3 +++
 sequencer.c          | 14 +++++++++++++-
 sequencer.h          |  1 +
 3 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 22394224faa..1ff07647af3 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -57,6 +57,9 @@ void append_todo_help(int command_count,
 "        create a merge commit using the original merge commit's\n"
 "        message (or the oneline, if no original merge commit was\n"
 "        specified); use -c <commit> to reword the commit message\n"
+"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
+"                      to this position in the new commits. The <ref> is\n"
+"                      updated at the end of the rebase\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
diff --git a/sequencer.c b/sequencer.c
index 575f9be5ea0..1d6442c9639 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1719,6 +1719,7 @@ static struct {
 	[TODO_LABEL] = { 'l', "label" },
 	[TODO_RESET] = { 't', "reset" },
 	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_UPDATE_REF] = { 'u', "update-ref" },
 	[TODO_NOOP] = { 0,   "noop" },
 	[TODO_DROP] = { 'd', "drop" },
 	[TODO_COMMENT] = { 0,   NULL },
@@ -2480,7 +2481,7 @@ static int parse_insn_line(struct repository *r, struct todo_item *item,
 			     command_to_string(item->command));
 
 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
-	    item->command == TODO_RESET) {
+	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
 		item->commit = NULL;
 		item->arg_offset = bol - buf;
 		item->arg_len = (int)(eol - bol);
@@ -4079,6 +4080,11 @@ leave_merge:
 	return ret;
 }
 
+static int do_update_ref(struct repository *r, const char *ref_name)
+{
+	return 0;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4454,6 +4460,12 @@ static int pick_commits(struct repository *r,
 				return error_with_patch(r, item->commit,
 							arg, item->arg_len,
 							opts, res, 0);
+		} else if (item->command == TODO_UPDATE_REF) {
+			struct strbuf ref = STRBUF_INIT;
+			strbuf_add(&ref, arg, item->arg_len);
+			if ((res = do_update_ref(r, ref.buf)))
+				reschedule = 1;
+			strbuf_release(&ref);
 		} else if (!is_noop(item->command))
 			return error(_("unknown command %d"), item->command);
 
diff --git a/sequencer.h b/sequencer.h
index 3ae541bb145..2cf5c1b9a38 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -95,6 +95,7 @@ enum todo_command {
 	TODO_LABEL,
 	TODO_RESET,
 	TODO_MERGE,
+	TODO_UPDATE_REF,
 	/* commands that do nothing but are counted for reporting progress */
 	TODO_NOOP,
 	TODO_DROP,
-- 
gitgitgadget

[PATCH v4 07/12] rebase: add --update-refs option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When working on a large feature, it can be helpful to break that feature
into multiple smaller parts that become reviewed in sequence. During
development or during review, a change to one part of the feature could
affect multiple of these parts. An interactive rebase can help adjust
the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the
feature, then rebasing the entire list of commits can create commits not
reachable from those "sub branches". It can take a manual step to update
those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
<ref>' steps to the todo file whenever a commit that is being rebased is
decorated with that <ref>. At the very end, the rebase process updates
all of the listed refs to the values stored during the rebase operation.

Be sure to iterate after any squashing or fixups are placed. Update the
branch only after those squashes and fixups are complete. This allows a
--fixup commit at the tip of the feature to apply correctly to the sub
branch, even if it is fixing up the most-recent commit in that part.

One potential problem here is that refs decorating commits that are
already marked as "fixup!" or "squash!" will not be included in this
list. Generally, the reordering of the "fixup!" and "squash!" is likely
to change the relative order of these refs, so it is not recommended.
The workflow here is intended to allow these kinds of commits at the tip
of the rebased branch while the other sub branches come along for the
ride without intervention.

This change update the documentation and builtin to accept the
--update-refs option as well as updating the todo file with the
'update-ref' commands. Tests are added to ensure that these todo
commands are added in the correct locations.

This change does _not_ include the actual behavior of tracking the
updated refs and writing the new ref values at the end of the rebase
process. That is deferred to a later change.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-rebase.txt  |   8 +++
 builtin/rebase.c              |   5 ++
 sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
 sequencer.h                   |   1 +
 t/t2407-worktree-heads.sh     |  22 +++++++
 t/t3404-rebase-interactive.sh |  70 ++++++++++++++++++++++
 6 files changed, 213 insertions(+)

diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 262fb01aec0..e7611b4089c 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	or point to a `squash! ...` or `fixup! ...` commit are not updated
+	in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
@@ -632,6 +639,7 @@ are incompatible with the following options:
  * --empty=
  * --reapply-cherry-picks
  * --edit-todo
+ * --update-refs
  * --root when used in combination with --onto
 
 In addition, the following pairs of options are incompatible:
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 7ab50cda2ad..56d82a52106 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update local refs that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
diff --git a/sequencer.c b/sequencer.c
index 1d6442c9639..e657862cda2 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -35,6 +35,7 @@
 #include "commit-reach.h"
 #include "rebase-interactive.h"
 #include "reset.h"
+#include "branch.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -5638,10 +5639,113 @@ static int skip_unnecessary_picks(struct repository *r,
 	return 0;
 }
 
+struct todo_add_branch_context {
+	struct todo_item *items;
+	size_t items_nr;
+	size_t items_alloc;
+	struct strbuf *buf;
+	struct commit *commit;
+	struct string_list refs_to_oids;
+};
+
+static int add_decorations_to_list(const struct commit *commit,
+				   struct todo_add_branch_context *ctx)
+{
+	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+
+	while (decoration) {
+		struct todo_item *item;
+		const char *path;
+		size_t base_offset = ctx->buf->len;
+
+		ALLOC_GROW(ctx->items,
+			ctx->items_nr + 1,
+			ctx->items_alloc);
+		item = &ctx->items[ctx->items_nr];
+		memset(item, 0, sizeof(*item));
+
+		/* If the branch is checked out, then leave a comment instead. */
+		if ((path = branch_checked_out(decoration->name))) {
+			item->command = TODO_COMMENT;
+			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
+				    decoration->name, path);
+		} else {
+			struct string_list_item *sti;
+			item->command = TODO_UPDATE_REF;
+			strbuf_addf(ctx->buf, "%s\n", decoration->name);
+
+			sti = string_list_insert(&ctx->refs_to_oids,
+						 decoration->name);
+			sti->util = oiddup(the_hash_algo->null_oid);
+		}
+
+		item->offset_in_buf = base_offset;
+		item->arg_offset = base_offset;
+		item->arg_len = ctx->buf->len - base_offset;
+		ctx->items_nr++;
+
+		decoration = decoration->next;
+	}
+
+	return 0;
+}
+
+/*
+ * For each 'pick' command, find out if the commit has a decoration in
+ * refs/heads/. If so, then add a 'label for-update-refs/' command.
+ */
+static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
+{
+	int i;
+	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
+	struct decoration_filter decoration_filter = {
+		.include_ref_pattern = &decorate_refs_include,
+		.exclude_ref_pattern = &decorate_refs_exclude,
+		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
+	};
+	struct todo_add_branch_context ctx = {
+		.buf = &todo_list->buf,
+		.refs_to_oids = STRING_LIST_INIT_DUP,
+	};
+
+	ctx.items_alloc = 2 * todo_list->nr + 1;
+	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
+
+	string_list_append(&decorate_refs_include, "refs/heads/");
+	load_ref_decorations(&decoration_filter, 0);
+
+	for (i = 0; i < todo_list->nr; ) {
+		struct todo_item *item = &todo_list->items[i];
+
+		/* insert ith item into new list */
+		ALLOC_GROW(ctx.items,
+			   ctx.items_nr + 1,
+			   ctx.items_alloc);
+
+		ctx.items[ctx.items_nr++] = todo_list->items[i++];
+
+		if (item->commit) {
+			ctx.commit = item->commit;
+			add_decorations_to_list(item->commit, &ctx);
+		}
+	}
+
+	string_list_clear(&ctx.refs_to_oids, 1);
+	free(todo_list->items);
+	todo_list->items = ctx.items;
+	todo_list->nr = ctx.items_nr;
+	todo_list->alloc = ctx.items_alloc;
+
+	return 0;
+}
+
 int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list)
 {
 	char shortonto[GIT_MAX_HEXSZ + 1];
@@ -5660,6 +5764,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
 	}
 
+	if (update_refs && todo_list_add_update_ref_commands(todo_list))
+		return -1;
+
 	if (autosquash && todo_list_rearrange_squash(todo_list))
 		return -1;
 
diff --git a/sequencer.h b/sequencer.h
index 2cf5c1b9a38..e671d7e0743 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 97f5c87f8c8..8a03f14df8d 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -164,4 +164,26 @@ test_expect_success 'refuse to overwrite when in error states' '
 	done
 '
 
+. "$TEST_DIRECTORY"/lib-rebase.sh
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
+	git commit --fixup HEAD~2 --allow-empty &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		! grep "update-refs" todo
+	) &&
+	git branch -f allow-update HEAD~2 &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		grep "update-ref refs/heads/allow-update" todo
+	)
+'
+
+# This must be the last test in this file
+test_expect_success '$EDITOR and friends are unchanged' '
+	test_editor_unchanged
+'
+
 test_done
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index f31afd4a547..3cd20733bc8 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1743,6 +1743,76 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
 	test_cmp_rev ORIG_HEAD test-orig-head@{1}
 '
 
+test_expect_success '--update-refs adds label and update-ref commands' '
+	git checkout -b update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git commit --allow-empty --fixup=third &&
+	git branch -f shared-tip &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		pick $(git log -1 --format=%h J) J
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h K) K
+		pick $(git log -1 --format=%h L) L
+		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h M) M
+		update-ref refs/heads/no-conflict-branch
+		update-ref refs/heads/shared-tip
+		EOF
+
+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo
+	)
+'
+
+test_expect_success '--update-refs adds commands with --rebase-merges' '
+	git checkout -b update-refs-with-merge no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git merge -m merge branch2 &&
+	git branch -f merge-branch &&
+	git commit --fixup=third --allow-empty &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		label onto
+		reset onto
+		pick $(git log -1 --format=%h branch2~1) F
+		pick $(git log -1 --format=%h branch2) I
+		update-ref refs/heads/branch2
+		label merge
+		reset onto
+		pick $(git log -1 --format=%h refs/heads/second) J
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h refs/heads/third~1) K
+		pick $(git log -1 --format=%h refs/heads/third) L
+		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h HEAD~2) M
+		update-ref refs/heads/no-conflict-branch
+		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
+		update-ref refs/heads/merge-branch
+		EOF
+
+		test_must_fail git rebase -i --autosquash \
+				   --rebase-merges=rebase-cousins \
+				   --update-refs primary >todo &&
+
+		test_cmp expect todo
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Elijah Newren]

On Tue, Jul 12, 2022 at 6:07 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <derrickstolee@github.com>
>
[...]
>
> +--update-refs::
> +--no-update-refs::
> +       Automatically force-update any branches that point to commits that
> +       are being rebased. Any branches that are checked out in a worktree
> +       or point to a `squash! ...` or `fixup! ...` commit are not updated
> +       in this way.

I think the second sentence here should be split.  In particular, I
don't think I understand the second half of the second sentence.  Do
you intend to say here that branches pointing to a `squash!` or
`fixup!` will instead update the first `pick` in the ancestry of such
a commit, rather than that such branches are entirely excluded from
any updates?  That's what I observed in my testing of your v3, at
least, and that's the behavior I'd expect this feature to implement,
but this documentation doesn't match.

[...]
> @@ -5660,6 +5764,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
>                 item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
>         }
>
> +       if (update_refs && todo_list_add_update_ref_commands(todo_list))
> +               return -1;
> +

As a tangent, I find it interesting that you add the update-ref
commands as a post-processing step rather than as a part of
sequencer_make_script().  I don't think you need to change anything,
but I am curious due to my git-replay work if you find it advantageous
to do it this way.

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Derrick Stolee]

On 7/16/2022 3:30 PM, Elijah Newren wrote:
> On Tue, Jul 12, 2022 at 6:07 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <derrickstolee@github.com>
>>
> [...]
>>
>> +--update-refs::
>> +--no-update-refs::
>> +       Automatically force-update any branches that point to commits that
>> +       are being rebased. Any branches that are checked out in a worktree
>> +       or point to a `squash! ...` or `fixup! ...` commit are not updated
>> +       in this way.
> 
> I think the second sentence here should be split.  In particular, I
> don't think I understand the second half of the second sentence.  Do
> you intend to say here that branches pointing to a `squash!` or
> `fixup!` will instead update the first `pick` in the ancestry of such
> a commit, rather than that such branches are entirely excluded from
> any updates?  That's what I observed in my testing of your v3, at
> least, and that's the behavior I'd expect this feature to implement,
> but this documentation doesn't match.

Good find. You're right that these don't match, and its in fact that
I expected it to work this way, but it doesn't.

I've added a branch to my tests that points to a fixup! that is not
the tip commit and use it to demonstrate that it is not reordered, but
_does_ appear in the 'update-ref <ref>' list.

I'll update the documentation to match this behavior, too. This case
is unlikely to happen much in practice, but I now believe it's better
to include the ref than to ignore it.
 
> [...]
>> @@ -5660,6 +5764,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
>>                 item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
>>         }
>>
>> +       if (update_refs && todo_list_add_update_ref_commands(todo_list))
>> +               return -1;
>> +
> 
> As a tangent, I find it interesting that you add the update-ref
> commands as a post-processing step rather than as a part of
> sequencer_make_script().  I don't think you need to change anything,
> but I am curious due to my git-replay work if you find it advantageous
> to do it this way.

I found it to be simple enough to do a scan of the steps directly
instead of injecting extra logic into the _make_script() method.
The simplest reason is that we would need to inject "update_refs"
into that method.

Thanks,
-Stolee

Re: [PATCH v4 07/12] rebase: add --update-refs option

[SZEDER Gábor]

On Tue, Jul 12, 2022 at 01:07:00PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> When working on a large feature, it can be helpful to break that feature
> into multiple smaller parts that become reviewed in sequence. During
> development or during review, a change to one part of the feature could
> affect multiple of these parts. An interactive rebase can help adjust
> the multi-part "story" of the branch.
> 
> However, if there are branches tracking the different parts of the
> feature, then rebasing the entire list of commits can create commits not
> reachable from those "sub branches". It can take a manual step to update
> those branches.
> 
> Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
> <ref>' steps to the todo file whenever a commit that is being rebased is
> decorated with that <ref>. At the very end, the rebase process updates
> all of the listed refs to the values stored during the rebase operation.
> 
> Be sure to iterate after any squashing or fixups are placed. Update the
> branch only after those squashes and fixups are complete. This allows a
> --fixup commit at the tip of the feature to apply correctly to the sub
> branch, even if it is fixing up the most-recent commit in that part.
> 
> One potential problem here is that refs decorating commits that are
> already marked as "fixup!" or "squash!" will not be included in this
> list. Generally, the reordering of the "fixup!" and "squash!" is likely
> to change the relative order of these refs, so it is not recommended.
> The workflow here is intended to allow these kinds of commits at the tip
> of the rebased branch while the other sub branches come along for the
> ride without intervention.
> 
> This change update the documentation and builtin to accept the
> --update-refs option as well as updating the todo file with the
> 'update-ref' commands. Tests are added to ensure that these todo
> commands are added in the correct locations.
> 
> This change does _not_ include the actual behavior of tracking the
> updated refs and writing the new ref values at the end of the rebase
> process. That is deferred to a later change.
> 
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>  Documentation/git-rebase.txt  |   8 +++
>  builtin/rebase.c              |   5 ++
>  sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
>  sequencer.h                   |   1 +
>  t/t2407-worktree-heads.sh     |  22 +++++++
>  t/t3404-rebase-interactive.sh |  70 ++++++++++++++++++++++
>  6 files changed, 213 insertions(+)
> 
> diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
> index 262fb01aec0..e7611b4089c 100644
> --- a/Documentation/git-rebase.txt
> +++ b/Documentation/git-rebase.txt
> @@ -609,6 +609,13 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
>  start would be overridden by the presence of
>  `rebase.rescheduleFailedExec=true` configuration.
>  
> +--update-refs::

So the option is called '--update-refs', but ...

> +--no-update-refs::
> +	Automatically force-update any branches that point to commits that

... its description talks about "branches".

> +	are being rebased. Any branches that are checked out in a worktree
> +	or point to a `squash! ...` or `fixup! ...` commit are not updated
> +	in this way.
> +
>  INCOMPATIBLE OPTIONS
>  --------------------
>  

> @@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
>  		OPT_BOOL(0, "autosquash", &options.autosquash,
>  			 N_("move commits that begin with "
>  			    "squash!/fixup! under -i")),
> +		OPT_BOOL(0, "update-refs", &options.update_refs,
> +			 N_("update local refs that point to commits "

And its short help talks about "local refs".

I think at least the documentation and short help should use
consistent terminology.

> +			    "that are being rebased")),
>  		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
>  			N_("GPG-sign commits"),
>  			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Derrick Stolee]

On 7/18/2022 5:05 AM, SZEDER Gábor wrote:
> On Tue, Jul 12, 2022 at 01:07:00PM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>

>> +--update-refs::
> 
> So the option is called '--update-refs', but ...
> 
>> +--no-update-refs::
>> +	Automatically force-update any branches that point to commits that
> 
> ... its description talks about "branches".

>> +		OPT_BOOL(0, "update-refs", &options.update_refs,
>> +			 N_("update local refs that point to commits "
> 
> And its short help talks about "local refs".
> 
> I think at least the documentation and short help should use
> consistent terminology.

Thanks for catching this. I think I should use "branches" here, but
keep the name "--update-refs". The biggest reason is that it provides
a nice parallel with the "update-ref" sequencer command. This command
allows updating _any_ ref, such as lightweight tags in refs/tags/*
or even refs in refs/my/namespace/*.

The --update-refs option doesn't create the commands to update tags
or refs in places other than refs/heads/*.

Thanks,
-Stolee

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> ... I think I should use "branches" here, but
> keep the name "--update-refs". The biggest reason is that it provides
> a nice parallel with the "update-ref" sequencer command. This command
> allows updating _any_ ref, such as lightweight tags in refs/tags/*
> or even refs in refs/my/namespace/*.
>
> The --update-refs option doesn't create the commands to update tags
> or refs in places other than refs/heads/*.

I guess it would make the choice of "branch" the most appropriate.

I was hoping that we can repoint refs in private namespaces that are
not branches with the option.  But as long as the underlying
"update-ref" instruction can be used by advanced users, that is OK.

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Derrick Stolee]

On 7/18/2022 3:35 PM, Junio C Hamano wrote:
> Derrick Stolee <derrickstolee@github.com> writes:
> 
>> ... I think I should use "branches" here, but
>> keep the name "--update-refs". The biggest reason is that it provides
>> a nice parallel with the "update-ref" sequencer command. This command
>> allows updating _any_ ref, such as lightweight tags in refs/tags/*
>> or even refs in refs/my/namespace/*.
>>
>> The --update-refs option doesn't create the commands to update tags
>> or refs in places other than refs/heads/*.
> 
> I guess it would make the choice of "branch" the most appropriate.
> 
> I was hoping that we can repoint refs in private namespaces that are
> not branches with the option.  But as long as the underlying
> "update-ref" instruction can be used by advanced users, that is OK.

I would like to keep the --update-refs name for a couple reasons:

1. 'update-ref' is the right name for the sequencer command. Having
   a parallel there is helpful for learning about the option.

2. We could extend the boolean '--update-refs' option into a more
   advanced multi-valued '--update-refs=<refspec>' option to allow
   advanced users to specify a ref namespace that they want included.

Thanks,
-Stolee

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

> On 7/18/2022 3:35 PM, Junio C Hamano wrote:
>> Derrick Stolee <derrickstolee@github.com> writes:
>> 
>>> ... I think I should use "branches" here, but
>>> keep the name "--update-refs". The biggest reason is that it provides
>>> a nice parallel with the "update-ref" sequencer command. This command
>>> allows updating _any_ ref, such as lightweight tags in refs/tags/*
>>> or even refs in refs/my/namespace/*.
>>>
>>> The --update-refs option doesn't create the commands to update tags
>>> or refs in places other than refs/heads/*.
>> 
>> I guess it would make the choice of "branch" the most appropriate.
>> 
>> I was hoping that we can repoint refs in private namespaces that are
>> not branches with the option.  But as long as the underlying
>> "update-ref" instruction can be used by advanced users, that is OK.
>
> I would like to keep the --update-refs name for a couple reasons:

I do not think anybody proposed to change the name of that option.

I was reacting to your "I should use branches here", with the
understanding that "here" is this place where you used "local refs".

>> +		OPT_BOOL(0, "update-refs", &options.update_refs,
>> +			 N_("update local refs that point to commits "

If "rebase --update-refs" uses "update-ref" insn (which is capable
of repointing non-branch refs) only for local branches, then the
help text for the "--update-refs" option can safely say "update
local branches" without being inaccurate.  That is where my "branch
is the most appropriate" comes from.

Re: [PATCH v4 07/12] rebase: add --update-refs option

[Derrick Stolee]

On 7/19/2022 12:44 PM, Junio C Hamano wrote:
> Derrick Stolee <derrickstolee@github.com> writes:
> 
>> On 7/18/2022 3:35 PM, Junio C Hamano wrote:
>>> Derrick Stolee <derrickstolee@github.com> writes:
>>>
>>>> ... I think I should use "branches" here, but
>>>> keep the name "--update-refs". The biggest reason is that it provides
>>>> a nice parallel with the "update-ref" sequencer command. This command
>>>> allows updating _any_ ref, such as lightweight tags in refs/tags/*
>>>> or even refs in refs/my/namespace/*.
>>>>
>>>> The --update-refs option doesn't create the commands to update tags
>>>> or refs in places other than refs/heads/*.
>>>
>>> I guess it would make the choice of "branch" the most appropriate.
>>>
>>> I was hoping that we can repoint refs in private namespaces that are
>>> not branches with the option.  But as long as the underlying
>>> "update-ref" instruction can be used by advanced users, that is OK.
>>
>> I would like to keep the --update-refs name for a couple reasons:
> 
> I do not think anybody proposed to change the name of that option.
> 
> I was reacting to your "I should use branches here", with the
> understanding that "here" is this place where you used "local refs".
> 
>>> +		OPT_BOOL(0, "update-refs", &options.update_refs,
>>> +			 N_("update local refs that point to commits "
> 
> If "rebase --update-refs" uses "update-ref" insn (which is capable
> of repointing non-branch refs) only for local branches, then the
> help text for the "--update-refs" option can safely say "update
> local branches" without being inaccurate.  That is where my "branch
> is the most appropriate" comes from.

Thanks for the clarification. Sorry for misunderstanding.

Thanks,
-Stolee

[PATCH v4 08/12] rebase: update refs from 'update-ref' commands

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change introduced the 'git rebase --update-refs' option
which added 'update-ref <ref>' commands to the todo list of an
interactive rebase.

Teach Git to record the HEAD position when reaching these 'update-ref'
commands. The ref/before/after triple is stored in the
$GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
file to avoid having other processes updating the refs in that file
while the rebase is in progress.

Not only do we update the file when the sequencer reaches these
'update-ref' commands, we then update the refs themselves at the end of
the rebase sequence. If the rebase is aborted before this final step,
then the refs are not updated. The 'before' value is used to ensure that
we do not accidentally obliterate a ref that was updated concurrently
(say, by an older version of Git or a third-party tool).

Now that the 'git rebase --update-refs' command is implemented to write
to the update-refs file, we can remove the fake construction of the
update-refs file from a test in t2407-worktree-heads.sh.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 113 +++++++++++++++++++++++++++++++++-
 t/t2407-worktree-heads.sh     |  21 ++-----
 t/t3404-rebase-interactive.sh |  17 +++++
 3 files changed, 134 insertions(+), 17 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index e657862cda2..2d89b3b727a 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -36,6 +36,7 @@
 #include "rebase-interactive.h"
 #include "reset.h"
 #include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -193,6 +194,19 @@ struct update_ref_record {
 	struct object_id after;
 };
 
+static struct update_ref_record *init_update_ref_record(const char *ref)
+{
+	struct update_ref_record *rec = xmalloc(sizeof(*rec));
+
+	oidcpy(&rec->before, null_oid());
+	oidcpy(&rec->after, null_oid());
+
+	/* This may fail, but that's fine, we will keep the null OID. */
+	read_ref(ref, &rec->before);
+
+	return rec;
+}
+
 static int git_sequencer_config(const char *k, const char *v, void *cb)
 {
 	struct replay_opts *opts = cb;
@@ -4081,11 +4095,102 @@ leave_merge:
 	return ret;
 }
 
-static int do_update_ref(struct repository *r, const char *ref_name)
+static int write_update_refs_state(struct string_list *refs_to_oids)
 {
+	int result = 0;
+	struct lock_file lock = LOCK_INIT;
+	FILE *fp = NULL;
+	struct string_list_item *item;
+	char *path;
+
+	if (!refs_to_oids->nr)
+		return 0;
+
+	path = rebase_path_update_refs(the_repository->gitdir);
+
+	if (safe_create_leading_directories(path)) {
+		result = error(_("unable to create leading directories of %s"),
+			       path);
+		goto cleanup;
+	}
+
+	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
+		result = error(_("another 'rebase' process appears to be running; "
+				 "'%s.lock' already exists"),
+			       path);
+		goto cleanup;
+	}
+
+	fp = fdopen_lock_file(&lock, "w");
+	if (!fp) {
+		result = error_errno(_("could not open '%s' for writing"), path);
+		rollback_lock_file(&lock);
+		goto cleanup;
+	}
+
+	for_each_string_list_item(item, refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+		fprintf(fp, "%s\n%s\n%s\n", item->string,
+			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
+	}
+
+	result = commit_lock_file(&lock);
+
+cleanup:
+	free(path);
+	return result;
+}
+
+static int do_update_ref(struct repository *r, const char *refname)
+{
+	struct string_list_item *item;
+	struct string_list list = STRING_LIST_INIT_DUP;
+
+	sequencer_get_update_refs_state(r->gitdir, &list);
+
+	for_each_string_list_item(item, &list) {
+		if (!strcmp(item->string, refname)) {
+			struct update_ref_record *rec = item->util;
+			read_ref("HEAD", &rec->after);
+			break;
+		}
+	}
+
+	write_update_refs_state(&list);
+	string_list_clear(&list, 1);
 	return 0;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	int res = 0;
+	struct string_list_item *item;
+	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
+	struct ref_store *refs = get_main_ref_store(r);
+
+	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
+
+	for_each_string_list_item(item, &refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+
+		if (oideq(&rec->after, the_hash_algo->null_oid)) {
+			/*
+			 * Ref was not updated. User may have deleted the
+			 * 'update-ref' step.
+			 */
+			continue;
+		}
+
+		res |= refs_update_ref(refs, "rewritten during rebase",
+				       item->string,
+				       &rec->after, &rec->before,
+				       0, UPDATE_REFS_MSG_ON_ERR);
+	}
+
+	string_list_clear(&refs_to_oids, 1);
+	return res;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4603,6 +4708,8 @@ cleanup_head_ref:
 		strbuf_release(&head_ref);
 	}
 
+	do_update_refs(r);
+
 	/*
 	 * Sequence of picks finished successfully; cleanup by
 	 * removing the .git/sequencer directory
@@ -5676,7 +5783,7 @@ static int add_decorations_to_list(const struct commit *commit,
 
 			sti = string_list_insert(&ctx->refs_to_oids,
 						 decoration->name);
-			sti->util = oiddup(the_hash_algo->null_oid);
+			sti->util = init_update_ref_record(decoration->name);
 		}
 
 		item->offset_in_buf = base_offset;
@@ -5732,6 +5839,8 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 		}
 	}
 
+	write_update_refs_state(&ctx.refs_to_oids);
+
 	string_list_clear(&ctx.refs_to_oids, 1);
 	free(todo_list->items);
 	todo_list->items = ctx.items;
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 8a03f14df8d..50815acd3e8 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -81,25 +81,16 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
-	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
-
-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
+	test_when_finished git -C wt-3 rebase --abort &&
 
-	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
-	refs/heads/fake-3
-	$(git rev-parse HEAD~1)
-	$(git rev-parse HEAD)
-	refs/heads/fake-4
-	$(git rev-parse HEAD)
-	$(git rev-parse HEAD)
-	EOF
+	git branch -f can-be-updated wt-3 &&
+	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
 
 	for i in 3 4
 	do
-		test_must_fail git branch -f fake-$i HEAD 2>err &&
-		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+		test_must_fail git branch -f can-be-updated HEAD 2>err &&
+		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
 			return 1
 	done
 '
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 3cd20733bc8..a37820fa728 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1813,6 +1813,23 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 	)
 '
 
+test_expect_success '--update-refs updates refs correctly' '
+	git checkout -B update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	test_commit extra2 fileX &&
+	git commit --amend --fixup=L &&
+
+	git rebase -i --autosquash --update-refs primary &&
+
+	test_cmp_rev HEAD~3 refs/heads/first &&
+	test_cmp_rev HEAD~3 refs/heads/second &&
+	test_cmp_rev HEAD~1 refs/heads/third &&
+	test_cmp_rev HEAD refs/heads/no-conflict-branch
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v4 08/12] rebase: update refs from 'update-ref' commands

[Phillip Wood]

Hi Stolee

On 12/07/2022 14:07, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> The previous change introduced the 'git rebase --update-refs' option
> which added 'update-ref <ref>' commands to the todo list of an
> interactive rebase.
> 
> Teach Git to record the HEAD position when reaching these 'update-ref'
> commands. The ref/before/after triple is stored in the
> $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
> file to avoid having other processes updating the refs in that file
> while the rebase is in progress.
> 
> Not only do we update the file when the sequencer reaches these
> 'update-ref' commands, we then update the refs themselves at the end of
> the rebase sequence. If the rebase is aborted before this final step,
> then the refs are not updated. The 'before' value is used to ensure that
> we do not accidentally obliterate a ref that was updated concurrently
> (say, by an older version of Git or a third-party tool).
> 
> Now that the 'git rebase --update-refs' command is implemented to write
> to the update-refs file, we can remove the fake construction of the
> update-refs file from a test in t2407-worktree-heads.sh.

This is looking good. I've left a few comments, mostly about error 
propagation. It's nice to see us recording the initial value of the ref 
when the todo list is created. It's also good to see this using a lock 
file. We could perhaps lock the file (with a timeout) when we read it in 
sequencer_get_update_refs_state() to avoid a race where a process is 
checking out a new branch in one worktree and another is preparing to 
rebase that branch in another worktree.

> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   sequencer.c                   | 113 +++++++++++++++++++++++++++++++++-
>   t/t2407-worktree-heads.sh     |  21 ++-----
>   t/t3404-rebase-interactive.sh |  17 +++++
>   3 files changed, 134 insertions(+), 17 deletions(-)
> 
> diff --git a/sequencer.c b/sequencer.c
> index e657862cda2..2d89b3b727a 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -36,6 +36,7 @@
>   #include "rebase-interactive.h"
>   #include "reset.h"
>   #include "branch.h"
> +#include "log-tree.h"
>   
>   #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>   
> @@ -193,6 +194,19 @@ struct update_ref_record {
>   	struct object_id after;
>   };
>   
> +static struct update_ref_record *init_update_ref_record(const char *ref)
> +{
> +	struct update_ref_record *rec = xmalloc(sizeof(*rec));
> +
> +	oidcpy(&rec->before, null_oid());
> +	oidcpy(&rec->after, null_oid());
> +
> +	/* This may fail, but that's fine, we will keep the null OID. */
> +	read_ref(ref, &rec->before);
> +
> +	return rec;
> +}
> +
>   static int git_sequencer_config(const char *k, const char *v, void *cb)
>   {
>   	struct replay_opts *opts = cb;
> @@ -4081,11 +4095,102 @@ leave_merge:
>   	return ret;
>   }
>   
> -static int do_update_ref(struct repository *r, const char *ref_name)
> +static int write_update_refs_state(struct string_list *refs_to_oids)
>   {
> +	int result = 0;
> +	struct lock_file lock = LOCK_INIT;
> +	FILE *fp = NULL;
> +	struct string_list_item *item;
> +	char *path;
> +
> +	if (!refs_to_oids->nr)
> +		return 0;
> +
> +	path = rebase_path_update_refs(the_repository->gitdir);
> +
> +	if (safe_create_leading_directories(path)) {
> +		result = error(_("unable to create leading directories of %s"),
> +			       path);
> +		goto cleanup;
> +	}
> +
> +	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
> +		result = error(_("another 'rebase' process appears to be running; "
> +				 "'%s.lock' already exists"),
> +			       path);
> +		goto cleanup;
> +	}
> +
> +	fp = fdopen_lock_file(&lock, "w");
> +	if (!fp) {
> +		result = error_errno(_("could not open '%s' for writing"), path);
> +		rollback_lock_file(&lock);
> +		goto cleanup;
> +	}
> +
> +	for_each_string_list_item(item, refs_to_oids) {
> +		struct update_ref_record *rec = item->util;
> +		fprintf(fp, "%s\n%s\n%s\n", item->string,
> +			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
> +	}
> +
> +	result = commit_lock_file(&lock);
> +
> +cleanup:
> +	free(path);
> +	return result;
> +}
> +
> +static int do_update_ref(struct repository *r, const char *refname)
> +{
> +	struct string_list_item *item;
> +	struct string_list list = STRING_LIST_INIT_DUP;
> +
> +	sequencer_get_update_refs_state(r->gitdir, &list);

We're ignoring any errors here and always returning 0 from this function.

> +
> +	for_each_string_list_item(item, &list) {
> +		if (!strcmp(item->string, refname)) {
> +			struct update_ref_record *rec = item->util;
> +			read_ref("HEAD", &rec->after);
> +			break;
> +		}
> +	}
> +
> +	write_update_refs_state(&list);
> +	string_list_clear(&list, 1);
>   	return 0;
>   }
>   
> +static int do_update_refs(struct repository *r)
> +{
> +	int res = 0;
> +	struct string_list_item *item;
> +	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
> +	struct ref_store *refs = get_main_ref_store(r);
> +
> +	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
> +
> +	for_each_string_list_item(item, &refs_to_oids) {
> +		struct update_ref_record *rec = item->util;
> +
> +		if (oideq(&rec->after, the_hash_algo->null_oid)) {
> +			/*
> +			 * Ref was not updated. User may have deleted the
> +			 * 'update-ref' step.
> +			 */

Unless we want to support users editing the todo list without using "git 
rebase --edit-todo" then by the end of the series it is a bug if we 
leave an entry in the update-refs file that has been removed from the 
todo list so I wander if we should remove this if().

> +			continue;
> +		}
> +
> +		res |= refs_update_ref(refs, "rewritten during rebase",
> +				       item->string,
> +				       &rec->after, &rec->before,
> +				       0, UPDATE_REFS_MSG_ON_ERR);
> +	}
> +
> +	string_list_clear(&refs_to_oids, 1);
> +	return res;
> +}
> +
>   static int is_final_fixup(struct todo_list *todo_list)
>   {
>   	int i = todo_list->current;
> @@ -4603,6 +4708,8 @@ cleanup_head_ref:
>   		strbuf_release(&head_ref);
>   	}
>   
> +	do_update_refs(r);

Should this be inside the "if (is_rebase_i(opts))" that is closed just 
above it? We're also ignoring the return value.

> +
>   	/*
>   	 * Sequence of picks finished successfully; cleanup by
>   	 * removing the .git/sequencer directory
> @@ -5676,7 +5783,7 @@ static int add_decorations_to_list(const struct commit *commit,
>   
>   			sti = string_list_insert(&ctx->refs_to_oids,
>   						 decoration->name);
> -			sti->util = oiddup(the_hash_algo->null_oid);
> +			sti->util = init_update_ref_record(decoration->name);
>   		}
>   
>   		item->offset_in_buf = base_offset;
> @@ -5732,6 +5839,8 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>   		}
>   	}
>   
> +	write_update_refs_state(&ctx.refs_to_oids);

We're ignoring the return value. Also I think 
todo_list_add_update_ref_commands() only ever returns 0.

Best Wishes

Phillip

> +
>   	string_list_clear(&ctx.refs_to_oids, 1);
>   	free(todo_list->items);
>   	todo_list->items = ctx.items;
> diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
> index 8a03f14df8d..50815acd3e8 100755
> --- a/t/t2407-worktree-heads.sh
> +++ b/t/t2407-worktree-heads.sh
> @@ -81,25 +81,16 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
>   	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
>   '
>   
> -test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
> -	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
> -
> -	mkdir -p .git/worktrees/wt-3/rebase-merge &&
> -	touch .git/worktrees/wt-3/rebase-merge/interactive &&
> +test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
> +	test_when_finished git -C wt-3 rebase --abort &&
>   
> -	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
> -	refs/heads/fake-3
> -	$(git rev-parse HEAD~1)
> -	$(git rev-parse HEAD)
> -	refs/heads/fake-4
> -	$(git rev-parse HEAD)
> -	$(git rev-parse HEAD)
> -	EOF
> +	git branch -f can-be-updated wt-3 &&
> +	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
>   
>   	for i in 3 4
>   	do
> -		test_must_fail git branch -f fake-$i HEAD 2>err &&
> -		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
> +		test_must_fail git branch -f can-be-updated HEAD 2>err &&
> +		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
>   			return 1
>   	done
>   '
> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index 3cd20733bc8..a37820fa728 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1813,6 +1813,23 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
>   	)
>   '
>   
> +test_expect_success '--update-refs updates refs correctly' '
> +	git checkout -B update-refs no-conflict-branch &&
> +	git branch -f base HEAD~4 &&
> +	git branch -f first HEAD~3 &&
> +	git branch -f second HEAD~3 &&
> +	git branch -f third HEAD~1 &&
> +	test_commit extra2 fileX &&
> +	git commit --amend --fixup=L &&
> +
> +	git rebase -i --autosquash --update-refs primary &&
> +
> +	test_cmp_rev HEAD~3 refs/heads/first &&
> +	test_cmp_rev HEAD~3 refs/heads/second &&
> +	test_cmp_rev HEAD~1 refs/heads/third &&
> +	test_cmp_rev HEAD refs/heads/no-conflict-branch
> +'
> +
>   # This must be the last test in this file
>   test_expect_success '$EDITOR and friends are unchanged' '
>   	test_editor_unchanged

Re: [PATCH v4 08/12] rebase: update refs from 'update-ref' commands

[Derrick Stolee]

On 7/15/2022 9:25 AM, Phillip Wood wrote:
>> Now that the 'git rebase --update-refs' command is implemented to write
>> to the update-refs file, we can remove the fake construction of the
>> update-refs file from a test in t2407-worktree-heads.sh.
> 
> This is looking good. I've left a few comments, mostly about error propagation. It's nice to see us recording the initial value of the ref when the todo list is created. It's also good to see this using a lock file. We could perhaps lock the file (with a timeout) when we read it in sequencer_get_update_refs_state() to avoid a race where a process is checking out a new branch in one worktree and another is preparing to rebase that branch in another worktree.

>> +static int do_update_ref(struct repository *r, const char *refname)
>> +{
>> +    struct string_list_item *item;
>> +    struct string_list list = STRING_LIST_INIT_DUP;
>> +
>> +    sequencer_get_update_refs_state(r->gitdir, &list);
> 
> We're ignoring any errors here and always returning 0 from this function.

Thanks. Will fix.
 
>> +
>> +    for_each_string_list_item(item, &list) {
>> +        if (!strcmp(item->string, refname)) {
>> +            struct update_ref_record *rec = item->util;
>> +            read_ref("HEAD", &rec->after);

This is the other place where we could have a failure.

>>   +static int do_update_refs(struct repository *r)
>> +{
>> +    int res = 0;
>> +    struct string_list_item *item;
>> +    struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
>> +    struct ref_store *refs = get_main_ref_store(r);
>> +
>> +    sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);

We need to check for failure here, too.

>> +    for_each_string_list_item(item, &refs_to_oids) {
>> +        struct update_ref_record *rec = item->util;
>> +
>> +        if (oideq(&rec->after, the_hash_algo->null_oid)) {
>> +            /*
>> +             * Ref was not updated. User may have deleted the
>> +             * 'update-ref' step.
>> +             */
> 
> Unless we want to support users editing the todo list without using "git rebase --edit-todo" then by the end of the series it is a bug if we leave an entry in the update-refs file that has been removed from the todo list so I wander if we should remove this if().

I think this is leftover from the previous version and will
never happen. If rec->after is null, then it would be removed
earlier when parsing the todo list.

>>   +    do_update_refs(r);
> 
> Should this be inside the "if (is_rebase_i(opts))" that is closed just above it? We're also ignoring the return value.

Couldn't hurt. Should be a no-op if not in interactive mode.

>>   +    write_update_refs_state(&ctx.refs_to_oids);
> 
> We're ignoring the return value. Also I think todo_list_add_update_ref_commands() only ever returns 0.

Thanks for your attention to detail here.

-Stolee

[PATCH v4 09/12] sequencer: rewrite update-refs as user edits todo list

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

An interactive rebase provides opportunities for the user to edit the
todo list. The --update-refs option initializes the list with some
'update-ref <ref>' steps, but the user could add these manually.
Further, the user could add or remove these steps during pauses in the
interactive rebase.

Add a new method, todo_list_filter_update_refs(), that scans a todo_list
and compares it to the stored update-refs file. There are two actions
that can happen at this point:

1. If a '<ref>/<before>/<after>' triple in the update-refs file does not
   have a matching 'update-ref <ref>' command in the todo-list _and_ the
   <after> value is the null OID, then remove that triple. Here, the
   user removed the 'update-ref <ref>' command before it was executed,
   since if it was executed then the <after> value would store the
   commit at that position.

2. If a 'update-ref <ref>' command in the todo-list does not have a
   matching '<ref>/<before>/<after>' triple in the update-refs file,
   then insert a new one. Store the <before> value to be the current
   OID pointed at by <ref>. This is handled inside of the
   init_update_ref_record() helper method.

We can test that this works by rewriting the todo-list several times in
the course of a rebase. Check that each ref is locked or unlocked for
updates after each todo-list update. We an also verify that the ref
update fails if a concurrent process updates one of the refs after the
rebase process records the "locked" ref location.

To help these tests, add a new 'set_replace_editor' helper that will
replace the todo-list with an exact file.

Reported-by: Phillip Wood <phillip.wood123@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c          |   6 ++
 sequencer.c                   |  96 +++++++++++++++++++++++
 sequencer.h                   |  12 +++
 t/lib-rebase.sh               |  15 ++++
 t/t3404-rebase-interactive.sh | 139 ++++++++++++++++++++++++++++++++++
 5 files changed, 268 insertions(+)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 1ff07647af3..7407c593191 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -146,6 +146,12 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list,
 		return -4;
 	}
 
+	/*
+	 * See if branches need to be added or removed from the update-refs
+	 * file based on the new todo list.
+	 */
+	todo_list_filter_update_refs(r, new_todo);
+
 	return 0;
 }
 
diff --git a/sequencer.c b/sequencer.c
index 2d89b3b727a..2808c027d68 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -4141,6 +4141,102 @@ cleanup:
 	return result;
 }
 
+/*
+ * Parse the update-refs file for the current rebase, then remove the
+ * refs that do not appear in the todo_list (and have not had updated
+ * values stored) and add refs that are in the todo_list but not
+ * represented in the update-refs file.
+ *
+ * If there are changes to the update-refs list, then write the new state
+ * to disk.
+ */
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list)
+{
+	int i;
+	int updated = 0;
+	struct string_list update_refs = STRING_LIST_INIT_DUP;
+
+	sequencer_get_update_refs_state(r->gitdir, &update_refs);
+
+	/*
+	 * For each item in the update_refs list, if it has no updated
+	 * value and does not appear in the todo_list, then remove it
+	 * from the update_refs list.
+	 */
+	for (i = 0; i < update_refs.nr; i++) {
+		int j;
+		int found = 0;
+		const char *ref = update_refs.items[i].string;
+		size_t reflen = strlen(ref);
+		struct update_ref_record *rec = update_refs.items[i].util;
+
+		/* OID already stored as updated. */
+		if (!is_null_oid(&rec->after))
+			continue;
+
+		for (j = 0; !found && j < todo_list->total_nr; j++) {
+			struct todo_item *item = &todo_list->items[j];
+			const char *arg = todo_list->buf.buf + item->arg_offset;
+
+			if (item->command != TODO_UPDATE_REF)
+				continue;
+
+			if (item->arg_len != reflen ||
+			    strncmp(arg, ref, reflen))
+				continue;
+
+			found = 1;
+		}
+
+		if (!found) {
+			free(update_refs.items[i].string);
+			free(update_refs.items[i].util);
+
+			update_refs.nr--;
+			MOVE_ARRAY(update_refs.items + i, update_refs.items + i + 1, update_refs.nr - i);
+
+			updated = 1;
+			i--;
+		}
+	}
+
+	/*
+	 * For each todo_item, check if its ref is in the update_refs list.
+	 * If not, then add it as an un-updated ref.
+	 */
+	for (i = 0; i < todo_list->total_nr; i++) {
+		struct todo_item *item = &todo_list->items[i];
+		const char *arg = todo_list->buf.buf + item->arg_offset;
+		int j, found = 0;
+
+		if (item->command != TODO_UPDATE_REF)
+			continue;
+
+		for (j = 0; !found && j < update_refs.nr; j++) {
+			const char *ref = update_refs.items[j].string;
+
+			found = strlen(ref) == item->arg_len &&
+				!strncmp(ref, arg, item->arg_len);
+		}
+
+		if (!found) {
+			struct string_list_item *inserted;
+			struct strbuf argref = STRBUF_INIT;
+
+			strbuf_add(&argref, arg, item->arg_len);
+			inserted = string_list_insert(&update_refs, argref.buf);
+			inserted->util = init_update_ref_record(argref.buf);
+			strbuf_release(&argref);
+			updated = 1;
+		}
+	}
+
+	if (updated)
+		write_update_refs_state(&update_refs);
+	string_list_clear(&update_refs, 1);
+}
+
 static int do_update_ref(struct repository *r, const char *refname)
 {
 	struct string_list_item *item;
diff --git a/sequencer.h b/sequencer.h
index e671d7e0743..98f3bcc1658 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -132,6 +132,18 @@ void todo_list_release(struct todo_list *todo_list);
 const char *todo_item_get_arg(struct todo_list *todo_list,
 			      struct todo_item *item);
 
+/*
+ * Parse the update-refs file for the current rebase, then remove the
+ * refs that do not appear in the todo_list (and have not had updated
+ * values stored) and add refs that are in the todo_list but not
+ * represented in the update-refs file.
+ *
+ * If there are changes to the update-refs list, then write the new state
+ * to disk.
+ */
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list);
+
 /* Call this to setup defaults before parsing command line options */
 void sequencer_init_config(struct replay_opts *opts);
 int sequencer_pick_revisions(struct repository *repo,
diff --git a/t/lib-rebase.sh b/t/lib-rebase.sh
index ec6b9b107da..b57541356bd 100644
--- a/t/lib-rebase.sh
+++ b/t/lib-rebase.sh
@@ -207,3 +207,18 @@ check_reworded_commits () {
 		>reword-log &&
 	test_cmp reword-expected reword-log
 }
+
+# usage: set_replace_editor <file>
+#
+# Replace the todo file with the exact contents of the given file.
+set_replace_editor () {
+	cat >script <<-\EOF &&
+	cat FILENAME >"$1"
+
+	echo 'rebase -i script after editing:'
+	cat "$1"
+	EOF
+
+	sed -e "s/FILENAME/$1/g" <script | write_script fake-editor.sh &&
+	test_set_editor "$(pwd)/fake-editor.sh"
+}
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index a37820fa728..7bfbd405ab8 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1830,6 +1830,145 @@ test_expect_success '--update-refs updates refs correctly' '
 	test_cmp_rev HEAD refs/heads/no-conflict-branch
 '
 
+test_expect_success 'respect user edits to update-ref steps' '
+	git checkout -B update-refs-break no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git branch -f unseen base &&
+
+	# First, we will add breaks to the expected todo file
+	cat >fake-todo-1 <<-EOF &&
+	pick $(git rev-parse HEAD~3)
+	break
+	update-ref refs/heads/second
+	update-ref refs/heads/first
+
+	pick $(git rev-parse HEAD~2)
+	pick $(git rev-parse HEAD~1)
+	update-ref refs/heads/third
+
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/no-conflict-branch
+	EOF
+
+	# Second, we will drop some update-refs commands (and move one)
+	cat >fake-todo-2 <<-EOF &&
+	update-ref refs/heads/second
+
+	pick $(git rev-parse HEAD~2)
+	update-ref refs/heads/third
+	pick $(git rev-parse HEAD~1)
+	break
+
+	pick $(git rev-parse HEAD)
+	EOF
+
+	# Third, we will:
+	# * insert a new one (new-branch),
+	# * re-add an old one (first), and
+	# * add a second instance of a previously-stored one (second)
+	cat >fake-todo-3 <<-EOF &&
+	update-ref refs/heads/unseen
+	update-ref refs/heads/new-branch
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/first
+	update-ref refs/heads/second
+	EOF
+
+	(
+		set_replace_editor fake-todo-1 &&
+		git rebase -i --update-refs primary &&
+
+		# These branches are currently locked.
+		for b in first second third no-conflict-branch
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		set_replace_editor fake-todo-2 &&
+		git rebase --edit-todo &&
+
+		# These branches are currently locked.
+		for b in second third
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		# These branches are currently unlocked for checkout.
+		for b in first no-conflict-branch
+		do
+			git worktree add wt-$b $b &&
+			git worktree remove wt-$b || return 1
+		done &&
+
+		git rebase --continue &&
+
+		set_replace_editor fake-todo-3 &&
+		git rebase --edit-todo &&
+
+		# These branches are currently locked.
+		for b in second third first unseen
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		# These branches are currently unlocked for checkout.
+		for b in no-conflict-branch
+		do
+			git worktree add wt-$b $b &&
+			git worktree remove wt-$b || return 1
+		done &&
+
+		git rebase --continue
+	) &&
+
+	test_cmp_rev HEAD~2 refs/heads/third &&
+	test_cmp_rev HEAD~1 refs/heads/unseen &&
+	test_cmp_rev HEAD~1 refs/heads/new-branch &&
+	test_cmp_rev HEAD refs/heads/first &&
+	test_cmp_rev HEAD refs/heads/second &&
+	test_cmp_rev HEAD refs/heads/no-conflict-branch
+'
+
+test_expect_success '--update-refs: check failed ref update' '
+	git checkout -B update-refs-error no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~2 &&
+	git branch -f third HEAD~1 &&
+
+	cat >fake-todo <<-EOF &&
+	pick $(git rev-parse HEAD~3)
+	break
+	update-ref refs/heads/first
+
+	pick $(git rev-parse HEAD~2)
+	update-ref refs/heads/second
+
+	pick $(git rev-parse HEAD~1)
+	update-ref refs/heads/third
+
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/no-conflict-branch
+	EOF
+
+	(
+		set_replace_editor fake-todo &&
+		git rebase -i --update-refs base
+	) &&
+
+	# At this point, the values of first, second, and third are
+	# recorded in the update-refs file. We will force-update the
+	# "second" ref, but "git branch -f" will not work because of
+	# the lock in the update-refs file.
+	git rev-parse third >.git/refs/heads/second &&
+
+	git rebase --continue 2>err &&
+	grep "update_ref failed for ref '\''refs/heads/second'\''" err
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v4 09/12] sequencer: rewrite update-refs as user edits todo list

[Phillip Wood]

Hi Stolee

On 12/07/2022 14:07, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> An interactive rebase provides opportunities for the user to edit the
> todo list. The --update-refs option initializes the list with some
> 'update-ref <ref>' steps, but the user could add these manually.
> Further, the user could add or remove these steps during pauses in the
> interactive rebase.
> 
> Add a new method, todo_list_filter_update_refs(), that scans a todo_list
> and compares it to the stored update-refs file. There are two actions
> that can happen at this point:
> 
> 1. If a '<ref>/<before>/<after>' triple in the update-refs file does not
>     have a matching 'update-ref <ref>' command in the todo-list _and_ the
>     <after> value is the null OID, then remove that triple. Here, the
>     user removed the 'update-ref <ref>' command before it was executed,
>     since if it was executed then the <after> value would store the
>     commit at that position.
> 
> 2. If a 'update-ref <ref>' command in the todo-list does not have a
>     matching '<ref>/<before>/<after>' triple in the update-refs file,
>     then insert a new one. Store the <before> value to be the current
>     OID pointed at by <ref>. This is handled inside of the
>     init_update_ref_record() helper method.
> 
> We can test that this works by rewriting the todo-list several times in
> the course of a rebase. Check that each ref is locked or unlocked for
> updates after each todo-list update. We an also verify that the ref
> update fails if a concurrent process updates one of the refs after the
> rebase process records the "locked" ref location.

Thanks for adding this and taking the time to write some good tests. 
When adding a new update-ref command to the todo list it would be nice 
to check if the ref is already checked. We could print a warning or 
force the user to re-edit the todo list as we do if they delete a pick 
and rebase.missingcommitscheck == error

The checks below are quadratic but we don't expect that many refs so 
that should be fine. I don't think it is worth the complexity of 
building a hash table or whatever at this stage.

Best Wishes

Phillip

> To help these tests, add a new 'set_replace_editor' helper that will
> replace the todo-list with an exact file.
> 
> Reported-by: Phillip Wood <phillip.wood123@gmail.com>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   rebase-interactive.c          |   6 ++
>   sequencer.c                   |  96 +++++++++++++++++++++++
>   sequencer.h                   |  12 +++
>   t/lib-rebase.sh               |  15 ++++
>   t/t3404-rebase-interactive.sh | 139 ++++++++++++++++++++++++++++++++++
>   5 files changed, 268 insertions(+)
> 
> diff --git a/rebase-interactive.c b/rebase-interactive.c
> index 1ff07647af3..7407c593191 100644
> --- a/rebase-interactive.c
> +++ b/rebase-interactive.c
> @@ -146,6 +146,12 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list,
>   		return -4;
>   	}
>   
> +	/*
> +	 * See if branches need to be added or removed from the update-refs
> +	 * file based on the new todo list.
> +	 */
> +	todo_list_filter_update_refs(r, new_todo);
> +
>   	return 0;
>   }
>   
> diff --git a/sequencer.c b/sequencer.c
> index 2d89b3b727a..2808c027d68 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -4141,6 +4141,102 @@ cleanup:
>   	return result;
>   }
>   
> +/*
> + * Parse the update-refs file for the current rebase, then remove the
> + * refs that do not appear in the todo_list (and have not had updated
> + * values stored) and add refs that are in the todo_list but not
> + * represented in the update-refs file.
> + *
> + * If there are changes to the update-refs list, then write the new state
> + * to disk.
> + */
> +void todo_list_filter_update_refs(struct repository *r,
> +				  struct todo_list *todo_list)
> +{
> +	int i;
> +	int updated = 0;
> +	struct string_list update_refs = STRING_LIST_INIT_DUP;
> +
> +	sequencer_get_update_refs_state(r->gitdir, &update_refs);
> +
> +	/*
> +	 * For each item in the update_refs list, if it has no updated
> +	 * value and does not appear in the todo_list, then remove it
> +	 * from the update_refs list.
> +	 */
> +	for (i = 0; i < update_refs.nr; i++) {
> +		int j;
> +		int found = 0;
> +		const char *ref = update_refs.items[i].string;
> +		size_t reflen = strlen(ref);
> +		struct update_ref_record *rec = update_refs.items[i].util;
> +
> +		/* OID already stored as updated. */
> +		if (!is_null_oid(&rec->after))
> +			continue;
> +
> +		for (j = 0; !found && j < todo_list->total_nr; j++) {
> +			struct todo_item *item = &todo_list->items[j];
> +			const char *arg = todo_list->buf.buf + item->arg_offset;
> +
> +			if (item->command != TODO_UPDATE_REF)
> +				continue;
> +
> +			if (item->arg_len != reflen ||
> +			    strncmp(arg, ref, reflen))
> +				continue;
> +
> +			found = 1;
> +		}
> +
> +		if (!found) {
> +			free(update_refs.items[i].string);
> +			free(update_refs.items[i].util);
> +
> +			update_refs.nr--;
> +			MOVE_ARRAY(update_refs.items + i, update_refs.items + i + 1, update_refs.nr - i);
> +
> +			updated = 1;
> +			i--;
> +		}
> +	}
> +
> +	/*
> +	 * For each todo_item, check if its ref is in the update_refs list.
> +	 * If not, then add it as an un-updated ref.
> +	 */
> +	for (i = 0; i < todo_list->total_nr; i++) {
> +		struct todo_item *item = &todo_list->items[i];
> +		const char *arg = todo_list->buf.buf + item->arg_offset;
> +		int j, found = 0;
> +
> +		if (item->command != TODO_UPDATE_REF)
> +			continue;
> +
> +		for (j = 0; !found && j < update_refs.nr; j++) {
> +			const char *ref = update_refs.items[j].string;
> +
> +			found = strlen(ref) == item->arg_len &&
> +				!strncmp(ref, arg, item->arg_len);
> +		}
> +
> +		if (!found) {
> +			struct string_list_item *inserted;
> +			struct strbuf argref = STRBUF_INIT;
> +
> +			strbuf_add(&argref, arg, item->arg_len);
> +			inserted = string_list_insert(&update_refs, argref.buf);
> +			inserted->util = init_update_ref_record(argref.buf);
> +			strbuf_release(&argref);
> +			updated = 1;
> +		}
> +	}
> +
> +	if (updated)
> +		write_update_refs_state(&update_refs);
> +	string_list_clear(&update_refs, 1);
> +}
> +
>   static int do_update_ref(struct repository *r, const char *refname)
>   {
>   	struct string_list_item *item;
> diff --git a/sequencer.h b/sequencer.h
> index e671d7e0743..98f3bcc1658 100644
> --- a/sequencer.h
> +++ b/sequencer.h
> @@ -132,6 +132,18 @@ void todo_list_release(struct todo_list *todo_list);
>   const char *todo_item_get_arg(struct todo_list *todo_list,
>   			      struct todo_item *item);
>   
> +/*
> + * Parse the update-refs file for the current rebase, then remove the
> + * refs that do not appear in the todo_list (and have not had updated
> + * values stored) and add refs that are in the todo_list but not
> + * represented in the update-refs file.
> + *
> + * If there are changes to the update-refs list, then write the new state
> + * to disk.
> + */
> +void todo_list_filter_update_refs(struct repository *r,
> +				  struct todo_list *todo_list);
> +
>   /* Call this to setup defaults before parsing command line options */
>   void sequencer_init_config(struct replay_opts *opts);
>   int sequencer_pick_revisions(struct repository *repo,
> diff --git a/t/lib-rebase.sh b/t/lib-rebase.sh
> index ec6b9b107da..b57541356bd 100644
> --- a/t/lib-rebase.sh
> +++ b/t/lib-rebase.sh
> @@ -207,3 +207,18 @@ check_reworded_commits () {
>   		>reword-log &&
>   	test_cmp reword-expected reword-log
>   }
> +
> +# usage: set_replace_editor <file>
> +#
> +# Replace the todo file with the exact contents of the given file.
> +set_replace_editor () {
> +	cat >script <<-\EOF &&
> +	cat FILENAME >"$1"
> +
> +	echo 'rebase -i script after editing:'
> +	cat "$1"
> +	EOF
> +
> +	sed -e "s/FILENAME/$1/g" <script | write_script fake-editor.sh &&
> +	test_set_editor "$(pwd)/fake-editor.sh"
> +}
> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index a37820fa728..7bfbd405ab8 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1830,6 +1830,145 @@ test_expect_success '--update-refs updates refs correctly' '
>   	test_cmp_rev HEAD refs/heads/no-conflict-branch
>   '
>   
> +test_expect_success 'respect user edits to update-ref steps' '
> +	git checkout -B update-refs-break no-conflict-branch &&
> +	git branch -f base HEAD~4 &&
> +	git branch -f first HEAD~3 &&
> +	git branch -f second HEAD~3 &&
> +	git branch -f third HEAD~1 &&
> +	git branch -f unseen base &&
> +
> +	# First, we will add breaks to the expected todo file
> +	cat >fake-todo-1 <<-EOF &&
> +	pick $(git rev-parse HEAD~3)
> +	break
> +	update-ref refs/heads/second
> +	update-ref refs/heads/first
> +
> +	pick $(git rev-parse HEAD~2)
> +	pick $(git rev-parse HEAD~1)
> +	update-ref refs/heads/third
> +
> +	pick $(git rev-parse HEAD)
> +	update-ref refs/heads/no-conflict-branch
> +	EOF
> +
> +	# Second, we will drop some update-refs commands (and move one)
> +	cat >fake-todo-2 <<-EOF &&
> +	update-ref refs/heads/second
> +
> +	pick $(git rev-parse HEAD~2)
> +	update-ref refs/heads/third
> +	pick $(git rev-parse HEAD~1)
> +	break
> +
> +	pick $(git rev-parse HEAD)
> +	EOF
> +
> +	# Third, we will:
> +	# * insert a new one (new-branch),
> +	# * re-add an old one (first), and
> +	# * add a second instance of a previously-stored one (second)
> +	cat >fake-todo-3 <<-EOF &&
> +	update-ref refs/heads/unseen
> +	update-ref refs/heads/new-branch
> +	pick $(git rev-parse HEAD)
> +	update-ref refs/heads/first
> +	update-ref refs/heads/second
> +	EOF
> +
> +	(
> +		set_replace_editor fake-todo-1 &&
> +		git rebase -i --update-refs primary &&
> +
> +		# These branches are currently locked.
> +		for b in first second third no-conflict-branch
> +		do
> +			test_must_fail git branch -f $b base || return 1
> +		done &&
> +
> +		set_replace_editor fake-todo-2 &&
> +		git rebase --edit-todo &&
> +
> +		# These branches are currently locked.
> +		for b in second third
> +		do
> +			test_must_fail git branch -f $b base || return 1
> +		done &&
> +
> +		# These branches are currently unlocked for checkout.
> +		for b in first no-conflict-branch
> +		do
> +			git worktree add wt-$b $b &&
> +			git worktree remove wt-$b || return 1
> +		done &&
> +
> +		git rebase --continue &&
> +
> +		set_replace_editor fake-todo-3 &&
> +		git rebase --edit-todo &&
> +
> +		# These branches are currently locked.
> +		for b in second third first unseen
> +		do
> +			test_must_fail git branch -f $b base || return 1
> +		done &&
> +
> +		# These branches are currently unlocked for checkout.
> +		for b in no-conflict-branch
> +		do
> +			git worktree add wt-$b $b &&
> +			git worktree remove wt-$b || return 1
> +		done &&
> +
> +		git rebase --continue
> +	) &&
> +
> +	test_cmp_rev HEAD~2 refs/heads/third &&
> +	test_cmp_rev HEAD~1 refs/heads/unseen &&
> +	test_cmp_rev HEAD~1 refs/heads/new-branch &&
> +	test_cmp_rev HEAD refs/heads/first &&
> +	test_cmp_rev HEAD refs/heads/second &&
> +	test_cmp_rev HEAD refs/heads/no-conflict-branch
> +'
> +
> +test_expect_success '--update-refs: check failed ref update' '
> +	git checkout -B update-refs-error no-conflict-branch &&
> +	git branch -f base HEAD~4 &&
> +	git branch -f first HEAD~3 &&
> +	git branch -f second HEAD~2 &&
> +	git branch -f third HEAD~1 &&
> +
> +	cat >fake-todo <<-EOF &&
> +	pick $(git rev-parse HEAD~3)
> +	break
> +	update-ref refs/heads/first
> +
> +	pick $(git rev-parse HEAD~2)
> +	update-ref refs/heads/second
> +
> +	pick $(git rev-parse HEAD~1)
> +	update-ref refs/heads/third
> +
> +	pick $(git rev-parse HEAD)
> +	update-ref refs/heads/no-conflict-branch
> +	EOF
> +
> +	(
> +		set_replace_editor fake-todo &&
> +		git rebase -i --update-refs base
> +	) &&
> +
> +	# At this point, the values of first, second, and third are
> +	# recorded in the update-refs file. We will force-update the
> +	# "second" ref, but "git branch -f" will not work because of
> +	# the lock in the update-refs file.
> +	git rev-parse third >.git/refs/heads/second &&
> +
> +	git rebase --continue 2>err &&
> +	grep "update_ref failed for ref '\''refs/heads/second'\''" err
> +'
> +
>   # This must be the last test in this file
>   test_expect_success '$EDITOR and friends are unchanged' '
>   	test_editor_unchanged

Re: [PATCH v4 09/12] sequencer: rewrite update-refs as user edits todo list

[Derrick Stolee]

On 7/15/2022 6:27 AM, Phillip Wood wrote:

>> We can test that this works by rewriting the todo-list several times in
>> the course of a rebase. Check that each ref is locked or unlocked for
>> updates after each todo-list update. We an also verify that the ref
>> update fails if a concurrent process updates one of the refs after the
>> rebase process records the "locked" ref location.
> 
> Thanks for adding this and taking the time to write some good tests. When adding a new update-ref command to the todo list it would be nice to check if the ref is already checked. We could print a warning or force the user to re-edit the todo list as we do if they delete a pick and rebase.missingcommitscheck == error

I agree that that would be a helpful feature instead of the user
discovering the issue at the end of the rebase (as they do with
the message from patch 12).
 
> The checks below are quadratic but we don't expect that many refs so that should be fine. I don't think it is worth the complexity of building a hash table or whatever at this stage.

A hash table would be the natural approach if we see users having
trouble from such high use of the feature.

I think both of these concerns would be excellent for a follow-up,
since they would shave off some rough edges. I hesitate to add them
to this series since it has been growing quite a bit already.

Thanks,
-Stolee

Re: [PATCH v4 09/12] sequencer: rewrite update-refs as user edits todo list

[Phillip Wood]

Hi Stolee

On 15/07/2022 14:13, Derrick Stolee wrote:
> On 7/15/2022 6:27 AM, Phillip Wood wrote:
> [...]
> I think both of these concerns would be excellent for a follow-up,
> since they would shave off some rough edges. I hesitate to add them
> to this series since it has been growing quite a bit already.

Yes that makes sense (and applies to pretty much all of my other 
suggestions as well). This series is looking good and it makes sense to 
get it merged and hopefully get some user feedback before making any 
tweaks. I'm going to off the list most of the next three weeks, so I 
probably wont reply promptly to any reroll. I think that what you've got 
in this version is pretty much ready so I'd be very happy to see this 
topic merged by the time I get back on the list.

Best Wishes

Phillip

> Thanks,
> -Stolee

Re: [PATCH v4 09/12] sequencer: rewrite update-refs as user edits todo list

[Elijah Newren]

On Tue, Jul 12, 2022 at 6:07 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <derrickstolee@github.com>
>
> An interactive rebase provides opportunities for the user to edit the
> todo list. The --update-refs option initializes the list with some
> 'update-ref <ref>' steps, but the user could add these manually.
> Further, the user could add or remove these steps during pauses in the
> interactive rebase.
>
> Add a new method, todo_list_filter_update_refs(), that scans a todo_list
> and compares it to the stored update-refs file. There are two actions
> that can happen at this point:
>
> 1. If a '<ref>/<before>/<after>' triple in the update-refs file does not
>    have a matching 'update-ref <ref>' command in the todo-list _and_ the
>    <after> value is the null OID, then remove that triple. Here, the
>    user removed the 'update-ref <ref>' command before it was executed,
>    since if it was executed then the <after> value would store the
>    commit at that position.
>
> 2. If a 'update-ref <ref>' command in the todo-list does not have a
>    matching '<ref>/<before>/<after>' triple in the update-refs file,
>    then insert a new one. Store the <before> value to be the current
>    OID pointed at by <ref>. This is handled inside of the
>    init_update_ref_record() helper method.
>
> We can test that this works by rewriting the todo-list several times in
> the course of a rebase. Check that each ref is locked or unlocked for
> updates after each todo-list update. We an also verify that the ref

s/an/can/ ?

[PATCH v4 10/12] rebase: add rebase.updateRefs config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change added the --update-refs command-line option.  For
users who always want this mode, create the rebase.updateRefs config
option which behaves the same way as rebase.autoSquash does with the
--autosquash option.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config/rebase.txt |  3 +++
 Documentation/git-rebase.txt    |  3 +++
 builtin/rebase.c                |  5 +++++
 t/t3404-rebase-interactive.sh   | 14 ++++++++++++++
 4 files changed, 25 insertions(+)

diff --git a/Documentation/config/rebase.txt b/Documentation/config/rebase.txt
index 8c979cb20f2..f19bd0e0407 100644
--- a/Documentation/config/rebase.txt
+++ b/Documentation/config/rebase.txt
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index e7611b4089c..1249f9ed617 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -615,6 +615,9 @@ start would be overridden by the presence of
 	are being rebased. Any branches that are checked out in a worktree
 	or point to a `squash! ...` or `fixup! ...` commit are not updated
 	in this way.
++
+If the configuration variable `rebase.updateRefs` is set, then this option
+can be used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 56d82a52106..8ebc98ea505 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 7bfbd405ab8..4b7b77a4123 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1768,6 +1768,12 @@ test_expect_success '--update-refs adds label and update-ref commands' '
 		EOF
 
 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
+
 		test_cmp expect todo
 	)
 '
@@ -1809,6 +1815,14 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 				   --rebase-merges=rebase-cousins \
 				   --update-refs primary >todo &&
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i \
+				   --rebase-merges=rebase-cousins \
+				   primary >todo &&
+
 		test_cmp expect todo
 	)
 '
-- 
gitgitgadget

[PATCH v4 11/12] sequencer: ignore HEAD ref under --update-refs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When using the 'git rebase -i --update-refs' option, the todo list is
populated with 'update-ref' commands for all tip refs in the history
that is being rebased. Refs that are checked out by some worktree are
instead added as a comment to warn the user that they will not be
updated.

Until now, this included the HEAD ref, which is being updated by the
rebase process itself, regardless of the --update-refs option. Remove
the comment in this case by ignoring any decorations that match the HEAD
ref.

Reported-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/sequencer.c b/sequencer.c
index 2808c027d68..82ef062d497 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -5855,12 +5855,25 @@ static int add_decorations_to_list(const struct commit *commit,
 				   struct todo_add_branch_context *ctx)
 {
 	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+	const char *head_ref = resolve_ref_unsafe("HEAD",
+						  RESOLVE_REF_READING,
+						  NULL,
+						  NULL);
 
 	while (decoration) {
 		struct todo_item *item;
 		const char *path;
 		size_t base_offset = ctx->buf->len;
 
+		/*
+		 * If the branch is the current HEAD, then it will be
+		 * updated by the default rebase behavior.
+		 */
+		if (head_ref && !strcmp(head_ref, decoration->name)) {
+			decoration = decoration->next;
+			continue;
+		}
+
 		ALLOC_GROW(ctx->items,
 			ctx->items_nr + 1,
 			ctx->items_alloc);
-- 
gitgitgadget

[PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When the user runs 'git rebase -i --update-refs', the end message still
says only

  Successfully rebased and updated <HEAD-ref>.

Update the sequencer to collect the successful (and unsuccessful) ref
updates due to the --update-refs option, so the end message now says

  Successfully rebased and updated <HEAD-ref>.
  Updated the following refs with --update-refs:
	refs/heads/first
	refs/heads/third
  Failed to update the following refs with --update-refs:
	refs/heads/second

To test this output, we need to be very careful to format the expected
error to drop the leading tab characters. Also, we need to be aware that
the verbose output from 'git rebase' is writing progress lines which
don't use traditional newlines but clear the line after every progress
item is complete. When opening the error file in an editor, these lines
are visible, but when looking at the diff in a terminal those lines
disappear because of the characters that delete the previous characters.
Use 'sed' to clear those progress lines and clear the tabs so we can get
an exact match on our expected output.

Reported-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 40 +++++++++++++++++++++++++++++------
 t/t3404-rebase-interactive.sh | 35 +++++++++++++++++++++++++++---
 2 files changed, 66 insertions(+), 9 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 82ef062d497..bdc67c66f3e 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -4257,17 +4257,20 @@ static int do_update_ref(struct repository *r, const char *refname)
 	return 0;
 }
 
-static int do_update_refs(struct repository *r)
+static int do_update_refs(struct repository *r, int quiet)
 {
 	int res = 0;
 	struct string_list_item *item;
 	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
 	struct ref_store *refs = get_main_ref_store(r);
+	struct strbuf update_msg = STRBUF_INIT;
+	struct strbuf error_msg = STRBUF_INIT;
 
 	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
 
 	for_each_string_list_item(item, &refs_to_oids) {
 		struct update_ref_record *rec = item->util;
+		int loop_res;
 
 		if (oideq(&rec->after, the_hash_algo->null_oid)) {
 			/*
@@ -4277,13 +4280,38 @@ static int do_update_refs(struct repository *r)
 			continue;
 		}
 
-		res |= refs_update_ref(refs, "rewritten during rebase",
-				       item->string,
-				       &rec->after, &rec->before,
-				       0, UPDATE_REFS_MSG_ON_ERR);
+		loop_res = refs_update_ref(refs, "rewritten during rebase",
+					   item->string,
+					   &rec->after, &rec->before,
+					   0, UPDATE_REFS_MSG_ON_ERR);
+		res |= loop_res;
+
+		if (quiet)
+			continue;
+
+		if (loop_res)
+			strbuf_addf(&error_msg, "\t%s\n", item->string);
+		else
+			strbuf_addf(&update_msg, "\t%s\n", item->string);
+	}
+
+	if (!quiet &&
+	    (update_msg.len || error_msg.len)) {
+		fprintf(stderr,
+			_("Updated the following refs with %s:\n%s"),
+			"--update-refs",
+			update_msg.buf);
+
+		if (res)
+			fprintf(stderr,
+				_("Failed to update the following refs with %s:\n%s"),
+				"--update-refs",
+				error_msg.buf);
 	}
 
 	string_list_clear(&refs_to_oids, 1);
+	strbuf_release(&update_msg);
+	strbuf_release(&error_msg);
 	return res;
 }
 
@@ -4804,7 +4832,7 @@ cleanup_head_ref:
 		strbuf_release(&head_ref);
 	}
 
-	do_update_refs(r);
+	do_update_refs(r, opts->quiet);
 
 	/*
 	 * Sequence of picks finished successfully; cleanup by
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 4b7b77a4123..ef902b5431f 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1836,12 +1836,26 @@ test_expect_success '--update-refs updates refs correctly' '
 	test_commit extra2 fileX &&
 	git commit --amend --fixup=L &&
 
-	git rebase -i --autosquash --update-refs primary &&
+	git rebase -i --autosquash --update-refs primary 2>err &&
 
 	test_cmp_rev HEAD~3 refs/heads/first &&
 	test_cmp_rev HEAD~3 refs/heads/second &&
 	test_cmp_rev HEAD~1 refs/heads/third &&
-	test_cmp_rev HEAD refs/heads/no-conflict-branch
+	test_cmp_rev HEAD refs/heads/no-conflict-branch &&
+
+	cat >expect <<-\EOF &&
+	Successfully rebased and updated refs/heads/update-refs.
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/second
+		refs/heads/third
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 test_expect_success 'respect user edits to update-ref steps' '
@@ -1980,7 +1994,22 @@ test_expect_success '--update-refs: check failed ref update' '
 	git rev-parse third >.git/refs/heads/second &&
 
 	git rebase --continue 2>err &&
-	grep "update_ref failed for ref '\''refs/heads/second'\''" err
+	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
+
+	cat >expect <<-\EOF &&
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/third
+	Failed to update the following refs with --update-refs:
+		refs/heads/second
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	tail -n 6 err >err.last &&
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err.last >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 # This must be the last test in this file
-- 
gitgitgadget

Re: [PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Phillip Wood]

On 12/07/2022 14:07, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> When the user runs 'git rebase -i --update-refs', the end message still
> says only
> 
>    Successfully rebased and updated <HEAD-ref>.
> 
> Update the sequencer to collect the successful (and unsuccessful) ref
> updates due to the --update-refs option, so the end message now says
> 
>    Successfully rebased and updated <HEAD-ref>.
>    Updated the following refs with --update-refs:
> 	refs/heads/first
> 	refs/heads/third
>    Failed to update the following refs with --update-refs:
> 	refs/heads/second
> 
> To test this output, we need to be very careful to format the expected
> error to drop the leading tab characters. Also, we need to be aware that
> the verbose output from 'git rebase' is writing progress lines which

s/is writing/writes/ ?

> don't use traditional newlines but clear the line after every progress
> item is complete.

I was a bit confused by the reference to "verbose output" in this 
paragraph. When the user passes --verbose then we do actually use NL, it 
is when the user does not pass verbose that we use CR instead.

> When opening the error file in an editor, these lines
> are visible, but when looking at the diff in a terminal those lines
> disappear because of the characters that delete the previous characters.
> Use 'sed' to clear those progress lines and clear the tabs so we can get
> an exact match on our expected output.

Thanks for the comprehensive commit message and for implementing an 
excellent suggestion from Elijah. I wonder if it makes sense to 
distinguish between the current branch and all the others when writing 
the update message as we do here or if all the refs should just be in a 
single list. I also think it doesn't matter much and we can change it 
later if we want.

 From the last test it looks like we are already printing something when 
we fail to update a ref (possibly this comes from the refs backend code) 
I don't think it hurts to print a summary of them all after that though.

> Reported-by: Elijah Newren <newren@gmail.com>
> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
> ---
>   sequencer.c                   | 40 +++++++++++++++++++++++++++++------
>   t/t3404-rebase-interactive.sh | 35 +++++++++++++++++++++++++++---
>   2 files changed, 66 insertions(+), 9 deletions(-)
> 
> diff --git a/sequencer.c b/sequencer.c
> index 82ef062d497..bdc67c66f3e 100644
> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -4257,17 +4257,20 @@ static int do_update_ref(struct repository *r, const char *refname)
>   	return 0;
>   }
>   
> -static int do_update_refs(struct repository *r)
> +static int do_update_refs(struct repository *r, int quiet)
>   {
>   	int res = 0;
>   	struct string_list_item *item;
>   	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
>   	struct ref_store *refs = get_main_ref_store(r);
> +	struct strbuf update_msg = STRBUF_INIT;
> +	struct strbuf error_msg = STRBUF_INIT;
>   
>   	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
>   
>   	for_each_string_list_item(item, &refs_to_oids) {
>   		struct update_ref_record *rec = item->util;
> +		int loop_res;
>   
>   		if (oideq(&rec->after, the_hash_algo->null_oid)) {
>   			/*
> @@ -4277,13 +4280,38 @@ static int do_update_refs(struct repository *r)
>   			continue;
>   		}
>   
> -		res |= refs_update_ref(refs, "rewritten during rebase",
> -				       item->string,
> -				       &rec->after, &rec->before,
> -				       0, UPDATE_REFS_MSG_ON_ERR);
> +		loop_res = refs_update_ref(refs, "rewritten during rebase",
> +					   item->string,
> +					   &rec->after, &rec->before,
> +					   0, UPDATE_REFS_MSG_ON_ERR);
> +		res |= loop_res;
> +
> +		if (quiet)
> +			continue;
> +
> +		if (loop_res)
> +			strbuf_addf(&error_msg, "\t%s\n", item->string);
> +		else
> +			strbuf_addf(&update_msg, "\t%s\n", item->string);
> +	}
> +
> +	if (!quiet &&

As you skip adding anything to the strbufs when quiet is true you don't 
really need this test

> +	    (update_msg.len || error_msg.len)) {
> +		fprintf(stderr,
> +			_("Updated the following refs with %s:\n%s"),
> +			"--update-refs",
> +			update_msg.buf);

This will be printed even if all the updates falied

> +
> +		if (res)
> +			fprintf(stderr,
> +				_("Failed to update the following refs with %s:\n%s"),
> +				"--update-refs",
> +				error_msg.buf);
>   	}
>   
>   	string_list_clear(&refs_to_oids, 1);
> +	strbuf_release(&update_msg);
> +	strbuf_release(&error_msg);
>   	return res;
>   }
>   
> @@ -4804,7 +4832,7 @@ cleanup_head_ref:
>   		strbuf_release(&head_ref);
>   	}
>   
> -	do_update_refs(r);
> +	do_update_refs(r, opts->quiet);
>   
>   	/*
>   	 * Sequence of picks finished successfully; cleanup by
> diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
> index 4b7b77a4123..ef902b5431f 100755
> --- a/t/t3404-rebase-interactive.sh
> +++ b/t/t3404-rebase-interactive.sh
> @@ -1836,12 +1836,26 @@ test_expect_success '--update-refs updates refs correctly' '
>   	test_commit extra2 fileX &&
>   	git commit --amend --fixup=L &&
>   
> -	git rebase -i --autosquash --update-refs primary &&
> +	git rebase -i --autosquash --update-refs primary 2>err &&
>   
>   	test_cmp_rev HEAD~3 refs/heads/first &&
>   	test_cmp_rev HEAD~3 refs/heads/second &&
>   	test_cmp_rev HEAD~1 refs/heads/third &&
> -	test_cmp_rev HEAD refs/heads/no-conflict-branch
> +	test_cmp_rev HEAD refs/heads/no-conflict-branch &&
> +
> +	cat >expect <<-\EOF &&
> +	Successfully rebased and updated refs/heads/update-refs.
> +	Updated the following refs with --update-refs:
> +		refs/heads/first
> +		refs/heads/no-conflict-branch
> +		refs/heads/second
> +		refs/heads/third
> +	EOF
> +
> +	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
> +	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
> +		<err >err.trimmed &&
> +	test_cmp expect err.trimmed
>   '
>   
>   test_expect_success 'respect user edits to update-ref steps' '
> @@ -1980,7 +1994,22 @@ test_expect_success '--update-refs: check failed ref update' '
>   	git rev-parse third >.git/refs/heads/second &&
>   
>   	git rebase --continue 2>err &&
> -	grep "update_ref failed for ref '\''refs/heads/second'\''" err
> +	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
> +
> +	cat >expect <<-\EOF &&
> +	Updated the following refs with --update-refs:
> +		refs/heads/first
> +		refs/heads/no-conflict-branch
> +		refs/heads/third
> +	Failed to update the following refs with --update-refs:
> +		refs/heads/second
> +	EOF
> +
> +	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
> +	tail -n 6 err >err.last &&

I'm curious as to why we need tail here but not in the test above.

This is looking good, I'd be happy enough to see it merged as is.

Best Wishes

Phillip

> +	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
> +		<err.last >err.trimmed &&
> +	test_cmp expect err.trimmed
>   '
>   
>   # This must be the last test in this file

Re: [PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Derrick Stolee]

On 7/15/2022 6:12 AM, Phillip Wood wrote:
> On 12/07/2022 14:07, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> When the user runs 'git rebase -i --update-refs', the end message still
>> says only
>>
>>    Successfully rebased and updated <HEAD-ref>.
>>
>> Update the sequencer to collect the successful (and unsuccessful) ref
>> updates due to the --update-refs option, so the end message now says
>>
>>    Successfully rebased and updated <HEAD-ref>.
>>    Updated the following refs with --update-refs:
>>     refs/heads/first
>>     refs/heads/third
>>    Failed to update the following refs with --update-refs:
>>     refs/heads/second
>>
>> To test this output, we need to be very careful to format the expected
>> error to drop the leading tab characters. Also, we need to be aware that
>> the verbose output from 'git rebase' is writing progress lines which
> 
> s/is writing/writes/ ?

That is an improvement. Thanks.
 
>> don't use traditional newlines but clear the line after every progress
>> item is complete.
> 
> I was a bit confused by the reference to "verbose output" in this paragraph. When the user passes --verbose then we do actually use NL, it is when the user does not pass verbose that we use CR instead.

I was mostly thinking that the summary message at the end is only shown
when the command has verbose output (which really means the default of
--no-quiet). There is no current way to have --no-progress --no-quiet
to keep the progress lines out of the output but still have the summary.

>> When opening the error file in an editor, these lines
>> are visible, but when looking at the diff in a terminal those lines
>> disappear because of the characters that delete the previous characters.
>> Use 'sed' to clear those progress lines and clear the tabs so we can get
>> an exact match on our expected output.
> 
> Thanks for the comprehensive commit message and for implementing an excellent suggestion from Elijah. I wonder if it makes sense to distinguish between the current branch and all the others when writing the update message as we do here or if all the refs should just be in a single list. I also think it doesn't matter much and we can change it later if we want.

I'm definitely open to suggestions, but I also think we should start
somewhere and see what users think. Since the mechanisms for updating the
refs are different, I felt it was appropriate to have different error
messages.

>> +    if (!quiet &&
> 
> As you skip adding anything to the strbufs when quiet is true you don't really need this test
> 
>> +        (update_msg.len || error_msg.len)) {
>> +        fprintf(stderr,
>> +            _("Updated the following refs with %s:\n%s"),
>> +            "--update-refs",
>> +            update_msg.buf);
> 
> This will be printed even if all the updates falied

These are good points! Instead, I should arrange this as

	if (update_msg.len)
		/* send the success message. */
	if (error_msg.len)
		/* send the failure message. */

>> +    cat >expect <<-\EOF &&
>> +    Updated the following refs with --update-refs:
>> +        refs/heads/first
>> +        refs/heads/no-conflict-branch
>> +        refs/heads/third
>> +    Failed to update the following refs with --update-refs:
>> +        refs/heads/second
>> +    EOF

(copying this message from earlier in your message for context)

> From the last test it looks like we are already printing something when we fail to update a ref (possibly this comes from the refs backend code) I don't think it hurts to print a summary of them all after that though.

>> +    # Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
>> +    tail -n 6 err >err.last &&
> 
> I'm curious as to why we need tail here but not in the test above.

I think I was trying to avoid testing the error messages from the
ref backend code. I bet that since this tail is here, the 'sed' is
no longer required.

>> +    sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
>> +        <err.last >err.trimmed &&
>> +    test_cmp expect err.trimmed

Thanks!
-Stolee

Re: [PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Elijah Newren]

On Fri, Jul 15, 2022 at 6:20 AM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 7/15/2022 6:12 AM, Phillip Wood wrote:
>
> > Thanks for the comprehensive commit message and for implementing an excellent suggestion from Elijah. I wonder if it makes sense to distinguish between the current branch and all the others when writing the update message as we do here or if all the refs should just be in a single list. I also think it doesn't matter much and we can change it later if we want.
>
> I'm definitely open to suggestions, but I also think we should start
> somewhere and see what users think. Since the mechanisms for updating the
> refs are different, I felt it was appropriate to have different error
> messages.

I think the separate messages are fine, but I'm a little surprised by
the wording of this rationale.  The mechanisms for updating the refs
are implementational details that are likely not known by the end
user, and aren't something they can tweak either.  As such, I don't
think the mechanism used should result in any differences in end-user
messages here.

However, one difference here is that one of the refs' changes (namely,
the one corresponding to HEAD) will also have its changes be reflected
in the current working directory and index.  That may be a good reason
to keep its update message separate from the rest of the refs.

But on a related tangent...

I'm still curious if people are going to be surprised to not see an
"update-ref" line in the todo list for the HEAD ref; I was at first,
and only understood the rationale for excluding it based on the
implementational details.  I know as far as the implementation goes
that if such a line did appear in the todo list, then you'd probably
just ignore it (other then ensuring it was the last line) and then let
the pre-existing mechanisms in rebase update the HEAD ref...but even
with that implementation "weirdness", I think incorporating it might
be less surprising to the end user than the current behavior.  Or
maybe I'm just knee-deep in implementation details anyway, and I'm
just guessing what others might think.  I don't feel strongly about
it, just thought I'd surface the thought.

Re: [PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Elijah Newren]

On Tue, Jul 12, 2022 at 6:07 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <derrickstolee@github.com>
>
> When the user runs 'git rebase -i --update-refs', the end message still
> says only
>
>   Successfully rebased and updated <HEAD-ref>.
>
> Update the sequencer to collect the successful (and unsuccessful) ref
> updates due to the --update-refs option, so the end message now says
>
>   Successfully rebased and updated <HEAD-ref>.
>   Updated the following refs with --update-refs:
>         refs/heads/first
>         refs/heads/third

This seems good.

>   Failed to update the following refs with --update-refs:
>         refs/heads/second

This is good, but I think it could be improved.  Could we also include
the commit to which rebase would have updated the branch to?  That
would allow the user to manually update it if they want, or at least
see a range-diff between what we would have updated it to and what it
now has.  Without that information, the user might have difficulty
correcting that branch.


Anyway, I've only been able to find a few minor things in reviewing
this series.  It looks really good overall, and I'm sure lots of
people will be making use of it!

Re: [PATCH v4 12/12] sequencer: notify user of --update-refs activity

[Derrick Stolee]

On 7/16/2022 6:09 PM, Elijah Newren wrote:
> On Tue, Jul 12, 2022 at 6:07 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> When the user runs 'git rebase -i --update-refs', the end message still
>> says only
>>
>>   Successfully rebased and updated <HEAD-ref>.
>>
>> Update the sequencer to collect the successful (and unsuccessful) ref
>> updates due to the --update-refs option, so the end message now says
>>
>>   Successfully rebased and updated <HEAD-ref>.
>>   Updated the following refs with --update-refs:
>>         refs/heads/first
>>         refs/heads/third
> 
> This seems good.
> 
>>   Failed to update the following refs with --update-refs:
>>         refs/heads/second
> 
> This is good, but I think it could be improved.  Could we also include
> the commit to which rebase would have updated the branch to?  That
> would allow the user to manually update it if they want, or at least
> see a range-diff between what we would have updated it to and what it
> now has.  Without that information, the user might have difficulty
> correcting that branch.

Would you mind if I left this as something for #leftoverbits? I
expect that a follow-up series will be necessary once we have more
user feedback. This isn't the only item delayed until after more
feedback.

Thanks,
-Stolee

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This series is based on ds/branch-checked-out.
> ...
> Updates in v4
> =============
>
> This version took longer than I'd hoped (I had less time to work on it than
> anticipated) but it also has some major updates. These major updates are
> direct responses to the significant review this series has received. Thank
> you!
>
>  * The update-refs file now stores "ref/before/after" triples (still
>    separated by lines). This allows us to store the "before" OID of a ref in
>    addition to the "after" that we will write to that ref at the end of the
>    rebase. This allows us to do a "force-with-lease" update. The
>    branch_checked_out() updates should prevent Git from updating those refs
>    while under the rebase, but older versions and third-party tools don't
>    have that protection.

Nice.

>  * The update-refs file is updated with every update to the todo-list file.
>    This allows for some advanced changes to the file, including removing,
>    adding, and duplicating 'update-ref' commands.
>  * The message at the end of the rebase process now lists which refs were
>    updated with the update-ref steps. This includes any ref updates that
>    fail.
>  * The branch_checked_out() tests now use 'git bisect' and 'git rebase' as
>    black-boxes instead of testing their internals directly.
>
> Here are the more minor updates:
>
>  * Dropped an unnecessary stat() call.
>  * Updated commit messages to include extra details, based on confusion in
>    last round.
>  * The HEAD branch no longer appears as a comment line in the initial todo
>    list.
>  * The update-refs file is now written using a lockfile.
>  * Tests now use test_cmp_rev.
>  * A memory leak ('path' variable) is resolved.

Interesting.

This is a tangent, but may serve as some food for thought.

When I queue (or develop myself) a topic that depends on another
topic, I often do

    $ git checkout --detach vX.Y.Z ;# choose an appropriate base
    $ git merge --into derived-topic base-topic
    $ develop develop (or "git am")

which would end up in

    vX.Y.Z -----M---A---B---C derived-topic
               /
     base-topic

so that "git log --first-parent --oneline master.." would show the
commits in the topic plus the fact that it depends on which other
topic recorded in a single merge commit.  A topic that depends on
two or more topics can be handled the same way.

One good thing about this arrangement, unlike the "totally linear"
one depicted at the top of your cover letter, is that it is easier
to rebuild each topic independently and the first-parent view is
still useful.  If you futz with the base topic in a totally linear
history, "log --decorate" of the derived topic would no longer tell
you where the old iteration of the base topic ended.

It would be very nice to see the update-ref feature (or something
like that) makes it easy to deal with such a topology, too.

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Derrick Stolee]

[-- Attachment #1: Type: text/plain, Size: 3976 bytes --]

On 7/12/22 11:37 AM, Junio C Hamano wrote:> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> This is a tangent, but may serve as some food for thought.
> 
> When I queue (or develop myself) a topic that depends on another
> topic, I often do
> 
>     $ git checkout --detach vX.Y.Z ;# choose an appropriate base
>     $ git merge --into derived-topic base-topic
>     $ develop develop (or "git am")
> 
> which would end up in
> 
>     vX.Y.Z -----M---A---B---C derived-topic
>                /
>      base-topic
> 
> so that "git log --first-parent --oneline master.." would show the
> commits in the topic plus the fact that it depends on which other
> topic recorded in a single merge commit.  A topic that depends on
> two or more topics can be handled the same way.
> 
> One good thing about this arrangement, unlike the "totally linear"
> one depicted at the top of your cover letter, is that it is easier
> to rebuild each topic independently and the first-parent view is
> still useful.  If you futz with the base topic in a totally linear
> history, "log --decorate" of the derived topic would no longer tell
> you where the old iteration of the base topic ended.
> 
> It would be very nice to see the update-ref feature (or something
> like that) makes it easy to deal with such a topology, too.

Your topology is an interesting one, but --update-refs isn't limited to
linear history.

(Begin aside)

One goal of this feature is to make it easier to manage the topics that
are being juggled in friendly forks. For example, git-for-windows/git and
microsoft/git have some topics that evolve version-to-version but might be
cleaned up and sent upstream. Both of these forks use 'git rebase
--rebase-merges=rebase-cousins' when consuming new upstream versions to
keep the merge structure of these new topics. However, we lose the branch
names and need to reconstruct them from the context of the merge commits.

With --update-refs, we can automatically rewrite the branches that are
included in these individual topics. That might make it simpler to extract
a series to send upstream.

One test in this series does test such a case with the --rebase-merges
option.

(End aside)

Back to your topology, I wonder what your rebase command looks like when
tracking those topics.

The goal of --update-refs is to help rebase multiple branches at the same
time, and with your example here, it would imply you want to rebase both
dependent topics.

 Before:

  A---B----C---M---Q1---Q2---Q3 <-- refs/heads/Q
   \          /
    P1--P2--P3 <-- refs/heads/P

 After rebasing both topics simultaneously (with 'git rebase --update-refs
 C' while Q is checked out):

  A---B---C---D---P1---P2---P3---Q1---Q2---Q3
			    ^              ^
			refs/heads/P  refs/heads/Q

But it seems what you mean to say is to update the merge commit M, which
means that the 'P' branch above has been updated independently of the 'Q'
branch, so we need to update 'Q' after-the-fact. I'm not sure what that
rebase would look like, indepdendent of updating refs.

Do you have an example rebase command that manipulates the commits the
way you want? Then I can better understand how the --update-refs could fit
in with that. (Or maybe the point of your tangent is that there isn't an
option.)

If instead we thought about an example like re-rolling the 'next' branch
entirely on top of the 'master' branch, then we have an example that is
closer to the friendly fork example. (I know this isn't a realistic
scenario since we don't rewrite the commits already merged to 'next', but
it's an interesting stress test.)

While having 'next' checked out, I ran

  git rebase -i --update-refs --rebase-merges=rebase-cousins master

and it updated all of the branches currently in the region master..next.
I've attached the output of

  git -c log.excludeDecoration="refs/remotes/*" \
        log --oneline --graph --boundary master..next

just to show what this looks like.

Thanks,
-Stolee

[-- Attachment #2: rebase-result.txt --]
[-- Type: text/plain, Size: 13575 bytes --]

*   bdf2d8e9e1 Merge branch 'll/curl-accept-language' into next
|\  
| * b0c4adcdd7 remote-curl: send Accept-Language header to server
* |   36fcf76053 Merge branch 'jk/diff-files-cleanup-fix' into next
|\ \  
| * | 04393ae7f7 diff-files: move misplaced cleanup label
* | |   3f6ccaf58c Merge branch 'rs/cocci-array-copy' into next
|\ \ \  
| * | | dc574132a0 cocci: avoid normalization rules for memcpy
* | | |   2125d3847d Merge branch 'jk/ref-filter-discard-commit-buffer' into next
|\ \ \ \  
| * | | | 359b01ca84 ref-filter: disable save_commit_buffer while traversing
* | | | |   fd0e10a90f Merge branch 'jk/clone-unborn-confusion' into next
|\ \ \ \ \  
| * | | | | 2eb5d38386 clone: move unborn head creation to update_head()
| * | | | | 73a0858224 clone: use remote branch if it matches default HEAD
| * | | | | 54a0c1896c clone: propagate empty remote HEAD even with other branches
| * | | | | 51fa61ba3e clone: drop extra newline from warning message
| | |/ / /  
| |/| | |   
* | | | |   05d6fbe0e5 Merge branch 'hx/lookup-commit-in-graph-fix' into next
|\ \ \ \ \  
| * | | | | 854d62d9fe t5330: remove run_with_limited_processses()
* | | | | |   5386228a28 Merge branch 'jc/resolve-undo' into next
|\ \ \ \ \ \  
| * | | | | | b237e81516 fsck: do not dereference NULL while checking resolve-undo data
* | | | | | |   a34a4435d9 Merge branch 'sg/multi-pack-index-parse-options-fix' into next
|\ \ \ \ \ \ \  
| * | | | | | | cc74afb83f multi-pack-index: simplify handling of unknown --options
| | |_|_|/ / /  
| |/| | | | |   
* | | | | | |   57b36b2e9b Merge branch 'bc/nettle-sha256' into next
|\ \ \ \ \ \ \  
| * | | | | | | e555735836 sha256: add support for Nettle
| |/ / / / / /  
* | | | | | |   11cf608d10 Merge branch 'kk/p4-client-name-encoding-fix' into next
|\ \ \ \ \ \ \  
| * | | | | | | 34f67c9619 git-p4: fix bug with encoding of p4 client name
| |/ / / / / /  
* | | | | | |   99a9342263 Merge branch 'jd/gpg-interface-trust-level-string' into next
|\ \ \ \ \ \ \  
| * | | | | | | 803978da49 gpg-interface: add function for converting trust level to string
| |/ / / / / /  
* | | | | | |   47942e9ac7 Merge branch 'ab/cocci-unused' into next
|\ \ \ \ \ \ \  
| * | | | | | | 06f5f8940c cocci: generalize "unused" rule to cover more than "strbuf"
| * | | | | | | 4f40f6cb73 cocci: add and apply a rule to find "unused" strbufs
| * | | | | | | 7a9a10b10e cocci: have "coccicheck{,-pending}" depend on "coccicheck-test"
| * | | | | | | f7ff6597a7 cocci: add a "coccicheck-test" target and test *.cocci rules
| * | | | | | | af0aa6904b Makefile & .gitignore: ignore & clean "git.res", not "*.res"
| * | | | | | | 7b63ea5750 Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS
* | | | | | | |   9720950472 Merge branch 'gc/submodule-use-super-prefix' into next
|\ \ \ \ \ \ \ \  
| * | | | | | | | 5ad87271cf submodule--helper: remove display path helper
| * | | | | | | | d7a714fddc submodule--helper update: use --super-prefix
| * | | | | | | | b0f8b21305 submodule--helper: remove unused SUPPORT_SUPER_PREFIX flags
| * | | | | | | | 58cec298f1 submodule--helper: use correct display path helper
| * | | | | | | | cb49e1e8d3 submodule--helper: don't recreate recursive prefix
| * | | | | | | | 618b8445d9 submodule--helper update: use display path helper
| * | | | | | | | 8fc36c39d9 submodule--helper tests: add missing "display path" coverage
| * | | | | | | |   c9e221b124 Merge branch 'ab/submodule-cleanup' into gc/submodule-use-super-prefix
| |\ \ \ \ \ \ \ \  
| | |_|/ / / / / /  
| |/| | | | | | |   
* | | | | | | | |   172d4d2a2b Merge branch 'en/merge-dual-dir-renames-fix' into next
|\ \ \ \ \ \ \ \ \  
| * | | | | | | | | 8fe673ee1d merge-ort: fix issue with dual rename and add/add conflict
| * | | | | | | | | 2ef5b00eba merge-ort: shuffle the computation and cleanup of potential collisions
| * | | | | | | | | b3468c2055 merge-ort: make a separate function for freeing struct collisions
| * | | | | | | | | a4719d9e46 merge-ort: small cleanups of check_for_directory_rename
| * | | | | | | | | 70fae629e6 t6423: add tests of dual directory rename plus add/add conflict
| | |_|_|_|_|/ / /  
| |/| | | | | | |   
* | | | | | | | |   9fc51f4bdd Merge branch 'ab/test-without-templates' into next
|\ \ \ \ \ \ \ \ \  
| * | | | | | | | | c8ab2aa9ce tests: don't assume a .git/info for .git/info/sparse-checkout
| * | | | | | | | | 913769a360 tests: don't assume a .git/info for .git/info/exclude
| * | | | | | | | | af4eb37df7 tests: don't assume a .git/info for .git/info/refs
| * | | | | | | | | 5383f7f9de tests: don't assume a .git/info for .git/info/attributes
| * | | | | | | | | dc7e8c3b3a tests: don't assume a .git/info for .git/info/grafts
| * | | | | | | | | fdc489c66c tests: don't depend on template-created .git/branches
| * | | | | | | | | 7a1cb29f61 t0008: don't rely on default ".git/info/exclude"
| |/ / / / / / / /  
* | | | | | | | |   8d446e7df8 Merge branch 'ab/build-gitweb' into next
|\ \ \ \ \ \ \ \ \  
| * | | | | | | | | a35258c62a gitweb/Makefile: add a "NO_GITWEB" parameter
| * | | | | | | | | d3b827408c Makefile: build 'gitweb' in the default target
| * | | | | | | | | affc3b755c gitweb/Makefile: include in top-level Makefile
| * | | | | | | | | 27438ef5e0 gitweb: remove "test" and "test-installed" targets
| * | | | | | | | | b82d66eb0c gitweb/Makefile: prepare to merge into top-level Makefile
| * | | | | | | | | 564ebde3d3 gitweb/Makefile: clear up and de-duplicate the gitweb.{css,js} vars
| * | | | | | | | | 1e08fa5e2b gitweb/Makefile: add a $(GITWEB_ALL) variable
| * | | | | | | | | 7decdb9b4a gitweb/Makefile: define all .PHONY prerequisites inline
* | | | | | | | | |   3ab689137a Merge branch 'ab/test-tool-leakfix' into next
|\ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | f40a693450 test-tool delta: fix a memory leak
| * | | | | | | | | | 34e691288d test-tool ref-store: fix a memory leak
| * | | | | | | | | | 9794633b4e test-tool bloom: fix memory leaks
| * | | | | | | | | | 1caaa858cc test-tool json-writer: fix memory leaks
| * | | | | | | | | | a20b0dc796 test-tool regex: call regfree(), fix memory leaks
| * | | | | | | | | | 1c343e5aef test-tool urlmatch-normalization: fix a memory leak
| * | | | | | | | | | 9afa46d4a6 test-tool {dump,scrap}-cache-tree: fix memory leaks
| * | | | | | | | | | e287a5b0a4 test-tool path-utils: fix a memory leak
| * | | | | | | | | | 330ca8501b test-tool test-hash: fix a memory leak
| | |_|/ / / / / / /  
| |/| | | | | | | |   
* | | | | | | | | |   b9183a05c5 Merge branch 'ab/leakfix' into next
|\ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | ece3974ba6 pull: fix a "struct oid_array" memory leak
| * | | | | | | | | | 27472b5195 cat-file: fix a common "struct object_context" memory leak
| * | | | | | | | | | 55916bba0f gc: fix a memory leak
| * | | | | | | | | | 33d0dda633 checkout: avoid "struct unpack_trees_options" leak
| * | | | | | | | | | e72e12cc02 merge-file: fix memory leaks on error path
| * | | | | | | | | | 480a0e30a7 merge-file: refactor for subsequent memory leak fix
| * | | | | | | | | | d90dafbe31 cat-file: fix a memory leak in --batch-command mode
| * | | | | | | | | | fd74ac95ac revert: free "struct replay_opts" members
| * | | | | | | | | | bc57ba1d54 submodule.c: free() memory from xgetcwd()
| * | | | | | | | | | 74a06a9f21 clone: fix memory leak in wanted_peer_refs()
| * | | | | | | | | | 99b6c45d8f check-ref-format: fix trivial memory leak
| |/ / / / / / / / /  
* | | | | | | | | |   542f8fec7e Merge branch 'jc/builtin-mv-move-array' into next
|\ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | 4901a36b4b builtin/mv.c: use the MOVE_ARRAY() macro instead of memmove()
| | |_|/ / / / / / /  
| |/| | | | | | | |   
* | | | | | | | | |   1b5d8c02cf Merge branch 'fr/vimdiff-layout-fix' into next
|\ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | a62efbaabb vimdiff: make layout engine more robust against user vim settings
| |/ / / / / / / / /  
* | | | | | | | | |   f022414e31 Merge branch 'hx/lookup-commit-in-graph-fix' into next
|\ \ \ \ \ \ \ \ \ \  
| | |_|_|_|_|_|/ / /  
| |/| | | | | | | |   
| * | | | | | | | | a4d60a6221 commit-graph.c: no lazy fetch in lookup_commit_in_graph()
| |/ / / / / / / /  
* | | | | | | | |   90790c4455 Merge branch 'ab/submodule-cleanup' into next
|\ \ \ \ \ \ \ \ \  
| | |_|_|/ / / / /  
| |/| | | | | | |   
| * | | | | | | | 5b893f7d81 git-sh-setup.sh: remove "say" function, change last users
| * | | | | | | | 2eec463739 git-submodule.sh: use "$quiet", not "$GIT_QUIET"
| * | | | | | | | b788fc671b submodule--helper: eliminate internal "--update" option
| * | | | | | | | 8f12108c29 submodule--helper: understand --checkout, --merge and --rebase synonyms
| * | | | | | | | 36d45163b6 submodule--helper: report "submodule" as our name in some "-h" output
| * | | | | | | | 6e556c412e submodule--helper: rename "absorb-git-dirs" to "absorbgitdirs"
| * | | | | | | | 0d68ee723e submodule update: remove "-v" option
| * | | | | | | | d9c7f69aaa submodule--helper: have --require-init imply --init
| * | | | | | | | da3aae9e84 git-submodule.sh: remove unused top-level "--branch" argument
| * | | | | | | | 757d092797 git-submodule.sh: make the "$cached" variable a boolean
| * | | | | | | | 960fad98e8 git-submodule.sh: remove unused $prefix variable
| * | | | | | | | 85775255f1 git-submodule.sh: remove unused sanitize_submodule_env()
* | | | | | | | |   d1ce5a9649 Merge branch 'sy/mv-out-of-cone' into next
|\ \ \ \ \ \ \ \ \  
| * | | | | | | | | b91a2b6594 mv: add check_dir_in_index() and solve general dir check issue
| * | | | | | | | | 24ea81d9ac mv: use flags mode for update_mode
| * | | | | | | | | 8a26a3915f mv: check if <destination> exists in index to handle overwriting
| * | | | | | | | | 6645b03ca5 mv: check if out-of-cone file exists in index with SKIP_WORKTREE bit
| * | | | | | | | | 7889755bae mv: decouple if/else-if checks using goto
| * | | | | | | | | 707fa2f76a mv: update sparsity after moving from out-of-cone to in-cone
| * | | | | | | | | 1143cc01b7 t1092: mv directory from out-of-cone to in-cone
| * | | | | | | | | 367844e5b7 t7002: add tests for moving out-of-cone file/directory
| | |_|/ / / / / /  
| |/| | | | | | |   
* | | | | | | | |   97316a96ed Merge branch 'hx/unpack-streaming' into next
|\ \ \ \ \ \ \ \ \  
| * | | | | | | | | aaf81223f4 unpack-objects: use stream_loose_object() to unpack large objects
| * | | | | | | | | 3c3ca0b0c1 core doc: modernize core.bigFileThreshold documentation
| * | | | | | | | | 2b6070ac4c object-file.c: add "stream_loose_object()" to handle large object
| * | | | | | | | | 21e7d88140 object-file.c: factor out deflate part of write_loose_object()
| * | | | | | | | | 97a9db6ffb object-file.c: refactor write_loose_object() to several steps
| * | | | | | | | | a1bf5ca29f unpack-objects: low memory footprint for get_data() in dry_run mode
* | | | | | | | | |   b4846d0daa Merge branch 'en/merge-tree' into next
|\ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | 7260e87248 git-merge-tree.txt: add a section on potentional usage mistakes
| * | | | | | | | | | 7976721d17 merge-tree: add a --allow-unrelated-histories flag
| * | | | | | | | | | 7c48b27822 merge-tree: allow `ls-files -u` style info to be NUL terminated
| * | | | | | | | | | de90581141 merge-ort: optionally produce machine-readable output
| * | | | | | | | | | cb2607759e merge-ort: store more specific conflict information
| * | | | | | | | | | 2715e8a931 merge-ort: make `path_messages` a strmap to a string_list
| * | | | | | | | | | 6debb7527b merge-ort: store messages in a list, not in a single strbuf
| * | | | | | | | | | b520bc6caa merge-tree: provide easy access to `ls-files -u` style info
| * | | | | | | | | | 7fa3338870 merge-tree: provide a list of which files have conflicts
| * | | | | | | | | | a4040cfa35 merge-ort: remove command-line-centric submodule message from merge-ort
| * | | | | | | | | | fae26ce79c merge-ort: provide a merge_get_conflicted_files() helper function
| * | | | | | | | | | a1a7811975 merge-tree: support including merge messages in output
| * | | | | | | | | | a34edae68a merge-ort: split out a separate display_update_messages() function
| * | | | | | | | | | 1f0c3a29da merge-tree: implement real merges
| * | | | | | | | | | 6ec755a0e1 merge-tree: add option parsing and initial shell for real merge function
| * | | | | | | | | | 55e48f6bf7 merge-tree: move logic for existing merge into new function
| * | | | | | | | | | 70176b7015 merge-tree: rename merge_trees() to trivial_merge_trees()
* | | | | | | | | | |   d340a03e89 Merge branch 'gg/worktree-from-the-above' into next
|\ \ \ \ \ \ \ \ \ \ \  
| * | | | | | | | | | | 299f88a3b3 dir: minor refactoring / clean-up
| * | | | | | | | | | | 992bb2eff9 dir: traverse into repository
| | |_|_|_|/ / / / / /  
| |/| | | | | | | | |   
* | | | | | | | | | |   a06ded0851 Merge branch 'jc/resolve-undo' into next
|\ \ \ \ \ \ \ \ \ \ \  
| |/ / / / / / / / / /  
|/| | | | | | | / / /   
| | |_|_|_|_|_|/ / /    
| |/| | | | | | | |     
| * | | | | | | | | 0c2d8a5b28 revision: mark blobs needed for resolve-undo as reachable
|/ / / / / / / / /  
| o / / / / / / / f770e9f396 Git 2.37-rc2
|  / / / / / / /  
| o / / / / / / ab336e8f1c Seventh batch
|  / / / / / /  
| | o / / / / 8168d5e9c2 Git 2.37-rc0
| |  / / / /  
| | o / / / 2668e3608e Sixth batch
| |  / / /  
| | o / / 30cc8d0f14 A regression fix for 2.37
| |  / /  
| o / / e4a4b31577 Git 2.37
|  / /  
o / / 4e2a4d1dd4 The second batch
 / /  
o / 54c8a7c379 revisions API: add a TODO for diff_free(&revs->diffopt)
 /  
o 1e59178e3f Sync with 'maint'

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Junio C Hamano]

Derrick Stolee <derrickstolee@github.com> writes:

>  Before:
>
>   A---B----C---M---Q1---Q2---Q3 <-- refs/heads/Q
>    \          /
>     P1--P2--P3 <-- refs/heads/P
>
>  After rebasing both topics simultaneously (with 'git rebase --update-refs
>  C' while Q is checked out):
>
>   A---B---C---D---P1---P2---P3---Q1---Q2---Q3
> 			    ^              ^
> 			refs/heads/P  refs/heads/Q
>
> But it seems what you mean to say is to update the merge commit M, which
> means that the 'P' branch above has been updated independently of the 'Q'
> branch, so we need to update 'Q' after-the-fact.

I am not aiming to flatten P and Q into a single strand of pearls.
That would defeat the point of "git log --oneline --first-parent"
that can be used to view "master..Q", whose output would be "at the
bottom the topic P lies there, and on top there are 3 patches".

P's may be your ds/branch-checked-out topic while Q's may be this
topic.  Other people may find bugs, improvements and a room for
unwanted churns in the former, and P may gain a few more commit,
in which case M thru Q3 needs to be rebuilt.

In a manual procedure, when I realize that P will gain a few
more patches (or gets rewritten):

 * find what other topics depend on P and make a mental note (i.e. Q
   needs to be rebuilt)

 * perform an equivalent of "git rebase -i --onto A P", but without
   using "git rebase".

   - git checkout P
   - git checkout --detach master...	;# reuse the same base
   - git am -s				;# apply
   - git rebase -i                      ;# minor fix-up while queuing
   - git range-diff @{-1}...		;# sanity check
   - make test				;# further sanity check
   - git checkout -B @{-1}		;# update P to the new round
   - git range-diff @{1}...             ;# final sanity check

   this rebuilds P1, P2, and P3 into a new series on A

 * for each topic that needs rebuilding (i.e. Q), find M and rebuild
   it

   - git checkout Q
   - git checkout --detach master...	;# reuse the same base
   - git merge --into Q P		;# recreate M with updated P
   - git rebase --onto HEAD M Q		;# rebuild Q
   - git checkout -B @{-1}		;# update Q to sit on top of new P
   - git range-diff @{1}...		;# sanity check (should be empty)


> I'm not sure what that rebase would look like, indepdendent of
> updating refs.

I suspect that a creative use of "git rebase --rebase-merge master
Q" should allow me to get there.  Here is an outline of the todo
list you'd get out of "git rebase --rebase-merge -i v2.37.0" while
the topic ds/rebase-update-ref is checked out:

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 
label onto

# Branch ds-branch-checked-out
reset 5ed49a75f3 # Merge branch 'os/fetch-check-not-current-branch'
pick 31ad6b61bd branch: add branch_checked_out() helper
pick d2ba271aad branch: check for bisects and rebases
...
pick 9bef0b1e6e branch: drop unused worktrees variable
label ds-branch-checked-out

reset onto
merge -C 7fefa1b68e ds-branch-checked-out # Merge branch 'ds/branch-checked-out' into ds/rebase-update-ref
pick a0bfa0ec53 t2407: test bisect and rebase as black-boxes
pick 43547f7a52 t2407: test branches currently using apply backend
...
pick 8b2a776cab sequencer: notify user of --update-refs activity
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 

So it would likely involve

 - "git checkout Q && git rebase -i --rebase-merge master" to get
   the todo list.

 - remove all "pick"s for existing commits on branch P, and replace
   them with a "break"

 - add an update-ref insn to update ds/branch-checked-out topic
   using the ds-branch-checked-out label at the end of the todo
   list.

 - exit the editor, and in the "break" session, run "am" to accept
   the new round of patches for P.

 - "git rebase --continue" to let the tip labeled as P and let the
   rest of the todo list rebuild Q

but I am not sure what should happen when there are more than one
dependent topic (i.e. in addition to Q, topic R also depends on P).
It also is unclear in the above procedure with "rebase-merge" what
to feed to "range-diff" in the sanity-checking step.  I could type
"git range-diff P..." but being able to use @{-1} is a lot handier.

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Elijah Newren]

On Thu, Jul 14, 2022 at 11:11 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Derrick Stolee <derrickstolee@github.com> writes:
>
> >  Before:
> >
> >   A---B----C---M---Q1---Q2---Q3 <-- refs/heads/Q
> >    \          /
> >     P1--P2--P3 <-- refs/heads/P
> >
> >  After rebasing both topics simultaneously (with 'git rebase --update-refs
> >  C' while Q is checked out):
> >
> >   A---B---C---D---P1---P2---P3---Q1---Q2---Q3
> >                           ^              ^
> >                       refs/heads/P  refs/heads/Q
> >
> > But it seems what you mean to say is to update the merge commit M, which
> > means that the 'P' branch above has been updated independently of the 'Q'
> > branch, so we need to update 'Q' after-the-fact.
>
> I am not aiming to flatten P and Q into a single strand of pearls.
> That would defeat the point of "git log --oneline --first-parent"
> that can be used to view "master..Q", whose output would be "at the
> bottom the topic P lies there, and on top there are 3 patches".
>
> P's may be your ds/branch-checked-out topic while Q's may be this
> topic.  Other people may find bugs, improvements and a room for
> unwanted churns in the former, and P may gain a few more commit,
> in which case M thru Q3 needs to be rebuilt.
>
> In a manual procedure, when I realize that P will gain a few
> more patches (or gets rewritten):
>
>  * find what other topics depend on P and make a mental note (i.e. Q
>    needs to be rebuilt)
>
>  * perform an equivalent of "git rebase -i --onto A P", but without
>    using "git rebase".
>
>    - git checkout P
>    - git checkout --detach master...    ;# reuse the same base
>    - git am -s                          ;# apply
>    - git rebase -i                      ;# minor fix-up while queuing
>    - git range-diff @{-1}...            ;# sanity check
>    - make test                          ;# further sanity check
>    - git checkout -B @{-1}              ;# update P to the new round
>    - git range-diff @{1}...             ;# final sanity check
>
>    this rebuilds P1, P2, and P3 into a new series on A
>
>  * for each topic that needs rebuilding (i.e. Q), find M and rebuild
>    it
>
>    - git checkout Q
>    - git checkout --detach master...    ;# reuse the same base
>    - git merge --into Q P               ;# recreate M with updated P
>    - git rebase --onto HEAD M Q         ;# rebuild Q
>    - git checkout -B @{-1}              ;# update Q to sit on top of new P
>    - git range-diff @{1}...             ;# sanity check (should be empty)

Thanks for sharing all these details, and the ones below.

> > I'm not sure what that rebase would look like, indepdendent of
> > updating refs.
>
> I suspect that a creative use of "git rebase --rebase-merge master
> Q" should allow me to get there.  Here is an outline of the todo
> list you'd get out of "git rebase --rebase-merge -i v2.37.0" while
> the topic ds/rebase-update-ref is checked out:
>
> ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
> label onto
>
> # Branch ds-branch-checked-out
> reset 5ed49a75f3 # Merge branch 'os/fetch-check-not-current-branch'
> pick 31ad6b61bd branch: add branch_checked_out() helper
> pick d2ba271aad branch: check for bisects and rebases
> ...
> pick 9bef0b1e6e branch: drop unused worktrees variable
> label ds-branch-checked-out
>
> reset onto
> merge -C 7fefa1b68e ds-branch-checked-out # Merge branch 'ds/branch-checked-out' into ds/rebase-update-ref
> pick a0bfa0ec53 t2407: test bisect and rebase as black-boxes
> pick 43547f7a52 t2407: test branches currently using apply backend
> ...
> pick 8b2a776cab sequencer: notify user of --update-refs activity
> ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
>
> So it would likely involve
>
>  - "git checkout Q && git rebase -i --rebase-merge master" to get
>    the todo list.
>
>  - remove all "pick"s for existing commits on branch P, and replace
>    them with a "break"
>
>  - add an update-ref insn to update ds/branch-checked-out topic
>    using the ds-branch-checked-out label at the end of the todo
>    list.
>
>  - exit the editor, and in the "break" session, run "am" to accept
>    the new round of patches for P.
>
>  - "git rebase --continue" to let the tip labeled as P and let the
>    rest of the todo list rebuild Q
>
> but I am not sure what should happen when there are more than one
> dependent topic (i.e. in addition to Q, topic R also depends on P).

Yeah, rebase's hard-coded HEAD assumption and including it in an
implicit range, makes it hard to get more general revision range
expressions.  If we had something that just let you specify your
revision range, then you could handle multiple topics, using syntax
something like:

   git replay --keep-base ^master P Q R

and then edit the resulting todo file (where --interactive and an
equivalent of --rebase-merges are both assumed).  You don't even
necessarily have to have any of P, Q, or R checked out.  In fact, if
we can use revision ranges, then perhaps it'd be even nicer to run

   git replay --keep-base --contained --ancestry-path P..seen

(where --contained says update all refs pointing to any commit
contained in the revision range, instead of just the explicitly listed
refs).  This would also remove all the picks for P for you, since it's
the left side of that range.  So, you'd just need to adjust the first
"reset" directive to an older commit, and add a "break" directive
right after it followed by an "update-ref" directive for P.  During
the "break", you can "git-am" the new patches to accept the new
version of P.  You wouldn't have to make a mental note of any other
branches (Q, R, S, etc.) that also need to be included, since they are
contained in seen and the --ancestry-path gets you just the right
range of commits, replaying them all including merges (and I've got
plans for improving the replaying of merges too).

However...

What if someone wants to edit the patches in topic P too, rather than
just eject and replace them?  There's no <revision range> I know of to
get that if the base of P is some very common commit, because
"--ancestry-path master..seen" will suddenly balloon the number of
topics involved rather dramatically.  I guess you could just take that
todo list and edit it (since most commits in the list would just be
replayed on the same base and end up fast-forwarding and thus not be
changed; only commits the user edited and their descendants would
change), but that todo list could be unworkably long  to meddle with.
I'm curious if anyone has ideas for that.

> It also is unclear in the above procedure with "rebase-merge" what
> to feed to "range-diff" in the sanity-checking step.  I could type
> "git range-diff P..." but being able to use @{-1} is a lot handier.

I think you could use "git range-diff seen@{1}...seen" after the
operation I outlined above.

Granted, that only helps sanity check the non-merge commits, since
range-diff currently ignores merges, but perhaps if we teach
range-diff to make use of --diff-merges=remerge then it could?

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Elijah Newren]

On Thu, Jul 14, 2022 at 7:50 AM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 7/12/22 11:37 AM, Junio C Hamano wrote:> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> > This is a tangent, but may serve as some food for thought.
> >
> > When I queue (or develop myself) a topic that depends on another
> > topic, I often do
> >
> >     $ git checkout --detach vX.Y.Z ;# choose an appropriate base
> >     $ git merge --into derived-topic base-topic
> >     $ develop develop (or "git am")
> >
> > which would end up in
> >
> >     vX.Y.Z -----M---A---B---C derived-topic
> >                /
> >      base-topic
> >
> > so that "git log --first-parent --oneline master.." would show the
> > commits in the topic plus the fact that it depends on which other
> > topic recorded in a single merge commit.  A topic that depends on
> > two or more topics can be handled the same way.
> >
> > One good thing about this arrangement, unlike the "totally linear"
> > one depicted at the top of your cover letter, is that it is easier
> > to rebuild each topic independently and the first-parent view is
> > still useful.  If you futz with the base topic in a totally linear
> > history, "log --decorate" of the derived topic would no longer tell
> > you where the old iteration of the base topic ended.
> >
> > It would be very nice to see the update-ref feature (or something
> > like that) makes it easy to deal with such a topology, too.
>
> Your topology is an interesting one, but --update-refs isn't limited to
> linear history.
>
> (Begin aside)
>
> One goal of this feature is to make it easier to manage the topics that
> are being juggled in friendly forks. For example, git-for-windows/git and
> microsoft/git have some topics that evolve version-to-version but might be
> cleaned up and sent upstream. Both of these forks use 'git rebase
> --rebase-merges=rebase-cousins' when consuming new upstream versions to
> keep the merge structure of these new topics. However, we lose the branch
> names and need to reconstruct them from the context of the merge commits.
>
> With --update-refs, we can automatically rewrite the branches that are
> included in these individual topics. That might make it simpler to extract
> a series to send upstream.
>
> One test in this series does test such a case with the --rebase-merges
> option.
>
> (End aside)
>
> Back to your topology, I wonder what your rebase command looks like when
> tracking those topics.
>
> The goal of --update-refs is to help rebase multiple branches at the same
> time, and with your example here, it would imply you want to rebase both
> dependent topics.
>
>  Before:
>
>   A---B----C---M---Q1---Q2---Q3 <-- refs/heads/Q
>    \          /
>     P1--P2--P3 <-- refs/heads/P
>
>  After rebasing both topics simultaneously (with 'git rebase --update-refs
>  C' while Q is checked out):
>
>   A---B---C---D---P1---P2---P3---Q1---Q2---Q3
>                             ^              ^
>                         refs/heads/P  refs/heads/Q
>
> But it seems what you mean to say is to update the merge commit M, which
> means that the 'P' branch above has been updated independently of the 'Q'
> branch, so we need to update 'Q' after-the-fact. I'm not sure what that
> rebase would look like, indepdendent of updating refs.

Since this is an aside, I'll take a chance to talk about stuff I'm
working on.  After

  git replay --onto main --contained main..Q

(where "--contained" seems to be similar to rebase's --update-refs
option, in that it assumes all refs pointing to history being
rewritten should also be updated), or the more explicit

  git replay --onto main ^main P Q

and assuming 'main' has one additional commit 'D' on top of 'C', you'd see

   A---B---C---D-----------M---Q1---Q2---Q3 <-- refs/heads/Q
               \          /
                P1--P2--P3 <-- refs/heads/P

In other words, both P and Q would be replayed but with relative
topology preserved.  Also, important changes in M (conflict fixups or
semantic fixes, etc.) would be preserved -- at least that's the plan.

> Do you have an example rebase command that manipulates the commits the
> way you want? Then I can better understand how the --update-refs could fit
> in with that. (Or maybe the point of your tangent is that there isn't an
> option.)

This would be nice.

> If instead we thought about an example like re-rolling the 'next' branch
> entirely on top of the 'master' branch, then we have an example that is
> closer to the friendly fork example. (I know this isn't a realistic
> scenario since we don't rewrite the commits already merged to 'next', but
> it's an interesting stress test.)
>
> While having 'next' checked out, I ran
>
>   git rebase -i --update-refs --rebase-merges=rebase-cousins master

Re-rolling 'next' might not be realistic, but re-rolling 'seen' isn't
quite as far fetched.  I think it'd be more likely to use
no-rebase-cousins, so:

    git checkout seen && git rebase -i --update-refs
[--rebase-merges=no-rebase-cousins] next

or

    git replay -i --contained --keep-base next..seen

> and it updated all of the branches currently in the region master..next.
> I've attached the output of
>
>   git -c log.excludeDecoration="refs/remotes/*" \
>         log --oneline --graph --boundary master..next
>
> just to show what this looks like.

One example that I thought might be useful, is tweaking a single topic
in 'seen', and updating 'seen' to reflect those updates.  This would
be something like

    git replay --keep-base --first-parent ^next seen topic_branch

Which would show you all the commits in the first-line history of
next..topic_branch plus the first-line history of next..seen, and let
you tweak that todo list.

Re: [PATCH v4 00/12] rebase: update branches in multi-part topic

[Phillip Wood]

On 12/07/2022 14:06, Derrick Stolee via GitGitGadget wrote:

I haven't looked at the first two patches but I think I read everything 
else that has changed in V4. It all looks good to me. I've left some 
comments on the later patches but I don't think there are any 
showstoppers and this version could be merged into next.

Thanks for working on this

Phillip

> This series is based on ds/branch-checked-out.
> 
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
> 
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'update-ref' commands into the
> TODO list. This is powered by the commit decoration machinery.
> 
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
> 
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> update-ref refs/heads/bundle-redo/fetch
> 
> pick 1e3f6546632 clone: add --bundle-uri option
> pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
> update-ref refs/heads/bundle-redo/clone
> 
> pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
> pick 3029c3aca15 bundle-uri: create base key-value pair parsing
> pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
> pick 92625a47673 bundle-uri: unit test "key=value" parsing
> pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
> pick 9d6809a8d53 bundle-uri: parse bundle list in config format
> pick 287a732b54c bundle-uri: fetch a list of bundles
> update-ref refs/heads/bundle-redo/list
> 
> pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
> pick 520204dcd1c bundle-uri client: add minimal NOOP client
> pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
> pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
> pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
> pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
> pick df255000b7e bundle-uri: download bundles from an advertised list
> pick d71beabf199 clone: unbundle the advertised bundles
> pick c9578391976 t5601: basic bundle URI tests
> # Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'
> 
> update-ref refs/heads/bundle-redo/advertise
> 
> 
> Here is an outline of the series:
> 
>   * Patch 1 updates some tests for branch_checked_out() to use 'git bisect'
>     and 'git rebase' as black-boxes instead of manually editing files inside
>     $GIT_DIR. (Thanks, Junio!)
>   * Patch 2 updates some tests for branch_checked_out() for the 'apply'
>     backend.
>   * Patch 3 updates branch_checked_out() to parse the
>     rebase-merge/update-refs file to block concurrent ref updates and
>     checkouts on branches selected by --update-refs.
>   * Patch 4 updates the todo list documentation to remove some unnecessary
>     dots in the 'merge' command. This makes it consistent with the 'fixup'
>     command before we document the 'update-ref' command.
>   * Patch 5 updates the definition of todo_command_info to use enum values as
>     array indices.
>   * Patches 6-8 implement the --update-refs logic itself.
>   * Patch 9 specifically updates the update-refs file every time the user
>     edits the todo-list (Thanks Phillip!)
>   * Patch 10 adds the rebase.updateRefs config option similar to
>     rebase.autoSquash.
>   * Patch 11 ignores the HEAD ref when creating the todo list instead of
>     making a comment (Thanks Elijah!)
>   * Patch 12 adds messaging to the end of the rebase stating which refs were
>     updated (Thanks Elijah!)
> 
> 
> Updates in v4
> =============
> 
> This version took longer than I'd hoped (I had less time to work on it than
> anticipated) but it also has some major updates. These major updates are
> direct responses to the significant review this series has received. Thank
> you!
> 
>   * The update-refs file now stores "ref/before/after" triples (still
>     separated by lines). This allows us to store the "before" OID of a ref in
>     addition to the "after" that we will write to that ref at the end of the
>     rebase. This allows us to do a "force-with-lease" update. The
>     branch_checked_out() updates should prevent Git from updating those refs
>     while under the rebase, but older versions and third-party tools don't
>     have that protection.
>   * The update-refs file is updated with every update to the todo-list file.
>     This allows for some advanced changes to the file, including removing,
>     adding, and duplicating 'update-ref' commands.
>   * The message at the end of the rebase process now lists which refs were
>     updated with the update-ref steps. This includes any ref updates that
>     fail.
>   * The branch_checked_out() tests now use 'git bisect' and 'git rebase' as
>     black-boxes instead of testing their internals directly.
> 
> Here are the more minor updates:
> 
>   * Dropped an unnecessary stat() call.
>   * Updated commit messages to include extra details, based on confusion in
>     last round.
>   * The HEAD branch no longer appears as a comment line in the initial todo
>     list.
>   * The update-refs file is now written using a lockfile.
>   * Tests now use test_cmp_rev.
>   * A memory leak ('path' variable) is resolved.
> 
> 
> Updates in v3
> =============
> 
>   * The branch_checked_out() API was extracted to its own topic and is now
>     the ds/branch-checked-out branch. This series is now based on that one.
>   * The for_each_decoration() API was removed, since it became trivial once
>     it did not take a commit directly.
>   * The branch_checked_out() tests did not verify the rebase-apply data (for
>     the apply backend), so that is fixed.
>   * Instead of using the 'label' command and a final 'update-refs' command in
>     the todo list, use a new 'update-ref ' command. This command updates the
>     rebase-merge/update-refs file with the OID of HEAD at these steps. At the
>     very end of the rebase sequence, those refs are updated to the stored OID
>     values (assuming that they were not removed by the user, in which case we
>     notice that the OID is the null OID and we do nothing).
>   * New tests are added.
>   * The todo-list comment documentation has some new formatting updates, but
>     also includes a description of 'update-refs' in this version.
> 
> 
> Updates in v2
> =============
> 
> As recommended by the excellent feedback, I have removed the 'exec' commands
> in favor of the 'label' commands and a new 'update-refs' command at the very
> end. This way, there is only one step that updates all of the refs at the
> end instead of updating refs during the rebase. If a user runs 'git rebase
> --abort' in the middle, then their refs are still where they need to be.
> 
> Based on some of the discussion, it seemed like one way to do this would be
> to have an 'update-ref ' command that would take the place of these 'label'
> commands. However, this would require two things that make it a bit awkward:
> 
>   1. We would need to replicate the storage of those positions during the
>      rebase. 'label' already does this pretty well. I've added the
>      "for-update-refs/" label to help here.
>   2. If we want to close out all of the refs as the rebase is finishing, then
>      that "step" becomes invisible to the user (and a bit more complicated to
>      insert). Thus, the 'update-refs' step performs this action. If the user
>      wants to do things after that step, then they can do so by editing the
>      TODO list.
> 
> Other updates:
> 
>   * The 'keep_decorations' parameter was renamed to 'update_refs'.
>   * I added tests for --rebase-merges=rebase-cousins to show how these labels
>     interact with other labels and merge commands.
>   * I changed the order of the insertion of these update-refs labels to be
>     before the fixups are rearranged. This fixes a bug where the tip commit
>     is a fixup! so its decorations are never inspected (and they would be in
>     the wrong place even if they were). The fixup! commands are properly
>     inserted between a pick and its following label command. Tests
>     demonstrate this is correct.
>   * Numerous style choices are updated based on feedback.
> 
> Thank you for all of the detailed review and ideas in this space. I
> appreciate any more ideas that can make this feature as effective as it can
> be.
> 
> Thanks, -Stolee
> 
> Derrick Stolee (12):
>    t2407: test bisect and rebase as black-boxes
>    t2407: test branches currently using apply backend
>    branch: consider refs under 'update-refs'
>    rebase-interactive: update 'merge' description
>    sequencer: define array with enum values
>    sequencer: add update-ref command
>    rebase: add --update-refs option
>    rebase: update refs from 'update-ref' commands
>    sequencer: rewrite update-refs as user edits todo list
>    rebase: add rebase.updateRefs config option
>    sequencer: ignore HEAD ref under --update-refs
>    sequencer: notify user of --update-refs activity
> 
>   Documentation/config/rebase.txt |   3 +
>   Documentation/git-rebase.txt    |  11 +
>   branch.c                        |  13 +
>   builtin/rebase.c                |  10 +
>   rebase-interactive.c            |  15 +-
>   sequencer.c                     | 469 +++++++++++++++++++++++++++++++-
>   sequencer.h                     |  23 ++
>   t/lib-rebase.sh                 |  15 +
>   t/t2407-worktree-heads.sh       | 103 +++++--
>   t/t3404-rebase-interactive.sh   | 269 ++++++++++++++++++
>   10 files changed, 887 insertions(+), 44 deletions(-)
> 
> 
> base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v4
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v4
> Pull-Request: https://github.com/gitgitgadget/git/pull/1247
> 
> Range-diff vs v3:
> 
>    -:  ----------- >  1:  9e53a27017a t2407: test bisect and rebase as black-boxes
>    1:  fbaedc7f1f0 !  2:  540a3be256f t2407: test branches currently using apply backend
>       @@ Commit message
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>         ## t/t2407-worktree-heads.sh ##
>       -@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite: worktree in bisect' '
>       - 	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
>       +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
>       + 	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
>         '
>         
>       --test_expect_success 'refuse to overwrite: worktree in rebase' '
>       -+test_expect_success 'refuse to overwrite: worktree in rebase (apply)' '
>       -+	test_when_finished rm -rf .git/worktrees/wt-*/rebase-apply &&
>       +-test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
>       ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (apply)' '
>       ++	test_when_finished git -C wt-2 rebase --abort &&
>        +
>       -+	mkdir -p .git/worktrees/wt-3/rebase-apply &&
>       -+	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-apply/head-name &&
>       -+	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-apply/onto &&
>       ++	# This will fail part-way through due to a conflict.
>       ++	test_must_fail git -C wt-2 rebase --apply conflict-2 &&
>        +
>       -+	test_must_fail git branch -f fake-1 HEAD 2>err &&
>       -+	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
>       ++	test_must_fail git branch -f wt-2 HEAD 2>err &&
>       ++	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
>        +'
>        +
>       -+test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
>       - 	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
>       ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (merge)' '
>       + 	test_when_finished git -C wt-2 rebase --abort &&
>         
>       - 	mkdir -p .git/worktrees/wt-3/rebase-merge &&
>       + 	# This will fail part-way through due to a conflict.
>    2:  2bc647b6fcd !  3:  bf301a054e3 branch: consider refs under 'update-refs'
>       @@ Commit message
>            branch_checked_out().
>        
>            The data store is a plaintext file inside the 'rebase-merge' directory
>       -    for that worktree. The file alternates refnames and OIDs. The OIDs will
>       -    be used to store the to-be-written values as the rebase progresses, but
>       -    can be ignored at the moment.
>       +    for that worktree. The file lists refnames followed by two OIDs, each on
>       +    separate lines. The OIDs will be used to store the original values of
>       +    the refs and the to-be-written values as the rebase progresses, but can
>       +    be ignored at the moment.
>        
>            Create a new sequencer_get_update_refs_state() method that parses this
>            file and populates a struct string_list with the ref-OID pairs. We can
>       @@ Commit message
>            method.
>        
>            We can test that this works without having Git write this file by
>       -    artificially creating one in our test script.
>       +    artificially creating one in our test script, at least until 'git rebase
>       +    --update-refs' is implemented and we can use it directly.
>        
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>       @@ branch.c: static void prepare_checked_out_branches(void)
>         	free_worktrees(worktrees);
>        
>         ## sequencer.c ##
>       +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>       +  */
>       + static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>       +
>       ++/*
>       ++ * The update-refs file stores a list of refs that will be updated at the end
>       ++ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
>       ++ * update the OIDs for the refs in this file, but the refs are not updated
>       ++ * until the end of the rebase sequence.
>       ++ *
>       ++ * rebase_path_update_refs() returns the path to this file for a given
>       ++ * worktree directory. For the current worktree, pass the_repository->gitdir.
>       ++ */
>       ++static char *rebase_path_update_refs(const char *wt_dir)
>       ++{
>       ++	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
>       ++}
>       ++
>       + /*
>       +  * The following files are written by git-rebase just after parsing the
>       +  * command-line.
>       +@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_no_reschedule_failed_exec, "rebase-merge/no-res
>       + static GIT_PATH_FUNC(rebase_path_drop_redundant_commits, "rebase-merge/drop_redundant_commits")
>       + static GIT_PATH_FUNC(rebase_path_keep_redundant_commits, "rebase-merge/keep_redundant_commits")
>       +
>       ++/**
>       ++ * A 'struct update_refs_record' represents a value in the update-refs
>       ++ * list. We use a string_list to map refs to these (before, after) pairs.
>       ++ */
>       ++struct update_ref_record {
>       ++	struct object_id before;
>       ++	struct object_id after;
>       ++};
>       ++
>       + static int git_sequencer_config(const char *k, const char *v, void *cb)
>       + {
>       + 	struct replay_opts *opts = cb;
>        @@ sequencer.c: int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
>         
>         	return 0;
>       @@ sequencer.c: int sequencer_determine_whence(struct repository *r, enum commit_wh
>        +				    struct string_list *refs)
>        +{
>        +	int result = 0;
>       -+	struct stat st;
>        +	FILE *fp = NULL;
>        +	struct strbuf ref = STRBUF_INIT;
>        +	struct strbuf hash = STRBUF_INIT;
>       -+	char *path = xstrfmt("%s/rebase-merge/update-refs", wt_dir);
>       ++	struct update_ref_record *rec = NULL;
>        +
>       -+	if (stat(path, &st))
>       -+		goto cleanup;
>       ++	char *path = rebase_path_update_refs(wt_dir);
>        +
>        +	fp = fopen(path, "r");
>        +	if (!fp)
>        +		goto cleanup;
>        +
>        +	while (strbuf_getline(&ref, fp) != EOF) {
>       -+		struct object_id oid;
>        +		struct string_list_item *item;
>        +
>       ++		CALLOC_ARRAY(rec, 1);
>       ++
>        +		if (strbuf_getline(&hash, fp) == EOF ||
>       -+		    get_oid_hex(hash.buf, &oid)) {
>       ++		    get_oid_hex(hash.buf, &rec->before)) {
>        +			warning(_("update-refs file at '%s' is invalid"),
>        +				  path);
>        +			result = -1;
>        +			goto cleanup;
>        +		}
>        +
>       -+		item = string_list_append(refs, ref.buf);
>       -+		item->util = oiddup(&oid);
>       ++		if (strbuf_getline(&hash, fp) == EOF ||
>       ++		    get_oid_hex(hash.buf, &rec->after)) {
>       ++			warning(_("update-refs file at '%s' is invalid"),
>       ++				  path);
>       ++			result = -1;
>       ++			goto cleanup;
>       ++		}
>       ++
>       ++		item = string_list_insert(refs, ref.buf);
>       ++		item->util = rec;
>       ++		rec = NULL;
>        +	}
>        +
>        +cleanup:
>        +	if (fp)
>        +		fclose(fp);
>        +	free(path);
>       ++	free(rec);
>        +	strbuf_release(&ref);
>        +	strbuf_release(&hash);
>        +	return result;
>       @@ sequencer.h: void sequencer_post_commit_cleanup(struct repository *r, int verbos
>         #endif /* SEQUENCER_H */
>        
>         ## t/t2407-worktree-heads.sh ##
>       -@@ t/t2407-worktree-heads.sh: TEST_PASSES_SANITIZE_LEAK=true
>       +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
>       + 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
>       + '
>         
>       - test_expect_success 'setup' '
>       - 	test_commit init &&
>       --	git branch -f fake-1 &&
>       --	git branch -f fake-2 &&
>       ++test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
>       ++	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
>       ++
>       ++	mkdir -p .git/worktrees/wt-3/rebase-merge &&
>       ++	touch .git/worktrees/wt-3/rebase-merge/interactive &&
>        +
>       -+	for i in 1 2 3 4
>       -+	do
>       -+		git branch -f fake-$i || return 1
>       -+	done &&
>       -
>       - 	for i in 1 2 3 4
>       - 	do
>       -@@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite: worktree in rebase (merge)' '
>       - 	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-merge/head-name &&
>       - 	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-merge/onto &&
>       -
>       --	test_must_fail git branch -f fake-1 HEAD 2>err &&
>       --	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
>        +	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
>        +	refs/heads/fake-3
>        +	$(git rev-parse HEAD~1)
>       ++	$(git rev-parse HEAD)
>        +	refs/heads/fake-4
>        +	$(git rev-parse HEAD)
>       ++	$(git rev-parse HEAD)
>        +	EOF
>        +
>       -+	for i in 1 3 4
>       ++	for i in 3 4
>        +	do
>        +		test_must_fail git branch -f fake-$i HEAD 2>err &&
>        +		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
>        +			return 1
>        +	done
>       - '
>       -
>       ++'
>       ++
>         test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
>       + 	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
>       + 	grep "refusing to fetch into branch '\''refs/heads/wt-3'\''" err &&
>    3:  669f4abd59e !  4:  dec95681d2b rebase-interactive: update 'merge' description
>       @@ Commit message
>            The 'merge' command description for the todo list documentation in an
>            interactive rebase has multiple lines. The lines other than the first
>            one start with dots ('.') while the similar multi-line documentation for
>       -    'fixup' does not.
>       +    'fixup' does not. This description only appears in the comment text of
>       +    the todo file during an interactive rebase.
>        
>            The 'merge' command was documented when interactive rebase was first
>            ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
>    4:  6528a50343f =  5:  b2c09600918 sequencer: define array with enum values
>    5:  e95ad41d355 =  6:  fa7ecb718cf sequencer: add update-ref command
>    6:  918b398d6a2 !  7:  3ec2cc922f9 rebase: add --update-refs option
>       @@ Commit message
>            'update-ref' commands. Tests are added to ensure that these todo
>            commands are added in the correct locations.
>        
>       -    A future change will update the behavior to actually update the refs
>       -    at the end of the rebase sequence.
>       +    This change does _not_ include the actual behavior of tracking the
>       +    updated refs and writing the new ref values at the end of the rebase
>       +    process. That is deferred to a later change.
>        
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>       @@ sequencer.c: static int skip_unnecessary_picks(struct repository *r,
>        +			item->command = TODO_UPDATE_REF;
>        +			strbuf_addf(ctx->buf, "%s\n", decoration->name);
>        +
>       -+			sti = string_list_append(&ctx->refs_to_oids,
>       ++			sti = string_list_insert(&ctx->refs_to_oids,
>        +						 decoration->name);
>        +			sti->util = oiddup(the_hash_algo->null_oid);
>        +		}
>       @@ t/t2407-worktree-heads.sh: test_expect_success 'refuse to overwrite when in erro
>        +		grep "update-ref refs/heads/allow-update" todo
>        +	)
>        +'
>       ++
>       ++# This must be the last test in this file
>       ++test_expect_success '$EDITOR and friends are unchanged' '
>       ++	test_editor_unchanged
>       ++'
>        +
>         test_done
>        
>    7:  72e0481b643 !  8:  fb5f64c5201 rebase: update refs from 'update-ref' commands
>       @@ Commit message
>            interactive rebase.
>        
>            Teach Git to record the HEAD position when reaching these 'update-ref'
>       -    commands. The ref/OID pair is stored in the
>       +    commands. The ref/before/after triple is stored in the
>            $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
>            file to avoid having other processes updating the refs in that file
>            while the rebase is in progress.
>       @@ Commit message
>            Not only do we update the file when the sequencer reaches these
>            'update-ref' commands, we then update the refs themselves at the end of
>            the rebase sequence. If the rebase is aborted before this final step,
>       -    then the refs are not updated.
>       +    then the refs are not updated. The 'before' value is used to ensure that
>       +    we do not accidentally obliterate a ref that was updated concurrently
>       +    (say, by an older version of Git or a third-party tool).
>       +
>       +    Now that the 'git rebase --update-refs' command is implemented to write
>       +    to the update-refs file, we can remove the fake construction of the
>       +    update-refs file from a test in t2407-worktree-heads.sh.
>        
>            Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>        
>       @@ sequencer.c
>         
>         #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
>         
>       -@@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
>       -  */
>       - static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
>       +@@ sequencer.c: struct update_ref_record {
>       + 	struct object_id after;
>       + };
>         
>       -+/*
>       -+ * The update-refs file stores a list of refs that will be updated at the end
>       -+ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
>       -+ * update the OIDs for the refs in this file, but the refs are not updated
>       -+ * until the end of the rebase sequence.
>       -+ */
>       -+static GIT_PATH_FUNC(rebase_path_update_refs, "rebase-merge/update-refs")
>       -+
>       - /*
>       -  * The following files are written by git-rebase just after parsing the
>       -  * command-line.
>       ++static struct update_ref_record *init_update_ref_record(const char *ref)
>       ++{
>       ++	struct update_ref_record *rec = xmalloc(sizeof(*rec));
>       ++
>       ++	oidcpy(&rec->before, null_oid());
>       ++	oidcpy(&rec->after, null_oid());
>       ++
>       ++	/* This may fail, but that's fine, we will keep the null OID. */
>       ++	read_ref(ref, &rec->before);
>       ++
>       ++	return rec;
>       ++}
>       ++
>       + static int git_sequencer_config(const char *k, const char *v, void *cb)
>       + {
>       + 	struct replay_opts *opts = cb;
>        @@ sequencer.c: leave_merge:
>         	return ret;
>         }
>         
>        -static int do_update_ref(struct repository *r, const char *ref_name)
>        +static int write_update_refs_state(struct string_list *refs_to_oids)
>       -+{
>       + {
>        +	int result = 0;
>       ++	struct lock_file lock = LOCK_INIT;
>        +	FILE *fp = NULL;
>        +	struct string_list_item *item;
>       -+	char *path = xstrdup(rebase_path_update_refs());
>       ++	char *path;
>       ++
>       ++	if (!refs_to_oids->nr)
>       ++		return 0;
>       ++
>       ++	path = rebase_path_update_refs(the_repository->gitdir);
>        +
>        +	if (safe_create_leading_directories(path)) {
>        +		result = error(_("unable to create leading directories of %s"),
>       @@ sequencer.c: leave_merge:
>        +		goto cleanup;
>        +	}
>        +
>       -+	fp = fopen(path, "w");
>       ++	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
>       ++		result = error(_("another 'rebase' process appears to be running; "
>       ++				 "'%s.lock' already exists"),
>       ++			       path);
>       ++		goto cleanup;
>       ++	}
>       ++
>       ++	fp = fdopen_lock_file(&lock, "w");
>        +	if (!fp) {
>        +		result = error_errno(_("could not open '%s' for writing"), path);
>       ++		rollback_lock_file(&lock);
>        +		goto cleanup;
>        +	}
>        +
>       -+	for_each_string_list_item(item, refs_to_oids)
>       -+		fprintf(fp, "%s\n%s\n", item->string, oid_to_hex(item->util));
>       ++	for_each_string_list_item(item, refs_to_oids) {
>       ++		struct update_ref_record *rec = item->util;
>       ++		fprintf(fp, "%s\n%s\n%s\n", item->string,
>       ++			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
>       ++	}
>       ++
>       ++	result = commit_lock_file(&lock);
>        +
>        +cleanup:
>       -+	if (fp)
>       -+		fclose(fp);
>       ++	free(path);
>        +	return result;
>        +}
>        +
>        +static int do_update_ref(struct repository *r, const char *refname)
>       - {
>       ++{
>        +	struct string_list_item *item;
>        +	struct string_list list = STRING_LIST_INIT_DUP;
>       -+	int found = 0;
>        +
>        +	sequencer_get_update_refs_state(r->gitdir, &list);
>        +
>        +	for_each_string_list_item(item, &list) {
>        +		if (!strcmp(item->string, refname)) {
>       -+			struct object_id oid;
>       -+			free(item->util);
>       -+			found = 1;
>       -+
>       -+			if (!read_ref("HEAD", &oid)) {
>       -+				item->util = oiddup(&oid);
>       -+				break;
>       -+			}
>       ++			struct update_ref_record *rec = item->util;
>       ++			read_ref("HEAD", &rec->after);
>       ++			break;
>        +		}
>        +	}
>        +
>       -+	if (!found) {
>       -+		struct object_id oid;
>       -+		item = string_list_append(&list, refname);
>       -+
>       -+		if (!read_ref("HEAD", &oid))
>       -+			item->util = oiddup(&oid);
>       -+		else
>       -+			item->util = oiddup(the_hash_algo->null_oid);
>       -+	}
>       -+
>        +	write_update_refs_state(&list);
>        +	string_list_clear(&list, 1);
>         	return 0;
>       @@ sequencer.c: leave_merge:
>        +	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
>        +
>        +	for_each_string_list_item(item, &refs_to_oids) {
>       -+		struct object_id *oid_to = item->util;
>       -+		struct object_id oid_from;
>       ++		struct update_ref_record *rec = item->util;
>        +
>       -+		if (oideq(oid_to, the_hash_algo->null_oid)) {
>       ++		if (oideq(&rec->after, the_hash_algo->null_oid)) {
>        +			/*
>        +			 * Ref was not updated. User may have deleted the
>        +			 * 'update-ref' step.
>       @@ sequencer.c: leave_merge:
>        +			continue;
>        +		}
>        +
>       -+		if (read_ref(item->string, &oid_from)) {
>       -+			/*
>       -+			 * The ref does not exist. The user probably
>       -+			 * inserted a new 'update-ref' step with a new
>       -+			 * branch name.
>       -+			 */
>       -+			oidcpy(&oid_from, the_hash_algo->null_oid);
>       -+		}
>       -+
>        +		res |= refs_update_ref(refs, "rewritten during rebase",
>       -+				item->string,
>       -+				oid_to, &oid_from,
>       -+				0, UPDATE_REFS_MSG_ON_ERR);
>       ++				       item->string,
>       ++				       &rec->after, &rec->before,
>       ++				       0, UPDATE_REFS_MSG_ON_ERR);
>        +	}
>        +
>        +	string_list_clear(&refs_to_oids, 1);
>       @@ sequencer.c: cleanup_head_ref:
>         	/*
>         	 * Sequence of picks finished successfully; cleanup by
>         	 * removing the .git/sequencer directory
>       +@@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
>       +
>       + 			sti = string_list_insert(&ctx->refs_to_oids,
>       + 						 decoration->name);
>       +-			sti->util = oiddup(the_hash_algo->null_oid);
>       ++			sti->util = init_update_ref_record(decoration->name);
>       + 		}
>       +
>       + 		item->offset_in_buf = base_offset;
>        @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>         		}
>         	}
>       @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo
>         	free(todo_list->items);
>         	todo_list->items = ctx.items;
>        
>       + ## t/t2407-worktree-heads.sh ##
>       +@@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
>       + 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
>       + '
>       +
>       +-test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
>       +-	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
>       +-
>       +-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
>       +-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
>       ++test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
>       ++	test_when_finished git -C wt-3 rebase --abort &&
>       +
>       +-	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
>       +-	refs/heads/fake-3
>       +-	$(git rev-parse HEAD~1)
>       +-	$(git rev-parse HEAD)
>       +-	refs/heads/fake-4
>       +-	$(git rev-parse HEAD)
>       +-	$(git rev-parse HEAD)
>       +-	EOF
>       ++	git branch -f can-be-updated wt-3 &&
>       ++	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
>       +
>       + 	for i in 3 4
>       + 	do
>       +-		test_must_fail git branch -f fake-$i HEAD 2>err &&
>       +-		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
>       ++		test_must_fail git branch -f can-be-updated HEAD 2>err &&
>       ++		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
>       + 			return 1
>       + 	done
>       + '
>       +
>         ## t/t3404-rebase-interactive.sh ##
>        @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands with --rebase-merges' '
>         	)
>         '
>         
>       -+compare_two_refs () {
>       -+	git rev-parse $1 >expect &&
>       -+	git rev-parse $2 >actual &&
>       -+	test_cmp expect actual
>       -+}
>       -+
>        +test_expect_success '--update-refs updates refs correctly' '
>        +	git checkout -B update-refs no-conflict-branch &&
>        +	git branch -f base HEAD~4 &&
>       @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs adds commands
>        +
>        +	git rebase -i --autosquash --update-refs primary &&
>        +
>       -+	compare_two_refs HEAD~3 refs/heads/first &&
>       -+	compare_two_refs HEAD~3 refs/heads/second &&
>       -+	compare_two_refs HEAD~1 refs/heads/third &&
>       -+	compare_two_refs HEAD refs/heads/no-conflict-branch
>       ++	test_cmp_rev HEAD~3 refs/heads/first &&
>       ++	test_cmp_rev HEAD~3 refs/heads/second &&
>       ++	test_cmp_rev HEAD~1 refs/heads/third &&
>       ++	test_cmp_rev HEAD refs/heads/no-conflict-branch
>        +'
>        +
>         # This must be the last test in this file
>    -:  ----------- >  9:  29c7c76805a sequencer: rewrite update-refs as user edits todo list
>    8:  d2cfdbfc431 = 10:  c0022d07579 rebase: add rebase.updateRefs config option
>    -:  ----------- > 11:  d53b4ff2cee sequencer: ignore HEAD ref under --update-refs
>    -:  ----------- > 12:  d5cd4b49e46 sequencer: notify user of --update-refs activity
> 

[PATCH v5 00/12] rebase: update branches in multi-part topic

[Derrick Stolee via GitGitGadget]

This series is based on ds/branch-checked-out.

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'update-ref' commands into the
TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
update-ref refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
update-ref refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
update-ref refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

update-ref refs/heads/bundle-redo/advertise


Here is an outline of the series:

 * Patch 1 updates some tests for branch_checked_out() to use 'git bisect'
   and 'git rebase' as black-boxes instead of manually editing files inside
   $GIT_DIR. (Thanks, Junio!)
 * Patch 2 updates some tests for branch_checked_out() for the 'apply'
   backend.
 * Patch 3 updates branch_checked_out() to parse the
   rebase-merge/update-refs file to block concurrent ref updates and
   checkouts on branches selected by --update-refs.
 * Patch 4 updates the todo list documentation to remove some unnecessary
   dots in the 'merge' command. This makes it consistent with the 'fixup'
   command before we document the 'update-ref' command.
 * Patch 5 updates the definition of todo_command_info to use enum values as
   array indices.
 * Patches 6-8 implement the --update-refs logic itself.
 * Patch 9 specifically updates the update-refs file every time the user
   edits the todo-list (Thanks Phillip!)
 * Patch 10 adds the rebase.updateRefs config option similar to
   rebase.autoSquash.
 * Patch 11 ignores the HEAD ref when creating the todo list instead of
   making a comment (Thanks Elijah!)
 * Patch 12 adds messaging to the end of the rebase stating which refs were
   updated (Thanks Elijah!)

During review, we have identified some areas that would be good for
#leftoverbits:

 * Warn the user when they add an 'update-ref ' command but is checked out
   in another worktree.
 * The checks in patch 9 are quadratic. They could be sped up using
   hashtables.
 * Consider whether we should include an 'update-ref ' command for the HEAD
   ref, so that all refs are updated in the same way. This might help
   confused users.
 * The error message for failed ref updates could include information on the
   commit IDs that would have been used. This can help the user fix the
   situation by updating the refs manually.
 * Modify the --update-refs option from a boolean to an
   optionally-string-parameter that specifies refspecs for the 'update-ref'
   commands.


Updates in v5
=============

 * Rename 'wt_dir' to 'wt_git_dir' for clarity.
 * The documented behavior around 'fixup!' and 'squash!' commits was
   incorrect, so update the commit message, documentation, and test to
   demonstrate the actual behavior.
 * Use CALLOC_ARRAY() to be more idiomatic.
 * Be much more careful about propagating errors.
 * Commit message typo: "We an" to "We can"
 * Remove unnecessary null OID check when writing refs, since those would
   already be removed by a previous step.


Updates in v4
=============

This version took longer than I'd hoped (I had less time to work on it than
anticipated) but it also has some major updates. These major updates are
direct responses to the significant review this series has received. Thank
you!

 * The update-refs file now stores "ref/before/after" triples (still
   separated by lines). This allows us to store the "before" OID of a ref in
   addition to the "after" that we will write to that ref at the end of the
   rebase. This allows us to do a "force-with-lease" update. The
   branch_checked_out() updates should prevent Git from updating those refs
   while under the rebase, but older versions and third-party tools don't
   have that protection.
 * The update-refs file is updated with every update to the todo-list file.
   This allows for some advanced changes to the file, including removing,
   adding, and duplicating 'update-ref' commands.
 * The message at the end of the rebase process now lists which refs were
   updated with the update-ref steps. This includes any ref updates that
   fail.
 * The branch_checked_out() tests now use 'git bisect' and 'git rebase' as
   black-boxes instead of testing their internals directly.

Here are the more minor updates:

 * Dropped an unnecessary stat() call.
 * Updated commit messages to include extra details, based on confusion in
   last round.
 * The HEAD branch no longer appears as a comment line in the initial todo
   list.
 * The update-refs file is now written using a lockfile.
 * Tests now use test_cmp_rev.
 * A memory leak ('path' variable) is resolved.


Updates in v3
=============

 * The branch_checked_out() API was extracted to its own topic and is now
   the ds/branch-checked-out branch. This series is now based on that one.
 * The for_each_decoration() API was removed, since it became trivial once
   it did not take a commit directly.
 * The branch_checked_out() tests did not verify the rebase-apply data (for
   the apply backend), so that is fixed.
 * Instead of using the 'label' command and a final 'update-refs' command in
   the todo list, use a new 'update-ref ' command. This command updates the
   rebase-merge/update-refs file with the OID of HEAD at these steps. At the
   very end of the rebase sequence, those refs are updated to the stored OID
   values (assuming that they were not removed by the user, in which case we
   notice that the OID is the null OID and we do nothing).
 * New tests are added.
 * The todo-list comment documentation has some new formatting updates, but
   also includes a description of 'update-refs' in this version.


Updates in v2
=============

As recommended by the excellent feedback, I have removed the 'exec' commands
in favor of the 'label' commands and a new 'update-refs' command at the very
end. This way, there is only one step that updates all of the refs at the
end instead of updating refs during the rebase. If a user runs 'git rebase
--abort' in the middle, then their refs are still where they need to be.

Based on some of the discussion, it seemed like one way to do this would be
to have an 'update-ref ' command that would take the place of these 'label'
commands. However, this would require two things that make it a bit awkward:

 1. We would need to replicate the storage of those positions during the
    rebase. 'label' already does this pretty well. I've added the
    "for-update-refs/" label to help here.
 2. If we want to close out all of the refs as the rebase is finishing, then
    that "step" becomes invisible to the user (and a bit more complicated to
    insert). Thus, the 'update-refs' step performs this action. If the user
    wants to do things after that step, then they can do so by editing the
    TODO list.

Other updates:

 * The 'keep_decorations' parameter was renamed to 'update_refs'.
 * I added tests for --rebase-merges=rebase-cousins to show how these labels
   interact with other labels and merge commands.
 * I changed the order of the insertion of these update-refs labels to be
   before the fixups are rearranged. This fixes a bug where the tip commit
   is a fixup! so its decorations are never inspected (and they would be in
   the wrong place even if they were). The fixup! commands are properly
   inserted between a pick and its following label command. Tests
   demonstrate this is correct.
 * Numerous style choices are updated based on feedback.

Thank you for all of the detailed review and ideas in this space. I
appreciate any more ideas that can make this feature as effective as it can
be.

Thanks, -Stolee

Derrick Stolee (12):
  t2407: test bisect and rebase as black-boxes
  t2407: test branches currently using apply backend
  branch: consider refs under 'update-refs'
  rebase-interactive: update 'merge' description
  sequencer: define array with enum values
  sequencer: add update-ref command
  rebase: add --update-refs option
  rebase: update refs from 'update-ref' commands
  sequencer: rewrite update-refs as user edits todo list
  rebase: add rebase.updateRefs config option
  sequencer: ignore HEAD ref under --update-refs
  sequencer: notify user of --update-refs activity

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  10 +
 branch.c                        |  13 +
 builtin/rebase.c                |  10 +
 rebase-interactive.c            |  15 +-
 sequencer.c                     | 474 +++++++++++++++++++++++++++++++-
 sequencer.h                     |  23 ++
 t/lib-rebase.sh                 |  15 +
 t/t2407-worktree-heads.sh       | 103 +++++--
 t/t3404-rebase-interactive.sh   | 273 ++++++++++++++++++
 10 files changed, 895 insertions(+), 44 deletions(-)


base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1247

Range-diff vs v4:

  1:  9e53a27017a =  1:  9e53a27017a t2407: test bisect and rebase as black-boxes
  2:  540a3be256f =  2:  540a3be256f t2407: test branches currently using apply backend
  3:  bf301a054e3 !  3:  1089a0edb73 branch: consider refs under 'update-refs'
     @@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-
      + * rebase_path_update_refs() returns the path to this file for a given
      + * worktree directory. For the current worktree, pass the_repository->gitdir.
      + */
     -+static char *rebase_path_update_refs(const char *wt_dir)
     ++static char *rebase_path_update_refs(const char *wt_git_dir)
      +{
     -+	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
     ++	return xstrfmt("%s/rebase-merge/update-refs", wt_git_dir);
      +}
      +
       /*
  4:  dec95681d2b =  4:  d1cce4f06aa rebase-interactive: update 'merge' description
  5:  b2c09600918 =  5:  4c086d477f0 sequencer: define array with enum values
  6:  fa7ecb718cf =  6:  7b3d6601960 sequencer: add update-ref command
  7:  3ec2cc922f9 !  7:  7efb55e4f14 rebase: add --update-refs option
     @@ Commit message
          --fixup commit at the tip of the feature to apply correctly to the sub
          branch, even if it is fixing up the most-recent commit in that part.
      
     -    One potential problem here is that refs decorating commits that are
     -    already marked as "fixup!" or "squash!" will not be included in this
     -    list. Generally, the reordering of the "fixup!" and "squash!" is likely
     -    to change the relative order of these refs, so it is not recommended.
     -    The workflow here is intended to allow these kinds of commits at the tip
     -    of the rebased branch while the other sub branches come along for the
     -    ride without intervention.
     -
          This change update the documentation and builtin to accept the
          --update-refs option as well as updating the todo file with the
          'update-ref' commands. Tests are added to ensure that these todo
     @@ Documentation/git-rebase.txt: provided. Otherwise an explicit `--no-reschedule-f
      +--no-update-refs::
      +	Automatically force-update any branches that point to commits that
      +	are being rebased. Any branches that are checked out in a worktree
     -+	or point to a `squash! ...` or `fixup! ...` commit are not updated
     -+	in this way.
     ++	are not updated in this way.
      +
       INCOMPATIBLE OPTIONS
       --------------------
     @@ builtin/rebase.c: int cmd_rebase(int argc, const char **argv, const char *prefix
       			 N_("move commits that begin with "
       			    "squash!/fixup! under -i")),
      +		OPT_BOOL(0, "update-refs", &options.update_refs,
     -+			 N_("update local refs that point to commits "
     ++			 N_("update branches that point to commits "
      +			    "that are being rebased")),
       		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
       			N_("GPG-sign commits"),
     @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
      +	git branch -f second HEAD~3 &&
      +	git branch -f third HEAD~1 &&
      +	git commit --allow-empty --fixup=third &&
     ++	git branch -f is-not-reordered &&
     ++	git commit --allow-empty --fixup=HEAD~4 &&
      +	git branch -f shared-tip &&
      +	(
      +		set_cat_todo_editor &&
      +
      +		cat >expect <<-EOF &&
      +		pick $(git log -1 --format=%h J) J
     ++		fixup $(git log -1 --format=%h update-refs) fixup! J # empty
      +		update-ref refs/heads/second
      +		update-ref refs/heads/first
      +		pick $(git log -1 --format=%h K) K
      +		pick $(git log -1 --format=%h L) L
     -+		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
     ++		fixup $(git log -1 --format=%h is-not-reordered) fixup! L # empty
      +		update-ref refs/heads/third
      +		pick $(git log -1 --format=%h M) M
      +		update-ref refs/heads/no-conflict-branch
     ++		update-ref refs/heads/is-not-reordered
      +		update-ref refs/heads/shared-tip
      +		EOF
      +
  8:  fb5f64c5201 !  8:  e7a91bdffbd rebase: update refs from 'update-ref' commands
     @@ sequencer.c: struct update_ref_record {
       
      +static struct update_ref_record *init_update_ref_record(const char *ref)
      +{
     -+	struct update_ref_record *rec = xmalloc(sizeof(*rec));
     ++	struct update_ref_record *rec;
     ++
     ++	CALLOC_ARRAY(rec, 1);
      +
      +	oidcpy(&rec->before, null_oid());
      +	oidcpy(&rec->after, null_oid());
     @@ sequencer.c: leave_merge:
       
      -static int do_update_ref(struct repository *r, const char *ref_name)
      +static int write_update_refs_state(struct string_list *refs_to_oids)
     - {
     ++{
      +	int result = 0;
      +	struct lock_file lock = LOCK_INIT;
      +	FILE *fp = NULL;
     @@ sequencer.c: leave_merge:
      +}
      +
      +static int do_update_ref(struct repository *r, const char *refname)
     -+{
     + {
      +	struct string_list_item *item;
      +	struct string_list list = STRING_LIST_INIT_DUP;
      +
     -+	sequencer_get_update_refs_state(r->gitdir, &list);
     ++	if (sequencer_get_update_refs_state(r->gitdir, &list))
     ++		return -1;
      +
      +	for_each_string_list_item(item, &list) {
      +		if (!strcmp(item->string, refname)) {
      +			struct update_ref_record *rec = item->util;
     -+			read_ref("HEAD", &rec->after);
     ++			if (read_ref("HEAD", &rec->after))
     ++				return -1;
      +			break;
      +		}
      +	}
     @@ sequencer.c: leave_merge:
      +	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
      +	struct ref_store *refs = get_main_ref_store(r);
      +
     -+	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
     ++	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
     ++		return res;
      +
      +	for_each_string_list_item(item, &refs_to_oids) {
      +		struct update_ref_record *rec = item->util;
      +
     -+		if (oideq(&rec->after, the_hash_algo->null_oid)) {
     -+			/*
     -+			 * Ref was not updated. User may have deleted the
     -+			 * 'update-ref' step.
     -+			 */
     -+			continue;
     -+		}
     -+
      +		res |= refs_update_ref(refs, "rewritten during rebase",
      +				       item->string,
      +				       &rec->after, &rec->before,
     @@ sequencer.c: leave_merge:
       {
       	int i = todo_list->current;
      @@ sequencer.c: cleanup_head_ref:
     + 
     + 		strbuf_release(&buf);
       		strbuf_release(&head_ref);
     ++
     ++		if (do_update_refs(r))
     ++			return -1;
       	}
       
     -+	do_update_refs(r);
     -+
       	/*
     - 	 * Sequence of picks finished successfully; cleanup by
     - 	 * removing the .git/sequencer directory
      @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
       
       			sti = string_list_insert(&ctx->refs_to_oids,
     @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
       		}
       
       		item->offset_in_buf = base_offset;
     +@@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
     +  */
     + static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
     + {
     +-	int i;
     ++	int i, res;
     + 	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
     + 	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
     + 	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
      @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
       		}
       	}
       
     -+	write_update_refs_state(&ctx.refs_to_oids);
     ++	res = write_update_refs_state(&ctx.refs_to_oids);
      +
       	string_list_clear(&ctx.refs_to_oids, 1);
     ++
     ++	if (res) {
     ++		/* we failed, so clean up the new list. */
     ++		free(ctx.items);
     ++		return res;
     ++	}
     ++
       	free(todo_list->items);
       	todo_list->items = ctx.items;
     + 	todo_list->nr = ctx.items_nr;
      
       ## t/t2407-worktree-heads.sh ##
      @@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
  9:  29c7c76805a !  9:  95e2bbcedb1 sequencer: rewrite update-refs as user edits todo list
     @@ Commit message
      
          We can test that this works by rewriting the todo-list several times in
          the course of a rebase. Check that each ref is locked or unlocked for
     -    updates after each todo-list update. We an also verify that the ref
     +    updates after each todo-list update. We can also verify that the ref
          update fails if a concurrent process updates one of the refs after the
          rebase process records the "locked" ref location.
      
 10:  c0022d07579 ! 10:  a73b02568f3 rebase: add rebase.updateRefs config option
     @@ Documentation/config/rebase.txt: rebase.autoStash::
      
       ## Documentation/git-rebase.txt ##
      @@ Documentation/git-rebase.txt: start would be overridden by the presence of
     + 	Automatically force-update any branches that point to commits that
       	are being rebased. Any branches that are checked out in a worktree
     - 	or point to a `squash! ...` or `fixup! ...` commit are not updated
     - 	in this way.
     + 	are not updated in this way.
      ++
      +If the configuration variable `rebase.updateRefs` is set, then this option
      +can be used to override and disable this setting.
 11:  d53b4ff2cee = 11:  2a6577974c7 sequencer: ignore HEAD ref under --update-refs
 12:  d5cd4b49e46 ! 12:  ec080ce1e90 sequencer: notify user of --update-refs activity
     @@ sequencer.c: static int do_update_ref(struct repository *r, const char *refname)
      +	struct strbuf update_msg = STRBUF_INIT;
      +	struct strbuf error_msg = STRBUF_INIT;
       
     - 	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
     + 	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
     + 		return res;
       
       	for_each_string_list_item(item, &refs_to_oids) {
       		struct update_ref_record *rec = item->util;
      +		int loop_res;
       
     - 		if (oideq(&rec->after, the_hash_algo->null_oid)) {
     - 			/*
     -@@ sequencer.c: static int do_update_refs(struct repository *r)
     - 			continue;
     - 		}
     - 
      -		res |= refs_update_ref(refs, "rewritten during rebase",
      -				       item->string,
      -				       &rec->after, &rec->before,
     @@ sequencer.c: static int do_update_refs(struct repository *r)
       }
       
      @@ sequencer.c: cleanup_head_ref:
     + 		strbuf_release(&buf);
       		strbuf_release(&head_ref);
     - 	}
       
     --	do_update_refs(r);
     -+	do_update_refs(r, opts->quiet);
     +-		if (do_update_refs(r))
     ++		if (do_update_refs(r, opts->quiet))
     + 			return -1;
     + 	}
       
     - 	/*
     - 	 * Sequence of picks finished successfully; cleanup by
      
       ## t/t3404-rebase-interactive.sh ##
      @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs correctly' '
     @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs c
       
       test_expect_success 'respect user edits to update-ref steps' '
      @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs: check failed ref update' '
     + 	# the lock in the update-refs file.
       	git rev-parse third >.git/refs/heads/second &&
       
     - 	git rebase --continue 2>err &&
     +-	git rebase --continue 2>err &&
      -	grep "update_ref failed for ref '\''refs/heads/second'\''" err
     ++	test_must_fail git rebase --continue 2>err &&
      +	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
      +
      +	cat >expect <<-\EOF &&

-- 
gitgitgadget

[PATCH v5 01/12] t2407: test bisect and rebase as black-boxes

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The tests added by d2ba271aad0 (branch: check for bisects and rebases,
2022-06-14) modified hidden state to verify the branch_checked_out()
helper. While this indeed checks that the method implementation is _as
designed_, it doesn't show that it is _correct_. Specifically, if 'git
bisect' or 'git rebase' change their back-end for preserving refs, then
these tests do not demonstrate that drift as a bug in
branch_checked_out().

Modify the tests in t2407 to actually rely on a paused bisect or rebase.
This requires adding the !SANITIZE_LEAK prereq for tests using those
builtins. The logic is still tested for leaks in the final test which
does set up that back-end directly for an error state that should not be
possible using Git commands.

Reported-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 t/t2407-worktree-heads.sh | 57 +++++++++++++++++++++------------------
 1 file changed, 31 insertions(+), 26 deletions(-)

diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index b6be42f74a2..100ab286d5c 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -7,13 +7,18 @@ TEST_PASSES_SANITIZE_LEAK=true
 
 test_expect_success 'setup' '
 	test_commit init &&
-	git branch -f fake-1 &&
-	git branch -f fake-2 &&
 
 	for i in 1 2 3 4
 	do
+		git checkout -b conflict-$i &&
+		echo "not I" >$i.t &&
+		git add $i.t &&
+		git commit -m "will conflict" &&
+
+		git checkout - &&
 		test_commit $i &&
 		git branch wt-$i &&
+		git branch fake-$i &&
 		git worktree add wt-$i wt-$i || return 1
 	done &&
 
@@ -44,26 +49,26 @@ test_expect_success 'refuse to overwrite: checked out in worktree' '
 	done
 '
 
-test_expect_success 'refuse to overwrite: worktree in bisect' '
-	test_when_finished rm -rf .git/worktrees/wt-*/BISECT_* &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
+	test_when_finished git -C wt-4 bisect reset &&
 
-	touch .git/worktrees/wt-4/BISECT_LOG &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/BISECT_START &&
+	# Set up a bisect so HEAD no longer points to wt-4.
+	git -C wt-4 bisect start &&
+	git -C wt-4 bisect bad wt-4 &&
+	git -C wt-4 bisect good wt-1 &&
 
-	test_must_fail git branch -f fake-2 HEAD 2>err &&
-	grep "cannot force update the branch '\''fake-2'\'' checked out at.*wt-4" err
+	test_must_fail git branch -f wt-4 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase' '
-	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
+	test_when_finished git -C wt-2 rebase --abort &&
 
-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
-	echo refs/heads/fake-1 >.git/worktrees/wt-3/rebase-merge/head-name &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-3/rebase-merge/onto &&
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-2 rebase conflict-2 &&
 
-	test_must_fail git branch -f fake-1 HEAD 2>err &&
-	grep "cannot force update the branch '\''fake-1'\'' checked out at.*wt-3" err
+	test_must_fail git branch -f wt-2 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
@@ -77,24 +82,24 @@ test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: worktree in bisect' '
-	test_when_finished rm -rf .git/worktrees/wt-*/BISECT_* &&
+	test_when_finished git -C wt-4 bisect reset &&
 
-	touch .git/worktrees/wt-4/BISECT_LOG &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/BISECT_START &&
+	# Set up a bisect so HEAD no longer points to wt-4.
+	git -C wt-4 bisect start &&
+	git -C wt-4 bisect bad wt-4 &&
+	git -C wt-4 bisect good wt-1 &&
 
-	test_must_fail git fetch server +refs/heads/fake-2:refs/heads/fake-2 2>err &&
+	test_must_fail git fetch server +refs/heads/wt-4:refs/heads/wt-4 2>err &&
 	grep "refusing to fetch into branch" err
 '
 
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: worktree in rebase' '
-	test_when_finished rm -rf .git/worktrees/wt-*/rebase-merge &&
+	test_when_finished git -C wt-3 rebase --abort &&
 
-	mkdir -p .git/worktrees/wt-4/rebase-merge &&
-	touch .git/worktrees/wt-4/rebase-merge/interactive &&
-	echo refs/heads/fake-1 >.git/worktrees/wt-4/rebase-merge/head-name &&
-	echo refs/heads/fake-2 >.git/worktrees/wt-4/rebase-merge/onto &&
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-3 rebase conflict-3 &&
 
-	test_must_fail git fetch server +refs/heads/fake-1:refs/heads/fake-1 2>err &&
+	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
 	grep "refusing to fetch into branch" err
 '
 
-- 
gitgitgadget

[PATCH v5 02/12] t2407: test branches currently using apply backend

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The tests in t2407 that verify the branch_checked_out() helper in the
case of bisects and rebases were added by 9347303db89 (branch: check for
bisects and rebases, 2022-06-08). However, that commit failed to check
for rebases that are using the 'apply' backend.

Add a test that checks the apply backend. The implementation was already
correct here, but it is good to have regression tests before modifying
the implementation further.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 t/t2407-worktree-heads.sh | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 100ab286d5c..a67ce5fb003 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -61,7 +61,17 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in bisect' '
 	grep "cannot force update the branch '\''wt-4'\'' checked out at.*wt-4" err
 '
 
-test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase' '
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (apply)' '
+	test_when_finished git -C wt-2 rebase --abort &&
+
+	# This will fail part-way through due to a conflict.
+	test_must_fail git -C wt-2 rebase --apply conflict-2 &&
+
+	test_must_fail git branch -f wt-2 HEAD 2>err &&
+	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
+'
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (merge)' '
 	test_when_finished git -C wt-2 rebase --abort &&
 
 	# This will fail part-way through due to a conflict.
-- 
gitgitgadget

[PATCH v5 03/12] branch: consider refs under 'update-refs'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The branch_checked_out() helper helps commands like 'git branch' and
'git fetch' from overwriting refs that are currently checked out in
other worktrees.

A future update to 'git rebase' will introduce a new '--update-refs'
option which will update the local refs that point to commits that are
being rebased. To avoid collisions as the rebase completes, we want to
make the future data store for these refs to be considered by
branch_checked_out().

The data store is a plaintext file inside the 'rebase-merge' directory
for that worktree. The file lists refnames followed by two OIDs, each on
separate lines. The OIDs will be used to store the original values of
the refs and the to-be-written values as the rebase progresses, but can
be ignored at the moment.

Create a new sequencer_get_update_refs_state() method that parses this
file and populates a struct string_list with the ref-OID pairs. We can
then use this list to add to the current_checked_out_branches strmap
used by branch_checked_out().

To properly navigate to the rebase directory for a given worktree,
extract the static strbuf_worktree_gitdir() method to a public API
method.

We can test that this works without having Git write this file by
artificially creating one in our test script, at least until 'git rebase
--update-refs' is implemented and we can use it directly.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 branch.c                  | 13 +++++++
 sequencer.c               | 74 +++++++++++++++++++++++++++++++++++++++
 sequencer.h               |  9 +++++
 t/t2407-worktree-heads.sh | 23 ++++++++++++
 4 files changed, 119 insertions(+)

diff --git a/branch.c b/branch.c
index 526e8237673..f252c4eef6c 100644
--- a/branch.c
+++ b/branch.c
@@ -365,6 +365,7 @@ static void prepare_checked_out_branches(void)
 		char *old;
 		struct wt_status_state state = { 0 };
 		struct worktree *wt = worktrees[i++];
+		struct string_list update_refs = STRING_LIST_INIT_DUP;
 
 		if (wt->is_bare)
 			continue;
@@ -400,6 +401,18 @@ static void prepare_checked_out_branches(void)
 			strbuf_release(&ref);
 		}
 		wt_status_state_free_buffers(&state);
+
+		if (!sequencer_get_update_refs_state(get_worktree_git_dir(wt),
+						     &update_refs)) {
+			struct string_list_item *item;
+			for_each_string_list_item(item, &update_refs) {
+				old = strmap_put(&current_checked_out_branches,
+						 item->string,
+						 xstrdup(wt->path));
+				free(old);
+			}
+			string_list_clear(&update_refs, 1);
+		}
 	}
 
 	free_worktrees(worktrees);
diff --git a/sequencer.c b/sequencer.c
index 8c3ed3532ac..7d131d06cc3 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -147,6 +147,20 @@ static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-onto")
  */
 static GIT_PATH_FUNC(rebase_path_refs_to_delete, "rebase-merge/refs-to-delete")
 
+/*
+ * The update-refs file stores a list of refs that will be updated at the end
+ * of the rebase sequence. The 'update-ref <ref>' commands in the todo file
+ * update the OIDs for the refs in this file, but the refs are not updated
+ * until the end of the rebase sequence.
+ *
+ * rebase_path_update_refs() returns the path to this file for a given
+ * worktree directory. For the current worktree, pass the_repository->gitdir.
+ */
+static char *rebase_path_update_refs(const char *wt_git_dir)
+{
+	return xstrfmt("%s/rebase-merge/update-refs", wt_git_dir);
+}
+
 /*
  * The following files are written by git-rebase just after parsing the
  * command-line.
@@ -169,6 +183,15 @@ static GIT_PATH_FUNC(rebase_path_no_reschedule_failed_exec, "rebase-merge/no-res
 static GIT_PATH_FUNC(rebase_path_drop_redundant_commits, "rebase-merge/drop_redundant_commits")
 static GIT_PATH_FUNC(rebase_path_keep_redundant_commits, "rebase-merge/keep_redundant_commits")
 
+/**
+ * A 'struct update_refs_record' represents a value in the update-refs
+ * list. We use a string_list to map refs to these (before, after) pairs.
+ */
+struct update_ref_record {
+	struct object_id before;
+	struct object_id after;
+};
+
 static int git_sequencer_config(const char *k, const char *v, void *cb)
 {
 	struct replay_opts *opts = cb;
@@ -5901,3 +5924,54 @@ int sequencer_determine_whence(struct repository *r, enum commit_whence *whence)
 
 	return 0;
 }
+
+int sequencer_get_update_refs_state(const char *wt_dir,
+				    struct string_list *refs)
+{
+	int result = 0;
+	FILE *fp = NULL;
+	struct strbuf ref = STRBUF_INIT;
+	struct strbuf hash = STRBUF_INIT;
+	struct update_ref_record *rec = NULL;
+
+	char *path = rebase_path_update_refs(wt_dir);
+
+	fp = fopen(path, "r");
+	if (!fp)
+		goto cleanup;
+
+	while (strbuf_getline(&ref, fp) != EOF) {
+		struct string_list_item *item;
+
+		CALLOC_ARRAY(rec, 1);
+
+		if (strbuf_getline(&hash, fp) == EOF ||
+		    get_oid_hex(hash.buf, &rec->before)) {
+			warning(_("update-refs file at '%s' is invalid"),
+				  path);
+			result = -1;
+			goto cleanup;
+		}
+
+		if (strbuf_getline(&hash, fp) == EOF ||
+		    get_oid_hex(hash.buf, &rec->after)) {
+			warning(_("update-refs file at '%s' is invalid"),
+				  path);
+			result = -1;
+			goto cleanup;
+		}
+
+		item = string_list_insert(refs, ref.buf);
+		item->util = rec;
+		rec = NULL;
+	}
+
+cleanup:
+	if (fp)
+		fclose(fp);
+	free(path);
+	free(rec);
+	strbuf_release(&ref);
+	strbuf_release(&hash);
+	return result;
+}
diff --git a/sequencer.h b/sequencer.h
index da64473636b..3ae541bb145 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -232,4 +232,13 @@ void sequencer_post_commit_cleanup(struct repository *r, int verbose);
 int sequencer_get_last_command(struct repository* r,
 			       enum replay_action *action);
 int sequencer_determine_whence(struct repository *r, enum commit_whence *whence);
+
+/**
+ * Append the set of ref-OID pairs that are currently stored for the 'git
+ * rebase --update-refs' feature if such a rebase is currently happening.
+ *
+ * Localized to a worktree's git dir.
+ */
+int sequencer_get_update_refs_state(const char *wt_dir, struct string_list *refs);
+
 #endif /* SEQUENCER_H */
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index a67ce5fb003..97f5c87f8c8 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -81,6 +81,29 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
+test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
+	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
+
+	mkdir -p .git/worktrees/wt-3/rebase-merge &&
+	touch .git/worktrees/wt-3/rebase-merge/interactive &&
+
+	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
+	refs/heads/fake-3
+	$(git rev-parse HEAD~1)
+	$(git rev-parse HEAD)
+	refs/heads/fake-4
+	$(git rev-parse HEAD)
+	$(git rev-parse HEAD)
+	EOF
+
+	for i in 3 4
+	do
+		test_must_fail git branch -f fake-$i HEAD 2>err &&
+		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+			return 1
+	done
+'
+
 test_expect_success !SANITIZE_LEAK 'refuse to fetch over ref: checked out' '
 	test_must_fail git fetch server +refs/heads/wt-3:refs/heads/wt-3 2>err &&
 	grep "refusing to fetch into branch '\''refs/heads/wt-3'\''" err &&
-- 
gitgitgadget

[PATCH v5 04/12] rebase-interactive: update 'merge' description

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The 'merge' command description for the todo list documentation in an
interactive rebase has multiple lines. The lines other than the first
one start with dots ('.') while the similar multi-line documentation for
'fixup' does not. This description only appears in the comment text of
the todo file during an interactive rebase.

The 'merge' command was documented when interactive rebase was first
ported to C in 145e05ac44b (rebase -i: rewrite append_todo_help() in C,
2018-08-10). These dots might have been carried over from the previous
shell implementation.

The 'fixup' command was documented more recently in 9e3cebd97cb (rebase
-i: add fixup [-C | -c] command, 2021-01-29).

Looking at the output in an editor, my personal opinion is that the dots
are unnecessary and noisy. Remove them now before adding more commands
with multi-line documentation.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 87649d0c016..22394224faa 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -54,9 +54,9 @@ void append_todo_help(int command_count,
 "l, label <label> = label current HEAD with a name\n"
 "t, reset <label> = reset HEAD to a label\n"
 "m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n"
-".       create a merge commit using the original merge commit's\n"
-".       message (or the oneline, if no original merge commit was\n"
-".       specified); use -c <commit> to reword the commit message\n"
+"        create a merge commit using the original merge commit's\n"
+"        message (or the oneline, if no original merge commit was\n"
+"        specified); use -c <commit> to reword the commit message\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
-- 
gitgitgadget

[PATCH v5 05/12] sequencer: define array with enum values

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The todo_command_info array defines which strings match with which
todo_command enum values. The array is defined in the same order as the
enum values, but if one changed without the other, then we would have
unexpected results.

Make it easier to see changes to the enum and this array by using the
enum values as the indices of the array.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 7d131d06cc3..2711182e43f 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1708,20 +1708,20 @@ static struct {
 	char c;
 	const char *str;
 } todo_command_info[] = {
-	{ 'p', "pick" },
-	{ 0,   "revert" },
-	{ 'e', "edit" },
-	{ 'r', "reword" },
-	{ 'f', "fixup" },
-	{ 's', "squash" },
-	{ 'x', "exec" },
-	{ 'b', "break" },
-	{ 'l', "label" },
-	{ 't', "reset" },
-	{ 'm', "merge" },
-	{ 0,   "noop" },
-	{ 'd', "drop" },
-	{ 0,   NULL }
+	[TODO_PICK] = { 'p', "pick" },
+	[TODO_REVERT] = { 0,   "revert" },
+	[TODO_EDIT] = { 'e', "edit" },
+	[TODO_REWORD] = { 'r', "reword" },
+	[TODO_FIXUP] = { 'f', "fixup" },
+	[TODO_SQUASH] = { 's', "squash" },
+	[TODO_EXEC] = { 'x', "exec" },
+	[TODO_BREAK] = { 'b', "break" },
+	[TODO_LABEL] = { 'l', "label" },
+	[TODO_RESET] = { 't', "reset" },
+	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_NOOP] = { 0,   "noop" },
+	[TODO_DROP] = { 'd', "drop" },
+	[TODO_COMMENT] = { 0,   NULL },
 };
 
 static const char *command_to_string(const enum todo_command command)
-- 
gitgitgadget

[PATCH v5 06/12] sequencer: add update-ref command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

Add the boilerplate for an "update-ref" command in the sequencer. This
connects to the current no-op do_update_ref() which will be filled in
after more connections are created.

The syntax in the todo list will be "update-ref <ref-name>" to signal
that we should store the current commit as the value for updating
<ref-name> at the end of the rebase.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c |  3 +++
 sequencer.c          | 14 +++++++++++++-
 sequencer.h          |  1 +
 3 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 22394224faa..1ff07647af3 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -57,6 +57,9 @@ void append_todo_help(int command_count,
 "        create a merge commit using the original merge commit's\n"
 "        message (or the oneline, if no original merge commit was\n"
 "        specified); use -c <commit> to reword the commit message\n"
+"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
+"                      to this position in the new commits. The <ref> is\n"
+"                      updated at the end of the rebase\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
diff --git a/sequencer.c b/sequencer.c
index 2711182e43f..0dc9c05c5bb 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -1719,6 +1719,7 @@ static struct {
 	[TODO_LABEL] = { 'l', "label" },
 	[TODO_RESET] = { 't', "reset" },
 	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_UPDATE_REF] = { 'u', "update-ref" },
 	[TODO_NOOP] = { 0,   "noop" },
 	[TODO_DROP] = { 'd', "drop" },
 	[TODO_COMMENT] = { 0,   NULL },
@@ -2480,7 +2481,7 @@ static int parse_insn_line(struct repository *r, struct todo_item *item,
 			     command_to_string(item->command));
 
 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
-	    item->command == TODO_RESET) {
+	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
 		item->commit = NULL;
 		item->arg_offset = bol - buf;
 		item->arg_len = (int)(eol - bol);
@@ -4079,6 +4080,11 @@ leave_merge:
 	return ret;
 }
 
+static int do_update_ref(struct repository *r, const char *ref_name)
+{
+	return 0;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4454,6 +4460,12 @@ static int pick_commits(struct repository *r,
 				return error_with_patch(r, item->commit,
 							arg, item->arg_len,
 							opts, res, 0);
+		} else if (item->command == TODO_UPDATE_REF) {
+			struct strbuf ref = STRBUF_INIT;
+			strbuf_add(&ref, arg, item->arg_len);
+			if ((res = do_update_ref(r, ref.buf)))
+				reschedule = 1;
+			strbuf_release(&ref);
 		} else if (!is_noop(item->command))
 			return error(_("unknown command %d"), item->command);
 
diff --git a/sequencer.h b/sequencer.h
index 3ae541bb145..2cf5c1b9a38 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -95,6 +95,7 @@ enum todo_command {
 	TODO_LABEL,
 	TODO_RESET,
 	TODO_MERGE,
+	TODO_UPDATE_REF,
 	/* commands that do nothing but are counted for reporting progress */
 	TODO_NOOP,
 	TODO_DROP,
-- 
gitgitgadget

[PATCH v5 07/12] rebase: add --update-refs option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When working on a large feature, it can be helpful to break that feature
into multiple smaller parts that become reviewed in sequence. During
development or during review, a change to one part of the feature could
affect multiple of these parts. An interactive rebase can help adjust
the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the
feature, then rebasing the entire list of commits can create commits not
reachable from those "sub branches". It can take a manual step to update
those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'update-ref
<ref>' steps to the todo file whenever a commit that is being rebased is
decorated with that <ref>. At the very end, the rebase process updates
all of the listed refs to the values stored during the rebase operation.

Be sure to iterate after any squashing or fixups are placed. Update the
branch only after those squashes and fixups are complete. This allows a
--fixup commit at the tip of the feature to apply correctly to the sub
branch, even if it is fixing up the most-recent commit in that part.

This change update the documentation and builtin to accept the
--update-refs option as well as updating the todo file with the
'update-ref' commands. Tests are added to ensure that these todo
commands are added in the correct locations.

This change does _not_ include the actual behavior of tracking the
updated refs and writing the new ref values at the end of the rebase
process. That is deferred to a later change.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-rebase.txt  |   7 +++
 builtin/rebase.c              |   5 ++
 sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
 sequencer.h                   |   1 +
 t/t2407-worktree-heads.sh     |  22 +++++++
 t/t3404-rebase-interactive.sh |  74 +++++++++++++++++++++++
 6 files changed, 216 insertions(+)

diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index 262fb01aec0..dfd847d6615 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -609,6 +609,12 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	are not updated in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
@@ -632,6 +638,7 @@ are incompatible with the following options:
  * --empty=
  * --reapply-cherry-picks
  * --edit-todo
+ * --update-refs
  * --root when used in combination with --onto
 
 In addition, the following pairs of options are incompatible:
diff --git a/builtin/rebase.c b/builtin/rebase.c
index 7ab50cda2ad..fd64897132a 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update branches that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
diff --git a/sequencer.c b/sequencer.c
index 0dc9c05c5bb..a4ee2799d6f 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -35,6 +35,7 @@
 #include "commit-reach.h"
 #include "rebase-interactive.h"
 #include "reset.h"
+#include "branch.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -5638,10 +5639,113 @@ static int skip_unnecessary_picks(struct repository *r,
 	return 0;
 }
 
+struct todo_add_branch_context {
+	struct todo_item *items;
+	size_t items_nr;
+	size_t items_alloc;
+	struct strbuf *buf;
+	struct commit *commit;
+	struct string_list refs_to_oids;
+};
+
+static int add_decorations_to_list(const struct commit *commit,
+				   struct todo_add_branch_context *ctx)
+{
+	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+
+	while (decoration) {
+		struct todo_item *item;
+		const char *path;
+		size_t base_offset = ctx->buf->len;
+
+		ALLOC_GROW(ctx->items,
+			ctx->items_nr + 1,
+			ctx->items_alloc);
+		item = &ctx->items[ctx->items_nr];
+		memset(item, 0, sizeof(*item));
+
+		/* If the branch is checked out, then leave a comment instead. */
+		if ((path = branch_checked_out(decoration->name))) {
+			item->command = TODO_COMMENT;
+			strbuf_addf(ctx->buf, "# Ref %s checked out at '%s'\n",
+				    decoration->name, path);
+		} else {
+			struct string_list_item *sti;
+			item->command = TODO_UPDATE_REF;
+			strbuf_addf(ctx->buf, "%s\n", decoration->name);
+
+			sti = string_list_insert(&ctx->refs_to_oids,
+						 decoration->name);
+			sti->util = oiddup(the_hash_algo->null_oid);
+		}
+
+		item->offset_in_buf = base_offset;
+		item->arg_offset = base_offset;
+		item->arg_len = ctx->buf->len - base_offset;
+		ctx->items_nr++;
+
+		decoration = decoration->next;
+	}
+
+	return 0;
+}
+
+/*
+ * For each 'pick' command, find out if the commit has a decoration in
+ * refs/heads/. If so, then add a 'label for-update-refs/' command.
+ */
+static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
+{
+	int i;
+	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
+	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
+	struct decoration_filter decoration_filter = {
+		.include_ref_pattern = &decorate_refs_include,
+		.exclude_ref_pattern = &decorate_refs_exclude,
+		.exclude_ref_config_pattern = &decorate_refs_exclude_config,
+	};
+	struct todo_add_branch_context ctx = {
+		.buf = &todo_list->buf,
+		.refs_to_oids = STRING_LIST_INIT_DUP,
+	};
+
+	ctx.items_alloc = 2 * todo_list->nr + 1;
+	ALLOC_ARRAY(ctx.items, ctx.items_alloc);
+
+	string_list_append(&decorate_refs_include, "refs/heads/");
+	load_ref_decorations(&decoration_filter, 0);
+
+	for (i = 0; i < todo_list->nr; ) {
+		struct todo_item *item = &todo_list->items[i];
+
+		/* insert ith item into new list */
+		ALLOC_GROW(ctx.items,
+			   ctx.items_nr + 1,
+			   ctx.items_alloc);
+
+		ctx.items[ctx.items_nr++] = todo_list->items[i++];
+
+		if (item->commit) {
+			ctx.commit = item->commit;
+			add_decorations_to_list(item->commit, &ctx);
+		}
+	}
+
+	string_list_clear(&ctx.refs_to_oids, 1);
+	free(todo_list->items);
+	todo_list->items = ctx.items;
+	todo_list->nr = ctx.items_nr;
+	todo_list->alloc = ctx.items_alloc;
+
+	return 0;
+}
+
 int complete_action(struct repository *r, struct replay_opts *opts, unsigned flags,
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list)
 {
 	char shortonto[GIT_MAX_HEXSZ + 1];
@@ -5660,6 +5764,9 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		item->arg_len = item->arg_offset = item->flags = item->offset_in_buf = 0;
 	}
 
+	if (update_refs && todo_list_add_update_ref_commands(todo_list))
+		return -1;
+
 	if (autosquash && todo_list_rearrange_squash(todo_list))
 		return -1;
 
diff --git a/sequencer.h b/sequencer.h
index 2cf5c1b9a38..e671d7e0743 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 97f5c87f8c8..8a03f14df8d 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -164,4 +164,26 @@ test_expect_success 'refuse to overwrite when in error states' '
 	done
 '
 
+. "$TEST_DIRECTORY"/lib-rebase.sh
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
+	git commit --fixup HEAD~2 --allow-empty &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		! grep "update-refs" todo
+	) &&
+	git branch -f allow-update HEAD~2 &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		grep "update-ref refs/heads/allow-update" todo
+	)
+'
+
+# This must be the last test in this file
+test_expect_success '$EDITOR and friends are unchanged' '
+	test_editor_unchanged
+'
+
 test_done
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index f31afd4a547..75c0b90fae1 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1743,6 +1743,80 @@ test_expect_success 'ORIG_HEAD is updated correctly' '
 	test_cmp_rev ORIG_HEAD test-orig-head@{1}
 '
 
+test_expect_success '--update-refs adds label and update-ref commands' '
+	git checkout -b update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git commit --allow-empty --fixup=third &&
+	git branch -f is-not-reordered &&
+	git commit --allow-empty --fixup=HEAD~4 &&
+	git branch -f shared-tip &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		pick $(git log -1 --format=%h J) J
+		fixup $(git log -1 --format=%h update-refs) fixup! J # empty
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h K) K
+		pick $(git log -1 --format=%h L) L
+		fixup $(git log -1 --format=%h is-not-reordered) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h M) M
+		update-ref refs/heads/no-conflict-branch
+		update-ref refs/heads/is-not-reordered
+		update-ref refs/heads/shared-tip
+		EOF
+
+		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo
+	)
+'
+
+test_expect_success '--update-refs adds commands with --rebase-merges' '
+	git checkout -b update-refs-with-merge no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git merge -m merge branch2 &&
+	git branch -f merge-branch &&
+	git commit --fixup=third --allow-empty &&
+	(
+		set_cat_todo_editor &&
+
+		cat >expect <<-EOF &&
+		label onto
+		reset onto
+		pick $(git log -1 --format=%h branch2~1) F
+		pick $(git log -1 --format=%h branch2) I
+		update-ref refs/heads/branch2
+		label merge
+		reset onto
+		pick $(git log -1 --format=%h refs/heads/second) J
+		update-ref refs/heads/second
+		update-ref refs/heads/first
+		pick $(git log -1 --format=%h refs/heads/third~1) K
+		pick $(git log -1 --format=%h refs/heads/third) L
+		fixup $(git log -1 --format=%h update-refs-with-merge) fixup! L # empty
+		update-ref refs/heads/third
+		pick $(git log -1 --format=%h HEAD~2) M
+		update-ref refs/heads/no-conflict-branch
+		merge -C $(git log -1 --format=%h HEAD~1) merge # merge
+		update-ref refs/heads/merge-branch
+		EOF
+
+		test_must_fail git rebase -i --autosquash \
+				   --rebase-merges=rebase-cousins \
+				   --update-refs primary >todo &&
+
+		test_cmp expect todo
+	)
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

[PATCH v5 08/12] rebase: update refs from 'update-ref' commands

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change introduced the 'git rebase --update-refs' option
which added 'update-ref <ref>' commands to the todo list of an
interactive rebase.

Teach Git to record the HEAD position when reaching these 'update-ref'
commands. The ref/before/after triple is stored in the
$GIT_DIR/rebase-merge/update-refs file. A previous change parsed this
file to avoid having other processes updating the refs in that file
while the rebase is in progress.

Not only do we update the file when the sequencer reaches these
'update-ref' commands, we then update the refs themselves at the end of
the rebase sequence. If the rebase is aborted before this final step,
then the refs are not updated. The 'before' value is used to ensure that
we do not accidentally obliterate a ref that was updated concurrently
(say, by an older version of Git or a third-party tool).

Now that the 'git rebase --update-refs' command is implemented to write
to the update-refs file, we can remove the fake construction of the
update-refs file from a test in t2407-worktree-heads.sh.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 120 +++++++++++++++++++++++++++++++++-
 t/t2407-worktree-heads.sh     |  21 ++----
 t/t3404-rebase-interactive.sh |  17 +++++
 3 files changed, 140 insertions(+), 18 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index a4ee2799d6f..ce8c7d8cd3a 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -36,6 +36,7 @@
 #include "rebase-interactive.h"
 #include "reset.h"
 #include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -193,6 +194,21 @@ struct update_ref_record {
 	struct object_id after;
 };
 
+static struct update_ref_record *init_update_ref_record(const char *ref)
+{
+	struct update_ref_record *rec;
+
+	CALLOC_ARRAY(rec, 1);
+
+	oidcpy(&rec->before, null_oid());
+	oidcpy(&rec->after, null_oid());
+
+	/* This may fail, but that's fine, we will keep the null OID. */
+	read_ref(ref, &rec->before);
+
+	return rec;
+}
+
 static int git_sequencer_config(const char *k, const char *v, void *cb)
 {
 	struct replay_opts *opts = cb;
@@ -4081,11 +4097,97 @@ leave_merge:
 	return ret;
 }
 
-static int do_update_ref(struct repository *r, const char *ref_name)
+static int write_update_refs_state(struct string_list *refs_to_oids)
+{
+	int result = 0;
+	struct lock_file lock = LOCK_INIT;
+	FILE *fp = NULL;
+	struct string_list_item *item;
+	char *path;
+
+	if (!refs_to_oids->nr)
+		return 0;
+
+	path = rebase_path_update_refs(the_repository->gitdir);
+
+	if (safe_create_leading_directories(path)) {
+		result = error(_("unable to create leading directories of %s"),
+			       path);
+		goto cleanup;
+	}
+
+	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
+		result = error(_("another 'rebase' process appears to be running; "
+				 "'%s.lock' already exists"),
+			       path);
+		goto cleanup;
+	}
+
+	fp = fdopen_lock_file(&lock, "w");
+	if (!fp) {
+		result = error_errno(_("could not open '%s' for writing"), path);
+		rollback_lock_file(&lock);
+		goto cleanup;
+	}
+
+	for_each_string_list_item(item, refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+		fprintf(fp, "%s\n%s\n%s\n", item->string,
+			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
+	}
+
+	result = commit_lock_file(&lock);
+
+cleanup:
+	free(path);
+	return result;
+}
+
+static int do_update_ref(struct repository *r, const char *refname)
 {
+	struct string_list_item *item;
+	struct string_list list = STRING_LIST_INIT_DUP;
+
+	if (sequencer_get_update_refs_state(r->gitdir, &list))
+		return -1;
+
+	for_each_string_list_item(item, &list) {
+		if (!strcmp(item->string, refname)) {
+			struct update_ref_record *rec = item->util;
+			if (read_ref("HEAD", &rec->after))
+				return -1;
+			break;
+		}
+	}
+
+	write_update_refs_state(&list);
+	string_list_clear(&list, 1);
 	return 0;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	int res = 0;
+	struct string_list_item *item;
+	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
+	struct ref_store *refs = get_main_ref_store(r);
+
+	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
+		return res;
+
+	for_each_string_list_item(item, &refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+
+		res |= refs_update_ref(refs, "rewritten during rebase",
+				       item->string,
+				       &rec->after, &rec->before,
+				       0, UPDATE_REFS_MSG_ON_ERR);
+	}
+
+	string_list_clear(&refs_to_oids, 1);
+	return res;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4601,6 +4703,9 @@ cleanup_head_ref:
 
 		strbuf_release(&buf);
 		strbuf_release(&head_ref);
+
+		if (do_update_refs(r))
+			return -1;
 	}
 
 	/*
@@ -5676,7 +5781,7 @@ static int add_decorations_to_list(const struct commit *commit,
 
 			sti = string_list_insert(&ctx->refs_to_oids,
 						 decoration->name);
-			sti->util = oiddup(the_hash_algo->null_oid);
+			sti->util = init_update_ref_record(decoration->name);
 		}
 
 		item->offset_in_buf = base_offset;
@@ -5696,7 +5801,7 @@ static int add_decorations_to_list(const struct commit *commit,
  */
 static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 {
-	int i;
+	int i, res;
 	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
 	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
 	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
@@ -5732,7 +5837,16 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 		}
 	}
 
+	res = write_update_refs_state(&ctx.refs_to_oids);
+
 	string_list_clear(&ctx.refs_to_oids, 1);
+
+	if (res) {
+		/* we failed, so clean up the new list. */
+		free(ctx.items);
+		return res;
+	}
+
 	free(todo_list->items);
 	todo_list->items = ctx.items;
 	todo_list->nr = ctx.items_nr;
diff --git a/t/t2407-worktree-heads.sh b/t/t2407-worktree-heads.sh
index 8a03f14df8d..50815acd3e8 100755
--- a/t/t2407-worktree-heads.sh
+++ b/t/t2407-worktree-heads.sh
@@ -81,25 +81,16 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
-	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
-
-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
+	test_when_finished git -C wt-3 rebase --abort &&
 
-	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
-	refs/heads/fake-3
-	$(git rev-parse HEAD~1)
-	$(git rev-parse HEAD)
-	refs/heads/fake-4
-	$(git rev-parse HEAD)
-	$(git rev-parse HEAD)
-	EOF
+	git branch -f can-be-updated wt-3 &&
+	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
 
 	for i in 3 4
 	do
-		test_must_fail git branch -f fake-$i HEAD 2>err &&
-		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+		test_must_fail git branch -f can-be-updated HEAD 2>err &&
+		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
 			return 1
 	done
 '
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 75c0b90fae1..2fffcf5d5fb 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1817,6 +1817,23 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 	)
 '
 
+test_expect_success '--update-refs updates refs correctly' '
+	git checkout -B update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	test_commit extra2 fileX &&
+	git commit --amend --fixup=L &&
+
+	git rebase -i --autosquash --update-refs primary &&
+
+	test_cmp_rev HEAD~3 refs/heads/first &&
+	test_cmp_rev HEAD~3 refs/heads/second &&
+	test_cmp_rev HEAD~1 refs/heads/third &&
+	test_cmp_rev HEAD refs/heads/no-conflict-branch
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v5 08/12] rebase: update refs from 'update-ref' commands

[Phillip Wood]

Hi Stolee

On 19/07/2022 19:33, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> +static int do_update_ref(struct repository *r, const char *refname)
>   {
> +	struct string_list_item *item;
> +	struct string_list list = STRING_LIST_INIT_DUP;
> +
> +	if (sequencer_get_update_refs_state(r->gitdir, &list))
> +		return -1;

This is good, we now return an error if we cannot read the update-refs file.

> +	for_each_string_list_item(item, &list) {
> +		if (!strcmp(item->string, refname)) {
> +			struct update_ref_record *rec = item->util;
> +			if (read_ref("HEAD", &rec->after))
> +				return -1;
> +			break;
> +		}
> +	}
> +
> +	write_update_refs_state(&list);

The return value of write_update_refs_state() should be propagated to 
the caller.

> +	string_list_clear(&list, 1);
>   	return 0;
>   }

Best Wishes

Phillip

[PATCH v5 09/12] sequencer: rewrite update-refs as user edits todo list

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

An interactive rebase provides opportunities for the user to edit the
todo list. The --update-refs option initializes the list with some
'update-ref <ref>' steps, but the user could add these manually.
Further, the user could add or remove these steps during pauses in the
interactive rebase.

Add a new method, todo_list_filter_update_refs(), that scans a todo_list
and compares it to the stored update-refs file. There are two actions
that can happen at this point:

1. If a '<ref>/<before>/<after>' triple in the update-refs file does not
   have a matching 'update-ref <ref>' command in the todo-list _and_ the
   <after> value is the null OID, then remove that triple. Here, the
   user removed the 'update-ref <ref>' command before it was executed,
   since if it was executed then the <after> value would store the
   commit at that position.

2. If a 'update-ref <ref>' command in the todo-list does not have a
   matching '<ref>/<before>/<after>' triple in the update-refs file,
   then insert a new one. Store the <before> value to be the current
   OID pointed at by <ref>. This is handled inside of the
   init_update_ref_record() helper method.

We can test that this works by rewriting the todo-list several times in
the course of a rebase. Check that each ref is locked or unlocked for
updates after each todo-list update. We can also verify that the ref
update fails if a concurrent process updates one of the refs after the
rebase process records the "locked" ref location.

To help these tests, add a new 'set_replace_editor' helper that will
replace the todo-list with an exact file.

Reported-by: Phillip Wood <phillip.wood123@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 rebase-interactive.c          |   6 ++
 sequencer.c                   |  96 +++++++++++++++++++++++
 sequencer.h                   |  12 +++
 t/lib-rebase.sh               |  15 ++++
 t/t3404-rebase-interactive.sh | 139 ++++++++++++++++++++++++++++++++++
 5 files changed, 268 insertions(+)

diff --git a/rebase-interactive.c b/rebase-interactive.c
index 1ff07647af3..7407c593191 100644
--- a/rebase-interactive.c
+++ b/rebase-interactive.c
@@ -146,6 +146,12 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list,
 		return -4;
 	}
 
+	/*
+	 * See if branches need to be added or removed from the update-refs
+	 * file based on the new todo list.
+	 */
+	todo_list_filter_update_refs(r, new_todo);
+
 	return 0;
 }
 
diff --git a/sequencer.c b/sequencer.c
index ce8c7d8cd3a..67812c0294f 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -4143,6 +4143,102 @@ cleanup:
 	return result;
 }
 
+/*
+ * Parse the update-refs file for the current rebase, then remove the
+ * refs that do not appear in the todo_list (and have not had updated
+ * values stored) and add refs that are in the todo_list but not
+ * represented in the update-refs file.
+ *
+ * If there are changes to the update-refs list, then write the new state
+ * to disk.
+ */
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list)
+{
+	int i;
+	int updated = 0;
+	struct string_list update_refs = STRING_LIST_INIT_DUP;
+
+	sequencer_get_update_refs_state(r->gitdir, &update_refs);
+
+	/*
+	 * For each item in the update_refs list, if it has no updated
+	 * value and does not appear in the todo_list, then remove it
+	 * from the update_refs list.
+	 */
+	for (i = 0; i < update_refs.nr; i++) {
+		int j;
+		int found = 0;
+		const char *ref = update_refs.items[i].string;
+		size_t reflen = strlen(ref);
+		struct update_ref_record *rec = update_refs.items[i].util;
+
+		/* OID already stored as updated. */
+		if (!is_null_oid(&rec->after))
+			continue;
+
+		for (j = 0; !found && j < todo_list->total_nr; j++) {
+			struct todo_item *item = &todo_list->items[j];
+			const char *arg = todo_list->buf.buf + item->arg_offset;
+
+			if (item->command != TODO_UPDATE_REF)
+				continue;
+
+			if (item->arg_len != reflen ||
+			    strncmp(arg, ref, reflen))
+				continue;
+
+			found = 1;
+		}
+
+		if (!found) {
+			free(update_refs.items[i].string);
+			free(update_refs.items[i].util);
+
+			update_refs.nr--;
+			MOVE_ARRAY(update_refs.items + i, update_refs.items + i + 1, update_refs.nr - i);
+
+			updated = 1;
+			i--;
+		}
+	}
+
+	/*
+	 * For each todo_item, check if its ref is in the update_refs list.
+	 * If not, then add it as an un-updated ref.
+	 */
+	for (i = 0; i < todo_list->total_nr; i++) {
+		struct todo_item *item = &todo_list->items[i];
+		const char *arg = todo_list->buf.buf + item->arg_offset;
+		int j, found = 0;
+
+		if (item->command != TODO_UPDATE_REF)
+			continue;
+
+		for (j = 0; !found && j < update_refs.nr; j++) {
+			const char *ref = update_refs.items[j].string;
+
+			found = strlen(ref) == item->arg_len &&
+				!strncmp(ref, arg, item->arg_len);
+		}
+
+		if (!found) {
+			struct string_list_item *inserted;
+			struct strbuf argref = STRBUF_INIT;
+
+			strbuf_add(&argref, arg, item->arg_len);
+			inserted = string_list_insert(&update_refs, argref.buf);
+			inserted->util = init_update_ref_record(argref.buf);
+			strbuf_release(&argref);
+			updated = 1;
+		}
+	}
+
+	if (updated)
+		write_update_refs_state(&update_refs);
+	string_list_clear(&update_refs, 1);
+}
+
 static int do_update_ref(struct repository *r, const char *refname)
 {
 	struct string_list_item *item;
diff --git a/sequencer.h b/sequencer.h
index e671d7e0743..98f3bcc1658 100644
--- a/sequencer.h
+++ b/sequencer.h
@@ -132,6 +132,18 @@ void todo_list_release(struct todo_list *todo_list);
 const char *todo_item_get_arg(struct todo_list *todo_list,
 			      struct todo_item *item);
 
+/*
+ * Parse the update-refs file for the current rebase, then remove the
+ * refs that do not appear in the todo_list (and have not had updated
+ * values stored) and add refs that are in the todo_list but not
+ * represented in the update-refs file.
+ *
+ * If there are changes to the update-refs list, then write the new state
+ * to disk.
+ */
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list);
+
 /* Call this to setup defaults before parsing command line options */
 void sequencer_init_config(struct replay_opts *opts);
 int sequencer_pick_revisions(struct repository *repo,
diff --git a/t/lib-rebase.sh b/t/lib-rebase.sh
index ec6b9b107da..b57541356bd 100644
--- a/t/lib-rebase.sh
+++ b/t/lib-rebase.sh
@@ -207,3 +207,18 @@ check_reworded_commits () {
 		>reword-log &&
 	test_cmp reword-expected reword-log
 }
+
+# usage: set_replace_editor <file>
+#
+# Replace the todo file with the exact contents of the given file.
+set_replace_editor () {
+	cat >script <<-\EOF &&
+	cat FILENAME >"$1"
+
+	echo 'rebase -i script after editing:'
+	cat "$1"
+	EOF
+
+	sed -e "s/FILENAME/$1/g" <script | write_script fake-editor.sh &&
+	test_set_editor "$(pwd)/fake-editor.sh"
+}
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 2fffcf5d5fb..bebf9b4def7 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1834,6 +1834,145 @@ test_expect_success '--update-refs updates refs correctly' '
 	test_cmp_rev HEAD refs/heads/no-conflict-branch
 '
 
+test_expect_success 'respect user edits to update-ref steps' '
+	git checkout -B update-refs-break no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	git branch -f unseen base &&
+
+	# First, we will add breaks to the expected todo file
+	cat >fake-todo-1 <<-EOF &&
+	pick $(git rev-parse HEAD~3)
+	break
+	update-ref refs/heads/second
+	update-ref refs/heads/first
+
+	pick $(git rev-parse HEAD~2)
+	pick $(git rev-parse HEAD~1)
+	update-ref refs/heads/third
+
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/no-conflict-branch
+	EOF
+
+	# Second, we will drop some update-refs commands (and move one)
+	cat >fake-todo-2 <<-EOF &&
+	update-ref refs/heads/second
+
+	pick $(git rev-parse HEAD~2)
+	update-ref refs/heads/third
+	pick $(git rev-parse HEAD~1)
+	break
+
+	pick $(git rev-parse HEAD)
+	EOF
+
+	# Third, we will:
+	# * insert a new one (new-branch),
+	# * re-add an old one (first), and
+	# * add a second instance of a previously-stored one (second)
+	cat >fake-todo-3 <<-EOF &&
+	update-ref refs/heads/unseen
+	update-ref refs/heads/new-branch
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/first
+	update-ref refs/heads/second
+	EOF
+
+	(
+		set_replace_editor fake-todo-1 &&
+		git rebase -i --update-refs primary &&
+
+		# These branches are currently locked.
+		for b in first second third no-conflict-branch
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		set_replace_editor fake-todo-2 &&
+		git rebase --edit-todo &&
+
+		# These branches are currently locked.
+		for b in second third
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		# These branches are currently unlocked for checkout.
+		for b in first no-conflict-branch
+		do
+			git worktree add wt-$b $b &&
+			git worktree remove wt-$b || return 1
+		done &&
+
+		git rebase --continue &&
+
+		set_replace_editor fake-todo-3 &&
+		git rebase --edit-todo &&
+
+		# These branches are currently locked.
+		for b in second third first unseen
+		do
+			test_must_fail git branch -f $b base || return 1
+		done &&
+
+		# These branches are currently unlocked for checkout.
+		for b in no-conflict-branch
+		do
+			git worktree add wt-$b $b &&
+			git worktree remove wt-$b || return 1
+		done &&
+
+		git rebase --continue
+	) &&
+
+	test_cmp_rev HEAD~2 refs/heads/third &&
+	test_cmp_rev HEAD~1 refs/heads/unseen &&
+	test_cmp_rev HEAD~1 refs/heads/new-branch &&
+	test_cmp_rev HEAD refs/heads/first &&
+	test_cmp_rev HEAD refs/heads/second &&
+	test_cmp_rev HEAD refs/heads/no-conflict-branch
+'
+
+test_expect_success '--update-refs: check failed ref update' '
+	git checkout -B update-refs-error no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~2 &&
+	git branch -f third HEAD~1 &&
+
+	cat >fake-todo <<-EOF &&
+	pick $(git rev-parse HEAD~3)
+	break
+	update-ref refs/heads/first
+
+	pick $(git rev-parse HEAD~2)
+	update-ref refs/heads/second
+
+	pick $(git rev-parse HEAD~1)
+	update-ref refs/heads/third
+
+	pick $(git rev-parse HEAD)
+	update-ref refs/heads/no-conflict-branch
+	EOF
+
+	(
+		set_replace_editor fake-todo &&
+		git rebase -i --update-refs base
+	) &&
+
+	# At this point, the values of first, second, and third are
+	# recorded in the update-refs file. We will force-update the
+	# "second" ref, but "git branch -f" will not work because of
+	# the lock in the update-refs file.
+	git rev-parse third >.git/refs/heads/second &&
+
+	git rebase --continue 2>err &&
+	grep "update_ref failed for ref '\''refs/heads/second'\''" err
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
-- 
gitgitgadget

Re: [PATCH v5 09/12] sequencer: rewrite update-refs as user edits todo list

[Phillip Wood]

Hi Stolee

On 19/07/2022 19:33, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <derrickstolee@github.com>
> 
> +void todo_list_filter_update_refs(struct repository *r,
> +				  struct todo_list *todo_list)
> +{
> +	int i;
> +	int updated = 0;
> +	struct string_list update_refs = STRING_LIST_INIT_DUP;
> +
> +	sequencer_get_update_refs_state(r->gitdir, &update_refs);
> +
> +	/*
> +	 * For each item in the update_refs list, if it has no updated
> +	 * value and does not appear in the todo_list, then remove it
> +	 * from the update_refs list.
> +	 */
> +	for (i = 0; i < update_refs.nr; i++) {
> +		int j;
> +		int found = 0;
> +		const char *ref = update_refs.items[i].string;
> +		size_t reflen = strlen(ref);
> +		struct update_ref_record *rec = update_refs.items[i].util;
> +
> +		/* OID already stored as updated. */
> +		if (!is_null_oid(&rec->after))
> +			continue;
> +
> +		for (j = 0; !found && j < todo_list->total_nr; j++) {
> +			struct todo_item *item = &todo_list->items[j];
> +			const char *arg = todo_list->buf.buf + item->arg_offset;
> +
> +			if (item->command != TODO_UPDATE_REF)
> +				continue;
> +
> +			if (item->arg_len != reflen ||
> +			    strncmp(arg, ref, reflen))
> +				continue;
> +
> +			found = 1;
> +		}
> +
> +		if (!found) {
> +			free(update_refs.items[i].string);
> +			free(update_refs.items[i].util);
> +
> +			update_refs.nr--;
> +			MOVE_ARRAY(update_refs.items + i, update_refs.items + i + 1, update_refs.nr - i);
> +
> +			updated = 1;
> +			i--;
> +		}
> +	}
> +
> +	/*
> +	 * For each todo_item, check if its ref is in the update_refs list.
> +	 * If not, then add it as an un-updated ref.
> +	 */
> +	for (i = 0; i < todo_list->total_nr; i++) {
> +		struct todo_item *item = &todo_list->items[i];
> +		const char *arg = todo_list->buf.buf + item->arg_offset;
> +		int j, found = 0;
> +
> +		if (item->command != TODO_UPDATE_REF)
> +			continue;
> +
> +		for (j = 0; !found && j < update_refs.nr; j++) {
> +			const char *ref = update_refs.items[j].string;
> +
> +			found = strlen(ref) == item->arg_len &&
> +				!strncmp(ref, arg, item->arg_len);
> +		}
> +
> +		if (!found) {
> +			struct string_list_item *inserted;
> +			struct strbuf argref = STRBUF_INIT;
> +
> +			strbuf_add(&argref, arg, item->arg_len);
> +			inserted = string_list_insert(&update_refs, argref.buf);
> +			inserted->util = init_update_ref_record(argref.buf);
> +			strbuf_release(&argref);
> +			updated = 1;
> +		}
> +	}
> +
> +	if (updated)
> +		write_update_refs_state(&update_refs);

We should return an error if write_update_refs_state() fails

Best Wishes

Phillip

[PATCH v5 10/12] rebase: add rebase.updateRefs config option

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

The previous change added the --update-refs command-line option.  For
users who always want this mode, create the rebase.updateRefs config
option which behaves the same way as rebase.autoSquash does with the
--autosquash option.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/config/rebase.txt |  3 +++
 Documentation/git-rebase.txt    |  3 +++
 builtin/rebase.c                |  5 +++++
 t/t3404-rebase-interactive.sh   | 14 ++++++++++++++
 4 files changed, 25 insertions(+)

diff --git a/Documentation/config/rebase.txt b/Documentation/config/rebase.txt
index 8c979cb20f2..f19bd0e0407 100644
--- a/Documentation/config/rebase.txt
+++ b/Documentation/config/rebase.txt
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt
index dfd847d6615..305255f1e15 100644
--- a/Documentation/git-rebase.txt
+++ b/Documentation/git-rebase.txt
@@ -614,6 +614,9 @@ start would be overridden by the presence of
 	Automatically force-update any branches that point to commits that
 	are being rebased. Any branches that are checked out in a worktree
 	are not updated in this way.
++
+If the configuration variable `rebase.updateRefs` is set, then this option
+can be used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
diff --git a/builtin/rebase.c b/builtin/rebase.c
index fd64897132a..dbb91895687 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index bebf9b4def7..1a27bb0626d 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1772,6 +1772,12 @@ test_expect_success '--update-refs adds label and update-ref commands' '
 		EOF
 
 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
+
 		test_cmp expect todo
 	)
 '
@@ -1813,6 +1819,14 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 				   --rebase-merges=rebase-cousins \
 				   --update-refs primary >todo &&
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i \
+				   --rebase-merges=rebase-cousins \
+				   primary >todo &&
+
 		test_cmp expect todo
 	)
 '
-- 
gitgitgadget

[PATCH v5 11/12] sequencer: ignore HEAD ref under --update-refs

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When using the 'git rebase -i --update-refs' option, the todo list is
populated with 'update-ref' commands for all tip refs in the history
that is being rebased. Refs that are checked out by some worktree are
instead added as a comment to warn the user that they will not be
updated.

Until now, this included the HEAD ref, which is being updated by the
rebase process itself, regardless of the --update-refs option. Remove
the comment in this case by ignoring any decorations that match the HEAD
ref.

Reported-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/sequencer.c b/sequencer.c
index 67812c0294f..1602649332b 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -5853,12 +5853,25 @@ static int add_decorations_to_list(const struct commit *commit,
 				   struct todo_add_branch_context *ctx)
 {
 	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+	const char *head_ref = resolve_ref_unsafe("HEAD",
+						  RESOLVE_REF_READING,
+						  NULL,
+						  NULL);
 
 	while (decoration) {
 		struct todo_item *item;
 		const char *path;
 		size_t base_offset = ctx->buf->len;
 
+		/*
+		 * If the branch is the current HEAD, then it will be
+		 * updated by the default rebase behavior.
+		 */
+		if (head_ref && !strcmp(head_ref, decoration->name)) {
+			decoration = decoration->next;
+			continue;
+		}
+
 		ALLOC_GROW(ctx->items,
 			ctx->items_nr + 1,
 			ctx->items_alloc);
-- 
gitgitgadget

[PATCH v5 12/12] sequencer: notify user of --update-refs activity

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <derrickstolee@github.com>

When the user runs 'git rebase -i --update-refs', the end message still
says only

  Successfully rebased and updated <HEAD-ref>.

Update the sequencer to collect the successful (and unsuccessful) ref
updates due to the --update-refs option, so the end message now says

  Successfully rebased and updated <HEAD-ref>.
  Updated the following refs with --update-refs:
	refs/heads/first
	refs/heads/third
  Failed to update the following refs with --update-refs:
	refs/heads/second

To test this output, we need to be very careful to format the expected
error to drop the leading tab characters. Also, we need to be aware that
the verbose output from 'git rebase' is writing progress lines which
don't use traditional newlines but clear the line after every progress
item is complete. When opening the error file in an editor, these lines
are visible, but when looking at the diff in a terminal those lines
disappear because of the characters that delete the previous characters.
Use 'sed' to clear those progress lines and clear the tabs so we can get
an exact match on our expected output.

Reported-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sequencer.c                   | 40 +++++++++++++++++++++++++++++------
 t/t3404-rebase-interactive.sh | 37 ++++++++++++++++++++++++++++----
 2 files changed, 67 insertions(+), 10 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 1602649332b..96e1d58ede8 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -4261,26 +4261,54 @@ static int do_update_ref(struct repository *r, const char *refname)
 	return 0;
 }
 
-static int do_update_refs(struct repository *r)
+static int do_update_refs(struct repository *r, int quiet)
 {
 	int res = 0;
 	struct string_list_item *item;
 	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
 	struct ref_store *refs = get_main_ref_store(r);
+	struct strbuf update_msg = STRBUF_INIT;
+	struct strbuf error_msg = STRBUF_INIT;
 
 	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
 		return res;
 
 	for_each_string_list_item(item, &refs_to_oids) {
 		struct update_ref_record *rec = item->util;
+		int loop_res;
 
-		res |= refs_update_ref(refs, "rewritten during rebase",
-				       item->string,
-				       &rec->after, &rec->before,
-				       0, UPDATE_REFS_MSG_ON_ERR);
+		loop_res = refs_update_ref(refs, "rewritten during rebase",
+					   item->string,
+					   &rec->after, &rec->before,
+					   0, UPDATE_REFS_MSG_ON_ERR);
+		res |= loop_res;
+
+		if (quiet)
+			continue;
+
+		if (loop_res)
+			strbuf_addf(&error_msg, "\t%s\n", item->string);
+		else
+			strbuf_addf(&update_msg, "\t%s\n", item->string);
+	}
+
+	if (!quiet &&
+	    (update_msg.len || error_msg.len)) {
+		fprintf(stderr,
+			_("Updated the following refs with %s:\n%s"),
+			"--update-refs",
+			update_msg.buf);
+
+		if (res)
+			fprintf(stderr,
+				_("Failed to update the following refs with %s:\n%s"),
+				"--update-refs",
+				error_msg.buf);
 	}
 
 	string_list_clear(&refs_to_oids, 1);
+	strbuf_release(&update_msg);
+	strbuf_release(&error_msg);
 	return res;
 }
 
@@ -4800,7 +4828,7 @@ cleanup_head_ref:
 		strbuf_release(&buf);
 		strbuf_release(&head_ref);
 
-		if (do_update_refs(r))
+		if (do_update_refs(r, opts->quiet))
 			return -1;
 	}
 
diff --git a/t/t3404-rebase-interactive.sh b/t/t3404-rebase-interactive.sh
index 1a27bb0626d..688b01e3eb6 100755
--- a/t/t3404-rebase-interactive.sh
+++ b/t/t3404-rebase-interactive.sh
@@ -1840,12 +1840,26 @@ test_expect_success '--update-refs updates refs correctly' '
 	test_commit extra2 fileX &&
 	git commit --amend --fixup=L &&
 
-	git rebase -i --autosquash --update-refs primary &&
+	git rebase -i --autosquash --update-refs primary 2>err &&
 
 	test_cmp_rev HEAD~3 refs/heads/first &&
 	test_cmp_rev HEAD~3 refs/heads/second &&
 	test_cmp_rev HEAD~1 refs/heads/third &&
-	test_cmp_rev HEAD refs/heads/no-conflict-branch
+	test_cmp_rev HEAD refs/heads/no-conflict-branch &&
+
+	cat >expect <<-\EOF &&
+	Successfully rebased and updated refs/heads/update-refs.
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/second
+		refs/heads/third
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 test_expect_success 'respect user edits to update-ref steps' '
@@ -1983,8 +1997,23 @@ test_expect_success '--update-refs: check failed ref update' '
 	# the lock in the update-refs file.
 	git rev-parse third >.git/refs/heads/second &&
 
-	git rebase --continue 2>err &&
-	grep "update_ref failed for ref '\''refs/heads/second'\''" err
+	test_must_fail git rebase --continue 2>err &&
+	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
+
+	cat >expect <<-\EOF &&
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/third
+	Failed to update the following refs with --update-refs:
+		refs/heads/second
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	tail -n 6 err >err.last &&
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err.last >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 # This must be the last test in this file
-- 
gitgitgadget

Re: [PATCH v5 00/12] rebase: update branches in multi-part topic

[Elijah Newren]

On Tue, Jul 19, 2022 at 11:33 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> This series is based on ds/branch-checked-out.
>
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
>
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'update-ref' commands into the
> TODO list. This is powered by the commit decoration machinery.
>
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
>
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> update-ref refs/heads/bundle-redo/fetch
>
> pick 1e3f6546632 clone: add --bundle-uri option
> pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
> update-ref refs/heads/bundle-redo/clone
>
> pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
> pick 3029c3aca15 bundle-uri: create base key-value pair parsing
> pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
> pick 92625a47673 bundle-uri: unit test "key=value" parsing
> pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
> pick 9d6809a8d53 bundle-uri: parse bundle list in config format
> pick 287a732b54c bundle-uri: fetch a list of bundles
> update-ref refs/heads/bundle-redo/list
>
> pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
> pick 520204dcd1c bundle-uri client: add minimal NOOP client
> pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
> pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
> pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
> pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
> pick df255000b7e bundle-uri: download bundles from an advertised list
> pick d71beabf199 clone: unbundle the advertised bundles
> pick c9578391976 t5601: basic bundle URI tests
> # Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'
>
> update-ref refs/heads/bundle-redo/advertise
>
>
> Here is an outline of the series:
>
>  * Patch 1 updates some tests for branch_checked_out() to use 'git bisect'
>    and 'git rebase' as black-boxes instead of manually editing files inside
>    $GIT_DIR. (Thanks, Junio!)
>  * Patch 2 updates some tests for branch_checked_out() for the 'apply'
>    backend.
>  * Patch 3 updates branch_checked_out() to parse the
>    rebase-merge/update-refs file to block concurrent ref updates and
>    checkouts on branches selected by --update-refs.
>  * Patch 4 updates the todo list documentation to remove some unnecessary
>    dots in the 'merge' command. This makes it consistent with the 'fixup'
>    command before we document the 'update-ref' command.
>  * Patch 5 updates the definition of todo_command_info to use enum values as
>    array indices.
>  * Patches 6-8 implement the --update-refs logic itself.
>  * Patch 9 specifically updates the update-refs file every time the user
>    edits the todo-list (Thanks Phillip!)
>  * Patch 10 adds the rebase.updateRefs config option similar to
>    rebase.autoSquash.
>  * Patch 11 ignores the HEAD ref when creating the todo list instead of
>    making a comment (Thanks Elijah!)
>  * Patch 12 adds messaging to the end of the rebase stating which refs were
>    updated (Thanks Elijah!)
>
> During review, we have identified some areas that would be good for
> #leftoverbits:
>
>  * Warn the user when they add an 'update-ref ' command but is checked out
>    in another worktree.
>  * The checks in patch 9 are quadratic. They could be sped up using
>    hashtables.
>  * Consider whether we should include an 'update-ref ' command for the HEAD
>    ref, so that all refs are updated in the same way. This might help
>    confused users.

Not necessarily so they are updated in the same way; the behind the
scenes mechanism could perhaps still be different.  Just so that if
the user looks for the "list of things being updated" they don't get
surprised that HEAD is missing.

>  * The error message for failed ref updates could include information on the
>    commit IDs that would have been used. This can help the user fix the
>    situation by updating the refs manually.
>  * Modify the --update-refs option from a boolean to an
>    optionally-string-parameter that specifies refspecs for the 'update-ref'
>    commands.

refspecs?  Is that the term you really mean here?


> Updates in v5
> =============
>
>  * Rename 'wt_dir' to 'wt_git_dir' for clarity.
>  * The documented behavior around 'fixup!' and 'squash!' commits was
>    incorrect, so update the commit message, documentation, and test to
>    demonstrate the actual behavior.
>  * Use CALLOC_ARRAY() to be more idiomatic.
>  * Be much more careful about propagating errors.
>  * Commit message typo: "We an" to "We can"
>  * Remove unnecessary null OID check when writing refs, since those would
>    already be removed by a previous step.

Thanks, I've read over the range-diff and these changes look good to
me.  One thing I'm curious about (sorry to bring this up so late):
"pick" commands come with the old commit hash.  Perhaps the update-ref
commands should too?  (e.g. "update-ref refs/heads/topic from
<OLDHASH>")

Re: [PATCH v5 00/12] rebase: update branches in multi-part topic

[Derrick Stolee]

On 7/21/2022 12:35 AM, Elijah Newren wrote:
> On Tue, Jul 19, 2022 at 11:33 AM Derrick Stolee via GitGitGadget

>> During review, we have identified some areas that would be good for
>> #leftoverbits:
>>
>>  * Warn the user when they add an 'update-ref ' command but is checked out
>>    in another worktree.
>>  * The checks in patch 9 are quadratic. They could be sped up using
>>    hashtables.
>>  * Consider whether we should include an 'update-ref ' command for the HEAD
>>    ref, so that all refs are updated in the same way. This might help
>>    confused users.
> 
> Not necessarily so they are updated in the same way; the behind the
> scenes mechanism could perhaps still be different.  Just so that if
> the user looks for the "list of things being updated" they don't get
> surprised that HEAD is missing.

Noted.
 
>>  * The error message for failed ref updates could include information on the
>>    commit IDs that would have been used. This can help the user fix the
>>    situation by updating the refs manually.
>>  * Modify the --update-refs option from a boolean to an
>>    optionally-string-parameter that specifies refspecs for the 'update-ref'
>>    commands.
> 
> refspecs?  Is that the term you really mean here?

Probably "ref namespaces" or "ref prefixes" would be better.

>> Updates in v5
>> =============
>>
>>  * Rename 'wt_dir' to 'wt_git_dir' for clarity.
>>  * The documented behavior around 'fixup!' and 'squash!' commits was
>>    incorrect, so update the commit message, documentation, and test to
>>    demonstrate the actual behavior.
>>  * Use CALLOC_ARRAY() to be more idiomatic.
>>  * Be much more careful about propagating errors.
>>  * Commit message typo: "We an" to "We can"
>>  * Remove unnecessary null OID check when writing refs, since those would
>>    already be removed by a previous step.
> 
> Thanks, I've read over the range-diff and these changes look good to
> me.  One thing I'm curious about (sorry to bring this up so late):
> "pick" commands come with the old commit hash.  Perhaps the update-ref
> commands should too?  (e.g. "update-ref refs/heads/topic from
> <OLDHASH>")

I don't personally see the value here other than to make it harder
for someone to add new commands (and confusing when wanting to
create a brand new ref). We could consider adding a comment in the
future without any backwards compatibility issues:

	update-ref refs/heads/my-ref # was 0123dead

Thanks,
-Stolee

Re: [PATCH v5 00/12] rebase: update branches in multi-part topic

[Elijah Newren]

On Thu, Jul 21, 2022 at 5:13 AM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 7/21/2022 12:35 AM, Elijah Newren wrote:
>
> > One thing I'm curious about (sorry to bring this up so late):
> > "pick" commands come with the old commit hash.  Perhaps the update-ref
> > commands should too?  (e.g. "update-ref refs/heads/topic from
> > <OLDHASH>")
>
> I don't personally see the value here other than to make it harder
> for someone to add new commands (and confusing when wanting to
> create a brand new ref).

I apologize if my late query caused frustration.  I was thinking it
made more sense to ask now than later and was genuinely curious for
thoughts on the idea, but perhaps it'd have been better if I just
didn't bring it up.  Sorry.

But, if you're curious why I was thinking about it...


There are four ways I can think of to handle the <oldvalue>: (A)
include it in the "update-ref" instruction, (B) manually get the
<oldvalue> at program invocation, (C) accept an increased race-window
by not grabbing <oldvalue> until the line is parsed, or (D) ignore
races and just don't provide <oldvalue> when updating refs.  If one
chooses (B), they can either (B1) pre-parse the entire todo list and
look up the current values of any refs mentioned in an update-ref
instruction, or (B2) record the value of all existing refs at
invocation.

Rebase is already paying the cost for pre-parsing the entire todo list
(i.e. B1).  In fact, you _also_ preemptively locking the refs
mentioned.  So, in a sense, rebase is already covered. However:
  * adding such a thing could potentially remove a small race window
(perhaps the user takes a while to edit the todo list, and the ref
somehow gets updated before they even finish that step; being able to
specify the oldref might help in such a case).
  * existing commands (pick, merge, etc.) give the oldrefs (or labels
to them), so having update-ref also do so might provide some "feel
good" consistency.

Perhaps those reasons are pretty weak for the rebase case, given you
already do B1.  However, `git replay` isn't going to do B1, and nor
will it preemptively lock refs.  I see `git replay` as a patch-based
analogue to the snapshot-based tools of fast-export and fast-import.
(fast-import doesn't pre-parse its entire input before starting to
process the first commit, nor does it preemptively lock refs.)  B2 is
not appealing because some repositories have gargantuan numbers of
refs.  Also, the update-ref-during-edit-of-todo rationale is more
important for `git replay`, because I want `git replay` to accept a
filename containing a todo list in lieu of a range of revisions to
replay -- perhaps users spent quite some time or programming effort
generating their todo list and it took some time to generate/review.
And that further undercuts both B1 and B2.  So, between all those
reasons, (A) is very appealing to me.

All that said, `git replay` already has other planned differences in
its todo list (e.g. a "reset" directive must be the first non-comment
directive, there's no separate "merge" directive, etc.), so if I have
to add another one, it's no big deal.

Your point about specifying <oldref> being seen as friction for some
users is well taken, and I'm thinking of making the "from <oldref>"
part of the instruction optional (though documenting that users are
accepting some race risk by switching from (A) to (C) when they omit
it).

> We could consider adding a comment in the
> future without any backwards compatibility issues:
>
>         update-ref refs/heads/my-ref # was 0123dead

Yes, this could certainly be added later (though none of my reasons
for wanting <oldref> were geared around pointers for the end user, so
this form wouldn't be helpful to me).

Also, if "from <oldref>" is optional, it could also be added later.


If you've read this far, let me just take a minute to again thank you
for your work on this --update-refs option to rebase.  It's some cool
work.  Also, thanks for listening to my random ideas and musings along
the way.  :-)

Re: [PATCH v5 00/12] rebase: update branches in multi-part topic

[Derrick Stolee]

On 7/21/2022 3:43 PM, Elijah Newren wrote:
> On Thu, Jul 21, 2022 at 5:13 AM Derrick Stolee <derrickstolee@github.com> wrote:
>>
>> On 7/21/2022 12:35 AM, Elijah Newren wrote:
>>
>>> One thing I'm curious about (sorry to bring this up so late):
>>> "pick" commands come with the old commit hash.  Perhaps the update-ref
>>> commands should too?  (e.g. "update-ref refs/heads/topic from
>>> <OLDHASH>")
>>
>> I don't personally see the value here other than to make it harder
>> for someone to add new commands (and confusing when wanting to
>> create a brand new ref).
> 
> I apologize if my late query caused frustration. 

I intended this as "I don't understand, please enlighten me", but
looking back now it definitely sounds more frustrated. Sorry about
that.

> I was thinking it
> made more sense to ask now than later and was genuinely curious for
> thoughts on the idea, but perhaps it'd have been better if I just
> didn't bring it up.  Sorry.
 
It's definitely the time to bring it up, because it will be harder
to change it once it's shipped.

> But, if you're curious why I was thinking about it...
> 
> 
> There are four ways I can think of to handle the <oldvalue>: (A)
> include it in the "update-ref" instruction, (B) manually get the
> <oldvalue> at program invocation, (C) accept an increased race-window
> by not grabbing <oldvalue> until the line is parsed, or (D) ignore
> races and just don't provide <oldvalue> when updating refs.  If one
> chooses (B), they can either (B1) pre-parse the entire todo list and
> look up the current values of any refs mentioned in an update-ref
> instruction, or (B2) record the value of all existing refs at
> invocation.
> 
> Rebase is already paying the cost for pre-parsing the entire todo list
> (i.e. B1).  In fact, you _also_ preemptively locking the refs
> mentioned.  So, in a sense, rebase is already covered. However:
>   * adding such a thing could potentially remove a small race window
> (perhaps the user takes a while to edit the todo list, and the ref
> somehow gets updated before they even finish that step; being able to
> specify the oldref might help in such a case).
>   * existing commands (pick, merge, etc.) give the oldrefs (or labels
> to them), so having update-ref also do so might provide some "feel
> good" consistency.
> 
> Perhaps those reasons are pretty weak for the rebase case, given you
> already do B1.  However, `git replay` isn't going to do B1, and nor
> will it preemptively lock refs.  I see `git replay` as a patch-based
> analogue to the snapshot-based tools of fast-export and fast-import.
> (fast-import doesn't pre-parse its entire input before starting to
> process the first commit, nor does it preemptively lock refs.)  B2 is
> not appealing because some repositories have gargantuan numbers of
> refs.  Also, the update-ref-during-edit-of-todo rationale is more
> important for `git replay`, because I want `git replay` to accept a
> filename containing a todo list in lieu of a range of revisions to
> replay -- perhaps users spent quite some time or programming effort
> generating their todo list and it took some time to generate/review.
> And that further undercuts both B1 and B2.  So, between all those
> reasons, (A) is very appealing to me.
> 
> All that said, `git replay` already has other planned differences in
> its todo list (e.g. a "reset" directive must be the first non-comment
> directive, there's no separate "merge" directive, etc.), so if I have
> to add another one, it's no big deal.

I'm happy that you're working on 'git replay'. I'm excited to see how
it works. Also, it would be helpful to keep these things in line as
much as possible.

> Your point about specifying <oldref> being seen as friction for some
> users is well taken, and I'm thinking of making the "from <oldref>"
> part of the instruction optional (though documenting that users are
> accepting some race risk by switching from (A) to (C) when they omit
> it).
>
>> We could consider adding a comment in the
>> future without any backwards compatibility issues:
>>
>>         update-ref refs/heads/my-ref # was 0123dead
> 
> Yes, this could certainly be added later (though none of my reasons
> for wanting <oldref> were geared around pointers for the end user, so
> this form wouldn't be helpful to me).
> 
> Also, if "from <oldref>" is optional, it could also be added later.

The optional value makes this something that we can add later, which
I think is good flexibility. I was thinking about doing the comment
thing as a follow-up (in addition to Phillip's error mode comments)
but maybe I'll explore this direction instead.
 
> If you've read this far, let me just take a minute to again thank you
> for your work on this --update-refs option to rebase.  It's some cool
> work.  Also, thanks for listening to my random ideas and musings along
> the way.  :-)

I really appreciate the attention you've given it. It's already
improved a lot due to your help.

Thanks,
-Stolee

Re: [PATCH v5 00/12] rebase: update branches in multi-part topic

[Phillip Wood]

Hi Stolee

I'm afraid I've only had time to have a quick look at patches 8 & 9. The 
error propagation is looking better but I did notice a couple of calls 
to write_update_refs_state() where we don't propagate the return value.

Best Wishes

Phillip

On 19/07/2022 19:33, Derrick Stolee via GitGitGadget wrote:
> This series is based on ds/branch-checked-out.
> 
> This is a feature I've wanted for quite a while. When working on the sparse
> index topic, I created a long RFC that actually broke into three topics for
> full review upstream. These topics were sequential, so any feedback on an
> earlier one required updates to the later ones. I would work on the full
> feature and use interactive rebase to update the full list of commits.
> However, I would need to update the branches pointing to those sub-topics.
> 
> This series adds a new --update-refs option to 'git rebase' (along with a
> rebase.updateRefs config option) that adds 'update-ref' commands into the
> TODO list. This is powered by the commit decoration machinery.
> 
> As an example, here is my in-progress bundle URI RFC split into subtopics as
> they appear during the TODO list of a git rebase -i --update-refs:
> 
> pick 2d966282ff3 docs: document bundle URI standard
> pick 31396e9171a remote-curl: add 'get' capability
> pick 54c6ab70f67 bundle-uri: create basic file-copy logic
> pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
> pick 6adaf842684 fetch: add --bundle-uri option
> pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
> update-ref refs/heads/bundle-redo/fetch
> 
> pick 1e3f6546632 clone: add --bundle-uri option
> pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
> update-ref refs/heads/bundle-redo/clone
> 
> pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
> pick 3029c3aca15 bundle-uri: create base key-value pair parsing
> pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
> pick 92625a47673 bundle-uri: unit test "key=value" parsing
> pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
> pick 9d6809a8d53 bundle-uri: parse bundle list in config format
> pick 287a732b54c bundle-uri: fetch a list of bundles
> update-ref refs/heads/bundle-redo/list
> 
> pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
> pick 520204dcd1c bundle-uri client: add minimal NOOP client
> pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
> pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
> pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
> pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
> pick df255000b7e bundle-uri: download bundles from an advertised list
> pick d71beabf199 clone: unbundle the advertised bundles
> pick c9578391976 t5601: basic bundle URI tests
> # Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'
> 
> update-ref refs/heads/bundle-redo/advertise
> 
> 
> Here is an outline of the series:
> 
>   * Patch 1 updates some tests for branch_checked_out() to use 'git bisect'
>     and 'git rebase' as black-boxes instead of manually editing files inside
>     $GIT_DIR. (Thanks, Junio!)
>   * Patch 2 updates some tests for branch_checked_out() for the 'apply'
>     backend.
>   * Patch 3 updates branch_checked_out() to parse the
>     rebase-merge/update-refs file to block concurrent ref updates and
>     checkouts on branches selected by --update-refs.
>   * Patch 4 updates the todo list documentation to remove some unnecessary
>     dots in the 'merge' command. This makes it consistent with the 'fixup'
>     command before we document the 'update-ref' command.
>   * Patch 5 updates the definition of todo_command_info to use enum values as
>     array indices.
>   * Patches 6-8 implement the --update-refs logic itself.
>   * Patch 9 specifically updates the update-refs file every time the user
>     edits the todo-list (Thanks Phillip!)
>   * Patch 10 adds the rebase.updateRefs config option similar to
>     rebase.autoSquash.
>   * Patch 11 ignores the HEAD ref when creating the todo list instead of
>     making a comment (Thanks Elijah!)
>   * Patch 12 adds messaging to the end of the rebase stating which refs were
>     updated (Thanks Elijah!)
> 
> During review, we have identified some areas that would be good for
> #leftoverbits:
> 
>   * Warn the user when they add an 'update-ref ' command but is checked out
>     in another worktree.
>   * The checks in patch 9 are quadratic. They could be sped up using
>     hashtables.
>   * Consider whether we should include an 'update-ref ' command for the HEAD
>     ref, so that all refs are updated in the same way. This might help
>     confused users.
>   * The error message for failed ref updates could include information on the
>     commit IDs that would have been used. This can help the user fix the
>     situation by updating the refs manually.
>   * Modify the --update-refs option from a boolean to an
>     optionally-string-parameter that specifies refspecs for the 'update-ref'
>     commands.
> 
> 
> Updates in v5
> =============
> 
>   * Rename 'wt_dir' to 'wt_git_dir' for clarity.
>   * The documented behavior around 'fixup!' and 'squash!' commits was
>     incorrect, so update the commit message, documentation, and test to
>     demonstrate the actual behavior.
>   * Use CALLOC_ARRAY() to be more idiomatic.
>   * Be much more careful about propagating errors.
>   * Commit message typo: "We an" to "We can"
>   * Remove unnecessary null OID check when writing refs, since those would
>     already be removed by a previous step.
> 
> 
> Updates in v4
> =============
> 
> This version took longer than I'd hoped (I had less time to work on it than
> anticipated) but it also has some major updates. These major updates are
> direct responses to the significant review this series has received. Thank
> you!
> 
>   * The update-refs file now stores "ref/before/after" triples (still
>     separated by lines). This allows us to store the "before" OID of a ref in
>     addition to the "after" that we will write to that ref at the end of the
>     rebase. This allows us to do a "force-with-lease" update. The
>     branch_checked_out() updates should prevent Git from updating those refs
>     while under the rebase, but older versions and third-party tools don't
>     have that protection.
>   * The update-refs file is updated with every update to the todo-list file.
>     This allows for some advanced changes to the file, including removing,
>     adding, and duplicating 'update-ref' commands.
>   * The message at the end of the rebase process now lists which refs were
>     updated with the update-ref steps. This includes any ref updates that
>     fail.
>   * The branch_checked_out() tests now use 'git bisect' and 'git rebase' as
>     black-boxes instead of testing their internals directly.
> 
> Here are the more minor updates:
> 
>   * Dropped an unnecessary stat() call.
>   * Updated commit messages to include extra details, based on confusion in
>     last round.
>   * The HEAD branch no longer appears as a comment line in the initial todo
>     list.
>   * The update-refs file is now written using a lockfile.
>   * Tests now use test_cmp_rev.
>   * A memory leak ('path' variable) is resolved.
> 
> 
> Updates in v3
> =============
> 
>   * The branch_checked_out() API was extracted to its own topic and is now
>     the ds/branch-checked-out branch. This series is now based on that one.
>   * The for_each_decoration() API was removed, since it became trivial once
>     it did not take a commit directly.
>   * The branch_checked_out() tests did not verify the rebase-apply data (for
>     the apply backend), so that is fixed.
>   * Instead of using the 'label' command and a final 'update-refs' command in
>     the todo list, use a new 'update-ref ' command. This command updates the
>     rebase-merge/update-refs file with the OID of HEAD at these steps. At the
>     very end of the rebase sequence, those refs are updated to the stored OID
>     values (assuming that they were not removed by the user, in which case we
>     notice that the OID is the null OID and we do nothing).
>   * New tests are added.
>   * The todo-list comment documentation has some new formatting updates, but
>     also includes a description of 'update-refs' in this version.
> 
> 
> Updates in v2
> =============
> 
> As recommended by the excellent feedback, I have removed the 'exec' commands
> in favor of the 'label' commands and a new 'update-refs' command at the very
> end. This way, there is only one step that updates all of the refs at the
> end instead of updating refs during the rebase. If a user runs 'git rebase
> --abort' in the middle, then their refs are still where they need to be.
> 
> Based on some of the discussion, it seemed like one way to do this would be
> to have an 'update-ref ' command that would take the place of these 'label'
> commands. However, this would require two things that make it a bit awkward:
> 
>   1. We would need to replicate the storage of those positions during the
>      rebase. 'label' already does this pretty well. I've added the
>      "for-update-refs/" label to help here.
>   2. If we want to close out all of the refs as the rebase is finishing, then
>      that "step" becomes invisible to the user (and a bit more complicated to
>      insert). Thus, the 'update-refs' step performs this action. If the user
>      wants to do things after that step, then they can do so by editing the
>      TODO list.
> 
> Other updates:
> 
>   * The 'keep_decorations' parameter was renamed to 'update_refs'.
>   * I added tests for --rebase-merges=rebase-cousins to show how these labels
>     interact with other labels and merge commands.
>   * I changed the order of the insertion of these update-refs labels to be
>     before the fixups are rearranged. This fixes a bug where the tip commit
>     is a fixup! so its decorations are never inspected (and they would be in
>     the wrong place even if they were). The fixup! commands are properly
>     inserted between a pick and its following label command. Tests
>     demonstrate this is correct.
>   * Numerous style choices are updated based on feedback.
> 
> Thank you for all of the detailed review and ideas in this space. I
> appreciate any more ideas that can make this feature as effective as it can
> be.
> 
> Thanks, -Stolee
> 
> Derrick Stolee (12):
>    t2407: test bisect and rebase as black-boxes
>    t2407: test branches currently using apply backend
>    branch: consider refs under 'update-refs'
>    rebase-interactive: update 'merge' description
>    sequencer: define array with enum values
>    sequencer: add update-ref command
>    rebase: add --update-refs option
>    rebase: update refs from 'update-ref' commands
>    sequencer: rewrite update-refs as user edits todo list
>    rebase: add rebase.updateRefs config option
>    sequencer: ignore HEAD ref under --update-refs
>    sequencer: notify user of --update-refs activity
> 
>   Documentation/config/rebase.txt |   3 +
>   Documentation/git-rebase.txt    |  10 +
>   branch.c                        |  13 +
>   builtin/rebase.c                |  10 +
>   rebase-interactive.c            |  15 +-
>   sequencer.c                     | 474 +++++++++++++++++++++++++++++++-
>   sequencer.h                     |  23 ++
>   t/lib-rebase.sh                 |  15 +
>   t/t2407-worktree-heads.sh       | 103 +++++--
>   t/t3404-rebase-interactive.sh   | 273 ++++++++++++++++++
>   10 files changed, 895 insertions(+), 44 deletions(-)
> 
> 
> base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v5
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v5
> Pull-Request: https://github.com/gitgitgadget/git/pull/1247
> 
> Range-diff vs v4:
> 
>    1:  9e53a27017a =  1:  9e53a27017a t2407: test bisect and rebase as black-boxes
>    2:  540a3be256f =  2:  540a3be256f t2407: test branches currently using apply backend
>    3:  bf301a054e3 !  3:  1089a0edb73 branch: consider refs under 'update-refs'
>       @@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-
>        + * rebase_path_update_refs() returns the path to this file for a given
>        + * worktree directory. For the current worktree, pass the_repository->gitdir.
>        + */
>       -+static char *rebase_path_update_refs(const char *wt_dir)
>       ++static char *rebase_path_update_refs(const char *wt_git_dir)
>        +{
>       -+	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
>       ++	return xstrfmt("%s/rebase-merge/update-refs", wt_git_dir);
>        +}
>        +
>         /*
>    4:  dec95681d2b =  4:  d1cce4f06aa rebase-interactive: update 'merge' description
>    5:  b2c09600918 =  5:  4c086d477f0 sequencer: define array with enum values
>    6:  fa7ecb718cf =  6:  7b3d6601960 sequencer: add update-ref command
>    7:  3ec2cc922f9 !  7:  7efb55e4f14 rebase: add --update-refs option
>       @@ Commit message
>            --fixup commit at the tip of the feature to apply correctly to the sub
>            branch, even if it is fixing up the most-recent commit in that part.
>        
>       -    One potential problem here is that refs decorating commits that are
>       -    already marked as "fixup!" or "squash!" will not be included in this
>       -    list. Generally, the reordering of the "fixup!" and "squash!" is likely
>       -    to change the relative order of these refs, so it is not recommended.
>       -    The workflow here is intended to allow these kinds of commits at the tip
>       -    of the rebased branch while the other sub branches come along for the
>       -    ride without intervention.
>       -
>            This change update the documentation and builtin to accept the
>            --update-refs option as well as updating the todo file with the
>            'update-ref' commands. Tests are added to ensure that these todo
>       @@ Documentation/git-rebase.txt: provided. Otherwise an explicit `--no-reschedule-f
>        +--no-update-refs::
>        +	Automatically force-update any branches that point to commits that
>        +	are being rebased. Any branches that are checked out in a worktree
>       -+	or point to a `squash! ...` or `fixup! ...` commit are not updated
>       -+	in this way.
>       ++	are not updated in this way.
>        +
>         INCOMPATIBLE OPTIONS
>         --------------------
>       @@ builtin/rebase.c: int cmd_rebase(int argc, const char **argv, const char *prefix
>         			 N_("move commits that begin with "
>         			    "squash!/fixup! under -i")),
>        +		OPT_BOOL(0, "update-refs", &options.update_refs,
>       -+			 N_("update local refs that point to commits "
>       ++			 N_("update branches that point to commits "
>        +			    "that are being rebased")),
>         		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
>         			N_("GPG-sign commits"),
>       @@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
>        +	git branch -f second HEAD~3 &&
>        +	git branch -f third HEAD~1 &&
>        +	git commit --allow-empty --fixup=third &&
>       ++	git branch -f is-not-reordered &&
>       ++	git commit --allow-empty --fixup=HEAD~4 &&
>        +	git branch -f shared-tip &&
>        +	(
>        +		set_cat_todo_editor &&
>        +
>        +		cat >expect <<-EOF &&
>        +		pick $(git log -1 --format=%h J) J
>       ++		fixup $(git log -1 --format=%h update-refs) fixup! J # empty
>        +		update-ref refs/heads/second
>        +		update-ref refs/heads/first
>        +		pick $(git log -1 --format=%h K) K
>        +		pick $(git log -1 --format=%h L) L
>       -+		fixup $(git log -1 --format=%h update-refs) fixup! L # empty
>       ++		fixup $(git log -1 --format=%h is-not-reordered) fixup! L # empty
>        +		update-ref refs/heads/third
>        +		pick $(git log -1 --format=%h M) M
>        +		update-ref refs/heads/no-conflict-branch
>       ++		update-ref refs/heads/is-not-reordered
>        +		update-ref refs/heads/shared-tip
>        +		EOF
>        +
>    8:  fb5f64c5201 !  8:  e7a91bdffbd rebase: update refs from 'update-ref' commands
>       @@ sequencer.c: struct update_ref_record {
>         
>        +static struct update_ref_record *init_update_ref_record(const char *ref)
>        +{
>       -+	struct update_ref_record *rec = xmalloc(sizeof(*rec));
>       ++	struct update_ref_record *rec;
>       ++
>       ++	CALLOC_ARRAY(rec, 1);
>        +
>        +	oidcpy(&rec->before, null_oid());
>        +	oidcpy(&rec->after, null_oid());
>       @@ sequencer.c: leave_merge:
>         
>        -static int do_update_ref(struct repository *r, const char *ref_name)
>        +static int write_update_refs_state(struct string_list *refs_to_oids)
>       - {
>       ++{
>        +	int result = 0;
>        +	struct lock_file lock = LOCK_INIT;
>        +	FILE *fp = NULL;
>       @@ sequencer.c: leave_merge:
>        +}
>        +
>        +static int do_update_ref(struct repository *r, const char *refname)
>       -+{
>       + {
>        +	struct string_list_item *item;
>        +	struct string_list list = STRING_LIST_INIT_DUP;
>        +
>       -+	sequencer_get_update_refs_state(r->gitdir, &list);
>       ++	if (sequencer_get_update_refs_state(r->gitdir, &list))
>       ++		return -1;
>        +
>        +	for_each_string_list_item(item, &list) {
>        +		if (!strcmp(item->string, refname)) {
>        +			struct update_ref_record *rec = item->util;
>       -+			read_ref("HEAD", &rec->after);
>       ++			if (read_ref("HEAD", &rec->after))
>       ++				return -1;
>        +			break;
>        +		}
>        +	}
>       @@ sequencer.c: leave_merge:
>        +	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
>        +	struct ref_store *refs = get_main_ref_store(r);
>        +
>       -+	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
>       ++	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
>       ++		return res;
>        +
>        +	for_each_string_list_item(item, &refs_to_oids) {
>        +		struct update_ref_record *rec = item->util;
>        +
>       -+		if (oideq(&rec->after, the_hash_algo->null_oid)) {
>       -+			/*
>       -+			 * Ref was not updated. User may have deleted the
>       -+			 * 'update-ref' step.
>       -+			 */
>       -+			continue;
>       -+		}
>       -+
>        +		res |= refs_update_ref(refs, "rewritten during rebase",
>        +				       item->string,
>        +				       &rec->after, &rec->before,
>       @@ sequencer.c: leave_merge:
>         {
>         	int i = todo_list->current;
>        @@ sequencer.c: cleanup_head_ref:
>       +
>       + 		strbuf_release(&buf);
>         		strbuf_release(&head_ref);
>       ++
>       ++		if (do_update_refs(r))
>       ++			return -1;
>         	}
>         
>       -+	do_update_refs(r);
>       -+
>         	/*
>       - 	 * Sequence of picks finished successfully; cleanup by
>       - 	 * removing the .git/sequencer directory
>        @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
>         
>         			sti = string_list_insert(&ctx->refs_to_oids,
>       @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
>         		}
>         
>         		item->offset_in_buf = base_offset;
>       +@@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
>       +  */
>       + static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>       + {
>       +-	int i;
>       ++	int i, res;
>       + 	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
>       + 	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
>       + 	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
>        @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
>         		}
>         	}
>         
>       -+	write_update_refs_state(&ctx.refs_to_oids);
>       ++	res = write_update_refs_state(&ctx.refs_to_oids);
>        +
>         	string_list_clear(&ctx.refs_to_oids, 1);
>       ++
>       ++	if (res) {
>       ++		/* we failed, so clean up the new list. */
>       ++		free(ctx.items);
>       ++		return res;
>       ++	}
>       ++
>         	free(todo_list->items);
>         	todo_list->items = ctx.items;
>       + 	todo_list->nr = ctx.items_nr;
>        
>         ## t/t2407-worktree-heads.sh ##
>        @@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
>    9:  29c7c76805a !  9:  95e2bbcedb1 sequencer: rewrite update-refs as user edits todo list
>       @@ Commit message
>        
>            We can test that this works by rewriting the todo-list several times in
>            the course of a rebase. Check that each ref is locked or unlocked for
>       -    updates after each todo-list update. We an also verify that the ref
>       +    updates after each todo-list update. We can also verify that the ref
>            update fails if a concurrent process updates one of the refs after the
>            rebase process records the "locked" ref location.
>        
>   10:  c0022d07579 ! 10:  a73b02568f3 rebase: add rebase.updateRefs config option
>       @@ Documentation/config/rebase.txt: rebase.autoStash::
>        
>         ## Documentation/git-rebase.txt ##
>        @@ Documentation/git-rebase.txt: start would be overridden by the presence of
>       + 	Automatically force-update any branches that point to commits that
>         	are being rebased. Any branches that are checked out in a worktree
>       - 	or point to a `squash! ...` or `fixup! ...` commit are not updated
>       - 	in this way.
>       + 	are not updated in this way.
>        ++
>        +If the configuration variable `rebase.updateRefs` is set, then this option
>        +can be used to override and disable this setting.
>   11:  d53b4ff2cee = 11:  2a6577974c7 sequencer: ignore HEAD ref under --update-refs
>   12:  d5cd4b49e46 ! 12:  ec080ce1e90 sequencer: notify user of --update-refs activity
>       @@ sequencer.c: static int do_update_ref(struct repository *r, const char *refname)
>        +	struct strbuf update_msg = STRBUF_INIT;
>        +	struct strbuf error_msg = STRBUF_INIT;
>         
>       - 	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
>       + 	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
>       + 		return res;
>         
>         	for_each_string_list_item(item, &refs_to_oids) {
>         		struct update_ref_record *rec = item->util;
>        +		int loop_res;
>         
>       - 		if (oideq(&rec->after, the_hash_algo->null_oid)) {
>       - 			/*
>       -@@ sequencer.c: static int do_update_refs(struct repository *r)
>       - 			continue;
>       - 		}
>       -
>        -		res |= refs_update_ref(refs, "rewritten during rebase",
>        -				       item->string,
>        -				       &rec->after, &rec->before,
>       @@ sequencer.c: static int do_update_refs(struct repository *r)
>         }
>         
>        @@ sequencer.c: cleanup_head_ref:
>       + 		strbuf_release(&buf);
>         		strbuf_release(&head_ref);
>       - 	}
>         
>       --	do_update_refs(r);
>       -+	do_update_refs(r, opts->quiet);
>       +-		if (do_update_refs(r))
>       ++		if (do_update_refs(r, opts->quiet))
>       + 			return -1;
>       + 	}
>         
>       - 	/*
>       - 	 * Sequence of picks finished successfully; cleanup by
>        
>         ## t/t3404-rebase-interactive.sh ##
>        @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs correctly' '
>       @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs c
>         
>         test_expect_success 'respect user edits to update-ref steps' '
>        @@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs: check failed ref update' '
>       + 	# the lock in the update-refs file.
>         	git rev-parse third >.git/refs/heads/second &&
>         
>       - 	git rebase --continue 2>err &&
>       +-	git rebase --continue 2>err &&
>        -	grep "update_ref failed for ref '\''refs/heads/second'\''" err
>       ++	test_must_fail git rebase --continue 2>err &&
>        +	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
>        +
>        +	cat >expect <<-\EOF &&
>