Re: [PATCH] technical doc: add a design doc for the evolve command

[Stefan Xenos <sxenos@google.com>]

> Am I correct to understand that the reason why a commit object is
> (ab|re)used to represent a meta-commit is because by doing so we
> would get connectivity (i.e. fetching & pushing would transfer all
> the associated objects along) for free, and by not representing it
> as a new and different object type, existing implementations can
> just pass them along without understanding what they are, and as
> long as these are not mixed as parts of the main history of the
> project (e.g. when enumerating commits that has aa7ce5 as its
> parents, because somebody else obsoleted aa7ce5 and you want to
> evolve anything that built on it, you do not want to mistake the
> above "meta" commit as a commit that is part of the ordinary history
> and rebuild on top of the new version of aa7ce5, which would lead to
> a disaster), everything would work just fine?

Yes, sir. That's it exactly. My first draft of the proposal suggested
creating a new top-level object type, but when I started digging
through the code I realized that the new object was so similar to a
commit that there was no need for a new type.

> Perhaps you'd use something like "presence of parent-type header
> marks that a commit is a meta-commit and not part of the main
> history".

Yes, that's called out explicitly as part of the proposal (see the
first sentence in the Parent-type subsection). Fsck would enforce this
invariant.

> How are these meta commits anchored so that it won't be reclaimed by
> repack?

They would either be anchored by a ref in the metas/ namespace (for
active changes currently under consideration by evolve) or by the
reflog (for recently deleted changes).

> I do not see any "parent" field used to chain them together,

They point to one another using the usual "parent" field present in
all commit objects. For an example of what the raw struct would look
like with parent pointers, see the top of the "Detailed design"
section or search the doc for the string <example_meta_commit>. For
examples of how the metacommits in a change graph would be connected
after various operations, see the "Commit" section and the "Merge"
section. Please let me know if any of these examples are
insufficiently explained or if there's any other examples you'd like
to see.

> but I do not think we can afford to spend one ref per meta
> commit, as refs are not designed to point into each and every object
> in the repository.

Agreed. This is actually one of the reasons I'm proposing the use of
chains of meta-commits as opposed to using a purely ref-based
approach. I describe several other ref-based approaches in the "Other
options considered" section, and I made essentially the same point
there. We only create refs in the metas/ namespace to point to the
head of each change, and the rest of the commits and metacommits used
by the graph are reached via the parent pointers in the metacommits.

> I have a moderately strong opposition against "origin" thing.  If
> aa7ce555 replaces d664309ee, in order for the tool to be able to
> "evolve" other histories that build on top of d664309ee, it only
> needs the history between aa7ce555 and d664309ee and it would not
> matter how aa7ce555 was built relative to its parent.

I see I haven't justified the "origin" thing well enough. I'll
elaborate in the document, but here's the short version. The "origin"
edges are needed to address several use-cases:

1. Gerrit needs to know about cherry picks.

This is one of the lesser-known things that it uses the change-id
footers for and if we want to be able to eliminate the gerrit
change-id footers we need to record and communicate information about
cherry-picks somehow. This is the main reason for the origin edges -
the early drafts of this proposal didn't have them but it came up when
I asked a kind Gerrit maintainer to whether the proposal would be
sufficient to eliminate gerrit's change-ids. However there may be
alternatives I didn't think of. If we were to omit the origin edges,
can you suggest an alternative way for git to record the fact that one
commit was cherry-picked from another and communicate this fact to
gerrit?

I see that I forgot to call out "replacing gerrit change-ids" as an
explicit goal. I'll add that to the doc.

2. Obsolescence across cherry-picks.

In your example, it *may* actually matter how aa7ce55 was constructed.
One such scenario is what I'm calling obsolescence across
cherry-picks. Let me describe the use-case for it:

Alice creates commit A1.

Bob cherry-picks A1 to another branch, producing B1. At this point,
Bob has a metacommit saying that A1 is the origin of B1.

Alice amends A1, producing A2. She shares this with Bob.

At this point, Bob probably wants to amend B1 to include whatever
bugfix Alice did in A2 since the thing he cherry-picked is now out of
date. That's what the obsolescence across cherry-picks feature does.
If bob runs evolve with this option enabled, the evolve command will
produce B2 by amending B1 with whatever diff Alice did between A1 and
A2... and this only works if we have origin edges. Without the origin
edge, the evolve command wouldn't know that B1 came from the now
out-of-date A1. The commit B2 that results from this would have both
origin and replacement edges. It replaces B1 but it was formed by
cherry-picking A2.

I'm currently unsure if obsolescence across cherry-picks should be on
or off by default. I was thinking of making it off-by-default
initially and then possibly flipping the default after users have a
chance to try it and give feedback.

3. Merge-base.

Origin edges will always point to a better merge base (ancestor for
three-way merges) than the content's parent. For example, consider
doing a cherry-pick followed by a rebase:

$ git checkout -b source
# Stage some stuff
$ git commit -m "A" && git tag A
# Stage some more stuff
$ git commit --amend -m "B" && git tag B
$ git checkout -b dest && git reset --hard HEAD^1
$ git cherry-pick A
$ git checkout source
$ git rebase dest

We cherry-picked an old version of a change, and then rebased a new
version of that same change onto the a branch containing the old one.
With today's code we'd pick A's parent commit as the common ancestor
and the user would have to resolve a bunch of merge conflicts since
both A and B are versions of the same patch and touch a lot of the
same lines. We know that B is the newer version but the merge tool
doesn't know that B is newer than A. If we had origin edges, we would
know that the latest commit on the dest branch was a cherry-pick of A
and would traverse its origin edge before the content edge to look for
ancestors. That would select A as the common ancestor, and since B
applies cleanly on top of A there would be no conflicts and the
automerge would most likely succeed.

Now, this may seem like a crazy thing to do. Why would you rebase two
different versions of the same change on top of one another? This
scenario is likely to crop up when users start using the change graph
to collaborate on the same WIP change. They'll be using rebase, merge,
and cherry-pick to resolve divergence and incorporate changes from
other users... so rebases and cherry-picks of different versions of
the same change would be commonplace.

> The user may
> have typed/developed it from scratch, the user may have borrowed 70%
> of its change from 7e1bbcd while remaining 30% was done from
> scratch, or it was a concatenation of the change made in 7e1bbcd and
> another commit.

I don't think the amount they developed from scratch invalidates any
of the three main use-cases (above). Even if we have no idea how many
manual edits were made, the origin parents would still be useful for
communication with gerrit, locating a better merge base, and giving
the user the option of obsolescence across cherry-picks. The proposal
calls for commands like squash merge to create multiple origin
parents, so concatenations *would* be recorded accurately if there was
ever a reason to treat them differently (I'm not sure any of my 3 main
use-cases need to).

> One half of my point being that we can do _without_ it, and in all
> cases, aa7ce555, if leaving the fact that it was derived from
> 7e1bbcd is so important, can mention that in its log message how it
> relates to the "origin" thing.

Log messages would be sufficient for communicating cherry-picks to the
user, but wouldn't address any of the driving use-cases for origin
parents which require it to be machine-readable.

Now, admittedly the obsolescence-across-cherry-picks and merge-base
use-cases are minor features due to the fact that cherry-picks and
squash merges are themselves uncommon. A lot of users would probably
never notice the difference. However, it would be very disappointing
if gerrit's change-ids needed to stick around just for the sake of
this one missing corner case.

> And the other half is that while I consider the "origin" thing is
> unnecessary for the above reasons, having it means we need to not
> just transfer the history reading to aa7ce555 and d664309ee (which
> are necessary anyway while we have histories to transplant from
> d664309ee to aa7ce555) but also have to pull in the history leading
> to 7e1bbcd and we cannot discard it.

I'll assume that by "history" you're referring to the change graph
(the metacommits) and not the branches (the commits), which would have
no origin edges or connection between replacements.

If the user has kept a change around in their metas namespace, it's an
indication that they (or their collaborators) are still working on it
and want its history to be retained. I don't necessarily see this as a
problem because if collaborators are still editing a change that the
local user cherry-picked, it's plausible that the change may be the
subject of a future obsolescence-over-cherry-pick in which case having
the history around is necessary.

You're right that our default position should be not to retain extra
objects unless there's a compelling reason to do so, and this proposal
should have explained that reason. Now that I've explained the reason
do you still have a strong objection to the "origin" parents, or have
I overlooked a use-case?

  - Stefan