Re: [Summit topic] Server-side merge/rebase: needs and wants?

[Christian Couder <christian.couder@gmail.com>]

On Tue, Nov 9, 2021 at 1:18 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>
> On Mon, Nov 08 2021, Taylor Blau wrote:
>
> > I was discussing this with Elijah today in IRC. I thought that I sent
> > the following message to the list, but somehow dropped it from the CC
> > list, and only sent it to Elijah and Johannes.
> >
> > Here it is in its entirety, this time copying the list.
> >
> > n Thu, Oct 21, 2021 at 01:56:06PM +0200, Johannes Schindelin wrote:
> >>  5.  The challenge is not necessarily the technical challenges, but the UX for
> >>      server tools that live “above” the git executable.
> >>
> >>      1. What kind of output is needed? Machine-readable error messages?
> >>
> >>      2. What Git objects must be created: a tree? A commit?
> >>
> >>      3. How to handle, report, and store conflicts? Index is not typically
> >>         available on the server.
> >
> > I looked a little bit more into what GitHub would need in order to make
> > the switch. For background, we currently perform merges and rebases
> > using libgit2 as the backend, for the obvious reason which is that in a
> > pre-ORT world we could not write an intermediate result without having
> > an index around.
> >
> > (As a fun aside, we used to expand our bare copy of a repository into a
> > temporary working directory, perform the merge there, and then delete
> > the directory. We definitely don't do that anymore ;)).
> >
> > It looks like our current libgit2 usage more-or-less returns an
> > (object_id, list<file>) tuple, where:
> >
> >   - a non-NULL object_id is the result of a successful (i.e.,
> >     conflict-free) merge; specifically the oid of the resulting root
> >     tree
> >
> >   - a NULL object_id and a non-empty list of files indicates that the
> >     merge could not be completed without manual conflict resolution, and
> >     the list of files indicates where the conflicts were
> >
> > When we try to process a conflicted merge, we display the list of files
> > where conflicts were present in the web UI. We do have a UI to resolve
> > conflicts, but we populate the contents of that UI by telling libgit2 to
> > perform the same merge on *just that file*, and writing out the file
> > with its conflict markers as the result (and sending that result out to
> > a web editor).
> >
> > So I think an ORT-powered server-side merge would have to be able to:
> >
> >   - write out the contents of a merge (with a tree, not a commit), and
> >     indicate whether or not that merge was successful with an exit code
> >
> >   - write out the list of files that had conflicts upon failure
> >
> > Given my limited knowledge of the ORT implementation, it seems like
> > writing out the conflicts themselves would be pretty easy. But GitHub
> > probably wouldn't use it, or at least not immediately, since we rely
> > heavily on being able to recreate the conflicts file-by-file as they are
> > needed.
> >
> > Anyway, I happened to be looking into all of this during the summit, but
> > never wrote any of it down. So I figured that this might be helpful in
> > case folks are interested in pursuing this further. If so, let me know
> > if there are any other questions about what GitHub might want on the
> > backend, and I'll try to answer as best I can.
>
> That's very informative, thanks.

Yeah, thanks!

> Not that "ort" won't me much better at this,

I think the optimizations in "ort" could still be useful. Wouldn't it
be nice if rename detection was optimized for example?

> but FWIW git-merge-tree
> sort of gets most of the way-ish to what you're describing already in
> terms of a command interface.

Yeah, but if the engine is not up to date, I am not sure it's worth it
to reuse it just for the current very limited command interface.

> I.e. I'm not the first or last to have (not for anything serious)
> implement a dry-run bare-repo merge with something like:
>
>     git merge-tree origin/master git-for-windows/main origin/seen >diff
>     # Better regex needed, but basically this
>     grep "^\+<<<<<<< \.our$" diff && conflict=t
>
> So with some parsing of that command output you can get a diff with one
> side or the other applied.

Yeah, it looks like it would be easy to add options like --ours,
--theirs, etc, to get only the part we are interested in. And we
already easily see if the merge conflicted or not from the current
output, as it seems to output:

"0 mode sha1 filename"

in case of a successful merge, and:

"1 mode sha1 filename"
"2 mode sha1 filename"
"3 mode sha1 filename"

in case of conflicts.

> From there it's a matter of applying the patch, and from there you'd get
> blobs/trees. which is painful from just having a diff & no index, so
> it's a common use-case of libgit2 for just such basic usage.
>
> But to the extent that we were talking about plumbing interfaces
> wouldn't basically a git-merge-tree on steroids (or extension thereof)
> do, i.e.:
>
>  * Ask it to merge X heads, returns whether it worked or not
>  * ... and can return a diff with conflict markers like this
>  * ... for just some <pathspec>
>  * ... maybe with the conflict already "resolved" one way or the other?
>  * ... optionally, after some markers write one/both sides, spew out the
>    relevant tree/blob OIDs
>  * ... which again, could be limited by the <pathspec> above.
>
> I'm thinking of something that basically works like git for-each-ref --format="". So:
>
>     git merge-tree --format="..." <heads> -- <pathspec>
>
> Where that <format> can be custom \0-delimited (or whatever) sections of
> payload that could have whatever combination of the above you'd need. I
> think git-for-each-ref is probably the best example we've got of a
> plumbing interface in this category, i.e. being able to extract
> arbitrary payloads via format specifiers & "path" (well, ref)
> limitation.

The current synopsis is:

git merge-tree <base-tree> <branch1> <branch2>

which is quite different from what you are proposing.

Given that it seems worth it to use a different underlying engine
(actually the "ort" one) than the current one, I think that it might
be better to start from scratch with a new command using the "ort"
engine.

> Elijah probably has much better ideas already, I'm just spitballing.

Yeah, I'd be interested in knowing Elijah's opinion on this. Although
maybe I misunderstood, but I thought that Elijah had plans to send
patches related to this to the list after v2.34.

> But if something like that worked it would be mostly a matter of
> stealing code from for-each-ref and the like, and then the <handwaiving>
> mapping that to ORT callbacks somehow.

Yeah, but what would be left from the original git merge-tree then?

Wouldn't it make more sense to start with a new command that has
roughly the same features as git merge-tree and a similar interface
(though maybe not quite the same as we could anticipate some future
extensions and maybe learn a bit from other commands), but uses "ort".
Then we could grow it as we want, without being burdened by the git
merge-tree legacy, in the same way as "ort" was developed without
being burdened by the recursive merge legacy?

> And then it could even learn a --batch mode, which with those formats
> could allow calling it without paying the price for command
> re-invocation, something like the update-ref/proposed cat-file interface
> discussed in another thread at [1].

Yeah, sure.