Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references

[Stefan Xenos <sxenos@google.com>]

Heya, Tejun!

Thanks for getting in touch. I agree with Junio that we probably
shouldn't be tracking the same information in two ways if we can think
of something that does both... so let's see if we can think of
something! The evolve proposal suggests having a metas/ namespace to
track ongoing changes is intended as a way to track the changes a user
is actively working on. If we were to use it to retroactively store
every historical cherry-pick in a large repository, I'm concerned it
may get too cluttered to use for its originally-intended purpose. I'm
also not sure how well things would perform with a huge number of
refs. The remainder of the proposal (using metacommits to store the
relationships) would work for the xref use-case, but we'd probably
want to tweak the way we store the root refs in order to do a good job
with the large number of inactive cherry-picks he probably wants. Any
code that is looking for cross-reference metadata could look in both
namespaces.

Conversely, if we were to tweak the xrefs proposal by adding the
obsolescence information that evolve needs, we'd be missing a place to
store the user's ongoing changes, a way to push and pull specific
changes, a way to describe alternate histories for a commit, and a
mechanism for preventing the commits needed by evolve from getting
garbage collected.

All the problems with both approaches are solve-able, though.

I spent a few hours discussing this with Stefan Beller last week and I
think we've come up with a variation on the evolve proposal that would
cover both our use-cases. Let me know what you think. To address the
cluttering of the metas namespace, we could introduce a new
"hiddenmetas" namespace. It would contain the same type of data
recorded in the metas namespace, but it would normally be hidden from
the user when they list their changes, etc. It would also be immune
from the automatic deletions that are applied to the "metas"
namespace. From your point of view, the critical bit is that it would
contain information about cherry-picks. That would address the
"user-visible clutter" problem. Utility methods that want to search
for cherry-picks would iterate over both namespaces.

For the performance problem, I think we could just live with it
temporarily and wait for the currently-ongoing reftable work since the
reftable proposal would address exactly this performance issue (by
allowing us to store and manipulate large numbers of refs
efficiently).

A nice characteristic of this approach is that it would also address a
problem with the evolve proposal that had concerned me: how to deal
with the filter-branch command, which would have created pretty much
the same problems regarding the large number of metadata refs for
changes the user isn't actively working on.

It might be nice to also consider and some alternatives, but I haven't
had enough time to think up more of them (and I haven't digested the
xrefs proposal sufficiently to suggest an enhanced version of it yet).
If anyone else has ideas for combining the xrefs and evolve use-cases,
having more alternatives to choose from is always better!

If you're okay with the "hiddenmetas" approach in principle, I'll
update the evolve proposal doc to include it. Once we work out how the
storage format will work, we can coordinate our implementation work.

  - Stefan




On Wed, Dec 12, 2018 at 7:46 PM Tejun Heo <tj@kernel.org> wrote:
>
> Hello, Junio, Stefan.
>
> On Thu, Dec 13, 2018 at 12:09:39PM +0900, Junio C Hamano wrote:
> > Please do not take the above as "don't do notes/xref-; instead read
> > from the 'meta commits'".  I do not have a preference between the
> > two proposed implementations.  The important thing is that we won't
> > end up with having to maintain two separate mechanisms that want to
> > keep track of essentially the same class of information.  FWIW I'd
> > be perfectly fine if the unification goes the other way, as long as
> > goals of both parties are met, and for that, I'd like to see you two
> > work together, or at least be aware of what each other is doing and
> > see if cross-pollination would result in a mutually better solution.
>
> So, from my POV, the only thing I want is being able to easily tell
> which commit got cherry picked where.  I really don't care how that's
> achieved.  Stefan, if there's anything I can do to push it forward,
> let me know and if you see anything useful in my patchset, please feel
> free to incorporate them in any way.
>
> Thanks.
>
> --
> tejun