[RFC PATCH] proposal for refs/tracking/* hierarchy

[RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

From: Jacob Keller <jacob.keller@gmail.com>

Historically, git has tracked the status of remote branches (heads) in
refs/remotes/<name>/*. This is necessary and useful as it allows users
to track the difference between their local work and the last known
status of the remote work.

Unfortunately this hierarchy is limited in that it only tracks branches.
Additionally, it is not feasible to extend it directly, because the
layout would become ambiguous. For example, if a user happened to have
a local branch named "notes" it could be confusing if you tried to add
"refs/remotes/origin/notes/<remote notes refs" as this would collide
with the already existing refs/remotes/origin/notes branch that existed.

Instead, lets add support for a new refs/tracking/* hierarchy which is
laid out in such a way to avoid this inconsistency. All refs in
"refs/tracking/<remote>/*" will include the complete ref, such that
dropping the "tracking/<remote>" part will give the exact ref name as it
is found in the upstream. Thus, we can track any ref here by simply
fetching it into refs/tracking/<remote>/*.

Add support to tell a new remote to start tracking remote hierarchies
via "--follow" which will track all refs under that section, ie:

git remote add --follow notes origin <url> will cause

+refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch
refspec for this remote.

This ensures that any future refs which want a method of sharing the
current remote status separate from local status could use
refs/tracking

A long term goal might be to deprecate refs/remotes/ in favor of
refs/tracking (possibly adding in magic to convert refs/remotes directly
into refs/tracking so that old stuff still works?).

Things not yet thought through:

1) maybe we should create a configurable default so that some known set
   of default refs get pulled (ie: notes, grafts, replace, etc?)
   Possibly with some sort of easy way to add or subtract from the list
   in config...

2) so far, there's no work done to figure out how to remove
   refs/tracking when a remote is dropped. I *think* we can just delete
   all refs under refs/tracking/<name> but I'm not completely certain

3) adding magic to complete refs under tracking, such as for git-notes
   or similar may wish to understand shorthand for referencing the
   remote version of notes

4) adding support for clone.. (this is weird because git-clone and
   git-remote don't both use the same flow for setup.. oops??)

5) tests, documentation etc. (This is primarily an RFC, with the goal of
   providing a known location for remote references such as notes to
   reside)

6) we probably want to enable notes and grafts and other similar things
   to use the remote tracking data if its available.

7) what about tags? Currently we fetch all tags into refs/tags directly,
   which is a bit awkward, if for example you create a local tag and
   a remote creates a tag with the same name, you simply don't get that
   new version. This is good, but now you have no idea how to tell if
   your tags are out of date or not. refs/tracking can partially resolve
   this since remote tags will always be "exactly" what the remote has.
   Unfortunately, I don't know how we'd resolve them into local tags...

Probably other things missing too...

Signed-off-by: Jacob Keller <jacob.keller@gmail.com>
---
 builtin/remote.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/builtin/remote.c b/builtin/remote.c
index f1a88fe26589..dffe3892be11 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -122,6 +122,16 @@ static void add_branch(const char *key, const char *branchname,
 	git_config_set_multivar(key, tmp->buf, "^$", 0);
 }
 
+static void add_tracking(const char *key, const char *namespace,
+			 const char *remotename, struct strbuf *tmp)
+{
+	strbuf_reset(tmp);
+	strbuf_addch(tmp, '+');
+	strbuf_addf(tmp, "refs/%s/*:refs/tracking/%s/%s/*",
+		    namespace, remotename, namespace);
+	git_config_set_multivar(key, tmp->buf, "^$", 0);
+}
+
 static const char mirror_advice[] =
 N_("--mirror is dangerous and deprecated; please\n"
    "\t use --mirror=fetch or --mirror=push instead");
@@ -149,6 +159,7 @@ static int add(int argc, const char **argv)
 	int fetch = 0, fetch_tags = TAGS_DEFAULT;
 	unsigned mirror = MIRROR_NONE;
 	struct string_list track = STRING_LIST_INIT_NODUP;
+	struct string_list follow = STRING_LIST_INIT_NODUP;
 	const char *master = NULL;
 	struct remote *remote;
 	struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
@@ -164,6 +175,8 @@ static int add(int argc, const char **argv)
 			    N_("or do not fetch any tag at all (--no-tags)"), TAGS_UNSET),
 		OPT_STRING_LIST('t', "track", &track, N_("branch"),
 				N_("branch(es) to track")),
+		OPT_STRING_LIST(0, "follow", &follow, N_("namespace"),
+				N_("refs namespaces to follow in refs/tracking")),
 		OPT_STRING('m', "master", &master, N_("branch"), N_("master branch")),
 		{ OPTION_CALLBACK, 0, "mirror", &mirror, N_("push|fetch"),
 			N_("set up remote as a mirror to push to or fetch from"),
@@ -207,6 +220,11 @@ static int add(int argc, const char **argv)
 		}
 	}
 
+	for (i = 0; i < follow.nr; i++) {
+		add_tracking(buf.buf, follow.items[i].string,
+			     name, &buf2);
+	}
+
 	if (mirror & MIRROR_PUSH) {
 		strbuf_reset(&buf);
 		strbuf_addf(&buf, "remote.%s.mirror", name);
-- 
2.13.0.615.gb09ed6e59a40

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

On Fri, Jun 23, 2017 at 6:52 AM, Jacob Keller <jacob.e.keller@intel.com> wrote:
> From: Jacob Keller <jacob.keller@gmail.com>
>
> Historically, git has tracked the status of remote branches (heads) in
> refs/remotes/<name>/*. This is necessary and useful as it allows users
> to track the difference between their local work and the last known
> status of the remote work.
>
> Unfortunately this hierarchy is limited in that it only tracks branches.
> Additionally, it is not feasible to extend it directly, because the
> layout would become ambiguous. For example, if a user happened to have
> a local branch named "notes" it could be confusing if you tried to add
> "refs/remotes/origin/notes/<remote notes refs" as this would collide
> with the already existing refs/remotes/origin/notes branch that existed.
>
> Instead, lets add support for a new refs/tracking/* hierarchy which is
> laid out in such a way to avoid this inconsistency. All refs in
> "refs/tracking/<remote>/*" will include the complete ref, such that
> dropping the "tracking/<remote>" part will give the exact ref name as it
> is found in the upstream. Thus, we can track any ref here by simply
> fetching it into refs/tracking/<remote>/*.
>
> Add support to tell a new remote to start tracking remote hierarchies
> via "--follow" which will track all refs under that section, ie:
>
> git remote add --follow notes origin <url> will cause
>
> +refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch
> refspec for this remote.
>
> This ensures that any future refs which want a method of sharing the
> current remote status separate from local status could use
> refs/tracking
>
> A long term goal might be to deprecate refs/remotes/ in favor of
> refs/tracking (possibly adding in magic to convert refs/remotes directly
> into refs/tracking so that old stuff still works?).
>
> Things not yet thought through:
>
> 1) maybe we should create a configurable default so that some known set
>    of default refs get pulled (ie: notes, grafts, replace, etc?)
>    Possibly with some sort of easy way to add or subtract from the list
>    in config...
>
> 2) so far, there's no work done to figure out how to remove
>    refs/tracking when a remote is dropped. I *think* we can just delete
>    all refs under refs/tracking/<name> but I'm not completely certain
>
> 3) adding magic to complete refs under tracking, such as for git-notes
>    or similar may wish to understand shorthand for referencing the
>    remote version of notes
>
> 4) adding support for clone.. (this is weird because git-clone and
>    git-remote don't both use the same flow for setup.. oops??)
>
> 5) tests, documentation etc. (This is primarily an RFC, with the goal of
>    providing a known location for remote references such as notes to
>    reside)
>
> 6) we probably want to enable notes and grafts and other similar things
>    to use the remote tracking data if its available.
>
> 7) what about tags? Currently we fetch all tags into refs/tags directly,
>    which is a bit awkward, if for example you create a local tag and
>    a remote creates a tag with the same name, you simply don't get that
>    new version. This is good, but now you have no idea how to tell if
>    your tags are out of date or not. refs/tracking can partially resolve
>    this since remote tags will always be "exactly" what the remote has.
>    Unfortunately, I don't know how we'd resolve them into local tags...
>

Oops:

8) if we decided to go with "all refs always get tracked in
refs/tracking" we probably want to either add a way to say "all refs
except refs/tracking ones" or we want a way for servers to (by
default) not advertise refs/tracking when clients fetch from them.
That's partially why I went with the easier "only some hierarchies
will get pulled by default" Otherwise, two remotes that fetch from
each other could create a never ending cycle of
refs/tracking/origin/tracking/origin/tracking/origin/ adding a new
layer every time they fetched.

Thanks,
Jake

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Junio C Hamano]

Jacob Keller <jacob.e.keller@intel.com> writes:

> Instead, lets add support for a new refs/tracking/* hierarchy which is
> laid out in such a way to avoid this inconsistency. All refs in
> "refs/tracking/<remote>/*" will include the complete ref, such that
> dropping the "tracking/<remote>" part will give the exact ref name as it
> is found in the upstream. Thus, we can track any ref here by simply
> fetching it into refs/tracking/<remote>/*.

I do think this overall is a good goal to aim for wrt "tracking",
i.e.  keeping a pristine copy of what we observed from the outside
world.  In addition to what you listed as "undecided" below,
however, I expect that this will highlight a lot of trouble in
"working together", i.e. reconciling the differences of various
world views and moving the project forward, in two orthogonal axes:

 - Things pointed at by some refs (e.g. notes/, but I think the
   ".gitmodules equivalent that is not tied to any particular commit
   in the superproject" Jonathan Nieder and friends advocate falls
   into the same category) do not correspond to the working tree
   contents, and merging their contents will always pose challenge
   when we need to prepare for conflict resolution.

 - Things pointed at by some other refs (e.g. tags/) do not make
   sense to be merged.  You may locally call a commit with a tag
   "wip", while your friends may use the same "wip" tag to point at
   a different one.  Both are valid, and more importantly, there is
   no point even trying to reconcile what the "wip" tag means in the
   project globally.

For the former, I expect that merging these non-working tree
contents will all have to require some specific tool that is
tailored for the meaning each hierarchy has, just like "git notes
merge" gives a solution that is very specific to the notes refs (I
do not know how well "notes merge" works in practice, though).

For the latter, having a separate tracking hierarchy is a strict
improvement from "tracking" point of view, but I think "working
together" also needs a well designed set of new look-up rules, and a
new convention.  For example, some tags may not want to be shared
(e.g. "wip" example above) even though they should be easy to access
by those who already have them (e.g. "git log ..wip" should work
without having to say "git log ..refs/local-tags/wip", even if we
decide to implement the "some tags may not want to be shared"
segregation by introducing a new hierarchy).  And also a shared tag
that is copied to "refs/tracking/origin/tags/v1.0" would need a way
better than "git log tracking/origin/tags/v1.0" for this mechanism
to be useful in everyday workflow.

Thanks.

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

On Fri, Jun 23, 2017 at 1:54 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jacob Keller <jacob.e.keller@intel.com> writes:
>
>> Instead, lets add support for a new refs/tracking/* hierarchy which is
>> laid out in such a way to avoid this inconsistency. All refs in
>> "refs/tracking/<remote>/*" will include the complete ref, such that
>> dropping the "tracking/<remote>" part will give the exact ref name as it
>> is found in the upstream. Thus, we can track any ref here by simply
>> fetching it into refs/tracking/<remote>/*.
>
> I do think this overall is a good goal to aim for wrt "tracking",
> i.e.  keeping a pristine copy of what we observed from the outside
> world.  In addition to what you listed as "undecided" below,
> however, I expect that this will highlight a lot of trouble in
> "working together", i.e. reconciling the differences of various
> world views and moving the project forward, in two orthogonal axes:
>

I agree, I think this is definitely a problem that we'd want/have to solve.

>  - Things pointed at by some refs (e.g. notes/, but I think the
>    ".gitmodules equivalent that is not tied to any particular commit
>    in the superproject" Jonathan Nieder and friends advocate falls
>    into the same category) do not correspond to the working tree
>    contents, and merging their contents will always pose challenge
>    when we need to prepare for conflict resolution.

This is part of why I started by looking at notes, since there is
already some code for notes merge, it just doesn't have a standard
place to store "remote" notes. I've wanted that ability to enable more
ease of sharing.

>
>  - Things pointed at by some other refs (e.g. tags/) do not make
>    sense to be merged.  You may locally call a commit with a tag
>    "wip", while your friends may use the same "wip" tag to point at
>    a different one.  Both are valid, and more importantly, there is
>    no point even trying to reconcile what the "wip" tag means in the
>    project globally.
>
> For the former, I expect that merging these non-working tree
> contents will all have to require some specific tool that is
> tailored for the meaning each hierarchy has, just like "git notes
> merge" gives a solution that is very specific to the notes refs (I
> do not know how well "notes merge" works in practice, though).

Agreed. That's also part of why I'm looking at doing it as a partial
"one hierarchy at a time" proposal, I've got a somewhat better format
for that coming Soon(TM)

>
> For the latter, having a separate tracking hierarchy is a strict
> improvement from "tracking" point of view, but I think "working
> together" also needs a well designed set of new look-up rules, and a
> new convention.  For example, some tags may not want to be shared
> (e.g. "wip" example above) even though they should be easy to access
> by those who already have them (e.g. "git log ..wip" should work
> without having to say "git log ..refs/local-tags/wip", even if we
> decide to implement the "some tags may not want to be shared"
> segregation by introducing a new hierarchy).  And also a shared tag
> that is copied to "refs/tracking/origin/tags/v1.0" would need a way
> better than "git log tracking/origin/tags/v1.0" for this mechanism
> to be useful in everyday workflow.
>
> Thanks.
>

My thoughts on tags are sort of the following:

keep the default semantics of tags get copied into our local
refs/tags, (if we're fetching them). However, create some mechanism
for easily showing when you have a local tag with the same name as a
remote tag which don't point at the same thing, by using the
refs/tracking. Possibly, any time you grab a tag, check and see if the
remote version is different and "do something" like complain, or show
both, or give some advice (possibly with a new tool to "pull in the
remote tag if you verify it's accurate, etc".

This way, you generally work with tags the same way but get added
protection for knowing when your local tags differ from what remotes
show?

Thanks,
Jake

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Marc Branchaud]

On 2017-06-23 04:54 PM, Junio C Hamano wrote:
> Jacob Keller <jacob.e.keller@intel.com> writes:
> 
>> Instead, lets add support for a new refs/tracking/* hierarchy which is
>> laid out in such a way to avoid this inconsistency. All refs in
>> "refs/tracking/<remote>/*" will include the complete ref, such that
>> dropping the "tracking/<remote>" part will give the exact ref name as it
>> is found in the upstream. Thus, we can track any ref here by simply
>> fetching it into refs/tracking/<remote>/*.
> 
> I do think this overall is a good goal to aim for wrt "tracking",
> i.e.  keeping a pristine copy of what we observed from the outside
> world.  In addition to what you listed as "undecided" below,
> however, I expect that this will highlight a lot of trouble in
> "working together", i.e. reconciling the differences of various
> world views and moving the project forward, in two orthogonal axes:
> 
>   - Things pointed at by some refs (e.g. notes/, but I think the
>     ".gitmodules equivalent that is not tied to any particular commit
>     in the superproject" Jonathan Nieder and friends advocate falls
>     into the same category) do not correspond to the working tree
>     contents, and merging their contents will always pose challenge
>     when we need to prepare for conflict resolution.

OTOH, we shouldn't need to do any conflict resolution on these 
"tracking" refs:  The remote side should update the refs properly. 
Nobody should make local changes to these refs.

I guess I'm advocating that git should only allow "tracking" refs to be 
updated by a fetch, or a successful push of a local, non-"tracking" ref.

		M.


>   - Things pointed at by some other refs (e.g. tags/) do not make
>     sense to be merged.  You may locally call a commit with a tag
>     "wip", while your friends may use the same "wip" tag to point at
>     a different one.  Both are valid, and more importantly, there is
>     no point even trying to reconcile what the "wip" tag means in the
>     project globally.
> 
> For the former, I expect that merging these non-working tree
> contents will all have to require some specific tool that is
> tailored for the meaning each hierarchy has, just like "git notes
> merge" gives a solution that is very specific to the notes refs (I
> do not know how well "notes merge" works in practice, though).
> 
> For the latter, having a separate tracking hierarchy is a strict
> improvement from "tracking" point of view, but I think "working
> together" also needs a well designed set of new look-up rules, and a
> new convention.  For example, some tags may not want to be shared
> (e.g. "wip" example above) even though they should be easy to access
> by those who already have them (e.g. "git log ..wip" should work
> without having to say "git log ..refs/local-tags/wip", even if we
> decide to implement the "some tags may not want to be shared"
> segregation by introducing a new hierarchy).  And also a shared tag
> that is copied to "refs/tracking/origin/tags/v1.0" would need a way
> better than "git log tracking/origin/tags/v1.0" for this mechanism
> to be useful in everyday workflow.
> 
> Thanks.
> 

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

> OTOH, we shouldn't need to do any conflict resolution on these
> "tracking" refs:  The remote side should update the refs
> properly. Nobody should make local changes to these refs.
>
> I guess I'm advocating that git should only allow "tracking" refs to
> be updated by a fetch, or a successful push of a local, non-"tracking"
> ref.

I do not think anybody is imagining new things to happen inside
refs/tracking other than by fetching, just like refs/remotes/ refs
behave like so.

What I was discussing was mostly the step next to the introduction
of tracking/.  Some things are useful merely by existing (e.g. tags
you got from remote, as long as you can easily refer to them, are
useful to you).  Some other things are cumbersome to use until you
manage to consolidate your views with theirs in order to make
collective progress, and they require merging (e.g. notes).