Syncing HEAD

Syncing HEAD

[Christian Couder]

Hi,

When cloning with --mirror, the clone gets its HEAD initialized with
the value HEAD has in its origin remote. After that if HEAD changes in
origin there is no simple way to sync HEAD at the same time as the
refs are synced.

It looks like the simplest way to sync HEAD is:

1) git remote show origin
2) parse "HEAD branch: XXX" from the output of the above command
3) git symbolic-ref HEAD refs/heads/XXX

It looks like it would be quite easy to add an option to `fetch` to
sync HEAD at the same time as regular refs are synced because every
fetch from an origin that uses a recent Git contains something like:

19:55:39.304976 pkt-line.c:80           packet:          git< YYYYYYYY
HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow
deepen-since deepen-not deepen-relative no-progress include-tag
multi_ack_detailed no-done symref=HEAD:refs/heads/test-1
agent=git/2.18.0

which in this example shows that HEAD is a symref to refs/heads/test-1
in origin.

Is there a reason why no such option already exists? Would it makes
sense to add one? Is there any reason why it's not a good idea? Or am
I missing something?

I am asking because GitLab uses HEAD in the bare repos it manages to
store the default branch against which the Merge Requests (same thing
as Pull Requests on GitHub) are created.

So when people want to keep 2 GitLab hosted repos in sync, GitLab
needs to sync HEADs too, not just the refs.

I think this could be useful to other setups than GitLab though.

Thanks,
Christian.

Re: Syncing HEAD

[Stefan Beller]

On Tue, Aug 14, 2018 at 1:09 PM Christian Couder
<christian.couder@gmail.com> wrote:
>
> Hi,
>
> When cloning with --mirror, the clone gets its HEAD initialized with
> the value HEAD has in its origin remote. After that if HEAD changes in
> origin there is no simple way to sync HEAD at the same time as the
> refs are synced.
>
> It looks like the simplest way to sync HEAD is:
>
> 1) git remote show origin
> 2) parse "HEAD branch: XXX" from the output of the above command
> 3) git symbolic-ref HEAD refs/heads/XXX
>
> It looks like it would be quite easy to add an option to `fetch` to
> sync HEAD at the same time as regular refs are synced because every
> fetch from an origin that uses a recent Git contains something like:
>
> 19:55:39.304976 pkt-line.c:80           packet:          git< YYYYYYYY
> HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow
> deepen-since deepen-not deepen-relative no-progress include-tag
> multi_ack_detailed no-done symref=HEAD:refs/heads/test-1
> agent=git/2.18.0
>
> which in this example shows that HEAD is a symref to refs/heads/test-1
> in origin.
>
> Is there a reason why no such option already exists? Would it makes
> sense to add one? Is there any reason why it's not a good idea? Or am
> I missing something?

I think it is a great idea to add that. IIRC there was some talk when
designing protocol v2 on how fetching of symrefs could be added later
on in the protocol, which is why I cc'd Brandon who did the work there.

> I am asking because GitLab uses HEAD in the bare repos it manages to
> store the default branch against which the Merge Requests (same thing
> as Pull Requests on GitHub) are created.
>
> So when people want to keep 2 GitLab hosted repos in sync, GitLab
> needs to sync HEADs too, not just the refs.
>
> I think this could be useful to other setups than GitLab though.

As said, I can see how that is useful; I recently came across some
HEAD bug related to submodules, and there we'd also have the application.

    git clone --recurse-submodules file://...

might clone the submodules that are in detached HEAD, which is totally
not a long term viable good HEAD, so a subsequent fetch might want
to change the detached HEAD in submodules or re-affix it to branches.

Unrelated/extended: I think it would be cool to mirror a repository even
more, e.g. it would be cool to be able to fetch (if configured as allowed)
the remote reflog, (not to be confused with you local reflog of the remote).
I think that work would be enabled once reftables are available, which you
have an eye on?

Re: Syncing HEAD

[Jeff King]

On Tue, Aug 14, 2018 at 10:09:37PM +0200, Christian Couder wrote:

> When cloning with --mirror, the clone gets its HEAD initialized with
> the value HEAD has in its origin remote. After that if HEAD changes in
> origin there is no simple way to sync HEAD at the same time as the
> refs are synced.
> 
> It looks like the simplest way to sync HEAD is:
> 
> 1) git remote show origin
> 2) parse "HEAD branch: XXX" from the output of the above command
> 3) git symbolic-ref HEAD refs/heads/XXX

How about:

  git remote set-head origin -a

?

> It looks like it would be quite easy to add an option to `fetch` to
> sync HEAD at the same time as regular refs are synced because every
> fetch from an origin that uses a recent Git contains something like:

I think the "remote set-head" option is not very discoverable, since
people are used to working with "fetch", making it the natural place to
look. Just like we ported "remote update" over to "fetch --all", I think
it would be sensible to have "fetch --update-head" or similar.

One tricky thing is that the name "refs/remotes/<remote>/HEAD" is only
special by convention, and that convention is known on the writing side
only by git-clone and git-remote. So obviously:

  git fetch --update-head https://example.com/

is nonsense. We don't even have a ref. What should:

  git config remote.origin.fetch refs/heads/*:refs/remotes/foo/*
  git fetch --update-head origin

do? Should it update based no the remote name, or based on the refspec?
What happens if there are several refspecs? Etc.

99% of the time those questions won't come up. But we should design so
that we do the obvious thing in those 99%, and something sane in the
other 1%.

-Peff

Re: Syncing HEAD

[Brandon Williams]

On 08/14, Stefan Beller wrote:
> On Tue, Aug 14, 2018 at 1:09 PM Christian Couder
> <christian.couder@gmail.com> wrote:
> >
> > Hi,
> >
> > When cloning with --mirror, the clone gets its HEAD initialized with
> > the value HEAD has in its origin remote. After that if HEAD changes in
> > origin there is no simple way to sync HEAD at the same time as the
> > refs are synced.
> >
> > It looks like the simplest way to sync HEAD is:
> >
> > 1) git remote show origin
> > 2) parse "HEAD branch: XXX" from the output of the above command
> > 3) git symbolic-ref HEAD refs/heads/XXX
> >
> > It looks like it would be quite easy to add an option to `fetch` to
> > sync HEAD at the same time as regular refs are synced because every
> > fetch from an origin that uses a recent Git contains something like:
> >
> > 19:55:39.304976 pkt-line.c:80           packet:          git< YYYYYYYY
> > HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow
> > deepen-since deepen-not deepen-relative no-progress include-tag
> > multi_ack_detailed no-done symref=HEAD:refs/heads/test-1
> > agent=git/2.18.0
> >
> > which in this example shows that HEAD is a symref to refs/heads/test-1
> > in origin.
> >
> > Is there a reason why no such option already exists? Would it makes
> > sense to add one? Is there any reason why it's not a good idea? Or am
> > I missing something?
> 
> I think it is a great idea to add that. IIRC there was some talk when
> designing protocol v2 on how fetching of symrefs could be added later
> on in the protocol, which is why I cc'd Brandon who did the work there.

Actually the functionality for fetching symrefs already exists (when
using protocol v2 of course).  Despite this functionality existing its
not being used right now.

When performing a v2 fetch the first thing that a client does is request
the list of refs (by doing an ls-refs request).  The output from ls-refs
(if asked) will included information about each ref including if they
are a symref and what ref they resolve to.  So really we just need to
plumb that information through fetch to actually update HEAD, or even
update other symrefs which exist on the server.

> 
> > I am asking because GitLab uses HEAD in the bare repos it manages to
> > store the default branch against which the Merge Requests (same thing
> > as Pull Requests on GitHub) are created.
> >
> > So when people want to keep 2 GitLab hosted repos in sync, GitLab
> > needs to sync HEADs too, not just the refs.
> >
> > I think this could be useful to other setups than GitLab though.
> 
> As said, I can see how that is useful; I recently came across some
> HEAD bug related to submodules, and there we'd also have the application.
> 
>     git clone --recurse-submodules file://...
> 
> might clone the submodules that are in detached HEAD, which is totally
> not a long term viable good HEAD, so a subsequent fetch might want
> to change the detached HEAD in submodules or re-affix it to branches.
> 
> Unrelated/extended: I think it would be cool to mirror a repository even
> more, e.g. it would be cool to be able to fetch (if configured as allowed)
> the remote reflog, (not to be confused with you local reflog of the remote).
> I think that work would be enabled once reftables are available, which you
> have an eye on?

-- 
Brandon Williams

Re: Syncing HEAD

[Jeff King]

On Tue, Aug 14, 2018 at 05:06:16PM -0400, Jeff King wrote:

> On Tue, Aug 14, 2018 at 10:09:37PM +0200, Christian Couder wrote:
> 
> > When cloning with --mirror, the clone gets its HEAD initialized with
> > the value HEAD has in its origin remote. After that if HEAD changes in
> > origin there is no simple way to sync HEAD at the same time as the
> > refs are synced.
> > 
> > It looks like the simplest way to sync HEAD is:
> > 
> > 1) git remote show origin
> > 2) parse "HEAD branch: XXX" from the output of the above command
> > 3) git symbolic-ref HEAD refs/heads/XXX
> 
> How about:
> 
>   git remote set-head origin -a
> 
> ?

Reading your message again, I see you actually care less about the
refs/remote placeholder and more about the actual HEAD in a bare repo.

In which case "git remote" isn't going to help, though its underlying
code has the algorithm you would want.

> One tricky thing is that the name "refs/remotes/<remote>/HEAD" is only
> special by convention, and that convention is known on the writing side
> only by git-clone and git-remote. So obviously:

And so here the convention is simpler, because we're talking about the
main HEAD. But we still have know if you want to do that, and not update
some refs/remotes/ symref in a bare repo.

So all of this really implies to me that you want to be able to say
"take this symref on the other side and update this one on the local
side". I.e., some way to tell a refspec "don't update the value, update
the symref destination". So imagine we made "~" the magic character for
"just the symrefs" (I picked that because it's not allowed in a
refname).

Then you could do what you want with:

  git config --add remote.origin.fetch ~HEAD:HEAD

and these two would be the same:

  git remote set-head origin -a
  git fetch origin ~HEAD:refs/remotes/origin/HEAD

And it would allow more exotic things, too, like:

  # always update the remote notion of HEAD on every fetch
  git config --add remote.origin.fetch ~HEAD:refs/remotes/origin/HEAD

  # update a non-HEAD symref we track for our own purposes
  git fetch origin ~refs/tags/LATEST:refs/tags/LATEST

  # or the same thing but using the usual refspec "dst defaults to src"
  # rule and dwim lookup magic
  git fetch origin ~LATEST

In protocol v0 we don't get symref reports from the other side over the
git protocol (except for HEAD), but we could use the same logic we use
for determining HEAD for older versions of Git: find a ref that points
to the same tip. Though I would say that unlike the existing code in
guess_remote_head(), we'd probably want to treat an ambiguity as an
error, and not just default to refs/heads/master.

-Peff

Re: Syncing HEAD

[Ævar Arnfjörð Bjarmason]

On Tue, Aug 14 2018, Christian Couder wrote:

> Hi,
>
> When cloning with --mirror, the clone gets its HEAD initialized with
> the value HEAD has in its origin remote. After that if HEAD changes in
> origin there is no simple way to sync HEAD at the same time as the
> refs are synced.
>
> It looks like the simplest way to sync HEAD is:
>
> 1) git remote show origin
> 2) parse "HEAD branch: XXX" from the output of the above command
> 3) git symbolic-ref HEAD refs/heads/XXX
>
> It looks like it would be quite easy to add an option to `fetch` to
> sync HEAD at the same time as regular refs are synced because every
> fetch from an origin that uses a recent Git contains something like:
>
> 19:55:39.304976 pkt-line.c:80           packet:          git< YYYYYYYY
> HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow
> deepen-since deepen-not deepen-relative no-progress include-tag
> multi_ack_detailed no-done symref=HEAD:refs/heads/test-1
> agent=git/2.18.0
>
> which in this example shows that HEAD is a symref to refs/heads/test-1
> in origin.
>
> Is there a reason why no such option already exists? Would it makes
> sense to add one? Is there any reason why it's not a good idea? Or am
> I missing something?
>
> I am asking because GitLab uses HEAD in the bare repos it manages to
> store the default branch against which the Merge Requests (same thing
> as Pull Requests on GitHub) are created.
>
> So when people want to keep 2 GitLab hosted repos in sync, GitLab
> needs to sync HEADs too, not just the refs.
>
> I think this could be useful to other setups than GitLab though.
>
> Thanks,
> Christian.

Isn't this some other facet of (or the same) bug I reported in
https://public-inbox.org/git/87bmcyfh67.fsf@evledraar.gmail.com/ ?

Re: Syncing HEAD

[Christian Couder]

On Tue, Aug 14, 2018 at 11:47 PM, Jeff King <peff@peff.net> wrote:
> On Tue, Aug 14, 2018 at 05:06:16PM -0400, Jeff King wrote:
>
>> On Tue, Aug 14, 2018 at 10:09:37PM +0200, Christian Couder wrote:
>>
>> > When cloning with --mirror, the clone gets its HEAD initialized with
>> > the value HEAD has in its origin remote. After that if HEAD changes in
>> > origin there is no simple way to sync HEAD at the same time as the
>> > refs are synced.
>> >
>> > It looks like the simplest way to sync HEAD is:
>> >
>> > 1) git remote show origin
>> > 2) parse "HEAD branch: XXX" from the output of the above command
>> > 3) git symbolic-ref HEAD refs/heads/XXX
>>
>> How about:
>>
>>   git remote set-head origin -a
>>
>> ?
>
> Reading your message again, I see you actually care less about the
> refs/remote placeholder and more about the actual HEAD in a bare repo.

Yeah, I am interesting in updating the actual HEAD in a bare repo.

> In which case "git remote" isn't going to help, though its underlying
> code has the algorithm you would want.

Ok, I will take a look at the algorithm.

>> One tricky thing is that the name "refs/remotes/<remote>/HEAD" is only
>> special by convention, and that convention is known on the writing side
>> only by git-clone and git-remote. So obviously:
>
> And so here the convention is simpler, because we're talking about the
> main HEAD. But we still have know if you want to do that, and not update
> some refs/remotes/ symref in a bare repo.

We could maybe look at the "remote.XXX.mirror" config option. If it is
set to "true", we could interpret that as meaning we are interested in
updating the main HEAD and not some refs/remotes/ symref.

> So all of this really implies to me that you want to be able to say
> "take this symref on the other side and update this one on the local
> side". I.e., some way to tell a refspec "don't update the value, update
> the symref destination". So imagine we made "~" the magic character for
> "just the symrefs" (I picked that because it's not allowed in a
> refname).
>
> Then you could do what you want with:
>
>   git config --add remote.origin.fetch ~HEAD:HEAD
>
> and these two would be the same:
>
>   git remote set-head origin -a
>   git fetch origin ~HEAD:refs/remotes/origin/HEAD
>
> And it would allow more exotic things, too, like:
>
>   # always update the remote notion of HEAD on every fetch
>   git config --add remote.origin.fetch ~HEAD:refs/remotes/origin/HEAD
>
>   # update a non-HEAD symref we track for our own purposes
>   git fetch origin ~refs/tags/LATEST:refs/tags/LATEST
>
>   # or the same thing but using the usual refspec "dst defaults to src"
>   # rule and dwim lookup magic
>   git fetch origin ~LATEST

And `git fetch origin ~HEAD` would sync the main HEAD?

Yeah, that looks like an interesting solution to the problem.

I wonder though if we should restrict the way `git fetch origin ~XXX`
searches the .git/ directory itself.

> In protocol v0 we don't get symref reports from the other side over the
> git protocol (except for HEAD), but we could use the same logic we use
> for determining HEAD for older versions of Git: find a ref that points
> to the same tip. Though I would say that unlike the existing code in
> guess_remote_head(), we'd probably want to treat an ambiguity as an
> error, and not just default to refs/heads/master.

I wonder what `git fetch origin ~refs/heads/*:refs/heads/*` should do.
Could it know which refs are symrefs using protocol v0? Should it
guess that refs with uppercase names are symrefs? Should we allow '*'
at all in those kinds of refspecs?

It looks like making "~" the magic character for "just the symrefs"
might be a good solution in the end, though we might want to restrict
it to protocol v2.
So perhaps something like `git fetch --update-head` that you suggest
in another email would be a good solution for now and for protocol v0.

Re: Syncing HEAD

[Jeff King]

On Wed, Aug 15, 2018 at 07:49:25AM +0200, Christian Couder wrote:

> > And so here the convention is simpler, because we're talking about the
> > main HEAD. But we still have know if you want to do that, and not update
> > some refs/remotes/ symref in a bare repo.
> 
> We could maybe look at the "remote.XXX.mirror" config option. If it is
> set to "true", we could interpret that as meaning we are interested in
> updating the main HEAD and not some refs/remotes/ symref.

Yeah, for the mirror case I think that would be sufficient, and that's a
subset of the larger problem. I'm not _totally_ opposed to solving just
this narrow case, but I think it would be great if we could solve the
larger problem.

> >   # or the same thing but using the usual refspec "dst defaults to src"
> >   # rule and dwim lookup magic
> >   git fetch origin ~LATEST
> 
> And `git fetch origin ~HEAD` would sync the main HEAD?

Yes, exactly.

> I wonder though if we should restrict the way `git fetch origin ~XXX`
> searches the .git/ directory itself.

The matching is done against the list of refs that the remote
advertises. So everything is under refs/ except for HEAD. If you tried
to do something funky with top-level refs like:

  git fetch origin ~MERGE_HEAD

it would always come up with "couldn't find remote ref MERGE_HEAD".

> I wonder what `git fetch origin ~refs/heads/*:refs/heads/*` should do.
> Could it know which refs are symrefs using protocol v0? Should it
> guess that refs with uppercase names are symrefs? Should we allow '*'
> at all in those kinds of refspecs?

That's an interesting question. I'd be tempted to say that it is an
error to use "~" with a wildcard ref, at least for the first version of
the patch. That way we don't back ourselves into a corner, and can make
it do something useful later.

I think one sane set of rules is:

 - for protocol v2+, where we know which remote refs are symrefs,
   transfer them as symrefs

 - for protocol v0, either transfer them as normal refs (except HEAD,
   which we always suspect of being a symref), or simply declare it
   an error

For the most part, though, I think people would be fine without
combining wildcards with the symref feature, and would just do:

  +refs/*:refs/*
  ~HEAD:HEAD

for a bare mirror, and:

  +refs/heads/*:refs/remotes/origin/*
  ~HEAD:refs/remotes/origin/HEAD

for an auto-updating non-bare remote.

> It looks like making "~" the magic character for "just the symrefs"
> might be a good solution in the end, though we might want to restrict
> it to protocol v2.
> So perhaps something like `git fetch --update-head` that you suggest
> in another email would be a good solution for now and for protocol v0.

You still have the problem with --update-head of where to store the
result. I think the semantics for a non-wildcard "~" are clear enough,
even with protocol v0, that it would be OK to start down that road.

A few final thoughts:

 - I like the look of "~", but there are not very many characters
   disallowed in refs, and we're using one of them. Another notable one
   is "^", from which we've built the "^{foo}" syntax elsewhere. So this
   could be something like "^{symref}HEAD:HEAD", which leaves room for
   new "^{}" types in the future. But man, that looks really ugly
   compared to "~HEAD:HEAD".

 - Is there a case for a symref update where we'd want to require a
   force-push? Maybe if the local side exists and is not already a
   symref?

 - What do we do if the other side isn't a symref (e.g., a detached
   HEAD)? Is that an error? Do we detach ourselves? Does it require a
   force?

-Peff

Re: Syncing HEAD

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> So all of this really implies to me that you want to be able to say
> "take this symref on the other side and update this one on the local
> side". I.e., some way to tell a refspec "don't update the value, update
> the symref destination". ...
> ...
>   git fetch origin ~HEAD:refs/remotes/origin/HEAD

We need to be a bit careful here.

You can define the meaning of the above sanely if you know that
refmap refs/heads/*:refs/remotes/origin/* is in effect for the
remote to read "My HEAD points at refs/heads/frotz" and interpret it
as "In order to match, I need to make my refs/remotes/origin/HEAD to
point at refs/remotes/origin/frotz".

Also, what should the above form of "git fetch" write in FETCH_HEAD?
Should "git pull origin ~HEAD:refs/remotes/origin/HEAD" run the fetch
and then merge it (which may have value of refs/remotes/origin/frotz)
to the current branch?  Should the underlying fetch be also fetching
the frotz branch from them at the same time, or do we attempt to merge
a possibly stale 'frotz' (which might not even have been there, the
last time we fetched from them)?

Re: Syncing HEAD

[Jeff King]

On Fri, Aug 17, 2018 at 09:28:59AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > So all of this really implies to me that you want to be able to say
> > "take this symref on the other side and update this one on the local
> > side". I.e., some way to tell a refspec "don't update the value, update
> > the symref destination". ...
> > ...
> >   git fetch origin ~HEAD:refs/remotes/origin/HEAD
> 
> We need to be a bit careful here.
> 
> You can define the meaning of the above sanely if you know that
> refmap refs/heads/*:refs/remotes/origin/* is in effect for the
> remote to read "My HEAD points at refs/heads/frotz" and interpret it
> as "In order to match, I need to make my refs/remotes/origin/HEAD to
> point at refs/remotes/origin/frotz".

Good point. I was thinking too much about the symlink itself and not its
destination. You need some way of mapping that destination, as well.

For Christian's case, it is really the "refs/*:refs/*" mapping that he
would want to emulate (because he'd be doing ~HEAD:HEAD). In some cases,
like that one, you could infer the mapping from the HEAD:HEAD itself (X
on the remote becomes X locally). But that does not work for the
refs/remotes case. You might infer from "~HEAD:refs/remotes/origin/HEAD"
that "X becomes refs/remotes/origin/X", but it is actually "refs/heads/X
becomes ...".

So yeah, this really does need pairing with the overall ref mapping.

I think that's doable even for the example I gave above, because we
could find those refspecs in the config. But:

  git fetch git://... ~HEAD:HEAD

does not have that information. We may or may not have fetched the
pointed-to ref previously.

What if this _required_ that the symref destination from the other side
also be something that we are fetching, and was otherwise an error? That
would avoid any config trickery. It does mean that "git fetch origin
~HEAD:HEAD" does not work.  But I think you'd generally want to pair it
with a fetch anyway. I.e., Either two configured refspecs, or if a
one-off fetch from a URL, fetching the branches and the symref at the
same time.

I suppose the one case it would not support is "I want to fetch HEAD as
a symref and whatever branch it points to, but I do not yet know what
that branch is". Or, I suppose, "I do not want to update any branches,
but just update my notion of HEAD" (i.e., what "remote set-head -a"
currently does).

> Also, what should the above form of "git fetch" write in FETCH_HEAD?
> Should "git pull origin ~HEAD:refs/remotes/origin/HEAD" run the fetch
> and then merge it (which may have value of refs/remotes/origin/frotz)
> to the current branch?  Should the underlying fetch be also fetching
> the frotz branch from them at the same time, or do we attempt to merge
> a possibly stale 'frotz' (which might not even have been there, the
> last time we fetched from them)?

I'd be tempted to say that a symref fetch writes nothing into FETCH_HEAD
at all. Which would make a bare "git pull origin ~HEAD" an error. I'm
sure there are cases it _could_ do something useful, but there are so
many where it doesn't make sense.

-Peff