Best way to update `HEAD` in mirrored repos

Best way to update `HEAD` in mirrored repos

[Rodrigo Silva Mendoza]

Hi all,

Given a repo cloned like so:
`git clone --mirror someRemote`

And then with its origin updated like so
`git remote set-url origin someRemote`

What would be the best way to update the `HEAD` ref in the clone, if
some change happens to the remote HEAD? Like say, the default branch
changing from `main` to `dev` or being renamed from `main` to
`main_2`.

What I've got that I think works
1. Get the ref from the origin that points to `HEAD`. Extract the ref
from the output.
2. Manually update the `HEAD` ref with the extracted output prior step

Like so:
```
$ git ls-remote --symref origin HEAD
ref: refs/heads/good_main_3     HEAD
0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD

$ git symbolic-ref HEAD refs/heads/good_main_3
```

Does this make sense?

The following are some of the things I've tried - they all fail to
update the `HEAD` file, or fail for some other reason.
`git fetch -p`
`git fetch -p -a`
`get fetch -p -a -u`
`git remote set-head origin -a`
   Fails with "error: Not a valid ref:
refs/remotes/origin/good_main_3". The ref it fails with is whatever
the remote HEAD is.
`git remote update --prune`

And my use case:
I have a set of git repos cloned/mirrored. They are then indexed for
search, using the `HEAD` ref. The indexer runs periodically, and
before indexing it always runs `get fetch -p` to update repos. For
each repo, to get the latest commit to index, I use the libgit2
equivalent of `git rev-parse HEAD^0`. Right now, previously cloned
repos whose HEAD is updated after the initial clone will fail at this
step, which causes them to not get indexed. Here's an example run, all
for the same repo `x`.

1. `git clone & set origin for repo x` (indexer first clones repo if
it doesn't exist)
2. `git rev-parse HEAD^0` - successfully outputs some commit sha
3. Github default branch is renamed from `main` to `main_3`
4. `git fetch -p` - indexer runs this to update
5. `git rev-parse HEAD^0` - now outputs "fatal: ambiguous argument
'HEAD^0': unknown revision or path not in the working tree."

Thanks for your time,

Rodrigo

Re: Best way to update `HEAD` in mirrored repos

[Junio C Hamano]

Rodrigo Silva Mendoza <rodrigosilvamendoza3@gmail.com> writes:

> What I've got that I think works
> 1. Get the ref from the origin that points to `HEAD`. Extract the ref
> from the output.
> 2. Manually update the `HEAD` ref with the extracted output prior step
>
> Like so:
> ```
> $ git ls-remote --symref origin HEAD
> ref: refs/heads/good_main_3     HEAD
> 0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
>
> $ git symbolic-ref HEAD refs/heads/good_main_3
> ```
>
> Does this make sense?

Yes.  That soudns like the right thing to do.

Re: Best way to update `HEAD` in mirrored repos

[Rodrigo Silva Mendoza]

Great, thanks for the confirmation. Appreciate it!

Rodrigo

On Wed, Jun 8, 2022 at 3:54 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Rodrigo Silva Mendoza <rodrigosilvamendoza3@gmail.com> writes:
>
> > What I've got that I think works
> > 1. Get the ref from the origin that points to `HEAD`. Extract the ref
> > from the output.
> > 2. Manually update the `HEAD` ref with the extracted output prior step
> >
> > Like so:
> > ```
> > $ git ls-remote --symref origin HEAD
> > ref: refs/heads/good_main_3     HEAD
> > 0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
> >
> > $ git symbolic-ref HEAD refs/heads/good_main_3
> > ```
> >
> > Does this make sense?
>
> Yes.  That soudns like the right thing to do.
>

Re: Best way to update `HEAD` in mirrored repos

[Johannes Schindelin]

Hi Rodrigo,

On Wed, 8 Jun 2022, Rodrigo Silva Mendoza wrote:

> `git remote set-head origin -a`
>    Fails with "error: Not a valid ref:
> refs/remotes/origin/good_main_3". The ref it fails with is whatever
> the remote HEAD is.

The reason is that `set-head` expects options to come before arguments,
like so:

	git remote set-head -a origin

Ciao,
Johannes

Re: Best way to update `HEAD` in mirrored repos

[Rodrigo Silva Mendoza]

Hi Johannes,

On Thu, Jun 9, 2022 at 1:02 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
>
> The reason is that `set-head` expects options to come before arguments,
> like so:
>
>         git remote set-head -a origin
>
> Ciao,
> Johannes


Hmm, that doesn't seem to work for me either - I get the same type of
error as before.
Here's a minimal repro:
    $ git clone git@github.com:git/gitscm-old.git --mirror
    $ cd gitscm-old.git
    $ git remote set-head -a origin
      error: Not a valid ref: refs/remotes/origin/master

Git version: git version 2.32.1 (Apple Git-133)

Let me know if I'm missing anything.

Cheers,
Rodrigo

Re: Best way to update `HEAD` in mirrored repos

[Johannes Schindelin]

Hi Rodrigo,

On Thu, 9 Jun 2022, Rodrigo Silva Mendoza wrote:

> On Thu, Jun 9, 2022 at 1:02 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > The reason is that `set-head` expects options to come before arguments,
> > like so:
> >
> >         git remote set-head -a origin
>
> Hmm, that doesn't seem to work for me either - I get the same type of
> error as before.
> Here's a minimal repro:
>     $ git clone git@github.com:git/gitscm-old.git --mirror

Here, you set this up as a fetch mirror. That is something I've missed
before. It is also important because it means that there is no
`refs/remotes/origin/HEAD`.

>     $ cd gitscm-old.git
>     $ git remote set-head -a origin
>       error: Not a valid ref: refs/remotes/origin/master

... yet that's exactly what `set-head` assumes. The following patch fixes
it for me:

-- snip --
diff --git a/builtin/remote.c b/builtin/remote.c
index eddd40c8f87..fead15adb97 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)

 static int set_head(int argc, const char **argv)
 {
-	int i, opt_a = 0, opt_d = 0, result = 0;
+	int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
 	struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
 	char *head_name = NULL;

@@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
 	};
 	argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
 			     0);
-	if (argc)
-		strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
+	if (argc) {
+		struct remote *remote = remote_get(argv[0]);
+
+		if (!remote || !remote->mirror)
+			strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
+		else {
+			is_mirror = 1;
+			strbuf_addstr(&buf, "HEAD");
+		}
+	}

 	if (!opt_a && !opt_d && argc == 2) {
 		head_name = xstrdup(argv[1]);
@@ -1383,7 +1391,10 @@ static int set_head(int argc, const char **argv)
 		usage_with_options(builtin_remote_sethead_usage, options);

 	if (head_name) {
-		strbuf_addf(&buf2, "refs/remotes/%s/%s", argv[0], head_name);
+		if (is_mirror)
+			strbuf_addf(&buf2, "refs/heads/%s", head_name);
+		else
+			strbuf_addf(&buf2, "refs/remotes/%s/%s", argv[0], head_name);
 		/* make sure it's valid */
 		if (!ref_exists(buf2.buf))
 			result |= error(_("Not a valid ref: %s"), buf2.buf);

-- snap --

Would you have a chance to build Git with this patch and verify that it
works for you, too?

Ciao,
Johannes

Re: Best way to update `HEAD` in mirrored repos

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> diff --git a/builtin/remote.c b/builtin/remote.c
> index eddd40c8f87..fead15adb97 100644
> --- a/builtin/remote.c
> +++ b/builtin/remote.c
> @@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)
>
>  static int set_head(int argc, const char **argv)
>  {
> -	int i, opt_a = 0, opt_d = 0, result = 0;
> +	int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
>  	struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
>  	char *head_name = NULL;
>
> @@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
>  	};
>  	argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
>  			     0);
> -	if (argc)
> -		strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +	if (argc) {
> +		struct remote *remote = remote_get(argv[0]);
> +
> +		if (!remote || !remote->mirror)
> +			strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +		else {
> +			is_mirror = 1;
> +			strbuf_addstr(&buf, "HEAD");
> +		}
> +	}

Good.

Regardless of the "mirror" issue, it makes tons of sense to ask the
remote API how the remote-tracking refs for the given remote is set
up, instead of assuming that it must be "refs/remotes/<remote>"
blindly like in the original code.

That way, we could even handle a case like so:

	[remote "frotz"]
		fetch = +refs/heads/*:refs/remotes/nitfol/*

Their HEAD should be mapped to refs/remotes/nitfol/HEAD on our end,
so set-head should be able to notice that, too, if we go further
with your "do not assume, instead ask remote API" approach.

Thanks.

Re: Best way to update `HEAD` in mirrored repos

[Rodrigo Silva Mendoza]

Hello,

Apologies for the delay, I'd never built Git from source so I put it
off until I had a chunk of time, assuming it might take me a bit -
thankfully it was painless.

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Here, you set this up as a fetch mirror. That is something I've missed
> before. It is also important because it means that there is no
> `refs/remotes/origin/HEAD`.

Ahhh, the reason it wasn't working before clicked for me now, thanks
for the explanation.

> diff --git a/builtin/remote.c b/builtin/remote.c
> index eddd40c8f87..fead15adb97 100644
> --- a/builtin/remote.c
> +++ b/builtin/remote.c
> @@ -1344,7 +1344,7 @@ static int show(int argc, const char **argv)
>
>  static int set_head(int argc, const char **argv)
>  {
> -     int i, opt_a = 0, opt_d = 0, result = 0;
> +     int i, opt_a = 0, opt_d = 0, result = 0, is_mirror = 0;
>       struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
>       char *head_name = NULL;
>
> @@ -1357,8 +1357,16 @@ static int set_head(int argc, const char **argv)
>       };
>       argc = parse_options(argc, argv, NULL, options, builtin_remote_sethead_usage,
>                            0);
> -     if (argc)
> -             strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +     if (argc) {
> +             struct remote *remote = remote_get(argv[0]);
> +
> +             if (!remote || !remote->mirror)
> +                     strbuf_addf(&buf, "refs/remotes/%s/HEAD", argv[0]);
> +             else {
> +                     is_mirror = 1;
> +                     strbuf_addstr(&buf, "HEAD");
> +             }
> +     }

> Would you have a chance to build Git with this patch and verify that it
> works for you, too?

Good news Johannes, your patch worked for me as well! Here's a run
with the patched code.

    $ git --version
    git version 2.37.0.rc0.dirty
    $ git clone git@github.com:xvandish/livegrep-fragment.git --mirror
    $ cd livegrep-fragment
    $ git symbolic-ref HEAD
    refs/heads/good_main_5
    $ // I changed the github default branch name at this moment
    $ git ls-remote --symref origin HEAD
    ref: refs/heads/good_main_5     HEAD
    0666a519f94b8500ab6f14bdf7c9c2e5ca7d5821        HEAD
    $ git fetch -p
    From github.com:xvandish/livegrep-fragment
    - [deleted]         (none)     -> good_main_4
    * [new branch]      good_main_5 -> good_main_5
    $ git remote set-head -a origin
    origin/HEAD set to good_main_5
    $ git symbolic-ref HEAD
    refs/heads/good_main_5


On Fri, Jun 10, 2022 at 2:53 PM Junio C Hamano <gitster@pobox.com> wrote:
> Good.
>
> Regardless of the "mirror" issue, it makes tons of sense to ask the
> remote API how the remote-tracking refs for the given remote is set
> up, instead of assuming that it must be "refs/remotes/<remote>"
> blindly like in the original code.
>
> That way, we could even handle a case like so:
>
>         [remote "frotz"]
>                 fetch = +refs/heads/*:refs/remotes/nitfol/*
>
> Their HEAD should be mapped to refs/remotes/nitfol/HEAD on our end,
> so set-head should be able to notice that, too, if we go further
> with your "do not assume, instead ask remote API" approach.
>
> Thanks.

This makes sense to me as well, as a continuation of the "ask remote " style.
I only have a very vague idea of how I'd implement this from a look at
the codebase, so I'd be of more
help testing things out. If this idea carries forward let me know if I
can be of help testing, or if you'd like me to
like me to try to put a patch up let me know.


Thanks to both of you for your time.

Re: Best way to update `HEAD` in mirrored repos

[Rodrigo Silva Mendoza]

On Tue, Jun 14, 2022 at 9:11 PM Rodrigo Silva Mendoza
<rodrigosilvamendoza3@gmail.com> wrote:

> Good news Johannes, your patch worked for me as well! Here's a run
> with the patched code.
>
>     $ git --version
>     git version 2.37.0.rc0.dirty
>     $ git clone git@github.com:xvandish/livegrep-fragment.git --mirror
>     $ cd livegrep-fragment
>     $ git symbolic-ref HEAD
>     refs/heads/good_main_5
....

Whoops, botched the output of the first `git symbolic-ref HEAD` call.
It should be:

    $ git symbolic-ref HEAD
    refs/heads/good_main_4

Cheers,
Rodrigo