[BUG] fetching all remote branches results in failed multiple updates

[BUG] fetching all remote branches results in failed multiple updates

[Alexander Huynh]

Hello all,

When running the latest release of git, I receive an error when attempting to
fetch all remote branches from a repo:

    fatal: multiple updates for ref 'refs/remotes/origin/maint' not allowed

The specific ref that it fails on changes depending on the repository, but the
end result is the repo isn't cloned.

The specific configuration that causes this bug is the `remote.origin.fetch`
option, specifically:

    [remote "origin"]
    	fetch = +refs/heads/*:refs/remotes/origin/*

These settings are listed as an example in "CONFIGURED REMOTE-TRACKING
BRANCHES" of git-fetch(1), as well as expanded upon in
https://stackoverflow.com/a/40739835.

I'm running git version 2.21.0.

Full reproduction steps are below:

    [root@chabuduo ~]# useradd -m git-test -s /bin/bash
    [root@chabuduo ~]# sudo --preserve-env=SSH_AUTH_SOCK -u git-test -i
    [git-test@chabuduo ~]$ git --version
    git version 2.21.0
    [git-test@chabuduo ~]$ cat > ~/.gitconfig
    [remote "origin"]
    	fetch = +refs/heads/*:refs/remotes/origin/*
    [git-test@chabuduo ~]$ cat ~/.gitconfig
    [remote "origin"]
    	fetch = +refs/heads/*:refs/remotes/origin/*
    [git-test@chabuduo ~]$ git clone git@github.com:git/git.git
    Cloning into 'git'...
    Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
    remote: Enumerating objects: 265113, done.
    remote: Total 265113 (delta 0), reused 0 (delta 0), pack-reused 265113
    Receiving objects: 100% (265113/265113), 113.20 MiB | 5.49 MiB/s, done.
    Resolving deltas: 100% (196542/196542), done.
    fatal: multiple updates for ref 'refs/remotes/origin/maint' not allowed

Re: [BUG] fetching all remote branches results in failed multiple updates

[SZEDER Gábor]

On Thu, Mar 07, 2019 at 09:44:47PM +0000, Alexander Huynh wrote:
> Hello all,
> 
> When running the latest release of git, I receive an error when attempting to
> fetch all remote branches from a repo:
> 
>     fatal: multiple updates for ref 'refs/remotes/origin/maint' not allowed
> 
> The specific ref that it fails on changes depending on the repository, but the
> end result is the repo isn't cloned.
> 
> The specific configuration that causes this bug is the `remote.origin.fetch`
> option, specifically:
> 
>     [remote "origin"]
>     	fetch = +refs/heads/*:refs/remotes/origin/*
> 
> These settings are listed as an example in "CONFIGURED REMOTE-TRACKING
> BRANCHES" of git-fetch(1), as well as expanded upon in
> https://stackoverflow.com/a/40739835.
> 
> I'm running git version 2.21.0.
> 
> Full reproduction steps are below:
> 
>     [root@chabuduo ~]# useradd -m git-test -s /bin/bash
>     [root@chabuduo ~]# sudo --preserve-env=SSH_AUTH_SOCK -u git-test -i
>     [git-test@chabuduo ~]$ git --version
>     git version 2.21.0
>     [git-test@chabuduo ~]$ cat > ~/.gitconfig
>     [remote "origin"]
>     	fetch = +refs/heads/*:refs/remotes/origin/*

Why do you do this?

First, this is the default refspec configuration that clone will fetch
and set up in the clone's config file anyway.  You don't have to set
it.

Second, setting repository-specific stuff like the default fetch
refspec in the global configuration is not such a good idea.

So the simplest workaround is:

  rm ~/.gitconfig; git clone git@github.com:git/git.git

>     [git-test@chabuduo ~]$ cat ~/.gitconfig
>     [remote "origin"]
>     	fetch = +refs/heads/*:refs/remotes/origin/*
>     [git-test@chabuduo ~]$ git clone git@github.com:git/git.git
>     Cloning into 'git'...
>     Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
>     remote: Enumerating objects: 265113, done.
>     remote: Total 265113 (delta 0), reused 0 (delta 0), pack-reused 265113
>     Receiving objects: 100% (265113/265113), 113.20 MiB | 5.49 MiB/s, done.
>     Resolving deltas: 100% (196542/196542), done.
>     fatal: multiple updates for ref 'refs/remotes/origin/maint' not allowed

Anyway, this is a side-effect of 515be83382 (clone: respect additional
configured fetch refspecs during initial fetch, 2018-11-14).  Since
then the same refspec comes up twice in remote->fetch, once from the
configuration and once the explicitly added default refspec.  And
since 'git clone' doesn't run a fully-fledged 'git fetch' and has its
own simple refs-to-refspec matching logic, it ends up trying to write
each ref twice, once for each of the two refspecs, in a single ref
transaction.

Re: [BUG] fetching all remote branches results in failed multiple updates

[Alexander Huynh]

On Fri, Mar 08, 2019 at 12:40:15AM +0100, SZEDER Gábor wrote:
> On Thu, Mar 07, 2019 at 09:44:47PM +0000, Alexander Huynh wrote:
> >     [git-test@chabuduo ~]$ cat > ~/.gitconfig
> >     [remote "origin"]
> >     	fetch = +refs/heads/*:refs/remotes/origin/*
>
> Why do you do this?
>
> First, this is the default refspec configuration that clone will fetch
> and set up in the clone's config file anyway.  You don't have to set
> it.

My mistake. I was under the assumption that this configuration variable was
needed for git to fetch all remote refs, and that this behaviour wasn't
default.

> Anyway, this is a side-effect of 515be83382 (clone: respect additional
> configured fetch refspecs during initial fetch, 2018-11-14).  Since
> then the same refspec comes up twice in remote->fetch, once from the
> configuration and once the explicitly added default refspec.  And
> since 'git clone' doesn't run a fully-fledged 'git fetch' and has its
> own simple refs-to-refspec matching logic, it ends up trying to write
> each ref twice, once for each of the two refspecs, in a single ref
> transaction.

Thanks for the clarification! One question: should we update the documentation
at https://git-scm.com/docs/git-fetch#CRTB to not list this as an example to
follow?

Thanks again,
Alex

Re: [BUG] fetching all remote branches results in failed multiple updates

[Junio C Hamano]

Alexander Huynh <alex@grande.coffee> writes:

> Thanks for the clarification! One question: should we update the documentation
> at https://git-scm.com/docs/git-fetch#CRTB to not list this as an example to
> follow?

The example itself is valuable.  The way it is labeled might be
misleading to some people (although personally I do not find it
so).  It says "Typically such a variable may look like this" and
shows the fetch refspec that is given to a repository cloned from
anywhere by default.

I think it might help to update the text to clearly state that the
sample one is what you get in a clone without doing anything
special.  The example is primarily there to show you how each part
of the fetch refspec is used in the operations described in the
following paragraphs,and it is not there to tell you that you have
to add duplicate yourself to make these things described in the
following paragraphs to happen.

Re: [BUG] fetching all remote branches results in failed multiple updates

[Alexander Huynh]

Hi all,

I appreciate the help offered in identifying what I was doing incorrectly.

Upon further investigation, I remembered why I added the original
`origin.fetch` parameter: I was compensating for shallow clones not fetching
all remote refs.

Here's the use case that I have:

    [root@chabuduo ~]# useradd -m git-test -s /bin/bash
    [root@chabuduo ~]# sudo --preserve-env=SSH_AUTH_SOCK -u git-test -i
    [git-test@chabuduo ~]$ git clone --depth=1 git@github.com:git/git.git
    Cloning into 'git'...
    The authenticity of host 'github.com (192.30.255.112)' can't be established.
    RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'github.com,192.30.255.112' (RSA) to the list of known hosts.
    remote: Enumerating objects: 3664, done.
    remote: Counting objects: 100% (3664/3664), done.
    remote: Compressing objects: 100% (3282/3282), done.
    remote: Total 3664 (delta 276), reused 1921 (delta 232), pack-reused 0
    Receiving objects: 100% (3664/3664), 8.31 MiB | 9.61 MiB/s, done.
    Resolving deltas: 100% (276/276), done.
    [git-test@chabuduo ~]$ cd git/
    [git-test@chabuduo git]$ git branch -a
    * master
      remotes/origin/HEAD -> origin/master
      remotes/origin/master

Running a shallow clone results in only the remote `HEAD` ref being fetched,
as observed by the last `git branch -a`.

No additional refs are fetched, even when promoting the shallow repository to
a full one:

    [git-test@chabuduo git]$ git pull --unshallow
    remote: Enumerating objects: 255783, done.
    remote: Counting objects: 100% (255777/255777), done.
    remote: Compressing objects: 100% (62079/62079), done.
    remote: Total 253423 (delta 193725), reused 249056 (delta 189413), pack-reused 0
    Receiving objects: 100% (253423/253423), 94.28 MiB | 33.96 MiB/s, done.
    Resolving deltas: 100% (193725/193725), completed with 2155 local objects.
    remote: Enumerating objects: 709, done.
    remote: Total 709 (delta 0), reused 0 (delta 0), pack-reused 709
    Receiving objects: 100% (709/709), 363.70 KiB | 12.12 MiB/s, done.
    From github.com:git/git
     * [new tag]             gitgui-0.10.0    -> gitgui-0.10.0
     * [new tag]             gitgui-0.10.1    -> gitgui-0.10.1
    ...
    Already up to date.
    [git-test@chabuduo git]$ git branch -a
    * master
      remotes/origin/HEAD -> origin/master
      remotes/origin/master

How I've managed to work around the issue is by force fetching all refs, as
explained in the example at https://git-scm.com/docs/git-fetch#CRTB:

    [git-test@chabuduo git]$ git fetch origin '+refs/heads/*:refs/remotes/origin/*'
    remote: Enumerating objects: 1218, done.
    remote: Counting objects: 100% (1218/1218), done.
    remote: Total 7699 (delta 1218), reused 1218 (delta 1218), pack-reused 6481
    Receiving objects: 100% (7699/7699), 4.31 MiB | 17.22 MiB/s, done.
    Resolving deltas: 100% (4881/4881), completed with 435 local objects.
    From github.com:git/git
     * [new branch]            maint      -> origin/maint
     * [new branch]            next       -> origin/next
     * [new branch]            pu         -> origin/pu
     * [new branch]            todo       -> origin/todo

This populates the missing references from a shallow clone:

    [git-test@chabuduo git]$ git branch -a
    * master
      remotes/origin/HEAD -> origin/master
      remotes/origin/maint
      remotes/origin/master
      remotes/origin/next
      remotes/origin/pu
      remotes/origin/todo

I've created an alias that allows for filling out refs for future shallow
clones:

    % git config --global alias.fetcha '!f() { remote="${1:-origin}"; git fetch "$remote" "+refs/heads/*:refs/remotes/$remote/*"; }; f'

I'm also interested in more efficient ways to obtain all refs from shallow
clones, if anyone has better ideas.

Thanks for taking the time to help me understand,
Alex

Re: [BUG] fetching all remote branches results in failed multiple updates

[Alexander Huynh]

Hi all,

Sorry to bother one last time. After a bit more digging, I found that the
problem was with my workflow.

Shallow clones set `remote.origin.fetch` to one branch, so what I was doing in
the past was overriding the respository-specific `remote.origin.fetch` with my
global config.

I've now changed my workflow to perform a shallow clone at first, `cd` into
the repository, and locally set `remote.origin.fetch` to
`+refs/heads/*:refs/remotes/origin/*`.

Thanks again,
Alex