False negative authentication with multiple accounts on a SSH-GIT server

False negative authentication with multiple accounts on a SSH-GIT server

[Harley Paterson]

Hello,

I've noticed an edge case in Git when a user has two Git accounts on a single server - as might be common for a personal account and a work account on a Git forge (Github, GitLab, Bitbucket...)

When attempting SSH login, `ssh` and Git will eagerly connect with the first matching key. This may or may not be the key right key for the repository the user needs. As a result, Git clones, pulls, and pushes will fail with the `Repository not found` if the wrong key is tried first.

For example, the user `alice` has two accounts on the host `git-server.com`. `alice`'s accounts are `alice-work`, and `alice-personal`. `alice-work` has access to the `foo` repository, and `alice-personal` has access to the `bar` repository.

`alice` attempts to clone `foo` with both her `alice-work` and `alice-personal` keys in her SSH Agent. SSH Agent does not define which key it will attempt first, so SSH may connect successfully to `git-server.com` with her `alice-personal` keys, which do not have access to the `foo` repository.

I'd be interested in your opinions for fixes to this. I am willing to make a patch, although my knowledge of the Git codebase isn't perfect.

- Should Git servers provide distinctly different error messages for `access denied`, and `repository does not exist`? Currently the server immediately closes the connection in this case, so `transport.c:handshake()` with fail when attempting to `discover_version()` because the reader hits the EOF. Perhaps the server could send a hypothetical access denied packet here, and a more appropriate error generated? 

Can anyone point me to where in the Git codebase the daemon receives and responds to these requests? I haven't found it yet, if I wanted to patch this.

- Should Git provide a `-i` option to allow the user to choose an SSH key, which could be added to the SSH subprocess's command line?

- Should Git attempt to iterate over all keys in the SSH Agent when the connection is setup, testing the connection to check if each connected key has access to the target repository, before giving up and reporting an error? This may be difficult looking at the current behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists paths to files (which could be plugged into `ssh -i`), just the key signature. Does anyone know of any SSH/SSH-Agent tricks which might help with this?

All the best,



H Paterson.

Re: False negative authentication with multiple accounts on a SSH-GIT server

[M. Buecher]

Hi there,

you can use GIT_SSH_COMMAND or config 'core.sshCommand'.
For more flexibility use wrappers like ssh-ident.

Recommended read: https://superuser.com/a/1616186/557798

Hope this helps
Kind regards
Maddes


On 2021-01-19 07:17, Harley Paterson wrote:
> Hello,
> 
> I've noticed an edge case in Git when a user has two Git accounts on a
> single server - as might be common for a personal account and a work
> account on a Git forge (Github, GitLab, Bitbucket...)
> 
> When attempting SSH login, `ssh` and Git will eagerly connect with the
> first matching key. This may or may not be the key right key for the
> repository the user needs. As a result, Git clones, pulls, and pushes
> will fail with the `Repository not found` if the wrong key is tried
> first.
> 
> For example, the user `alice` has two accounts on the host
> `git-server.com`. `alice`'s accounts are `alice-work`, and
> `alice-personal`. `alice-work` has access to the `foo` repository, and
> `alice-personal` has access to the `bar` repository.
> 
> `alice` attempts to clone `foo` with both her `alice-work` and
> `alice-personal` keys in her SSH Agent. SSH Agent does not define
> which key it will attempt first, so SSH may connect successfully to
> `git-server.com` with her `alice-personal` keys, which do not have
> access to the `foo` repository.
> 
> I'd be interested in your opinions for fixes to this. I am willing to
> make a patch, although my knowledge of the Git codebase isn't perfect.
> 
> - Should Git servers provide distinctly different error messages for
> `access denied`, and `repository does not exist`? Currently the server
> immediately closes the connection in this case, so
> `transport.c:handshake()` with fail when attempting to
> `discover_version()` because the reader hits the EOF. Perhaps the
> server could send a hypothetical access denied packet here, and a more
> appropriate error generated?
> 
> Can anyone point me to where in the Git codebase the daemon receives
> and responds to these requests? I haven't found it yet, if I wanted to
> patch this.
> 
> - Should Git provide a `-i` option to allow the user to choose an SSH
> key, which could be added to the SSH subprocess's command line?
> 
> - Should Git attempt to iterate over all keys in the SSH Agent when
> the connection is setup, testing the connection to check if each
> connected key has access to the target repository, before giving up
> and reporting an error? This may be difficult looking at the current
> behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists paths
> to files (which could be plugged into `ssh -i`), just the key
> signature. Does anyone know of any SSH/SSH-Agent tricks which might
> help with this?
> 
> All the best,
> 
> 
> 
> H Paterson.

Re: False negative authentication with multiple accounts on a SSH-GIT server

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 4216 bytes --]

On 2021-01-19 at 06:17:21, Harley Paterson wrote:
> Hello,
> 
> I've noticed an edge case in Git when a user has two Git accounts on a
> single server - as might be common for a personal account and a work
> account on a Git forge (Github, GitLab, Bitbucket...)
> 
> When attempting SSH login, `ssh` and Git will eagerly connect with the
> first matching key. This may or may not be the key right key for the
> repository the user needs. As a result, Git clones, pulls, and pushes
> will fail with the `Repository not found` if the wrong key is tried
> first.
> 
> For example, the user `alice` has two accounts on the host
> `git-server.com`. `alice`'s accounts are `alice-work`, and
> `alice-personal`. `alice-work` has access to the `foo` repository, and
> `alice-personal` has access to the `bar` repository.

Yes, this is because SSH authentication happens before any command is
run.  The server never knows what resource is being requested until the
user is already authenticated.

> `alice` attempts to clone `foo` with both her `alice-work` and
> `alice-personal` keys in her SSH Agent. SSH Agent does not define
> which key it will attempt first, so SSH may connect successfully to
> `git-server.com` with her `alice-personal` keys, which do not have
> access to the `foo` repository.
> 
> I'd be interested in your opinions for fixes to this. I am willing to
> make a patch, although my knowledge of the Git codebase isn't perfect.

We've documented how to deal with situation properly in the FAQ, which
you can see at gitfaq(7) or
https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt-HowdoIusemultipleaccountswiththesamehostingproviderusingSSH

Is there some reason that this doesn't work for you?

> - Should Git servers provide distinctly different error messages for
>   `access denied`, and `repository does not exist`? Currently the
>   server immediately closes the connection in this case, so
>   `transport.c:handshake()` with fail when attempting to
>   `discover_version()` because the reader hits the EOF. Perhaps the
>   server could send a hypothetical access denied packet here, and a
>   more appropriate error generated?

Unfortunately, this leaks whether a repository exists.  If Company XYZ
has a repository for each of its clients, it then becomes easy to see if
Company XYZ is doing work for a particular client by trying to see if a
repository exists.  This would be bad and a huge violation of privacy,
so nobody is likely do to that.  I can tell you as one of the staff who
maintains the GitHub Git service, we're not likely to change the
behavior, and I suspect nobody else is, either.

> Can anyone point me to where in the Git codebase the daemon receives
> and responds to these requests? I haven't found it yet, if I wanted to
> patch this.
> 
> - Should Git provide a `-i` option to allow the user to choose an SSH
>   key, which could be added to the SSH subprocess's command line?

You can use GIT_SSH_COMMAND or `core.sshcommand` to set this.  We don't
provide such an option because the user is not necessarily using
OpenSSH and different SSH programs provide this feature differently.

> - Should Git attempt to iterate over all keys in the SSH Agent when
>   the connection is setup, testing the connection to check if each
>   connected key has access to the target repository, before giving up
>   and reporting an error? This may be difficult looking at the current
>   behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists
>   paths to files (which could be plugged into `ssh -i`), just the key
>   signature. Does anyone know of any SSH/SSH-Agent tricks which might
>   help with this?

For historical reasons, we retry SSH connections in Git LFS and users
frequently complain about being prompted for their password multiple
times before giving up, so I suspect this would not be a good experience
for the user.  Users may also have multiple keys and have configured
their agent to prompt them for a passphrase before using the key, and
I'm pretty sure those users will complain very loudly if we do this.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: False negative authentication with multiple accounts on a SSH-GIT server

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 20 2021, brian m. carlson wrote:

> On 2021-01-19 at 06:17:21, Harley Paterson wrote:
>> Hello,
>> 
>> I've noticed an edge case in Git when a user has two Git accounts on a
>> single server - as might be common for a personal account and a work
>> account on a Git forge (Github, GitLab, Bitbucket...)
>> 
>> When attempting SSH login, `ssh` and Git will eagerly connect with the
>> first matching key. This may or may not be the key right key for the
>> repository the user needs. As a result, Git clones, pulls, and pushes
>> will fail with the `Repository not found` if the wrong key is tried
>> first.
>> 
>> For example, the user `alice` has two accounts on the host
>> `git-server.com`. `alice`'s accounts are `alice-work`, and
>> `alice-personal`. `alice-work` has access to the `foo` repository, and
>> `alice-personal` has access to the `bar` repository.
>
> Yes, this is because SSH authentication happens before any command is
> run.  The server never knows what resource is being requested until the
> user is already authenticated.
>
>> `alice` attempts to clone `foo` with both her `alice-work` and
>> `alice-personal` keys in her SSH Agent. SSH Agent does not define
>> which key it will attempt first, so SSH may connect successfully to
>> `git-server.com` with her `alice-personal` keys, which do not have
>> access to the `foo` repository.
>> 
>> I'd be interested in your opinions for fixes to this. I am willing to
>> make a patch, although my knowledge of the Git codebase isn't perfect.
>
> We've documented how to deal with situation properly in the FAQ, which
> you can see at gitfaq(7) or
> https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt-HowdoIusemultipleaccountswiththesamehostingproviderusingSSH
>
> Is there some reason that this doesn't work for you?
>
>> - Should Git servers provide distinctly different error messages for
>>   `access denied`, and `repository does not exist`? Currently the
>>   server immediately closes the connection in this case, so
>>   `transport.c:handshake()` with fail when attempting to
>>   `discover_version()` because the reader hits the EOF. Perhaps the
>>   server could send a hypothetical access denied packet here, and a
>>   more appropriate error generated?
>
> Unfortunately, this leaks whether a repository exists.  If Company XYZ
> has a repository for each of its clients, it then becomes easy to see if
> Company XYZ is doing work for a particular client by trying to see if a
> repository exists.

I wonder how many hosting providers are confident that the code involved
in this isn't vulnerable to a timing attack.

> This would be bad and a huge violation of privacy, so nobody is likely
> do to that.  I can tell you as one of the staff who maintains the
> GitHub Git service, we're not likely to change the behavior, and I
> suspect nobody else is, either.

Let's be clear, it's a violation of privacy in the case that involves a
private repo.

But the fact that when you e.g. clone git@github.com:git/git.git with no
valid key and get a "Permission denied" is just an emergent effect of
how people string together openssh & their own auth solutions in the
wild.

You can just as well let the dialog proceed down to a shell without a
valid key or password, and then check that the user wants to run "git
upload-pack git/git.git", and then decide "sure, it's a public repo" and
proceed.

Indeed that's what e.g. GitHub does when you make up a username &
password to clone a public repo over https.

    # Under OpenSSH, See https://tools.ietf.org/html/rfc4252#section-5.2
    Match User git
        PermitEmptyPasswords yes
        AuthenticationMethods none

Why would a site like GitHub treat ssh differently than https here? I
think it just comes down to openssh's support for use-case being a bit
of a pain to configure.

I don't know of a way to do that *and* pass down to e.g. a login shell
wrapper script if/how the user was authenticated.

Hrm, maybe I take that back. I suppose you can make the default shell
for the "git" user "git-shell-public" and set the rw login shell to
"git-shell-private" via an Authorized{Keys,Principals}Command". But I
haven't tested it.

Anyway, just saying that there's bad error messages for good reasons,
and bad error messages for lazyness/not wanting to deal with the edge
case.

What Harley Paterson is complaining about is sometimes in the "good
reasons" bucket, but sometimes just server operator lazyness, or not
caring about this marginal edge case enough to provide a better error
message.

Re: False negative authentication with multiple accounts on a SSH-GIT server

[Jeff King]

On Wed, Jan 20, 2021 at 01:42:56PM +0100, Ævar Arnfjörð Bjarmason wrote:

> > Unfortunately, this leaks whether a repository exists.  If Company XYZ
> > has a repository for each of its clients, it then becomes easy to see if
> > Company XYZ is doing work for a particular client by trying to see if a
> > repository exists.
> 
> I wonder how many hosting providers are confident that the code involved
> in this isn't vulnerable to a timing attack.

I would say "not very certain" in the case of GitHub. I don't recall any
analysis of the timing ever having happened.

> Why would a site like GitHub treat ssh differently than https here? I
> think it just comes down to openssh's support for use-case being a bit
> of a pain to configure.

GitHub doesn't use openssh at all; ssh sessions are terminated at a
custom load-balancing/proxy layer daemon.

I agree that servers in general probably could accept this notion of "no
valid user" for ssh sessions. I just think it's something that nobody
has asked for. I.e., if you are not planning to be an authenticated
user, then why use ssh in the first place? Certainly you could ask "why
not", but AFAIK this is the first time it has come up.

There is one protocol issue I'm not sure of, though. My understanding of
ssh auth is that the client offers keys in sequence, and the server says
"yes" or "no" on each. We'd want to continue saying "no" on keys that we
don't know about, so the client may keep trying other keys to which we
might say "yes". But after having seen all of the keys, what then?

We would want the server to say "OK, even though we don't like any of
your keys, let's start a session anyway". Is that possible within the
protocol? I think having the server say "OK, now try password auth" is
not a good idea. If the client asks for a dummy password, that is likely
to be either confusing, or disrupt sessions without a human present.

But I don't know the ssh protocol very well, so perhaps my understanding
of what is possible totally wrong. https has this much easier because
unauthenticated requests are totally normal, and the server sees the
auth along with the whole request at the same time.

-Peff

Re: False negative authentication with multiple accounts on a SSH-GIT server

[Harley Paterson]

Repost: Accedentialy forgot to CC this to the mailing list.

>> Unfortunately, this leaks whether a repository exists.  If Company XYZ
>> has a repository for each of its clients, it then becomes easy to see if
>> Company XYZ is doing work for a particular client by trying to see if a
>> repository exists.
> 
> I wonder how many hosting providers are confident that the code involved
> in this isn't vulnerable to a timing attack.
>
> I would say "not very certain" in the case of GitHub. I don't recall any
> analysis of the timing ever having happened.



I have to agree with this... It's certainly a risk, but if an attacker has enough
information to guess the repository names/URLs without triggering a rate
limiter, they already have a pretty good idea who Company XYZ is likely working
for, and what on.

HP.