[BUG] GIT_SSH_COMMAND is not being decomposed

[BUG] GIT_SSH_COMMAND is not being decomposed

[Randall S. Becker]

I am encountering a problem on one of our NonStop platform variants where
the GIT_SSH_COMMAND string is not being broken into constituent parts. This
is causing SSH to not run properly. As background, SSH is not in a standard
location and has non-standard required arguments. This also occurs with
core.sshCommand. The situation is:

git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'

which correctly sets .gitconfig as:

[core]
        sshCommand = /G/system/zssh/sshossz5 -Q

When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch

We get the partial trace:
14:19:56.027088 trace: built-in: git fetch
14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
user@host

The same trace on our systems that actually do work results in:
14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q' -G
user@host

I need help resolving why this is happening (as in where to look and debug
the situation).

Urgently,
Randall

-- Brief whoami:
NonStop developer since approximately 211288444200000000
UNIX developer since approximately 421664400
-- In my real life, I talk too much.

Re: [BUG] GIT_SSH_COMMAND is not being decomposed

[Ævar Arnfjörð Bjarmason]

On Sat, Apr 13 2019, Randall S. Becker wrote:

> I am encountering a problem on one of our NonStop platform variants where
> the GIT_SSH_COMMAND string is not being broken into constituent parts. This
> is causing SSH to not run properly. As background, SSH is not in a standard
> location and has non-standard required arguments. This also occurs with
> core.sshCommand. The situation is:
>
> git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'
>
> which correctly sets .gitconfig as:
>
> [core]
>         sshCommand = /G/system/zssh/sshossz5 -Q
>
> When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch
>
> We get the partial trace:
> 14:19:56.027088 trace: built-in: git fetch
> 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
> user@host
>
> The same trace on our systems that actually do work results in:
> 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q' -G
> user@host
>
> I need help resolving why this is happening (as in where to look and debug
> the situation).

This doesn't seem to be documented *explicitly* (except between the
lines & inferred), but it's only supported to pass a *command* there,
i.e. the path of the ssh binary. See the code around get_ssh_command()
in connect.c. The whole env/config value we look up gets passed as one.

So if you need arguments you need to create a wrapper script and set ssh
command to that script.

RE: [BUG] GIT_SSH_COMMAND is not being decomposed

[Randall S. Becker]

On April 13, 2019 16:40, Ævar Arnfjörð Bjarmason wrote:
> On Sat, Apr 13 2019, Randall S. Becker wrote:
> 
> > I am encountering a problem on one of our NonStop platform variants
> > where the GIT_SSH_COMMAND string is not being broken into constituent
> > parts. This is causing SSH to not run properly. As background, SSH is
> > not in a standard location and has non-standard required arguments.
> > This also occurs with core.sshCommand. The situation is:
> >
> > git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'
> >
> > which correctly sets .gitconfig as:
> >
> > [core]
> >         sshCommand = /G/system/zssh/sshossz5 -Q
> >
> > When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch
> >
> > We get the partial trace:
> > 14:19:56.027088 trace: built-in: git fetch
> > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
> > user@host
> >
> > The same trace on our systems that actually do work results in:
> > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q' -G
> > user@host
> >
> > I need help resolving why this is happening (as in where to look and
> > debug the situation).
> 
> This doesn't seem to be documented *explicitly* (except between the lines &
> inferred), but it's only supported to pass a *command* there, i.e. the path of
> the ssh binary. See the code around get_ssh_command() in connect.c. The
> whole env/config value we look up gets passed as one.
> 
> So if you need arguments you need to create a wrapper script and set ssh
> command to that script.

Thanks. I didn't know that, because this technique worked for years on the older platform variant. My wrapper works fine on the newer system.

With appreciation,
Randall

Re: [BUG] GIT_SSH_COMMAND is not being decomposed

[SZEDER Gábor]

On Sat, Apr 13, 2019 at 10:39:35PM +0200, Ævar Arnfjörð Bjarmason wrote:
> 
> On Sat, Apr 13 2019, Randall S. Becker wrote:
> 
> > I am encountering a problem on one of our NonStop platform variants where
> > the GIT_SSH_COMMAND string is not being broken into constituent parts. This
> > is causing SSH to not run properly. As background, SSH is not in a standard
> > location and has non-standard required arguments. This also occurs with
> > core.sshCommand. The situation is:
> >
> > git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'
> >
> > which correctly sets .gitconfig as:
> >
> > [core]
> >         sshCommand = /G/system/zssh/sshossz5 -Q
> >
> > When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch
> >
> > We get the partial trace:
> > 14:19:56.027088 trace: built-in: git fetch
> > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
> > user@host
> >
> > The same trace on our systems that actually do work results in:
> > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q' -G
> > user@host
> >
> > I need help resolving why this is happening (as in where to look and debug
> > the situation).
> 
> This doesn't seem to be documented *explicitly* (except between the
> lines & inferred), but it's only supported to pass a *command* there,
> i.e. the path of the ssh binary.

'man git' it quite explicit about this:

  $GIT_SSH_COMMAND takes precedence over $GIT_SSH, and is interpreted
  by the shell, which allows additional arguments to be included.
  $GIT_SSH on the other hand must be just the path to a program (which
  can be a wrapper shell script, if additional arguments are needed).

Quick test shows that the implementation agrees with the
documentation:

  $ GIT_TRACE=2 GIT_SSH_COMMAND='/usr/bin/ssh -v' git push -n github
  23:39:02.048870 git.c:419               trace: built-in: git push -n github
  23:39:02.060821 run-command.c:643       trace: run_command: unset GIT_PREFIX; '/usr/bin/ssh -v' git@github.com 'git-receive-pack '\''/szeder/git'\'''
  OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
  debug1: Reading configuration data /home/szeder/.ssh/config
  <... snipt rest of the verbose ssh output ...>

And the config setting works, too:

  $ GIT_TRACE=2 git -c core.sshCommand='/usr/bin/ssh -v' push -n github
  23:42:55.277776 git.c:439               trace: built-in: git push -n github
  23:42:55.285149 run-command.c:663       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_PREFIX; '/usr/bin/ssh -v' git@github.com 'git-receive-pack '\''/szeder/git'\'''
  OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
  debug1: Reading configuration data /home/szeder/.ssh/config
  <...>

Note that in both cases the trace shows '/usr/bin/ssh -v', IOW neither
$GIT_SSH_COMMAND nor 'core.sshCommand' are broken up.

But this is just an avarage Linux box, so perhaps this is a
NonStop-specific issue?


> See the code around get_ssh_command()
> in connect.c. The whole env/config value we look up gets passed as one.
> 
> So if you need arguments you need to create a wrapper script and set ssh
> command to that script.

RE: [BUG] GIT_SSH_COMMAND is not being decomposed

[Randall S. Becker]

On April 13, 2019 17:48, SZEDER Gábor wrote:
> To: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Cc: Randall S. Becker <rsbecker@nexbridge.com>; git@vger.kernel.org
> Subject: Re: [BUG] GIT_SSH_COMMAND is not being decomposed
> 
> On Sat, Apr 13, 2019 at 10:39:35PM +0200, Ævar Arnfjörð Bjarmason wrote:
> > On Sat, Apr 13 2019, Randall S. Becker wrote:
> > > I am encountering a problem on one of our NonStop platform variants
> > > where the GIT_SSH_COMMAND string is not being broken into
> > > constituent parts. This is causing SSH to not run properly. As
> > > background, SSH is not in a standard location and has non-standard
> > > required arguments. This also occurs with core.sshCommand. The
> situation is:
> > >
> > > git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'
> > >
> > > which correctly sets .gitconfig as:
> > >
> > > [core]
> > >         sshCommand = /G/system/zssh/sshossz5 -Q
> > >
> > > When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch
> > >
> > > We get the partial trace:
> > > 14:19:56.027088 trace: built-in: git fetch
> > > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
> > > user@host
> > >
> > > The same trace on our systems that actually do work results in:
> > > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q'
> > > -G user@host
> > >
> > > I need help resolving why this is happening (as in where to look and
> > > debug the situation).
> >
> > This doesn't seem to be documented *explicitly* (except between the
> > lines & inferred), but it's only supported to pass a *command* there,
> > i.e. the path of the ssh binary.
> 
> 'man git' it quite explicit about this:
> 
>   $GIT_SSH_COMMAND takes precedence over $GIT_SSH, and is interpreted
>   by the shell, which allows additional arguments to be included.
>   $GIT_SSH on the other hand must be just the path to a program (which
>   can be a wrapper shell script, if additional arguments are needed).
> 
> Quick test shows that the implementation agrees with the
> documentation:
> 
>   $ GIT_TRACE=2 GIT_SSH_COMMAND='/usr/bin/ssh -v' git push -n github
>   23:39:02.048870 git.c:419               trace: built-in: git push -n github
>   23:39:02.060821 run-command.c:643       trace: run_command: unset
> GIT_PREFIX; '/usr/bin/ssh -v' git@github.com 'git-receive-pack
> '\''/szeder/git'\'''
>   OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
>   debug1: Reading configuration data /home/szeder/.ssh/config
>   <... snipt rest of the verbose ssh output ...>
> 
> And the config setting works, too:
> 
>   $ GIT_TRACE=2 git -c core.sshCommand='/usr/bin/ssh -v' push -n github
>   23:42:55.277776 git.c:439               trace: built-in: git push -n github
>   23:42:55.285149 run-command.c:663       trace: run_command: unset
> GIT_CONFIG_PARAMETERS GIT_PREFIX; '/usr/bin/ssh -v' git@github.com 'git-
> receive-pack '\''/szeder/git'\'''
>   OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
>   debug1: Reading configuration data /home/szeder/.ssh/config
>   <...>
> 
> Note that in both cases the trace shows '/usr/bin/ssh -v', IOW neither
> $GIT_SSH_COMMAND nor 'core.sshCommand' are broken up.
> 
> But this is just an avarage Linux box, so perhaps this is a NonStop-specific
> issue?

I'm sure it is a NonStop issue. It is interesting that 2.21.0 broke the string apart on the older NonStop variant and not the newer one. But looking at the code, I can't imagine how the string was broken up into parts. Makes no sense at all with xstrdup() and argv_array_push(). 

> > See the code around get_ssh_command()
> > in connect.c. The whole env/config value we look up gets passed as one.
> >
> > So if you need arguments you need to create a wrapper script and set
> > ssh command to that script.

Going forward, I'm going to use (and strongly recommend) a wrapper on both NonStop variants. It's the right way to go, and not only a trivial script but makes configuring communication with upstream servers much easier (there are potentially multiple TCP/IP stacks and multiple SSH d databases available that need to be selected on the sshoss command line). Managing all that in one place is easier than having each user worry about it changing over time.

Thanks again,
Randall

RE: [BUG] GIT_SSH_COMMAND is not being decomposed

[Randall S. Becker]

On April 13, 2019 17:48, SZEDER Gábor wrote:
> On Sat, Apr 13, 2019 at 10:39:35PM +0200, Ævar Arnfjörð Bjarmason wrote:
> > On Sat, Apr 13 2019, Randall S. Becker wrote:
> >
> > > I am encountering a problem on one of our NonStop platform variants
> > > where the GIT_SSH_COMMAND string is not being broken into
> > > constituent parts. This is causing SSH to not run properly. As
> > > background, SSH is not in a standard location and has non-standard
> > > required arguments. This also occurs with core.sshCommand. The
> situation is:
> > >
> > > git config --global core.sshCommand '/G/system/zssh/sshossz5 -Q'
> > >
> > > which correctly sets .gitconfig as:
> > >
> > > [core]
> > >         sshCommand = /G/system/zssh/sshossz5 -Q
> > >
> > > When git is run with GIT_TRACE=true GIT_PACKET_TRACE=true git fetch
> > >
> > > We get the partial trace:
> > > 14:19:56.027088 trace: built-in: git fetch
> > > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5 -Q' -G
> > > user@host
> > >
> > > The same trace on our systems that actually do work results in:
> > > 14:19:56.029895 trace: run_command: '/G/system/zssh/sshossz5' '-Q'
> > > -G user@host
> > >
> > > I need help resolving why this is happening (as in where to look and
> > > debug the situation).
> >
> > This doesn't seem to be documented *explicitly* (except between the
> > lines & inferred), but it's only supported to pass a *command* there,
> > i.e. the path of the ssh binary.
> 
> 'man git' it quite explicit about this:
> 
>   $GIT_SSH_COMMAND takes precedence over $GIT_SSH, and is interpreted
>   by the shell, which allows additional arguments to be included.
>   $GIT_SSH on the other hand must be just the path to a program (which
>   can be a wrapper shell script, if additional arguments are needed).
> 
> Quick test shows that the implementation agrees with the
> documentation:
> 
>   $ GIT_TRACE=2 GIT_SSH_COMMAND='/usr/bin/ssh -v' git push -n github
>   23:39:02.048870 git.c:419               trace: built-in: git push -n github
>   23:39:02.060821 run-command.c:643       trace: run_command: unset
> GIT_PREFIX; '/usr/bin/ssh -v' git@github.com 'git-receive-pack
> '\''/szeder/git'\'''
>   OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
>   debug1: Reading configuration data /home/szeder/.ssh/config
>   <... snipt rest of the verbose ssh output ...>
> 
> And the config setting works, too:
> 
>   $ GIT_TRACE=2 git -c core.sshCommand='/usr/bin/ssh -v' push -n github
>   23:42:55.277776 git.c:439               trace: built-in: git push -n github
>   23:42:55.285149 run-command.c:663       trace: run_command: unset
> GIT_CONFIG_PARAMETERS GIT_PREFIX; '/usr/bin/ssh -v' git@github.com
> 'git-receive-pack '\''/szeder/git'\'''
>   OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
>   debug1: Reading configuration data /home/szeder/.ssh/config
>   <...>
> 
> Note that in both cases the trace shows '/usr/bin/ssh -v', IOW neither
> $GIT_SSH_COMMAND nor 'core.sshCommand' are broken up.
> 
> But this is just an avarage Linux box, so perhaps this is a NonStop-specific
> issue?
> 
> 
> > See the code around get_ssh_command()
> > in connect.c. The whole env/config value we look up gets passed as one.
> >
> > So if you need arguments you need to create a wrapper script and set
> > ssh command to that script.

What is strange is that GIT_SSH_COMMAND='/usr/bin/ssh -v' should not execute if we are just looking at an object path. It should be broken into '/usr/bin/ssh' and '-v' otherwise spawn* or exec* will not execute it. I'm still trying to understand why I can successfully do things like the following:

$ GIT_SSH_COMMAND="ssh -i ~/.ssh/myid" git fetch

on virtually any platform at my disposal (Windows, Ubuntu, MacOS, the older NonStop variant), and have that work with no problem. Somewhere after get_ssh_command(), the command is being interpreted it its parts either as a shell or something else (still trying to find that).

Re: [BUG] GIT_SSH_COMMAND is not being decomposed

[Andreas Schwab]

On Apr 15 2019, "Randall S. Becker" <rsbecker@nexbridge.com> wrote:

> on virtually any platform at my disposal (Windows, Ubuntu, MacOS, the
> older NonStop variant), and have that work with no problem. Somewhere
> after get_ssh_command(), the command is being interpreted it its parts
> either as a shell or something else (still trying to find that).

See run-command.c:prepare_shell_cmd, if the command contains shell meta
characters it is passed to sh -c without further quoting.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [BUG] GIT_SSH_COMMAND is not being decomposed

[Johannes Sixt]

Am 15.04.19 um 21:20 schrieb Randall S. Becker:
> What is strange is that GIT_SSH_COMMAND='/usr/bin/ssh -v' should not
> execute if we are just looking at an object path. It should be broken
> into '/usr/bin/ssh' and '-v' otherwise spawn* or exec* will not
> execute it. I'm still trying to understand why I can successfully do
> things like the following:>
> $ GIT_SSH_COMMAND="ssh -i ~/.ssh/myid" git fetch
> 
> on virtually any platform at my disposal (Windows, Ubuntu, MacOS, the
> older NonStop variant), and have that work with no problem. Somewhere
> after get_ssh_command(), the command is being interpreted it its
> parts either as a shell or something else (still trying to find
> that).
Have a look at fill_ssh_args() and determine_ssh_variant() in connect.c.
Quite a lot is going on there. I don't see immediately why it doesn't
follow the usual logic in your case, though.

-- Hannes

RE: [BUG] GIT_SSH_COMMAND is not being decomposed

[Randall S. Becker]

On April 15, 2019 17:14, Andreas Schwab wrote:
> On Apr 15 2019, "Randall S. Becker" <rsbecker@nexbridge.com> wrote:
> 
> > on virtually any platform at my disposal (Windows, Ubuntu, MacOS, the
> > older NonStop variant), and have that work with no problem. Somewhere
> > after get_ssh_command(), the command is being interpreted it its parts
> > either as a shell or something else (still trying to find that).
> 
> See run-command.c:prepare_shell_cmd, if the command contains shell meta
> characters it is passed to sh -c without further quoting.
> 
> Andreas.

Well crap. That explains far too much about what is happening. 😊. One of the special parameters is specified as -S \$ZSSH2 (example, referring to the process name - which begin with $ and have to be escaped with \). This obviously triggers the alternate path and has been problematic. On the older systems, we found fewer (a.k.a just about none) uses of this parameter, so never encountered it. On the newer systems, virtually everyone is using -S. Ergo, behavioural differences. That explains a whole lot of why we need a wrapper script. Thanks for the pointer to the strcspn() reference. I can stop obsessing about this (thanks too Johannes, Szeder, Ævar) for the help.

As a suggestion, with people who know how to escape stuff properly (or not), perhaps we can select the alternate behaviour explicitly using a core.sshIgnoreEscape=true/false option. Thoughts on that?

Regards,
Randall

Re: [BUG] GIT_SSH_COMMAND is not being decomposed

[Junio C Hamano]

"Randall S. Becker" <rsbecker@nexbridge.com> writes:

> As a suggestion, with people who know how to escape stuff properly
> (or not), perhaps we can select the alternate behaviour explicitly
> using a core.sshIgnoreEscape=true/false option. Thoughts on that?

The semantics of prepare_shell_cmd() is, regardless of any "funny
characters" on the command line, the spawned command MUST behave AS
IF it was run via the shell.  The strcspn() trick is there merely as
a low-level optimization so that we do not have to say

	sh -c a-single-token

which would be exactly the same as running

	a-single-token

The most typical use of that strcspn() trick is to ensure that

	GIT_SSH_COMMAND="the-command and its arguments"

would not attempt to run a command with a long and funny name
"the-command and its arguments" somewhere on the $PATH without any
parameter, and instead run

	sh -c "the-command and its arguments"

i.e. run the "the-command" with three parameters (and perhaps more
built-in parameters prepared by the caller in the ssh connection
codepath).