General support for ! in git-config values

General support for ! in git-config values

[Ævar Arnfjörð Bjarmason]

alias.* values can start with !, in which case a shell command will be
executed with the arguments you give.

For a program I'm working on (git-deploy) I'd like to have this as a
general facility, i.e. users can specify either:

    foo.bar = value

Or:

    foo.bar = !cat /some/path

I'm wondering why git-config doesn't do this already, if there's no
reason in particular I can just patch it in, either as a new option:

    git config --with-exec --get foo.bar

Or as the default behavior.

Re: General support for ! in git-config values

[Jeff King]

On Wed, Feb 01, 2012 at 06:33:47PM +0100, Ævar Arnfjörð Bjarmason wrote:

> For a program I'm working on (git-deploy) I'd like to have this as a
> general facility, i.e. users can specify either:
> 
>     foo.bar = value
> 
> Or:
> 
>     foo.bar = !cat /some/path
> 
> I'm wondering why git-config doesn't do this already, if there's no
> reason in particular I can just patch it in, either as a new option:
> 
>     git config --with-exec --get foo.bar

I'm not clear on what you want --with-exec to do. By default, config
values are strings. I would expect the "!" to be a special marker that
the caller would recognize in the string, and then act appropriately.

So if I were implementing git aliases in the shell, the code would look
like:

  v=$(git config alias.$alias)
  case "$v" in
  "")
          die "no such alias: $alias" ;;
  "!*)
          cmd="${v#!}" ;;
  *)
          cmd="git $v" ;;
  esac
  eval "$cmd"

I.e., everything pertaining to "!" happens after we get the config
string. So what is it that you want "git config --with-exec" to do?

-Peff

Re: General support for ! in git-config values

[Ævar Arnfjörð Bjarmason]

On Wed, Feb 1, 2012 at 19:40, Jeff King <peff@peff.net> wrote:
> On Wed, Feb 01, 2012 at 06:33:47PM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> For a program I'm working on (git-deploy) I'd like to have this as a
>> general facility, i.e. users can specify either:
>>
>>     foo.bar = value
>>
>> Or:
>>
>>     foo.bar = !cat /some/path
>>
>> I'm wondering why git-config doesn't do this already, if there's no
>> reason in particular I can just patch it in, either as a new option:
>>
>>     git config --with-exec --get foo.bar
>
> I'm not clear on what you want --with-exec to do. By default, config
> values are strings. I would expect the "!" to be a special marker that
> the caller would recognize in the string, and then act appropriately.
>
> So if I were implementing git aliases in the shell, the code would look
> like:
>
>  v=$(git config alias.$alias)
>  case "$v" in
>  "")
>          die "no such alias: $alias" ;;
>  "!*)
>          cmd="${v#!}" ;;
>  *)
>          cmd="git $v" ;;
>  esac
>  eval "$cmd"
>
> I.e., everything pertaining to "!" happens after we get the config
> string. So what is it that you want "git config --with-exec" to do?

I agree that that's how it should work, I just suggested --with-exec
in case anyone complained about the backwards compatibility issue of
changing the meaning of "!" for existing configs.

Re: General support for ! in git-config values

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>> I.e., everything pertaining to "!" happens after we get the config
>> string. So what is it that you want "git config --with-exec" to do?
>
> I agree that that's how it should work, I just suggested --with-exec
> in case anyone complained about the backwards compatibility issue of
> changing the meaning of "!" for existing configs.

Now you made me utterly confused.

What "backwards compatibility" issue do you have in mind?  If I name
myself '!me' with "user.name = !me", do I suddenly break backwards
compatibility of Git with previous versions somehow? If so how?

The --with-exec option you talk about seems to me the option about the
backward compatibility of the _calling script_ of "git config".  The old
version of git-blorl script may have used foo.bar as a mere string, but a
new version of it may (optionally) interpret foo.bar's value in a special
way when it begins with a "!", introducing a backward compatibility issues
to git-blorl script, and users who want to take advantage of that feature
may want to run it as "git-blorl --with-exec", and the relevant part of
the "git-blorl" script might look like this:

	foo_bar=$(git config foo.bar)
        case "$with_exec,$foo_bar" in
        yes,\!*)
        	foo_bar=$(eval ${foo_bar#\!}) ;;
	esac
        ... then do something interesting using $foo_bar ...

Re: General support for ! in git-config values

[demerphq]

On 1 February 2012 23:21, Junio C Hamano <gitster@pobox.com> wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>>> I.e., everything pertaining to "!" happens after we get the config
>>> string. So what is it that you want "git config --with-exec" to do?
>>
>> I agree that that's how it should work, I just suggested --with-exec
>> in case anyone complained about the backwards compatibility issue of
>> changing the meaning of "!" for existing configs.
>
> Now you made me utterly confused.
>
> What "backwards compatibility" issue do you have in mind?  If I name
> myself '!me' with "user.name = !me", do I suddenly break backwards
> compatibility of Git with previous versions somehow? If so how?

(I am piping up because Ævar and I discussed this idea earlier today
while we discussed git-deploy)

If Ævar's proposal was implemented without a --with-exec switch then

user.name = !me

would set your name to the result of executing the "me" command, which
likely would have the same result as not having set your name at all,
but presumably with a warning about a bad command.

> The --with-exec option you talk about seems to me the option about the
> backward compatibility of the _calling script_ of "git config".

By backward compatibility he meant that anything not providing the
--with-exec option to git-config would continue to behave as before,
and anything that did would get the new more powerful behavior, where
! at the front of a value signified that it meant a command to be
executed whose output would be used as the value.

I think it would be convenient to be able to enable this in the config
file itself. Consider a bunch of devs share a box. Their full name and
email address can be located via a LDAP lookup. So if we could enable
Ævar's proposal in the config then we can automagically have their
commits have the correct username and email if we were to put in the
/etc/gitconfig something like this:

config.enable_executable_options=true
user.name=!ldap_full_name
user.email=!ldap_email

So now, user.name and user.email don't have to be configured by the
users. They just get an account on the box and all their commits have
the right details, no fuss.

I am aware that there are other ways of solving /this/ particular
problem, so it specifically is not a big deal, but still I think the
idea makes a lot of sense, I can imagine it being used in a lot of
interesting ways that are not obvious right away.

cheers,
Yves



-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[Junio C Hamano]

demerphq <demerphq@gmail.com> writes:

> If Ævar's proposal was implemented without a --with-exec switch then
>
> user.name = !me
>
> would set your name to the result of executing the "me" command, which
> likely would have the same result as not having set your name at all,
> but presumably with a warning about a bad command.
>
> user.name=!ldap_full_name
> user.email=!ldap_email

I didn't get the impression that that was what he was proposing, but
regardless of who's suggesting it, the above does not make any sense,
from the syntax point of view.

When you say "alias.co = !git checkout --foo-bar", you are not saying
"instead of usual aliases like 'alias.co = checkout' that defines what
string replaces the 'co' alias, please run !whatever command, capture
its output, and that is the _value_ of alias.co variable, i.e. the command
line that is going to be run".

Re: General support for ! in git-config values

[demerphq]

On 2 February 2012 01:27, Junio C Hamano <gitster@pobox.com> wrote:
> demerphq <demerphq@gmail.com> writes:
>
>> If Ævar's proposal was implemented without a --with-exec switch then
>>
>> user.name = !me
>>
>> would set your name to the result of executing the "me" command, which
>> likely would have the same result as not having set your name at all,
>> but presumably with a warning about a bad command.
>>
>> user.name=!ldap_full_name
>> user.email=!ldap_email
>
> I didn't get the impression that that was what he was proposing, but
> regardless of who's suggesting it, the above does not make any sense,
> from the syntax point of view.

Does that mean that from a utility point of view it does?

As for the suggestion and who is making it, this came up because Ævar
and I were discussing git-deploy and a scenario much like I described,
where we use tags to mark out rollouts and we have multiple users
using a common staging server and git repo to do the rollouts from, we
want the tags to have proper details, so we have to go through some
hoops to ensure that users have properly configured things before the
do their first rollout. Since Ævar was aware of the "!" notation for
aliases and thought it made some kind of sense to reuse the syntax
elsewhere he decided to see what people thought. I was not aware of
the notation until he pointed it out. I don't use git-alias.

> When you say "alias.co = !git checkout --foo-bar", you are not saying
> "instead of usual aliases like 'alias.co = checkout' that defines what
> string replaces the 'co' alias, please run !whatever command, capture
> its output, and that is the _value_ of alias.co variable, i.e. the command
> line that is going to be run".

I think I see your point. Using the same "!" notation for "execute
this command and exit" and "execute this command and use its result as
the config value" might be confusing.

Do you have a suggestion for a better syntax? Backtick quoting perhaps?

cheers,
yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[Junio C Hamano]

demerphq <demerphq@gmail.com> writes:

> On 2 February 2012 01:27, Junio C Hamano <gitster@pobox.com> wrote:
>> demerphq <demerphq@gmail.com> writes:
>>
>>> user.name=!ldap_full_name
>>> user.email=!ldap_email
>>
>> I didn't get the impression that that was what he was proposing, but
>> regardless of who's suggesting it, the above does not make any sense,
>> from the syntax point of view.
>
> Does that mean that from a utility point of view it does?

Not really.

I do not think whatever "utility" value outweighs the hassle of having to
think through the ramifications (including but not limited to security) of
running arbitrary user command every time a value is looked up.

Re: General support for ! in git-config values

[Junio C Hamano]

demerphq <demerphq@gmail.com> writes:

>> When you say "alias.co = !git checkout --foo-bar", you are not saying
>> "instead of usual aliases like 'alias.co = checkout' that defines what
>> string replaces the 'co' alias, please run !whatever command, capture
>> its output, and that is the _value_ of alias.co variable, i.e. the command
>> line that is going to be run".
>
> I think I see your point. Using the same "!" notation for "execute
> this command and exit" and "execute this command and use its result as
> the config value" might be confusing.

I am afraid that you are not seeing it.  In "alias.co = !whatever", bang
does *NOT* mean "execute this command and exit".

Re: General support for ! in git-config values

[demerphq]

On 2 February 2012 02:39, Junio C Hamano <gitster@pobox.com> wrote:
> demerphq <demerphq@gmail.com> writes:
>
>> On 2 February 2012 01:27, Junio C Hamano <gitster@pobox.com> wrote:
>>> demerphq <demerphq@gmail.com> writes:
>>>
>>>> user.name=!ldap_full_name
>>>> user.email=!ldap_email
>>>
>>> I didn't get the impression that that was what he was proposing, but
>>> regardless of who's suggesting it, the above does not make any sense,
>>> from the syntax point of view.
>>
>> Does that mean that from a utility point of view it does?
>
> Not really.
>
> I do not think whatever "utility" value outweighs the hassle of having to
> think through the ramifications (including but not limited to security) of
> running arbitrary user command every time a value is looked up.

Why is that your problem? If I have to enable it then isn't that my choice?

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[demerphq]

On 2 February 2012 02:41, Junio C Hamano <gitster@pobox.com> wrote:
> demerphq <demerphq@gmail.com> writes:
>
>>> When you say "alias.co = !git checkout --foo-bar", you are not saying
>>> "instead of usual aliases like 'alias.co = checkout' that defines what
>>> string replaces the 'co' alias, please run !whatever command, capture
>>> its output, and that is the _value_ of alias.co variable, i.e. the command
>>> line that is going to be run".
>>
>> I think I see your point. Using the same "!" notation for "execute
>> this command and exit" and "execute this command and use its result as
>> the config value" might be confusing.
>
> I am afraid that you are not seeing it.  In "alias.co = !whatever", bang
> does *NOT* mean "execute this command and exit".

Then I have been unclear. I understand what this does:

$ git config alias.foo '!echo foo'
$ git foo
foo

And if you were trying to say that having "!" denote "execute this
command and use its return as the config value" would be confusing as
it is different from its meaning for aliases then I see your point.

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[Jeff King]

On Thu, Feb 02, 2012 at 02:57:14AM +0100, demerphq wrote:

> > Not really.
> >
> > I do not think whatever "utility" value outweighs the hassle of having to
> > think through the ramifications (including but not limited to security) of
> > running arbitrary user command every time a value is looked up.
> 
> Why is that your problem? If I have to enable it then isn't that my choice?

>From a security perspective, you want to make sure that people who
aren't interested in your feature don't accidentally trigger it. E.g.,
imagine I currently run a locked-down git repo but execute some commands
on your behalf, and I allow you to set a few "known safe" config options
like user.email. Even though I am not interested in your feature,
respecting "!rm -rf /" in the user.email you give me would be a bad
thing.

It's not an insurmountable problem. There could be options to turn it
on, or turn it off, or whatever. Or we could shrug and say that config
is already dangerous to let other people set (which it is already, but
only for some options). But those are the sorts of ramifications that
need to be thought through.

(Another one is that with our current strategy, we actually read and
parse the config files multiple times. Should your program get run many
times?).

-Peff

Re: General support for ! in git-config values

[demerphq]

On 2 February 2012 03:38, Jeff King <peff@peff.net> wrote:
> On Thu, Feb 02, 2012 at 02:57:14AM +0100, demerphq wrote:
>
>> > Not really.
>> >
>> > I do not think whatever "utility" value outweighs the hassle of having to
>> > think through the ramifications (including but not limited to security) of
>> > running arbitrary user command every time a value is looked up.
>>
>> Why is that your problem? If I have to enable it then isn't that my choice?
>
> From a security perspective, you want to make sure that people who
> aren't interested in your feature don't accidentally trigger it. E.g.,
> imagine I currently run a locked-down git repo but execute some commands
> on your behalf, and I allow you to set a few "known safe" config options
> like user.email. Even though I am not interested in your feature,
> respecting "!rm -rf /" in the user.email you give me would be a bad
> thing.

Like I said, I do not think this should be enabled by default, I think
it should be possible to enable it config wide. So unless this
scenario involves getting the owner of the locked down repo to enable
a config option they know nothing about, in which case I would say
there are easier attacks -- someone that stupid probably could be
talked into telling you their root password. :-)

> It's not an insurmountable problem. There could be options to turn it
> on, or turn it off, or whatever.

My thought exactly. Anyone paranoid about security would never enable
this feature. Those who are comfortable with the security issues
could.

> Or we could shrug and say that config
> is already dangerous to let other people set (which it is already, but
> only for some options).

I think that since it could be set up to be determined by the user,
that it would be no more dangerous than any other option.

> But those are the sorts of ramifications that
> need to be thought through.

I understand that. All I can say is $work uses git on a pretty large
scale, 100+ devs etc, and we use it to manage our rollout processes
which we use a lot (I cant say how often but a lot). So if it would be
useful to us it probably would be useful to others.

> (Another one is that with our current strategy, we actually read and
> parse the config files multiple times. Should your program get run many
> times?).

Again I would say this is not git's problem. If it should not be run
multiple times it is up to the user to figure out an alternative.

The general design of git seems to me to be based around providing
building blocks that people can use to build new and interesting tools
on top of, and so it seems counter to that philosophy to reject an
feature based on speculative security issues that really can't be
decided in advance but must instead be decided on a case by case
basis.

cheers,
Yves





-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[Jeff King]

On Thu, Feb 02, 2012 at 10:44:05AM +0100, demerphq wrote:

> The general design of git seems to me to be based around providing
> building blocks that people can use to build new and interesting tools
> on top of, and so it seems counter to that philosophy to reject an
> feature based on speculative security issues that really can't be
> decided in advance but must instead be decided on a case by case
> basis.

I can't speak for Junio, but I am certainly not rejecting it. Only
saying that it needs to be thought through, and the utility weighed
against the costs. So far I haven't seen an actual patch to comment on
(or even a proposed syntax beyond starting a string with "!", which I
think is a non-starter due to conflicting with existing uses), nor have
I seen a concrete use case (you mentioned pulling the name/email from
ldap, but you also mentioned that there are lots of other ways of
solving that particular problem, so it's not especially compelling).

I'd be happy to hear a more concrete proposal.

-Peff

Re: General support for ! in git-config values

[demerphq]

On 2 February 2012 10:54, Jeff King <peff@peff.net> wrote:
> On Thu, Feb 02, 2012 at 10:44:05AM +0100, demerphq wrote:
>
>> The general design of git seems to me to be based around providing
>> building blocks that people can use to build new and interesting tools
>> on top of, and so it seems counter to that philosophy to reject an
>> feature based on speculative security issues that really can't be
>> decided in advance but must instead be decided on a case by case
>> basis.
>
> I can't speak for Junio, but I am certainly not rejecting it. Only
> saying that it needs to be thought through, and the utility weighed
> against the costs.

Of course. I totally understand. I have written mails saying stuff
like this myself. :-)

> So far I haven't seen an actual patch to comment on
> (or even a proposed syntax beyond starting a string with "!", which I
> think is a non-starter due to conflicting with existing uses),

I understand. I think we will probably use backtick quoting in git-deploy. So

deploy.prefix=`cat /etc/SERVER_ROLE`

will execute cat /etc/SERVER_ROLE and use the results as the value of
the config option.

> nor have
> I seen a concrete use case (you mentioned pulling the name/email from
> ldap, but you also mentioned that there are lots of other ways of
> solving that particular problem, so it's not especially compelling).

One place that it would be useful for us in git-deploy would be to
detect the tag prefix for the rollout we are doing. Every staging
server already has a file that contains this value. We would like to
make it easy for people to configure the tool to either use the value
provided, or to use something like  `cat /etc/SERVER_ROLE` instead.
Anyway, from that POV I could totally understand "so do that in
git-deploy". Since the tool is written in perl we have to wrap
git-config anyway, so it easy to add a special case for ourselves.

But I still think the general idea is pretty useful, the ldap example
is IMO a cleaner solution than the alternatives, and a variant that I
think is much harder to do currently come to mind right away: setting
the user.email automatically depending on where in your tree a git
repo was located, so that when I work on repo underneath  /CPAN/ it
uses my CPAN address, and when I work in my /work/ tree it uses my
$work address, etc, without me having to configure it repo by repo.
(This has bitten more than once in the past)

> I'd be happy to hear a more concrete proposal.

I will be mostly afk the next week so I will leave that to Avar if he
wants to pursue it.

cheers,
Yves


-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: General support for ! in git-config values

[Kyle Moffett]

On Thu, Feb 2, 2012 at 02:21, demerphq <demerphq@gmail.com> wrote:
>> So far I haven't seen an actual patch to comment on
>> (or even a proposed syntax beyond starting a string with "!", which I
>> think is a non-starter due to conflicting with existing uses),
>
> I understand. I think we will probably use backtick quoting in git-deploy. So
>
> deploy.prefix=`cat /etc/SERVER_ROLE`
>
> will execute cat /etc/SERVER_ROLE and use the results as the value of
> the config option.

Alternatively, you could extend the recent proposal for GIT config
"include" statements so that something like this works:

[include]
    exec = echo "deploy.prefix = `cat /etc/SERVER_ROLE`"
    exec = /usr/local/bin/git-config-for-ldap-user

Thoughts?

Cheers,
Kyle Moffett

Re: General support for ! in git-config values

[Junio C Hamano]

Kyle Moffett <kyle@moffetthome.net> writes:

> Alternatively, you could extend the recent proposal for GIT config
> "include" statements so that something like this works:
>
> [include]
>     exec = echo "deploy.prefix = `cat /etc/SERVER_ROLE`"
>     exec = /usr/local/bin/git-config-for-ldap-user

Erh...

Running known stuff from your own .git/config may be justifiable as "at
your own risk", but if we consider sources that are not under your direct
control, such as /etc/gitconfig and whatever your project encourages you
to include from your .git/config,... eek.

Re: General support for ! in git-config values

[Kyle Moffett]

On Thu, Feb 2, 2012 at 22:11, Junio C Hamano <gitster@pobox.com> wrote:
> Kyle Moffett <kyle@moffetthome.net> writes:
>
>> Alternatively, you could extend the recent proposal for GIT config
>> "include" statements so that something like this works:
>>
>> [include]
>>     exec = echo "deploy.prefix = `cat /etc/SERVER_ROLE`"
>>     exec = /usr/local/bin/git-config-for-ldap-user
>
> Erh...
>
> Running known stuff from your own .git/config may be justifiable as "at
> your own risk", but if we consider sources that are not under your direct
> control, such as /etc/gitconfig and whatever your project encourages you
> to include from your .git/config,... eek.

Well yes, but running commands from .git/config is exactly what the OP
requested, and if it applies to .git/config it should also be
applicable to other trusted include sources too, no?

Perhaps allow config files to perform a "trusted" include, EG:
[include]
    trusted_exec = /usr/local/bin/site-specific-config-program
    blob = v1.0:src/gitconfig

By default, the only files which would be trusted are /etc/gitconfig,
~/.gitconfig, and .git/config (but ONLY if it has the same owner and
mode go-w), and they would only pass trust on to other files if they
use "trusted_*" include lines.

Also, since "include" is intended to introduce a
non-backwards-compatible change in behavior, perhaps a totally
different format should be used, EG:

$include exec_trusted /usr/local/bin/site-specific-config-program
$include blob v1.0:src/gitconfig

Something that would cause noticeable warnings in older versions of
git instead of silently ignoring the desired config includes.

Just a few thoughts.

Cheers,
Kyle Moffett

Re: General support for ! in git-config values

[Jeff King]

On Thu, Feb 02, 2012 at 09:08:46PM -0800, Kyle Moffett wrote:

> > I understand. I think we will probably use backtick quoting in git-deploy. So
> >
> > deploy.prefix=`cat /etc/SERVER_ROLE`
> >
> > will execute cat /etc/SERVER_ROLE and use the results as the value of
> > the config option.
> 
> Alternatively, you could extend the recent proposal for GIT config
> "include" statements so that something like this works:
> 
> [include]
>     exec = echo "deploy.prefix = `cat /etc/SERVER_ROLE`"
>     exec = /usr/local/bin/git-config-for-ldap-user
> 
> Thoughts?

I am still undecided on whether the utility of the idea is worth the
potential hassle, but syntactically I like that better. It does put a
little more burden on the called program to handle things like quoting,
though.

-Peff

Re: General support for ! in git-config values

[Jeff King]

On Thu, Feb 02, 2012 at 10:11:27PM -0800, Junio C Hamano wrote:

> Kyle Moffett <kyle@moffetthome.net> writes:
> 
> > Alternatively, you could extend the recent proposal for GIT config
> > "include" statements so that something like this works:
> >
> > [include]
> >     exec = echo "deploy.prefix = `cat /etc/SERVER_ROLE`"
> >     exec = /usr/local/bin/git-config-for-ldap-user
> 
> Erh...
> 
> Running known stuff from your own .git/config may be justifiable as "at
> your own risk", but if we consider sources that are not under your direct
> control, such as /etc/gitconfig and whatever your project encourages you
> to include from your .git/config,... eek.

For normal use, I don't see this as a big deal. They could also be
specifying diff.external, which would run arbitrary code (and who
doesn't run "git diff" once in a while?).

I see it as a bigger issue for sites which serve repositories on behalf
of their users, and already take care never to use porcelain commands
which will run arbitrary code from the config by default (e.g., gitweb
carefully uses diff plumbing for this reason). Introducing such an
option provides a mechanism for users who control the config of the
served repositories to execute code as the user running git-daemon or
gitweb.

-Peff