Can git choose perl at runtime?

Can git choose perl at runtime?

[John Passaro]

I recently submitted my first patch using OSX and found the experience
frustrating, for reasons that have come up on the list before,
concerning git-send-email and perl dependencies that you need to be
root to update.

Last seen here:
https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/

The struggle is that Mac's package manager Homebrew has opted,
apparently with some finality, to no longer support linking to a user
perl at build time. PERL_PATH is hard-coded to link to the system
perl, which means the user needs sudo to install the SSL libraries
required for send-email. So for send-email to work, you need to either
sudo cpan or build git yourself. The obvious solution here would be to
do /usr/bin/env perl, but in the above message Aevar pointed out
pitfalls with that.

It seems that choosing perl at compile time necessarily comes with
tradeoffs. So I wonder if there is a way we can support choosing a
perl at runtime without breaking the existing mechanism of linking to
perl at compile time.

I'm picturing adding an executable "git-perl" to libexec that checks
config core.perlPath and envvar GIT_PERL_PATH, in some order. Having
chosen one of these or the build-time PERL_PATH as a last resort, it
exec's the correct perl executable.

Then relevant scripts (e.g. git-add--interactive, git-send-email)
invoke git-perl instead of /usr/bin/perl, and the makefile no longer
replaces that with PERL_PATH -- instead that will be used at runtime
via git-perl when we can be sure the user does not explicitly prefer
something different.

That does mean we have a new command to support and document: "git
perl". If it is preferred to keep this hidden as an implementation
detail, we could call the executable something like "util-git-perl"
instead so that it doesn't show up when scanning libexec for git
commands.

Does this seem like a good idea? I'd be happy to work on a patch.

Re: Can git choose perl at runtime?

[Jonathan Nieder]

Hi John,

John Passaro wrote:

> https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/
>
> The struggle is that Mac's package manager Homebrew has opted,
> apparently with some finality, to no longer support linking to a user
> perl at build time. PERL_PATH is hard-coded to link to the system
> perl, which means the user needs sudo to install the SSL libraries
> required for send-email. So for send-email to work, you need to either
> sudo cpan or build git yourself. The obvious solution here would be to
> do /usr/bin/env perl, but in the above message Aevar pointed out
> pitfalls with that.
>
> It seems that choosing perl at compile time necessarily comes with
> tradeoffs. So I wonder if there is a way we can support choosing a
> perl at runtime without breaking the existing mechanism of linking to
> perl at compile time.

I haven't carefully looked at your exact proposal, but I just wanted
to offer you my support: yes, I would love to see some solution.
Thanks for looking into it.

It would let me remove this bit of horror from my local build script:

 APIVER_EXPR='@{[sub{use Config; $$Config{api_version}}->()]}'
 XCODE_PERL="/Applications/Xcode.app/Contents/Developer/Library/Perl/5.$APIVER_EXPR/darwin-thread-multi-2level"
 make ... PERLLIB_EXTRA="$XCODE_PERL"

(My apologies.)

[...]
> That does mean we have a new command to support and document: "git
> perl". If it is preferred to keep this hidden as an implementation
> detail, we could call the executable something like "util-git-perl"
> instead so that it doesn't show up when scanning libexec for git
> commands.

Typically we handle this kind of thing by putting a double-dash in
the command name.  See git-sh--setup, for example.

Thanks and hope that helps,
Jonathan

Re: Can git choose perl at runtime?

[Carlo Arenas]

On Tue, Dec 18, 2018 at 7:29 PM John Passaro <john.a.passaro@gmail.com> wrote:
>
> I recently submitted my first patch using OSX and found the experience
> frustrating, for reasons that have come up on the list before,
> concerning git-send-email and perl dependencies that you need to be
> root to update.

you can install them somewhere else (your homedir, for example) and
then instruct perl to look for them there by setting the PERL5LIB
environment variable

Carlo

Re: Can git choose perl at runtime?

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 19 2018, Carlo Arenas wrote:

> On Tue, Dec 18, 2018 at 7:29 PM John Passaro <john.a.passaro@gmail.com> wrote:
>>
>> I recently submitted my first patch using OSX and found the experience
>> frustrating, for reasons that have come up on the list before,
>> concerning git-send-email and perl dependencies that you need to be
>> root to update.
>
> you can install them somewhere else (your homedir, for example) and
> then instruct perl to look for them there by setting the PERL5LIB
> environment variable

Note that this is different. Cases I can think of:

 1. You have an entirely different perl + modules somewhere. E.g. maybe
    on OSX /usr/bin/perl v.s. some homebrew version of perl+CPAN. My WIP
    https://public-inbox.org/git/87a7l1fx8x.fsf@evledraar.gmail.com/
    addresses this.

 2. You're happy with /usr/bin/perl (or whatever git is compiled with),
    but miss some module(s). That's your suggestion here, but note that
    in this case you usually need a compiler (E-Mail SSL libs etc.),
    which may not be what the user wants.

    I.e. they can install a new perl+modules from their package manager
    easily, but can't as easily build their own modules for a system
    perl.

 3. Using a /usr/bin/perl + e.g. homebrew CPAN libs via a "modules over
    here" facility similar to #2 is likely to segfault (different ABI
    versions).

I think we're good if we just have #1 and if people have the #2 use-case
an additional core.perlLibs config of stuff to prepend to @INC (or maybe
entirly override, least we run into the segfault case in #3).

For that last bit see also 7a7bfc7adc ("perl: treat PERLLIB_EXTRA as an
extra path again", 2018-01-02). I.e. there's the use case of "@INC
instead of" and "@INC extra".

But probably you're happy with just #1 for now....

Re: Can git choose perl at runtime?

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 19 2018, John Passaro wrote:

> I recently submitted my first patch using OSX and found the experience
> frustrating, for reasons that have come up on the list before,
> concerning git-send-email and perl dependencies that you need to be
> root to update.
>
> Last seen here:
> https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/
>
> The struggle is that Mac's package manager Homebrew has opted,
> apparently with some finality, to no longer support linking to a user
> perl at build time. PERL_PATH is hard-coded to link to the system
> perl, which means the user needs sudo to install the SSL libraries
> required for send-email. So for send-email to work, you need to either
> sudo cpan or build git yourself. The obvious solution here would be to
> do /usr/bin/env perl, but in the above message Aevar pointed out
> pitfalls with that.
>
> It seems that choosing perl at compile time necessarily comes with
> tradeoffs. So I wonder if there is a way we can support choosing a
> perl at runtime without breaking the existing mechanism of linking to
> perl at compile time.
>
> I'm picturing adding an executable "git-perl" to libexec that checks
> config core.perlPath and envvar GIT_PERL_PATH, in some order. Having
> chosen one of these or the build-time PERL_PATH as a last resort, it
> exec's the correct perl executable.
>
> Then relevant scripts (e.g. git-add--interactive, git-send-email)
> invoke git-perl instead of /usr/bin/perl, and the makefile no longer
> replaces that with PERL_PATH -- instead that will be used at runtime
> via git-perl when we can be sure the user does not explicitly prefer
> something different.
>
> That does mean we have a new command to support and document: "git
> perl". If it is preferred to keep this hidden as an implementation
> detail, we could call the executable something like "util-git-perl"
> instead so that it doesn't show up when scanning libexec for git
> commands.
>
> Does this seem like a good idea? I'd be happy to work on a patch.

I see no problem with this. As I noted in my message you linked to doing
this unconditionally is a bad idea, but we can just do it with a config,
e.g. this works:

    diff --git a/perl/header_templates/fixed_prefix.template.pl b/perl/header_templates/fixed_prefix.template.pl
    index 857b4391a4..f96e2ecd11 100644
    --- a/perl/header_templates/fixed_prefix.template.pl
    +++ b/perl/header_templates/fixed_prefix.template.pl
    @@ -1 +1,7 @@
    +BEGIN {
    +    chomp(my $perlPath = `git config --get core.perlPath`);;
    +    if ($perlPath and $^X ne $perlPath) {
    +       exec($perlPath, $0, @ARGV);
    +    }
    +}
     use lib (split(/@@PATHSEP@@/, $ENV{GITPERLLIB} || '@@INSTLIBDIR@@'));

Here you just optionally set core.perlPath in your config and if set
it'll chainload to the new interpreter you point at.

I leave wondering if you also want a setting for @INC there, dealing
with perl/header_templates/runtime_prefix.template.pl and docs/tests as
an exercise for the reader :)

Re: Can git choose perl at runtime?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On Tue, Dec 18, 2018 at 10:09:14PM -0500, John Passaro wrote:
> I recently submitted my first patch using OSX and found the experience
> frustrating, for reasons that have come up on the list before,
> concerning git-send-email and perl dependencies that you need to be
> root to update.
> 
> Last seen here:
> https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/
> 
> The struggle is that Mac's package manager Homebrew has opted,
> apparently with some finality, to no longer support linking to a user
> perl at build time. PERL_PATH is hard-coded to link to the system
> perl, which means the user needs sudo to install the SSL libraries
> required for send-email. So for send-email to work, you need to either
> sudo cpan or build git yourself. The obvious solution here would be to
> do /usr/bin/env perl, but in the above message Aevar pointed out
> pitfalls with that.
> 
> It seems that choosing perl at compile time necessarily comes with
> tradeoffs. So I wonder if there is a way we can support choosing a
> perl at runtime without breaking the existing mechanism of linking to
> perl at compile time.
> 
> I'm picturing adding an executable "git-perl" to libexec that checks
> config core.perlPath and envvar GIT_PERL_PATH, in some order. Having
> chosen one of these or the build-time PERL_PATH as a last resort, it
> exec's the correct perl executable.
> 
> Then relevant scripts (e.g. git-add--interactive, git-send-email)
> invoke git-perl instead of /usr/bin/perl, and the makefile no longer
> replaces that with PERL_PATH -- instead that will be used at runtime
> via git-perl when we can be sure the user does not explicitly prefer
> something different.

How do git send-email and git svn work in such a case? They depend on
the Git and Git::SVN modules being in place, so if you use a Perl other
than the one you built Git with, they won't be present (or they'll be
present, but potentially with the wrong version).
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Can git choose perl at runtime?

[Ævar Arnfjörð Bjarmason]

On Fri, Dec 21 2018, brian m. carlson wrote:

> On Tue, Dec 18, 2018 at 10:09:14PM -0500, John Passaro wrote:
>> I recently submitted my first patch using OSX and found the experience
>> frustrating, for reasons that have come up on the list before,
>> concerning git-send-email and perl dependencies that you need to be
>> root to update.
>>
>> Last seen here:
>> https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/
>>
>> The struggle is that Mac's package manager Homebrew has opted,
>> apparently with some finality, to no longer support linking to a user
>> perl at build time. PERL_PATH is hard-coded to link to the system
>> perl, which means the user needs sudo to install the SSL libraries
>> required for send-email. So for send-email to work, you need to either
>> sudo cpan or build git yourself. The obvious solution here would be to
>> do /usr/bin/env perl, but in the above message Aevar pointed out
>> pitfalls with that.
>>
>> It seems that choosing perl at compile time necessarily comes with
>> tradeoffs. So I wonder if there is a way we can support choosing a
>> perl at runtime without breaking the existing mechanism of linking to
>> perl at compile time.
>>
>> I'm picturing adding an executable "git-perl" to libexec that checks
>> config core.perlPath and envvar GIT_PERL_PATH, in some order. Having
>> chosen one of these or the build-time PERL_PATH as a last resort, it
>> exec's the correct perl executable.
>>
>> Then relevant scripts (e.g. git-add--interactive, git-send-email)
>> invoke git-perl instead of /usr/bin/perl, and the makefile no longer
>> replaces that with PERL_PATH -- instead that will be used at runtime
>> via git-perl when we can be sure the user does not explicitly prefer
>> something different.
>
> How do git send-email and git svn work in such a case? They depend on
> the Git and Git::SVN modules being in place, so if you use a Perl other
> than the one you built Git with, they won't be present (or they'll be
> present, but potentially with the wrong version).

Yeah this is one of the things I was alluding to in
<87a7l1fx8x.fsf@evledraar.gmail.com>.

We don't ship any C bindings, so our libs end up in
e.g. /usr/share/perl5, some custom-built perls will have that in their
@INC still, no idea if any of this OSX stuff does.

But otherwise we'd either need to give the user a way to override
PERL5LIB (or they can do it themselves...), or better yet continue what
I started in 20d2a30f8f ("Makefile: replace perl/Makefile.PL with simple
make rules", 2017-12-10) and make our perl stuff entirely decoupled from
the system install.

E.g. Linux distros would probably still override that and install our
*.pm stuff in their usual Perl places, but by default we could just have
a libexec/perl directory with all this stuff, and find our libraries
there, then it won't matter if we chainload to a new Perl interpreter,
we'll still find the libs in the same place.

We could also turn RUNTIME_PREFIX on by default, it already fixes this
by proxy.

Re: Can git choose perl at runtime?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3349 bytes --]

On Sun, Dec 23, 2018 at 11:05:46PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Fri, Dec 21 2018, brian m. carlson wrote:
> > How do git send-email and git svn work in such a case? They depend on
> > the Git and Git::SVN modules being in place, so if you use a Perl other
> > than the one you built Git with, they won't be present (or they'll be
> > present, but potentially with the wrong version).
> 
> Yeah this is one of the things I was alluding to in
> <87a7l1fx8x.fsf@evledraar.gmail.com>.
> 
> We don't ship any C bindings, so our libs end up in
> e.g. /usr/share/perl5, some custom-built perls will have that in their
> @INC still, no idea if any of this OSX stuff does.
> 
> But otherwise we'd either need to give the user a way to override
> PERL5LIB (or they can do it themselves...), or better yet continue what
> I started in 20d2a30f8f ("Makefile: replace perl/Makefile.PL with simple
> make rules", 2017-12-10) and make our perl stuff entirely decoupled from
> the system install.
> 
> E.g. Linux distros would probably still override that and install our
> *.pm stuff in their usual Perl places, but by default we could just have
> a libexec/perl directory with all this stuff, and find our libraries
> there, then it won't matter if we chainload to a new Perl interpreter,
> we'll still find the libs in the same place.

This wouldn't fix the fact that we still need modules like Net::SMTP,
Authen::SASL, and IO::Socket::SSL (because these days every provider
forces TLS on the submission port). Since those are going to come from
the distributor, letting people override the Perl path to some arbitrary
path will mean that those modules may not be installed.

I also think that the situation you want with relocatable modules is
only going to be useful for people who custom-install their own Git,
which is not most people. Nobody shipping a packaged version of Git is
going to install modules in a custom Git-specific path (since they can't
be loaded by other software), so everyone who want to use a custom Perl
will already be compiling a custom Git and can just specify the Perl
they want to use.

My concern is, more generally, that this situation is going to lead to
hard-to-troubleshoot user support issues. I routinely answer questions
on Stack Overflow and I see all sorts of cases where users who may be
great programmers but are mostly unfamiliar with Git end up in bad
situations.

For example, at a previous job, we shipped a newer Git and Perl, which
were installed in a custom path (so definitely not using relocatable
modules). If this option were enabled and user used the newer Git, which
was installed in a custom path, but the system Perl, things would
definitely be broken, since the system Perl would almost certainly have
none of the right modules (or, if it did, they'd be grossly out of
date). A lot of the users who would run into this issue are less
technical, and so wouldn't know how to fix it.

We've traditionally shied away from specifying things like
"#!/usr/bin/env perl" specifically for this reason: because people will
often have custom-compiled versions of interpreters that don't meet our
needs. I'm not seeing how this is significantly different.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Can git choose perl at runtime?

[Ævar Arnfjörð Bjarmason]

On Sun, Dec 23 2018, brian m. carlson wrote:

> On Sun, Dec 23, 2018 at 11:05:46PM +0100, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Fri, Dec 21 2018, brian m. carlson wrote:
>> > How do git send-email and git svn work in such a case? They depend on
>> > the Git and Git::SVN modules being in place, so if you use a Perl other
>> > than the one you built Git with, they won't be present (or they'll be
>> > present, but potentially with the wrong version).
>>
>> Yeah this is one of the things I was alluding to in
>> <87a7l1fx8x.fsf@evledraar.gmail.com>.
>>
>> We don't ship any C bindings, so our libs end up in
>> e.g. /usr/share/perl5, some custom-built perls will have that in their
>> @INC still, no idea if any of this OSX stuff does.
>>
>> But otherwise we'd either need to give the user a way to override
>> PERL5LIB (or they can do it themselves...), or better yet continue what
>> I started in 20d2a30f8f ("Makefile: replace perl/Makefile.PL with simple
>> make rules", 2017-12-10) and make our perl stuff entirely decoupled from
>> the system install.
>>
>> E.g. Linux distros would probably still override that and install our
>> *.pm stuff in their usual Perl places, but by default we could just have
>> a libexec/perl directory with all this stuff, and find our libraries
>> there, then it won't matter if we chainload to a new Perl interpreter,
>> we'll still find the libs in the same place.
>
> This wouldn't fix the fact that we still need modules like Net::SMTP,
> Authen::SASL, and IO::Socket::SSL (because these days every provider
> forces TLS on the submission port). Since those are going to come from
> the distributor, letting people override the Perl path to some arbitrary
> path will mean that those modules may not be installed.

Yeah, but my reading (which may be wrong) of John Passaro's E-Mail
upthread
(<CAJdN7Kioa22xrDP2ssZXmBbu7KDkcr2MQCUDW=Tzm5ydzeChBQ@mail.gmail.com>)
is that for some users this is the path of least resistance to getting
git-send-email et al working for whatever reason.

I.e. they'd like to install a git version that's compiled against
/usr/bin/perl, and then after the fact ask git to point to some other
better working perl installation with all of the above compiled.

> I also think that the situation you want with relocatable modules is
> only going to be useful for people who custom-install their own Git,
> which is not most people. Nobody shipping a packaged version of Git is
> going to install modules in a custom Git-specific path (since they can't
> be loaded by other software), so everyone who want to use a custom Perl
> will already be compiling a custom Git and can just specify the Perl
> they want to use.

I mean that we could just make RUNTIME_PREFIX our default behavior, it
would simplify things by only carrying one "how do we find stuff" mode,
and in this case nicely solve this whole problem of your custom perl not
having git's perl modules in its @INC (but having everything else,
e.g. Authen::SASL available).

> My concern is, more generally, that this situation is going to lead to
> hard-to-troubleshoot user support issues. I routinely answer questions
> on Stack Overflow and I see all sorts of cases where users who may be
> great programmers but are mostly unfamiliar with Git end up in bad
> situations.
>
> For example, at a previous job, we shipped a newer Git and Perl, which
> were installed in a custom path (so definitely not using relocatable
> modules). If this option were enabled and user used the newer Git, which
> was installed in a custom path, but the system Perl, things would
> definitely be broken, since the system Perl would almost certainly have
> none of the right modules (or, if it did, they'd be grossly out of
> date). A lot of the users who would run into this issue are less
> technical, and so wouldn't know how to fix it.

No comment on this other than: "whoever's itch this actually is and who
picks up my POC patch will need to address this to brian's satisfaction"
:)

> We've traditionally shied away from specifying things like
> "#!/usr/bin/env perl" specifically for this reason: because people will
> often have custom-compiled versions of interpreters that don't meet our
> needs. I'm not seeing how this is significantly different.

Because "#!/usr/bin/env perl" would break git if you were just playing
around with a custom perl for some other reason, e.g. perl web
development. I agree that this wouldn't be acceptable (as see in my
https://public-inbox.org/git/878t55qga6.fsf@evledraar.gmail.com/ that
John linked to).

Whereas what's being proposed here is some way to specifically tell git
via configuration that it should use a runtime configured perl instead
of the compile-time one.

Re: Can git choose perl at runtime?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

On Mon, Dec 24, 2018 at 12:34:12AM +0100, Ævar Arnfjörð Bjarmason wrote:
> Yeah, but my reading (which may be wrong) of John Passaro's E-Mail
> upthread
> (<CAJdN7Kioa22xrDP2ssZXmBbu7KDkcr2MQCUDW=Tzm5ydzeChBQ@mail.gmail.com>)
> is that for some users this is the path of least resistance to getting
> git-send-email et al working for whatever reason.

I think we should just ask Homebrew to ship a functional, complete Git.
If they need to use the Homebrew Perl to install modules instead of the
system Perl, then they should do that. It sounds like we're engineering
a feature that lets users shoot themselves in the foot to work around an
easily fixable distributor issue.

I'm sympathetic to the difficult job of distributors, but Git is not an
unreasonable upstream, and I don't think it's unfair to ask Homebrew to
solve this problem in their distribution.

I've commented on the issue on their GitHub repository, asking them to
address this problem.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Can git choose perl at runtime?

[John Passaro]

On Sun, Dec 23, 2018 at 9:20 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> I think we should just ask Homebrew to ship a functional, complete Git.

They are doing just that:
https://github.com/Homebrew/homebrew-core/pull/35446
If for some reason this patch doesn't make it in, I'll keep bugging
them about it, but this does seem like the preferable solution.
Thank you everybody for chiming in.