[PATCH] CodingGuidelines: explicitly allow "local" for test scripts

[PATCH] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

01d3a526 (t0000: check whether the shell supports the "local"
keyword, 2017-10-26) raised a test balloon to see if those who build
and test Git use a platform without a shell that lacks support for
the "local" keyword.  After two years, 7f0b5908 (t0000: reword
comments for "local" test, 2019-08-08) documented that "local"
keyword, even though is outside POSIX, is allowed in our test
scripts.

Let's write it in the CodingGuidelines, too.  It might be tempting
to allow it in scripted Porcelains (we have avoided getting them
contaminiated by "local" so far), but they are on their way out and
getting rewritten in C.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/CodingGuidelines | 5 +++++
 1 file changed, 5 insertions(+)

diff --git c/Documentation/CodingGuidelines w/Documentation/CodingGuidelines
index 45465bc0c9..ea70676a30 100644
--- c/Documentation/CodingGuidelines
+++ w/Documentation/CodingGuidelines
@@ -175,6 +175,11 @@ For shell scripts specifically (not exhaustive):
 
    does not have such a problem.
 
+ - Even though "local" is not part of POSIX, we make heavy use of it
+   in our test suite.  We do not use it in scripted Porcelains, and
+   hopefully nobody starts using "local" before they are reimplemented
+   in C ;-)
+
 
 For C programs:
 

Re: [PATCH] CodingGuidelines: explicitly allow "local" for test scripts

[Eric Sunshine]

On Mon, May 3, 2021 at 12:11 AM Junio C Hamano <gitster@pobox.com> wrote:
> 01d3a526 (t0000: check whether the shell supports the "local"
> keyword, 2017-10-26) raised a test balloon to see if those who build
> and test Git use a platform without a shell that lacks support for
> the "local" keyword. [...]

Double negative whammy. Should be either:

    use a platform with a shell that lacks support

or:

    use a platform without a shell supporting "local"

Or I'm confused and misunderstanding what was written.

Re: [PATCH] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> Double negative whammy. Should be either:
>
>     use a platform with a shell that lacks support
>
> or:
>
>     use a platform without a shell supporting "local"
>
> Or I'm confused and misunderstanding what was written.

No, I just wasn't paying much attention to what I was writing X-<.
Thanks for spotting.

[PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

01d3a526 (t0000: check whether the shell supports the "local"
keyword, 2017-10-26) raised a test balloon to see if those who build
and test Git use a platform with a shell that lacks support for the
"local" keyword.  After two years, 7f0b5908 (t0000: reword comments
for "local" test, 2019-08-08) documented that "local" keyword, even
though is outside POSIX, is allowed in our test scripts.

Let's write it in the CodingGuidelines, too.  It might be tempting
to allow it in scripted Porcelains (we have avoided getting them
contaminiated by "local" so far), but they are on their way out and
getting rewritten in C.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/CodingGuidelines | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
index 45465bc0c9..ea70676a30 100644
--- a/Documentation/CodingGuidelines
+++ b/Documentation/CodingGuidelines
@@ -175,6 +175,11 @@ For shell scripts specifically (not exhaustive):
 
    does not have such a problem.
 
+ - Even though "local" is not part of POSIX, we make heavy use of it
+   in our test suite.  We do not use it in scripted Porcelains, and
+   hopefully nobody starts using "local" before they are reimplemented
+   in C ;-)
+
 
 For C programs:
 
-- 
2.31.1-579-ge1192275af

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Ævar Arnfjörð Bjarmason]

On Mon, May 03 2021, Junio C Hamano wrote:

> 01d3a526 (t0000: check whether the shell supports the "local"
> keyword, 2017-10-26) raised a test balloon to see if those who build
> and test Git use a platform with a shell that lacks support for the
> "local" keyword.  After two years, 7f0b5908 (t0000: reword comments
> for "local" test, 2019-08-08) documented that "local" keyword, even
> though is outside POSIX, is allowed in our test scripts.
>
> Let's write it in the CodingGuidelines, too.  It might be tempting
> to allow it in scripted Porcelains (we have avoided getting them
> contaminiated by "local" so far), but they are on their way out and
> getting rewritten in C.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>  Documentation/CodingGuidelines | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
> index 45465bc0c9..ea70676a30 100644
> --- a/Documentation/CodingGuidelines
> +++ b/Documentation/CodingGuidelines
> @@ -175,6 +175,11 @@ For shell scripts specifically (not exhaustive):
>  
>     does not have such a problem.
>  
> + - Even though "local" is not part of POSIX, we make heavy use of it
> +   in our test suite.  We do not use it in scripted Porcelains, and
> +   hopefully nobody starts using "local" before they are reimplemented
> +   in C ;-)
> +

Is there any portability reason to avoid "local" in the porcelains? I
don't have any plans for using it, but I don't see why we'd explicitly
forbid it.

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> Is there any portability reason to avoid "local" in the porcelains? I
> don't have any plans for using it, but I don't see why we'd explicitly
> forbid it.

Things that are not even in POSIX are forbidden unless explicitly
allowed.

In general, he way we encouraged people to think has been "don't use
it, it is not even in POSIX" and "even if it is in POSIX, we know
the support by platform/implementation X is broken, so let's not use
it".  It has been successfully helped us to stay out of portability
troubles.

There may be a few tiny cases where we said "practically everybody
we care about has it, even though it is not in POSIX, and it makes
our life so vastly be better" to explicitly allow some feature,
though.

And "local"?  Not absolutely essential, unless you are doing a
library that you want to avoid stepping on users' toes.  Besides, we
are no longer adding scripted Porcelains left and right---rather,
people are actively rewriting them.

Thanks.

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Ævar Arnfjörð Bjarmason]

On Tue, May 04 2021, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> Is there any portability reason to avoid "local" in the porcelains? I
>> don't have any plans for using it, but I don't see why we'd explicitly
>> forbid it.
>
> Things that are not even in POSIX are forbidden unless explicitly
> allowed.
>
> In general, he way we encouraged people to think has been "don't use
> it, it is not even in POSIX" and "even if it is in POSIX, we know
> the support by platform/implementation X is broken, so let's not use
> it".  It has been successfully helped us to stay out of portability
> troubles.
>
> There may be a few tiny cases where we said "practically everybody
> we care about has it, even though it is not in POSIX, and it makes
> our life so vastly be better" to explicitly allow some feature,
> though.
>
> And "local"?  Not absolutely essential, unless you are doing a
> library that you want to avoid stepping on users' toes.  Besides, we
> are no longer adding scripted Porcelains left and right---rather,
> people are actively rewriting them.

[You mailed me off-list with "Did I forget to say it is not even
POSIX?", I had this reply to that, also applicable here]:

BEGIN QUOTE

I'm aware that it's not in POSIX. What I'm getting at is that anyone who
doesn't support this must have been failing t0000-basic.sh since
01d3a526ad9 (t0000: check whether the shell supports the "local"
keyword, 2017-10-26), released with v2.16.0.

And all tests of any kind since 78dc08875cd (test-lib: allow short
options to be bundled, 2020-03-22), perhaps earlier, I didn't trace the
full includes, but that's the first use in test-lib.sh itself. Released
with v2.27.0.

From my own cross-platform testing and us not having any reports about
this I very much suspect that this is one of those not-in-POSIX but in
practice supported everywhere, or at least anywhere Git has been ported
to.

So just saying it's OK to use unconditionally of "in t/?" should be
fine, like e.g. some of the C89-plus-XYZ features.

END QUOTE

Afterwards it occurred to me that your stance on in-tree *.sh goes
against e.g. brian's opinion on it expressed in reply to [1], i.e. that
we might justify future new *.sh built-ins.

In any case, as noted above I really don't care much about using "local"
in any *.sh built-in. I just don't think this "We do not use it in
scripted Porcelains[...]" clause is needed at all.

It's effectively synonymous with saying "we still want to support git on
platforms that are so broken they can't even run a single test in our
test suite".

1. https://lore.kernel.org/git/patch-1.1-83266f30b67-20210417T084346Z-avarab@gmail.com/

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> It's effectively synonymous with saying "we still want to support git on
> platforms that are so broken they can't even run a single test in our
> test suite".

Not really.  Those on such a platform would (rightly) say that it is
the test suite that is broken and out of compliance.

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Ævar Arnfjörð Bjarmason]

On Tue, May 04 2021, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> It's effectively synonymous with saying "we still want to support git on
>> platforms that are so broken they can't even run a single test in our
>> test suite".
>
> Not really.  Those on such a platform would (rightly) say that it is
> the test suite that is broken and out of compliance.

Indeed. But the lack of any reports about that suggests that in practice
this is universally supported enough to be a hard dependency.

In any case, it's clear you don't agree and you manage the patch
queue. So I'll leave it at that.

My aim here was to discover if we had any reason to think that "local"
was less universally implemented than other POSIX/C89-plus features we
rely on. It seems that it's not.

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Felipe Contreras]

Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 04 2021, Junio C Hamano wrote:
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> >
> >> It's effectively synonymous with saying "we still want to support git on
> >> platforms that are so broken they can't even run a single test in our
> >> test suite".
> >
> > Not really.  Those on such a platform would (rightly) say that it is
> > the test suite that is broken and out of compliance.
> 
> Indeed. But the lack of any reports about that suggests that in practice
> this is universally supported enough to be a hard dependency.

It is universally supported but not always in the same way, which is why
it hasn't managed to become part of POSIX.

I for example stumbled upon an issue where 'local x' does not work the
same in bash and zsh, triggering a huge discussion on the zsh mailing
list.

However, if you set a value (e.g. 'local x=') then you shouldn't have
any problems.

  f () { local x; echo ${x-unset}; }; f # this behavior varies

For a full discussion check [1].

[1] https://www.austingroupbugs.net/bug_view_page.php?bug_id=767

-- 
Felipe Contreras

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, May 04 2021, Junio C Hamano wrote:
> 
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> >
> >> It's effectively synonymous with saying "we still want to support git on
> >> platforms that are so broken they can't even run a single test in our
> >> test suite".
> >
> > Not really.  Those on such a platform would (rightly) say that it is
> > the test suite that is broken and out of compliance.
> 
> Indeed. But the lack of any reports about that suggests that in practice
> this is universally supported enough to be a hard dependency.
> 
> In any case, it's clear you don't agree and you manage the patch
> queue. So I'll leave it at that.

I don't feel very strongly, but I would be fine with requiring local.
One of the main reasons the Austin Group is having trouble standardizing
it is because some shells implement it with lexical scoping and some use
dynamic scoping, but if we try not to make too many assumptions, we'll
probably be okay.

> My aim here was to discover if we had any reason to think that "local"
> was less universally implemented than other POSIX/C89-plus features we
> rely on. It seems that it's not.

"local" is missing in AT&T ksh.  As far as I'm aware, all of the other
major open source shells support it: bash, dash, mksh, posh, and zsh,
plus the default shells on most BSDs, so there are options for people
who would like to use Git on systems with a less capable shell.  As a
practical matter, that means someone on a proprietary Unix or possibly a
non-Unix system.  In the latter case, we've only seen Plan 9, I believe,
which is so devoid of reasonably functional basic Unix tools that it's
probably hopeless, and therefore we really only need to consider Windows
and Unix systems.

I should point out that we also make several non-POSIX assumptions about
shell behavior in our testsuite.  I fixed one in c64368e3a2 ("t9001:
avoid including non-trailing NUL bytes in variables", 2019-11-27), but
the other one we make is that all components of a pipeline are run in
subshells, which is not true of AT&T ksh or zsh (in zsh mode), which run
the last item in the main shell.  This assumption used to break running
zsh on our testsuite, but the developers recently accepted a patch to
make zsh in sh mode emulate what all other sh implementations do because
this assumption is so widespread that, as a practical matter, many
things break in such a case (the Git testsuite being one of them, but
also things like Debian's debconf).

So I'm okay with requiring a little more than POSIX behavior here
because as a practical matter we already do and POSIX permits a wide
variety of behavior which is never implemented (e.g., running something
_other_ than the last element in a pipeline in the main shell) and which
we could not practically test.  I agree that we should aim for targets
which provide excellent compatibility and that when in doubt, we should
look to POSIX.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> One of the main reasons the Austin Group is having trouble standardizing
> it is because some shells implement it with lexical scoping and some use
> dynamic scoping, but if we try not to make too many assumptions, we'll
> probably be okay.

Thanks for raising that one.  As long as we make sure that our
script would work with anybody's "local", I do not care very much.
I just didn't think we want to spend the effort to explain that in
the documentation and enforce it---compared to the effort, just
saying "no" is certainly easier.

But if somebody wants an incremental patch on top to spell out how
"local" is allowed to be used in our scripts, that's fine by me.

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Felipe Contreras]

brian m. carlson wrote:
> On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote:
> > My aim here was to discover if we had any reason to think that "local"
> > was less universally implemented than other POSIX/C89-plus features we
> > rely on. It seems that it's not.
> 
> "local" is missing in AT&T ksh.

It's not missing, it's supported only in "functions", which have a
different syntax in ksh:

  function f { local x="foo"; echo $x; }; f

-- 
Felipe Contreras

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On 2021-05-05 at 00:08:17, Felipe Contreras wrote:
> brian m. carlson wrote:
> > On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote:
> > > My aim here was to discover if we had any reason to think that "local"
> > > was less universally implemented than other POSIX/C89-plus features we
> > > rely on. It seems that it's not.
> > 
> > "local" is missing in AT&T ksh.
> 
> It's not missing, it's supported only in "functions", which have a
> different syntax in ksh:
> 
>   function f { local x="foo"; echo $x; }; f

I believe there's a bash compatibility mode that enables this, but I
don't see support for it in the version Debian ships[0], which doesn't
enable that mode:

  ksh -c 'function f { local x="foo"; echo $x; }; f'
  ksh: f[1]: local: not found [No such file or directory]

It's also possible that it may be aliased to "typeset" on some systems,
in which case it may indeed work.

Regardless of the specifics, I think we can both agree that it doesn't
work in sh-style functions, so for Git's purposes, AT&T ksh does not
meet our needs in terms of support for local.

[0] I believe Debian's philosophy on this is that they ship bash, so
    programs that need bashisms can just use bash.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Felipe Contreras]

brian m. carlson wrote:
> On 2021-05-05 at 00:08:17, Felipe Contreras wrote:
> > brian m. carlson wrote:
> > > On 2021-05-04 at 15:09:54, Ævar Arnfjörð Bjarmason wrote:
> > > > My aim here was to discover if we had any reason to think that "local"
> > > > was less universally implemented than other POSIX/C89-plus features we
> > > > rely on. It seems that it's not.
> > > 
> > > "local" is missing in AT&T ksh.
> > 
> > It's not missing, it's supported only in "functions", which have a
> > different syntax in ksh:
> > 
> >   function f { local x="foo"; echo $x; }; f
> 
> I believe there's a bash compatibility mode that enables this, but I
> don't see support for it in the version Debian ships[0], which doesn't
> enable that mode:
> 
>   ksh -c 'function f { local x="foo"; echo $x; }; f'
>   ksh: f[1]: local: not found [No such file or directory]

I tried to build ksh myself to make sure, and apparently there's a mess
of different versions of it. I manually applied some patches and tried
three times before giving up.

It's clear the Debian version and the Arch Linux version are completely
different.

> Regardless of the specifics, I think we can both agree that it doesn't
> work in sh-style functions, so for Git's purposes, AT&T ksh does not
> meet our needs in terms of support for local.

Indeed. But it also probably doesn't meet many other needs. A quick
check shows 54 failed tests just on t0000-basic.sh. I don't think it's
something we should even consider.

Cheers.

-- 
Felipe Contreras

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Jeff King]

On Tue, May 04, 2021 at 10:59:02PM -0500, Felipe Contreras wrote:

> > I believe there's a bash compatibility mode that enables this, but I
> > don't see support for it in the version Debian ships[0], which doesn't
> > enable that mode:
> > 
> >   ksh -c 'function f { local x="foo"; echo $x; }; f'
> >   ksh: f[1]: local: not found [No such file or directory]
> 
> I tried to build ksh myself to make sure, and apparently there's a mess
> of different versions of it. I manually applied some patches and tried
> three times before giving up.
> 
> It's clear the Debian version and the Arch Linux version are completely
> different.
> 
> > Regardless of the specifics, I think we can both agree that it doesn't
> > work in sh-style functions, so for Git's purposes, AT&T ksh does not
> > meet our needs in terms of support for local.
> 
> Indeed. But it also probably doesn't meet many other needs. A quick
> check shows 54 failed tests just on t0000-basic.sh. I don't think it's
> something we should even consider.

Yeah, we have visited this off and on over the years (links below).
There are some modernized ksh variants that work OK (like mksh), but I
think we've declared ksh93 as not worth it.

-Peff

[0] https://lore.kernel.org/git/31108626.20150508231514@gmail.com/
[1] https://lore.kernel.org/git/xmqqinxt3kwq.fsf@gitster.mtv.corp.google.com/

Re: [PATCH v2] CodingGuidelines: explicitly allow "local" for test scripts

[Felipe Contreras]

Jeff King wrote:
> On Tue, May 04, 2021 at 10:59:02PM -0500, Felipe Contreras wrote:
> > Indeed. But it also probably doesn't meet many other needs. A quick
> > check shows 54 failed tests just on t0000-basic.sh. I don't think it's
> > something we should even consider.
> 
> Yeah, we have visited this off and on over the years (links below).
> There are some modernized ksh variants that work OK (like mksh), but I
> think we've declared ksh93 as not worth it.

Just to clarify what was said on that thread: ksh93 is not a version.
There is at least ksh93u+ and ksh93v-, which are *completely* different.

-- 
Felipe Contreras