Solaris and sed: Too many commands, last: s/\n//

Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

Hi Everyone,

I enabled self tests for Solaris. Solaris has some anemic utilities so
I put /usr/gnu/bin first on-path.

make test is resulting in a lot of:

gmake -C templates  SHELL_PATH='/bin/bash' PERL_PATH='/usr/bin/perl'
gmake[1]: Entering directory
`/export/home/jwalton/Build-Scripts/git-2.21.0/templates'
: no custom templates yet
gmake[1]: Leaving directory
`/export/home/jwalton/Build-Scripts/git-2.21.0/templates'
gmake -C t/ all
gmake[1]: Entering directory `/export/home/jwalton/Build-Scripts/git-2.21.0/t'
rm -f -r 'test-results'
sed: Too many commands, last: s/\n//
--- chainlint/arithmetic-expansion.expect       2019-02-24
11:31:46.000000000 -0500
+++ chainlinttmp/arithmetic-expansion.actual    2019-03-11
12:20:16.880610011 -0400
@@ -1,9 +0,0 @@
-(
-       foo &&
-       bar=$((42 + 1)) &&
-       baz
->) &&
-(
-?!AMP?!        bar=$((42 + 1))
-       baz
->)
sed: Too many commands, last: s/\n//
--- chainlint/bash-array.expect 2019-02-24 11:31:46.000000000 -0500
+++ chainlinttmp/bash-array.actual      2019-03-11 12:20:16.902186323 -0400
@@ -1,10 +0,0 @@
-(
-       foo &&
-       bar=(gumbo stumbo wumbo) &&
-       baz
->) &&
-(
-       foo &&
-       bar=${#bar[@]} &&
-       baz
->)
sed: Too many commands, last: s/\n//

A more complete output is available at https://pastebin.com/gpZMUVmQ,
but it is more of the same.

Solaris in a VM sucks. I can provide SSH access to the hardware if
anyone is interested. It is just an Solaris i86pc on an older Ivy
Bridge.

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Eric Sunshine]

[cc:+Ævar]

On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> I enabled self tests for Solaris. Solaris has some anemic utilities so
> I put /usr/gnu/bin first on-path.

The first question is if you are really running GNU 'sed'? My guess is
"no, it's still picking up Solaris's 'sed'".

> gmake -C templates  SHELL_PATH='/bin/bash' PERL_PATH='/usr/bin/perl'
> sed: Too many commands, last: s/\n//
> --- chainlint/arithmetic-expansion.expect       2019-02-24

Ævar had taken a stab at getting this to work on Solaris back in
September 2018. He did fix one or two issues, but the one you report
here was not addressed. He ultimately abandoned the attempt, I think,
after Junio's reminder that the project had long ago declared
Solaris's tools as a lost cause. See the relevant messages:

http://public-inbox.org/git/87bm9cs5y6.fsf@evledraar.gmail.com/
http://public-inbox.org/git/CAPig+cRrwW662Hzp8V1sH51x8qYxPyqG=VHpOkfd-=RwUSSZbw@mail.gmail.com
http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com
http://public-inbox.org/git/xmqqzhwv51wy.fsf@gitster-ct.c.googlers.com

> Solaris in a VM sucks. I can provide SSH access to the hardware if
> anyone is interested. It is just an Solaris i86pc on an older Ivy
> Bridge.

I wouldn't mind taking a look at it, though I don't promise anything,
and I suspect the only way forward is by ensuring that the GNU or XPG
tools are used instead of the Solaris ones.

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> [cc:+Ævar]
>
> On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > I put /usr/gnu/bin first on-path.
>
> The first question is if you are really running GNU 'sed'? My guess is
> "no, it's still picking up Solaris's 'sed'".

I believe so. After modifying PATH, command -v returns:

    Solaris tools:
         sed: /usr/gnu/bin/sed
         awk: /usr/gnu/bin/awk
        grep: /usr/gnu/bin/grep

(This was added to my scripts to confirm).

Maybe Git would benefit from SED, AWK and GREP variables like PERL.

> ...
> > Solaris in a VM sucks. I can provide SSH access to the hardware if
> > anyone is interested. It is just an Solaris i86pc on an older Ivy
> > Bridge.
>
> I wouldn't mind taking a look at it, though I don't promise anything,
> and I suspect the only way forward is by ensuring that the GNU or XPG
> tools are used instead of the Solaris ones.

Send over your authorized_keys. You will connect with:

    esunshine@151.196.22.177 -p 1523

I'm in a dynamic IP address block. You will have to ping me on
occasion to get the updated IP address.

Some other machines you may be interested in:

  * PowerMac G5, PPC big-endian with OS X 10.5 (port 1522)
  * MacBook late 2012, x86_64 with OS X 10.9 (port 1524)
  * Intel Goldmont with SHA extensions (port 1526)

Andy Polyakov uses the PowerMac for tuning his ASM used in OpenSSL.

If you want to speedup SHA-1 (re: unaligned accesses) then try the
Goldmont machine. SHA-1 runs at 1.8 cycles per byte on Goldmont.
Here's the compression function ready for copy/paste:
https://github.com/noloader/SHA-Intrinsics

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Eric Sunshine]

On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote:
> On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > I put /usr/gnu/bin first on-path.
> >
> > The first question is if you are really running GNU 'sed'? My guess is
> > "no, it's still picking up Solaris's 'sed'".
> 
> I believe so. After modifying PATH, command -v returns:
> 
>     Solaris tools:
>          sed: /usr/gnu/bin/sed
>          awk: /usr/gnu/bin/awk
>         grep: /usr/gnu/bin/grep
> 
> (This was added to my scripts to confirm).

Can you doubly confirm by having t/Makefile tell us itself? I ask because that error message in your original email is exactly the error spit out by Solaris 'sed'. Perhaps apply this patch:

diff --git a/t/Makefile b/t/Makefile
index c83fd18861..d22eff4463 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -70,6 +70,7 @@ clean-chainlint:
 
 check-chainlint:
 	@mkdir -p '$(CHAINLINTTMP_SQ)' && \
+	printf 'SED VERSION: ' && sed --version && \
 	err=0 && \
 	for i in $(CHAINLINTTESTS); do \
 		$(CHAINLINT) <chainlint/$$i.test | \

If it reports "SED VERSION: sed (GNU sed) x.y", then we can feel reasonably assured that it's using GNU 'sed', however, if it errors out or prints something else, then it's not picking up your PATH for some reason.

> Maybe Git would benefit from SED, AWK and GREP variables like PERL.

Very possibly.

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote:
> > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > > I put /usr/gnu/bin first on-path.
> > >
> > > The first question is if you are really running GNU 'sed'? My guess is
> > > "no, it's still picking up Solaris's 'sed'".
> >
> > I believe so. After modifying PATH, command -v returns:
> >
> >     Solaris tools:
> >          sed: /usr/gnu/bin/sed
> >          awk: /usr/gnu/bin/awk
> >         grep: /usr/gnu/bin/grep
> >
> > (This was added to my scripts to confirm).
> ...
>
> > Maybe Git would benefit from SED, AWK and GREP variables like PERL.
>
> Very possibly.

Another potential workaround is to use Perl. Perl is already a
prerequisite, it get passed to the test gear through PERL_PATH, and it
avoids Solaris' anemic tools.

Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch.

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> >
> > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote:
> > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > > > I put /usr/gnu/bin first on-path.
> > > >
> > > > The first question is if you are really running GNU 'sed'? My guess is
> > > > "no, it's still picking up Solaris's 'sed'".
> > >
> > > I believe so. After modifying PATH, command -v returns:
> > >
> > >     Solaris tools:
> > >          sed: /usr/gnu/bin/sed
> > >          awk: /usr/gnu/bin/awk
> > >         grep: /usr/gnu/bin/grep
> > >
> > > (This was added to my scripts to confirm).
> > ...
> >
> > > Maybe Git would benefit from SED, AWK and GREP variables like PERL.
> >
> > Very possibly.
>
> Another potential workaround is to use Perl. Perl is already a
> prerequisite, it get passed to the test gear through PERL_PATH, and it
> avoids Solaris' anemic tools.
>
> Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch.

Oh man, you're using GNU make. I thought Git was using that anemic
Posix Make. See attached.

I think Solaris provides an older gawk. Is this an easier problem:

    awk: chainlint.sed:88: :squash
    awk: chainlint.sed:88: ^ syntax error
    awk: chainlint.sed:91:  s/\\\n//
    awk: chainlint.sed:91:    ^ backslash not last character on line
    Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
    Usage: awk [POSIX or GNU style options] [--] 'program' file ...

Jeff

[-- Attachment #2: git2.patch --]
[-- Type: application/octet-stream, Size: 2346 bytes --]

--- t/Makefile
+++ t/Makefile
@@ -11,11 +11,25 @@
 TEST_SHELL_PATH ?= $(SHELL_PATH)
 PERL_PATH ?= /usr/bin/perl
 TAR ?= $(TAR)
+AWK ?= $(AWK)
+SED ?= $(SED)
+GREP ?= $(GREP)
 RM ?= rm -f
 PROVE ?= prove
 DEFAULT_TEST_TARGET ?= test
 TEST_LINT ?= test-lint
 
+# Fix Solaris tools. GNU tools at /usr/gnu/bin. Posix tools /usr/xpg4/bin.
+ifneq ($(wildcard /usr/gnu/bin/grep),)
+  GREP := /usr/gnu/bin/grep
+endif
+ifneq ($(wildcard /usr/gnu/bin/sed),)
+  SED := /usr/gnu/bin/sed
+endif
+ifneq ($(wildcard /usr/gnu/bin/awk),)
+  SED := /usr/gnu/bin/awk
+endif
+
 ifdef TEST_OUTPUT_DIRECTORY
 TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
 CHAINLINTTMP = $(TEST_OUTPUT_DIRECTORY)/chainlinttmp
@@ -35,7 +49,7 @@
 TGITWEB = $(sort $(wildcard t95[0-9][0-9]-*.sh))
 THELPERS = $(sort $(filter-out $(T),$(wildcard *.sh)))
 CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))
-CHAINLINT = sed -f chainlint.sed
+CHAINLINT = $(SED) -f chainlint.sed
 
 all: $(DEFAULT_TEST_TARGET)
 
@@ -44,8 +58,8 @@
 
 failed:
 	@failed=$$(cd '$(TEST_RESULTS_DIRECTORY_SQ)' && \
-		grep -l '^failed [1-9]' *.counts | \
-		sed -n 's/\.counts$$/.sh/p') && \
+		$(GREP) -l '^failed [1-9]' *.counts | \
+		$(SED) -n 's/\.counts$$/.sh/p') && \
 	test -z "$$failed" || $(MAKE) $$failed
 
 prove: pre-clean check-chainlint $(TEST_LINT)
@@ -73,7 +87,7 @@
 	err=0 && \
 	for i in $(CHAINLINTTESTS); do \
 		$(CHAINLINT) <chainlint/$$i.test | \
-		sed -e '/^# LINT: /d' >'$(CHAINLINTTMP_SQ)'/$$i.actual && \
+		$(SED) -e '/^# LINT: /d' >'$(CHAINLINTTMP_SQ)'/$$i.actual && \
 		diff -u chainlint/$$i.expect '$(CHAINLINTTMP_SQ)'/$$i.actual || err=1; \
 	done && exit $$err
 
@@ -81,7 +95,7 @@
 	test-lint-filenames
 
 test-lint-duplicates:
-	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+	@dups=`echo $(T) | tr ' ' '\n' | $(SED) 's/-.*//' | sort | uniq -d` && \
 		test -z "$$dups" || { \
 		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
 
@@ -97,7 +111,7 @@
 	@# We do *not* pass a glob to ls-files but use grep instead, to catch
 	@# non-ASCII characters (which are quoted within double-quotes)
 	@bad="$$(git -c core.quotepath=true ls-files 2>/dev/null | \
-			grep '["*:<>?\\|]')"; \
+			$(GREP) '["*:<>?\\|]')"; \
 		test -z "$$bad" || { \
 		echo >&2 "non-portable file name(s): $$bad"; exit 1; }
 

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote:
> >
> > On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > >
> > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote:
> > > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > > > > I put /usr/gnu/bin first on-path.
> > > > >
> > > > > The first question is if you are really running GNU 'sed'? My guess is
> > > > > "no, it's still picking up Solaris's 'sed'".
> > > >
> > > > I believe so. After modifying PATH, command -v returns:
> > > >
> > > >     Solaris tools:
> > > >          sed: /usr/gnu/bin/sed
> > > >          awk: /usr/gnu/bin/awk
> > > >         grep: /usr/gnu/bin/grep
> > > >
> > > > (This was added to my scripts to confirm).
> > > ...
> > >
> > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL.
> > >
> > > Very possibly.
> >
> > Another potential workaround is to use Perl. Perl is already a
> > prerequisite, it get passed to the test gear through PERL_PATH, and it
> > avoids Solaris' anemic tools.
> >
> > Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch.
>
> Oh man, you're using GNU make. I thought Git was using that anemic
> Posix Make. See attached.
>
> I think Solaris provides an older gawk. Is this an easier problem:
>
>     awk: chainlint.sed:88: :squash
>     awk: chainlint.sed:88: ^ syntax error
>     awk: chainlint.sed:91:  s/\\\n//
>     awk: chainlint.sed:91:    ^ backslash not last character on line
>     Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
>     Usage: awk [POSIX or GNU style options] [--] 'program' file ...

My bad , there was a typo... 'awk' got assigned to SED variable.

This patch works as expected.

Jeff

[-- Attachment #2: git3.patch --]
[-- Type: application/octet-stream, Size: 2656 bytes --]

diff --git a/t/Makefile b/t/Makefile
index c83fd18861..10abb1c34c 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -11,11 +11,25 @@ SHELL_PATH ?= $(SHELL)
 TEST_SHELL_PATH ?= $(SHELL_PATH)
 PERL_PATH ?= /usr/bin/perl
 TAR ?= $(TAR)
+AWK ?= $(AWK)
+SED ?= $(SED)
+GREP ?= $(GREP)
 RM ?= rm -f
 PROVE ?= prove
 DEFAULT_TEST_TARGET ?= test
 TEST_LINT ?= test-lint
 
+# Fix Solaris tools. These are Posix. GNU tools located at /usr/gnu/bin.
+ifneq ($(wildcard /usr/gnu/bin/grep),)
+  GREP := /usr/gnu/bin/grep
+endif
+ifneq ($(wildcard /usr/gnu/bin/sed),)
+  SED := /usr/gnu/bin/sed
+endif
+ifneq ($(wildcard /usr/gnu/bin/awk),)
+  SED := /usr/gnu/bin/awk
+endif
+
 ifdef TEST_OUTPUT_DIRECTORY
 TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
 CHAINLINTTMP = $(TEST_OUTPUT_DIRECTORY)/chainlinttmp
@@ -35,7 +49,7 @@ T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
 TGITWEB = $(sort $(wildcard t95[0-9][0-9]-*.sh))
 THELPERS = $(sort $(filter-out $(T),$(wildcard *.sh)))
 CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))
-CHAINLINT = sed -f chainlint.sed
+CHAINLINT = $(SED) -f chainlint.sed
 
 all: $(DEFAULT_TEST_TARGET)
 
@@ -44,8 +58,8 @@ test: pre-clean check-chainlint $(TEST_LINT)
 
 failed:
 	@failed=$$(cd '$(TEST_RESULTS_DIRECTORY_SQ)' && \
-		grep -l '^failed [1-9]' *.counts | \
-		sed -n 's/\.counts$$/.sh/p') && \
+		$(GREP) -l '^failed [1-9]' *.counts | \
+		$(SED) -n 's/\.counts$$/.sh/p') && \
 	test -z "$$failed" || $(MAKE) $$failed
 
 prove: pre-clean check-chainlint $(TEST_LINT)
@@ -73,7 +87,7 @@ check-chainlint:
 	err=0 && \
 	for i in $(CHAINLINTTESTS); do \
 		$(CHAINLINT) <chainlint/$$i.test | \
-		sed -e '/^# LINT: /d' >'$(CHAINLINTTMP_SQ)'/$$i.actual && \
+		$(SED) -e '/^# LINT: /d' >'$(CHAINLINTTMP_SQ)'/$$i.actual && \
 		diff -u chainlint/$$i.expect '$(CHAINLINTTMP_SQ)'/$$i.actual || err=1; \
 	done && exit $$err
 
@@ -81,7 +95,7 @@ test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax \
 	test-lint-filenames
 
 test-lint-duplicates:
-	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+	@dups=`echo $(T) | tr ' ' '\n' | $(SED) 's/-.*//' | sort | uniq -d` && \
 		test -z "$$dups" || { \
 		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
 
@@ -97,7 +111,7 @@ test-lint-filenames:
 	@# We do *not* pass a glob to ls-files but use grep instead, to catch
 	@# non-ASCII characters (which are quoted within double-quotes)
 	@bad="$$(git -c core.quotepath=true ls-files 2>/dev/null | \
-			grep '["*:<>?\\|]')"; \
+			$(GREP) '["*:<>?\\|]')"; \
 		test -z "$$bad" || { \
 		echo >&2 "non-portable file name(s): $$bad"; exit 1; }
 

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 10:11 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote:
> >
> > On Mon, Mar 11, 2019 at 9:07 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > >
> > > On Mon, Mar 11, 2019 at 6:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > > >
> > > > On Mon, Mar 11, 2019 at 05:43:55PM -0400, Jeffrey Walton wrote:
> > > > > On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > > > > > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > > > > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > > > > > I put /usr/gnu/bin first on-path.
> > > > > >
> > > > > > The first question is if you are really running GNU 'sed'? My guess is
> > > > > > "no, it's still picking up Solaris's 'sed'".
> > > > >
> > > > > I believe so. After modifying PATH, command -v returns:
> > > > >
> > > > >     Solaris tools:
> > > > >          sed: /usr/gnu/bin/sed
> > > > >          awk: /usr/gnu/bin/awk
> > > > >         grep: /usr/gnu/bin/grep
> > > > >
> > > > > (This was added to my scripts to confirm).
> > > > ...
> > > >
> > > > > Maybe Git would benefit from SED, AWK and GREP variables like PERL.
> > > >
> > > > Very possibly.
> > >
> > > Another potential workaround is to use Perl. Perl is already a
> > > prerequisite, it get passed to the test gear through PERL_PATH, and it
> > > avoids Solaris' anemic tools.
> > >
> > > Sadly my Perl sucks. You would be disappointed if I tried to whip up a patch.
> >
> > Oh man, you're using GNU make. I thought Git was using that anemic
> > Posix Make. See attached.
> >
> > I think Solaris provides an older gawk. Is this an easier problem:
> >
> >     awk: chainlint.sed:88: :squash
> >     awk: chainlint.sed:88: ^ syntax error
> >     awk: chainlint.sed:91:  s/\\\n//
> >     awk: chainlint.sed:91:    ^ backslash not last character on line
> >     Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
> >     Usage: awk [POSIX or GNU style options] [--] 'program' file ...
>
> My bad , there was a typo... 'awk' got assigned to SED variable.
>
> This patch works as expected.

https://github.com/git/git/pull/582

Re: Solaris and sed: Too many commands, last: s/\n//

[Eric Sunshine]

On Mon, Mar 11, 2019 at 10:11:41PM -0400, Jeffrey Walton wrote:
> On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > Oh man, you're using GNU make. I thought Git was using that anemic
> > Posix Make. See attached.
> >
> > I think Solaris provides an older gawk. Is this an easier problem:
> >
> >     awk: chainlint.sed:88: :squash
> >     awk: chainlint.sed:88: ^ syntax error
> >     awk: chainlint.sed:91:  s/\\\n//
> >     awk: chainlint.sed:91:    ^ backslash not last character on line
> >     Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
> >     Usage: awk [POSIX or GNU style options] [--] 'program' file ...
> 
> My bad , there was a typo... 'awk' got assigned to SED variable.
> 
> This patch works as expected.
>
> diff --git a/t/Makefile b/t/Makefile
> @@ -11,11 +11,25 @@ SHELL_PATH ?= $(SHELL)
>  TAR ?= $(TAR)
> +AWK ?= $(AWK)
> +SED ?= $(SED)
> +GREP ?= $(GREP)
>  
> +# Fix Solaris tools. These are Posix. GNU tools located at /usr/gnu/bin.
> +ifneq ($(wildcard /usr/gnu/bin/grep),)
> +  GREP := /usr/gnu/bin/grep
> +endif
> +ifneq ($(wildcard /usr/gnu/bin/sed),)
> +  SED := /usr/gnu/bin/sed
> +endif
> +ifneq ($(wildcard /usr/gnu/bin/awk),)
> +  SED := /usr/gnu/bin/awk
> +endif

I think the last assignment ought to be "AWK := ...".

Anyhow, this sort of platform-specific tool customization is typically done by config.mak.uname in the top-level Git directory. In fact, there's already a section for SunOS:

    ifeq ($(uname_S),SunOS)
        ...
        SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin
        ...

Prepending /usr/gnu/bin to SANE_TOOL_PATH might be a good idea as a first step toward fixing the problem you're seeing on Solaris, however, as Ævar mentioned in [1], SANE_TOOL_PATH isn't presently consulted when running tests. But, as he suggested, fixing the tests to respect SANE_TOOL_PATH might be a good solution overall.

So, rather than making platform-specific customizations to t/Makefile, an arguably better solution would be to update config.mak.uname to add /usr/gnu/bin to SANE_TOOL_PATH and then update the test system to respect that value (thus, these GREP, SED, AWK specializations can be avoided).

[1]: http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com/

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 10:45 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Mon, Mar 11, 2019 at 10:11:41PM -0400, Jeffrey Walton wrote:
> > On Mon, Mar 11, 2019 at 9:55 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > Oh man, you're using GNU make. I thought Git was using that anemic
> > > Posix Make. See attached.
> > >
> > > I think Solaris provides an older gawk. Is this an easier problem:
> > >
> > >     awk: chainlint.sed:88: :squash
> > >     awk: chainlint.sed:88: ^ syntax error
> > >     awk: chainlint.sed:91:  s/\\\n//
> > >     awk: chainlint.sed:91:    ^ backslash not last character on line
> > >     Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
> > >     Usage: awk [POSIX or GNU style options] [--] 'program' file ...
> >
> > My bad , there was a typo... 'awk' got assigned to SED variable.
> >
> > This patch works as expected.
> >
> > diff --git a/t/Makefile b/t/Makefile
> > @@ -11,11 +11,25 @@ SHELL_PATH ?= $(SHELL)
> >  TAR ?= $(TAR)
> > +AWK ?= $(AWK)
> > +SED ?= $(SED)
> > +GREP ?= $(GREP)
> >
> > +# Fix Solaris tools. These are Posix. GNU tools located at /usr/gnu/bin.
> > +ifneq ($(wildcard /usr/gnu/bin/grep),)
> > +  GREP := /usr/gnu/bin/grep
> > +endif
> > +ifneq ($(wildcard /usr/gnu/bin/sed),)
> > +  SED := /usr/gnu/bin/sed
> > +endif
> > +ifneq ($(wildcard /usr/gnu/bin/awk),)
> > +  SED := /usr/gnu/bin/awk
> > +endif
>
> I think the last assignment ought to be "AWK := ...".
>
> Anyhow, this sort of platform-specific tool customization is typically done by config.mak.uname in the top-level Git directory. In fact, there's already a section for SunOS:
>
>     ifeq ($(uname_S),SunOS)
>         ...
>         SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin
>         ...
>
> Prepending /usr/gnu/bin to SANE_TOOL_PATH might be a good idea as a first step toward fixing the problem you're seeing on Solaris, however, as Ćvar mentioned in [1], SANE_TOOL_PATH isn't presently consulted when running tests. But, as he suggested, fixing the tests to respect SANE_TOOL_PATH might be a good solution overall.
>
> So, rather than making platform-specific customizations to t/Makefile, an arguably better solution would be to update config.mak.uname to add /usr/gnu/bin to SANE_TOOL_PATH and then update the test system to respect that value (thus, these GREP, SED, AWK specializations can be avoided).

I may have been blowing away SANE_BUILD_PATH.

My build script calls:

   SANE_BUILD_PATH="<my updated tool path> \
   PKG_CONFIG_PATH="..." \
   ... \
./configure \
    --prefix=<my new tool location> \
    --libdir="..." \
    ...

I set SANE_BUILD_PATH in the environment because I want Git to use the
new tools being built for testing.

Or, if Git is blowing away my SANE_BUILD_PATH , then that might
explain the encoding failures. The updated tools, like new iConv and
Unistring, are not on PATH. The updated tools are located at
/var/sanitize/bin, and are passed through SANE_BUILD_PATH .

This might also explain the intermittent crashes I see. Old tools from
/usr/bin are used, but new libraries from /var/sanitize/lib are used
because of LD_LIBRARY_PATH. Gotta love those fucking path problems
that have plagued Linux for the last 30 years. The idiot who figured
it was a good idea to compile and link against one library, and then
load the wrong library at runtime, should get a platinum Darwin award.

It may be better to set SANE_TOOL_PATH something like:

    SANE_TOOL_PATH = "$SANE_TOOL_PATH:/usr/xpg6/bin:/usr/xpg4/bin"

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> [cc:+Ævar]
>
> On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > I put /usr/gnu/bin first on-path.
>
> The first question is if you are really running GNU 'sed'? My guess is
> "no, it's still picking up Solaris's 'sed'".

Yeah, you're right:

$ make test V=1
...

gmake -C t/ all
gmake[1]: Entering directory `/export/home/jwalton/Build-Scripts/git-2.21.0/t'
rm -f -r 'test-results'
SED VERSION: sed: illegal option -- version

I wonder why ... Let me check if I am exporting PATH.

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Jeffrey Walton]

On Mon, Mar 11, 2019 at 7:52 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Mon, Mar 11, 2019 at 5:15 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> >
> > [cc:+Ævar]
> >
> > On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
> > > I enabled self tests for Solaris. Solaris has some anemic utilities so
> > > I put /usr/gnu/bin first on-path.
> >
> > The first question is if you are really running GNU 'sed'? My guess is
> > "no, it's still picking up Solaris's 'sed'".
>
> Yeah, you're right:
>
> $ make test V=1
> ...
>
> gmake -C t/ all
> gmake[1]: Entering directory `/export/home/jwalton/Build-Scripts/git-2.21.0/t'
> rm -f -r 'test-results'
> SED VERSION: sed: illegal option -- version
>
> I wonder why ... Let me check if I am exporting PATH.

I was not exporting PATH, so that was my mistake.

However, PATH is now exported but I get the same result.

Is the build system clearing the environment?

Jeff

Re: Solaris and sed: Too many commands, last: s/\n//

[Ævar Arnfjörð Bjarmason]

On Mon, Mar 11 2019, Eric Sunshine wrote:

> [cc:+Ævar]
>
> On Mon, Mar 11, 2019 at 4:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
>> I enabled self tests for Solaris. Solaris has some anemic utilities so
>> I put /usr/gnu/bin first on-path.
>
> The first question is if you are really running GNU 'sed'? My guess is
> "no, it's still picking up Solaris's 'sed'".
>
>> gmake -C templates  SHELL_PATH='/bin/bash' PERL_PATH='/usr/bin/perl'
>> sed: Too many commands, last: s/\n//
>> --- chainlint/arithmetic-expansion.expect       2019-02-24
>
> Ævar had taken a stab at getting this to work on Solaris back in
> September 2018. He did fix one or two issues, but the one you report
> here was not addressed. He ultimately abandoned the attempt, I think,
> after Junio's reminder that the project had long ago declared
> Solaris's tools as a lost cause. See the relevant messages:
>
> http://public-inbox.org/git/87bm9cs5y6.fsf@evledraar.gmail.com/
> http://public-inbox.org/git/CAPig+cRrwW662Hzp8V1sH51x8qYxPyqG=VHpOkfd-=RwUSSZbw@mail.gmail.com
> http://public-inbox.org/git/877ek0rymz.fsf@evledraar.gmail.com
> http://public-inbox.org/git/xmqqzhwv51wy.fsf@gitster-ct.c.googlers.com
>
>> Solaris in a VM sucks. I can provide SSH access to the hardware if
>> anyone is interested. It is just an Solaris i86pc on an older Ivy
>> Bridge.
>
> I wouldn't mind taking a look at it, though I don't promise anything,
> and I suspect the only way forward is by ensuring that the GNU or XPG
> tools are used instead of the Solaris ones.

FWIW I gave up on Solaris sed after trying to fix it the last time
around.

If you need another box to test it on you can apply for a GCC Farm
account: https://cfarm.tetaneutral.net/machines/list/