[BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Randall S. Becker]

Hi All,

We are running the regression suite (as we always have) in Jenkins and are
finding that t0500 is basically a complete failure. I cannot recreate it
using bash where we have a real terminal, but without one, the t0500
consistently fails. I wonder whether any other platform is experiencing the
same situation.

We also find that we cannot run the main test suite as a background job - we
get segmentation faults that I have yet to isolate, but when run in the
foreground, no problem.

I wanted to get this out there because 2.24.0 goes too far as this did not
happen in 2.23.0 or any release prior.

Sincerely,
Randall

-- Brief whoami:
NonStop developer since approximately 211288444200000000
UNIX developer since approximately 421664400
-- In my real life, I talk too much.

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Johannes Schindelin]

Hi Randall,

On Thu, 24 Oct 2019, Randall S. Becker wrote:

> We are running the regression suite (as we always have) in Jenkins and are
> finding that t0500 is basically a complete failure. I cannot recreate it
> using bash where we have a real terminal, but without one, the t0500
> consistently fails. I wonder whether any other platform is experiencing the
> same situation.

This says that a test failed, but it lacks the verbose log, so there is
little else left to the willing reader than wild guesses that are almost
certainly completely missing the culprit.

In other words: verbose log, please. That is, the output of the test
script when run with `-i -v -x` (or `-i -V -x` in which case you will
want to analyze the contents of
`t/test-results/t0500-progress-display.out` and if that does not help,
paste it into a reply).

Ciao,
Johannes

>
> We also find that we cannot run the main test suite as a background job - we
> get segmentation faults that I have yet to isolate, but when run in the
> foreground, no problem.
>
> I wanted to get this out there because 2.24.0 goes too far as this did not
> happen in 2.23.0 or any release prior.
>
> Sincerely,
> Randall
>
> -- Brief whoami:
> NonStop developer since approximately 211288444200000000
> UNIX developer since approximately 421664400
> -- In my real life, I talk too much.
>
>
>
>

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Randall S. Becker]

On October 28, 2019 9:07 AM, Johannes Schindelin wrote:
> On Thu, 24 Oct 2019, Randall S. Becker wrote:
> > We are running the regression suite (as we always have) in Jenkins and
> > are finding that t0500 is basically a complete failure. I cannot
> > recreate it using bash where we have a real terminal, but without one,
> > the t0500 consistently fails. I wonder whether any other platform is
> > experiencing the same situation.
> 
> This says that a test failed, but it lacks the verbose log, so there is
little else
> left to the willing reader than wild guesses that are almost certainly
> completely missing the culprit.
> 
> In other words: verbose log, please. That is, the output of the test
script
> when run with `-i -v -x` (or `-i -V -x` in which case you will want to
analyze the
> contents of `t/test-results/t0500-progress-display.out` and if that does
not
> help, paste it into a reply).

Here is the verbose output from subtest 2 that failed. Sorry it took so
long, we had to arrange a different Jenkins job to test this in isolation.
Format issue?

Cheers,
Randall

expecting success of 0500.2 'progress display with total': 
	cat >expect <<-\EOF &&
	Working hard:  33% (1/3)<CR>
	Working hard:  66% (2/3)<CR>
	Working hard: 100% (3/3)<CR>
	Working hard: 100% (3/3), done.
	EOF

	cat >in <<-\EOF &&
	progress 1
	progress 2
	progress 3
	EOF
	test-tool progress --total=3 "Working hard" <in 2>stderr &&

	show_cr <stderr >out &&
	test_i18ncmp expect out

+ cat
+ 1> expect 0< /tmp/sh1811939370.4
+ cat
+ 1> in 0< /tmp/sh1811939370.5
+ test-tool progress --total=3 Working hard
+ 0< in 2> stderr
+ show_cr
+ 0< stderr 1> out
+ test_i18ncmp expect out
--- expect	2019-10-28 14:11:40 +0000
+++ out	2019-10-28 14:11:41 +0000
@@ -1,4 +1,2 @@
-Working hard:  33% (1/3)<CR>
-Working hard:  66% (2/3)<CR>
-Working hard: 100% (3/3)<CR>
-Working hard: 100% (3/3), done.
+Working hard:   0% (1/12884901888)<CR>
+Working hard:   0% (3/12884901888), done.
error: last command exited with $?=1
not ok 2 - progress display with total
#	
#		cat >expect <<-\EOF &&
#		Working hard:  33% (1/3)<CR>
#		Working hard:  66% (2/3)<CR>
#		Working hard: 100% (3/3)<CR>
#		Working hard: 100% (3/3), done.
#		EOF
#	
#		cat >in <<-\EOF &&
#		progress 1
#		progress 2
#		progress 3
#		EOF
#		test-tool progress --total=3 "Working hard" <in 2>stderr &&
#	
#		show_cr <stderr >out &&
#		test_i18ncmp expect out
#	

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[SZEDER Gábor]

On Mon, Oct 28, 2019 at 10:15:31AM -0400, Randall S. Becker wrote:
> Here is the verbose output from subtest 2 that failed. Sorry it took so
> long, we had to arrange a different Jenkins job to test this in isolation.
> Format issue?
> 
> Cheers,
> Randall
> 
> expecting success of 0500.2 'progress display with total': 
> 	cat >expect <<-\EOF &&
> 	Working hard:  33% (1/3)<CR>
> 	Working hard:  66% (2/3)<CR>
> 	Working hard: 100% (3/3)<CR>
> 	Working hard: 100% (3/3), done.
> 	EOF
> 
> 	cat >in <<-\EOF &&
> 	progress 1
> 	progress 2
> 	progress 3
> 	EOF
> 	test-tool progress --total=3 "Working hard" <in 2>stderr &&
> 
> 	show_cr <stderr >out &&
> 	test_i18ncmp expect out
> 
> + cat
> + 1> expect 0< /tmp/sh1811939370.4

Wow :)

> + cat
> + 1> in 0< /tmp/sh1811939370.5
> + test-tool progress --total=3 Working hard
> + 0< in 2> stderr
> + show_cr
> + 0< stderr 1> out

Another wow.

This is unrelated to the test failure, but makes me rather curious:
you run the test script with '-x', that's why we see the trace of
commands executed during the test.  Interestingly, though, we don't
see any trace of commands executed within any shell function invoked
in the test, i.e. from show_cr above and test_i18ncmp below.

What kind of shell is this?

> + test_i18ncmp expect out
> --- expect	2019-10-28 14:11:40 +0000
> +++ out	2019-10-28 14:11:41 +0000
> @@ -1,4 +1,2 @@
> -Working hard:  33% (1/3)<CR>
> -Working hard:  66% (2/3)<CR>
> -Working hard: 100% (3/3)<CR>
> -Working hard: 100% (3/3), done.
> +Working hard:   0% (1/12884901888)<CR>
> +Working hard:   0% (3/12884901888), done.

Weird, this looks exactly like the big-endian test failure that was
fixed in 2b6f6ea1bd (test-progress: fix test failures on big-endian
systems, 2019-10-20), but that is already in 2.24.0-rc1 (but not yet
in -rc0).

https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-3a566ce689a5@physik.fu-berlin.de/

Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?

> error: last command exited with $?=1
> not ok 2 - progress display with total
> #	
> #		cat >expect <<-\EOF &&
> #		Working hard:  33% (1/3)<CR>
> #		Working hard:  66% (2/3)<CR>
> #		Working hard: 100% (3/3)<CR>
> #		Working hard: 100% (3/3), done.
> #		EOF
> #	
> #		cat >in <<-\EOF &&
> #		progress 1
> #		progress 2
> #		progress 3
> #		EOF
> #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> #	
> #		show_cr <stderr >out &&
> #		test_i18ncmp expect out
> #	
> 

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Randall S. Becker]

On October 28, 2019 10:52 AM, SZEDER Gábor wrote:
> On Mon, Oct 28, 2019 at 10:15:31AM -0400, Randall S. Becker wrote:
> > Here is the verbose output from subtest 2 that failed. Sorry it took
> > so long, we had to arrange a different Jenkins job to test this in isolation.
> > Format issue?
> >
> > Cheers,
> > Randall
> >
> > expecting success of 0500.2 'progress display with total':
> > 	cat >expect <<-\EOF &&
> > 	Working hard:  33% (1/3)<CR>
> > 	Working hard:  66% (2/3)<CR>
> > 	Working hard: 100% (3/3)<CR>
> > 	Working hard: 100% (3/3), done.
> > 	EOF
> >
> > 	cat >in <<-\EOF &&
> > 	progress 1
> > 	progress 2
> > 	progress 3
> > 	EOF
> > 	test-tool progress --total=3 "Working hard" <in 2>stderr &&
> >
> > 	show_cr <stderr >out &&
> > 	test_i18ncmp expect out
> >
> > + cat
> > + 1> expect 0< /tmp/sh1811939370.4
> 
> Wow :)
> 
> > + cat
> > + 1> in 0< /tmp/sh1811939370.5
> > + test-tool progress --total=3 Working hard 0< in 2> stderr show_cr 0<
> > + stderr 1> out
> 
> Another wow.
> 
> This is unrelated to the test failure, but makes me rather curious:
> you run the test script with '-x', that's why we see the trace of commands
> executed during the test.  Interestingly, though, we don't see any trace of
> commands executed within any shell function invoked in the test, i.e. from
> show_cr above and test_i18ncmp below.
> 
> What kind of shell is this?

The test output requested by Johannes was to use -i -v -x. We use ksh and have for years to test git. The command used was done within a Jenkins environment and was:

cd t && sh t0500-progress-display.sh -i -v -x

> 
> > + test_i18ncmp expect out
> > --- expect	2019-10-28 14:11:40 +0000
> > +++ out	2019-10-28 14:11:41 +0000
> > @@ -1,4 +1,2 @@
> > -Working hard:  33% (1/3)<CR>
> > -Working hard:  66% (2/3)<CR>
> > -Working hard: 100% (3/3)<CR>
> > -Working hard: 100% (3/3), done.
> > +Working hard:   0% (1/12884901888)<CR>
> > +Working hard:   0% (3/12884901888), done.
> 
> Weird, this looks exactly like the big-endian test failure that was fixed in
> 2b6f6ea1bd (test-progress: fix test failures on big-endian systems, 2019-10-
> 20), but that is already in 2.24.0-rc1 (but not yet in -rc0).
> 
> https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> 3a566ce689a5@physik.fu-berlin.de/
> 
> Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?

NonStop is big-endian. When t0500 is run from an interactive terminal, it passes. This failure seems to be a result of being in a disconnected terminal situation typical of Jenkins, not that the test result makes any sense with that being the only difference. t0500 did not exist in 2.23.0, our last build, so I can't easily get that answer. Our Jenkins is based off the master branch, so it's a hard to revert in our pipeline without a serious amount of work - that and without 2b6f6ea1bd, other things break if I remember from August.

Does the printf format use positional arguments (%digit$)? That has known issues on the platform.

FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is 0x300000000, surprisingly.

> > error: last command exited with $?=1
> > not ok 2 - progress display with total
> > #
> > #		cat >expect <<-\EOF &&
> > #		Working hard:  33% (1/3)<CR>
> > #		Working hard:  66% (2/3)<CR>
> > #		Working hard: 100% (3/3)<CR>
> > #		Working hard: 100% (3/3), done.
> > #		EOF
> > #
> > #		cat >in <<-\EOF &&
> > #		progress 1
> > #		progress 2
> > #		progress 3
> > #		EOF
> > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > #
> > #		show_cr <stderr >out &&
> > #		test_i18ncmp expect out
> > #
> >

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[SZEDER Gábor]

On Mon, Oct 28, 2019 at 11:20:48AM -0400, Randall S. Becker wrote:
> > > + test_i18ncmp expect out
> > > --- expect	2019-10-28 14:11:40 +0000
> > > +++ out	2019-10-28 14:11:41 +0000
> > > @@ -1,4 +1,2 @@
> > > -Working hard:  33% (1/3)<CR>
> > > -Working hard:  66% (2/3)<CR>
> > > -Working hard: 100% (3/3)<CR>
> > > -Working hard: 100% (3/3), done.
> > > +Working hard:   0% (1/12884901888)<CR>
> > > +Working hard:   0% (3/12884901888), done.
> > 
> > Weird, this looks exactly like the big-endian test failure that was fixed in
> > 2b6f6ea1bd (test-progress: fix test failures on big-endian systems, 2019-10-
> > 20), but that is already in 2.24.0-rc1 (but not yet in -rc0).
> > 
> > https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> > 3a566ce689a5@physik.fu-berlin.de/
> > 
> > Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?
> 
> NonStop is big-endian. When t0500 is run from an interactive
> terminal, it passes. This failure seems to be a result of being in a
> disconnected terminal situation typical of Jenkins, not that the
> test result makes any sense with that being the only difference.
> t0500 did not exist in 2.23.0, our last build,

Didn't you build v2.24.0-rc0?

> so I can't easily get
> that answer. Our Jenkins is based off the master branch, so it's a
> hard to revert in our pipeline without a serious amount of work -
> that and without 2b6f6ea1bd, other things break if I remember from
> August.
> 
> Does the printf format use positional arguments (%digit$)? That has
> known issues on the platform.

The output in question is formatted in display() in progress.c:124:

        strbuf_addf(counters_sb,
                    "%3u%% (%"PRIuMAX"/%"PRIuMAX")%s", percent,
                    (uintmax_t)n, (uintmax_t)progress->total,
                    tp);

where both 'n' and 'progress->total' are uint64_t, and 'PRIuMAX' is
'llu'.  All of these are widely used throughout the code base.

> FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is
> 0x300000000, surprisingly.

It's not surprising, basically that's what 2b6f6ea1bd is all about.

> > > error: last command exited with $?=1
> > > not ok 2 - progress display with total
> > > #
> > > #		cat >expect <<-\EOF &&
> > > #		Working hard:  33% (1/3)<CR>
> > > #		Working hard:  66% (2/3)<CR>
> > > #		Working hard: 100% (3/3)<CR>
> > > #		Working hard: 100% (3/3), done.
> > > #		EOF
> > > #
> > > #		cat >in <<-\EOF &&
> > > #		progress 1
> > > #		progress 2
> > > #		progress 3
> > > #		EOF
> > > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > > #
> > > #		show_cr <stderr >out &&
> > > #		test_i18ncmp expect out
> > > #
> > >
> 

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Randall S. Becker]

On October 29, 2019 6:16 AM, SZEDER Gábor wrote:
> On Mon, Oct 28, 2019 at 11:20:48AM -0400, Randall S. Becker wrote:
> > > > + test_i18ncmp expect out
> > > > --- expect	2019-10-28 14:11:40 +0000
> > > > +++ out	2019-10-28 14:11:41 +0000
> > > > @@ -1,4 +1,2 @@
> > > > -Working hard:  33% (1/3)<CR>
> > > > -Working hard:  66% (2/3)<CR>
> > > > -Working hard: 100% (3/3)<CR>
> > > > -Working hard: 100% (3/3), done.
> > > > +Working hard:   0% (1/12884901888)<CR>
> > > > +Working hard:   0% (3/12884901888), done.
> > >
> > > Weird, this looks exactly like the big-endian test failure that was
> > > fixed in 2b6f6ea1bd (test-progress: fix test failures on big-endian
> > > systems, 2019-10- 20), but that is already in 2.24.0-rc1 (but not yet in -
> rc0).
> > >
> > > https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> > > 3a566ce689a5@physik.fu-berlin.de/
> > >
> > > Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?
> >
> > NonStop is big-endian. When t0500 is run from an interactive terminal,
> > it passes. This failure seems to be a result of being in a
> > disconnected terminal situation typical of Jenkins, not that the test
> > result makes any sense with that being the only difference.
> > t0500 did not exist in 2.23.0, our last build,
> 
> Didn't you build v2.24.0-rc0?

Reporter bonehead moment. Ok, the problem is in v2.24.0-rc0, *NOT* rc1. I am deeply sorry for the confusion. I ran both build test cycles and t0500 in rc1 worked. We are running the full rc1 cycle now.

> > so I can't easily get
> > that answer. Our Jenkins is based off the master branch, so it's a
> > hard to revert in our pipeline without a serious amount of work - that
> > and without 2b6f6ea1bd, other things break if I remember from August.
> >
> > Does the printf format use positional arguments (%digit$)? That has
> > known issues on the platform.
> 
> The output in question is formatted in display() in progress.c:124:
> 
>         strbuf_addf(counters_sb,
>                     "%3u%% (%"PRIuMAX"/%"PRIuMAX")%s", percent,
>                     (uintmax_t)n, (uintmax_t)progress->total,
>                     tp);
> 
> where both 'n' and 'progress->total' are uint64_t, and 'PRIuMAX' is 'llu'.  All
> of these are widely used throughout the code base.
> 
> > FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is
> > 0x300000000, surprisingly.
> 
> It's not surprising, basically that's what 2b6f6ea1bd is all about.
> 
> > > > error: last command exited with $?=1 not ok 2 - progress display
> > > > with total #
> > > > #		cat >expect <<-\EOF &&
> > > > #		Working hard:  33% (1/3)<CR>
> > > > #		Working hard:  66% (2/3)<CR>
> > > > #		Working hard: 100% (3/3)<CR>
> > > > #		Working hard: 100% (3/3), done.
> > > > #		EOF
> > > > #
> > > > #		cat >in <<-\EOF &&
> > > > #		progress 1
> > > > #		progress 2
> > > > #		progress 3
> > > > #		EOF
> > > > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > > > #
> > > > #		show_cr <stderr >out &&
> > > > #		test_i18ncmp expect out
> > > > #

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

[Randall S. Becker]

FYI: git 2.24.0-rc2 passes on the NonStop TNS/E platform - except for the usual t9001 and t9020 that never do and can't.

Thanks to everyone for all the hard work!
Randall

-- Brief whoami:
 NonStop developer since approximately 211288444200000000
 UNIX developer since approximately 421664400
-- In my real life, I talk too much.