Re: [PATCH] tests: add a special setup where prerequisites fail

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Tue, May 14 2019, Johannes Schindelin wrote:

> Hi Ævar,
>
> On Mon, 13 May 2019, Ævar Arnfjörð Bjarmason wrote:
>
>> As discussed in [1] there's a regression in the "pu" branch now
>> because a new test implicitly assumed that a previous test guarded by
>> a prerequisite had been run. Add a "GIT_TEST_FAIL_PREREQS" special
>> test setup where we'll skip (nearly) all tests guarded by
>> prerequisites, allowing us to easily emulate those platform where we
>> don't run these tests.
>>
>> As noted in the documentation I'm adding I'm whitelisting the SYMLINKS
>> prerequisite for now. A lot of tests started failing if we lied about
>> not supporting symlinks. It's also unlikely that we'll have a failing
>> test due to a hard dependency on symlinks without that being the
>> obvious cause, so for now it's not worth the effort to make it work.
>
> I don't know... In Git for Windows, the SYMLINKS prereq is not met.
>
> (Side note: Windows 10 already supports symlinks for quite some time, even
> for non-admin developers, but the fact that Git's test suite is
> implemented in shell script bites us yet one more time: MSYS2 has a
> completely different idea what symlinks are. It uses the "system file" bit
> that only exists on Windows, and if that is set, reads the beginning of
> the file, and if that reads "!<symlink>" (interpreted as ASCII), then the
> rest of that system file is interpreted as the symlink target.)
>
> So it makes me worry if you say that you had to exclude the SYMLINK
> prereq. Maybe all the dependent tests have different prereqs that just so
> happen *also* not to be met on Windows?

I should have clarified this. What I mean is that it leaves SYMLINK
alone, i.e. what was breaking is that on Linux we don't deal with
pretending that we don't support symlinks, these tests should still work
just fine on Windows (or anywhere SYMLINKS is genuinely false), since
there we actually back up our promise not to support them.

>> 1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1905131531000.44@tvgsbejvaqbjf.bet/
>>
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>
>> On Mon, May 13 2019, Johannes Schindelin wrote:
>>
>> > [...]
>> > Namely, when test cases 51 and 52 are skipped because of a missing GPG
>> > prerequisite [*1*], and those two are obviously required to run for the
>> > `git merge to fail in your test case, as you can very easily verify by
>> > downloading the artifact containing the `trash directory.t7600-merge`
>> > directory and re-running the last steps on Linux (where the `git -c
>> > rerere.enabled=true merge master` *succeeds*).
>> >
>> > In fact, you can very, very easily emulate the whole situation on your box
>> > by running:
>> >
>> > 	sh t7600-merge.sh -i -v -x --run=1-50,53-59
>> >
>> > And then you can fix your test case so that it does not need to rely on
>> > test cases that may, or may not, have run previously.
>>
>> I think it would be better to more pro-actively spot this sort of
>> thing in the future, so here's a patch to do that. It passes on
>> "master", but fails on "pu" due to the issue with the one test being
>> discussed here.
>
> It does drive me nuts that the `--run=<N>` option exists (thank you,
> Slavica, for teaching me!) and is so poorly supported by our test suite.
>
> For example, if t7600.59 fails, it would make a ton of sense to run
>
> 	sh t7600-merge.sh -i -v -x --run=59
>
> right?
>
> Except that we frequently have at least one "test case" whose only purpose
> is to set things up.
>
> But we never declare explicitly "test case 59 requires test case 51 to run
> first".
>
> We do not even declare the test cases: we execute them immediately. So we
> would not even be able to juggle them about, e.g. run them in reverse
> (which would otherwise be the easiest way to get rid of almost all side
> effects).
>
> I think in the long run, we'll have to drag Git's test suite into the 21st
> century (kicking and screaming, I'm sure), to have a more declarative
> style, with those features that one might know from Mocha, Jest, JUnit,
> xUnit.NET, etc.
>
> Back to your patch: it only catches prereq problems. But the `--run=59`
> thing would still not be addressed.

Yeah that would be a nice-to-have but unrelated thing (and a much bigger
task). We'd need to split up "this is setup code" v.s. "this is a test",
or make each individual test declare its dependency graph, which could
get quite verbose.

> What would you think about a mode where random test cases are skipped? It
> would have to make sure to provide a way to recreate the problem, e.g.
> giving a string that defines exactly which test cases were skipped.
>
> I am *sure* that tons of test scripts would fail with that, and we would
> probably have to special-case the `setup` "test cases", and we would have
> to clean up quite a few scripts to *not* execute random stuff outside of
> `test_expect_*`...

I think it would be neat, but unrelated to and overkill for spotting the
practical problem we have now, which is that we *know* we skip some of
this now on some platforms/setups due to prereqs.

>> diff --git a/t/README b/t/README
>> index 6404f33e19..9747971d58 100644
>> --- a/t/README
>> +++ b/t/README
>> @@ -334,6 +334,15 @@ that cannot be easily covered by a few specific test cases. These
>>  could be enabled by running the test suite with correct GIT_TEST_
>>  environment set.
>>
>> +GIT_TEST_FAIL_PREREQS<non-empty?> fails all prerequisites. This is
>
> Did you mean to insert `=` after `GIT_TEST_FAIL_PREREQS`?

Yeah, will fix.

>> diff --git a/t/t4202-log.sh b/t/t4202-log.sh
>> index 819c24d10e..c20209324c 100755
>> --- a/t/t4202-log.sh
>> +++ b/t/t4202-log.sh
>> @@ -352,7 +352,7 @@ test_expect_success 'log with grep.patternType configuration and command line' '
>>  	test_cmp expect actual
>>  '
>>
>> -test_expect_success 'log with various grep.patternType configurations & command-lines' '
>> +test_expect_success !FAIL_PREREQS 'log with various grep.patternType configurations & command-lines' '
>
> Is this an indication of a bug in this test case?
>
>>  	git init pattern-type &&
>>  	(
>>  		cd pattern-type &&
>> diff --git a/t/t7405-submodule-merge.sh b/t/t7405-submodule-merge.sh
>> index 7855bd8648..aa33978ed2 100755
>> --- a/t/t7405-submodule-merge.sh
>> +++ b/t/t7405-submodule-merge.sh
>> @@ -417,7 +417,7 @@ test_expect_failure 'directory/submodule conflict; keep submodule clean' '
>>  	)
>>  '
>>
>> -test_expect_failure 'directory/submodule conflict; should not treat submodule files as untracked or in the way' '
>> +test_expect_failure !FAIL_PREREQS 'directory/submodule conflict; should not treat submodule files as untracked or in the way' '
>
> Same here?
>
>>  	test_when_finished "git -C directory-submodule/path reset --hard" &&
>>  	test_when_finished "git -C directory-submodule reset --hard" &&
>>  	(
>> diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
>> index 2e1bb61b41..7d7b396c23 100755
>> --- a/t/t7810-grep.sh
>> +++ b/t/t7810-grep.sh
>> @@ -412,7 +412,7 @@ do
>>  		test_cmp expected actual
>>  	'
>>
>> -	test_expect_success !PCRE "grep $L with grep.patterntype=perl errors without PCRE" '
>> +	test_expect_success !FAIL_PREREQS,!PCRE "grep $L with grep.patterntype=perl errors without PCRE" '
>
> And here?
>
>>  		test_must_fail git -c grep.patterntype=perl grep "foo.*bar"
>>  	'
>>
>> @@ -1234,7 +1234,7 @@ test_expect_success PCRE 'grep --perl-regexp pattern' '
>>  	test_cmp expected actual
>>  '
>>
>> -test_expect_success !PCRE 'grep --perl-regexp pattern errors without PCRE' '
>> +test_expect_success !FAIL_PREREQS,!PCRE 'grep --perl-regexp pattern errors without PCRE' '
>
> And here?
>
>>  	test_must_fail git grep --perl-regexp "foo.*bar"
>>  '
>>
>> @@ -1249,7 +1249,7 @@ test_expect_success LIBPCRE2 "grep -P with (*NO_JIT) doesn't error out" '
>>
>>  '
>>
>> -test_expect_success !PCRE 'grep -P pattern errors without PCRE' '
>> +test_expect_success !FAIL_PREREQS,!PCRE 'grep -P pattern errors without PCRE' '
>
> And here?

To all of the above: Not a bug, I guess it could be refactored as an
unrelated fix, but it's tests of the form "without this prereq we should
die", so of course if I lie about not having it where I *do* have it it
won't die.

>>  	test_must_fail git grep -P "foo.*bar"
>>  '
>>
>> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
>> index 8270de74be..0367cec5fd 100644
>> --- a/t/test-lib-functions.sh
>> +++ b/t/test-lib-functions.sh
>> @@ -309,6 +309,26 @@ test_unset_prereq () {
>>  }
>>
>>  test_set_prereq () {
>> +	if test -n "$GIT_TEST_FAIL_PREREQS"
>> +	then
>> +		case "$1" in
>> +		# The "!" case is handled below with
>> +		# test_unset_prereq()
>> +		!*)
>> +			;;
>> +		# (Temporary?) whitelist of things we can't easily
>> +		# pretend not to support
>> +		SYMLINKS)
>> +			;;
>> +		# Inspecting whether GIT_TEST_FAIL_PREREQS is on
>> +		# should be unaffected.
>> +		FAIL_PREREQS)
>> +			;;
>> +		*)
>> +			return
>> +		esac
>> +	fi
>> +
>
> I would probably have done that on the reading side rather than the
> writing side ;-)

I started out by doing that (I'll note so in the v2 commit message), but
found it much harier, in that function we need to deal with the lazy
prereqs v.s. non-lazy, so I'd need to change multiple branches of that
or refactor it, all that stuff calls test_set_prereq() after it
discovers what the prereq is, so doing it here is much easier &
straightforward.

>>  	case "$1" in
>>  	!*)
>>  		test_unset_prereq "${1#!}"
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index 908ddb9c46..6fabafebb3 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1607,3 +1607,7 @@ test_lazy_prereq SHA1 '
>>  test_lazy_prereq REBASE_P '
>>  	test -z "$GIT_TEST_SKIP_REBASE_P"
>>  '
>> +
>> +test_lazy_prereq FAIL_PREREQS '
>> +	test -n "$GIT_TEST_FAIL_PREREQS"
>> +'
>> --
>> 2.21.0.1020.gf2820cf01a
>>
>>