[PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[SZEDER Gábor]

Some test prereqs depend on other prereqs, so in a couple of cases we
have nested prereqs that look something like this:

  test_lazy_prereq FOO '
      test_have_prereq BAR &&
      check-foo
  '

This can be problematic, because lazy prereqs are evaluated in the
'$TRASH_DIRECTORY/prereq-test-dir' directory, which is the same for
every prereq, and which is automatically removed after the prereq has
been evaluated.  So if the inner prereq (BAR above) is a lazy prereq
that hasn't been evaluated yet, then after its evaluation the
'prereq-test-dir' shared with the outer prereq will be removed.
Consequently, 'check-foo' will find itself in a non-existing
directory, and won't be able to create/access any files in its cwd,
which could result in an unfulfilled outer prereq.

Luckily, this doesn't affect any of our current nested prereqs, either
because the inner prereq is not a lazy prereq (e.g. MINGW, CYGWIN or
PERL), or because the outer prereq happens to be checked without
touching any paths in its cwd (GPGSM and RFC1991 in 'lib-gpg.sh').

So to prevent nested prereqs from interfering with each other let's
evaluate each prereq in its own dedicated directory by appending the
prereq's name to the directory name, e.g. 'prereq-test-dir-SYMLINKS'.
In the test we check not only that the prereq test dir is still there,
but also that the inner prereq can't mess with the outer prereq's
files.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---

This came up while looking into:

  https://public-inbox.org/git/nycvar.QRO.7.76.6.2011152252520.18437@tvgsbejvaqbjf.bet/

because this did become an issue when I made the JGIT prereq depend on
SHA1.

 t/t0000-basic.sh        | 21 +++++++++++++++++++++
 t/test-lib-functions.sh |  6 +++---
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 923281af93..4031a0e3fc 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -831,6 +831,27 @@ then
 	exit 1
 fi
 
+test_lazy_prereq NESTED_INNER '
+	>inner &&
+	rm -f outer
+'
+test_lazy_prereq NESTED_PREREQ '
+	>outer &&
+	test_have_prereq NESTED_INNER &&
+	echo "can create new file in cwd" >file &&
+	test -f outer &&
+	test ! -f inner
+'
+test_expect_success NESTED_PREREQ 'evaluating nested lazy prereqs dont interfere with each other' '
+	nestedworks=yes
+'
+
+if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" && test "$nestedworks" != yes
+then
+	say 'bug in test framework: nested lazy prerequisites do not work'
+	exit 1
+fi
+
 test_expect_success 'lazy prereqs do not turn off tracing' "
 	run_sub_test_lib_test lazy-prereq-and-tracing \
 		'lazy prereqs and -x' -v -x <<-\\EOF &&
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 8d59b90348..94395b9807 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -474,15 +474,15 @@ test_lazy_prereq () {
 
 test_run_lazy_prereq_ () {
 	script='
-mkdir -p "$TRASH_DIRECTORY/prereq-test-dir" &&
+mkdir -p "$TRASH_DIRECTORY/prereq-test-dir-'"$1"'" &&
 (
-	cd "$TRASH_DIRECTORY/prereq-test-dir" &&'"$2"'
+	cd "$TRASH_DIRECTORY/prereq-test-dir-'"$1"'" &&'"$2"'
 )'
 	say >&3 "checking prerequisite: $1"
 	say >&3 "$script"
 	test_eval_ "$script"
 	eval_ret=$?
-	rm -rf "$TRASH_DIRECTORY/prereq-test-dir"
+	rm -rf "$TRASH_DIRECTORY/prereq-test-dir-$1"
 	if test "$eval_ret" = 0; then
 		say >&3 "prerequisite $1 ok"
 	else
-- 
2.29.2.612.g41c40d3f73

[PATCH 2/2] tests: fix description of 'test_set_prereq'

[SZEDER Gábor]

'test_set_prereq's description claims that prereqs can be specified to
'test_expect_code', but that is not the case (it is not meant to run a
test _case_, but a git command), so remove it.

OTOH that description doesn't mention 'test_external' and
'test_external_without_stderr' that do accept prereqs, so mention
them.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 t/test-lib-functions.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 94395b9807..21b2ea55b7 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -423,7 +423,7 @@ write_script () {
 # - Explicitly using test_have_prereq.
 #
 # - Implicitly by specifying the prerequisite tag in the calls to
-#   test_expect_{success,failure,code}.
+#   test_expect_{success,failure} and test_external{,_without_stderr}.
 #
 # The single parameter is the prerequisite tag (a simple word, in all
 # capital letters by convention).
-- 
2.29.2.612.g41c40d3f73

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> This can be problematic, because lazy prereqs are evaluated in the
> '$TRASH_DIRECTORY/prereq-test-dir' directory, which is the same for
> every prereq, and which is automatically removed after the prereq has
> been evaluated.  So if the inner prereq (BAR above) is a lazy prereq
> that hasn't been evaluated yet, then after its evaluation the
> 'prereq-test-dir' shared with the outer prereq will be removed.

OK.

> Consequently, 'check-foo' will find itself in a non-existing
> directory, and won't be able to create/access any files in its cwd,
> which could result in an unfulfilled outer prereq.

I have a feeling that this would be filesystem dependent what
happens in an unlinked directory.  But in any case, if an outer lazy
prereq creates a file in the test directory, tries to evaluate
another lazy prereq (which would use the same directory and then
would remove everything in it and the directory itself when done),
and then expects that the file it created before evaluating the
inner lazy prereq is still there, it would not work, so I think this
is a good change.  I just think "won't be able to" is a bit too
strong here ("may not be able to (depending on the filesystem)", I
would buy).

> +test_lazy_prereq NESTED_INNER '
> +	>inner &&
> +	rm -f outer
> +'
> +test_lazy_prereq NESTED_PREREQ '
> +	>outer &&
> +	test_have_prereq NESTED_INNER &&
> +	echo "can create new file in cwd" >file &&
> +	test -f outer &&
> +	test ! -f inner
> +'

And the existence check for outer is exactly what I wrote above.
That would portably break without separte directory.  I do not know
if "can create new file" step would fail portably.

> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
> index 8d59b90348..94395b9807 100644
> --- a/t/test-lib-functions.sh
> +++ b/t/test-lib-functions.sh
> @@ -474,15 +474,15 @@ test_lazy_prereq () {
>  
>  test_run_lazy_prereq_ () {
>  	script='
> -mkdir -p "$TRASH_DIRECTORY/prereq-test-dir" &&
> +mkdir -p "$TRASH_DIRECTORY/prereq-test-dir-'"$1"'" &&
>  (
> -	cd "$TRASH_DIRECTORY/prereq-test-dir" &&'"$2"'
> +	cd "$TRASH_DIRECTORY/prereq-test-dir-'"$1"'" &&'"$2"'
>  )'

Qoting rules are a bit tricky here, but I think it does not matter
too much, as $1 (name or prereq) won't have $IFS or "'" or anything
funny in it.

>  	say >&3 "checking prerequisite: $1"
>  	say >&3 "$script"
>  	test_eval_ "$script"
>  	eval_ret=$?
> -	rm -rf "$TRASH_DIRECTORY/prereq-test-dir"
> +	rm -rf "$TRASH_DIRECTORY/prereq-test-dir-$1"

And this is obviously safe no matter what is in $1 ;-)

>  	if test "$eval_ret" = 0; then
>  		say >&3 "prerequisite $1 ok"
>  	else

Looks good.

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Jeff King]

On Wed, Nov 18, 2020 at 08:04:13PM +0100, SZEDER Gábor wrote:

> So to prevent nested prereqs from interfering with each other let's
> evaluate each prereq in its own dedicated directory by appending the
> prereq's name to the directory name, e.g. 'prereq-test-dir-SYMLINKS'.
> In the test we check not only that the prereq test dir is still there,
> but also that the inner prereq can't mess with the outer prereq's
> files.

That sounds reasonable. I do wonder, though, whether simply creating the
prereq directory in the _current_ directory would be sufficient. Then
you'd get prereq-test-dir/prereq-test-dir for a nested invocation. But
the prereqs aren't supposed to care about which specific directory
they're in.

I.e., your test passes equally well with just:

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 59bbf75e83..f5dc6801d9 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -474,15 +474,15 @@ test_lazy_prereq () {
 
 test_run_lazy_prereq_ () {
 	script='
-mkdir -p "$TRASH_DIRECTORY/prereq-test-dir" &&
+mkdir -p "prereq-test-dir" &&
 (
-	cd "$TRASH_DIRECTORY/prereq-test-dir" &&'"$2"'
+	cd "prereq-test-dir" &&'"$2"'
 )'
 	say >&3 "checking prerequisite: $1"
 	say >&3 "$script"
 	test_eval_ "$script"
 	eval_ret=$?
-	rm -rf "$TRASH_DIRECTORY/prereq-test-dir"
+	rm -rf "prereq-test-dir"
 	if test "$eval_ret" = 0; then
 		say >&3 "prerequisite $1 ok"
 	else

though I guess it is not really much simpler (it avoids the funky
quoting around $1 in the embedded script, but we already have that for
$2). And perhaps debugging is easier with a more predictable and
descriptive directory name.

> +test_lazy_prereq NESTED_INNER '
> +	>inner &&
> +	rm -f outer
> +'
> +test_lazy_prereq NESTED_PREREQ '
> +	>outer &&
> +	test_have_prereq NESTED_INNER &&
> +	echo "can create new file in cwd" >file &&
> +	test -f outer &&
> +	test ! -f inner
> +'
> +test_expect_success NESTED_PREREQ 'evaluating nested lazy prereqs dont interfere with each other' '
> +	nestedworks=yes
> +'
> +
> +if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" && test "$nestedworks" != yes
> +then
> +	say 'bug in test framework: nested lazy prerequisites do not work'
> +	exit 1
> +fi

I was surprised to see this bare exit, because I know we have some
functions (run_sub_test_*) to help with testing the framework itself. It
looks like the other prereq tests don't use it either, though. I wonder
if there is a technical reason, or if they were simply added at a
different time. (Either way, I am OK for your new test to match the
surrounding ones like you have here).

-Peff

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Jeff King]

On Thu, Nov 19, 2020 at 10:58:24AM -0500, Jeff King wrote:

> > +if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" && test "$nestedworks" != yes
> > +then
> > +	say 'bug in test framework: nested lazy prerequisites do not work'
> > +	exit 1
> > +fi
> 
> I was surprised to see this bare exit, because I know we have some
> functions (run_sub_test_*) to help with testing the framework itself. It
> looks like the other prereq tests don't use it either, though. I wonder
> if there is a technical reason, or if they were simply added at a
> different time. (Either way, I am OK for your new test to match the
> surrounding ones like you have here).

I took a look at converting some of the existing tests. This seems to
work. It's a bit longer to read, perhaps, but I kind of like that the
expected outcome is all laid out. It also pollutes the test output less
(e.g., if you wanted to count up skipped tests in the whole suite, you'd
get a bunch of noise from t0000 for these uninteresting skips).

Thoughts? I think this is something I'd do on top of your patch.

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index f4ba2e8c85..f369af76be 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -759,43 +759,50 @@ test_expect_success '--run invalid range end' "
 	EOF_ERR
 "
 
-
-test_set_prereq HAVEIT
-haveit=no
-test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '
-	test_have_prereq HAVEIT &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success DONTHAVEIT 'unmet prerequisite causes test to be skipped' '
-	donthaveit=no
-'
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit != yesyes
-then
-	say "bug in test framework: prerequisite tags do not work reliably"
-	exit 1
-fi
-
-test_set_prereq HAVETHIS
-haveit=no
-test_expect_success HAVETHIS,HAVEIT 'test runs if prerequisites are satisfied' '
-	test_have_prereq HAVEIT &&
-	test_have_prereq HAVETHIS &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success HAVEIT,DONTHAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveit=no
-'
-donthaveiteither=yes
-test_expect_success DONTHAVEIT,HAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveiteither=no
+test_expect_success 'tests respect prerequisites' '
+	run_sub_test_lib_test prereqs "tests respect prereqs" <<-\EOF &&
+
+	test_set_prereq HAVEIT
+	haveit=no
+	test_expect_success HAVEIT "prereq is satisfied" "
+		test_have_prereq HAVEIT &&
+		haveit=yes
+	"
+
+	donthaveit=yes
+	test_expect_success DONTHAVEIT "prereq not satisfied" "
+		donthaveit=no
+	"
+
+	test_set_prereq HAVETHIS
+	haveit=no
+	test_expect_success HAVETHIS,HAVEIT "multiple prereqs" "
+		test_have_prereq HAVEIT &&
+		test_have_prereq HAVETHIS &&
+		haveit=yes
+	"
+
+	donthaveit=yes
+	test_expect_success HAVEIT,DONTHAVEIT "mixed prereqs (yes,no)" "
+		donthaveit=no
+	"
+
+	donthaveiteither=yes
+	test_expect_success DONTHAVEIT,HAVEIT "mixed prereqs (no,yes)" "
+		donthaveiteither=no
+	"
+	test_done
+	EOF
+	check_sub_test_lib_test prereqs <<-\EOF
+	ok 1 - prereq is satisfied
+	ok 2 # skip prereq not satisfied (missing DONTHAVEIT)
+	ok 3 - multiple prereqs
+	ok 4 # skip mixed prereqs (yes,no) (missing DONTHAVEIT of HAVEIT,DONTHAVEIT)
+	ok 5 # skip mixed prereqs (no,yes) (missing DONTHAVEIT of DONTHAVEIT,HAVEIT)
+	# passed all 5 test(s)
+	1..5
+	EOF
 '
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit$donthaveiteither != yesyesyes
-then
-	say "bug in test framework: multiple prerequisite tags do not work reliably"
-	exit 1
-fi
 
 test_lazy_prereq LAZY_TRUE true
 havetrue=no

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I took a look at converting some of the existing tests. This seems to
> work. It's a bit longer to read, perhaps, but I kind of like that the
> expected outcome is all laid out. It also pollutes the test output less
> (e.g., if you wanted to count up skipped tests in the whole suite, you'd
> get a bunch of noise from t0000 for these uninteresting skips).
>
> Thoughts? I think this is something I'd do on top of your patch.

Yes, it looks nice as the expectation is expressed much clearly.

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Jeff King]

On Thu, Nov 19, 2020 at 11:50:26AM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > I took a look at converting some of the existing tests. This seems to
> > work. It's a bit longer to read, perhaps, but I kind of like that the
> > expected outcome is all laid out. It also pollutes the test output less
> > (e.g., if you wanted to count up skipped tests in the whole suite, you'd
> > get a bunch of noise from t0000 for these uninteresting skips).
> >
> > Thoughts? I think this is something I'd do on top of your patch.
> 
> Yes, it looks nice as the expectation is expressed much clearly.

OK, then here's the whole thing. I ended up with a few more cleanups,
too. This is all on top of Gábor's patches. It's conceptually
independent, but the textual wrangling was annoying enough it didn't
make any sense to require you to do it again during merging. ;) Plus I
do not think either topic is high-risk nor urgent enough to worry too
much about one blocking the other.

The diffstat is scary, but it's mostly the final patch, which is pretty
mechanical.

  [1/4]: t0000: keep clean-up tests together
  [2/4]: t0000: run prereq tests inside sub-test
  [3/4]: t0000: run cleaning test inside sub-test
  [4/4]: t0000: consistently use single quotes for outer tests

 t/t0000-basic.sh | 570 +++++++++++++++++++++++------------------------
 1 file changed, 284 insertions(+), 286 deletions(-)

-Peff

[PATCH 1/4] t0000: keep clean-up tests together

[Jeff King]

We check that test_when_finished cleans up after a test, and that it
runs even after a failure. Those two were originally adjacent, but got
split apart by the new test added in 477dcaddb6 (tests: do not let lazy
prereqs inside `test_expect_*` turn off tracing, 2020-03-26), and then
further by more lazy-prereq tests. Let's move them back together.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t0000-basic.sh | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index f4ba2e8c85..d49b5dd4ac 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -829,17 +829,6 @@ then
 	exit 1
 fi
 
-clean=no
-test_expect_success 'tests clean up after themselves' '
-	test_when_finished clean=yes
-'
-
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $clean != yes
-then
-	say "bug in test framework: basic cleanup command does not work reliably"
-	exit 1
-fi
-
 test_lazy_prereq NESTED_INNER '
 	>inner &&
 	rm -f outer
@@ -874,6 +863,17 @@ test_expect_success 'lazy prereqs do not turn off tracing' "
 	grep 'echo trace' lazy-prereq-and-tracing/err
 "
 
+clean=no
+test_expect_success 'tests clean up after themselves' '
+	test_when_finished clean=yes
+'
+
+if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $clean != yes
+then
+	say "bug in test framework: basic cleanup command does not work reliably"
+	exit 1
+fi
+
 test_expect_success 'tests clean up even on failures' "
 	run_sub_test_lib_test_err \
 		failing-cleanup 'Failing tests with cleanup commands' <<-\\EOF &&
-- 
2.29.2.730.g3e418f96ba

[PATCH 2/4] t0000: run prereq tests inside sub-test

[Jeff King]

We test the behavior of prerequisites in t0000 by setting up fake ones
in the main test script, trying to run some tests, and then seeing if
those tests impacted the environment correctly. If they didn't, then we
write a message and manually call exit.

Instead, let's push these down into a sub-test, like many of the other
tests covering the framework itself. This has a few advantages:

  - it does not pollute the test output with mention of skipped tests
    (that we know are uninteresting -- the point of the test was to see
    that these are skipped).

  - when running in a TAP harness, we get a useful test failure message
    (whereas when the script exits early, a tool like "prove" simply
    says "Dubious, test returned 1").

  - we do not have to worry about different test environments, such as
    when GIT_TEST_FAIL_PREREQS_INTERNAL is set. Our sub-test helpers
    already give us a known environment.

  - the tests themselves are a bit easier to read, as we can just check
    the test-framework output to see what happened (and get the usual
    test_cmp diff if it failed)

A few notes on the implementation:

  - we could do one sub-test per each individual test_expect_success. I
    broke it up here into a few logical groups, as I think this makes it
    more readable

  - the original tests modified environment variables inside the test
    bodies. Instead, I've used "true" as the body of a test we expect to
    run and "false" otherwise. Technically this does not confirm that
    the body of the "true" test actually ran. We are trusting the
    framework output to believe that it truly ran, which is sufficient
    for these tests. And I think the end result is much simpler to
    follow.

  - the nested_prereq test uses a few bare "test -f" calls; I converted
    these to our usual test_path_is_* helpers while moving the code
    around.

Signed-off-by: Jeff King <peff@peff.net>
---
The diff is pretty gnarly. I recommend just reading the before/after
sections (I wish there was a way to convince diff _not_ to generate the
minimal diff; two big blocks would have been more readable).

 t/t0000-basic.sh | 149 ++++++++++++++++++++++-------------------------
 1 file changed, 69 insertions(+), 80 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index d49b5dd4ac..3d36a87610 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -759,96 +759,85 @@ test_expect_success '--run invalid range end' "
 	EOF_ERR
 "
 
+test_expect_success 'tests respect prerequisites' '
+	run_sub_test_lib_test prereqs "tests respect prereqs" <<-\EOF &&
 
-test_set_prereq HAVEIT
-haveit=no
-test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '
-	test_have_prereq HAVEIT &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success DONTHAVEIT 'unmet prerequisite causes test to be skipped' '
-	donthaveit=no
-'
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit != yesyes
-then
-	say "bug in test framework: prerequisite tags do not work reliably"
-	exit 1
-fi
+	test_set_prereq HAVEIT
+	test_expect_success HAVEIT "prereq is satisfied" "true"
+	test_expect_success "have_prereq works" "
+		test_have_prereq HAVEIT
+	"
+	test_expect_success DONTHAVEIT "prereq not satisfied" "false"
 
-test_set_prereq HAVETHIS
-haveit=no
-test_expect_success HAVETHIS,HAVEIT 'test runs if prerequisites are satisfied' '
-	test_have_prereq HAVEIT &&
-	test_have_prereq HAVETHIS &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success HAVEIT,DONTHAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveit=no
-'
-donthaveiteither=yes
-test_expect_success DONTHAVEIT,HAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveiteither=no
-'
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit$donthaveiteither != yesyesyes
-then
-	say "bug in test framework: multiple prerequisite tags do not work reliably"
-	exit 1
-fi
+	test_set_prereq HAVETHIS
+	test_expect_success HAVETHIS,HAVEIT "multiple prereqs" "true"
+	test_expect_success HAVEIT,DONTHAVEIT "mixed prereqs (yes,no)" "false"
+	test_expect_success DONTHAVEIT,HAVEIT "mixed prereqs (no,yes)" "false"
 
-test_lazy_prereq LAZY_TRUE true
-havetrue=no
-test_expect_success LAZY_TRUE 'test runs if lazy prereq is satisfied' '
-	havetrue=yes
-'
-donthavetrue=yes
-test_expect_success !LAZY_TRUE 'missing lazy prereqs skip tests' '
-	donthavetrue=no
+	test_done
+	EOF
+
+	check_sub_test_lib_test prereqs <<-\EOF
+	ok 1 - prereq is satisfied
+	ok 2 - have_prereq works
+	ok 3 # skip prereq not satisfied (missing DONTHAVEIT)
+	ok 4 - multiple prereqs
+	ok 5 # skip mixed prereqs (yes,no) (missing DONTHAVEIT of HAVEIT,DONTHAVEIT)
+	ok 6 # skip mixed prereqs (no,yes) (missing DONTHAVEIT of DONTHAVEIT,HAVEIT)
+	# passed all 6 test(s)
+	1..6
+	EOF
 '
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a "$havetrue$donthavetrue" != yesyes
-then
-	say 'bug in test framework: lazy prerequisites do not work'
-	exit 1
-fi
+test_expect_success 'tests respect lazy prerequisites' '
+	run_sub_test_lib_test lazy-prereqs "respect lazy prereqs" <<-\EOF &&
 
-test_lazy_prereq LAZY_FALSE false
-nothavefalse=no
-test_expect_success !LAZY_FALSE 'negative lazy prereqs checked' '
-	nothavefalse=yes
-'
-havefalse=yes
-test_expect_success LAZY_FALSE 'missing negative lazy prereqs will skip' '
-	havefalse=no
-'
+	test_lazy_prereq LAZY_TRUE true
+	test_expect_success LAZY_TRUE "lazy prereq is satisifed" "true"
+	test_expect_success !LAZY_TRUE "negative lazy prereq" "false"
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a "$nothavefalse$havefalse" != yesyes
-then
-	say 'bug in test framework: negative lazy prerequisites do not work'
-	exit 1
-fi
+	test_lazy_prereq LAZY_FALSE false
+	test_expect_success LAZY_FALSE "lazy prereq not satisfied" "false"
+	test_expect_success !LAZY_FALSE "negative false prereq" "true"
 
-test_lazy_prereq NESTED_INNER '
-	>inner &&
-	rm -f outer
-'
-test_lazy_prereq NESTED_PREREQ '
-	>outer &&
-	test_have_prereq NESTED_INNER &&
-	echo "can create new file in cwd" >file &&
-	test -f outer &&
-	test ! -f inner
-'
-test_expect_success NESTED_PREREQ 'evaluating nested lazy prereqs dont interfere with each other' '
-	nestedworks=yes
+	test_done
+	EOF
+
+	check_sub_test_lib_test lazy-prereqs <<-\EOF
+	ok 1 - lazy prereq is satisifed
+	ok 2 # skip negative lazy prereq (missing !LAZY_TRUE)
+	ok 3 # skip lazy prereq not satisfied (missing LAZY_FALSE)
+	ok 4 - negative false prereq
+	# passed all 4 test(s)
+	1..4
+	EOF
 '
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" && test "$nestedworks" != yes
-then
-	say 'bug in test framework: nested lazy prerequisites do not work'
-	exit 1
-fi
+test_expect_success 'nested lazy prerequisites' '
+	run_sub_test_lib_test nested-lazy "nested lazy prereqs" <<-\EOF &&
+
+	test_lazy_prereq NESTED_INNER "
+		>inner &&
+		rm -f outer
+	"
+	test_lazy_prereq NESTED_PREREQ "
+		>outer &&
+		test_have_prereq NESTED_INNER &&
+		echo can create new file in cwd >file &&
+		test_path_is_file outer &&
+		test_path_is_missing inner
+	"
+	test_expect_success NESTED_PREREQ "evaluate nested prereq" "true"
+
+	test_done
+	EOF
+
+	check_sub_test_lib_test nested-lazy <<-\EOF
+	ok 1 - evaluate nested prereq
+	# passed all 1 test(s)
+	1..1
+	EOF
+'
 
 test_expect_success 'lazy prereqs do not turn off tracing' "
 	run_sub_test_lib_test lazy-prereq-and-tracing \
-- 
2.29.2.730.g3e418f96ba

[PATCH 3/4] t0000: run cleaning test inside sub-test

[Jeff King]

Our check of test_when_finished is done directly in the main script, and
if we failed to clean, we complain and exit immediately. It's nicer to
signal a test failure here, for a few reasons:

  - this gives better output to the user when run under a TAP harness
    like "prove"

  - constency; it's the only test left in the file that behaves this way

  - half of its "if" conditional is nonsense anyway; it picked up a
    reference to GIT_TEST_FAIL_PREREQS internal in dfe1a17df9 (tests:
    add a special setup where prerequisites fail, 2019-05-13) along with
    its neighbors, even though it has nothing to do with that flag

We could actually do this without a sub-test at all, and just put our
two tests (one to do cleanup, and one to check that it happened) in the
main script. But doing it in a subtest is conceptually cleaner (from the
perspective of the main test script, we are checking only one thing),
and it remains consistent with the "cleanup when failing" test directly
after it, which has to happen in a sub-test (to avoid the main script
complaining of the failed test).

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t0000-basic.sh | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 3d36a87610..502375bdf6 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -852,16 +852,25 @@ test_expect_success 'lazy prereqs do not turn off tracing' "
 	grep 'echo trace' lazy-prereq-and-tracing/err
 "
 
-clean=no
 test_expect_success 'tests clean up after themselves' '
-	test_when_finished clean=yes
-'
+	run_sub_test_lib_test cleanup "test with cleanup" <<-\EOF &&
+	clean=no
+	test_expect_success "do cleanup" "
+		test_when_finished clean=yes
+	"
+	test_expect_success "cleanup happened" "
+		test $clean = yes
+	"
+	test_done
+	EOF
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $clean != yes
-then
-	say "bug in test framework: basic cleanup command does not work reliably"
-	exit 1
-fi
+	check_sub_test_lib_test cleanup <<-\EOF
+	ok 1 - do cleanup
+	ok 2 - cleanup happened
+	# passed all 2 test(s)
+	1..2
+	EOF
+'
 
 test_expect_success 'tests clean up even on failures' "
 	run_sub_test_lib_test_err \
-- 
2.29.2.730.g3e418f96ba

[PATCH 4/4] t0000: consistently use single quotes for outer tests

[Jeff King]

When we use the sub-test helpers, we end defining one shell snippet
inside another shell snippet. So if we use single-quotes for the outer
snippet, we have to use double-quotes within the inner snippet (it's
included as here-doc within the outer snippet, but using a single quote
would end the outer snippet early). Or vice versa we can use double
quotes for the outer snippet, but then single quotes in the inner.

We have some of each in the script, and neither is wrong. But it would
be nice to be consistent unless there is a good reason not to. Using
single quotes for the outer script is preferable, because it requires
less metacharacter quoting overall. For example, in:

  test_expect_success 'outer' '
	run_sub_test_lib_test ...  <<-\EOF
		echo $foo &&
		test_expect_success "inner" "
			echo \$bar
		"
	EOF
  '

we need only quote inside "inner", but not inside "outer" or the
here-doc. Whereas if we flip them, we have to quote in both places:

  test_expect_success 'outer' "
	run_sub_test_lib_test ...  <<-\EOF
		echo \$foo &&
		test_expect_success 'inner' '
			echo \$bar
		'
	EOF
  "

The exception is when we need a literal single-quote in an expected
output here-doc. There we can either use outer double-quotes, or just
use ${SQ} within the doc. I chose the latter for consistency (within
this test, but also with other test scripts that face the same problem).

There is one other interesting case, which is some tests that do:

  test_expect_success ... "
	do_something --run='"'!3'"'
  "

This is rather confusing to read, but is correct. The outer script sees
'!3' in single-quotes, as does the eval'd snippet. This is perhaps being
overly cautious. In many interactive shells, an exclamation triggers
history expansion even inside double quotes, but that is not generally
true in non-interactive shells.

There's some conflicting information here. Commit 784ce03d55 (t4216:
avoid unnecessary subshell in test_bloom_filters_not_used, 2020-05-19)
reports it as a problem with OpenBSD 6.7's /bin/sh. However, we have
many instances in this script of prereqs like !LAZY_TRUE, which haven't
been a problem. I left them un-escaped here to test out this theory.
It's much nicer if we can not worry about this as a portability issue,
so it's worth knowing.

Signed-off-by: Jeff King <peff@peff.net>
---
I have often thought that test_expect_* should take the snippet on stdin
via a here-doc, which removes one layer of quoting and makes this all
even simpler. We could obviously do that with a helper here, but I
didn't want to stray too far from usual practice within the rest of the
test suite.

 t/t0000-basic.sh | 380 +++++++++++++++++++++++------------------------
 1 file changed, 190 insertions(+), 190 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 502375bdf6..a6e570d674 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -135,77 +135,77 @@ check_sub_test_lib_test_err () {
 	)
 }
 
-test_expect_success 'pretend we have a fully passing test suite' "
-	run_sub_test_lib_test full-pass '3 passing tests' <<-\\EOF &&
+test_expect_success 'pretend we have a fully passing test suite' '
+	run_sub_test_lib_test full-pass "3 passing tests" <<-\EOF &&
 	for i in 1 2 3
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test full-pass <<-\\EOF
+	check_sub_test_lib_test full-pass <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 - passing test #3
 	> # passed all 3 test(s)
 	> 1..3
 	EOF
-"
+'
 
-test_expect_success 'pretend we have a partially passing test suite' "
+test_expect_success 'pretend we have a partially passing test suite' '
 	run_sub_test_lib_test_err \
-		partial-pass '2/3 tests passing' <<-\\EOF &&
-	test_expect_success 'passing test #1' 'true'
-	test_expect_success 'failing test #2' 'false'
-	test_expect_success 'passing test #3' 'true'
+		partial-pass "2/3 tests passing" <<-\EOF &&
+	test_expect_success "passing test #1" "true"
+	test_expect_success "failing test #2" "false"
+	test_expect_success "passing test #3" "true"
 	test_done
 	EOF
-	check_sub_test_lib_test partial-pass <<-\\EOF
+	check_sub_test_lib_test partial-pass <<-\EOF
 	> ok 1 - passing test #1
 	> not ok 2 - failing test #2
 	#	false
 	> ok 3 - passing test #3
 	> # failed 1 among 3 test(s)
 	> 1..3
 	EOF
-"
+'
 
-test_expect_success 'pretend we have a known breakage' "
-	run_sub_test_lib_test failing-todo 'A failing TODO test' <<-\\EOF &&
-	test_expect_success 'passing test' 'true'
-	test_expect_failure 'pretend we have a known breakage' 'false'
+test_expect_success 'pretend we have a known breakage' '
+	run_sub_test_lib_test failing-todo "A failing TODO test" <<-\EOF &&
+	test_expect_success "passing test" "true"
+	test_expect_failure "pretend we have a known breakage" "false"
 	test_done
 	EOF
-	check_sub_test_lib_test failing-todo <<-\\EOF
+	check_sub_test_lib_test failing-todo <<-\EOF
 	> ok 1 - passing test
 	> not ok 2 - pretend we have a known breakage # TODO known breakage
 	> # still have 1 known breakage(s)
 	> # passed all remaining 1 test(s)
 	> 1..2
 	EOF
-"
+'
 
-test_expect_success 'pretend we have fixed a known breakage' "
-	run_sub_test_lib_test passing-todo 'A passing TODO test' <<-\\EOF &&
-	test_expect_failure 'pretend we have fixed a known breakage' 'true'
+test_expect_success 'pretend we have fixed a known breakage' '
+	run_sub_test_lib_test passing-todo "A passing TODO test" <<-\EOF &&
+	test_expect_failure "pretend we have fixed a known breakage" "true"
 	test_done
 	EOF
-	check_sub_test_lib_test passing-todo <<-\\EOF
+	check_sub_test_lib_test passing-todo <<-\EOF
 	> ok 1 - pretend we have fixed a known breakage # TODO known breakage vanished
 	> # 1 known breakage(s) vanished; please update test(s)
 	> 1..1
 	EOF
-"
+'
 
-test_expect_success 'pretend we have fixed one of two known breakages (run in sub test-lib)' "
+test_expect_success 'pretend we have fixed one of two known breakages (run in sub test-lib)' '
 	run_sub_test_lib_test partially-passing-todos \
-		'2 TODO tests, one passing' <<-\\EOF &&
-	test_expect_failure 'pretend we have a known breakage' 'false'
-	test_expect_success 'pretend we have a passing test' 'true'
-	test_expect_failure 'pretend we have fixed another known breakage' 'true'
+		"2 TODO tests, one passing" <<-\EOF &&
+	test_expect_failure "pretend we have a known breakage" "false"
+	test_expect_success "pretend we have a passing test" "true"
+	test_expect_failure "pretend we have fixed another known breakage" "true"
 	test_done
 	EOF
-	check_sub_test_lib_test partially-passing-todos <<-\\EOF
+	check_sub_test_lib_test partially-passing-todos <<-\EOF
 	> not ok 1 - pretend we have a known breakage # TODO known breakage
 	> ok 2 - pretend we have a passing test
 	> ok 3 - pretend we have fixed another known breakage # TODO known breakage vanished
@@ -214,17 +214,17 @@ test_expect_success 'pretend we have fixed one of two known breakages (run in su
 	> # passed all remaining 1 test(s)
 	> 1..3
 	EOF
-"
+'
 
-test_expect_success 'pretend we have a pass, fail, and known breakage' "
+test_expect_success 'pretend we have a pass, fail, and known breakage' '
 	run_sub_test_lib_test_err \
-		mixed-results1 'mixed results #1' <<-\\EOF &&
-	test_expect_success 'passing test' 'true'
-	test_expect_success 'failing test' 'false'
-	test_expect_failure 'pretend we have a known breakage' 'false'
+		mixed-results1 "mixed results #1" <<-\EOF &&
+	test_expect_success "passing test" "true"
+	test_expect_success "failing test" "false"
+	test_expect_failure "pretend we have a known breakage" "false"
 	test_done
 	EOF
-	check_sub_test_lib_test mixed-results1 <<-\\EOF
+	check_sub_test_lib_test mixed-results1 <<-\EOF
 	> ok 1 - passing test
 	> not ok 2 - failing test
 	> #	false
@@ -233,24 +233,24 @@ test_expect_success 'pretend we have a pass, fail, and known breakage' "
 	> # failed 1 among remaining 2 test(s)
 	> 1..3
 	EOF
-"
+'
 
-test_expect_success 'pretend we have a mix of all possible results' "
+test_expect_success 'pretend we have a mix of all possible results' '
 	run_sub_test_lib_test_err \
-		mixed-results2 'mixed results #2' <<-\\EOF &&
-	test_expect_success 'passing test' 'true'
-	test_expect_success 'passing test' 'true'
-	test_expect_success 'passing test' 'true'
-	test_expect_success 'passing test' 'true'
-	test_expect_success 'failing test' 'false'
-	test_expect_success 'failing test' 'false'
-	test_expect_success 'failing test' 'false'
-	test_expect_failure 'pretend we have a known breakage' 'false'
-	test_expect_failure 'pretend we have a known breakage' 'false'
-	test_expect_failure 'pretend we have fixed a known breakage' 'true'
+		mixed-results2 "mixed results #2" <<-\EOF &&
+	test_expect_success "passing test" "true"
+	test_expect_success "passing test" "true"
+	test_expect_success "passing test" "true"
+	test_expect_success "passing test" "true"
+	test_expect_success "failing test" "false"
+	test_expect_success "failing test" "false"
+	test_expect_success "failing test" "false"
+	test_expect_failure "pretend we have a known breakage" "false"
+	test_expect_failure "pretend we have a known breakage" "false"
+	test_expect_failure "pretend we have fixed a known breakage" "true"
 	test_done
 	EOF
-	check_sub_test_lib_test mixed-results2 <<-\\EOF
+	check_sub_test_lib_test mixed-results2 <<-\EOF
 	> ok 1 - passing test
 	> ok 2 - passing test
 	> ok 3 - passing test
@@ -269,7 +269,7 @@ test_expect_success 'pretend we have a mix of all possible results' "
 	> # failed 3 among remaining 7 test(s)
 	> 1..10
 	EOF
-"
+'
 
 test_expect_success C_LOCALE_OUTPUT 'test --verbose' '
 	run_sub_test_lib_test_err \
@@ -321,39 +321,39 @@ test_expect_success 'test --verbose-only' '
 	EOF
 '
 
-test_expect_success 'GIT_SKIP_TESTS' "
+test_expect_success 'GIT_SKIP_TESTS' '
 	(
-		GIT_SKIP_TESTS='git.2' && export GIT_SKIP_TESTS &&
+		GIT_SKIP_TESTS="git.2" && export GIT_SKIP_TESTS &&
 		run_sub_test_lib_test git-skip-tests-basic \
-			'GIT_SKIP_TESTS' <<-\\EOF &&
+			"GIT_SKIP_TESTS" <<-\EOF &&
 		for i in 1 2 3
 		do
-			test_expect_success \"passing test #\$i\" 'true'
+			test_expect_success "passing test #$i" "true"
 		done
 		test_done
 		EOF
-		check_sub_test_lib_test git-skip-tests-basic <<-\\EOF
+		check_sub_test_lib_test git-skip-tests-basic <<-\EOF
 		> ok 1 - passing test #1
 		> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
 		> ok 3 - passing test #3
 		> # passed all 3 test(s)
 		> 1..3
 		EOF
 	)
-"
+'
 
-test_expect_success 'GIT_SKIP_TESTS several tests' "
+test_expect_success 'GIT_SKIP_TESTS several tests' '
 	(
-		GIT_SKIP_TESTS='git.2 git.5' && export GIT_SKIP_TESTS &&
+		GIT_SKIP_TESTS="git.2 git.5" && export GIT_SKIP_TESTS &&
 		run_sub_test_lib_test git-skip-tests-several \
-			'GIT_SKIP_TESTS several tests' <<-\\EOF &&
+			"GIT_SKIP_TESTS several tests" <<-\EOF &&
 		for i in 1 2 3 4 5 6
 		do
-			test_expect_success \"passing test #\$i\" 'true'
+			test_expect_success "passing test #$i" "true"
 		done
 		test_done
 		EOF
-		check_sub_test_lib_test git-skip-tests-several <<-\\EOF
+		check_sub_test_lib_test git-skip-tests-several <<-\EOF
 		> ok 1 - passing test #1
 		> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
 		> ok 3 - passing test #3
@@ -364,20 +364,20 @@ test_expect_success 'GIT_SKIP_TESTS several tests' "
 		> 1..6
 		EOF
 	)
-"
+'
 
-test_expect_success 'GIT_SKIP_TESTS sh pattern' "
+test_expect_success 'GIT_SKIP_TESTS sh pattern' '
 	(
-		GIT_SKIP_TESTS='git.[2-5]' && export GIT_SKIP_TESTS &&
+		GIT_SKIP_TESTS="git.[2-5]" && export GIT_SKIP_TESTS &&
 		run_sub_test_lib_test git-skip-tests-sh-pattern \
-			'GIT_SKIP_TESTS sh pattern' <<-\\EOF &&
+			"GIT_SKIP_TESTS sh pattern" <<-\EOF &&
 		for i in 1 2 3 4 5 6
 		do
-			test_expect_success \"passing test #\$i\" 'true'
+			test_expect_success "passing test #$i" "true"
 		done
 		test_done
 		EOF
-		check_sub_test_lib_test git-skip-tests-sh-pattern <<-\\EOF
+		check_sub_test_lib_test git-skip-tests-sh-pattern <<-\EOF
 		> ok 1 - passing test #1
 		> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
 		> ok 3 # skip passing test #3 (GIT_SKIP_TESTS)
@@ -388,56 +388,56 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' "
 		> 1..6
 		EOF
 	)
-"
+'
 
-test_expect_success 'GIT_SKIP_TESTS entire suite' "
+test_expect_success 'GIT_SKIP_TESTS entire suite' '
 	(
-		GIT_SKIP_TESTS='git' && export GIT_SKIP_TESTS &&
+		GIT_SKIP_TESTS="git" && export GIT_SKIP_TESTS &&
 		run_sub_test_lib_test git-skip-tests-entire-suite \
-			'GIT_SKIP_TESTS entire suite' <<-\\EOF &&
+			"GIT_SKIP_TESTS entire suite" <<-\EOF &&
 		for i in 1 2 3
 		do
-			test_expect_success \"passing test #\$i\" 'true'
+			test_expect_success "passing test #$i" "true"
 		done
 		test_done
 		EOF
-		check_sub_test_lib_test git-skip-tests-entire-suite <<-\\EOF
+		check_sub_test_lib_test git-skip-tests-entire-suite <<-\EOF
 		> 1..0 # SKIP skip all tests in git
 		EOF
 	)
-"
+'
 
-test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
+test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' '
 	(
-		GIT_SKIP_TESTS='notgit' && export GIT_SKIP_TESTS &&
+		GIT_SKIP_TESTS="notgit" && export GIT_SKIP_TESTS &&
 		run_sub_test_lib_test git-skip-tests-unmatched-suite \
-			'GIT_SKIP_TESTS does not skip unmatched suite' <<-\\EOF &&
+			"GIT_SKIP_TESTS does not skip unmatched suite" <<-\EOF &&
 		for i in 1 2 3
 		do
-			test_expect_success \"passing test #\$i\" 'true'
+			test_expect_success "passing test #$i" "true"
 		done
 		test_done
 		EOF
-		check_sub_test_lib_test git-skip-tests-unmatched-suite <<-\\EOF
+		check_sub_test_lib_test git-skip-tests-unmatched-suite <<-\EOF
 		> ok 1 - passing test #1
 		> ok 2 - passing test #2
 		> ok 3 - passing test #3
 		> # passed all 3 test(s)
 		> 1..3
 		EOF
 	)
-"
+'
 
-test_expect_success '--run basic' "
+test_expect_success '--run basic' '
 	run_sub_test_lib_test run-basic \
-		'--run basic' --run='1,3,5' <<-\\EOF &&
+		"--run basic" --run="1,3,5" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-basic <<-\\EOF
+	check_sub_test_lib_test run-basic <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 # skip passing test #2 (--run)
 	> ok 3 - passing test #3
@@ -447,18 +447,18 @@ test_expect_success '--run basic' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with a range' "
+test_expect_success '--run with a range' '
 	run_sub_test_lib_test run-range \
-		'--run with a range' --run='1-3' <<-\\EOF &&
+		"--run with a range" --run="1-3" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-range <<-\\EOF
+	check_sub_test_lib_test run-range <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 - passing test #3
@@ -468,18 +468,18 @@ test_expect_success '--run with a range' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with two ranges' "
+test_expect_success '--run with two ranges' '
 	run_sub_test_lib_test run-two-ranges \
-		'--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
+		"--run with two ranges" --run="1-2,5-6" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-two-ranges <<-\\EOF
+	check_sub_test_lib_test run-two-ranges <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -489,18 +489,18 @@ test_expect_success '--run with two ranges' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with a left open range' "
+test_expect_success '--run with a left open range' '
 	run_sub_test_lib_test run-left-open-range \
-		'--run with a left open range' --run='-3' <<-\\EOF &&
+		"--run with a left open range" --run="-3" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-left-open-range <<-\\EOF
+	check_sub_test_lib_test run-left-open-range <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 - passing test #3
@@ -510,18 +510,18 @@ test_expect_success '--run with a left open range' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with a right open range' "
+test_expect_success '--run with a right open range' '
 	run_sub_test_lib_test run-right-open-range \
-		'--run with a right open range' --run='4-' <<-\\EOF &&
+		"--run with a right open range" --run="4-" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-right-open-range <<-\\EOF
+	check_sub_test_lib_test run-right-open-range <<-\EOF
 	> ok 1 # skip passing test #1 (--run)
 	> ok 2 # skip passing test #2 (--run)
 	> ok 3 # skip passing test #3 (--run)
@@ -531,18 +531,18 @@ test_expect_success '--run with a right open range' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with basic negation' "
+test_expect_success '--run with basic negation' '
 	run_sub_test_lib_test run-basic-neg \
-		'--run with basic negation' --run='"'!3'"' <<-\\EOF &&
+		"--run with basic negation" --run="!3" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-basic-neg <<-\\EOF
+	check_sub_test_lib_test run-basic-neg <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -552,18 +552,18 @@ test_expect_success '--run with basic negation' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run with two negations' "
+test_expect_success '--run with two negations' '
 	run_sub_test_lib_test run-two-neg \
-		'--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
+		"--run with two negations" --run="!3,!6" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-two-neg <<-\\EOF
+	check_sub_test_lib_test run-two-neg <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -573,18 +573,18 @@ test_expect_success '--run with two negations' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run a range and negation' "
+test_expect_success '--run a range and negation' '
 	run_sub_test_lib_test run-range-and-neg \
-		'--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
+		"--run a range and negation" --run="-4,!2" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-range-and-neg <<-\\EOF
+	check_sub_test_lib_test run-range-and-neg <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 # skip passing test #2 (--run)
 	> ok 3 - passing test #3
@@ -594,18 +594,18 @@ test_expect_success '--run a range and negation' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run range negation' "
+test_expect_success '--run range negation' '
 	run_sub_test_lib_test run-range-neg \
-		'--run range negation' --run='"'!1-3'"' <<-\\EOF &&
+		"--run range negation" --run="!1-3" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-range-neg <<-\\EOF
+	check_sub_test_lib_test run-range-neg <<-\EOF
 	> ok 1 # skip passing test #1 (--run)
 	> ok 2 # skip passing test #2 (--run)
 	> ok 3 # skip passing test #3 (--run)
@@ -615,19 +615,19 @@ test_expect_success '--run range negation' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run include, exclude and include' "
+test_expect_success '--run include, exclude and include' '
 	run_sub_test_lib_test run-inc-neg-inc \
-		'--run include, exclude and include' \
-		--run='"'1-5,!1-3,2'"' <<-\\EOF &&
+		"--run include, exclude and include" \
+		--run="1-5,!1-3,2" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-inc-neg-inc <<-\\EOF
+	check_sub_test_lib_test run-inc-neg-inc <<-\EOF
 	> ok 1 # skip passing test #1 (--run)
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -637,19 +637,19 @@ test_expect_success '--run include, exclude and include' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run include, exclude and include, comma separated' "
+test_expect_success '--run include, exclude and include, comma separated' '
 	run_sub_test_lib_test run-inc-neg-inc-comma \
-		'--run include, exclude and include, comma separated' \
-		--run=1-5,\!1-3,2 <<-\\EOF &&
+		"--run include, exclude and include, comma separated" \
+		--run=1-5,!1-3,2 <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-inc-neg-inc-comma <<-\\EOF
+	check_sub_test_lib_test run-inc-neg-inc-comma <<-\EOF
 	> ok 1 # skip passing test #1 (--run)
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -659,19 +659,19 @@ test_expect_success '--run include, exclude and include, comma separated' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run exclude and include' "
+test_expect_success '--run exclude and include' '
 	run_sub_test_lib_test run-neg-inc \
-		'--run exclude and include' \
-		--run='"'!3-,5'"' <<-\\EOF &&
+		"--run exclude and include" \
+		--run="!3-,5" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-neg-inc <<-\\EOF
+	check_sub_test_lib_test run-neg-inc <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 - passing test #2
 	> ok 3 # skip passing test #3 (--run)
@@ -681,19 +681,19 @@ test_expect_success '--run exclude and include' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run empty selectors' "
+test_expect_success '--run empty selectors' '
 	run_sub_test_lib_test run-empty-sel \
-		'--run empty selectors' \
-		--run='1,,3,,,5' <<-\\EOF &&
+		"--run empty selectors" \
+		--run="1,,3,,,5" <<-\EOF &&
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"passing test #\$i\" 'true'
+		test_expect_success "passing test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-empty-sel <<-\\EOF
+	check_sub_test_lib_test run-empty-sel <<-\EOF
 	> ok 1 - passing test #1
 	> ok 2 # skip passing test #2 (--run)
 	> ok 3 - passing test #3
@@ -703,20 +703,20 @@ test_expect_success '--run empty selectors' "
 	> # passed all 6 test(s)
 	> 1..6
 	EOF
-"
+'
 
-test_expect_success '--run substring selector' "
+test_expect_success '--run substring selector' '
 	run_sub_test_lib_test run-substring-selector \
-		'--run empty selectors' \
-		--run='relevant' <<-\\EOF &&
-	test_expect_success \"relevant test\" 'true'
+		"--run empty selectors" \
+		--run="relevant" <<-\EOF &&
+	test_expect_success "relevant test" "true"
 	for i in 1 2 3 4 5 6
 	do
-		test_expect_success \"other test #\$i\" 'true'
+		test_expect_success "other test #$i" "true"
 	done
 	test_done
 	EOF
-	check_sub_test_lib_test run-substring-selector <<-\\EOF
+	check_sub_test_lib_test run-substring-selector <<-\EOF
 	> ok 1 - relevant test
 	> ok 2 # skip other test #1 (--run)
 	> ok 3 # skip other test #2 (--run)
@@ -727,37 +727,37 @@ test_expect_success '--run substring selector' "
 	> # passed all 7 test(s)
 	> 1..7
 	EOF
-"
+'
 
-test_expect_success '--run keyword selection' "
+test_expect_success '--run keyword selection' '
 	run_sub_test_lib_test_err run-inv-range-start \
-		'--run invalid range start' \
-		--run='a-5' <<-\\EOF &&
-	test_expect_success \"passing test #1\" 'true'
+		"--run invalid range start" \
+		--run="a-5" <<-\EOF &&
+	test_expect_success "passing test #1" "true"
 	test_done
 	EOF
 	check_sub_test_lib_test_err run-inv-range-start \
-		<<-\\EOF_OUT 3<<-\\EOF_ERR
+		<<-\EOF_OUT 3<<-EOF_ERR
 	> FATAL: Unexpected exit with code 1
 	EOF_OUT
-	> error: --run: invalid non-numeric in range start: 'a-5'
+	> error: --run: invalid non-numeric in range start: ${SQ}a-5${SQ}
 	EOF_ERR
-"
+'
 
-test_expect_success '--run invalid range end' "
+test_expect_success '--run invalid range end' '
 	run_sub_test_lib_test_err run-inv-range-end \
-		'--run invalid range end' \
-		--run='1-z' <<-\\EOF &&
-	test_expect_success \"passing test #1\" 'true'
+		"--run invalid range end" \
+		--run="1-z" <<-\EOF &&
+	test_expect_success "passing test #1" "true"
 	test_done
 	EOF
 	check_sub_test_lib_test_err run-inv-range-end \
-		<<-\\EOF_OUT 3<<-\\EOF_ERR
+		<<-\EOF_OUT 3<<-EOF_ERR
 	> FATAL: Unexpected exit with code 1
 	EOF_OUT
-	> error: --run: invalid non-numeric in range end: '1-z'
+	> error: --run: invalid non-numeric in range end: ${SQ}1-z${SQ}
 	EOF_ERR
-"
+'
 
 test_expect_success 'tests respect prerequisites' '
 	run_sub_test_lib_test prereqs "tests respect prereqs" <<-\EOF &&
@@ -839,18 +839,18 @@ test_expect_success 'nested lazy prerequisites' '
 	EOF
 '
 
-test_expect_success 'lazy prereqs do not turn off tracing' "
+test_expect_success 'lazy prereqs do not turn off tracing' '
 	run_sub_test_lib_test lazy-prereq-and-tracing \
-		'lazy prereqs and -x' -v -x <<-\\EOF &&
+		"lazy prereqs and -x" -v -x <<-\EOF &&
 	test_lazy_prereq LAZY true
 
-	test_expect_success lazy 'test_have_prereq LAZY && echo trace'
+	test_expect_success lazy "test_have_prereq LAZY && echo trace"
 
 	test_done
 	EOF
 
-	grep 'echo trace' lazy-prereq-and-tracing/err
-"
+	grep "echo trace" lazy-prereq-and-tracing/err
+'
 
 test_expect_success 'tests clean up after themselves' '
 	run_sub_test_lib_test cleanup "test with cleanup" <<-\EOF &&
@@ -872,20 +872,20 @@ test_expect_success 'tests clean up after themselves' '
 	EOF
 '
 
-test_expect_success 'tests clean up even on failures' "
+test_expect_success 'tests clean up even on failures' '
 	run_sub_test_lib_test_err \
-		failing-cleanup 'Failing tests with cleanup commands' <<-\\EOF &&
-	test_expect_success 'tests clean up even after a failure' '
+		failing-cleanup "Failing tests with cleanup commands" <<-\EOF &&
+	test_expect_success "tests clean up even after a failure" "
 		touch clean-after-failure &&
 		test_when_finished rm clean-after-failure &&
 		(exit 1)
-	'
-	test_expect_success 'failure to clean up causes the test to fail' '
+	"
+	test_expect_success "failure to clean up causes the test to fail" "
 		test_when_finished \"(exit 2)\"
-	'
+	"
 	test_done
 	EOF
-	check_sub_test_lib_test failing-cleanup <<-\\EOF
+	check_sub_test_lib_test failing-cleanup <<-\EOF
 	> not ok 1 - tests clean up even after a failure
 	> #	Z
 	> #	touch clean-after-failure &&
@@ -894,30 +894,30 @@ test_expect_success 'tests clean up even on failures' "
 	> #	Z
 	> not ok 2 - failure to clean up causes the test to fail
 	> #	Z
-	> #	test_when_finished \"(exit 2)\"
+	> #	test_when_finished "(exit 2)"
 	> #	Z
 	> # failed 2 among 2 test(s)
 	> 1..2
 	EOF
-"
+'
 
-test_expect_success 'test_atexit is run' "
+test_expect_success 'test_atexit is run' '
 	run_sub_test_lib_test_err \
-		atexit-cleanup 'Run atexit commands' -i <<-\\EOF &&
-	test_expect_success 'tests clean up even after a failure' '
+		atexit-cleanup "Run atexit commands" -i <<-\EOF &&
+	test_expect_success "tests clean up even after a failure" "
 		> ../../clean-atexit &&
 		test_atexit rm ../../clean-atexit &&
 		> ../../also-clean-atexit &&
 		test_atexit rm ../../also-clean-atexit &&
 		> ../../dont-clean-atexit &&
 		(exit 1)
-	'
+	"
 	test_done
 	EOF
 	test_path_is_file dont-clean-atexit &&
 	test_path_is_missing clean-atexit &&
 	test_path_is_missing also-clean-atexit
-"
+'
 
 test_expect_success 'test_oid provides sane info by default' '
 	test_oid zero >actual &&
-- 
2.29.2.730.g3e418f96ba

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> OK, then here's the whole thing. I ended up with a few more cleanups,
> too. This is all on top of Gábor's patches. It's conceptually
> independent, but the textual wrangling was annoying enough it didn't
> make any sense to require you to do it again during merging. ;) Plus I
> do not think either topic is high-risk nor urgent enough to worry too
> much about one blocking the other.

I'll ask you to do the last step maybe in a few weeks; the range
notation tests seem to have changed since where I queued Gábor's
patches and where [4/4] is based on (yours is based on newer
codebase).

> The diffstat is scary, but it's mostly the final patch, which is pretty
> mechanical.

Yup, and the result is much easier to read.

Thanks.

>   [1/4]: t0000: keep clean-up tests together
>   [2/4]: t0000: run prereq tests inside sub-test
>   [3/4]: t0000: run cleaning test inside sub-test
>   [4/4]: t0000: consistently use single quotes for outer tests
>
>  t/t0000-basic.sh | 570 +++++++++++++++++++++++------------------------
>  1 file changed, 284 insertions(+), 286 deletions(-)
>
> -Peff

Re: [PATCH 1/2] tests: make sure nested lazy prereqs work reliably

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, Nov 18, 2020 at 08:04:13PM +0100, SZEDER Gábor wrote:
>
>> So to prevent nested prereqs from interfering with each other let's
>> evaluate each prereq in its own dedicated directory by appending the
>> prereq's name to the directory name, e.g. 'prereq-test-dir-SYMLINKS'.
>> In the test we check not only that the prereq test dir is still there,
>> but also that the inner prereq can't mess with the outer prereq's
>> files.
>
> That sounds reasonable. I do wonder, though, whether simply creating the
> prereq directory in the _current_ directory would be sufficient. Then
> you'd get prereq-test-dir/prereq-test-dir for a nested invocation. But
> the prereqs aren't supposed to care about which specific directory
> they're in.

True.  That does sound conceptually simpler.  As we've already seen
the patch, I do not mind too deeply either way, though.

Thanks.

[PATCH 0/4] t0000 cleanups

[Jeff King]

This is a re-send of some cleanups from:

  https://lore.kernel.org/git/20201120001458.GA274082@coredump.intra.peff.net/

There were some textual conflicts back then which would have been
tedious to resolve, so we punted. Now that the requisite commits are all
in master, this can be applied there without any conflicts.

  [1/4]: t0000: keep clean-up tests together
  [2/4]: t0000: run prereq tests inside sub-test
  [3/4]: t0000: run cleaning test inside sub-test
  [4/4]: t0000: consistently use single quotes for outer tests

 t/t0000-basic.sh | 570 +++++++++++++++++++++++------------------------
 1 file changed, 284 insertions(+), 286 deletions(-)

-Peff

[PATCH 2/4] t0000: run prereq tests inside sub-test

[Jeff King]

We test the behavior of prerequisites in t0000 by setting up fake ones
in the main test script, trying to run some tests, and then seeing if
those tests impacted the environment correctly. If they didn't, then we
write a message and manually call exit.

Instead, let's push these down into a sub-test, like many of the other
tests covering the framework itself. This has a few advantages:

  - it does not pollute the test output with mention of skipped tests
    (that we know are uninteresting -- the point of the test was to see
    that these are skipped).

  - when running in a TAP harness, we get a useful test failure message
    (whereas when the script exits early, a tool like "prove" simply
    says "Dubious, test returned 1").

  - we do not have to worry about different test environments, such as
    when GIT_TEST_FAIL_PREREQS_INTERNAL is set. Our sub-test helpers
    already give us a known environment.

  - the tests themselves are a bit easier to read, as we can just check
    the test-framework output to see what happened (and get the usual
    test_cmp diff if it failed)

A few notes on the implementation:

  - we could do one sub-test per each individual test_expect_success. I
    broke it up here into a few logical groups, as I think this makes it
    more readable

  - the original tests modified environment variables inside the test
    bodies. Instead, I've used "true" as the body of a test we expect to
    run and "false" otherwise. Technically this does not confirm that
    the body of the "true" test actually ran. We are trusting the
    framework output to believe that it truly ran, which is sufficient
    for these tests. And I think the end result is much simpler to
    follow.

  - the nested_prereq test uses a few bare "test -f" calls; I converted
    these to our usual test_path_is_* helpers while moving the code
    around.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t0000-basic.sh | 149 ++++++++++++++++++++++-------------------------
 1 file changed, 69 insertions(+), 80 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index d49b5dd4ac..3d36a87610 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -759,96 +759,85 @@ test_expect_success '--run invalid range end' "
 	EOF_ERR
 "
 
+test_expect_success 'tests respect prerequisites' '
+	run_sub_test_lib_test prereqs "tests respect prereqs" <<-\EOF &&
 
-test_set_prereq HAVEIT
-haveit=no
-test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '
-	test_have_prereq HAVEIT &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success DONTHAVEIT 'unmet prerequisite causes test to be skipped' '
-	donthaveit=no
-'
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit != yesyes
-then
-	say "bug in test framework: prerequisite tags do not work reliably"
-	exit 1
-fi
+	test_set_prereq HAVEIT
+	test_expect_success HAVEIT "prereq is satisfied" "true"
+	test_expect_success "have_prereq works" "
+		test_have_prereq HAVEIT
+	"
+	test_expect_success DONTHAVEIT "prereq not satisfied" "false"
 
-test_set_prereq HAVETHIS
-haveit=no
-test_expect_success HAVETHIS,HAVEIT 'test runs if prerequisites are satisfied' '
-	test_have_prereq HAVEIT &&
-	test_have_prereq HAVETHIS &&
-	haveit=yes
-'
-donthaveit=yes
-test_expect_success HAVEIT,DONTHAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveit=no
-'
-donthaveiteither=yes
-test_expect_success DONTHAVEIT,HAVEIT 'unmet prerequisites causes test to be skipped' '
-	donthaveiteither=no
-'
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a $haveit$donthaveit$donthaveiteither != yesyesyes
-then
-	say "bug in test framework: multiple prerequisite tags do not work reliably"
-	exit 1
-fi
+	test_set_prereq HAVETHIS
+	test_expect_success HAVETHIS,HAVEIT "multiple prereqs" "true"
+	test_expect_success HAVEIT,DONTHAVEIT "mixed prereqs (yes,no)" "false"
+	test_expect_success DONTHAVEIT,HAVEIT "mixed prereqs (no,yes)" "false"
 
-test_lazy_prereq LAZY_TRUE true
-havetrue=no
-test_expect_success LAZY_TRUE 'test runs if lazy prereq is satisfied' '
-	havetrue=yes
-'
-donthavetrue=yes
-test_expect_success !LAZY_TRUE 'missing lazy prereqs skip tests' '
-	donthavetrue=no
+	test_done
+	EOF
+
+	check_sub_test_lib_test prereqs <<-\EOF
+	ok 1 - prereq is satisfied
+	ok 2 - have_prereq works
+	ok 3 # skip prereq not satisfied (missing DONTHAVEIT)
+	ok 4 - multiple prereqs
+	ok 5 # skip mixed prereqs (yes,no) (missing DONTHAVEIT of HAVEIT,DONTHAVEIT)
+	ok 6 # skip mixed prereqs (no,yes) (missing DONTHAVEIT of DONTHAVEIT,HAVEIT)
+	# passed all 6 test(s)
+	1..6
+	EOF
 '
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a "$havetrue$donthavetrue" != yesyes
-then
-	say 'bug in test framework: lazy prerequisites do not work'
-	exit 1
-fi
+test_expect_success 'tests respect lazy prerequisites' '
+	run_sub_test_lib_test lazy-prereqs "respect lazy prereqs" <<-\EOF &&
 
-test_lazy_prereq LAZY_FALSE false
-nothavefalse=no
-test_expect_success !LAZY_FALSE 'negative lazy prereqs checked' '
-	nothavefalse=yes
-'
-havefalse=yes
-test_expect_success LAZY_FALSE 'missing negative lazy prereqs will skip' '
-	havefalse=no
-'
+	test_lazy_prereq LAZY_TRUE true
+	test_expect_success LAZY_TRUE "lazy prereq is satisifed" "true"
+	test_expect_success !LAZY_TRUE "negative lazy prereq" "false"
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" -a "$nothavefalse$havefalse" != yesyes
-then
-	say 'bug in test framework: negative lazy prerequisites do not work'
-	exit 1
-fi
+	test_lazy_prereq LAZY_FALSE false
+	test_expect_success LAZY_FALSE "lazy prereq not satisfied" "false"
+	test_expect_success !LAZY_FALSE "negative false prereq" "true"
 
-test_lazy_prereq NESTED_INNER '
-	>inner &&
-	rm -f outer
-'
-test_lazy_prereq NESTED_PREREQ '
-	>outer &&
-	test_have_prereq NESTED_INNER &&
-	echo "can create new file in cwd" >file &&
-	test -f outer &&
-	test ! -f inner
-'
-test_expect_success NESTED_PREREQ 'evaluating nested lazy prereqs dont interfere with each other' '
-	nestedworks=yes
+	test_done
+	EOF
+
+	check_sub_test_lib_test lazy-prereqs <<-\EOF
+	ok 1 - lazy prereq is satisifed
+	ok 2 # skip negative lazy prereq (missing !LAZY_TRUE)
+	ok 3 # skip lazy prereq not satisfied (missing LAZY_FALSE)
+	ok 4 - negative false prereq
+	# passed all 4 test(s)
+	1..4
+	EOF
 '
 
-if test -z "$GIT_TEST_FAIL_PREREQS_INTERNAL" && test "$nestedworks" != yes
-then
-	say 'bug in test framework: nested lazy prerequisites do not work'
-	exit 1
-fi
+test_expect_success 'nested lazy prerequisites' '
+	run_sub_test_lib_test nested-lazy "nested lazy prereqs" <<-\EOF &&
+
+	test_lazy_prereq NESTED_INNER "
+		>inner &&
+		rm -f outer
+	"
+	test_lazy_prereq NESTED_PREREQ "
+		>outer &&
+		test_have_prereq NESTED_INNER &&
+		echo can create new file in cwd >file &&
+		test_path_is_file outer &&
+		test_path_is_missing inner
+	"
+	test_expect_success NESTED_PREREQ "evaluate nested prereq" "true"
+
+	test_done
+	EOF
+
+	check_sub_test_lib_test nested-lazy <<-\EOF
+	ok 1 - evaluate nested prereq
+	# passed all 1 test(s)
+	1..1
+	EOF
+'
 
 test_expect_success 'lazy prereqs do not turn off tracing' "
 	run_sub_test_lib_test lazy-prereq-and-tracing \
-- 
2.30.0.758.gfe500d6872