git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
	"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH] test-lib: have --immediate emit valid TAP on failure
Date: Wed, 23 Mar 2022 21:51:31 +0100	[thread overview]
Message-ID: <patch-1.1-47b236139e6-20220323T204410Z-avarab@gmail.com> (raw)

Change the "--immediate" option so that it emits valid TAP on
failure. Before this it would omit the required plan at the end,
e.g. under SANITIZE=leak we'd show a "No plan found in TAP output"
error from "prove":

    $ prove t0006-date.sh ::  --immediate
    t0006-date.sh .. Dubious, test returned 1 (wstat 256, 0x100)
    Failed 1/22 subtests

    Test Summary Report
    -------------------
    t0006-date.sh (Wstat: 256 Tests: 22 Failed: 1)
      Failed test:  22
      Non-zero exit status: 1
      Parse errors: No plan found in TAP output
    Files=1, Tests=22,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.18 cusr  0.06 csys =  0.27 CPU)
    Result: FAIL

Now we'll emit output that doesn't result in TAP parsing failures:

    $ prove t0006-date.sh ::  --immediate
    t0006-date.sh .. Dubious, test returned 1 (wstat 256, 0x100)
    Failed 1/22 subtests

    Test Summary Report
    -------------------
    t0006-date.sh (Wstat: 256 Tests: 22 Failed: 1)
      Failed test:  22
      Non-zero exit status: 1
    Files=1, Tests=22,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.19 cusr  0.05 csys =  0.26 CPU)
    Result: FAIL

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---

Junio: When this is merged to "seen" there's a conflict like:

	<<<<<<< HEAD
		if test -n "$immediate"
		then
			say_color error "1..$test_count"
			_error_exit
		fi
	=======
		test "$immediate" = "" || _error_exit
		finalize_test_case_output failure "$failure_label" "$@"
	>>>>>>> origin/seen

The "solution" is to end up with:

	test_failure_ () {
		failure_label=$1
		test_failure=$(($test_failure + 1))
		say_color error "not ok $test_count - $1"
		shift
		printf '%s\n' "$*" | sed -e 's/^/#	/'
		if test -n "$immediate"
		then
			say_color error "1..$test_count"
			_error_exit
		fi
		finalize_test_case_output failure "$failure_label" "$@"
	}

Those scare quotes being because the "finalize_test_case_output" is in
the wrong place both before and after, but that's an existing bug in
eb53a5b8047 (ci: call `finalize_test_case_output` a little later,
2022-03-01), i.e. that alternate output mechanism wasn't doing the
right thing on --immediate output.

But this change and that merge resolution doesn't make that any worse,
so hopefully this can be queued anyway. Thanks!
		
 t/t0000-basic.sh | 13 +++++++++++++
 t/test-lib.sh    |  6 +++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 9dcbf518a78..17a268ccd1b 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -101,6 +101,19 @@ test_expect_success 'subtest: 2/3 tests passing' '
 	EOF
 '
 
+test_expect_success 'subtest: --immediate' '
+	run_sub_test_lib_test_err partial-pass \
+		--immediate &&
+	check_sub_test_lib_test_err partial-pass \
+		<<-\EOF_OUT 3<<-EOF_ERR
+	> ok 1 - passing test #1
+	> not ok 2 - failing test #2
+	> #	false
+	> 1..2
+	EOF_OUT
+	EOF_ERR
+'
+
 test_expect_success 'subtest: a failing TODO test' '
 	write_and_run_sub_test_lib_test failing-todo <<-\EOF &&
 	test_expect_success "passing test" "true"
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 515b1af7ed4..4373f7d70b5 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -806,7 +806,11 @@ test_failure_ () {
 	say_color error "not ok $test_count - $1"
 	shift
 	printf '%s\n' "$*" | sed -e 's/^/#	/'
-	test "$immediate" = "" || _error_exit
+	if test -n "$immediate"
+	then
+		say_color error "1..$test_count"
+		_error_exit
+	fi
 }
 
 test_known_broken_ok_ () {
-- 
2.35.1.1452.ga7cfc89151f


             reply	other threads:[~2022-03-23 20:51 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-23 20:51 Ævar Arnfjörð Bjarmason [this message]
2022-03-24 13:38 ` [PATCH] test-lib: have --immediate emit valid TAP on failure Johannes Schindelin
2022-03-24 13:48   ` A "why TAP?" manifesto (was: [PATCH] test-lib: have --immediate emit valid TAP on failure) Ævar Arnfjörð Bjarmason
2022-03-24 17:26     ` A "why TAP?" manifesto Junio C Hamano
2022-03-28 15:50       ` Ævar Arnfjörð Bjarmason
2022-03-24 21:57     ` A "why TAP?" manifesto (was: [PATCH] test-lib: have --immediate emit valid TAP on failure) Taylor Blau

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=patch-1.1-47b236139e6-20220323T204410Z-avarab@gmail.com \
    --to=avarab@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).