git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: git@vger.kernel.org
Cc: Adam Spiers <git@adamspiers.org>, Jeff King <peff@peff.net>
Subject: [PATCH v7 7/7] tests: paint unexpectedly fixed known breakages in bold red
Date: Thu, 20 Dec 2012 19:12:38 -0800	[thread overview]
Message-ID: <1356059558-23479-8-git-send-email-gitster@pobox.com> (raw)
In-Reply-To: <1356059558-23479-1-git-send-email-gitster@pobox.com>
In-Reply-To: <CAOkDyE9tDYRYzojzNnjWsT7UygxMAurHqLSDGA66_LMPD2Wmnw@mail.gmail.com>

From: Adam Spiers <git@adamspiers.org>

Change color of unexpectedly fixed known breakages to bold red.  An
unexpectedly passing test indicates that the test code is somehow
broken or out of sync with the code it is testing.  Either way this is
an error which is potentially as bad as a failing test, and as such is
no longer portrayed as a pass in the output.

Signed-off-by: Adam Spiers <git@adamspiers.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 t/t0000-basic.sh | 30 ++++++++++++++++++++++++------
 t/test-lib.sh    | 13 +++++++++----
 2 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 384b0ae..8973d2c 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -145,13 +145,31 @@ test_expect_success 'pretend we have fixed a known breakage' "
 	test_done
 	EOF
 	check_sub_test_lib_test passing-todo <<-\\EOF
-	> ok 1 - pretend we have fixed a known breakage # TODO known breakage
-	> # fixed 1 known breakage(s)
-	> # passed all 1 test(s)
+	> ok 1 - pretend we have fixed a known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
 	> 1..1
 	EOF
 "
 
+test_expect_success 'pretend we have fixed one of two known breakages (run in sub test-lib)' "
+	run_sub_test_lib_test partially-passing-todos \
+		'2 TODO tests, one passing' <<-\\EOF &&
+	test_expect_failure 'pretend we have a known breakage' 'false'
+	test_expect_success 'pretend we have a passing test' 'true'
+	test_expect_failure 'pretend we have fixed another known breakage' 'true'
+	test_done
+	EOF
+	check_sub_test_lib_test partially-passing-todos <<-\\EOF
+	> not ok 1 - pretend we have a known breakage # TODO known breakage
+	> ok 2 - pretend we have a passing test
+	> ok 3 - pretend we have fixed another known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
+	> # still have 1 known breakage(s)
+	> # passed all remaining 1 test(s)
+	> 1..3
+	EOF
+"
+
 test_expect_success 'pretend we have a pass, fail, and known breakage' "
 	test_must_fail run_sub_test_lib_test \
 		mixed-results1 'mixed results #1' <<-\\EOF &&
@@ -199,10 +217,10 @@ test_expect_success 'pretend we have a mix of all possible results' "
 	> #	false
 	> not ok 8 - pretend we have a known breakage # TODO known breakage
 	> not ok 9 - pretend we have a known breakage # TODO known breakage
-	> ok 10 - pretend we have fixed a known breakage # TODO known breakage
-	> # fixed 1 known breakage(s)
+	> ok 10 - pretend we have fixed a known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
 	> # still have 2 known breakage(s)
-	> # failed 3 among remaining 8 test(s)
+	> # failed 3 among remaining 7 test(s)
 	> 1..10
 	EOF
 "
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 8b75c9a..30a0937 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -278,7 +278,7 @@ test_failure_ () {
 
 test_known_broken_ok_ () {
 	test_fixed=$(($test_fixed+1))
-	say_color "" "ok $test_count - $@ # TODO known breakage"
+	say_color error "ok $test_count - $@ # TODO known breakage vanished"
 }
 
 test_known_broken_failure_ () {
@@ -373,13 +373,18 @@ test_done () {
 
 	if test "$test_fixed" != 0
 	then
-		say_color pass "# fixed $test_fixed known breakage(s)"
+		say_color error "# $test_fixed known breakage(s) vanished; please update test(s)"
 	fi
 	if test "$test_broken" != 0
 	then
 		say_color warn "# still have $test_broken known breakage(s)"
-		msg="remaining $(($test_count-$test_broken)) test(s)"
+	fi
+	if test "$test_broken" != 0 || test "$test_fixed" != 0
+	then
+		test_remaining=$(( $test_count - $test_broken - $test_fixed ))
+		msg="remaining $test_remaining test(s)"
 	else
+		test_remaining=$test_count
 		msg="$test_count test(s)"
 	fi
 	case "$test_failure" in
@@ -393,7 +398,7 @@ test_done () {
 
 		if test $test_external_has_tap -eq 0
 		then
-			if test $test_count -gt 0
+			if test $test_remaining -gt 0
 			then
 				say_color pass "# passed all $msg"
 			fi
-- 
1.8.1.rc2.225.g8d36ab4

      parent reply	other threads:[~2012-12-21  3:13 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-12-16 18:28 [PATCH v6 0/7] make test output coloring more intuitive Adam Spiers
2012-12-16 18:28 ` [PATCH v6 1/7] tests: test number comes first in 'not ok $count - $message' Adam Spiers
2012-12-16 18:28 ` [PATCH v6 2/7] tests: paint known breakages in bold yellow Adam Spiers
2012-12-16 18:28 ` [PATCH v6 3/7] tests: paint skipped tests in bold blue Adam Spiers
2012-12-16 18:28 ` [PATCH v6 4/7] tests: change info messages from yellow/brown to bold cyan Adam Spiers
2012-12-16 18:28 ` [PATCH v6 5/7] tests: refactor mechanics of testing in a sub test-lib Adam Spiers
2012-12-16 18:28 ` [PATCH v6 6/7] tests: test the test framework more thoroughly Adam Spiers
2012-12-16 18:28 ` [PATCH v6 7/7] tests: paint unexpectedly fixed known breakages in bold red Adam Spiers
2012-12-16 18:54 ` [PATCH v6 0/7] make test output coloring more intuitive Junio C Hamano
2012-12-16 19:01   ` Adam Spiers
2012-12-16 23:11     ` Junio C Hamano
2012-12-20 15:34     ` Jeff King
2012-12-20 15:44       ` Adam Spiers
2012-12-20 16:11         ` Jeff King
2012-12-20 18:08           ` Adam Spiers
2012-12-20 19:21           ` Junio C Hamano
2012-12-20 19:50             ` Jeff King
2012-12-20 23:28             ` Adam Spiers
2012-12-21  3:12               ` [PATCH v7 0/7] coloring test output after traffic signal Junio C Hamano
2012-12-21  8:15                 ` Jeff King
2012-12-21 10:00                   ` Adam Spiers
2012-12-21  3:12               ` [PATCH v7 1/7] tests: test number comes first in 'not ok $count - $message' Junio C Hamano
2012-12-21  3:12               ` [PATCH v7 2/7] tests: paint known breakages in yellow Junio C Hamano
2012-12-21  8:51                 ` Stefano Lattarini
2012-12-21 15:46                   ` Junio C Hamano
2012-12-21 16:59                     ` Stefano Lattarini
2012-12-21  3:12               ` [PATCH v7 3/7] tests: paint skipped tests in blue Junio C Hamano
2012-12-21  3:12               ` [PATCH v7 4/7] tests: change info messages from yellow/brown to cyan Junio C Hamano
2012-12-21  3:12               ` [PATCH v7 5/7] tests: refactor mechanics of testing in a sub test-lib Junio C Hamano
2012-12-21  3:12               ` [PATCH v7 6/7] tests: test the test framework more thoroughly Junio C Hamano
2012-12-21  3:12               ` Junio C Hamano [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1356059558-23479-8-git-send-email-gitster@pobox.com \
    --to=gitster@pobox.com \
    --cc=git@adamspiers.org \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).