From: Adam Spiers <git@adamspiers.org>
To: git list <git@vger.kernel.org>
Subject: [PATCH v6 6/7] tests: test the test framework more thoroughly
Date: Sun, 16 Dec 2012 18:28:14 +0000 [thread overview]
Message-ID: <1355682495-22382-7-git-send-email-git@adamspiers.org> (raw)
In-Reply-To: <1355682495-22382-1-git-send-email-git@adamspiers.org>
Add 5 new full test suite runs each with a different number of
passing/failing/broken/fixed tests, in order to ensure that the
correct exit code and output are generated in each case. As before,
these are run in a subdirectory to avoid disrupting the metrics for
the parent tests.
Signed-off-by: Adam Spiers <git@adamspiers.org>
---
t/t0000-basic.sh | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 104 insertions(+)
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index fc5200f..5c1dde0 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -79,6 +79,55 @@ check_sub_test_lib_test () {
)
}
+test_expect_success 'pretend we have a fully passing test suite' "
+ run_sub_test_lib_test full-pass '3 passing tests' <<-\\EOF &&
+ for i in 1 2 3; do
+ test_expect_success \"passing test #\$i\" 'true'
+ done
+ test_done
+ EOF
+ check_sub_test_lib_test full-pass <<-\\EOF
+ > ok 1 - passing test #1
+ > ok 2 - passing test #2
+ > ok 3 - passing test #3
+ > # passed all 3 test(s)
+ > 1..3
+ EOF
+"
+
+test_expect_success 'pretend we have a partially passing test suite' "
+ test_must_fail run_sub_test_lib_test \
+ partial-pass '2/3 tests passing' <<-\\EOF &&
+ test_expect_success 'passing test #1' 'true'
+ test_expect_success 'failing test #2' 'false'
+ test_expect_success 'passing test #3' 'true'
+ test_done
+ EOF
+ check_sub_test_lib_test partial-pass <<-\\EOF
+ > ok 1 - passing test #1
+ > not ok 2 - failing test #2
+ # false
+ > ok 3 - passing test #3
+ > # failed 1 among 3 test(s)
+ > 1..3
+ EOF
+"
+
+test_expect_success 'pretend we have a known breakage' "
+ run_sub_test_lib_test failing-todo 'A failing TODO test' <<-\\EOF &&
+ test_expect_success 'passing test' 'true'
+ test_expect_failure 'pretend we have a known breakage' 'false'
+ test_done
+ EOF
+ check_sub_test_lib_test failing-todo <<-\\EOF
+ > ok 1 - passing test
+ > not ok 2 - pretend we have a known breakage # TODO known breakage
+ > # still have 1 known breakage(s)
+ > # passed all remaining 1 test(s)
+ > 1..2
+ EOF
+"
+
test_expect_success 'pretend we have fixed a known breakage' "
run_sub_test_lib_test passing-todo 'A passing TODO test' <<-\\EOF &&
test_expect_failure 'pretend we have fixed a known breakage' 'true'
@@ -92,6 +141,61 @@ test_expect_success 'pretend we have fixed a known breakage' "
EOF
"
+test_expect_success 'pretend we have a pass, fail, and known breakage' "
+ test_must_fail run_sub_test_lib_test \
+ mixed-results1 'mixed results #1' <<-\\EOF &&
+ test_expect_success 'passing test' 'true'
+ test_expect_success 'failing test' 'false'
+ test_expect_failure 'pretend we have a known breakage' 'false'
+ test_done
+ EOF
+ check_sub_test_lib_test mixed-results1 <<-\\EOF
+ > ok 1 - passing test
+ > not ok 2 - failing test
+ > # false
+ > not ok 3 - pretend we have a known breakage # TODO known breakage
+ > # still have 1 known breakage(s)
+ > # failed 1 among remaining 2 test(s)
+ > 1..3
+ EOF
+"
+
+test_expect_success 'pretend we have a mix of all possible results' "
+ test_must_fail run_sub_test_lib_test \
+ mixed-results2 'mixed results #2' <<-\\EOF &&
+ test_expect_success 'passing test' 'true'
+ test_expect_success 'passing test' 'true'
+ test_expect_success 'passing test' 'true'
+ test_expect_success 'passing test' 'true'
+ test_expect_success 'failing test' 'false'
+ test_expect_success 'failing test' 'false'
+ test_expect_success 'failing test' 'false'
+ test_expect_failure 'pretend we have a known breakage' 'false'
+ test_expect_failure 'pretend we have a known breakage' 'false'
+ test_expect_failure 'pretend we have fixed a known breakage' 'true'
+ test_done
+ EOF
+ check_sub_test_lib_test mixed-results2 <<-\\EOF
+ > ok 1 - passing test
+ > ok 2 - passing test
+ > ok 3 - passing test
+ > ok 4 - passing test
+ > not ok 5 - failing test
+ > # false
+ > not ok 6 - failing test
+ > # false
+ > not ok 7 - failing test
+ > # false
+ > not ok 8 - pretend we have a known breakage # TODO known breakage
+ > not ok 9 - pretend we have a known breakage # TODO known breakage
+ > ok 10 - pretend we have fixed a known breakage # TODO known breakage
+ > # fixed 1 known breakage(s)
+ > # still have 2 known breakage(s)
+ > # failed 3 among remaining 8 test(s)
+ > 1..10
+ EOF
+"
+
test_set_prereq HAVEIT
haveit=no
test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '
--
1.7.12.1.396.g53b3ea9
next prev parent reply other threads:[~2012-12-16 18:39 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-16 18:28 [PATCH v6 0/7] make test output coloring more intuitive Adam Spiers
2012-12-16 18:28 ` [PATCH v6 1/7] tests: test number comes first in 'not ok $count - $message' Adam Spiers
2012-12-16 18:28 ` [PATCH v6 2/7] tests: paint known breakages in bold yellow Adam Spiers
2012-12-16 18:28 ` [PATCH v6 3/7] tests: paint skipped tests in bold blue Adam Spiers
2012-12-16 18:28 ` [PATCH v6 4/7] tests: change info messages from yellow/brown to bold cyan Adam Spiers
2012-12-16 18:28 ` [PATCH v6 5/7] tests: refactor mechanics of testing in a sub test-lib Adam Spiers
2012-12-16 18:28 ` Adam Spiers [this message]
2012-12-16 18:28 ` [PATCH v6 7/7] tests: paint unexpectedly fixed known breakages in bold red Adam Spiers
2012-12-16 18:54 ` [PATCH v6 0/7] make test output coloring more intuitive Junio C Hamano
2012-12-16 19:01 ` Adam Spiers
2012-12-16 23:11 ` Junio C Hamano
2012-12-20 15:34 ` Jeff King
2012-12-20 15:44 ` Adam Spiers
2012-12-20 16:11 ` Jeff King
2012-12-20 18:08 ` Adam Spiers
2012-12-20 19:21 ` Junio C Hamano
2012-12-20 19:50 ` Jeff King
2012-12-20 23:28 ` Adam Spiers
2012-12-21 3:12 ` [PATCH v7 0/7] coloring test output after traffic signal Junio C Hamano
2012-12-21 8:15 ` Jeff King
2012-12-21 10:00 ` Adam Spiers
2012-12-21 3:12 ` [PATCH v7 1/7] tests: test number comes first in 'not ok $count - $message' Junio C Hamano
2012-12-21 3:12 ` [PATCH v7 2/7] tests: paint known breakages in yellow Junio C Hamano
2012-12-21 8:51 ` Stefano Lattarini
2012-12-21 15:46 ` Junio C Hamano
2012-12-21 16:59 ` Stefano Lattarini
2012-12-21 3:12 ` [PATCH v7 3/7] tests: paint skipped tests in blue Junio C Hamano
2012-12-21 3:12 ` [PATCH v7 4/7] tests: change info messages from yellow/brown to cyan Junio C Hamano
2012-12-21 3:12 ` [PATCH v7 5/7] tests: refactor mechanics of testing in a sub test-lib Junio C Hamano
2012-12-21 3:12 ` [PATCH v7 6/7] tests: test the test framework more thoroughly Junio C Hamano
2012-12-21 3:12 ` [PATCH v7 7/7] tests: paint unexpectedly fixed known breakages in bold red Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1355682495-22382-7-git-send-email-git@adamspiers.org \
--to=git@adamspiers.org \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).