From: Josh Steadmon <steadmon@google.com>
To: Christian Couder <christian.couder@gmail.com>
Cc: git@vger.kernel.org, phillip.wood123@gmail.com,
linusa@google.com, calvinwan@google.com, gitster@pobox.com,
rsbecker@nexbridge.com
Subject: Re: [PATCH v8 2/3] unit tests: add TAP unit test framework
Date: Wed, 1 Nov 2023 15:54:42 -0700 [thread overview]
Message-ID: <ZULXMhD0ajESkde5@google.com> (raw)
In-Reply-To: <CAP8UFD3eY_i36YO0OcpAp9ey5KO0q-PrwvjSLRXKYQb=iZ8JCA@mail.gmail.com>
On 2023.10.27 22:15, Christian Couder wrote:
> On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon <steadmon@google.com> wrote:
> >
> > From: Phillip Wood <phillip.wood@dunelm.org.uk>
> >
> > This patch contains an implementation for writing unit tests with TAP
> > output. Each test is a function that contains one or more checks. The
> > test is run with the TEST() macro and if any of the checks fail then the
> > test will fail. A complete program that tests STRBUF_INIT would look
> > like
> >
> > #include "test-lib.h"
> > #include "strbuf.h"
> >
> > static void t_static_init(void)
> > {
> > struct strbuf buf = STRBUF_INIT;
> >
> > check_uint(buf.len, ==, 0);
> > check_uint(buf.alloc, ==, 0);
> > check_char(buf.buf[0], ==, '\0');
> > }
> >
> > int main(void)
> > {
> > TEST(t_static_init(), "static initialization works);
> >
> > return test_done();
> > }
> >
> > The output of this program would be
> >
> > ok 1 - static initialization works
> > 1..1
> >
> > If any of the checks in a test fail then they print a diagnostic message
> > to aid debugging and the test will be reported as failing. For example a
> > failing integer check would look like
> >
> > # check "x >= 3" failed at my-test.c:102
>
> I wonder if it would be a bit better to say that the test was an
> integer test for example with "check_int(x >= 3) failed ..."
>
> > # left: 2
> > # right: 3
>
> I like "expected" and "actual" better than "left" and "right", not
> sure how it's possible to have that in a way consistent with the shell
> tests though.
I also prefer expected/actual, but I don't think it's possible where we
accept arbitrary operators, and I don't want to plumb a flag through to
specify whether to display left/right vs expected/actual.
> > not ok 1 - x is greater than or equal to three
> >
> > There are a number of check functions implemented so far. check() checks
> > a boolean condition, check_int(), check_uint() and check_char() take two
> > values to compare and a comparison operator. check_str() will check if
> > two strings are equal. Custom checks are simple to implement as shown in
> > the comments above test_assert() in test-lib.h.
>
> Yeah, nice.
>
> > Tests can be skipped with test_skip() which can be supplied with a
> > reason for skipping which it will print. Tests can print diagnostic
> > messages with test_msg(). Checks that are known to fail can be wrapped
> > in TEST_TODO().
>
> Maybe TEST_TOFIX() would be a bit more clear, but "TODO" is something
> that is more likely to be searched for than "TOFIX", so Ok.
>
> > There are a couple of example test programs included in this
> > patch. t-basic.c implements some self-tests and demonstrates the
> > diagnostic output for failing test. The output of this program is
> > checked by t0080-unit-test-output.sh. t-strbuf.c shows some example
> > unit tests for strbuf.c
> >
> > The unit tests will be built as part of the default "make all" target,
> > to avoid bitrot. If you wish to build just the unit tests, you can run
> > "make build-unit-tests". To run the tests, you can use "make unit-tests"
> > or run the test binaries directly, as in "./t/unit-tests/bin/t-strbuf".
>
> Nice!
>
> > +unit-tests-prove:
> > + @echo "*** prove - unit tests ***"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)
>
> Nice, but DEFAULT_TEST_TARGET=prove isn't used. So not sure how
> important or relevant the 'prove' related sections are in the
> Documentation/technical/unit-tests.txt file introduced by the previous
> patch.
The "unit-tests" target runs DEFAULT_UNIT_TEST_TARGET, which can be
overridden to "unit-tests-prove".
> > +int test_assert(const char *location, const char *check, int ok)
> > +{
> > + assert(ctx.running);
> > +
> > + if (ctx.result == RESULT_SKIP) {
> > + test_msg("skipping check '%s' at %s", check, location);
> > + return 1;
> > + } else if (!ctx.todo) {
>
> I think it would be a bit clearer without the "else" above and with
> the "if (!ctx.todo) {" starting on a new line.
Fixed in v9.
> > + if (ok) {
> > + test_pass();
> > + } else {
> > + test_msg("check \"%s\" failed at %s", check, location);
> > + test_fail();
> > + }
> > + }
> > +
> > + return !!ok;
> > +}
>
> Otherwise it looks good to me.
Thanks for the review!
next prev parent reply other threads:[~2023-11-01 22:55 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20230517-unit-tests-v2-v2-0-8c1b50f75811@google.com>
2023-06-30 22:51 ` [PATCH v4] unit tests: Add a project plan document Josh Steadmon
2023-07-01 0:42 ` Junio C Hamano
2023-07-01 1:03 ` Junio C Hamano
2023-08-07 23:07 ` [PATCH v5] " Josh Steadmon
2023-08-14 13:29 ` Phillip Wood
2023-08-15 22:55 ` Josh Steadmon
2023-08-17 9:05 ` Phillip Wood
2023-08-16 23:50 ` [PATCH v6 0/3] Add unit test framework and project plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-17 0:12 ` Junio C Hamano
2023-08-17 0:41 ` Junio C Hamano
2023-08-17 18:34 ` Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 0/3] Add unit test framework and project plan Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-18 0:12 ` Junio C Hamano
2023-09-22 20:05 ` Junio C Hamano
2023-09-24 13:57 ` phillip.wood123
2023-09-25 18:57 ` Junio C Hamano
2023-10-06 22:58 ` Josh Steadmon
2023-10-09 17:37 ` Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 20:38 ` [PATCH v7 0/3] Add unit test framework and project plan Junio C Hamano
2023-08-24 20:11 ` Josh Steadmon
2023-09-13 18:14 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 " Josh Steadmon
2023-10-09 22:21 ` [PATCH v8 1/3] unit tests: Add a project plan document Josh Steadmon
2023-10-10 8:57 ` Oswald Buddenhagen
2023-10-11 21:14 ` Josh Steadmon
2023-10-11 23:05 ` Oswald Buddenhagen
2023-11-01 17:31 ` Josh Steadmon
2023-10-27 20:12 ` Christian Couder
2023-11-01 17:47 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-10-11 21:42 ` Junio C Hamano
2023-10-16 13:43 ` [PATCH v8 2.5/3] fixup! " Phillip Wood
2023-10-16 16:41 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:48 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-27 20:15 ` [PATCH v8 2/3] " Christian Couder
2023-11-01 22:54 ` Josh Steadmon [this message]
2023-10-09 22:21 ` [PATCH v8 3/3] ci: run unit tests in CI Josh Steadmon
2023-10-09 23:50 ` [PATCH v8 0/3] Add unit test framework and project plan Junio C Hamano
2023-10-19 15:21 ` [PATCH 0/3] CMake unit test fixups Phillip Wood
2023-10-19 15:21 ` [PATCH 1/3] fixup! cmake: also build unit tests Phillip Wood
2023-10-19 15:21 ` [PATCH 2/3] fixup! artifacts-tar: when including `.dll` files, don't forget the unit-tests Phillip Wood
2023-10-19 15:21 ` [PATCH 3/3] fixup! cmake: handle also unit tests Phillip Wood
2023-10-19 19:19 ` [PATCH 0/3] CMake unit test fixups Junio C Hamano
2023-10-16 10:07 ` [PATCH v8 0/3] Add unit test framework and project plan phillip.wood123
2023-11-01 23:09 ` Josh Steadmon
2023-10-27 20:26 ` Christian Couder
2023-11-01 23:31 ` [PATCH v9 " Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-03 21:54 ` Christian Couder
2023-11-09 17:51 ` Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 3/3] ci: run unit tests in CI Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 0/3] Add unit test framework and project plan Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-09 23:15 ` Junio C Hamano
2023-11-09 18:50 ` [PATCH v10 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 3/3] ci: run unit tests in CI Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZULXMhD0ajESkde5@google.com \
--to=steadmon@google.com \
--cc=calvinwan@google.com \
--cc=christian.couder@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=linusa@google.com \
--cc=phillip.wood123@gmail.com \
--cc=rsbecker@nexbridge.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).