From: Josh Steadmon <steadmon@google.com>
To: phillip.wood@dunelm.org.uk
Cc: git@vger.kernel.org, linusa@google.com, calvinwan@google.com,
gitster@pobox.com
Subject: Re: [PATCH v5] unit tests: Add a project plan document
Date: Tue, 15 Aug 2023 15:55:05 -0700 [thread overview]
Message-ID: <ZNwCSdrbuG3K/iMP@google.com> (raw)
In-Reply-To: <90651b2b-6a7a-2ed0-716a-fc87adc957ba@gmail.com>
On 2023.08.14 14:29, Phillip Wood wrote:
> Hi Josh
>
> On 08/08/2023 00:07, Josh Steadmon wrote:
> >
> > Reviewers can help this series progress by discussing whether it's
> > acceptable to rely on `prove` as a test harness for unit tests. We
> > support this for the current shell tests suite, but it is not strictly
> > required.
>
> If possible it would be good to be able to run individual test programs
> without a harness as we can for our integration tests. For running more than
> one test program in parallel I think it is fine to require a harness.
Sounds good. This is working in v6 which I hope to send to the list
soon.
> I don't have a strong preference for which harness we use so long as it
> provides a way to (a) run tests that previously failed tests first and (b)
> run slow tests first. I do have a strong preference for using the same
> harness for both the unit tests and the integration tests so developers
> don't have to learn two different tools. Unless there is a problem with
> prove it would probably make sense just to keep using that as the project
> test harness.
To be clear, it sounds like both of these can be done with `prove`
(using the various --state settings) without any further support from
our unit tests, right? I see that we do have a "failed" target for
re-running integration tests, but that relies on some test-lib.sh
features that currently have no equivalent in the unit test framework.
> > TODOs remaining:
> > - Discuss pre-existing harnesses for the current test suite
> > - Figure out how to evaluate frameworks on additional OSes such as *BSD
> > and NonStop
>
> We have .cirrus.yml in tree which I think gitgitgadget uses to run our test
> suite on freebsd so we could leverage that for the unit tests. As for
> NonStop I think we'd just have to rely on Randall running the tests as he
> does now for the integration tests.
Thanks for the pointer. I've updated .cirrus.yml (as well as the GitHub
CI configs) for v6.
> > Changes in v5:
> > - Add comparison point "License".
> > - Discuss feature priorities
> > - Drop frameworks:
> > - Incompatible licenses: libtap, cmocka
> > - Missing source: MyTAP
> > - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit
> > - Drop comparison point "Coverage reports": this can generally be
> > handled by tools such as `gcov` regardless of the framework used.
> > - Drop comparison point "Inline tests": there didn't seem to be
> > strong interest from reviewers for this feature.
> > - Drop comparison point "Scheduling / re-running": this was not
> > supported by any of the main contenders, and is generally better
> > handled by the harness rather than framework.
> > - Drop comparison point "Lazy test planning": this was supported by
> > all frameworks that provide TAP output.
>
> These changes all sound sensible to me
>
> The trimmed down table is most welcome. The custom implementation supports
> partial parallel execution. It shouldn't be too difficult to add some signal
> handling to it depending on what we want it to do.
>
> It sounds like we're getting to the point where we have pinned down our
> requirements and the available alternatives well enough to make a decision.
Yes, v6 will include your TAP implementation (I assume you are still OK
if I include your patch in this series?).
> Best Wishes
>
> Phillip
>
next prev parent reply other threads:[~2023-08-15 22:55 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20230517-unit-tests-v2-v2-0-8c1b50f75811@google.com>
2023-06-30 22:51 ` [PATCH v4] unit tests: Add a project plan document Josh Steadmon
2023-07-01 0:42 ` Junio C Hamano
2023-07-01 1:03 ` Junio C Hamano
2023-08-07 23:07 ` [PATCH v5] " Josh Steadmon
2023-08-14 13:29 ` Phillip Wood
2023-08-15 22:55 ` Josh Steadmon [this message]
2023-08-17 9:05 ` Phillip Wood
2023-08-16 23:50 ` [PATCH v6 0/3] Add unit test framework and project plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-17 0:12 ` Junio C Hamano
2023-08-17 0:41 ` Junio C Hamano
2023-08-17 18:34 ` Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 0/3] Add unit test framework and project plan Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-18 0:12 ` Junio C Hamano
2023-09-22 20:05 ` Junio C Hamano
2023-09-24 13:57 ` phillip.wood123
2023-09-25 18:57 ` Junio C Hamano
2023-10-06 22:58 ` Josh Steadmon
2023-10-09 17:37 ` Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 20:38 ` [PATCH v7 0/3] Add unit test framework and project plan Junio C Hamano
2023-08-24 20:11 ` Josh Steadmon
2023-09-13 18:14 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 " Josh Steadmon
2023-10-09 22:21 ` [PATCH v8 1/3] unit tests: Add a project plan document Josh Steadmon
2023-10-10 8:57 ` Oswald Buddenhagen
2023-10-11 21:14 ` Josh Steadmon
2023-10-11 23:05 ` Oswald Buddenhagen
2023-11-01 17:31 ` Josh Steadmon
2023-10-27 20:12 ` Christian Couder
2023-11-01 17:47 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-10-11 21:42 ` Junio C Hamano
2023-10-16 13:43 ` [PATCH v8 2.5/3] fixup! " Phillip Wood
2023-10-16 16:41 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:48 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-27 20:15 ` [PATCH v8 2/3] " Christian Couder
2023-11-01 22:54 ` Josh Steadmon
2023-10-09 22:21 ` [PATCH v8 3/3] ci: run unit tests in CI Josh Steadmon
2023-10-09 23:50 ` [PATCH v8 0/3] Add unit test framework and project plan Junio C Hamano
2023-10-19 15:21 ` [PATCH 0/3] CMake unit test fixups Phillip Wood
2023-10-19 15:21 ` [PATCH 1/3] fixup! cmake: also build unit tests Phillip Wood
2023-10-19 15:21 ` [PATCH 2/3] fixup! artifacts-tar: when including `.dll` files, don't forget the unit-tests Phillip Wood
2023-10-19 15:21 ` [PATCH 3/3] fixup! cmake: handle also unit tests Phillip Wood
2023-10-19 19:19 ` [PATCH 0/3] CMake unit test fixups Junio C Hamano
2023-10-16 10:07 ` [PATCH v8 0/3] Add unit test framework and project plan phillip.wood123
2023-11-01 23:09 ` Josh Steadmon
2023-10-27 20:26 ` Christian Couder
2023-11-01 23:31 ` [PATCH v9 " Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-03 21:54 ` Christian Couder
2023-11-09 17:51 ` Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 3/3] ci: run unit tests in CI Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 0/3] Add unit test framework and project plan Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-09 23:15 ` Junio C Hamano
2023-11-09 18:50 ` [PATCH v10 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 3/3] ci: run unit tests in CI Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZNwCSdrbuG3K/iMP@google.com \
--to=steadmon@google.com \
--cc=calvinwan@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=linusa@google.com \
--cc=phillip.wood@dunelm.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).