From: Christian Couder <christian.couder@gmail.com>
To: Josh Steadmon <steadmon@google.com>
Cc: git@vger.kernel.org, phillip.wood123@gmail.com,
linusa@google.com, calvinwan@google.com, gitster@pobox.com,
rsbecker@nexbridge.com
Subject: Re: [PATCH v8 1/3] unit tests: Add a project plan document
Date: Fri, 27 Oct 2023 22:12:12 +0200 [thread overview]
Message-ID: <CAP8UFD26X4MPbJs4KfNOgicLMb-wiuFZj3Hw17acMmmc_=vcqQ@mail.gmail.com> (raw)
In-Reply-To: <81c5148a1267b8f9ce432a950340f0fa16b4d773.1696889530.git.steadmon@google.com>
On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon <steadmon@google.com> wrote:
>
> In our current testing environment, we spend a significant amount of
> effort crafting end-to-end tests for error conditions that could easily
> be captured by unit tests (or we simply forgo some hard-to-setup and
> rare error conditions). Describe what we hope to accomplish by
> implementing unit tests, and explain some open questions and milestones.
> Discuss desired features for test frameworks/harnesses, and provide a
> preliminary comparison of several different frameworks.
Nit: Not sure why the test framework comparison is "preliminary" as we
have actually selected a unit test framework and are adding it in the
next patch of the series. I understand that this was perhaps written
before the choice was made, but maybe we might want to update that
now.
> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
> new file mode 100644
> index 0000000000..b7a89cc838
> --- /dev/null
> +++ b/Documentation/technical/unit-tests.txt
> @@ -0,0 +1,220 @@
> += Unit Testing
> +
> +In our current testing environment, we spend a significant amount of effort
> +crafting end-to-end tests for error conditions that could easily be captured by
> +unit tests (or we simply forgo some hard-to-setup and rare error conditions).
> +Unit tests additionally provide stability to the codebase and can simplify
> +debugging through isolation. Writing unit tests in pure C, rather than with our
> +current shell/test-tool helper setup, simplifies test setup, simplifies passing
> +data around (no shell-isms required), and reduces testing runtime by not
> +spawning a separate process for every test invocation.
> +
> +We believe that a large body of unit tests, living alongside the existing test
> +suite, will improve code quality for the Git project.
I agree with that.
> +== Choosing a framework
> +
> +We believe the best option is to implement a custom TAP framework for the Git
> +project. We use a version of the framework originally proposed in
> +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[1].
Nit: Logically I would think that our opinion should come after the
comparison and be backed by it.
> +== Choosing a test harness
> +
> +During upstream discussion, it was occasionally noted that `prove` provides many
> +convenient features, such as scheduling slower tests first, or re-running
> +previously failed tests.
> +
> +While we already support the use of `prove` as a test harness for the shell
> +tests, it is not strictly required. The t/Makefile allows running shell tests
> +directly (though with interleaved output if parallelism is enabled). Git
> +developers who wish to use `prove` as a more advanced harness can do so by
> +setting DEFAULT_TEST_TARGET=prove in their config.mak.
> +
> +We will follow a similar approach for unit tests: by default the test
> +executables will be run directly from the t/Makefile, but `prove` can be
> +configured with DEFAULT_UNIT_TEST_TARGET=prove.
Nice that it can be used.
The rest of the file looks good.
next prev parent reply other threads:[~2023-10-27 20:12 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20230517-unit-tests-v2-v2-0-8c1b50f75811@google.com>
2023-06-30 22:51 ` [PATCH v4] unit tests: Add a project plan document Josh Steadmon
2023-07-01 0:42 ` Junio C Hamano
2023-07-01 1:03 ` Junio C Hamano
2023-08-07 23:07 ` [PATCH v5] " Josh Steadmon
2023-08-14 13:29 ` Phillip Wood
2023-08-15 22:55 ` Josh Steadmon
2023-08-17 9:05 ` Phillip Wood
2023-08-16 23:50 ` [PATCH v6 0/3] Add unit test framework and project plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-17 0:12 ` Junio C Hamano
2023-08-17 0:41 ` Junio C Hamano
2023-08-17 18:34 ` Josh Steadmon
2023-08-16 23:50 ` [PATCH v6 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 0/3] Add unit test framework and project plan Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 1/3] unit tests: Add a project plan document Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-08-18 0:12 ` Junio C Hamano
2023-09-22 20:05 ` Junio C Hamano
2023-09-24 13:57 ` phillip.wood123
2023-09-25 18:57 ` Junio C Hamano
2023-10-06 22:58 ` Josh Steadmon
2023-10-09 17:37 ` Josh Steadmon
2023-08-17 18:37 ` [PATCH v7 3/3] ci: run unit tests in CI Josh Steadmon
2023-08-17 20:38 ` [PATCH v7 0/3] Add unit test framework and project plan Junio C Hamano
2023-08-24 20:11 ` Josh Steadmon
2023-09-13 18:14 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 " Josh Steadmon
2023-10-09 22:21 ` [PATCH v8 1/3] unit tests: Add a project plan document Josh Steadmon
2023-10-10 8:57 ` Oswald Buddenhagen
2023-10-11 21:14 ` Josh Steadmon
2023-10-11 23:05 ` Oswald Buddenhagen
2023-11-01 17:31 ` Josh Steadmon
2023-10-27 20:12 ` Christian Couder [this message]
2023-11-01 17:47 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-09 22:21 ` [PATCH v8 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-10-11 21:42 ` Junio C Hamano
2023-10-16 13:43 ` [PATCH v8 2.5/3] fixup! " Phillip Wood
2023-10-16 16:41 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:48 ` Junio C Hamano
2023-11-01 17:54 ` Josh Steadmon
2023-11-01 23:49 ` Junio C Hamano
2023-10-27 20:15 ` [PATCH v8 2/3] " Christian Couder
2023-11-01 22:54 ` Josh Steadmon
2023-10-09 22:21 ` [PATCH v8 3/3] ci: run unit tests in CI Josh Steadmon
2023-10-09 23:50 ` [PATCH v8 0/3] Add unit test framework and project plan Junio C Hamano
2023-10-19 15:21 ` [PATCH 0/3] CMake unit test fixups Phillip Wood
2023-10-19 15:21 ` [PATCH 1/3] fixup! cmake: also build unit tests Phillip Wood
2023-10-19 15:21 ` [PATCH 2/3] fixup! artifacts-tar: when including `.dll` files, don't forget the unit-tests Phillip Wood
2023-10-19 15:21 ` [PATCH 3/3] fixup! cmake: handle also unit tests Phillip Wood
2023-10-19 19:19 ` [PATCH 0/3] CMake unit test fixups Junio C Hamano
2023-10-16 10:07 ` [PATCH v8 0/3] Add unit test framework and project plan phillip.wood123
2023-11-01 23:09 ` Josh Steadmon
2023-10-27 20:26 ` Christian Couder
2023-11-01 23:31 ` [PATCH v9 " Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-03 21:54 ` Christian Couder
2023-11-09 17:51 ` Josh Steadmon
2023-11-01 23:31 ` [PATCH v9 3/3] ci: run unit tests in CI Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 0/3] Add unit test framework and project plan Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 1/3] unit tests: Add a project plan document Josh Steadmon
2023-11-09 23:15 ` Junio C Hamano
2023-11-09 18:50 ` [PATCH v10 2/3] unit tests: add TAP unit test framework Josh Steadmon
2023-11-09 18:50 ` [PATCH v10 3/3] ci: run unit tests in CI Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAP8UFD26X4MPbJs4KfNOgicLMb-wiuFZj3Hw17acMmmc_=vcqQ@mail.gmail.com' \
--to=christian.couder@gmail.com \
--cc=calvinwan@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=linusa@google.com \
--cc=phillip.wood123@gmail.com \
--cc=rsbecker@nexbridge.com \
--cc=steadmon@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).