git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Denton Liu <liu.denton@gmail.com>,
	Johannes Schindelin <johannes.schindelin@gmx.de>,
	Junio C Hamano <gitster@pobox.com>,
	Johannes Schindelin <johannes.schindelin@gmx.de>
Subject: [PATCH v3 10/13] test-tool run-command: learn to run (parts of) the testsuite
Date: Fri, 04 Oct 2019 08:09:33 -0700 (PDT)	[thread overview]
Message-ID: <d2c4973431e36b7b7b9f05ae30d96541d5fc1938.1570201763.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.288.v3.git.gitgitgadget@gmail.com>

From: Johannes Schindelin <johannes.schindelin@gmx.de>

Git for Windows jumps through hoops to provide a development environment
that allows to build Git and to run its test suite. To that end, an
entire MSYS2 system, including GNU make and GCC is offered as "the Git
for Windows SDK". It does come at a price: an initial download of said
SDK weighs in with several hundreds of megabytes, and the unpacked SDK
occupies ~2GB of disk space.

A much more native development environment on Windows is Visual Studio.
To help contributors use that environment, we already have a Makefile
target `vcxproj` that generates a commit with project files (and other
generated files), and Git for Windows' `vs/master` branch is
continuously re-generated using that target.

The idea is to allow building Git in Visual Studio, and to run
individual tests using a Portable Git.

The one missing thing is a way to run the entire test suite: neither
`make` nor `prove` are required to run Git, therefore Git for Windows
does not support those commands in the Portable Git.

To help with that, add a simple test helper that exercises the
`run_processes_parallel()` function to allow for running test scripts in
parallel (which is really necessary, especially on Windows, as Git's
test suite takes such a long time to run).

This will also come in handy for the upcoming change to our Azure
Pipeline: we will use this helper in a Portable Git to test the Visual
Studio build of Git.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/helper/test-run-command.c | 153 ++++++++++++++++++++++++++++++++++++
 1 file changed, 153 insertions(+)

diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c
index 2cc93bb69c..ead6dc611a 100644
--- a/t/helper/test-run-command.c
+++ b/t/helper/test-run-command.c
@@ -10,9 +10,14 @@
 
 #include "test-tool.h"
 #include "git-compat-util.h"
+#include "cache.h"
 #include "run-command.h"
 #include "argv-array.h"
 #include "strbuf.h"
+#include "parse-options.h"
+#include "string-list.h"
+#include "thread-utils.h"
+#include "wildmatch.h"
 #include <string.h>
 #include <errno.h>
 
@@ -50,11 +55,159 @@ static int task_finished(int result,
 	return 1;
 }
 
+struct testsuite {
+	struct string_list tests, failed;
+	int next;
+	int quiet, immediate, verbose, verbose_log, trace, write_junit_xml;
+};
+#define TESTSUITE_INIT \
+	{ STRING_LIST_INIT_DUP, STRING_LIST_INIT_DUP, -1, 0, 0, 0, 0, 0, 0 }
+
+static int next_test(struct child_process *cp, struct strbuf *err, void *cb,
+		     void **task_cb)
+{
+	struct testsuite *suite = cb;
+	const char *test;
+	if (suite->next >= suite->tests.nr)
+		return 0;
+
+	test = suite->tests.items[suite->next++].string;
+	argv_array_pushl(&cp->args, "sh", test, NULL);
+	if (suite->quiet)
+		argv_array_push(&cp->args, "--quiet");
+	if (suite->immediate)
+		argv_array_push(&cp->args, "-i");
+	if (suite->verbose)
+		argv_array_push(&cp->args, "-v");
+	if (suite->verbose_log)
+		argv_array_push(&cp->args, "-V");
+	if (suite->trace)
+		argv_array_push(&cp->args, "-x");
+	if (suite->write_junit_xml)
+		argv_array_push(&cp->args, "--write-junit-xml");
+
+	strbuf_addf(err, "Output of '%s':\n", test);
+	*task_cb = (void *)test;
+
+	return 1;
+}
+
+static int test_finished(int result, struct strbuf *err, void *cb,
+			 void *task_cb)
+{
+	struct testsuite *suite = cb;
+	const char *name = (const char *)task_cb;
+
+	if (result)
+		string_list_append(&suite->failed, name);
+
+	strbuf_addf(err, "%s: '%s'\n", result ? "FAIL" : "SUCCESS", name);
+
+	return 0;
+}
+
+static int test_failed(struct strbuf *out, void *cb, void *task_cb)
+{
+	struct testsuite *suite = cb;
+	const char *name = (const char *)task_cb;
+
+	string_list_append(&suite->failed, name);
+	strbuf_addf(out, "FAILED TO START: '%s'\n", name);
+
+	return 0;
+}
+
+static const char * const testsuite_usage[] = {
+	"test-run-command testsuite [<options>] [<pattern>...]",
+	NULL
+};
+
+static int testsuite(int argc, const char **argv)
+{
+	struct testsuite suite = TESTSUITE_INIT;
+	int max_jobs = 1, i, ret;
+	DIR *dir;
+	struct dirent *d;
+	struct option options[] = {
+		OPT_BOOL('i', "immediate", &suite.immediate,
+			 "stop at first failed test case(s)"),
+		OPT_INTEGER('j', "jobs", &max_jobs, "run <N> jobs in parallel"),
+		OPT_BOOL('q', "quiet", &suite.quiet, "be terse"),
+		OPT_BOOL('v', "verbose", &suite.verbose, "be verbose"),
+		OPT_BOOL('V', "verbose-log", &suite.verbose_log,
+			 "be verbose, redirected to a file"),
+		OPT_BOOL('x', "trace", &suite.trace, "trace shell commands"),
+		OPT_BOOL(0, "write-junit-xml", &suite.write_junit_xml,
+			 "write JUnit-style XML files"),
+		OPT_END()
+	};
+
+	memset(&suite, 0, sizeof(suite));
+	suite.tests.strdup_strings = suite.failed.strdup_strings = 1;
+
+	argc = parse_options(argc, argv, NULL, options,
+			testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
+
+	if (max_jobs <= 0)
+		max_jobs = online_cpus();
+
+	dir = opendir(".");
+	if (!dir)
+		die("Could not open the current directory");
+	while ((d = readdir(dir))) {
+		const char *p = d->d_name;
+
+		if (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||
+		    !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||
+		    !ends_with(p, ".sh"))
+			continue;
+
+		/* No pattern: match all */
+		if (!argc) {
+			string_list_append(&suite.tests, p);
+			continue;
+		}
+
+		for (i = 0; i < argc; i++)
+			if (!wildmatch(argv[i], p, 0)) {
+				string_list_append(&suite.tests, p);
+				break;
+			}
+	}
+	closedir(dir);
+
+	if (!suite.tests.nr)
+		die("No tests match!");
+	if (max_jobs > suite.tests.nr)
+		max_jobs = suite.tests.nr;
+
+	fprintf(stderr, "Running %d tests (%d at a time)\n",
+		suite.tests.nr, max_jobs);
+
+	ret = run_processes_parallel(max_jobs, next_test, test_failed,
+				     test_finished, &suite);
+
+	if (suite.failed.nr > 0) {
+		ret = 1;
+		fprintf(stderr, "%d tests failed:\n\n", suite.failed.nr);
+		for (i = 0; i < suite.failed.nr; i++)
+			fprintf(stderr, "\t%s\n", suite.failed.items[i].string);
+	}
+
+	string_list_clear(&suite.tests, 0);
+	string_list_clear(&suite.failed, 0);
+
+	return !!ret;
+}
+
 int cmd__run_command(int argc, const char **argv)
 {
 	struct child_process proc = CHILD_PROCESS_INIT;
 	int jobs;
 
+	if (argc > 1 && !strcmp(argv[1], "testsuite"))
+		exit(testsuite(argc - 1, argv + 1));
+
 	if (argc < 3)
 		return 1;
 	while (!strcmp(argv[1], "env")) {
-- 
gitgitgadget


  parent reply	other threads:[~2019-10-04 15:09 UTC|newest]

Thread overview: 72+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-26  8:30 [PATCH 00/13] ci: include a Visual Studio build & test in our Azure Pipeline Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 01/13] push: do not pretend to return `int` from `die_push_simple()` Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 02/13] msvc: avoid using minus operator on unsigned types Johannes Schindelin via GitGitGadget
2019-09-26 17:20   ` Denton Liu
2019-09-26 21:01     ` Johannes Schindelin
2019-09-26 23:57       ` Denton Liu
2019-09-30  9:50         ` Johannes Schindelin
2019-09-26  8:30 ` [PATCH 03/13] winansi: use FLEX_ARRAY to avoid compiler warning Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 04/13] compat/win32/path-utils.h: add #include guards Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 05/13] msvc: ignore some libraries when linking Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 06/13] msvc: handle DEVELOPER=1 Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 07/13] msvc: work around a bug in GetEnvironmentVariable() Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 08/13] vcxproj: only copy `git-remote-http.exe` once it was built Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 09/13] vcxproj: include more generated files Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 10/13] test-tool run-command: learn to run (parts of) the testsuite Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 11/13] tests: let --immediate and --write-junit-xml play well together Johannes Schindelin via GitGitGadget
2019-09-28 22:22   ` Junio C Hamano
2019-09-30  9:52     ` Johannes Schindelin
2019-09-26  8:30 ` [PATCH 12/13] ci: really use shallow clones on Azure Pipelines Johannes Schindelin via GitGitGadget
2019-09-26  8:30 ` [PATCH 13/13] ci: also build and test with MS Visual Studio " Johannes Schindelin via GitGitGadget
2019-09-30  9:55 ` [PATCH v2 00/13] ci: include a Visual Studio build & test in our Azure Pipeline Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 02/13] msvc: avoid using minus operator on unsigned types Johannes Schindelin via GitGitGadget
2019-10-03 22:44     ` Junio C Hamano
2019-10-04  9:55       ` Johannes Schindelin
2019-10-04 17:09         ` Johannes Sixt
2019-10-04 21:24           ` Johannes Schindelin
2019-10-06  0:02             ` Junio C Hamano
2019-10-06 10:53               ` Johannes Sixt
2019-10-08 12:04                 ` Johannes Schindelin
2019-10-08 21:13                   ` Johannes Sixt
2019-09-30  9:55   ` [PATCH v2 01/13] push: do not pretend to return `int` from `die_push_simple()` Johannes Schindelin via GitGitGadget
2019-10-03 22:37     ` Junio C Hamano
2019-10-04  9:36       ` Johannes Schindelin
2019-09-30  9:55   ` [PATCH v2 03/13] winansi: use FLEX_ARRAY to avoid compiler warning Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 04/13] compat/win32/path-utils.h: add #include guards Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 05/13] msvc: ignore some libraries when linking Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 06/13] msvc: handle DEVELOPER=1 Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 07/13] msvc: work around a bug in GetEnvironmentVariable() Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 09/13] vcxproj: include more generated files Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 08/13] vcxproj: only copy `git-remote-http.exe` once it was built Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 10/13] test-tool run-command: learn to run (parts of) the testsuite Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 11/13] tests: let --immediate and --write-junit-xml play well together Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 13/13] ci: also build and test with MS Visual Studio on Azure Pipelines Johannes Schindelin via GitGitGadget
2019-09-30  9:55   ` [PATCH v2 12/13] ci: really use shallow clones " Johannes Schindelin via GitGitGadget
2019-10-04 15:09   ` [PATCH v3 00/13] ci: include a Visual Studio build & test in our Azure Pipeline Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 01/13] push: do not pretend to return `int` from `die_push_simple()` Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 02/13] msvc: avoid using minus operator on unsigned types Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 03/13] winansi: use FLEX_ARRAY to avoid compiler warning Johannes Schindelin via GitGitGadget
2019-10-07 19:16       ` Alban Gruin
2019-10-04 15:09     ` [PATCH v3 04/13] compat/win32/path-utils.h: add #include guards Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 05/13] msvc: ignore some libraries when linking Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 06/13] msvc: handle DEVELOPER=1 Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 07/13] msvc: work around a bug in GetEnvironmentVariable() Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 08/13] vcxproj: only copy `git-remote-http.exe` once it was built Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 09/13] vcxproj: include more generated files Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` Johannes Schindelin via GitGitGadget [this message]
2019-10-04 15:09     ` [PATCH v3 11/13] tests: let --immediate and --write-junit-xml play well together Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 12/13] ci: really use shallow clones on Azure Pipelines Johannes Schindelin via GitGitGadget
2019-10-04 15:09     ` [PATCH v3 13/13] ci: also build and test with MS Visual Studio " Johannes Schindelin via GitGitGadget
2019-10-06  0:19     ` [PATCH v3 00/13] ci: include a Visual Studio build & test in our Azure Pipeline Junio C Hamano
2019-10-06 10:45       ` Johannes Schindelin
2019-10-06 20:38         ` Johannes Schindelin
2019-10-07  1:14           ` Junio C Hamano
2019-10-07 21:51             ` Johannes Schindelin
2019-10-08  2:19               ` Junio C Hamano
2019-10-08 12:46                 ` Johannes Schindelin
2019-10-09 13:57                   ` Philip Oakley
2019-10-10  9:03                     ` Johannes Schindelin
2019-10-10 10:12                       ` Philip Oakley
2019-10-07  0:59         ` Junio C Hamano
2019-10-07 16:08           ` Thomas Gummerer
2019-10-11 22:06             ` Johannes Schindelin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d2c4973431e36b7b7b9f05ae30d96541d5fc1938.1570201763.git.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=johannes.schindelin@gmx.de \
    --cc=liu.denton@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).