[PATCH v4 0/8] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests

["SZEDER Gábor" <szeder.dev@gmail.com>]

To recap: this patch series tries to make reproducing rare failures in
flaky tests easier: it adds the '--stress' option to our test library
to run the test script repeatedly in multiple parallel jobs, in the
hope that the increased load creates enough variance in the timing of
the test's commands that such a failure is eventually triggered.

There is an issue with v3: by parsing all command line options before
re-executing the test script for '--tee' or '--verbose-log', we not
only parse the options twice (which is harmless), but check the
shell's version for untraceable test scripts twice as well.
Consequently, the warning about ignoring '-x' are issued twice:

  $ ./t1510-repo-setup.sh -x --tee
  warning: ignoring -x; './t1510-repo-setup.sh' is untraceable without BASH_XTRACEFD
  warning: ignoring -x; './t1510-repo-setup.sh' is untraceable without BASH_XTRACEFD
  ok 1 - #0: nonbare repo, no explicit configuration
  <...>

Furthermore, when TEST_SHELL_PATH is not /bin/sh but is set to Bash
v4.1 or later, then the above command issues one warning as it's run
with /bin/sh from the shebang line, but then re-execs itself with Bash
and does print the '-x' tracing output as well.

This version of the patch series fixes the above issue by checking the
shell's version after '--tee' and '--verbose-log' re-execue the test
script.  To do so it makes more sense to move the 'test-lib: extract
Bash version check for '-x' tracing' patch to the beginning of the
series.  This move, however, made the range-diff quite noisy, hence
the interdiff below, as it describes the changes since v3 much better.

It's based on 'sg/test-bash-version-fix'.


SZEDER Gábor (8):
  test-lib: translate SIGTERM and SIGHUP to an exit
  test-lib: extract Bash version check for '-x' tracing
  test-lib: parse options in a for loop to keep $@ intact
  test-lib: parse command line options earlier
  test-lib: consolidate naming of test-results paths
  test-lib: set $TRASH_DIRECTORY earlier
  test-lib-functions: introduce the 'test_set_port' helper function
  test-lib: add the '--stress' option to run a test repeatedly under
    load

 t/README                 |  19 +-
 t/lib-git-daemon.sh      |   2 +-
 t/lib-git-p4.sh          |   9 +-
 t/lib-git-svn.sh         |   2 +-
 t/lib-httpd.sh           |   2 +-
 t/t0410-partial-clone.sh |   1 -
 t/t5512-ls-remote.sh     |   2 +-
 t/test-lib-functions.sh  |  39 +++++
 t/test-lib.sh            | 365 ++++++++++++++++++++++++++-------------
 9 files changed, 308 insertions(+), 133 deletions(-)

Interdiff:
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 3f0b0c07df..a1abb1177a 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -180,29 +180,6 @@ then
 	immediate=t
 fi
 
-if test -n "$trace" && test -n "$test_untraceable"
-then
-	# '-x' tracing requested, but this test script can't be reliably
-	# traced, unless it is run with a Bash version supporting
-	# BASH_XTRACEFD (introduced in Bash v4.1).
-	if test -n "$BASH_VERSION" && eval '
-	     test ${BASH_VERSINFO[0]} -gt 4 || {
-	       test ${BASH_VERSINFO[0]} -eq 4 &&
-	       test ${BASH_VERSINFO[1]} -ge 1
-	     }
-	   '
-	then
-		: Executed by a Bash version supporting BASH_XTRACEFD.  Good.
-	else
-		echo >&2 "warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD"
-		trace=
-	fi
-fi
-if test -n "$trace" && test -z "$verbose_log"
-then
-	verbose=t
-fi
-
 TEST_STRESS_JOB_SFX="${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}"
 TEST_NAME="$(basename "$0" .sh)"
 TEST_RESULTS_DIR="$TEST_OUTPUT_DIRECTORY/test-results"
@@ -322,6 +299,34 @@ then
 	exit
 fi
 
+if test -n "$trace" && test -n "$test_untraceable"
+then
+	# '-x' tracing requested, but this test script can't be reliably
+	# traced, unless it is run with a Bash version supporting
+	# BASH_XTRACEFD (introduced in Bash v4.1).
+	#
+	# Perform this version check _after_ the test script was
+	# potentially re-executed with $TEST_SHELL_PATH for '--tee' or
+	# '--verbose-log', so the right shell is checked and the
+	# warning is issued only once.
+	if test -n "$BASH_VERSION" && eval '
+	     test ${BASH_VERSINFO[0]} -gt 4 || {
+	       test ${BASH_VERSINFO[0]} -eq 4 &&
+	       test ${BASH_VERSINFO[1]} -ge 1
+	     }
+	   '
+	then
+		: Executed by a Bash version supporting BASH_XTRACEFD.  Good.
+	else
+		echo >&2 "warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD"
+		trace=
+	fi
+fi
+if test -n "$trace" && test -z "$verbose_log"
+then
+	verbose=t
+fi
+
 # For repeatability, reset the environment to known value.
 # TERM is sanitized below, after saving color control sequences.
 LANG=C
-- 
2.20.1.151.gec613c4b75