[PATCH v6 13/14] CI: make --github-workflow-markup ci-config, off by default

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

In preceding commits the --github-workflow-markup output was made the
default under GitHub CI.

There's a few outstanding issues with that new output target however:

A. As noted in [1] it's much slower in some common cases, i.e. the
   page rendering time (under Firefox & Chrome debugging) goes from ~20s
   to on the order of 60-80s (less pronounced when not debugging, but
   still in that ballpark). See also [2] for tests on an earlier iteration
   of the series.

B. Preceding commits in this series to omit non-failing test output
   may have mitigated that somewhat, i.e. skipping "ok" and "skip" tests
   in the *.markup files (although [1] is a benchmark with that "ok"
   change).

   Per [3] there are some CI users that prefer to see only the full
   verbose output, i.e. we can't in the general case assume that we
   can emit only the trace output for the failing test, and that's
   going to give us the information we need to debug the test. It
   might be failing because of earlier setup, or from trace output
   outside of any test (i.e. in the main body of the file).

   Users such as [3] could still just consult the *.out output, but
   then we'd make them download the archive instead of viewing the
   output in the web UX.

C. Per B above and [3] an earlier change in this series modified
   the *.out output to change the *.out output inadvertently while trying
   to modify the *.markup output. See the commit message of "ci(github):
   avoid printing test case preamble twice" earlier in this series.

It thus make sense for now to make the "ci/print-test-failures.sh"
output be the "raw" one. I.e. the *.out output we'd get before
--github-workflow-markup was implemented.

To make it use ci/print-test-failures.sh we can create a
"ci/config/print-test-failures-output-type" script on the "ci-config"
branch that does e.g.:

	#!/bin/sh
	echo github

That script will (like the existing "check-ref") get the ref as an
argument, so we can in the future turn this on experimentally for
"seen", and flip the default in the future. The script can also emit
nothing to get whatever our idea is of the default (currently "raw").

Note the corresponding change to ci/print-test-failures.sh, i.e. we'll
do the right thing vis-a-vis what we "zoom in" to depending on the
setting. If it's "raw" we'd like a failed "make test" to "zoom in" to
that step, if it's "github" we'd like the to open the subsequent
"ci/print-test-failures.sh" step instead.

1. https://lore.kernel.org/git/220523.86ee0kzjix.gmgdl@evledraar.gmail.com/
2. https://lore.kernel.org/git/220222.86tucr6kz5.gmgdl@evledraar.gmail.com/
3. https://lore.kernel.org/git/20210309175249.GE3590451@szeder.dev/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 .github/workflows/main.yml | 24 ++++++++++++++++++++++++
 ci/lib.sh                  | 14 +++++++++++++-
 ci/print-test-failures.sh  |  7 +++++--
 3 files changed, 42 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 6835e942280..4195f37c1ba 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -8,6 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       enabled: ${{ steps.check-ref.outputs.enabled }}${{ steps.skip-if-redundant.outputs.enabled }}
+      print-test-failures-output-type: ${{ steps.print-test-failures-output-type.outputs.type }}
     steps:
       - name: try to clone ci-config branch
         run: |
@@ -22,6 +23,21 @@ jobs:
             config-repo &&
           cd config-repo &&
           git checkout HEAD -- ci/config || : ignore
+      - id: print-test-failures-output-type
+        name: check what output type ci/print-test-failures.sh uses
+        run: |
+          type_default=raw
+          type=$type_default
+
+          if test -x config-repo/ci/config/print-test-failures-output-type
+          then
+            type=$(config-repo/ci/config/print-test-failures-output-type '${{ github.ref }}')
+            if test -z "$type"
+            then
+              type=$type_default
+            fi
+          fi
+          echo "::set-output name=type::$type"
       - id: check-ref
         name: check whether CI is enabled for ref
         run: |
@@ -124,6 +140,8 @@ jobs:
     - uses: git-for-windows/setup-git-for-windows-sdk@v1
     - run: ci/lib.sh --test
       shell: bash
+      env:
+        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
     - name: select tests
       run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10
       shell: bash
@@ -213,6 +231,8 @@ jobs:
       run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz
     - run: ci/lib.sh --test
       shell: bash
+      env:
+        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
     - name: select tests
       run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10
       shell: bash
@@ -265,6 +285,8 @@ jobs:
     - run: ci/lib.sh --build
     - run: make
     - run: ci/lib.sh --test
+      env:
+        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
     - run: make test
       if: success()
     - run: ci/print-test-failures.sh
@@ -302,6 +324,8 @@ jobs:
     - run: make
     - run: ci/lib.sh --test
       if: success() && matrix.vector.skip-tests != 'yes'
+      env:
+        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
     - run: make test
       if: success() && matrix.vector.skip-tests != 'yes'
     - run: ci/print-test-failures.sh
diff --git a/ci/lib.sh b/ci/lib.sh
index 422399b1305..b148d3f209f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -95,6 +95,7 @@ if test -n "$GITHUB_ENV"
 then
 	echo "CONFIG: GITHUB_ENV=$GITHUB_ENV" >&2
 fi
+echo "CONFIG: GIT_CI_PTF_OUTPUT_TYPE=$GIT_CI_PTF_OUTPUT_TYPE" >&2
 echo >&2
 
 # Helper functions
@@ -189,7 +190,18 @@ MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
 case "$CI_TYPE" in
 github-actions)
 	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
-	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
+	GIT_TEST_OPTS="--verbose-log -x"
+	if test -n "$GIT_CI_PTF_OUTPUT_TYPE"
+	then
+		# For later use in ci/print-test-failures.sh
+		setenv --test GIT_CI_PTF_OUTPUT_TYPE "$GIT_CI_PTF_OUTPUT_TYPE"
+
+		case "$GIT_CI_PTF_OUTPUT_TYPE" in
+		github)
+			GIT_TEST_OPTS="$GIT_TEST_OPTS --github-workflow-markup"
+			;;
+		esac
+	fi
 	test Windows != "$RUNNER_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh
index 72fae2b0cc4..1e0738e0058 100755
--- a/ci/print-test-failures.sh
+++ b/ci/print-test-failures.sh
@@ -9,10 +9,13 @@ set -e
 . ${0%/*}/lib-tput.sh
 
 github_workflow_markup=auto
-case "$CI_TYPE" in
-github-actions)
+case "$GIT_CI_PTF_OUTPUT_TYPE" in
+github)
 	github_workflow_markup=t
 	;;
+raw)
+	github_workflow_markup=
+	;;
 esac
 
 while test $# != 0
-- 
2.36.1.1045.gf356b5617dd