git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: "Carlo Marcelo Arenas Belón" <carenas@gmail.com>
Cc: git@vger.kernel.org, alban.gruin@gmail.com, congdanhqx@gmail.com,
	johannes.schindelin@gmx.de
Subject: Re: [RFC PATCH] t: move metadata into TAP test description
Date: Fri, 15 May 2020 09:38:06 -0700	[thread overview]
Message-ID: <xmqqpnb5m9w1.fsf@gitster.c.googlers.com> (raw)
In-Reply-To: <20200515150041.22873-1-carenas@gmail.com> ("Carlo Marcelo Arenas =?utf-8?Q?Bel=C3=B3n=22's?= message of "Fri, 15 May 2020 08:00:41 -0700")

Carlo Marcelo Arenas Belón  <carenas@gmail.com> writes:

> 662f9cf154 (tests: when run in Bash, annotate test failures with file
> name/line number, 2020-04-11) adds metadata to the TAP output to help
> identify the location of the failed test, but does it in a way that
> break the TAP format and therefore confuses prove.
>
> Move the metadata to the description to workaround the issue and
> change the regex from 676eb0c1ce (ci: add a problem matcher for GitHub
> Actions, 2020-04-11) to match.
>
> Reported-by: Alban Gruin <alban.gruin@gmail.com>
> ---
>  ci/git-problem-matcher.json | 10 +++++-----
>  t/test-lib.sh               |  6 +++---
>  2 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/ci/git-problem-matcher.json b/ci/git-problem-matcher.json
> index 506dfbd97f..e10e88bba1 100644
> --- a/ci/git-problem-matcher.json
> +++ b/ci/git-problem-matcher.json
> @@ -4,11 +4,11 @@
>              "owner": "git-test-suite",
>              "pattern": [
>                  {
> -                    "regexp": "^([^ :]+\\.sh):(\\d+): (error|warning|info):\\s+(.*)$",
> -                    "file": 1,
> -                    "line": 2,
> -                    "severity": 3,
> -                    "message": 4
> +                    "regexp": "^(.*)(error|warning|info):\\([^ :]+\\.sh):(\\d+)\\)$",

Do we ever run our tests in ci environments with a shell other than
bash?  Would we still work fine without the "error:<F>:<L>:" suffix?
I guess we just live with degraded output in such a case, and is
much better than breaking TAP when tests are run under bash.

The tail part of this pattern (starting at error/warn/info) is tight
enough that the overly loose (.*) for the message part may still
match the "rest of the message that has arbitrary string" just fine,
but it might be less error prone to add an unusual letter (e.g. "|")
in the output between "not ok $test_count - $1" and the output from
the file_lineno() helper to make sure that we won't end up eating a
word "error" etc. at the end of the test title.  I dunno.

> +                    "file": 3,
> +                    "line": 4,
> +                    "severity": 2,
> +                    "message": 1
>                  }
>              ]
>          }
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index baf94546da..d5f59ab3bf 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -682,8 +682,8 @@ file_lineno () {
>  		for i in ${!BASH_SOURCE[*]}
>  		do
>  			case $i,"${BASH_SOURCE[$i]##*/}" in
> -			0,t[0-9]*.sh) echo "t/${BASH_SOURCE[$i]}:$LINENO: ${1+$1: }"; return;;
> -			*,t[0-9]*.sh) echo "t/${BASH_SOURCE[$i]}:${BASH_LINENO[$(($i-1))]}: ${1+$1: }"; return;;
> +			0,t[0-9]*.sh) echo "(${1+$1:}t/${BASH_SOURCE[$i]}:$LINENO)"; return;;
> +			*,t[0-9]*.sh) echo "(${1+$1:}t/${BASH_SOURCE[$i]}:${BASH_LINENO[$(($i-1))]})"; return;;
>  			esac
>  		done
>  	'
> @@ -734,7 +734,7 @@ test_failure_ () {
>  		write_junit_xml_testcase "$1" "      $junit_insert"
>  	fi
>  	test_failure=$(($test_failure + 1))
> -	say_color error "$(file_lineno error)not ok $test_count - $1"
> +	say_color error "not ok $test_count - $1$(file_lineno error)"
>  	shift
>  	printf '%s\n' "$*" | sed -e 's/^/#	/'
>  	test "$immediate" = "" || { finalize_junit_xml; GIT_EXIT_OK=t; exit 1; }

  parent reply	other threads:[~2020-05-15 16:38 UTC|newest]

Thread overview: 110+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-31 12:48 [PATCH 0/5] ci: replace our Azure Pipeline by GitHub Actions Johannes Schindelin via GitGitGadget
2020-03-31 12:48 ` [PATCH 1/5] ci/lib: if CI type is unknown, show the environment variables Johannes Schindelin via GitGitGadget
2020-03-31 12:48 ` [PATCH 2/5] ci/lib: allow running in GitHub Actions Johannes Schindelin via GitGitGadget
2020-04-03  8:46   ` SZEDER Gábor
2020-04-04 20:08     ` Johannes Schindelin
2020-04-05  0:01       ` Danh Doan
2020-04-07 22:41         ` Johannes Schindelin
2020-04-10 15:31       ` SZEDER Gábor
2020-04-10 19:10         ` Junio C Hamano
2020-04-12 21:42           ` Johannes Schindelin
2020-04-12 22:12             ` Junio C Hamano
2020-04-12 22:25               ` Junio C Hamano
2020-03-31 12:48 ` [PATCH 3/5] ci: configure GitHub Actions for CI/PR Johannes Schindelin via GitGitGadget
2020-04-03 22:07   ` SZEDER Gábor
2020-04-04 17:59     ` Johannes Schindelin
2020-04-04 23:55       ` Danh Doan
2020-03-31 12:48 ` [PATCH 4/5] README: add a build badge for the GitHub Actions runs Johannes Schindelin via GitGitGadget
2020-03-31 12:48 ` [PATCH 5/5] ci: retire the Azure Pipelines definition Johannes Schindelin via GitGitGadget
2020-03-31 14:46 ` [PATCH 0/5] ci: replace our Azure Pipeline by GitHub Actions Danh Doan
2020-03-31 18:47   ` Junio C Hamano
2020-04-01 15:55 ` [PATCH v2 0/6] " Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 1/6] ci/lib: if CI type is unknown, show the environment variables Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 2/6] ci/lib: allow running in GitHub Actions Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 3/6] ci/lib: set TERM environment variable if not exist Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 4/6] ci: configure GitHub Actions for CI/PR Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 5/6] README: add a build badge for the GitHub Actions runs Đoàn Trần Công Danh
2020-04-01 15:55   ` [PATCH v2 6/6] ci: retire the Azure Pipelines definition Đoàn Trần Công Danh
2020-04-01 21:23   ` [PATCH v2 0/6] ci: replace our Azure Pipeline by GitHub Actions Junio C Hamano
2020-04-02  0:14     ` Danh Doan
2020-04-02 13:16   ` [PATCH v3 " Đoàn Trần Công Danh
2020-04-02 13:16     ` [PATCH v3 1/6] ci/lib: if CI type is unknown, show the environment variables Đoàn Trần Công Danh
2020-04-02 13:16     ` [PATCH v3 2/6] ci/lib: allow running in GitHub Actions Đoàn Trần Công Danh
2020-04-02 13:16     ` [PATCH v3 3/6] ci/lib: set TERM environment variable if not exist Đoàn Trần Công Danh
2020-04-02 13:16     ` [PATCH v3 4/6] ci: configure GitHub Actions for CI/PR Đoàn Trần Công Danh
2020-04-02 13:16     ` [PATCH v3 5/6] README: add a build badge for the GitHub Actions runs Đoàn Trần Công Danh
2020-04-02 13:17     ` [PATCH v3 6/6] ci: retire the Azure Pipelines definition Đoàn Trần Công Danh
2020-04-08  4:05 ` [PATCH v4 00/12] ci: replace our Azure Pipeline by GitHub Actions Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 01/12] ci/lib: if CI type is unknown, show the environment variables Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 02/12] ci/lib: allow running in GitHub Actions Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 03/12] ci/lib: set TERM environment variable if not exist Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 04/12] ci: fix the `jobname` of the `GETTEXT_POISON` job Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 05/12] ci: explicit install all required packages Đoàn Trần Công Danh
2020-04-10 15:53     ` SZEDER Gábor
2020-04-10 16:07       ` Danh Doan
2020-04-10 16:21         ` Junio C Hamano
2020-04-08  4:05   ` [PATCH v4 06/12] ci: run gem with sudo to install asciidoctor Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 07/12] ci: configure GitHub Actions for CI/PR Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 08/12] README: add a build badge for the GitHub Actions runs Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 09/12] ci: retire the Azure Pipelines definition Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 10/12] tests: when run in Bash, annotate test failures with file name/line number Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 11/12] ci: add a problem matcher for GitHub Actions Đoàn Trần Công Danh
2020-04-08  4:05   ` [PATCH v4 12/12] ci: let GitHub Actions upload failed tests' directories Đoàn Trần Công Danh
2020-04-09 21:19   ` [PATCH v4 00/12] ci: replace our Azure Pipeline by GitHub Actions Junio C Hamano
2020-04-10 14:34     ` Johannes Schindelin
2020-04-10 14:37       ` Johannes Schindelin
2020-04-10 17:35         ` Danh Doan
2020-04-10 15:42       ` Junio C Hamano
2020-04-10 17:41         ` Danh Doan
2020-04-16  0:49           ` Junio C Hamano
2020-04-16  1:28             ` fixing ci failure of 'pu' with the es/bugreport topic Junio C Hamano
2020-04-16  1:55               ` Emily Shaffer
2020-04-16  3:20                 ` Junio C Hamano
2020-04-16  3:45                 ` Elijah Newren
2020-04-16  4:10                   ` Emily Shaffer
2020-04-16  4:57                     ` Junio C Hamano
2020-04-16 11:26                   ` Danh Doan
2020-04-16 12:05                     ` Johannes Schindelin
2020-04-16 12:08             ` [PATCH v4 00/12] ci: replace our Azure Pipeline by GitHub Actions Johannes Schindelin
2020-04-16 15:55               ` Junio C Hamano
2020-04-10 17:18 ` [PATCH v5 " Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 01/12] ci/lib: if CI type is unknown, show the environment variables Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 02/12] ci/lib: allow running in GitHub Actions Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 03/12] ci/lib: set TERM environment variable if not exist Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 04/12] ci: fix the `jobname` of the `GETTEXT_POISON` job Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 05/12] ci: explicit install all required packages Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 06/12] ci: run gem with sudo to install asciidoctor Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 07/12] ci: configure GitHub Actions for CI/PR Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 08/12] README: add a build badge for the GitHub Actions runs Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 09/12] ci: retire the Azure Pipelines definition Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 10/12] tests: when run in Bash, annotate test failures with file name/line number Đoàn Trần Công Danh
2020-05-04 17:46     ` Carlo Marcelo Arenas Belón
2020-05-04 23:25       ` Danh Doan
2020-05-05  0:35         ` Junio C Hamano
2020-05-06  7:30           ` Carlo Marcelo Arenas Belón
2020-05-06 12:54             ` Johannes Schindelin
2020-05-06 13:46               ` Carlo Marcelo Arenas Belón
2020-05-06 14:33                 ` Johannes Schindelin
2020-05-07  6:10                   ` Carlo Marcelo Arenas Belón
2020-05-06 16:33                 ` Junio C Hamano
2020-05-05  0:54         ` Carlo Marcelo Arenas Belón
2020-05-15 13:16     ` Alban Gruin
2020-05-15 15:00       ` [RFC PATCH] t: move metadata into TAP test description Carlo Marcelo Arenas Belón
2020-05-15 15:08         ` Eric Sunshine
2020-05-15 15:38         ` Alban Gruin
2020-05-15 15:45           ` Carlo Marcelo Arenas Belón
2020-05-15 16:50             ` Junio C Hamano
2020-05-15 17:14               ` Carlo Marcelo Arenas Belón
2020-05-15 17:23                 ` Junio C Hamano
2020-05-15 22:42                   ` Johannes Schindelin
2020-05-15 22:57                     ` Junio C Hamano
2020-05-15 17:21               ` [PATCH 0/2] Unbreak TAP output under bash Junio C Hamano
2020-05-15 17:21                 ` [PATCH 1/2] Revert "t/test_lib: avoid naked bash arrays in file_lineno" Junio C Hamano
2020-05-15 17:21                 ` [PATCH 2/2] Revert "tests: when run in Bash, annotate test failures with file name/line number" Junio C Hamano
2020-05-15 16:38         ` Junio C Hamano [this message]
2020-05-15 17:22           ` [RFC PATCH] t: move metadata into TAP test description Carlo Marcelo Arenas Belón
2020-05-15 19:04         ` Alban Gruin
2020-05-15 15:28       ` [PATCH v5 10/12] tests: when run in Bash, annotate test failures with file name/line number Carlo Marcelo Arenas Belón
2020-05-15 15:33         ` Alban Gruin
2020-04-10 17:18   ` [PATCH v5 11/12] ci: add a problem matcher for GitHub Actions Đoàn Trần Công Danh
2020-04-10 17:18   ` [PATCH v5 12/12] ci: let GitHub Actions upload failed tests' directories Đoàn Trần Công Danh

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqqpnb5m9w1.fsf@gitster.c.googlers.com \
    --to=gitster@pobox.com \
    --cc=alban.gruin@gmail.com \
    --cc=carenas@gmail.com \
    --cc=congdanhqx@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=johannes.schindelin@gmx.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).