Re: [PATCH 0/6 + 1] test-lib: make --verbose output valid TAP

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Tue, Mar 09 2021, SZEDER Gábor wrote:

> On Tue, Mar 09, 2021 at 05:02:12PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> The eventual goal not included in this series is to have multiple
>> output targets, and e.g. convert ci/print-test-failures.sh to use a
>> TAP parser.
>> 
>> Machine-readable "TAP --verbose -x" output can bring us a lot of nice
>> things down the line, I have some local WIP code that's a smarter
>> version of ci/print-test-failures.sh that knows how to spew out only
>> the output relevant to the failing test(s).
>
> I wonder what you mean by the word "relevant" here, as I can't imagine
> how you could possibly identify what is relevant for a failing test
> and what isn't.  If you didn't at all meant "relevant", but that it
> will show only the output of the failing test(s), then this is a bad
> example.  Our test cases depend too much on previous test cases, and a
> failure of one test can be the result of a change in a previous
> successful tests.
>
> Therefore, any such change to 'ci/print-test-failures.sh' will have my
> firm NACK.

On e.g. the github CI every step in the run is an collapsable button, so
we could have our cake and eat it too here.

It seems to me like a sane default would be to have an equivalent to
"print-test-failures.sh" that only prints the --verbose output for the
failing tests, and a "print-all-output-for-failing-tests.sh" or whatever
which gave you the full output.

Even then, there seem to me to be some low hanging fruit for
abbreviating even that output. E.g. if you we have 100 tests and we
failed only on the 5th, isn't the --verbose -x output up to and
including the 5th going to be enough, or do we need it for the other 95?

There's also ways to unambiguously abbreviate the output, and that's now
easy to implement as shown in my other just-sent reply to you.

E.g. a lot of our test setup is really repetitive, so under -x you're
sometimes looking at the same 5-10+ lines of test setup. We could just
do a couple of passes over the output and change output like:

    + foo
    + bar
    + he
    + foo
    + bar
    + llo
    + foo
    + bar
    + world

to:

    $1 = +foo
         +bar
    + he
    $1
    + llo
    $1
    + world

Or whatever, I'm not advocating for the specific syntax there, just for
the output to be nicer.