Re: A "why TAP?" manifesto

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Thu, Mar 24 2022, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> == It "can be parsed"
>> == Human readable
>> == It's trivial to produce
>> == It's more flexible
>> == It doesn't matter if nobody else uses it
>> == Everyone uses it
>
> I agree with all of what you said above.
>
> But do non-human readers really want to read output from an
> "--immediate" session?  Don't they rather see the whole thing?

That test run will still take longer, especially if you're expecting
widespread failures.

The immediate use-case I have for this is that I have unsubmitted
patches to implement a "check" mode for SANITIZE=leak, where it runs
with --immediate and "fails" tests that pass under SANITIZE=leak but
aren't marked with "TEST_PASSES_SANITIZE_LEAK=true", and passes (and
runs to completion completely) those tests that do have
"TEST_PASSES_SANITIZE_LEAK=true" but pass with SANITIZE=leak (and fails
those that would fail there, no "fail inversion").

That sort of thing is very useful to for any tests where we mark certain
other tests as being OK under a given mode, and except there to be a 1=1
correspondance.

The reason for the --immediate there is an optimization, it takes a long
time to run with SANITIZE=leak, and if any single failure is enough to
declare the test bad --immediate is useful.

Now, the TAP tooling is quite tolerant of such bad output, another point
in its favor over something like JUnit (I'm assuming most libraries
using it would pass it through a generic XML parser, whose error
handling is either "ok" or "ARGHL!" :).

I.e. it'll give you a full parse, but just have a minor complaint about
the lacking end marker, getting rid of that output & parse nit is nice.

> The only "Huh?" I had with the original patch that started the
> thread was that TAP output currently does not work well with the
> "--verbose" option, and I've never run our tests with the
> "--immediate" option without the "--verbose" option to see where and
> how the first breakage happens and what is left behind in the trash
> directory, i.e. to bootstrap an interactive debugging session.
>
> But not being useful for the use case of human reader running
> interactively to get the leftover state does not mean improvement
> for other use cases is not welcome.

I was sad to see that ability go away, without the patches I referenced
running through a TAP parser with --verbose works for almost all ouf our
test suite, so IMO we were a bit overzelous in adding that early abort
for it.

But I do have patches to make it pass 100% guaranteed, so depending on
your interest ... :)

I was planning to queue that behind some of the outstanding test lib
changes it would conflict with, e.g. this one & the outstanding CI
topics.