Re: [PATCH 1/6] Make test-lib.sh emit valid TAP format

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Fri, May 7, 2010 at 20:17, Jakub Narebski <jnareb@gmail.com> wrote:
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> TAP, the the Test Anything Protocol, is a simple text-based interface
>> between testing modules in a test harness. test-lib.sh's output was
>> already very close to being valid TAP. This change brings it all the
>> way there.
>>
>> The advantage of using TAP is that any program that reads the format
>> (a "test harness") can run the tests. The most popular of these is the
>> prove(1) utility that comes with Perl. It can run tests in parallel,
>> display colored output, format the output to console, file, HTML etc.,
>> and much more.
>>
>> On my quad Xeon server running the test suite with `make test` takes
>> 21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
>> 5 minutes.
>>
>> With parallel tests the whole test suite doesn't have to stall because
>> tests like t3404-rebase-interactive.sh take a long time.
>
> I would have thought that it would be better for git test suite to
> enable TAP output with --tap switch.  On the other hand changing
> output to TAP, replacing old format, would be less code to maintain.

That was my thinking as well. TAP is human readable like the old
format, the only difference is that TAP is standardized and can be
read by a wide variety of existing programs. For comparison:

Before the patch:

    $ ./t0005-signals.sh
    *   ok 1: sigchain works
    * passed all 1 test(s)

After:

    $ ./t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

Or alternatively:

    $ prove ./t0005-signals.sh
    ./t0005-signals.sh .. ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.01
cusr  0.02 csys =  0.06 CPU)
    Result: PASS

> I am not sure if testing with 'prove' and TAP output is compatibile
> with all current git test suite options, i.e. --debug, --tee,
> --verbose and --immediate, and with GIT_SKIP_TESTS environmental
> variable.

If you pass things like the --verbose output through prove it'll
work. This is because TAP specifies that any unknown output can be
ignored (i.e. lines that don't match /^(?:not)? ok/).

> Also valuable way of checking where the error occurs in the test,
> namely 'sh -x ./tXXXX-test.sh' would not work, I think, with
> 'prove'.

This doesn't break any method of currently running tests (e.g. running
them manually). You don't *have* to run prove, you just can if you
want to.

> Note also that having Perl (and 'prove') installed is not requirement
> for git runnig, and I think it should not be requirement for git
> development.

It should not be. Using fancy TAP formatting tools (whether it's prove
or something else) is purely optional.

> P.S. I think that such series would be better with the cover letter.

I'll make sure to submit one in the future if I send a large patch
series again, sorry.