Re: [PATCH 6/6] fixup! reftable: rest of library

[Jeff King <peff@peff.net>]

On Wed, Dec 02, 2020 at 12:01:49PM +0100, Han-Wen Nienhuys wrote:

> > I'm not sure if the long-term goal is to have this opaque unit-test
> > program or not. If it is, I was likewise going to suggest that its
> > ad-hoc output be replaced with TAP. But it looks like on your branch
> > that "test-tool reftable" does not produce output at all. So I may be a
> > bit behind on what the current state and forward plans are...
> 
> The most important requirement is that something fails if the
> unittests don't work. I surmised that that meant running tests from
> test-helper in some way, so this is what happens now. Looking for
> "unit.?test" across the git codebase didn't turn up much info. Happy
> to explore other solutions if you can give me pointers.

Normally we do a combination of:

  - git plumbing exposes scriptable commands, and we make sure those
    work. This is a much coarser-grained unit than testing individual C
    functions, but produces resilient tests because that interface is
    user-visible and stable (and in fact seeing test breakages is often
    a sign that one will be breaking real users).

    These are obviously driven by shell script tests emulating what
    users might run.

  - for unit tests of individual data types where it's not appropriate
    to have a user-facing command, we often add a test-tool helper that
    exposes specific functions (e.g., t/helper/test-date.c exposes date
    parsing and formatting routines, test-oid-array.c exercises methods
    of that data type, etc).

    The C parts of these are usually generic, and then are driven by
    shell scripts providing the actual data in individual tests (and
    handling success/failure, skipping tests, etc).

    This is still coarser than you might get unit-testing inside C.
    E.g., you could not generally check the difference between passing
    an empty array vs NULL to a function. But our philosophy has
    generally been _not_ to test at that level. The C interfaces are
    internal, and Git is not that big a project. If there's a function
    whose caller does something unexpected, it's usually easier to fix
    the caller and add a test that triggers the caller's code path.

I agree that a test that simply runs a bunch of C code and either exits
with failure or success is better than nothing, in the sense of finding
tests. And wrapping that with a single test_expect_success does that.
But it's unfortunate that we get none of the fine-grained control that
the test suite provides, nor much support in debugging failing tests.

One middle ground would be for a battery of C tests to output
TAP-compatible output (outputting "ok 1 - foo works", and "not ok 2 -
bar does not work", etc). That at least gives more info on what fails,
and does it in a way that the rest of the test suite can interpret
(though not manipulate).

-Peff