Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI

[Jeff King <peff@peff.net>]

On Fri, Jul 16, 2021 at 08:56:04PM +0200, Ævar Arnfjörð Bjarmason wrote:

> > I think you can do the same level of skipping with GIT_TEST_SKIP,
> > though. My argument was just that adding a new mechanism does not make
> > sense when we already have one. I.e., running:
> >
> >   GIT_SKIP_TESTS='
> >     t[123456789]*
> >     t0[^0]*
> >     t00[^016]*
> >     t000[469]
> >     t001[2459]
> >     t006[0248]
> >   ' make SANITIZE=leak test
> >
> > works already to do the same thing. The only thing we might want is a
> > nicer syntax (e.g., to allow positive and negative patterns, or to read
> > from a file). But that would benefit all users of GIT_SKIP_TESTS, not
> > just people interested in leaks.
> 
> A glob in this series is t13*config*, you can't do that with
> GIT_SKIP_TESTS because it only includes the numeric part of the test,
> i.e. t1300, not t1300-config, or t1306-xdg-files.

That seems like a feature that GIT_SKIP_TESTS could learn (though IMHO
just using the test number in your patterns is sufficient).

> I would like to be able to compile with it and run "make test" without a
> wall of failures by default, i.e. we should be able to tell regressions
> from known-OK to get anywhere with it, but that's orthagonal to the
> exact mechanism.

Right, I definitely agree on the goal. I just don't see the need to add
a new, very-specific mechanism. The skip-list above is gross and
obviously not something you'd want to type. Driving it from a ci script
is not too bad, but I agree people who want to leak-check locally would
want an easy way to use it, too. That's why I suggested extending it to
a file that could be easily specified (and possibly even auto-triggered
in the Makefile by SANITIZE=leak).

> > With GIT_SKIP_TESTS you obviously don't get a message saying "try
> > skipping this test" when it fails. :) But IMHO that is not that big a
> > deal. You'll get a test failure with good LSan output. If you are
> > working on expanding leak-checker coverage, you already know about your
> > options for skipping. If you're adding a new test that leaks, you might
> > consider fixing the leak (though not always, if it's far from code
> > you're touching).
> 
> I do think it makes sense as a test mode test-lib.sh is aware of,
> e.g. on obvious next step is to not fail everything right away, but just
> let the test run and log all failures to a file, then e.g. fail one test
> at the end, or if we're running in that mode collate all the callstacks
> and emit a summary for the whole test run.

That's a more compelling reason, if we did implement that feature. My
hope was that all of this would be a temporary state, though, and we'd
get to a point where you can simply run "make SANITIZE=leak test" and
actually run all of the tests. And then such a feature would not be that
interesting, because failures would be rare and cause for immediate
human attention.

-Peff