Re: GSoC 2019: Git's application submitted

[Duy Nguyen <pclouds@gmail.com>]

On Wed, Mar 6, 2019 at 11:49 AM Jeff King <peff@peff.net> wrote:
>
> On Tue, Mar 05, 2019 at 07:04:59PM +0700, Duy Nguyen wrote:
>
> > On Mon, Feb 4, 2019 at 4:17 PM Christian Couder
> > <christian.couder@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > There are now ideas, micro-projects and organization application pages
> > > for GSoC 2019 on https://git.github.io/
> > >
> > > It would be nice to have a few more project ideas.
> >
> > Not sure if it's too late now. Anyway this could be something fun to
> > do: support C-based tests in our test suite.
> >
> > A while back I noticed some test running very long because it was
> > trying a lot of input combination. The actual logic is not much, but
> > because of the increasing number of test cases, overhead goes off the
> > roof. The last part is probably not true, but Windows port I think is
> > hit much harder than what I experience, and I think Dscho did complain
> > about it.
> >
> > So what this project does is somehow allow people to write test cases
> > in C instead of shell. Imagine replacing t3070-wildmatch.sh with a
> > binary program t3070-wildmatch that behaves the same way. This test
> > framework needs to support the same basic feature set as test-lib.sh:
> > TAP output, test results summary, maybe -i and --valgrind... To
> > demonstrate that the test framework works, one of these long test
> > files should be rewritten in C. I'm sure there's one that is simple to
> > rewrite.
> >
> > I'm pretty sure I had some fun with this idea and made some prototype
> > but I couldn't find it. If I do, I'll post the link here.
>
> In my experience, it's nicer to have a tool written in C that can be
> driven by arbitrary input. That makes it easy to write new test cases,
> because you just have to write in some easy domain-specific format
> instead of embedding the test data in C code.
>
> And many of our tests do work like that (in fact, many of the Git
> plumbing tools function as that). E.g., test-date gives you direct
> access to the low-level routines, and we feed it a variety of dates.
>
> That doesn't help with the cost of invoking that tool over and over,
> though, once per test case. I wonder if we could have some kind of
> hybrid. I.e., where t3070 is still a shell script, but it primarily
> consists of running one big binary, like:
>
>   test-wildmatch <<-\EOF
>   case 1
>   case 2
>   ...etc
>   EOF
>
> but with one added twist: test-wildmatch would actually generate TAP
> output for each test, rather than just returning 0/1 for each success or
> failure, and being embedded in a test_expect_success.

Yeah. I mean, converting a full file is just one of the ways to go.
The exact design is up to whoever implements it.

> It seems like that would even be pretty easy to do, with the exception
> of the numbering.

I'd like some niceties from test-lib.sh though. -i should be supported
to stop at the first failure. -v would be nice if possible. Skipping
tsets also.

> It would be nice if we could intermingle this kind of
> "chunk of C tests" with normal tests, but we'd have to figure out how
> many tests it ran and increment our shell-script's counter
> appropriately.

Even if you convert a full file, you need to do that anyway, to make
sure final test run summary is correct (at least when TAP is not used)

This kind of framework, I think also helps write new unit tests. There
are cases where creating the right condition to trigger just one
function is lots of work and easy to break. Sometimes it's easier to
do everything in C when full repository is not involved.
-- 
Duy