Re: [PATCH] test-lib: add ability to cap the runtime of tests

[Jeff King <peff@peff.net>]

On Mon, Jun 05, 2017 at 10:55:42AM +0900, Junio C Hamano wrote:

> I do not expect any single person to tackle the splitting.  I just
> wished that a patch inspired by this patch (or better yet, a new
> version of this patch) made the tail end of "make test" output to
> read like this:
> 
>    ...
>    [18:32:44] t9400-git-cvsserver-server.sh ...... ok    18331 ms
>    [18:32:49] t9402-git-cvsserver-refs.sh ........ ok    22902 ms
>    [18:32:49] t9200-git-cvsexportcommit.sh ....... ok    25163 ms
>    [18:32:51]
>    All tests successful.
>    Files=785, Tests=16928, 122 wallclock secs ( ...
>    Result: PASS
> 
>    * The following tests took longer than 15 seconds to run.  We
>      may want to look into splitting them into smaller files.
> 
>    t3404-rebase-interactive.sh ...    19 secs
>    t9001-send-email.sh ...........    22 secs
>    t9402-git-cvsserver-refs.sh ...    22 secs
>    t9200-git-cvsexportcommit.sh ..    25 secs
> 
> when the hidden feature is _not_ used, so that wider set of people
> will be forced to see that some tests take inordinate amount of
> time, and entice at least some of them to look into it.

If you use "prove", it already records this information, and it can
print it with "--timer".

I don't use that myself, though. What's much more interesting (but which
I haven't found a way to get prove to do out-of-the-box) is to show the
longest tests after the fact. I gave a perl snippet to do so in

  http://public-inbox.org/git/20161019205638.m3ytxozzmeh47ml2@sigill.intra.peff.net/

I've actually played with this splitting before, but on my quad-core
(plus hyperthreading) box, I could never get it to make any improvement
once --state=slow was used. The longest test for me is 28s, but the
whole suite takes 50s to run. The slow tests get front-loaded, and then
by the end we have lots of little tests to hand out to each processor
and they all stay busy.

The main difference with Ævar's run is that he has a huge number of
processors. So I'm in favor of more splitting, but I also doubt that
even most Git developers would see any improvement. Or maybe people
really do have monstrous boxes. I dunno.

The biggest changes I've seen in my runs are:

  1. Use "prove --state=slow,save"; even with a few processors it makes
     a big difference.

  2. Point --root at a RAM disk.

  3. I started using Michael's git-test[1], which I have testing each
     commit on the current branch in the background[2]. That gives me an
     early warning when there's a failure (I have it play a sad trombone
     sound, since it's in a minimized terminal), and when I do ask the
     tests to run, quite often it can answer "all tests pass" out of
     the cache.

-Peff

[1] https://github.com/mhagger/git-test

[2] My hacky script is at https://github.com/peff/git/blob/meta/ci