Re: How to handle long running tests in nptl.

[Florian Weimer <fweimer@redhat.com>]

* Carlos O'Donell:

> On 9/12/19 9:21 AM, Stefan Liebler wrote:
>> Hi,
>> 
>> the nptl tests are currently running in sequence. Some of them are running very long:
>> -tst-cond16: 20s
>> -tst-cond17: 20s
>> -tst-cond18: 20s
>> -tst-rwlock-tryrdlock-stall: 20s
>> -tst-mutex10: 16s
>> -tst-rwlock20: 10s
>> -tst-rwlock-trywrlock-stall: 10s
>> -tst-rwlock-pwn: 10s
>> 
>> The listed tests are responsible for over two minutes of runtime of "make check". They all are running a test in a loop for a large amount of iterations or seconds in order to trigger e.g. a race condition.
>> 
>> How to handle those long running tests with respect of "make check" runtime?
>> - reduce runtime by reducing number of iterations or seconds
>> - move those tests to "make xcheck"
>> - reduce runtime while running "make check" and rerun with unchanged runtime in "make xcheck"
>> - change nothing
>> - other ideas?
>
> The use of 'make xcheck' is for tests that need specific persmissions
> to run, like root.

It's also used for tests that require many minutes to run or require
special firewall settings, e.g., resolv/tst-resolv-qtypes.

Unfortunately, we do not capture data of legitimate test failures, so
it's hard to tell how valuable such tests are.

Many of the timeout-heavy nptl tests do not actually need a quiet
system, though, so they could be moved to a separate subdirectory that
contains only such tests.  Or we could add some markup to the tests and
add a more intelligent test scheduler.

Thanks,
Florian