Module suggestion

Module suggestion

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

Gnulib contains a module named valgrind that allows the easy use of
Valgrind for tests whenever it is supported by the build system.

I would suggest to add a similar module named timeout that sets the
variable TIMEOUT with suitable defaults whenever the GNU coreutils timeout
program (or an equivalent) is available.

This would allow test runners to be unconditionally prefixed with
$(TIMEOUT) so that on supporting systems, tests are killed (and an error is
reported) whenever they run for too long (mostly because of a logic error
causing an infinite loop).

Thanks,

Marc

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

Re: Module suggestion: timeout

[Bruno Haible]

Hi Marc,

Marc Nieper-Wißkirchen wrote:
> Gnulib contains a module named valgrind

More precisely, it's called 'valgrind-tests'. See
https://www.gnu.org/software/gnulib/manual/html_node/Using-valgrind-automatically.html

> that allows the easy use of
> Valgrind for tests whenever it is supported by the build system.
> 
> I would suggest to add a similar module named timeout that sets the
> variable TIMEOUT with suitable defaults whenever the GNU coreutils timeout
> program (or an equivalent) is available.
> 
> This would allow test runners to be unconditionally prefixed with
> $(TIMEOUT) so that on supporting systems, tests are killed (and an error is
> reported) whenever they run for too long (mostly because of a logic error
> causing an infinite loop).

The situations are a bit different:

1) For the timeout, there is a coding idiom that needs little code and works
also when the 'timeout' program is not present:

#if HAVE_DECL_ALARM
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);
#endif

2) For the timeout, different tests need different timeouts. Whereas valgrind
typically does not need per-program arguments.

3) If a way to enable a timeout exists, it is typically never useful to run a
unit test without a timeout. Whereas if valgrind is available, it is still
useful to run a unit test without valgrind:
  - the unit test runs much faster without valgrind,
  - the unit test might not be prepared to cope with the extra '==...' lines
    output to stderr.

So, I don't think the "let's treat timeout like valgrind" approach is going
to work. Instead, you need to design a way to deal with timeouts, independently.

Bruno

Re: Module suggestion: timeout

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> So, I don't think the "let's treat timeout like valgrind" approach is going
> to work. Instead, you need to design a way to deal with timeouts, independently.

Hi!  I think Marc's request for functionality to introduce timeouts for
self-tests is a good one.  However I reach the same conclusion as Bruno,
that having a module like valgrind-tests is probably not the best way to
solve it.  To me, having a timeout seems like an essential feature of a
self-test framework.  I know automake isn't primarily a self-test
framework, but it has concepts for it and the test framework has been
improved significantly over the years, so I think adding a timeout
functionality to automake makes sense.  What do bug-automake people
think?

The functionality could be conditioned on the coreutils 'timeout' tool,
and if that tool exists, and appears to work, running all self-tests
under that tool could be done automatically.  The default self-test
timeout be quite generous (say 17 hours?) but it should be easy to
modify both by end-user and project developer.  If we want to be
conservative, the functionality could be opt-in initially, and then
after a few years become the default behaviour.

Thoughts?

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Module suggestion: timeout

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 2459 bytes --]

Dear Bruno, dear Simon,

thank you for your replies.

I understand that valgrind-tests and the proposed "timeout-tests" solution
are not completely equivalent. Nevertheless, I still think that some
timeout functionality provided by Gnulib would be useful.

Bruno's solution

**
#if HAVE_DECL_ALARM
  /* Declare failure if test takes too long, by using default abort
     caused by SIGALRM.  */
  int alarm_value = 600;
  signal (SIGALRM, SIG_DFL);
  alarm (alarm_value);
#endif
**

works for unit tests that have been written specifically for the project.
It doesn't help, though, if I want to test (production) binaries that are
to be installed because I generally don't want testing code inside them.

Moreover, use cases for a baked-in timeout are not restricted to tests. For
example, I may want to restrict the build time of certain components in
situations where a logical error may lead to infinite build times (a simple
example is that of a Scheme compiler used as a build tool; thanks to
Turing-completeness of Scheme macros, such a build may not terminate).

Marc

Am Fr., 30. Apr. 2021 um 11:24 Uhr schrieb Simon Josefsson <
simon@josefsson.org>:

> Bruno Haible <bruno@clisp.org> writes:
>
> > So, I don't think the "let's treat timeout like valgrind" approach is
> going
> > to work. Instead, you need to design a way to deal with timeouts,
> independently.
>
> Hi!  I think Marc's request for functionality to introduce timeouts for
> self-tests is a good one.  However I reach the same conclusion as Bruno,
> that having a module like valgrind-tests is probably not the best way to
> solve it.  To me, having a timeout seems like an essential feature of a
> self-test framework.  I know automake isn't primarily a self-test
> framework, but it has concepts for it and the test framework has been
> improved significantly over the years, so I think adding a timeout
> functionality to automake makes sense.  What do bug-automake people
> think?
>
> The functionality could be conditioned on the coreutils 'timeout' tool,
> and if that tool exists, and appears to work, running all self-tests
> under that tool could be done automatically.  The default self-test
> timeout be quite generous (say 17 hours?) but it should be easy to
> modify both by end-user and project developer.  If we want to be
> conservative, the functionality could be opt-in initially, and then
> after a few years become the default behaviour.
>
> Thoughts?
>
> /Simon
>

[-- Attachment #2: Type: text/html, Size: 4079 bytes --]

Re: Module suggestion: timeout

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

Marc Nieper-Wißkirchen <marc.nieper+gnu@gmail.com> writes:

> Moreover, use cases for a baked-in timeout are not restricted to tests. For
> example, I may want to restrict the build time of certain components in
> situations where a logical error may lead to infinite build times (a simple
> example is that of a Scheme compiler used as a build tool; thanks to
> Turing-completeness of Scheme macros, such a build may not terminate).

This makes me believe even stronger that the functionality ought to be
provided by automake natively -- it seems the desired functionality is
not only timeouts for self-tests but timeouts for all operations.

Implementing this for self-tests in automake would probably be quite
simple, but implementing it for all operations is probably more
complicated.  Maybe it should be two separate features.  I have wanted
timeouts for self-tests but rarely for building.  If the second is more
complicated to implement, maybe starting with the first will be
sufficient and useful.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Module suggestion: timeout

[Marc Nieper-Wißkirchen]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

Adding timeout to all build operations automatically is probably too much
and too intrusive.

But in any case, it should be easy to add such a timeout to certain build
operations (e.g. downloading, running a compiler for which the halting
problem cannot be solved, testing a production binary, ...).

Am Fr., 30. Apr. 2021 um 11:43 Uhr schrieb Simon Josefsson <
simon@josefsson.org>:

> Marc Nieper-Wißkirchen <marc.nieper+gnu@gmail.com> writes:
>
> > Moreover, use cases for a baked-in timeout are not restricted to tests.
> For
> > example, I may want to restrict the build time of certain components in
> > situations where a logical error may lead to infinite build times (a
> simple
> > example is that of a Scheme compiler used as a build tool; thanks to
> > Turing-completeness of Scheme macros, such a build may not terminate).
>
> This makes me believe even stronger that the functionality ought to be
> provided by automake natively -- it seems the desired functionality is
> not only timeouts for self-tests but timeouts for all operations.
>
> Implementing this for self-tests in automake would probably be quite
> simple, but implementing it for all operations is probably more
> complicated.  Maybe it should be two separate features.  I have wanted
> timeouts for self-tests but rarely for building.  If the second is more
> complicated to implement, maybe starting with the first will be
> sufficient and useful.
>
> /Simon
>

[-- Attachment #2: Type: text/html, Size: 2064 bytes --]

Re: bug#48113: Module suggestion: timeout

[Karl Berry]

    What do bug-automake people think?

For myself, I have no objection to sprinkling timeout commands through
the Automake test infrastructure wherever appropriate. It's not ever
going to rise to the top of my own list of things to do, though, so if
it's going to happen, you or someone will have to write the patch.

Of course I don't speak for Jim, but from what he's said in the past,
I suspect he is in a similar situation.

No one else has come forward to work on Automake, despite my plea
(https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html),
so I guess that's where we are.
    
    the functionality could be opt-in initially, 

Certainly.

    and then after a few years become the default behaviour.

Personally, I think it should be opt-in forever. People could easily
have test suites that need to run for days. I prefer not to
unnecessarily break compatibility.

Thanks,
Karl

Re: bug#48113: Module suggestion: timeout

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

severity 48113 wishlist
retitle 48113 Self-test timeout functionality
thanks

Karl Berry <karl@freefriends.org> writes:

>     What do bug-automake people think?
>
> For myself, I have no objection to sprinkling timeout commands through
> the Automake test infrastructure wherever appropriate. It's not ever
> going to rise to the top of my own list of things to do, though, so if
> it's going to happen, you or someone will have to write the patch.
>
> Of course I don't speak for Jim, but from what he's said in the past,
> I suspect he is in a similar situation.

Thanks for confirming that it isn't an obviously bad idea.  Tagging this
as a wishlist bug, and improving the bug title a bit.

> No one else has come forward to work on Automake, despite my plea
> (https://lists.gnu.org/archive/html/automake/2021-03/msg00018.html),
> so I guess that's where we are.

Ah right -- I'll see if I can help in any way, but will respond
separately.

>     the functionality could be opt-in initially, 
>
> Certainly.
>
>     and then after a few years become the default behaviour.
>
> Personally, I think it should be opt-in forever. People could easily
> have test suites that need to run for days. I prefer not to
> unnecessarily break compatibility.

Yes, I probably agree with this -- chosing the default timeout setting
is difficult as it will most likely just cause problems for some people
with long-running tests, and not solve the initial problem for some
other people (that have small projects and a timeout of 5 minutes is
what you want).  Better leave this up the each project to decide.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]