Initial Travis smoke testing

Initial Travis smoke testing

[Jeffrey Walton]

Hi Everyone/Bruno/Paul,

The initial Travis smoke test went in today. It is on my testing fork
at https://github.com/noloader/gnulib.

The Travis configuration file is at
https://github.com/noloader/gnulib/blob/master/.travis.yml. The Travis
test results are at
https://travis-ci.com/github/noloader/gnulib/builds for the time
being.

Each Travis job runs 32 Gnulib module tests. The first 32 are taken
using head -n 32. module_list has approximately 725 testable modules.

module_list=$(find modules -name '*tests' | sed -e 's|^modules/||g' -e
's|-tests$||g' | sort | head -n 32)

The big TODO is, how to partition the list so every module is tested
at least once a week or every two weeks.

Does anyone have an idea about partitioning the tests? (I have some,
but I don't want to taint suggestions).

Jeff

Re: Initial Travis smoke testing

[Jeffrey Walton]

On Sat, Mar 28, 2020 at 5:32 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> Hi Everyone/Bruno/Paul,
>
> The initial Travis smoke test went in today. It is on my testing fork
> at https://github.com/noloader/gnulib.
>
> The Travis configuration file is at
> https://github.com/noloader/gnulib/blob/master/.travis.yml. The Travis
> test results are at
> https://travis-ci.com/github/noloader/gnulib/builds for the time
> being.

Bruno, Paul,

Email me your GitHub credentials (if you have an account). I'll make
you collaborators on the testing fork. Collaborators on repos is like
being an administrator with write privileges. You can sync with
upstream as needed and make other changes as needed. There's no need
to waste time with endless messages making change requests.

Jeff

Re: Initial Travis smoke testing

[Bruno Haible]

Hi Jeffrey,

> The initial Travis smoke test went in today. It is on my testing fork
> at https://github.com/noloader/gnulib.

I appreciate that you are going forward!

> The big TODO is, how to partition the list so every module is tested
> at least once a week or every two weeks.
> 
> Does anyone have an idea about partitioning the tests? (I have some,
> but I don't want to taint suggestions).

An idea could be as follows:

  - Fix a number N of test runs for which you are going to store the
    list of tested modules.
  - For a new test run:
    1. Take the set of all modules, minus the modules that were selected
       in the last N test runs.
    2. If this set is empty, print "N is too large", decrement N, and go
       to step 1.
    3. Otherwise select up to, say, 5 elements of this module set at random.
    4. Invoke './gnulib-tool --create-testdir --single-configure ...'
       with these module names.
    5. gnulib-tool prints the full list of modules. Store them in the
       list of tested modules.

This way, modules at the "top" of the dependency hierarchy will not
be tested frequently, but through step 1, a certain amount of redundancy
is avoided.

If this doesn't work because you can't store things between runs, I would
vote for some clever algorithm on the dependency graph.

Bruno

Re: Initial Travis smoke testing

[Jeffrey Walton]

On Sat, Mar 28, 2020 at 6:19 PM Bruno Haible <bruno@clisp.org> wrote:
>
> Hi Jeffrey,
>
> > The initial Travis smoke test went in today. It is on my testing fork
> > at https://github.com/noloader/gnulib.
>
> I appreciate that you are going forward!
>
> > The big TODO is, how to partition the list so every module is tested
> > at least once a week or every two weeks.
> >
> > Does anyone have an idea about partitioning the tests? (I have some,
> > but I don't want to taint suggestions).
>
> An idea could be as follows:
>
>   - Fix a number N of test runs for which you are going to store the
>     list of tested modules.
>   - For a new test run:
>     1. Take the set of all modules, minus the modules that were selected
>        in the last N test runs.
>     2. If this set is empty, print "N is too large", decrement N, and go
>        to step 1.
>     3. Otherwise select up to, say, 5 elements of this module set at random.
>     4. Invoke './gnulib-tool --create-testdir --single-configure ...'
>        with these module names.
>     5. gnulib-tool prints the full list of modules. Store them in the
>        list of tested modules.
>
> This way, modules at the "top" of the dependency hierarchy will not
> be tested frequently, but through step 1, a certain amount of redundancy
> is avoided.
>
> If this doesn't work because you can't store things between runs, I would
> vote for some clever algorithm on the dependency graph.

Yep, that's the problem. No state between runs.

Jeff