Hi,

On Tue, 12 Feb 2019, SZEDER Gábor wrote:

> On Tue, Feb 12, 2019 at 02:14:56PM +0100, Ævar Arnfjörð Bjarmason wrote:
> > 
> > On Tue, Feb 12 2019, Duy Nguyen wrote:
> > 
> > > On Tue, Feb 12, 2019 at 7:43 PM Duy Nguyen <pclouds@gmail.com> wrote:
> > >> The test failures on NetBSD and Solaris/Sparc, not sure if we can do
> > >> anything without test logs or access to these systems.
> > >
> > > Actually if you could tweak your ci script a bit to run tests with -v,
> > > that would help.
> > 
> > I vaguely remember doing that and running into some issue where it
> > truncated the output, so e.g. I wouldn't see compile warnings on AIX
> > because of the firehose of subsequent test output.
> > 
> > But yeah, having this in some smart way would be great. I'd be most keen
> > to just work towards offloading this to some smarter test runner as
> > noted to Johannes upthread.
> > 
> > I.e. a good test_for(SHA1, params) function would run the tests with
> > "prove", and e.g. spot that tests so-and-so failed, and then run those
> > specific ones with -v -x.
> 
> Just follow suit of what we have been doing on Travis CI since the
> very beginning: run tests with '--verbose-log' to begin with, and then
> dump the logfiles of any failed tests, i.e. where the content of
> 'test-results/t1234-foo.exit' is not '0'.

Indeed, and this is how you can do it at home:

	ci/run-build-and-tests.sh || {
           ci/print-test-failures.sh
           exit 1
       }

Erm, that's how it *used* to be possible at home, but I broke that! Now it
will tell you that it "Could not identify CI type". I guess we should fix
this by introducing a new arm to the `if test true = "$TRAVIS"` construct,
to allow for manual runs (or for your CI builds where you specify the
parameters such as CI_TYPE, CI_BRANCH, CI_COMMIT and CI_OS_NAME via
environment variables).

To get you unblocked, you could rewrite ci/lib.sh on the fly:

	mv ci/lib.sh ci/lib.sh.orig
	sed "s/^if.*TRAVIS/if true; then :; el/" >ci/lib.sh <ci/lib.sh.orig

Ciao,
Dscho

> 
> Re-running a failed test is not a good idea, as it won't help if the
> test managed to fail because of a rare flakiness.
> 
> 
> > That's how I was going to fix the log overflow problem, but I'd much
> > rather not continue hacking on this gitlab-gccfarm-specific thing, and
> > instead work towards something more general.
>