Hi Ævar,

On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:

> On Tue, May 14 2019, Johannes Schindelin wrote:
>
> > On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:
> >
> >> On Tue, May 14 2019, Johannes Schindelin wrote:
> >>
> >> > What would you think about a mode where random test cases are
> >> > skipped? It would have to make sure to provide a way to recreate
> >> > the problem, e.g. giving a string that defines exactly which test
> >> > cases were skipped.
> >> >
> >> > I am *sure* that tons of test scripts would fail with that, and we
> >> > would probably have to special-case the `setup` "test cases", and
> >> > we would have to clean up quite a few scripts to *not* execute
> >> > random stuff outside of `test_expect_*`...
> >>
> >> I think it would be neat, but unrelated to and overkill for spotting
> >> the practical problem we have now, which is that we *know* we skip
> >> some of this now on some platforms/setups due to prereqs.
> >
> > I understand, but I am still worried that this is a lot of work for an
> > incomplete fix.
> >
> > For example, the t7600-merge.sh test script that set off this
> > conversation has two prereqs that are unmet on Windows: GPG and
> > EXECKEEPSPID. On Azure Pipelines' macOS agents, it is only GPG that is
> > unmet. So switching off all prereqs would not help macOS with e.g. a
> > bug where the GPG test cases are skipped but the EXECKEEPSPID test
> > case is not.
>
> It won't catch such cases, but will catch cases where a later new test
> assumes that whatever the state of the test repo it gets is what's
> always going to be there. In practice I think that'll catch most such
> issues.

That's fair. It will also raise awareness of these issues, which should
also have a quite beneficial effect.

Given that your patch is not even large, I think you're right, it *is* a
lot of bang for the buck.

Ciao,
Dscho