Re: [PATCH] tests: add an optional test to test git-annex

[Junio C Hamano <gitster@pobox.com>]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>> Well, it is one thing to place git-annex under CI to make sure its
>> latest and greatest works together well with our latest and greatest
>> (and it may be something we want to see happen), but driving its
>> tests from our testsuite sounds like a tail wagging the dog, at
>> least to me.
>
> To me this is just a question of:
>
> * Is it the case that git-annex tests for a lot of edge cases we don't
> test for: Yes, probably. As evidenced by them spotting this
> regression, and not us.

And I'd encourage them to keep doing so.

> * We can (and should) add a test for the specific breakage we caused
> in 2.13.0, but that's no replacement for other things annex may be
> covering & we may be missing which'll catch future breakages.
>
> * It's a pretty established practice to test a library (git) along
> with its consumers (e.g. annex) before a major release.

I am not so sure about the division of labor.  What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers.  If they are also a moving target, then
I do not think it is worth it.  After all, we are *not* in the
business of testing these consumers.

Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development.  If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.