Re: [PATCH] tests: add an optional test to test git-annex

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Wed, May 17, 2017 at 4:45 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> Add an optional test to test git-annex. It's guarded by a new
>> EXTERNAL_TESTS environment variable. Running this test takes me 10
>> minutes.

[Re-arranged your mail because it worked better with my reply]

> I do not mind at all to place the simple reproduction recipe Joey
> posted as a new test in our test suite, though.  That kind of test
> that catches changes to externally visible behaviour surely belongs
> to our test suite.

This is not a replacement for having an isolated test for the issue
Joey noted. We should have a separate patch for that, but I did not
have time/interest in writing that up. This change is orthagonal to
that.

> Well, it is one thing to place git-annex under CI to make sure its
> latest and greatest works together well with our latest and greatest
> (and it may be something we want to see happen), but driving its
> tests from our testsuite sounds like a tail wagging the dog, at
> least to me.

To me this is just a question of:

* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.

* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.

* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.

* This allows us to do that at minimal cost. I think it makes sense to
add this and integration tests for other similar utilities if they're
similarly easy to integrate.