Re: test &&-chain lint

[Junio C Hamano <gitster@pobox.com>]

Jeff King <peff@peff.net> writes:

> [+cc Jonathan, whose patch I apparently subconsciously copied]
>
> On Thu, Mar 19, 2015 at 10:08:51PM -0400, Jeff King wrote:
>
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c096778..02a03d5 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -524,6 +524,21 @@ test_eval_ () {
>>  test_run_ () {
>>  	test_cleanup=:
>>  	expecting_failure=$2
>> +
>> +	if test -n "$GIT_TEST_CHAIN_LINT"; then
>> +		# 117 is unlikely to match the exit code of
>> +		# another part of the chain
>> +		test_eval_ "(exit 117) && $1"
>> +		if test "$?" != 117; then
>> +			# all bets are off for continuing with other tests;
>> +			# we expected none of the rest of the test commands to
>> +			# run, but at least some did. Who knows what weird
>> +			# state we're in? Just bail, and the user can diagnose
>> +			# by running in --verbose mode
>> +			error "bug in the test script: broken &&-chain"
>> +		fi
>> +	fi
>> +
>>  	setup_malloc_check
>>  	test_eval_ "$1"
>>  	eval_ret=$?
>> 
>> This turns up an appalling number of failures, but AFAICT they are all
>> "real" in the sense that the &&-chains are broken. In some cases these
>> are real, but in others the tests are of an older style where they did
>> not expect some early commands to fail (and we would catch their bogus
>> output if they did). E.g., in the patch below, I think the first one is
>> a real potential bug, and the other two are mostly noise. I do not mind
>> setting a rule and fixing all of them, though.
>> 
>> I seem to recall people looked at doing this sort of lint a while ago,
>> but we never ended up committing anything. I wonder if it was because of
>> all of these "false positives".
>
> This turns out to be rather annoying to grep for in the list archives,
> but I found at least one discussion:
>
>   http://article.gmane.org/gmane.comp.version-control.git/235913
>
> I don't know why we didn't follow it up then. Perhaps because the patch
> there (which is rather similar to what I have above) was not
> conditional, so whole chunks of the test suite needed fixing. There are
> enough problems that we would probably want to do this conditionally,
> fix them over time, and then finally flip the feature on by default.

Hmmm, they do look similar and unfamiliar ;-)  It happened while I
was offline, it seems.

Thanks for working on this---I think test-chain-lint should become
one of the pre-acceptance test on my end when it gets applied.