git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Stefan Beller <sbeller@google.com>
To: Jeff King <peff@peff.net>
Cc: "git@vger.kernel.org" <git@vger.kernel.org>
Subject: Re: [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp
Date: Fri, 6 Oct 2017 12:41:22 -0700	[thread overview]
Message-ID: <CAGZ79kb4N2442HNWJuviATpOwiQ-qoGOgK+6Ys2JQMuS_kvjzg@mail.gmail.com> (raw)
In-Reply-To: <20171006192252.3a27enzim5yurhe5@sigill.intra.peff.net>

On Fri, Oct 6, 2017 at 12:22 PM, Jeff King <peff@peff.net> wrote:
> On Fri, Oct 06, 2017 at 12:00:05PM -0700, Stefan Beller wrote:
>
>> The `test_must_fail` should only be used to indicate a git command is
>> failing. `test_cmp` is not a git command, such that it doesn't need the
>> special treatment of `test_must_fail` (which e.g. includes checking for
>> segfault)
>
> Hmph. "test_must_fail test_cmp" is a weird thing for somebody to write.
> And your patch is obviously an improvement, but I have to wonder if some
> of these make any sense.

Thanks for actually thinking about the problem. I just searched and replaced
the combination of test_must_fail and test_cmp mechanically after I noticed
one such occurrence whilst preparing the next patch (order of test_cmp args).

>
> If we're expecting some outcome, then it's reasonable to say:
>
>   1. The output should look exactly like this. (test_cmp)
>
>   2. The output should look something like this. (grep)
>
>   3. The output should _not_ mention this (! grep)
>
> But "the output should not look exactly like this" doesn't seem very
> robust. It's likely to give a false success due to small changes (or
> translations), or even bugs in the script.

I agree that the case "should not look like exactly this" is most likely
indicating a weak test.


> Running ./t3504 with "-v" (with or without your patch) shows:
>
>   --- expect    2017-10-06 19:14:43.677840120 +0000
>   +++ foo       2017-10-06 19:14:43.705840120 +0000
>   @@ -1 +1 @@
>   -fatal: cherry-pick: --no-rerere-autoupdate cannot be used with --continue
>   +foo-dev
>
> Which just seems like a bug.  Did the original author mean foo-expect?

This was originally written by a non regular in 2008. I don't think
they remember even if we'd ask.

I think we'd want to not resolve it to foo-dev here,
(so ideally we'd test for

<<<<
foo
====
foo-dev
>>>>

but just testing that we do not blindly resolve to foo seems ok-ish)

Thanks for spotting this!

> It's hard to tell, as we are just reusing expectations from previous
> tests.
>
>> diff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh
>> index 02106c9226..7178b917ce 100755
>> --- a/t/t5512-ls-remote.sh
>> +++ b/t/t5512-ls-remote.sh
>> @@ -56,7 +56,7 @@ test_expect_success 'use "origin" when no remote specified' '
>>
>>  test_expect_success 'suppress "From <url>" with -q' '
>>       git ls-remote -q 2>actual_err &&
>> -     test_must_fail test_cmp exp_err actual_err
>> +     ! test_cmp exp_err actual_err
>>  '
>
> This one seems like "test_18ngrep ! ^From" would be more appropriate. Or
> even "test_must_be_empty".

Going by the test title, I agree.

>> diff --git a/t/t5612-clone-refspec.sh b/t/t5612-clone-refspec.sh
>> index fac5a73851..5f9ad51929 100755
>> --- a/t/t5612-clone-refspec.sh
>> +++ b/t/t5612-clone-refspec.sh
>> @@ -87,7 +87,7 @@ test_expect_success 'by default no tags will be kept updated' '
>>               git for-each-ref refs/tags >../actual
>>       ) &&
>>       git for-each-ref refs/tags >expect &&
>> -     test_must_fail test_cmp expect actual &&
>> +     ! test_cmp expect actual &&
>>       test_line_count = 2 actual
>
> Here we check that no updates happened due to a fetch because we see
> that the tags in the fetched repo do not match the tags in the parent
> repo. That actually seems pretty legitimate. But I think:
>
>   git for-each-ref refs/tags >before
>   git fetch
>   git for-each-ref refs/tags >after
>   test_cmp before after
>
> would be more straightforward.
>
>> diff --git a/t/t7508-status.sh b/t/t7508-status.sh
>> index 93f162a4f7..1644866571 100755
>> --- a/t/t7508-status.sh
>> +++ b/t/t7508-status.sh
>> @@ -1532,7 +1532,7 @@ test_expect_success '"status.branch=true" same as "-b"' '
>>  test_expect_success '"status.branch=true" different from "--no-branch"' '
>>       git status -s --no-branch  >expected_nobranch &&
>>       git -c status.branch=true status -s >actual &&
>> -     test_must_fail test_cmp expected_nobranch actual
>> +     ! test_cmp expected_nobranch actual
>>  '
>
> Shouldn't this be comparing it positively to the output with "--branch"?
>
>>  test_expect_success '"status.branch=true" weaker than "--no-branch"' '
>> diff --git a/t/t9164-git-svn-dcommit-concurrent.sh b/t/t9164-git-svn-dcommit-concurrent.sh
>> index d8464d4218..5cd6b40432 100755
>> --- a/t/t9164-git-svn-dcommit-concurrent.sh
>> +++ b/t/t9164-git-svn-dcommit-concurrent.sh
>> @@ -92,7 +92,7 @@ test_expect_success 'check if post-commit hook creates a concurrent commit' '
>>               echo 1 >> file &&
>>               svn_cmd commit -m "changing file" &&
>>               svn_cmd up &&
>> -             test_must_fail test_cmp auto_updated_file au_file_saved
>> +             ! test_cmp auto_updated_file au_file_saved
>>       )
>>  '
>
> This one looked complicated, so I leave it as an exercise for the
> reader. :)

eh, I was hoping to not stirr up a controversy, but treating this as a
drive-by patch "making the tests a better place, one tiny step at a time".

Thanks,
Stefan

>
> -Peff

  reply	other threads:[~2017-10-06 19:41 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-06 19:00 [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp Stefan Beller
2017-10-06 19:00 ` [PATCH 2/2] tests: fix diff order arguments in test_cmp Stefan Beller
2017-10-06 22:01   ` Jonathan Nieder
2017-10-06 22:10     ` Stefan Beller
2017-10-07  1:18       ` Junio C Hamano
2017-10-07  2:29   ` Junio C Hamano
2017-10-06 19:22 ` [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp Jeff King
2017-10-06 19:41   ` Stefan Beller [this message]
2017-10-06 21:59   ` Jonathan Nieder
2017-10-07  2:12   ` Junio C Hamano
2017-10-06 20:21 ` Jonathan Nieder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAGZ79kb4N2442HNWJuviATpOwiQ-qoGOgK+6Ys2JQMuS_kvjzg@mail.gmail.com \
    --to=sbeller@google.com \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).