From: Jeff King <peff@peff.net>
To: Stefan Beller <sbeller@google.com>
Cc: git@vger.kernel.org
Subject: Re: [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp
Date: Fri, 6 Oct 2017 15:22:52 -0400 [thread overview]
Message-ID: <20171006192252.3a27enzim5yurhe5@sigill.intra.peff.net> (raw)
In-Reply-To: <20171006190006.19623-1-sbeller@google.com>
On Fri, Oct 06, 2017 at 12:00:05PM -0700, Stefan Beller wrote:
> The `test_must_fail` should only be used to indicate a git command is
> failing. `test_cmp` is not a git command, such that it doesn't need the
> special treatment of `test_must_fail` (which e.g. includes checking for
> segfault)
Hmph. "test_must_fail test_cmp" is a weird thing for somebody to write.
And your patch is obviously an improvement, but I have to wonder if some
of these make any sense.
If we're expecting some outcome, then it's reasonable to say:
1. The output should look exactly like this. (test_cmp)
2. The output should look something like this. (grep)
3. The output should _not_ mention this (! grep)
But "the output should not look exactly like this" doesn't seem very
robust. It's likely to give a false success due to small changes (or
translations), or even bugs in the script.
For instance:
> diff --git a/t/t3504-cherry-pick-rerere.sh b/t/t3504-cherry-pick-rerere.sh
> index a267b2d144..c31141c471 100755
> --- a/t/t3504-cherry-pick-rerere.sh
> +++ b/t/t3504-cherry-pick-rerere.sh
> @@ -95,7 +95,7 @@ test_expect_success 'cherry-pick --rerere-autoupdate more than once' '
> test_expect_success 'cherry-pick conflict without rerere' '
> test_config rerere.enabled false &&
> test_must_fail git cherry-pick master &&
> - test_must_fail test_cmp expect foo
> + ! test_cmp expect foo
> '
Running ./t3504 with "-v" (with or without your patch) shows:
--- expect 2017-10-06 19:14:43.677840120 +0000
+++ foo 2017-10-06 19:14:43.705840120 +0000
@@ -1 +1 @@
-fatal: cherry-pick: --no-rerere-autoupdate cannot be used with --continue
+foo-dev
Which just seems like a bug. Did the original author mean foo-expect?
It's hard to tell, as we are just reusing expectations from previous
tests.
> diff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh
> index 02106c9226..7178b917ce 100755
> --- a/t/t5512-ls-remote.sh
> +++ b/t/t5512-ls-remote.sh
> @@ -56,7 +56,7 @@ test_expect_success 'use "origin" when no remote specified' '
>
> test_expect_success 'suppress "From <url>" with -q' '
> git ls-remote -q 2>actual_err &&
> - test_must_fail test_cmp exp_err actual_err
> + ! test_cmp exp_err actual_err
> '
This one seems like "test_18ngrep ! ^From" would be more appropriate. Or
even "test_must_be_empty".
> diff --git a/t/t5612-clone-refspec.sh b/t/t5612-clone-refspec.sh
> index fac5a73851..5f9ad51929 100755
> --- a/t/t5612-clone-refspec.sh
> +++ b/t/t5612-clone-refspec.sh
> @@ -87,7 +87,7 @@ test_expect_success 'by default no tags will be kept updated' '
> git for-each-ref refs/tags >../actual
> ) &&
> git for-each-ref refs/tags >expect &&
> - test_must_fail test_cmp expect actual &&
> + ! test_cmp expect actual &&
> test_line_count = 2 actual
Here we check that no updates happened due to a fetch because we see
that the tags in the fetched repo do not match the tags in the parent
repo. That actually seems pretty legitimate. But I think:
git for-each-ref refs/tags >before
git fetch
git for-each-ref refs/tags >after
test_cmp before after
would be more straightforward.
> diff --git a/t/t7508-status.sh b/t/t7508-status.sh
> index 93f162a4f7..1644866571 100755
> --- a/t/t7508-status.sh
> +++ b/t/t7508-status.sh
> @@ -1532,7 +1532,7 @@ test_expect_success '"status.branch=true" same as "-b"' '
> test_expect_success '"status.branch=true" different from "--no-branch"' '
> git status -s --no-branch >expected_nobranch &&
> git -c status.branch=true status -s >actual &&
> - test_must_fail test_cmp expected_nobranch actual
> + ! test_cmp expected_nobranch actual
> '
Shouldn't this be comparing it positively to the output with "--branch"?
> test_expect_success '"status.branch=true" weaker than "--no-branch"' '
> diff --git a/t/t9164-git-svn-dcommit-concurrent.sh b/t/t9164-git-svn-dcommit-concurrent.sh
> index d8464d4218..5cd6b40432 100755
> --- a/t/t9164-git-svn-dcommit-concurrent.sh
> +++ b/t/t9164-git-svn-dcommit-concurrent.sh
> @@ -92,7 +92,7 @@ test_expect_success 'check if post-commit hook creates a concurrent commit' '
> echo 1 >> file &&
> svn_cmd commit -m "changing file" &&
> svn_cmd up &&
> - test_must_fail test_cmp auto_updated_file au_file_saved
> + ! test_cmp auto_updated_file au_file_saved
> )
> '
This one looked complicated, so I leave it as an exercise for the
reader. :)
-Peff
next prev parent reply other threads:[~2017-10-06 19:23 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-06 19:00 [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp Stefan Beller
2017-10-06 19:00 ` [PATCH 2/2] tests: fix diff order arguments in test_cmp Stefan Beller
2017-10-06 22:01 ` Jonathan Nieder
2017-10-06 22:10 ` Stefan Beller
2017-10-07 1:18 ` Junio C Hamano
2017-10-07 2:29 ` Junio C Hamano
2017-10-06 19:22 ` Jeff King [this message]
2017-10-06 19:41 ` [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp Stefan Beller
2017-10-06 21:59 ` Jonathan Nieder
2017-10-07 2:12 ` Junio C Hamano
2017-10-06 20:21 ` Jonathan Nieder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171006192252.3a27enzim5yurhe5@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=sbeller@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).