From: Johannes Schindelin <Johannes.Schindelin@gmx.de>
To: Junio C Hamano <gitster@pobox.com>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
git@vger.kernel.org, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
"Anthony Ramine" <n.oxyde@gmail.com>,
"Johannes Sixt" <j6t@kdbg.org>,
"Adam Dinwoodie" <adam@dinwoodie.org>
Subject: Re: [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper
Date: Sat, 6 Jan 2018 13:51:45 +0100 (STD) [thread overview]
Message-ID: <nycvar.QRO.7.76.6.1801061337020.1337@wbunaarf-fpuvaqryva.tvgsbejvaqbjf.bet> (raw)
In-Reply-To: <xmqqzi5raogu.fsf@gitster.mtv.corp.google.com>
[-- Attachment #1: Type: text/plain, Size: 3590 bytes --]
Hi Junio,
On Fri, 5 Jan 2018, Junio C Hamano wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
> > Skip the newly added file creation tests on Windows proper, these
> > already work under Cygwin, but as that involves a significant
> > emulation layer the results are different under Windows proper with
> > MinGW.
> > ...
> > diff --git a/t/t3070-wildmatch.sh b/t/t3070-wildmatch.sh
> > index f606f91acb..50a53e7a62 100755
> > --- a/t/t3070-wildmatch.sh
> > +++ b/t/t3070-wildmatch.sh
> > @@ -7,6 +7,14 @@ test_description='wildmatch tests'
> > create_test_file() {
> > file=$1
> >
> > + # These tests limp along under Cygwin, but the failures on
> > + # native Windows are still too many. Skip file tests there
> > + # until they're sorted out.
> > + if test_have_prereq MINGW
> > + then
> > + return 1
> > + fi
> > +
>
> That looks to be a nuclear option.
Indeed:
# still have 84 known breakage(s)
# failed 52 among remaining 1428 test(s)
1..1512
Let's just switch it off completely because 52 of those 1512 test cases
break only on Windows? Pretty heavy-handed.
But do read on.
> For now it may be suffice, but somehow it feels that it goes directly
> against Dscho's wish to treat (or pretend to treat) Windows as
> first-class citizens and/or just like other platforms.
It not only goes against my wish to treat Windows as a normal citizen
instead of like Rey's parents, it also goes against my wish to have a
focused and meaningful test suite. Nobody likes to run tests that take too
long. And look at this:
...
ok 1511 - ipathmatch: match 'Z' '[Z-y]'
ok 1512 - ipathmatch(ls): match '[Z-y]' 'Z'
# still have 84 known breakage(s)
# failed 52 among remaining 1428 test(s)
1..1512
real 5m51.432s
user 0m33.986s
sys 2m13.162s
Yep. It takes *over eight minutes*.
And this is a *fast* machine.
Why? Because of the completely overboard testing of all kinds of
*potential* breakages *at some point* in the future.
I understand that Ævar wants to increase test coverage. I am sympathetic
to this cause.
But I completely disagree that increasing the test coverage beyond reason
is a good thing. Tests *can* take too long, and they do, in practice, and
the results are always problematic: every developer who has to deal with
test suites that take too long... does not run them. As simple as that.
And then you have *zero* test coverage. Pretty stupid, eh? And contrary to
your intentions, too.
So yes, I think that it has been proven beyond any doubt that t3070 takes
*waaaaaaaaaay* too long on Windows, for dubitable benefit.
I could see a reasonable compromise where
- an extensive test matrix is hidden behind an EXPENSIVE_WILDMATCH prereq,
- the test matrix would be written in a much more understandable way, i.e.
using English words rather than "1"s. If need be, there could be blocks
("test this with case-sensitive matching", "skip case-sensitive matching",
you get the idea),
- these magic skippings of certain test cases (where the (non-traced)
`create_test_file()` function returns 1 for certain things) still would
need to go away, in favor probably of prereqs and/or blocks where flags
are set accordingly in a preamble,
- by default, i.e. without the EXPENSIVE_WILDMATCH prereq, it should test
a *tiny* subset that is indicative of the most likely bugs.
As it is, I like 8/7 in the present form for all the wrong reasons: it
protects me from the damage a full t3070 would do to me.
Ciao,
Dscho
next prev parent reply other threads:[~2018-01-06 12:51 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-23 21:30 [PATCH 0/6] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 1/6] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 2/6] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 3/6] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 4/6] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 5/6] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 6/6] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-24 9:24 ` Johannes Sixt
2017-12-24 11:06 ` Ævar Arnfjörð Bjarmason
2017-12-24 11:51 ` Johannes Sixt
2017-12-25 0:28 ` [PATCH v2 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-28 22:48 ` Junio C Hamano
2017-12-28 23:49 ` Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 " Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-29 0:16 ` Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 20:28 ` Junio C Hamano
2017-12-25 0:28 ` [PATCH v2 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-25 9:26 ` Johannes Sixt
2017-12-27 19:07 ` Junio C Hamano
2018-01-03 13:02 ` Adam Dinwoodie
2018-01-03 13:31 ` Ævar Arnfjörð Bjarmason
2018-01-03 14:41 ` Adam Dinwoodie
2018-01-03 19:14 ` Ævar Arnfjörð Bjarmason
2018-01-04 11:50 ` Adam Dinwoodie
2018-01-04 19:26 ` [PATCH v4 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 00/10] " Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 01/10] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 02/10] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 03/10] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 04/10] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 05/10] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 06/10] wildmatch test: use test_must_fail, not ! for test-wildmatch Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 07/10] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 08/10] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 09/10] test-lib: add an EXPENSIVE_ON_WINDOWS prerequisite Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 10/10] wildmatch test: mark test as EXPENSIVE_ON_WINDOWS Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-05 16:41 ` Johannes Schindelin
2018-01-05 19:08 ` Ævar Arnfjörð Bjarmason
2018-01-05 20:48 ` Johannes Schindelin
2018-01-05 22:12 ` [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper Ævar Arnfjörð Bjarmason
2018-01-05 23:13 ` Junio C Hamano
2018-01-06 12:51 ` Johannes Schindelin [this message]
2018-01-06 13:32 ` Ævar Arnfjörð Bjarmason
2018-01-06 20:46 ` Johannes Schindelin
2018-01-08 12:46 ` Johannes Schindelin
2018-01-08 18:49 ` Junio C Hamano
2018-01-07 2:51 ` Duy Nguyen
2018-01-08 12:25 ` Johannes Schindelin
2018-01-10 9:07 ` Duy Nguyen
2018-01-10 10:38 ` Adam Dinwoodie
2018-01-10 10:52 ` Duy Nguyen
2018-01-10 20:24 ` Johannes Schindelin
2018-01-11 9:25 ` Duy Nguyen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=nycvar.QRO.7.76.6.1801061337020.1337@wbunaarf-fpuvaqryva.tvgsbejvaqbjf.bet \
--to=johannes.schindelin@gmx.de \
--cc=adam@dinwoodie.org \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=n.oxyde@gmail.com \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).