From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: "Junio C Hamano" <gitster@pobox.com>,
git@vger.kernel.org, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
"Anthony Ramine" <n.oxyde@gmail.com>,
"Johannes Sixt" <j6t@kdbg.org>,
"Adam Dinwoodie" <adam@dinwoodie.org>
Subject: Re: [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper
Date: Sat, 06 Jan 2018 14:32:34 +0100 [thread overview]
Message-ID: <87mv1raz9p.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <nycvar.QRO.7.76.6.1801061337020.1337@wbunaarf-fpuvaqryva.tvgsbejvaqbjf.bet>
On Sat, Jan 06 2018, Johannes Schindelin jotted:
> Hi Junio,
>
> On Fri, 5 Jan 2018, Junio C Hamano wrote:
>
>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>>
>> > Skip the newly added file creation tests on Windows proper, these
>> > already work under Cygwin, but as that involves a significant
>> > emulation layer the results are different under Windows proper with
>> > MinGW.
>> > ...
>> > diff --git a/t/t3070-wildmatch.sh b/t/t3070-wildmatch.sh
>> > index f606f91acb..50a53e7a62 100755
>> > --- a/t/t3070-wildmatch.sh
>> > +++ b/t/t3070-wildmatch.sh
>> > @@ -7,6 +7,14 @@ test_description='wildmatch tests'
>> > create_test_file() {
>> > file=$1
>> >
>> > + # These tests limp along under Cygwin, but the failures on
>> > + # native Windows are still too many. Skip file tests there
>> > + # until they're sorted out.
>> > + if test_have_prereq MINGW
>> > + then
>> > + return 1
>> > + fi
>> > +
>>
>> That looks to be a nuclear option.
>
> Indeed:
>
> # still have 84 known breakage(s)
> # failed 52 among remaining 1428 test(s)
> 1..1512
>
> Let's just switch it off completely because 52 of those 1512 test cases
> break only on Windows? Pretty heavy-handed.
Can you please provide me with the output of the test under -v -x -d
from github.com:avar/git.git wildmatch-refactor-8 branch?
It has some improvements that should make things a bit faster for you,
but also most of the logic is now in test_expect_success so -x is more
helpful.
I should then be able to fix these 52 remaining cases (and at least 1 of
them should be fixed already).
> But do read on.
>
>> For now it may be suffice, but somehow it feels that it goes directly
>> against Dscho's wish to treat (or pretend to treat) Windows as
>> first-class citizens and/or just like other platforms.
>
> It not only goes against my wish to treat Windows as a normal citizen
> instead of like Rey's parents, it also goes against my wish to have a
> focused and meaningful test suite. Nobody likes to run tests that take too
> long. And look at this:
>
> ...
> ok 1511 - ipathmatch: match 'Z' '[Z-y]'
> ok 1512 - ipathmatch(ls): match '[Z-y]' 'Z'
> # still have 84 known breakage(s)
> # failed 52 among remaining 1428 test(s)
> 1..1512
>
> real 5m51.432s
> user 0m33.986s
> sys 2m13.162s
>
> Yep. It takes *over eight minutes*.
>
> And this is a *fast* machine.
>
> Why? Because of the completely overboard testing of all kinds of
> *potential* breakages *at some point* in the future.
>
> I understand that Ævar wants to increase test coverage. I am sympathetic
> to this cause.
>
> But I completely disagree that increasing the test coverage beyond reason
> is a good thing. Tests *can* take too long, and they do, in practice, and
> the results are always problematic: every developer who has to deal with
> test suites that take too long... does not run them. As simple as that.
> And then you have *zero* test coverage. Pretty stupid, eh? And contrary to
> your intentions, too.
>
> So yes, I think that it has been proven beyond any doubt that t3070 takes
> *waaaaaaaaaay* too long on Windows, for dubitable benefit.
>
> I could see a reasonable compromise where
>
> - an extensive test matrix is hidden behind an EXPENSIVE_WILDMATCH prereq,
>
> - the test matrix would be written in a much more understandable way, i.e.
> using English words rather than "1"s. If need be, there could be blocks
> ("test this with case-sensitive matching", "skip case-sensitive matching",
> you get the idea),
I take your point with the readability of some of this stuff, and will
get around to fixing that before the next submission.
> - these magic skippings of certain test cases (where the (non-traced)
> `create_test_file()` function returns 1 for certain things) still would
> need to go away, in favor probably of prereqs and/or blocks where flags
> are set accordingly in a preamble,
This is a lot of work for dubious benefit. I'm not going to try to
maintain some exhaustive mapping that's potentially going to be a hash
4-levels deep of:
os.os_version.fs.vs_version
And that's before we'd get to the potential 6-level crazyness of:
os.os_version.os_flags.fs.vs_version.fs_flags
It's way easier and more portable (even to OSs or FSs nobody's invented
yet) to just test whether a file can be created, and if not skip the
test.
As I explained in 20180105221222.28867-1-avarab@gmail.com the actual
benefit of this test is that as much as possible is tested
*somewhere*. There's no reason to suspect that e.g. Linux will overnight
make the character ":" illegal in filenames and we'll be the only ones
who notice it.
Since we don't treat ":" or any other character differently for the
purposes of glob matching on Windows, but *do* treat the raw glob
matching differently than calling wildmatch() directly, as the current
tests do, I really don't see what the point of this exercises would be.
> - by default, i.e. without the EXPENSIVE_WILDMATCH prereq, it should test
> a *tiny* subset that is indicative of the most likely bugs.
> As it is, I like 8/7 in the present form for all the wrong reasons: it
> protects me from the damage a full t3070 would do to me.
This test takes around 8 seconds to run on a 3.4GHz i7 on Linux, and
around 4 seconds on a 2.6GHz i5 on my SSD laptop on Linux.
I agree that there should be some prerequisite to skip this on Windows
by default since 6 minutes is clearly excessive (although I think you'll
find it runs a bit faster in the branch above), but that should be
something like:
test_lazy_prereq EXPENSIVE_ON_WINDOWS '
test -n "$GIT_TEST_LONG" || test_have_prereq !MINGW,!CYGWIN
'
As the long runtime is not inherent to the test, but to excessive
slowness caused by certain operations on certain platforms, or maybe it
should be EXPENSIVE_ON_SLOW_FS or EXPENSIVE_IF_FORKING_IS_SLOW or if
we'd like to generalize it.
next prev parent reply other threads:[~2018-01-06 13:32 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-23 21:30 [PATCH 0/6] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 1/6] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 2/6] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 3/6] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 4/6] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 5/6] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 6/6] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-24 9:24 ` Johannes Sixt
2017-12-24 11:06 ` Ævar Arnfjörð Bjarmason
2017-12-24 11:51 ` Johannes Sixt
2017-12-25 0:28 ` [PATCH v2 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-28 22:48 ` Junio C Hamano
2017-12-28 23:49 ` Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 " Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 23:28 ` [PATCH v3 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-29 0:16 ` Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-25 0:28 ` [PATCH v2 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 20:28 ` Junio C Hamano
2017-12-25 0:28 ` [PATCH v2 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-25 9:26 ` Johannes Sixt
2017-12-27 19:07 ` Junio C Hamano
2018-01-03 13:02 ` Adam Dinwoodie
2018-01-03 13:31 ` Ævar Arnfjörð Bjarmason
2018-01-03 14:41 ` Adam Dinwoodie
2018-01-03 19:14 ` Ævar Arnfjörð Bjarmason
2018-01-04 11:50 ` Adam Dinwoodie
2018-01-04 19:26 ` [PATCH v4 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 00/10] " Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 01/10] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 02/10] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 03/10] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 04/10] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 05/10] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 06/10] wildmatch test: use test_must_fail, not ! for test-wildmatch Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 07/10] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 08/10] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 09/10] test-lib: add an EXPENSIVE_ON_WINDOWS prerequisite Ævar Arnfjörð Bjarmason
2018-01-30 21:21 ` [PATCH v5 10/10] wildmatch test: mark test as EXPENSIVE_ON_WINDOWS Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-04 19:26 ` [PATCH v4 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-05 16:41 ` Johannes Schindelin
2018-01-05 19:08 ` Ævar Arnfjörð Bjarmason
2018-01-05 20:48 ` Johannes Schindelin
2018-01-05 22:12 ` [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper Ævar Arnfjörð Bjarmason
2018-01-05 23:13 ` Junio C Hamano
2018-01-06 12:51 ` Johannes Schindelin
2018-01-06 13:32 ` Ævar Arnfjörð Bjarmason [this message]
2018-01-06 20:46 ` Johannes Schindelin
2018-01-08 12:46 ` Johannes Schindelin
2018-01-08 18:49 ` Junio C Hamano
2018-01-07 2:51 ` Duy Nguyen
2018-01-08 12:25 ` Johannes Schindelin
2018-01-10 9:07 ` Duy Nguyen
2018-01-10 10:38 ` Adam Dinwoodie
2018-01-10 10:52 ` Duy Nguyen
2018-01-10 20:24 ` Johannes Schindelin
2018-01-11 9:25 ` Duy Nguyen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87mv1raz9p.fsf@evledraar.gmail.com \
--to=avarab@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=adam@dinwoodie.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=n.oxyde@gmail.com \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).