git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: "Junio C Hamano" <gitster@pobox.com>,
	git@vger.kernel.org, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
	"Anthony Ramine" <n.oxyde@gmail.com>,
	"Johannes Sixt" <j6t@kdbg.org>,
	"Adam Dinwoodie" <adam@dinwoodie.org>
Subject: Re: [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper
Date: Sat, 06 Jan 2018 14:32:34 +0100	[thread overview]
Message-ID: <87mv1raz9p.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <nycvar.QRO.7.76.6.1801061337020.1337@wbunaarf-fpuvaqryva.tvgsbejvaqbjf.bet>


On Sat, Jan 06 2018, Johannes Schindelin jotted:

> Hi Junio,
>
> On Fri, 5 Jan 2018, Junio C Hamano wrote:
>
>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>>
>> > Skip the newly added file creation tests on Windows proper, these
>> > already work under Cygwin, but as that involves a significant
>> > emulation layer the results are different under Windows proper with
>> > MinGW.
>> > ...
>> > diff --git a/t/t3070-wildmatch.sh b/t/t3070-wildmatch.sh
>> > index f606f91acb..50a53e7a62 100755
>> > --- a/t/t3070-wildmatch.sh
>> > +++ b/t/t3070-wildmatch.sh
>> > @@ -7,6 +7,14 @@ test_description='wildmatch tests'
>> >  create_test_file() {
>> >  	file=$1
>> >
>> > +	# These tests limp along under Cygwin, but the failures on
>> > +	# native Windows are still too many. Skip file tests there
>> > +	# until they're sorted out.
>> > +	if test_have_prereq MINGW
>> > +	then
>> > +		return 1
>> > +	fi
>> > +
>>
>> That looks to be a nuclear option.
>
> Indeed:
>
> 	# still have 84 known breakage(s)
> 	# failed 52 among remaining 1428 test(s)
> 	1..1512
>
> Let's just switch it off completely because 52 of those 1512 test cases
> break only on Windows? Pretty heavy-handed.

Can you please provide me with the output of the test under -v -x -d
from github.com:avar/git.git wildmatch-refactor-8 branch?

It has some improvements that should make things a bit faster for you,
but also most of the logic is now in test_expect_success so -x is more
helpful.

I should then be able to fix these 52 remaining cases (and at least 1 of
them should be fixed already).

> But do read on.
>
>> For now it may be suffice, but somehow it feels that it goes directly
>> against Dscho's wish to treat (or pretend to treat) Windows as
>> first-class citizens and/or just like other platforms.
>
> It not only goes against my wish to treat Windows as a normal citizen
> instead of like Rey's parents, it also goes against my wish to have a
> focused and meaningful test suite. Nobody likes to run tests that take too
> long. And look at this:
>
> 	...
> 	ok 1511 - ipathmatch: match 'Z' '[Z-y]'
> 	ok 1512 - ipathmatch(ls): match '[Z-y]' 'Z'
> 	# still have 84 known breakage(s)
> 	# failed 52 among remaining 1428 test(s)
> 	1..1512
>
> 	real    5m51.432s
> 	user    0m33.986s
> 	sys     2m13.162s
>
> Yep. It takes *over eight minutes*.
>
> And this is a *fast* machine.
>
> Why? Because of the completely overboard testing of all kinds of
> *potential* breakages *at some point* in the future.
>
> I understand that Ævar wants to increase test coverage. I am sympathetic
> to this cause.
>
> But I completely disagree that increasing the test coverage beyond reason
> is a good thing. Tests *can* take too long, and they do, in practice, and
> the results are always problematic: every developer who has to deal with
> test suites that take too long... does not run them. As simple as that.
> And then you have *zero* test coverage. Pretty stupid, eh? And contrary to
> your intentions, too.
>
> So yes, I think that it has been proven beyond any doubt that t3070 takes
> *waaaaaaaaaay* too long on Windows, for dubitable benefit.
>
> I could see a reasonable compromise where
>
> - an extensive test matrix is hidden behind an EXPENSIVE_WILDMATCH prereq,
>
> - the test matrix would be written in a much more understandable way, i.e.
>   using English words rather than "1"s. If need be, there could be blocks
>   ("test this with case-sensitive matching", "skip case-sensitive matching",
>   you get the idea),

I take your point with the readability of some of this stuff, and will
get around to fixing that before the next submission.

> - these magic skippings of certain test cases (where the (non-traced)
>   `create_test_file()` function returns 1 for certain things) still would
>   need to go away, in favor probably of prereqs and/or blocks where flags
>   are set accordingly in a preamble,

This is a lot of work for dubious benefit. I'm not going to try to
maintain some exhaustive mapping that's potentially going to be a hash
4-levels deep of:

    os.os_version.fs.vs_version

And that's before we'd get to the potential 6-level crazyness of:

    os.os_version.os_flags.fs.vs_version.fs_flags

It's way easier and more portable (even to OSs or FSs nobody's invented
yet) to just test whether a file can be created, and if not skip the
test.

As I explained in 20180105221222.28867-1-avarab@gmail.com the actual
benefit of this test is that as much as possible is tested
*somewhere*. There's no reason to suspect that e.g. Linux will overnight
make the character ":" illegal in filenames and we'll be the only ones
who notice it.

Since we don't treat ":" or any other character differently for the
purposes of glob matching on Windows, but *do* treat the raw glob
matching differently than calling wildmatch() directly, as the current
tests do, I really don't see what the point of this exercises would be.

> - by default, i.e. without the EXPENSIVE_WILDMATCH prereq, it should test
>   a *tiny* subset that is indicative of the most likely bugs.

> As it is, I like 8/7 in the present form for all the wrong reasons: it
> protects me from the damage a full t3070 would do to me.

This test takes around 8 seconds to run on a 3.4GHz i7 on Linux, and
around 4 seconds on a 2.6GHz i5 on my SSD laptop on Linux.

I agree that there should be some prerequisite to skip this on Windows
by default since 6 minutes is clearly excessive (although I think you'll
find it runs a bit faster in the branch above), but that should be
something like:

    test_lazy_prereq EXPENSIVE_ON_WINDOWS '
            test -n "$GIT_TEST_LONG" || test_have_prereq !MINGW,!CYGWIN
    '

As the long runtime is not inherent to the test, but to excessive
slowness caused by certain operations on certain platforms, or maybe it
should be EXPENSIVE_ON_SLOW_FS or EXPENSIVE_IF_FORKING_IS_SLOW or if
we'd like to generalize it.

  reply	other threads:[~2018-01-06 13:32 UTC|newest]

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-12-23 21:30 [PATCH 0/6] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 1/6] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 2/6] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 3/6] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 4/6] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 5/6] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-23 21:30 ` [PATCH 6/6] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-24  9:24   ` Johannes Sixt
2017-12-24 11:06     ` Ævar Arnfjörð Bjarmason
2017-12-24 11:51       ` Johannes Sixt
2017-12-25  0:28 ` [PATCH v2 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2017-12-28 22:48   ` Junio C Hamano
2017-12-28 23:49     ` Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 " Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 23:28   ` [PATCH v3 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-29  0:16     ` Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2017-12-25  0:28 ` [PATCH v2 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2017-12-28 20:28   ` Junio C Hamano
2017-12-25  0:28 ` [PATCH v2 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2017-12-25  9:26   ` Johannes Sixt
2017-12-27 19:07   ` Junio C Hamano
2018-01-03 13:02   ` Adam Dinwoodie
2018-01-03 13:31     ` Ævar Arnfjörð Bjarmason
2018-01-03 14:41       ` Adam Dinwoodie
2018-01-03 19:14         ` Ævar Arnfjörð Bjarmason
2018-01-04 11:50           ` Adam Dinwoodie
2018-01-04 19:26             ` [PATCH v4 0/7] increase wildmatch test coverage Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 00/10] " Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 01/10] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 02/10] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 03/10] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 04/10] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 05/10] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 06/10] wildmatch test: use test_must_fail, not ! for test-wildmatch Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 07/10] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 08/10] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 09/10] test-lib: add an EXPENSIVE_ON_WINDOWS prerequisite Ævar Arnfjörð Bjarmason
2018-01-30 21:21               ` [PATCH v5 10/10] wildmatch test: mark test as EXPENSIVE_ON_WINDOWS Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 1/7] wildmatch test: indent with tabs, not spaces Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 2/7] wildmatch test: use more standard shell style Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 3/7] wildmatch test: don't try to vertically align our output Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 4/7] wildmatch test: use a paranoia pattern from nul_match() Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 5/7] wildmatch test: remove dead fnmatch() test code Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 6/7] wildmatch test: perform all tests under all wildmatch() modes Ævar Arnfjörð Bjarmason
2018-01-04 19:26             ` [PATCH v4 7/7] wildmatch test: create & test files on disk in addition to in-memory Ævar Arnfjörð Bjarmason
2018-01-05 16:41               ` Johannes Schindelin
2018-01-05 19:08                 ` Ævar Arnfjörð Bjarmason
2018-01-05 20:48                   ` Johannes Schindelin
2018-01-05 22:12                     ` [PATCH v4 8/7] wildmatch test: skip file creation tests on Windows proper Ævar Arnfjörð Bjarmason
2018-01-05 23:13                       ` Junio C Hamano
2018-01-06 12:51                         ` Johannes Schindelin
2018-01-06 13:32                           ` Ævar Arnfjörð Bjarmason [this message]
2018-01-06 20:46                             ` Johannes Schindelin
2018-01-08 12:46                             ` Johannes Schindelin
2018-01-08 18:49                             ` Junio C Hamano
2018-01-07  2:51                           ` Duy Nguyen
2018-01-08 12:25                             ` Johannes Schindelin
2018-01-10  9:07                               ` Duy Nguyen
2018-01-10 10:38                                 ` Adam Dinwoodie
2018-01-10 10:52                                   ` Duy Nguyen
2018-01-10 20:24                                 ` Johannes Schindelin
2018-01-11  9:25                                   ` Duy Nguyen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87mv1raz9p.fsf@evledraar.gmail.com \
    --to=avarab@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=adam@dinwoodie.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=j6t@kdbg.org \
    --cc=n.oxyde@gmail.com \
    --cc=pclouds@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).