[PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[Ævar Arnfjörð Bjarmason]

When t7610-mergetool.sh runs without failures the git://example.com
submodule URLs will never be used. That's because we "git submodule
add" it, but then manually populate them so that subsequent "git
submodule update -N" won't attempt to clone it, only update it without
fetching.

But if we fail in an earlier test it'll have the knock-on effect of
having later tests hang on that "git submodule update -N" as we
attempt to clone this repository from example.com.

This can be reproduced on "master" by running the test with
SANITIZE=leak without "--immediate". With
"GIT_TEST_PASSING_SANITIZE_LEAK=true" (which the linux-leaks job uses)
we'll skip the test entirely. So we'll only run into this when running
it manually, or with the "GIT_TEST_PASSING_SANITIZE_LEAK=check" mode.

That's not because the failure has anything to do with leak detection
per-se. It just so happens that we have a leak that'll fail before
we've managed to fully set these up, and therefore "git submodule
update -N" ends up spawning "git clone".

Let's instead continue lying about the origin of this submodule by
providing a URL for it that doesn't work, but now one that *really*
doesn't work: /dev/null. If the test is passing we won't ever use
this, and if we have knock-on failures we'll fail early, instead of
waiting for a timeout.

The behavior of "-N" here might be surprising to some, since it's
explained as "[if you use -N we] don’t fetch new objects from the
remote site". But (perhaps counter-intuitively) it's only talking
about if it needs to do so via "git fetch". In this case we'll end up
spawning a "git clone", as we have no submodule set up.

See ff7f089ed10 (mergetool: Teach about submodules, 2011-04-13) for
the commit that implemented these "example.com" tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t7610-mergetool.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh
index 8cc64729adb..b1ba0d9a088 100755
--- a/t/t7610-mergetool.sh
+++ b/t/t7610-mergetool.sh
@@ -33,7 +33,7 @@ test_expect_success 'setup' '
 		git add foo &&
 		git commit -m "Add foo"
 	) &&
-	git submodule add git://example.com/submod submod &&
+	git submodule add /dev/null submod &&
 	git add file1 "spaced name" file1[1-4] subdir/file3 .gitmodules submod &&
 	git commit -m "add initial versions" &&
 
@@ -614,7 +614,7 @@ test_expect_success 'submodule in subdirectory' '
 		)
 	) &&
 	test_when_finished "rm -rf subdir/subdir_module" &&
-	git submodule add git://example.com/subsubmodule subdir/subdir_module &&
+	git submodule add /dev/null subdir/subdir_module &&
 	git add subdir/subdir_module &&
 	git commit -m "add submodule in subdirectory" &&
 
-- 
2.38.0.1452.g710f45c7951

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[Taylor Blau]

On Sat, Nov 05, 2022 at 12:54:21PM +0100, Ævar Arnfjörð Bjarmason wrote:
> The behavior of "-N" here might be surprising to some, since it's
> explained as "[if you use -N we] don’t fetch new objects from the
> remote site". But (perhaps counter-intuitively) it's only talking
> about if it needs to do so via "git fetch". In this case we'll end up
> spawning a "git clone", as we have no submodule set up.

Makes sense, though I'm not sure I agree this is worth patching as I
currently understand it.

If I run t7610 today with '--run=2-' (IOW, skipping the setup test), I
am definitely going to get failures. And I don't think we should have
every subsequent test depend on having run anything containing "setup"
before it. That is, it is not surprising that we will see some test
failures when carving up and running portions of the test instead of the
whole file.

I don't think this is substantively any different than that. Whether we
don't complete the "setup" test because we had some leak (and ran the
test suite with the appropriate LSan options), or because we neglected
to run it at all, I don't think there is a significant difference.

Either way, the end-state of the test repository isn't guaranteed to
match the intent of the "setup" test.

If this is the only such problem in-tree, sure, I think it's fine to
patch. But I would wager that there are *many* more than just this one
lurking, and patching all of them would be less straightforward than
this one.

So... I don't know. I'm certainly leaning negative on this patch, but if
you have some compelling reason that I'm missing, I'm all-ears.

Thanks,
Taylor

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[Ævar Arnfjörð Bjarmason]

On Wed, Nov 09 2022, Taylor Blau wrote:

> On Sat, Nov 05, 2022 at 12:54:21PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> The behavior of "-N" here might be surprising to some, since it's
>> explained as "[if you use -N we] don’t fetch new objects from the
>> remote site". But (perhaps counter-intuitively) it's only talking
>> about if it needs to do so via "git fetch". In this case we'll end up
>> spawning a "git clone", as we have no submodule set up.
>
> Makes sense, though I'm not sure I agree this is worth patching as I
> currently understand it.
>
> If I run t7610 today with '--run=2-' (IOW, skipping the setup test), I
> am definitely going to get failures. And I don't think we should have
> every subsequent test depend on having run anything containing "setup"
> before it. That is, it is not surprising that we will see some test
> failures when carving up and running portions of the test instead of the
> whole file.
>
> I don't think this is substantively any different than that. Whether we
> don't complete the "setup" test because we had some leak (and ran the
> test suite with the appropriate LSan options), or because we neglected
> to run it at all, I don't think there is a significant difference.
>
> Either way, the end-state of the test repository isn't guaranteed to
> match the intent of the "setup" test.
>
> If this is the only such problem in-tree, sure, I think it's fine to
> patch. But I would wager that there are *many* more than just this one
> lurking, and patching all of them would be less straightforward than
> this one.
>
> So... I don't know. I'm certainly leaning negative on this patch, but if
> you have some compelling reason that I'm missing, I'm all-ears.

I agree with that in general, but the expected failure case for all
those other things that are broken with the missing setup is just that
we'll predictably error out immediately.

Whereas in this case we'll end up hanging on a connect() to some box
that IANA's maintaining for the example.com landing page, and after we
reach the connect() timeout we move onto the next failing test, which
will do that again. The test takes a long time to finally report an
overall failure.

I think that's worth fixing in general, aside from the leak
detection. I.e. yes in general we aren't good about running tests 3..9
successfully if the 2nd test fails.

But we generally just fail some or all of 3..9 pretty fast, and don't
start taking 20 minutes to run the test, when it took 10s before (or
whatever).

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[Taylor Blau]

On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
> But we generally just fail some or all of 3..9 pretty fast, and don't
> start taking 20 minutes to run the test, when it took 10s before (or
> whatever).

OK. I still think that in principle this is indistinguishable from not
running a setup test to completion.

*But*: I'm willing to treat them differently since instead of
manifesting in an immediate failure later on in the suite, we hang for a
substantial period of time.

So I'm content to merge this down, but I don't think it's worth
searching out for more of these in the future.

Thanks,
Taylor

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

[Taylor Blau]

On Wed, Nov 09, 2022 at 09:17:25PM -0500, Taylor Blau wrote:
> On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
> > But we generally just fail some or all of 3..9 pretty fast, and don't
> > start taking 20 minutes to run the test, when it took 10s before (or
> > whatever).
>
> OK. I still think that in principle this is indistinguishable from not
> running a setup test to completion.
>
> *But*: I'm willing to treat them differently since instead of
> manifesting in an immediate failure later on in the suite, we hang for a
> substantial period of time.
>
> So I'm content to merge this down, but I don't think it's worth
> searching out for more of these in the future.

Having merged this down to 'next', it looks like there is some CI
fallout on the Windows tests, see:

  https://github.com/ttaylorr/git/actions/runs/3473324797/jobs/5805324776

I am not sure how I might have caught this earlier not having a Windows
machine myself. Regardless, let's make sure that it is fixed up before
this graduates.

Thanks,
Taylor

[PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

[Ævar Arnfjörð Bjarmason]

On MinGW the "/dev/null" is translated to "nul" on command-lines, even
though as in this case it'll never end up referring to an actual file.

So on Windows the fix for the previous "example.com" timeout issue in
8354cf752ec (t7610: fix flaky timeout issue, don't clone from
example.com, 2022-11-05) would yield:

  fatal: repo URL: 'nul' must be absolute or begin with ./|../

Let's evade this yet again by prefixing this with "file://", which
makes this pass in the Windows CI.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---

On Tue, Nov 15 2022, Taylor Blau wrote:

> On Wed, Nov 09, 2022 at 09:17:25PM -0500, Taylor Blau wrote:
>> On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
>> > But we generally just fail some or all of 3..9 pretty fast, and don't
>> > start taking 20 minutes to run the test, when it took 10s before (or
>> > whatever).
>>
>> OK. I still think that in principle this is indistinguishable from not
>> running a setup test to completion.
>>
>> *But*: I'm willing to treat them differently since instead of
>> manifesting in an immediate failure later on in the suite, we hang for a
>> substantial period of time.
>>
>> So I'm content to merge this down, but I don't think it's worth
>> searching out for more of these in the future.
>
> Having merged this down to 'next', it looks like there is some CI
> fallout on the Windows tests, see:
>
>   https://github.com/ttaylorr/git/actions/runs/3473324797/jobs/5805324776
>
> I am not sure how I might have caught this earlier not having a Windows
> machine myself. Regardless, let's make sure that it is fixed up before
> this graduates.

Sorry again :( I think my CI was quite queued up at the time, and I
figured surely *this* won't have any porability issues, but forgot
about MinGW's sneaky /dev/null string-replacement behavior.

Windows CI passes with this:
https://github.com/avar/git/actions/runs/3473817195

 t/t7610-mergetool.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh
index b1ba0d9a088..7b957022f1a 100755
--- a/t/t7610-mergetool.sh
+++ b/t/t7610-mergetool.sh
@@ -33,7 +33,7 @@ test_expect_success 'setup' '
 		git add foo &&
 		git commit -m "Add foo"
 	) &&
-	git submodule add /dev/null submod &&
+	git submodule add file:///dev/null submod &&
 	git add file1 "spaced name" file1[1-4] subdir/file3 .gitmodules submod &&
 	git commit -m "add initial versions" &&
 
@@ -614,7 +614,7 @@ test_expect_success 'submodule in subdirectory' '
 		)
 	) &&
 	test_when_finished "rm -rf subdir/subdir_module" &&
-	git submodule add /dev/null subdir/subdir_module &&
+	git submodule add file:///dev/null subdir/subdir_module &&
 	git add subdir/subdir_module &&
 	git commit -m "add submodule in subdirectory" &&
 
-- 
2.38.0.1473.g172bcc0511c

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

[Taylor Blau]

On Wed, Nov 16, 2022 at 12:40:14AM +0100, Ævar Arnfjörð Bjarmason wrote:
> On MinGW the "/dev/null" is translated to "nul" on command-lines, even
> though as in this case it'll never end up referring to an actual file.
>
> So on Windows the fix for the previous "example.com" timeout issue in
> 8354cf752ec (t7610: fix flaky timeout issue, don't clone from
> example.com, 2022-11-05) would yield:
>
>   fatal: repo URL: 'nul' must be absolute or begin with ./|../
>
> Let's evade this yet again by prefixing this with "file://", which
> makes this pass in the Windows CI.

Thanks very much. I just picked this up on top of my 'ab/t7610-timeout'
branch and pushed it out to ttaylorr/git to double-check your results
against CI. (Not that I don't trust you, of course, but I'm feeling like
I should be extra-cautious ;-)).

Assuming that all looks good, then I'll push out a new version of 'next'
and dependent branches to the usual spots so that we can get 'next' back
to green.

Thanks for working on this so quickly.

Thanks,
Taylor

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

[Ævar Arnfjörð Bjarmason]

On Tue, Nov 15 2022, Taylor Blau wrote:

> On Wed, Nov 16, 2022 at 12:40:14AM +0100, Ævar Arnfjörð Bjarmason wrote:
>> On MinGW the "/dev/null" is translated to "nul" on command-lines, even
>> though as in this case it'll never end up referring to an actual file.
>>
>> So on Windows the fix for the previous "example.com" timeout issue in
>> 8354cf752ec (t7610: fix flaky timeout issue, don't clone from
>> example.com, 2022-11-05) would yield:
>>
>>   fatal: repo URL: 'nul' must be absolute or begin with ./|../
>>
>> Let's evade this yet again by prefixing this with "file://", which
>> makes this pass in the Windows CI.
>
> Thanks very much. I just picked this up on top of my 'ab/t7610-timeout'
> branch and pushed it out to ttaylorr/git to double-check your results
> against CI. (Not that I don't trust you, of course, but I'm feeling like
> I should be extra-cautious ;-)).
>
> Assuming that all looks good, then I'll push out a new version of 'next'
> and dependent branches to the usual spots so that we can get 'next' back
> to green.
>
> Thanks for working on this so quickly.

And thank for picking it up! Looks like "next" is passing CI now:
https://github.com/git/git/actions/runs/3475751137/jobs/5810308794

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

[Taylor Blau]

On Thu, Nov 17, 2022 at 11:08:08AM +0100, Ævar Arnfjörð Bjarmason wrote:
> And thank for picking it up! Looks like "next" is passing CI now:
> https://github.com/git/git/actions/runs/3475751137/jobs/5810308794

No problem. I dislike pushing out 'next' without a corresponding WC
report, but I dislike 'next' failing CI even more. So I felt like here
it was OK.

Thanks,
Taylor