Re: ab/cmake-nix-and-ci, was Re: [PATCH] test-lib.sh: discover "git" in subdirs of "contrib/buildsystems/out"

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Fri, Dec 09 2022, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
>> Junio, maybe you could clarify your take on this? As project lead, it is
>> your decision to define how Git uses Continuous Builds, and how the
>> project handles failed CI runs.
>
> I have pretty much been with what Peff and Taylor said in the thread
> already ever since we added CMake support to help Windows/VS folks.
> I agree with you that we do not need to run it for Linux or macOS,
> and if the promised/hoped-for benefit, i.e. that running them on
> non-Windows build would uncover issues that are common across the
> platform and help Windows, is not something that is likely to
> materialize, I'd prefer to see our resources (CI time and developer
> attention) not spent on that.

I think this should be addressed by the "I count less than 20 lines in a
~1.1k line recipe that are really "MSVC"-specific" in the sibling
mail[1]. I.e. the large majority of it is generic recipe code that's run
on all platforms.

> I do not think "how the project handles filed CI runs" is a very big
> issue.  I often ignore partial failures (e.g. "winVS(n) test job
> triggered rate limit") and the only annoyance I feel is that such a
> temporary failure contribute one more message to my trash mailbox,
> and I can learn to do the same for a test that marked as failed due
> to linux-cmake-ctest job.  I expect that regular contributors are
> doing the same pretty much.
>
> How blocking is a CI failure for drive-by contributors who use GGG?
> While I do not necessarily value drive-by contributions as much as
> you do, if such "an unimportant failure we can ignore" discourages
> those coming from GGG route, that would be unfortunate, exactly
> because they may not have contributed anything to the failures.
> This is not just cmake-ctest, but the leak checking job where a new
> use of a tool that is known to be leaky in a test can turn a test
> that has been passing to fail.  If we can mark failures in selected
> jobs as non-blocking, we definitely should do so.

I realize that we've been digressing to the larger topic of what to do
with CI in general, but I don't think the question of whether say
"win+VS build" should "soft fail" is something we should conflate with
this "ab/cmake-nix-and-ci" topic.

It doesn't change the status quo there, and I think is a net
improvement.

Even if we make the CI for anything cmake-related soft-fail, this topic
will still help to get it back up to speed, as you'll be able to run the
full cmake+ctest chain on non-Windows.

> Between keeping and marking linux-cmake-ctest as non-blocking, and
> removing it altogether, I am inclined to say that I'd favor the
> latter for the reasons I explained earlier in this message.  But to
> help casual contributors coming via GGG, we would anyway need to (1)
> allow submitting even with failing tests, and (2) tell them that it
> is OK to do so.  Which means it is not the end of the world, from
> the point of view of helping casual developers, if we had kept these
> brittle CI jobs like linux-cmake-ctest and linux-leaks.

I can peel off the commit that adds the "linux-cmake-ctest" CI job from
this series, or even just make it do the equivalent of:

	cmake && make || echo oops, we're broken

So it'll "soft-fail" (AFAIK GitHub CI, unlike GitLab[2] doesn't support
a native way to "soft-fail").

But I don't think that doing that would help without *also* making
"win+VS {build,test}" soft-fail. I.e. if "linux-cmake-ctest" *and*
"win+VS" (with all else passing) you can be pretty sure it's a generic
cmake problem.

If only one or the other is failing somewhere in cmake having the
"linux-cmake-ctest" job now will help narrow down whether it's a
platform-specific cmake issue.

So, just let me know what you'd prefer, but I think per the above even
if you're impatient with cmake failures the "linux-cmake-ctest" job
should help spend less time on them.

1. https://lore.kernel.org/git/221208.86wn726qcv.gmgdl@evledraar.gmail.com/
2. https://docs.gitlab.com/ee/ci/yaml/#allow_failure -- In the UX:
   green=passing, red=failing, yellow=soft-fail