Re: [PATCH v2 0/9] ci: make Git's GitHub workflow output much more helpful

[Junio C Hamano <gitster@pobox.com>]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> * print the verbose logs only for the failed test cases (to massively cut
>>   down on the size of the log, particularly when there's only a couple
>>   failures in a test file with a lot of passing tests).
>
> That's an amazingly simple trick to improve the speed by a ton, indeed.
> Thank you for this splendid idea!
>
>> * skip printing the full text of the test in 'finalize_test_case_output'
>>   when creating the group, i.e., use '$1' instead of '$*' (in both passing
>>   and failing tests, this information is already printed via some other
>>   means).
>>
>> If you wanted to make sure a user could still access the full failure logs
>> (i.e., including the "ok" test results), you could print a link to the
>> artifacts page as well - that way, all of the information we currently
>> provide to users can still be found somewhere.
>
> That's a good point, I added that hint to the output (the link is
> unfortunately not available at the time we print that advice).

https://github.com/git/git/runs/6539786128 shows that all in-flight
topics merged to 'seen', except for the ds/bundle-uri-more, passes
the linux-leaks job.  The ds/bundle-uri-more topic introduces some
leaks to commands that happen to be used in tests that are marked as
leak-checker clean, making the job fail.

Which makes a great guinea pig for the CI output improvement topic.

So, I created two variants of 'seen' with this linux-leaks breakage.
One is with the js/ci-github-workflow-markup topic on this thread.
The other one is with the ab/ci-github-workflow-markup topic (which
uses a preliminary clean-up ab/ci-setup-simplify topic as its base).
They should show the identical test results and failures.

And here are their output:

 - https://github.com/git/git/runs/6539835065
 - https://github.com/git/git/runs/6539900608

If I recall correctly, the selling point of the ab/* variant over
js/* variant was that it would give quicker UI response compared to
the former, but other than that, both variants' UI are supposed to
be as newbie friendly as the other.

When I tried the former, it reacted too poorly to my attempt to
scroll (with mouse scroll wheel, if it makes a difference) that
sometimes I was staring a blank dark-gray space for a few seconds
waiting for it to be filled by something, which was a bit jarring
experience.  When I tried the latter, it didn't show anything to
help diagnosing the details of the breakage in "run make test" step
and the user needed to know "print test failures" needs to be looked
at, which I am not sure is an inherent limitation of the approach.
After the single extra click, navigating the test output to find the
failed steps among many others that succeeded was not a very pleasant
experience.

Those who are interested in UX experiment may want to visit these
two output to see how usable each of these is for themselves.

Thanks.