From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Victoria Dye <vdye@github.com>
Cc: Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com>,
git@vger.kernel.org, Eric Sunshine <sunshine@sunshineco.com>,
Johannes Schindelin <johannes.schindelin@gmx.de>
Subject: Re: [PATCH v2 0/9] ci: make Git's GitHub workflow output much more helpful
Date: Fri, 25 Mar 2022 10:02:07 +0100 [thread overview]
Message-ID: <220325.864k3mmm2x.gmgdl@evledraar.gmail.com> (raw)
In-Reply-To: <6b83bb83-32b9-20c9-fa02-c1c3170351c3@github.com>
On Thu, Mar 24 2022, Victoria Dye wrote:
> Johannes Schindelin via GitGitGadget wrote:
> [...]
>> Is this the best UI we can have for test failures in CI runs? I hope we can
>> do better. Having said that, this patch series presents a pretty good start,
>> and offers a basis for future improvements.
>>
>
> I think these are really valuable improvements over our current state, but I
> also understand the concerns about performance elsewhere in this thread
> (it's really slow to load for me as well, and scrolling/expanding the log
> groups can be a bit glitchy in my browser). That said, I think there are a
> couple ways you could improve the load time without sacrificing the (very
> helpful) improvements you've made to error log visibility. I experimented a
> bit (example result [1]) and came up with some things that may help:
>
> * group errors by test file, rather than by test case (to reduce
> parsing/rendering time for lots of groups).
> * print the verbose logs only for the failed test cases (to massively cut
> down on the size of the log, particularly when there's only a couple
> failures in a test file with a lot of passing tests).
> * skip printing the full text of the test in 'finalize_test_case_output'
> when creating the group, i.e., use '$1' instead of '$*' (in both passing
> and failing tests, this information is already printed via some other
> means).
>
> If you wanted to make sure a user could still access the full failure logs
> (i.e., including the "ok" test results), you could print a link to the
> artifacts page as well - that way, all of the information we currently
> provide to users can still be found somewhere.
>
> [1] https://github.com/vdye/git/runs/5666973267
Thanks a lot for trying to address those concerns.
I took a look at this and it definitely performs better, although in
this case the overall output is ~3k lines.
I'd be curious to see how it performs on some of the cases discussed in
earlier threads of >~50k lines, although it looks like in this case that
would require failures to be really widespread in the test suite.
I just looked at this briefly, but looking at the branch I see you
removed the "checking known breakage of[...]" etc. from the non-GitHub
markdown output, I didn't spot how that was related/needed.
>> Johannes Schindelin (9):
>> ci: fix code style
>> ci/run-build-and-tests: take a more high-level view
>> ci: make it easier to find failed tests' logs in the GitHub workflow
>> ci/run-build-and-tests: add some structure to the GitHub workflow
>> output
>> tests: refactor --write-junit-xml code
>> test(junit): avoid line feeds in XML attributes
>> ci: optionally mark up output in the GitHub workflow
>> ci: use `--github-workflow-markup` in the GitHub workflow
>> ci: call `finalize_test_case_output` a little later
>>
>
> The organization of these commits makes the series a bit confusing to read,
> mainly due to the JUnit changes in the middle. Patches 5-6 don't appear to
> be dependent on patches 1-4, so they could be moved to the beginning (after
> patch 1). With that change, I think this series would flow more smoothly:
> "Cleanup/non-functional change" -> "JUnit XML improvements" -> "Log UX
> improvements".
Have you had a change to look at the approach my suggestion of an
alternate approach to the early part of this series takes?:
https://lore.kernel.org/git/cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com/
I.e. to not build up ci/lib.sh to know to group the "build" etc. within
the "run-build-and-test" step, but instead just to pull those to the
top-level by running separate build & test steps.
next prev parent reply other threads:[~2022-03-25 9:13 UTC|newest]
Thread overview: 98+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-24 18:56 [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful Johannes Schindelin via GitGitGadget
2022-01-24 18:56 ` [PATCH 1/9] ci: fix code style Johannes Schindelin via GitGitGadget
2022-01-24 18:56 ` [PATCH 2/9] ci/run-build-and-tests: take a more high-level view Johannes Schindelin via GitGitGadget
2022-01-24 23:22 ` Eric Sunshine
2022-01-25 14:34 ` Johannes Schindelin
2022-01-24 18:56 ` [PATCH 3/9] ci: make it easier to find failed tests' logs in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-01-25 23:48 ` Ævar Arnfjörð Bjarmason
2022-01-24 18:56 ` [PATCH 4/9] ci/run-build-and-tests: add some structure to the GitHub workflow output Johannes Schindelin via GitGitGadget
2022-02-23 12:13 ` Phillip Wood
2022-02-25 13:40 ` Johannes Schindelin
2022-01-24 18:56 ` [PATCH 5/9] tests: refactor --write-junit-xml code Johannes Schindelin via GitGitGadget
2022-01-26 0:10 ` Ævar Arnfjörð Bjarmason
2022-01-24 18:56 ` [PATCH 6/9] test(junit): avoid line feeds in XML attributes Johannes Schindelin via GitGitGadget
2022-01-24 18:56 ` [PATCH 7/9] ci: optionally mark up output in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-01-24 18:56 ` [PATCH 8/9] ci: use `--github-workflow-markup` " Johannes Schindelin via GitGitGadget
2022-01-24 18:56 ` [PATCH 9/9] ci: call `finalize_test_case_output` a little later Johannes Schindelin via GitGitGadget
2022-01-26 0:25 ` [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful Ævar Arnfjörð Bjarmason
2022-01-27 16:31 ` CI "grouping" within jobs v.s. lighter split-out jobs (was: [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful) Ævar Arnfjörð Bjarmason
2022-02-19 23:46 ` [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful Johannes Schindelin
2022-02-20 2:44 ` Junio C Hamano
2022-02-20 15:25 ` Johannes Schindelin
2022-02-21 8:09 ` Ævar Arnfjörð Bjarmason
2022-02-22 10:26 ` Johannes Schindelin
2022-02-20 12:47 ` Ævar Arnfjörð Bjarmason
2022-02-22 10:30 ` Johannes Schindelin
2022-02-22 13:31 ` Ævar Arnfjörð Bjarmason
2022-02-23 12:07 ` Phillip Wood
2022-02-25 12:39 ` Ævar Arnfjörð Bjarmason
2022-02-25 14:10 ` Johannes Schindelin
2022-02-25 18:16 ` Junio C Hamano
2022-02-26 18:43 ` Junio C Hamano
2022-03-01 2:59 ` Junio C Hamano
2022-03-01 6:35 ` Junio C Hamano
2022-03-01 10:18 ` Johannes Schindelin
2022-03-01 16:52 ` Junio C Hamano
2022-03-01 10:10 ` Johannes Schindelin
2022-03-01 16:57 ` Junio C Hamano
2022-03-01 10:20 ` Johannes Schindelin
2022-03-04 7:38 ` win+VS environment has "cut" but not "paste"? Junio C Hamano
2022-03-04 9:04 ` Ævar Arnfjörð Bjarmason
2022-03-07 15:51 ` Johannes Schindelin
2022-03-07 17:05 ` Junio C Hamano
2022-03-09 13:02 ` Johannes Schindelin
2022-03-10 15:23 ` Ævar Arnfjörð Bjarmason
2022-03-07 15:48 ` Johannes Schindelin
2022-03-07 16:58 ` Junio C Hamano
2022-03-02 10:58 ` [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful Phillip Wood
2022-03-07 16:07 ` Johannes Schindelin
2022-03-07 17:11 ` Junio C Hamano
2022-03-09 11:44 ` Ævar Arnfjörð Bjarmason
2022-03-07 17:12 ` Phillip Wood
2022-03-01 10:24 ` [PATCH v2 " Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 1/9] ci: fix code style Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 2/9] ci/run-build-and-tests: take a more high-level view Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 3/9] ci: make it easier to find failed tests' logs in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 4/9] ci/run-build-and-tests: add some structure to the GitHub workflow output Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 5/9] tests: refactor --write-junit-xml code Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 6/9] test(junit): avoid line feeds in XML attributes Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 7/9] ci: optionally mark up output in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 8/9] ci: use `--github-workflow-markup` " Johannes Schindelin via GitGitGadget
2022-03-01 10:24 ` [PATCH v2 9/9] ci: call `finalize_test_case_output` a little later Johannes Schindelin via GitGitGadget
2022-03-01 19:07 ` [PATCH v2 0/9] ci: make Git's GitHub workflow output much more helpful Junio C Hamano
2022-03-02 12:22 ` Ævar Arnfjörð Bjarmason
2022-03-07 15:57 ` Johannes Schindelin
2022-03-07 16:05 ` Ævar Arnfjörð Bjarmason
2022-03-07 17:36 ` Junio C Hamano
2022-03-09 10:56 ` Ævar Arnfjörð Bjarmason
2022-03-09 13:20 ` Johannes Schindelin
2022-03-09 19:39 ` Junio C Hamano
2022-03-09 19:47 ` Ævar Arnfjörð Bjarmason
2022-03-25 0:48 ` Victoria Dye
2022-03-25 9:02 ` Ævar Arnfjörð Bjarmason [this message]
2022-03-25 18:38 ` Victoria Dye
2022-05-21 21:42 ` Johannes Schindelin
2022-05-21 23:05 ` Junio C Hamano
2022-05-22 18:48 ` Johannes Schindelin
2022-05-22 19:10 ` Junio C Hamano
2022-05-23 12:58 ` Johannes Schindelin
2022-05-22 23:27 ` Junio C Hamano
2022-05-23 18:55 ` Junio C Hamano
2022-05-23 19:21 ` Johannes Schindelin
2022-05-23 9:05 ` Ævar Arnfjörð Bjarmason
2022-05-23 18:41 ` Johannes Schindelin
2022-05-24 8:40 ` Ævar Arnfjörð Bjarmason
2022-05-21 22:18 ` [PATCH v3 00/12] " Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 01/12] ci: fix code style Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 02/12] tests: refactor --write-junit-xml code Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 03/12] test(junit): avoid line feeds in XML attributes Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 04/12] ci/run-build-and-tests: take a more high-level view Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 05/12] ci: make it easier to find failed tests' logs in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 06/12] ci/run-build-and-tests: add some structure to the GitHub workflow output Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 07/12] ci: optionally mark up output in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 08/12] ci(github): skip the logs of the successful test cases Johannes Schindelin via GitGitGadget
2022-05-24 10:47 ` Ævar Arnfjörð Bjarmason
2022-05-21 22:18 ` [PATCH v3 09/12] ci(github): avoid printing test case preamble twice Victoria Dye via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 10/12] ci: use `--github-workflow-markup` in the GitHub workflow Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 11/12] ci(github): mention where the full logs can be found Johannes Schindelin via GitGitGadget
2022-05-21 22:18 ` [PATCH v3 12/12] ci: call `finalize_test_case_output` a little later Johannes Schindelin via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=220325.864k3mmm2x.gmgdl@evledraar.gmail.com \
--to=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=johannes.schindelin@gmx.de \
--cc=sunshine@sunshineco.com \
--cc=vdye@github.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).