From: Johannes Schindelin <Johannes.Schindelin@gmx.de>
To: git-for-windows@googlegroups.com
Cc: git@vger.kernel.org
Subject: Continuous Testing of Git on Windows
Date: Fri, 10 Feb 2017 13:24:20 +0100 (CET) [thread overview]
Message-ID: <alpine.DEB.2.20.1702101241210.3496@virtualbox> (raw)
Hi team,
at the GitMerge conference, I heard a couple of times "I do not bother
reporting bugs in `pu` or `next` on MacOSX and Windows anymore, only Linux
seems to be truly supported" or some variation thereof.
This is a strong indicator that we have some room for improvement here.
Active readers of the Git mailing list will not be surprised to read that
I think we have to react to build/test failures quicker, that it is not
enough to declare it okay for those integration branches to fail the test
suite or even to fail to build[*1*].
In that vein, I worked quite a bit on "Continuous Integration", or more
appropriately: Continuous Testing. That is, I created an ensemble of jobs
that build & test the four integration tests of upstream Git (`maint`,
`master`, `next` and `pu`) to verify the *basic* validity of their
respective current revisions.
Most CI integrations these days require custom configuration files to be
committed, and certain knobs to be twisted on GitHub (which I cannot turn
because I have no special privileges on git/git). After struggling with
making it work *somehow* anyway (even trying to get in touch with Travis,
but they have not bothered to reply to my requests in over a year...), I
decided to go with the Visual Studio Team Services (or short, VSTS; it
does come in handy that it is developed by distant colleagues of mine, so
they *have* to reply to my requests) where the CI configuration can be
separated from the source code.
The entire setup is a little bit more complex than your grandfather's CI
setup: it has to orchestrate five separate Git repositories, two of them
generated and updated from live 32/64-bit Git for Windows SDKs, using a
custom pool of build agents due to high resource demands, using three
separate Git for Windows installations to support 32/64-bit as well as
updating git.exe via `git pull`[*2*].
There is currently only one downside to that setup: the ability to have
publicly accessible build logs on VSTS is still in development.
This is not *such* a big downside: if the MacOSX/Linux CI based on
Travis[*3*] is any indicator, few people, if any, give a flying,
fornicating fly about public build logs.
However, we should strive to improve our software development practices,
and one such well-known Best Practice is to use Continuous Testing more
effectively, i.e. *not* to ignore it.
That is why I taught the Git for Windows CI job that tests the four
upstream Git integration branches to *also* bisect test breakages and then
upload comments to the identified commit on GitHub. See an example here
(the identified breakage seems to have disappeared in the meantime):
https://github.com/git/git/commit/5a12b3d76973#commitcomment-20802488
The code that generates this comment can be seen here:
https://github.com/git-for-windows/build-extra/blob/50c392c7d107/please.sh#L1648-L1665
So here is hoping to a quicker turnaround from breakage to fix in the
future!
Ciao,
Johannes
P.S.: I realize that these commit comments may *still* be ignored, but I
simply was not yet ready to annoy everybody by having automated mails sent
out.
Footnote *1*: It would be kind of okay if, say, `pu` would simply pick up
*all* patches so that they would not be forgotten. But that is not the
case. Even worse: it was stated recently that the expectation is that the
*submitters* of patches find bugs in their code, that the patches should
essentially be bug-free by the time they were submitted. This reasoning
falls flat on its face considering the very real failures, of course,
demonstrating our dear need for Continuous Testing.
Footnote *2*: I will describe the entire setup, including links to the
involved repositories, in a separate mail at a later stage.
Footnote *3*: Look at https://travis-ci.org/git/git/builds, and be happy
if you have a red/green deficiency so you cannot see all that red.
next reply other threads:[~2017-02-10 12:33 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-10 12:24 Johannes Schindelin [this message]
2017-02-13 23:46 ` Continuous Testing of Git on Windows Junio C Hamano
2017-02-14 20:55 ` [git-for-windows] " Johannes Schindelin
2017-02-14 21:08 ` Junio C Hamano
2017-02-14 23:00 ` Christian Couder
2017-02-14 23:11 ` Junio C Hamano
2017-02-14 23:27 ` Philip Oakley
2017-02-14 23:35 ` Junio C Hamano
2017-02-15 17:31 ` Philip Oakley
2017-02-15 21:26 ` Junio C Hamano
2017-02-15 23:33 ` Philip Oakley
2017-02-16 1:33 ` Junio C Hamano
2017-02-15 22:19 ` Philip Oakley
2017-02-15 22:19 ` Philip Oakley
2017-02-15 14:22 ` Johannes Schindelin
2017-02-15 23:57 ` Philip Oakley
2017-02-16 0:20 ` Junio C Hamano
2017-02-18 11:49 ` Philip Oakley
2017-02-15 14:07 ` Johannes Schindelin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.20.1702101241210.3496@virtualbox \
--to=johannes.schindelin@gmx.de \
--cc=git-for-windows@googlegroups.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).