[PATCH v2] travis-ci: build and test Git on Windows

[PATCH v2] travis-ci: build and test Git on Windows

[Lars Schneider]

Most Git developers work on Linux and they have no way to know if their
changes would break the Git for Windows build. Let's fix that by adding
a job to TravisCI that builds and tests Git on Windows. Unfortunately,
TravisCI does not support Windows.

Therefore, we did the following:
* Johannes Schindelin set up a Visual Studio Team Services build
  sponsored by Microsoft and made it accessible via an Azure Function
  that speaks a super-simple API. We made TravisCI use this API to
  trigger a build, wait until its completion, and print the build and
  test results.
* A Windows build and test run takes up to 3h and TravisCI has a timeout
  after 50min for Open Source projects. Since the TravisCI job does not
  use heavy CPU/memory/etc. resources, the friendly TravisCI folks
  extended the job timeout for git/git to 3h.

Things, that would need to be done:
* Someone with write access to https://travis-ci.org/git/git would need
  to add the secret token as "GFW_CI_TOKEN" variable in the TravisCI
  repository setting [1]. Afterwards the build should just work.

Things, that might need to be done:
* The Windows box can only process a single build at a time. A second
  Windows build would need to wait until the first finishes. This
  waiting time and the build time after the wait could exceed the 3h
  threshold. If this is a problem, then it is likely to happen every day
  as usually multiple branches are pushed at the same time (pu/next/
  master/maint). I cannot test this as my TravisCI account has the 50min
  timeout. One solution could be to limit the number of concurrent
  TravisCI jobs [2].

[1] https://docs.travis-ci.com/user/environment-variables#Defining-Variables-in-Repository-Settings
[2] https://docs.travis-ci.com/user/customizing-the-build#Limiting-Concurrent-Builds

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---

Hi,

I think I addressed all issues from the v1 review (see interdiff below)
with one exception. The script still uses bash instead of sh. Something
about this does not work in sh:
    --output >(sed "$(printf '1s/^\xef\xbb\xbf//')" >cat >&3)

Does anyone know how to make this sh compatible?

Thanks,
Lars


Notes:
    Base Ref: master
    Web-Diff: https://github.com/larsxschneider/git/commit/7c527f3571
    Checkout: git fetch https://github.com/larsxschneider/git travisci/win-v2 && git checkout 7c527f3571

    Interdiff (v1..v2):

    diff --git a/.travis.yml b/.travis.yml
    index a7e98ae519..c757a111ce 100644
    --- a/.travis.yml
    +++ b/.travis.yml
    @@ -48,7 +48,7 @@ matrix:
           script:
             - >
               test "$TRAVIS_REPO_SLUG" != "git/git" ||
    -          ci/run-windows-build.sh $GFW_CI_TOKEN $TRAVIS_BRANCH $(git rev-parse HEAD)
    +          ci/run-windows-build.sh $TRAVIS_BRANCH $(git rev-parse HEAD)
           after_failure:
         - env: Linux32
           os: linux
    diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
    index 324a9ea4e6..234a235c77 100755
    --- a/ci/run-windows-build.sh
    +++ b/ci/run-windows-build.sh
    @@ -1,30 +1,49 @@
     #!/usr/bin/env bash
     #
     # Script to trigger the a Git for Windows build and test run.
    -# Pass a token, the branch (only branches on https://github.com/git/git)
    -# are supported), and a commit hash.
    +# Set the $GFW_CI_TOKEN as environment variable.
    +# Pass the branch (only branches on https://github.com/git/git are
    +# supported) and a commit hash.
     #

    -[ $# -eq 3 ] || (echo "Unexpected number of parameters" && exit 1)
    +test $# -ne 2 && echo "Unexpected number of parameters" && exit 1
    +test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit

    -TOKEN=$1
    -BRANCH=$2
    -COMMIT=$3
    +BRANCH=$1
    +COMMIT=$2

     gfwci () {
    -	curl \
    -		-H "Authentication: Bearer $TOKEN" \
    -		--silent --retry 5 \
    -		"https://git-for-windows-ci.azurewebsites.net/api/TestNow?$1" |
    -	sed "$(printf '1s/^\xef\xbb\xbf//')"  # Remove the Byte Order Mark
    +	local CURL_ERROR_CODE HTTP_CODE
    +	exec 3>&1
    +	HTTP_CODE=$(curl \
    +		-H "Authentication: Bearer $GFW_CI_TOKEN" \
    +		--silent --retry 5 --write-out '%{HTTP_CODE}' \
    +		--output >(sed "$(printf '1s/^\xef\xbb\xbf//')" >cat >&3) \
    +		"https://git-for-windows-ci.azurewebsites.net/api/TestNow?$1" \
    +	)
    +	CURL_ERROR_CODE=$?
    +	if test $CURL_ERROR_CODE -ne 0
    +	then
    +		return $CURL_ERROR_CODE
    +	fi
    +	if test "$HTTP_CODE" -ge 400 && test "$HTTP_CODE" -lt 600
    +	then
    +		return 127
    +	fi
     }

     # Trigger build job
     BUILD_ID=$(gfwci "action=trigger&branch=$BRANCH&commit=$COMMIT&skipTests=false")
    +if test $? -ne 0
    +then
    +	echo "Unable to trigger Visual Studio Team Services Build"
    +	echo "$BUILD_ID"
    +	exit 1
    +fi

     # Check if the $BUILD_ID contains a number
     case $BUILD_ID in
    -	''|*[!0-9]*) echo $BUILD_ID && exit 1
    +''|*[!0-9]*) echo "Unexpected build number: $BUILD_ID" && exit 1
     esac

     echo "Visual Studio Team Services Build #${BUILD_ID}"
    @@ -36,13 +55,13 @@ while true
     do
     	LAST_STATUS=$STATUS
     	STATUS=$(gfwci "action=status&buildId=$BUILD_ID")
    -	[ "$STATUS" == "$LAST_STATUS" ] || printf "\nStatus: $STATUS "
    +	test "$STATUS" = "$LAST_STATUS" || printf "\nStatus: $STATUS "
     	printf "."

    -	case $STATUS in
    +	case "$STATUS" in
     	inProgress|postponed|notStarted) sleep 10               ;; # continue
     	         "completed: succeeded") RESULT="success"; break;; # success
    -		                              *) echo "Unknown: $STATUS"; break;; # failure
    +	*) echo "Unhandled status: $STATUS";               break;; # failure
     	esac
     done

    @@ -52,4 +71,5 @@ echo ""
     gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-

     # Set exit code for TravisCI
    -[ "$RESULT" == "success" ]
    +test "$RESULT" = "success"
    +

    \0

 .travis.yml             | 11 ++++++++
 ci/run-windows-build.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
 create mode 100755 ci/run-windows-build.sh

diff --git a/.travis.yml b/.travis.yml
index 591cc57b80..c757a111ce 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,17 @@ env:

 matrix:
   include:
+    - env: Windows
+      os: linux
+      compiler:
+      addons:
+      before_install:
+      before_script:
+      script:
+        - >
+          test "$TRAVIS_REPO_SLUG" != "git/git" ||
+          ci/run-windows-build.sh $TRAVIS_BRANCH $(git rev-parse HEAD)
+      after_failure:
     - env: Linux32
       os: linux
       services:
diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
new file mode 100755
index 0000000000..234a235c77
--- /dev/null
+++ b/ci/run-windows-build.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+#
+# Script to trigger the a Git for Windows build and test run.
+# Set the $GFW_CI_TOKEN as environment variable.
+# Pass the branch (only branches on https://github.com/git/git are
+# supported) and a commit hash.
+#
+
+test $# -ne 2 && echo "Unexpected number of parameters" && exit 1
+test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit
+
+BRANCH=$1
+COMMIT=$2
+
+gfwci () {
+	local CURL_ERROR_CODE HTTP_CODE
+	exec 3>&1
+	HTTP_CODE=$(curl \
+		-H "Authentication: Bearer $GFW_CI_TOKEN" \
+		--silent --retry 5 --write-out '%{HTTP_CODE}' \
+		--output >(sed "$(printf '1s/^\xef\xbb\xbf//')" >cat >&3) \
+		"https://git-for-windows-ci.azurewebsites.net/api/TestNow?$1" \
+	)
+	CURL_ERROR_CODE=$?
+	if test $CURL_ERROR_CODE -ne 0
+	then
+		return $CURL_ERROR_CODE
+	fi
+	if test "$HTTP_CODE" -ge 400 && test "$HTTP_CODE" -lt 600
+	then
+		return 127
+	fi
+}
+
+# Trigger build job
+BUILD_ID=$(gfwci "action=trigger&branch=$BRANCH&commit=$COMMIT&skipTests=false")
+if test $? -ne 0
+then
+	echo "Unable to trigger Visual Studio Team Services Build"
+	echo "$BUILD_ID"
+	exit 1
+fi
+
+# Check if the $BUILD_ID contains a number
+case $BUILD_ID in
+''|*[!0-9]*) echo "Unexpected build number: $BUILD_ID" && exit 1
+esac
+
+echo "Visual Studio Team Services Build #${BUILD_ID}"
+
+# Wait until build job finished
+STATUS=
+RESULT=
+while true
+do
+	LAST_STATUS=$STATUS
+	STATUS=$(gfwci "action=status&buildId=$BUILD_ID")
+	test "$STATUS" = "$LAST_STATUS" || printf "\nStatus: $STATUS "
+	printf "."
+
+	case "$STATUS" in
+	inProgress|postponed|notStarted) sleep 10               ;; # continue
+	         "completed: succeeded") RESULT="success"; break;; # success
+	*) echo "Unhandled status: $STATUS";               break;; # failure
+	esac
+done
+
+# Print log
+echo ""
+echo ""
+gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
+
+# Set exit code for TravisCI
+test "$RESULT" = "success"
+

base-commit: afd6726309f57f532b4b989a75c1392359c611cc
--
2.12.1

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Daniel Stenberg]

On Fri, 24 Mar 2017, Lars Schneider wrote:

> Most Git developers work on Linux and they have no way to know if their 
> changes would break the Git for Windows build. Let's fix that by adding a 
> job to TravisCI that builds and tests Git on Windows. Unfortunately, 
> TravisCI does not support Windows.

Forgive me for bursting in and possibly repeating what you've already 
discussed. I just read about the limitations for doing windows builds via 
travis so I thought I'd at least let you know that you can avoid those 
limitations without too much work:

Two alternative approaches would be:

1. use appveyor.com, as that is a Travis-like service for Windows. We do our
    windows-builds in the curl project using that.

2. run your own buildbot and submit data using the regular github hook and
    have buildbot submit the results back (it has a plugin that can do that).
    We do solaris-builds in the curl project using that method (thanks to
    opencsw.org) and some additional windows-builds thanks to private
    individuals.

-- 

  / daniel.haxx.se

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Lars Schneider]

> On 24 Mar 2017, at 12:48, Daniel Stenberg <daniel@haxx.se> wrote:
> 
> On Fri, 24 Mar 2017, Lars Schneider wrote:
> 
>> Most Git developers work on Linux and they have no way to know if their changes would break the Git for Windows build. Let's fix that by adding a job to TravisCI that builds and tests Git on Windows. Unfortunately, TravisCI does not support Windows.
> 
> Forgive me for bursting in and possibly repeating what you've already discussed. I just read about the limitations for doing windows builds via travis so I thought I'd at least let you know that you can avoid those limitations without too much work:

You are not bursting in - I am happy to get your feedback! Thanks a lot for CURL BTW :-)


> Two alternative approaches would be:
> 
> 1. use appveyor.com, as that is a Travis-like service for Windows. We do our
>   windows-builds in the curl project using that.

The Git for Windows build and tests are *really* resources intensive and they
take a lot of setup time. AFAIK we would run into timeouts with AppVeyor.
Maybe Sebastian or Dscho know details?


> 2. run your own buildbot and submit data using the regular github hook and
>   have buildbot submit the results back (it has a plugin that can do that).
>   We do solaris-builds in the curl project using that method (thanks to
>   opencsw.org) and some additional windows-builds thanks to private
>   individuals.

We could do that! However, the idea was to have the entire build status 
for all platforms in one place.


- Lars

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Daniel Stenberg]

On Fri, 24 Mar 2017, Lars Schneider wrote:

>> 2. run your own buildbot and submit data using the regular github hook and
>>   have buildbot submit the results back (it has a plugin that can do that).
>>   We do solaris-builds in the curl project using that method (thanks to
>>   opencsw.org) and some additional windows-builds thanks to private
>>   individuals.
>
> We could do that! However, the idea was to have the entire build status for 
> all platforms in one place.

As the status is (or at least can be) passed back to github, all CI jobs are 
shown at the same place: below each commit and each pull-request...

-- 

  / daniel.haxx.se

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Sebastian Schuberth]

On Fri, Mar 24, 2017 at 1:35 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:

>> 1. use appveyor.com, as that is a Travis-like service for Windows. We do our
>>   windows-builds in the curl project using that.
>
> The Git for Windows build and tests are *really* resources intensive and they
> take a lot of setup time. AFAIK we would run into timeouts with AppVeyor.
> Maybe Sebastian or Dscho know details?

At lot of setup time, in terms of installing the Git for Windows SDK,
could probably be saved by only installing the required packages on
top of MSYS2 that's already installed in AppVeyor nodes [1].

[1] https://www.appveyor.com/docs/build-environment/#mingw-msys-cygwin

-- 
Sebastian Schuberth

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Jeff King]

On Fri, Mar 24, 2017 at 12:37:47PM +0100, Lars Schneider wrote:

> I think I addressed all issues from the v1 review (see interdiff below)
> with one exception. The script still uses bash instead of sh. Something
> about this does not work in sh:
>     --output >(sed "$(printf '1s/^\xef\xbb\xbf//')" >cat >&3)
> 
> Does anyone know how to make this sh compatible?

Process substitution is a bash-ism. Just looking at this snippet I would
ask why you aren't just using stdout, but from the whole diff it looks
like you're using --write-out and want to keep the streams separate.

The POSIX shell way would be to make your own named pipe rather than
anonymous one:

  mkfifo data
  sed ... <data &
  curl ... --output data
  rm data

But I don't think relying on bash is that big a deal, as long as the
script is clearly marked with the correct #!-line (which it is).

The ">cat" isn't doing AFAICT, though.

-Peff

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> I think I addressed all issues from the v1 review (see interdiff below)
> with one exception. The script still uses bash instead of sh. Something
> about this does not work in sh:
>     --output >(sed "$(printf '1s/^\xef\xbb\xbf//')" >cat >&3)
>
> Does anyone know how to make this sh compatible?

Making this runnable under any shell is not a goal; if your script
does rely on that bash-ism, then leave it as-is and mark it with
#!/bin/bash or somesuch.

Thanks.

Re: [PATCH v2] travis-ci: build and test Git on Windows

[Johannes Schindelin]

Hi,

On Fri, 24 Mar 2017, Sebastian Schuberth wrote:

> On Fri, Mar 24, 2017 at 1:35 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
> 
> >> 1. use appveyor.com, as that is a Travis-like service for Windows. We do our
> >>   windows-builds in the curl project using that.
> >
> > The Git for Windows build and tests are *really* resources intensive and they
> > take a lot of setup time. AFAIK we would run into timeouts with AppVeyor.
> > Maybe Sebastian or Dscho know details?
> 
> At lot of setup time, in terms of installing the Git for Windows SDK,
> could probably be saved by only installing the required packages on
> top of MSYS2 that's already installed in AppVeyor nodes [1].
> 
> [1] https://www.appveyor.com/docs/build-environment/#mingw-msys-cygwin

And that is indeed what Git for Windows' Git fork has configured. Sadly,
the job timed out too often for me, even if I disabled the tests.

As of recent, things turned more green, with the build taking only about 3
minutes. Of course, re-enabling the tests would let us run smack into the
20 minutes time out, *and* we still would have to disable the tests that
fail because MSYS2's maintainer ignored the PRs trying to integrate Git
for Windows' patches to the MSYS2 runtime.

Example for a green job:
https://ci.appveyor.com/project/dscho/git/build/1.0.676

Ciao,
Johannes