[PATCH v1 0/3] travis-ci: run Git bisect on failed tests

[PATCH v1 0/3] travis-ci: run Git bisect on failed tests

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

This patch series run Git bisect on failing tests on Travis CI.

Example output on Travis CI:
https://travis-ci.org/larsxschneider/git/jobs/132049871

Plaintext example output:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/132049871/log.txt

Please scroll all the way down to see the bisect output.

Cheers,
Lars


Lars Schneider (3):
  travis-ci: move "after_failure" code to dedicated file in /ci
  travis-ci: disable verbose test output
  travis-ci: run Git bisect on failed tests

 .travis.yml       | 14 ++--------
 ci/test-report.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 12 deletions(-)
 create mode 100755 ci/test-report.sh

--
2.5.1

[PATCH v1 1/3] travis-ci: move "after_failure" code to dedicated file in /ci

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Move the code and adjust it to the Git shell script coding guidelines.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml       | 12 +-----------
 ci/test-report.sh | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 11 deletions(-)
 create mode 100755 ci/test-report.sh

diff --git a/.travis.yml b/.travis.yml
index adab5b8..a93ecb3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,17 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    for TEST_EXIT in t/test-results/*.exit;
-    do
-      if [ "$(cat "$TEST_EXIT")" != "0" ];
-      then
-        TEST_OUT="${TEST_EXIT%exit}out";
-        echo "------------------------------------------------------------------------";
-        echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)";
-        echo "------------------------------------------------------------------------";
-        cat "${TEST_OUT}";
-      fi;
-    done;
+    ./ci/test-report.sh
 
 notifications:
   email: false
diff --git a/ci/test-report.sh b/ci/test-report.sh
new file mode 100755
index 0000000..d08a999
--- /dev/null
+++ b/ci/test-report.sh
@@ -0,0 +1,19 @@
+#!/bin/sh
+#
+# Print test results
+#
+for TEST_EXIT in t/test-results/*.exit
+do
+	if test "$(cat "$TEST_EXIT")" != "0"
+	then
+		TEST="${TEST_EXIT%.exit}"
+		TEST_OUT="${TEST}.out"
+		echo "------------------------------------------------------------------------"
+		echo "  $(tput setaf 1)${TEST} Output$(tput sgr0)"
+		echo "------------------------------------------------------------------------"
+		cat "$TEST_OUT"
+		echo ""
+		echo ""
+	fi
+done
+
-- 
2.5.1

[PATCH v1 2/3] travis-ci: disable verbose test output

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

The verbose output clutters the Travis CI webview and is not really
useful since test debugging usually happens on a local machine.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index a93ecb3..81d2027 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -29,7 +29,7 @@ env:
     - LINUX_GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
-    - GIT_TEST_OPTS="--verbose --tee"
+    - GIT_TEST_OPTS="--tee"
     - GIT_TEST_CLONE_2GB=YesPlease
     # t9810 occasionally fails on Travis CI OS X
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
-- 
2.5.1

[PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Junio usually pushes many commits at once to the public "pu"/"next"/
"master" branches. If a test fails then it is not obvious what commit
caused the failure. Therefore we run Git bisect with the merge base
between the failing rev and its more stable branch ("next" for "pu",
"master" for "next", and "maint" for "master") as good ref to find the
offending commit. This is only enabled on "github.com/git/git" because
there we can assume that all relevant branches are up to date.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml       |  2 +-
 ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 81d2027..922807b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,7 +101,7 @@ script: make --quiet test
 after_failure:
   - >
     : '<-- Click here to see detailed test output!                                                        ';
-    ./ci/test-report.sh
+    ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH;
 
 notifications:
   email: false
diff --git a/ci/test-report.sh b/ci/test-report.sh
index d08a999..8f7adad 100755
--- a/ci/test-report.sh
+++ b/ci/test-report.sh
@@ -1,5 +1,12 @@
 #!/bin/sh
 #
+# Print test results and run Git bisect on failed tests.
+#
+REPO_ORG_NAME=$1
+CURRENT_BRANCH_NAME=$2
+
+
+#
 # Print test results
 #
 for TEST_EXIT in t/test-results/*.exit
@@ -17,3 +24,61 @@ do
 	fi
 done
 
+
+#
+# Run Git bisect
+#
+run_bisect () {
+	TEST_SCRIPT=$1
+	BAD_REV=$2
+	GOOD_RV=$3
+	TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
+	cat > "$TMPDIR/bisect-run.sh" <<EOF
+
+EOF
+	chmod +x "$TMPDIR/bisect-run.sh"
+	git bisect start $BAD_REV $GOOD_RV
+	git bisect run "$TMPDIR/bisect-run.sh"
+	if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
+	then
+		cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
+	else
+		# If the test file does not exist or the build fails then tell
+		# Git bisect to skip the commit.
+		exit 125
+	fi
+	git bisect reset >/dev/null 2>&1
+}
+
+case "$CURRENT_BRANCH_NAME" in
+	master) STABLE_BRANCH="maint";;
+	next)   STABLE_BRANCH="master";;
+	pu)     STABLE_BRANCH="next";;
+esac
+
+if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
+then
+	BAD_REV=$(git rev-parse HEAD)
+
+	# Travis CI clones are shallow. It is possible that the last good revision
+	# was not fetched, yet. Therefore we need to fetch all commits on the
+	# stable branch.
+	git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
+	git fetch --unshallow --quiet
+	LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
+
+	for TEST_EXIT in t/test-results/*.exit
+	do
+		if test "$(cat "$TEST_EXIT")" != "0"
+		then
+			TEST="${TEST_EXIT%.exit}"
+			TEST_SCRIPT=${TEST#t/test-results/}
+			echo "------------------------------------------------------------------------"
+			echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
+			echo "------------------------------------------------------------------------"
+			run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
+			echo ""
+			echo ""
+		fi
+	done
+fi
-- 
2.5.1

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[Christian Couder]

On Sun, May 22, 2016 at 1:00 PM,  <larsxschneider@gmail.com> wrote:

[...]

> +#
> +# Run Git bisect
> +#
> +run_bisect () {
> +       TEST_SCRIPT=$1
> +       BAD_REV=$2
> +       GOOD_RV=$3
> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
> +
> +EOF
> +       chmod +x "$TMPDIR/bisect-run.sh"
> +       git bisect start $BAD_REV $GOOD_RV
> +       git bisect run "$TMPDIR/bisect-run.sh"
> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
> +       then
> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
> +       else
> +               # If the test file does not exist or the build fails then tell
> +               # Git bisect to skip the commit.
> +               exit 125
> +       fi

Shouldn't all the above "if ... fi" be in the here document creating
"$TMPDIR/bisect-run.sh"?

> +       git bisect reset >/dev/null 2>&1
> +}

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[Lars Schneider]

> On 22 May 2016, at 17:35, Christian Couder <christian.couder@gmail.com> wrote:
> 
> On Sun, May 22, 2016 at 1:00 PM,  <larsxschneider@gmail.com> wrote:
> 
> [...]
> 
>> +#
>> +# Run Git bisect
>> +#
>> +run_bisect () {
>> +       TEST_SCRIPT=$1
>> +       BAD_REV=$2
>> +       GOOD_RV=$3
>> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
>> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
>> +
>> +EOF
>> +       chmod +x "$TMPDIR/bisect-run.sh"
>> +       git bisect start $BAD_REV $GOOD_RV
>> +       git bisect run "$TMPDIR/bisect-run.sh"
>> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
>> +       then
>> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
>> +       else
>> +               # If the test file does not exist or the build fails then tell
>> +               # Git bisect to skip the commit.
>> +               exit 125
>> +       fi
> 
> Shouldn't all the above "if ... fi" be in the here document creating
> "$TMPDIR/bisect-run.sh"?
Ohh. Absolutely... I wonder what happened. I'll post a v2.

Thank you,
Lars


> 
>> +       git bisect reset >/dev/null 2>&1
>> +}

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[Pranit Bauva]

Hey Lars,

On Sun, May 22, 2016 at 4:30 PM,  <larsxschneider@gmail.com> wrote:
> From: Lars Schneider <larsxschneider@gmail.com>
>
> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  .travis.yml       |  2 +-
>  ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 81d2027..922807b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -101,7 +101,7 @@ script: make --quiet test
>  after_failure:
>    - >
>      : '<-- Click here to see detailed test output!                                                        ';
> -    ./ci/test-report.sh
> +    ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH;
>
>  notifications:
>    email: false
> diff --git a/ci/test-report.sh b/ci/test-report.sh
> index d08a999..8f7adad 100755
> --- a/ci/test-report.sh
> +++ b/ci/test-report.sh
> @@ -1,5 +1,12 @@
>  #!/bin/sh
>  #
> +# Print test results and run Git bisect on failed tests.
> +#
> +REPO_ORG_NAME=$1
> +CURRENT_BRANCH_NAME=$2
> +
> +
> +#
>  # Print test results
>  #
>  for TEST_EXIT in t/test-results/*.exit
> @@ -17,3 +24,61 @@ do
>         fi
>  done
>
> +
> +#
> +# Run Git bisect
> +#
> +run_bisect () {
> +       TEST_SCRIPT=$1
> +       BAD_REV=$2
> +       GOOD_RV=$3
> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
> +       cat > "$TMPDIR/bisect-run.sh" <<EOF

If you are doing a re-roll, then you could probably fix the style issue.
' cat >"$TMPDIR..."  '

> +
> +EOF
> +       chmod +x "$TMPDIR/bisect-run.sh"
> +       git bisect start $BAD_REV $GOOD_RV
> +       git bisect run "$TMPDIR/bisect-run.sh"
> +       if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1
> +       then
> +               cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1
> +       else
> +               # If the test file does not exist or the build fails then tell
> +               # Git bisect to skip the commit.
> +               exit 125
> +       fi
> +       git bisect reset >/dev/null 2>&1
> +}
> +
> +case "$CURRENT_BRANCH_NAME" in
> +       master) STABLE_BRANCH="maint";;
> +       next)   STABLE_BRANCH="master";;
> +       pu)     STABLE_BRANCH="next";;
> +esac
> +
> +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH
> +then
> +       BAD_REV=$(git rev-parse HEAD)
> +
> +       # Travis CI clones are shallow. It is possible that the last good revision
> +       # was not fetched, yet. Therefore we need to fetch all commits on the
> +       # stable branch.
> +       git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH"
> +       git fetch --unshallow --quiet
> +       LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH")
> +
> +       for TEST_EXIT in t/test-results/*.exit
> +       do
> +               if test "$(cat "$TEST_EXIT")" != "0"
> +               then
> +                       TEST="${TEST_EXIT%.exit}"
> +                       TEST_SCRIPT=${TEST#t/test-results/}
> +                       echo "------------------------------------------------------------------------"
> +                       echo "  $(tput setaf 1)${TEST} Bisect$(tput sgr0)"
> +                       echo "------------------------------------------------------------------------"
> +                       run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV
> +                       echo ""
> +                       echo ""
> +               fi
> +       done
> +fi

Regards,
Pranit Bauva

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[Junio C Hamano]

larsxschneider@gmail.com writes:

> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.

Just FYI, 'git log next..pu' is not a very interesting range unless
you also use the "--no-merges" option.

Is there a good way to tell the CI that, instead of testing 'pu' (or
a specific branch in general), test all new commits that appear on
the first-parent chain between 'master'..'pu'?

It would be ideal if CI can do this every time 'master' and 'pu'
gets updated.

 - Find all commits that appear on the first-parent chain between
   'master..pu'.

 - For each of them:

   - Test it.  This tests the merge result and can catch breakages
     introduced by mismerging.

   - Also test its second parent if it is a merge.  This tests the
     tip of each topic branch in isolation.  CI gets a bonus point
     if it can remember that the tip of the topic did not move and
     the commit already passed the test to skip it ;-)

 - Optionally, for those topics that failed the "tip of the topic
   branch" test, "bisect master..$topic" to see where it breaks.

The thing is, "bisect" works only when you have a _single_ cause of
breakage, but because 'pu' is a pile of unsorted raw material yet to
be sifted into dirt and gem, once you see a failure at the tip of
'pu', mechanical 'bisect run' may not be a very useful tool, as you
do not know if there is only one breakage you are looking for.

Because the unit of integration we use is a topic branch (not an
individual commit on a topic branch), "this topic is broken", and
"this topic itself may be good, but merging it breaks 'pu'" are far
more interesting and relevant piece of information than "this commit
is what broke 'pu'".

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

[Lars Schneider]

> On 22 May 2016, at 19:21, Pranit Bauva <pranit.bauva@gmail.com> wrote:
> 
> Hey Lars,
> 
> On Sun, May 22, 2016 at 4:30 PM,  <larsxschneider@gmail.com> wrote:
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
...
>> +
>> +#
>> +# Run Git bisect
>> +#
>> +run_bisect () {
>> +       TEST_SCRIPT=$1
>> +       BAD_REV=$2
>> +       GOOD_RV=$3
>> +       TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null)
>> +       cat > "$TMPDIR/bisect-run.sh" <<EOF
> 
> If you are doing a re-roll, then you could probably fix the style issue.
> ' cat >"$TMPDIR..."  '

Good eye :-) I will fix this, too.

Thank you,
Lars

Re: [PATCH v1 2/3] travis-ci: disable verbose test output

[Jeff King]

On Sun, May 22, 2016 at 01:00:55PM +0200, larsxschneider@gmail.com wrote:

> From: Lars Schneider <larsxschneider@gmail.com>
> 
> The verbose output clutters the Travis CI webview and is not really
> useful since test debugging usually happens on a local machine.

I have not really been using the Travis CI results, so perhaps my
opinion does not count. But in other systems, I have found that the more
verbose the CI output, the better, simply because you will inevitably be
faced with a test that breaks on CI and not your local machine, and you
will have no way to get more details.

I don't know if Travis provides a better way to hide the output in the
non-failing cases.

-Peff