[PATCH 0/2] Travis CI: skip commits with successfully built and tested trees

[PATCH 0/2] Travis CI: skip commits with successfully built and tested trees

[SZEDER Gábor]

Travis CI dutifully builds and tests each new branch tip, even if its
tree has previously been successfully built and tested.  This happens
often enough in contributors' workflows, when a work-in-progress
branch is rebased changing e.g. only commit messages or the order or
number of commits while leaving the resulting code intact, and is then
pushed to a Travis CI-enabled GitHub fork.

This is wasting Travis CI's resources and is sometimes scary-annoying
when the new tip commit with a tree identical to the previous,
successfully tested one is suddenly reported in red, because one of
the OSX build jobs happened to exceed the time limit yet again.

These two patches extend our Travis CI build scripts to skip building
commits whose trees have previously been successfully built and tested.


These patches should go on top of the "Rest of the Travis CI fixes" patch
series, just posted at:

  https://public-inbox.org/git/20171227163603.13313-1-szeder.dev@gmail.com/


SZEDER Gábor (2):
  travis-ci: don't try to create the cache directory unnecessarily
  travis-ci: record and skip successfully built trees

 ci/lib-travisci.sh        | 43 +++++++++++++++++++++++++++++++++++++++++++
 ci/run-linux32-docker.sh  |  2 ++
 ci/run-static-analysis.sh |  2 ++
 ci/run-tests.sh           |  3 ++-
 ci/run-windows-build.sh   |  2 ++
 ci/test-documentation.sh  |  2 ++
 6 files changed, 53 insertions(+), 1 deletion(-)

-- 
2.15.1.500.g54ea76cc4

[PATCH 1/2] travis-ci: don't try to create the cache directory unnecessarily

[SZEDER Gábor]

Travis CI creates that directory for us anyway, even when a previous
cache doesn't exist for the current build job.

In itself it doesn't hurt to try, of course, but the following patch
will access the Travis CI cache much earlier in the build process, and
then creating the cache directory this late might cause confusion.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/run-tests.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/ci/run-tests.sh b/ci/run-tests.sh
index f0c743de9..ccdfc2b9d 100755
--- a/ci/run-tests.sh
+++ b/ci/run-tests.sh
@@ -5,6 +5,5 @@
 
 . ${0%/*}/lib-travisci.sh
 
-mkdir -p $HOME/travis-cache
 ln -s $HOME/travis-cache/.prove t/.prove
 make --quiet test
-- 
2.15.1.500.g54ea76cc4

Re: [PATCH 1/2] travis-ci: don't try to create the cache directory unnecessarily

[Jonathan Nieder]

SZEDER Gábor wrote:

> Travis CI creates that directory for us anyway, even when a previous
> cache doesn't exist for the current build job.
>
> In itself it doesn't hurt to try, of course, but the following patch
> will access the Travis CI cache much earlier in the build process, and
> then creating the cache directory this late might cause confusion.
>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
>  ci/run-tests.sh | 1 -
>  1 file changed, 1 deletion(-)

Is this behavior documented anywhere?
https://docs.travis-ci.com/user/caching#Arbitrary-directories doesn't
say anything about it.

Thanks,
Jonathan

Re: [PATCH 1/2] travis-ci: don't try to create the cache directory unnecessarily

[SZEDER Gábor]

On Wed, Dec 27, 2017 at 8:46 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> SZEDER Gábor wrote:
>
>> Travis CI creates that directory for us anyway, even when a previous
>> cache doesn't exist for the current build job.
>>
>> In itself it doesn't hurt to try, of course, but the following patch
>> will access the Travis CI cache much earlier in the build process, and
>> then creating the cache directory this late might cause confusion.
>>
>> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
>> ---
>>  ci/run-tests.sh | 1 -
>>  1 file changed, 1 deletion(-)
>
> Is this behavior documented anywhere?
> https://docs.travis-ci.com/user/caching#Arbitrary-directories doesn't
> say anything about it.

No, I'm afraid it isn't explicitly mentioned.
I seem to remember an example implicitly relying on it, though, but
can't find it anymore, so either misremembered or misunderstood one of
the examples.
OK, then I'll move this 'mkdir' to 'ci/lib-travisci.sh', to ensure that
the cache directory exists in all build jobs.

Gábor

Re: [PATCH 1/2] travis-ci: don't try to create the cache directory unnecessarily

[Jonathan Nieder]

SZEDER Gábor wrote:
> On Wed, Dec 27, 2017 at 8:46 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > SZEDER Gábor wrote:

>>> Travis CI creates that directory for us anyway, even when a previous
>>> cache doesn't exist for the current build job.
>>>
>>> In itself it doesn't hurt to try, of course, but the following patch
>>> will access the Travis CI cache much earlier in the build process, and
>>> then creating the cache directory this late might cause confusion.
[...]
>> Is this behavior documented anywhere?
>> https://docs.travis-ci.com/user/caching#Arbitrary-directories doesn't
>> say anything about it.
>
> No, I'm afraid it isn't explicitly mentioned.
> I seem to remember an example implicitly relying on it, though, but
> can't find it anymore, so either misremembered or misunderstood one of
> the examples.
> OK, then I'll move this 'mkdir' to 'ci/lib-travisci.sh', to ensure that
> the cache directory exists in all build jobs.

Thanks.  Sounds good to me.

Another alternative would be to contact Travis CI folks to get the
behavior documented more clearly.  But here an early 'mkdir' doesn't
hurt, so it seems simplest to do it.

Sincerely,
Jonathan

[PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

Travis CI dutifully builds and tests each new branch tip, even if its
tree has previously been successfully built and tested.  This happens
often enough in contributors' workflows, when a work-in-progress
branch is rebased changing e.g. only commit messages or the order or
number of commits while leaving the resulting code intact, and is then
pushed to a Travis CI-enabled GitHub fork.

This is wasting Travis CI's resources and is sometimes scary-annoying
when the new tip commit with a tree identical to the previous,
successfully tested one is suddenly reported in red, because one of
the OSX build jobs happened to exceed the time limit yet again.

So extend our Travis CI build scripts to skip building commits whose
trees have previously been successfully built and tested.  Use the
Travis CI cache feature to keep a record of the object names of trees
that tested successfully, in a plain and simple flat text file, one
line per tree object name.  Append the current tree's object name at
the end of every successful build job to this file, along with a bit
of additional info about the build job (commit object name, Travis CI
job number and id).  Check, using a simple grep invocation, in each
build job whether the current commit's tree is already in there, and
skip the build if it is.  Include a message in the skipped build job's
trace log, containing the URL to the build job successfully testing
that tree for the first time and instructions on how to force a
re-build.  Catch the case when a build job, which successfully built
and tested a particular tree for the first time, is restarted and omit
the URL of the previous build job's trace log, as in this case it's
the same build job and the trace log has just been overwritten.

Using an ever-growing flat text file might seem like asking for
trouble on the long run, but it's perfectly adequate for this purpose.
Contributors' topic branches are short-lived in general, so this file
won't grow large enough to cause any issues.  Grepping through several
tens of thousands such lines is sufficiently fast, so not even
git/git's forever living integration branches will cause scalability
issues with the current rate of ~1 push/day for a couple of decades.
And even if we reach the point that this file grows too big, the
caches can be deleted on Travis CI's web interface.

Note: this won't kick in if two identical trees are on two different
branches, because Travis CI caches are not shared between build jobs
of different branches.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/lib-travisci.sh        | 43 +++++++++++++++++++++++++++++++++++++++++++
 ci/run-linux32-docker.sh  |  2 ++
 ci/run-static-analysis.sh |  2 ++
 ci/run-tests.sh           |  2 ++
 ci/run-windows-build.sh   |  2 ++
 ci/test-documentation.sh  |  2 ++
 6 files changed, 53 insertions(+)

diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 348fe3c3c..05e73123f 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -21,6 +21,48 @@ skip_branch_tip_with_tag () {
 	fi
 }
 
+good_trees_file="$HOME/travis-cache/good-trees"
+
+# Save some info about the current commit's tree, so we can skip the build
+# job if we encounter the same tree again and can provide a useful info
+# message.
+save_good_tree () {
+	echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
+}
+
+# Skip the build job if the same tree has already been built and tested
+# successfully before (e.g. because the branch got rebased, changing only
+# the commit messages).
+skip_good_tree () {
+	if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
+	then
+		# haven't seen this tree yet; continue the build job
+		return
+	fi
+
+	echo "$good_tree_info" | {
+		read tree prev_good_commit prev_good_job_number prev_good_job_id
+
+		if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
+		then
+			cat <<-EOF
+			Skipping build job for commit $TRAVIS_COMMIT.
+			This commit has already been built and tested successfully by this build job.
+			To force a re-build delete the branch's cache and then hit 'Restart job'.
+			EOF
+		else
+			cat <<-EOF
+			Skipping build job for commit $TRAVIS_COMMIT.
+			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
+			The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
+			To force a re-build delete the branch's cache and then hit 'Restart job'.
+			EOF
+		fi
+	}
+
+	exit 0
+}
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -28,6 +70,7 @@ skip_branch_tip_with_tag () {
 set -ex
 
 skip_branch_tip_with_tag
+skip_good_tree
 
 if test -z "$jobname"
 then
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index 3a8b2ba42..870a41246 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -22,3 +22,5 @@ docker run \
 	--volume "${HOME}/travis-cache:/tmp/travis-cache" \
 	daald/ubuntu32:xenial \
 	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+
+save_good_tree
diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
index 68dd0f080..fe4ee4e06 100755
--- a/ci/run-static-analysis.sh
+++ b/ci/run-static-analysis.sh
@@ -6,3 +6,5 @@
 . ${0%/*}/lib-travisci.sh
 
 make coccicheck
+
+save_good_tree
diff --git a/ci/run-tests.sh b/ci/run-tests.sh
index ccdfc2b9d..eb5ba4058 100755
--- a/ci/run-tests.sh
+++ b/ci/run-tests.sh
@@ -7,3 +7,5 @@
 
 ln -s $HOME/travis-cache/.prove t/.prove
 make --quiet test
+
+save_good_tree
diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
index 86999268a..d99a180e5 100755
--- a/ci/run-windows-build.sh
+++ b/ci/run-windows-build.sh
@@ -99,3 +99,5 @@ gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
 
 # Set exit code for TravisCI
 test "$RESULT" = "success"
+
+save_good_tree
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
index 7a0a848e8..3d62e6c95 100755
--- a/ci/test-documentation.sh
+++ b/ci/test-documentation.sh
@@ -25,3 +25,5 @@ sed '/^GIT_VERSION = / d' stderr.log
 ! test -s stderr.log
 test -s Documentation/git.html
 grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
+
+save_good_tree
-- 
2.15.1.500.g54ea76cc4

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Lars Schneider]

> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> Travis CI dutifully builds and tests each new branch tip, even if its
> tree has previously been successfully built and tested.  This happens
> often enough in contributors' workflows, when a work-in-progress
> branch is rebased changing e.g. only commit messages or the order or
> number of commits while leaving the resulting code intact, and is then
> pushed to a Travis CI-enabled GitHub fork.
> 
> This is wasting Travis CI's resources and is sometimes scary-annoying
> when the new tip commit with a tree identical to the previous,
> successfully tested one is suddenly reported in red, because one of
> the OSX build jobs happened to exceed the time limit yet again.
> 
> So extend our Travis CI build scripts to skip building commits whose
> trees have previously been successfully built and tested.  Use the
> Travis CI cache feature to keep a record of the object names of trees
> that tested successfully, in a plain and simple flat text file, one
> line per tree object name.  Append the current tree's object name at
> the end of every successful build job to this file, along with a bit
> of additional info about the build job (commit object name, Travis CI
> job number and id).  Check, using a simple grep invocation, in each
> build job whether the current commit's tree is already in there, and
> skip the build if it is.  Include a message in the skipped build job's
> trace log, containing the URL to the build job successfully testing
> that tree for the first time and instructions on how to force a
> re-build.  Catch the case when a build job, which successfully built
> and tested a particular tree for the first time, is restarted and omit
> the URL of the previous build job's trace log, as in this case it's
> the same build job and the trace log has just been overwritten.
> 
> Using an ever-growing flat text file might seem like asking for
> trouble on the long run, but it's perfectly adequate for this purpose.
> Contributors' topic branches are short-lived in general, so this file
> won't grow large enough to cause any issues.  Grepping through several
> tens of thousands such lines is sufficiently fast, so not even
> git/git's forever living integration branches will cause scalability
> issues with the current rate of ~1 push/day for a couple of decades.
> And even if we reach the point that this file grows too big, the
> caches can be deleted on Travis CI's web interface.
> 
> Note: this won't kick in if two identical trees are on two different
> branches, because Travis CI caches are not shared between build jobs
> of different branches.

This is a nice idea. If there are no other objections to this approach 
then I will forward it to the TravisCI folks. Maybe they are willing to 
implement this natively to safe some more cycles beyond */git ;-)


> 
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
> ci/lib-travisci.sh        | 43 +++++++++++++++++++++++++++++++++++++++++++
> ci/run-linux32-docker.sh  |  2 ++
> ci/run-static-analysis.sh |  2 ++
> ci/run-tests.sh           |  2 ++
> ci/run-windows-build.sh   |  2 ++
> ci/test-documentation.sh  |  2 ++
> 6 files changed, 53 insertions(+)
> 
> diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
> index 348fe3c3c..05e73123f 100755
> --- a/ci/lib-travisci.sh
> +++ b/ci/lib-travisci.sh
> @@ -21,6 +21,48 @@ skip_branch_tip_with_tag () {
> 	fi
> }
> 
> +good_trees_file="$HOME/travis-cache/good-trees"
> +
> +# Save some info about the current commit's tree, so we can skip the build
> +# job if we encounter the same tree again and can provide a useful info
> +# message.
> +save_good_tree () {
> +	echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
> +}
> +
> +# Skip the build job if the same tree has already been built and tested
> +# successfully before (e.g. because the branch got rebased, changing only
> +# the commit messages).
> +skip_good_tree () {
> +	if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
> +	then
> +		# haven't seen this tree yet; continue the build job
> +		return
> +	fi
> +
> +	echo "$good_tree_info" | {
> +		read tree prev_good_commit prev_good_job_number prev_good_job_id
> +
> +		if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"

Under what circumstances would that be true?

Nit: One unintended space after = ?!

> +		then
> +			cat <<-EOF
> +			Skipping build job for commit $TRAVIS_COMMIT.
> +			This commit has already been built and tested successfully by this build job.
> +			To force a re-build delete the branch's cache and then hit 'Restart job'.
> +			EOF
> +		else
> +			cat <<-EOF
> +			Skipping build job for commit $TRAVIS_COMMIT.
> +			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
> +			The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
> +			To force a re-build delete the branch's cache and then hit 'Restart job'.
> +			EOF

Maybe add a few newlines before and after EOF to make the text more stand out?
Or print it in a different color? Maybe red?

See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625

Thanks,
Lars


> +		fi
> +	}
> +
> +	exit 0
> +}
> +
> # Set 'exit on error' for all CI scripts to let the caller know that
> # something went wrong.
> # Set tracing executed commands, primarily setting environment variables
> @@ -28,6 +70,7 @@ skip_branch_tip_with_tag () {
> set -ex
> 
> skip_branch_tip_with_tag
> +skip_good_tree
> 
> if test -z "$jobname"
> then
> diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
> index 3a8b2ba42..870a41246 100755
> --- a/ci/run-linux32-docker.sh
> +++ b/ci/run-linux32-docker.sh
> @@ -22,3 +22,5 @@ docker run \
> 	--volume "${HOME}/travis-cache:/tmp/travis-cache" \
> 	daald/ubuntu32:xenial \
> 	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
> +
> +save_good_tree
> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
> index 68dd0f080..fe4ee4e06 100755
> --- a/ci/run-static-analysis.sh
> +++ b/ci/run-static-analysis.sh
> @@ -6,3 +6,5 @@
> . ${0%/*}/lib-travisci.sh
> 
> make coccicheck
> +
> +save_good_tree
> diff --git a/ci/run-tests.sh b/ci/run-tests.sh
> index ccdfc2b9d..eb5ba4058 100755
> --- a/ci/run-tests.sh
> +++ b/ci/run-tests.sh
> @@ -7,3 +7,5 @@
> 
> ln -s $HOME/travis-cache/.prove t/.prove
> make --quiet test
> +
> +save_good_tree
> diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
> index 86999268a..d99a180e5 100755
> --- a/ci/run-windows-build.sh
> +++ b/ci/run-windows-build.sh
> @@ -99,3 +99,5 @@ gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
> 
> # Set exit code for TravisCI
> test "$RESULT" = "success"
> +
> +save_good_tree
> diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
> index 7a0a848e8..3d62e6c95 100755
> --- a/ci/test-documentation.sh
> +++ b/ci/test-documentation.sh
> @@ -25,3 +25,5 @@ sed '/^GIT_VERSION = / d' stderr.log
> ! test -s stderr.log
> test -s Documentation/git.html
> grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
> +
> +save_good_tree
> -- 
> 2.15.1.500.g54ea76cc4
> 

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

On Wed, Dec 27, 2017 at 8:15 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
>> +# Skip the build job if the same tree has already been built and tested
>> +# successfully before (e.g. because the branch got rebased, changing only
>> +# the commit messages).
>> +skip_good_tree () {
>> +     if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
>> +     then
>> +             # haven't seen this tree yet; continue the build job
>> +             return
>> +     fi
>> +
>> +     echo "$good_tree_info" | {
>> +             read tree prev_good_commit prev_good_job_number prev_good_job_id
>> +
>> +             if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
>
> Under what circumstances would that be true?

When the user hits 'Restart job' on the Travis CI web interface,
$TRAVI_JOB_NUMBER and _ID remain the same in the restarted build job as
they were in the original.
So the condition is true when the user hits 'Restart job' on a build job
that was the first to successfully build and test the current tree.

> Nit: One unintended space after = ?!

Ok.

>> +             then
>> +                     cat <<-EOF
>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>> +                     This commit has already been built and tested successfully by this build job.
>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>> +                     EOF
>> +             else
>> +                     cat <<-EOF
>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>> +                     This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
>> +                     The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>> +                     EOF
>
> Maybe add a few newlines before and after EOF to make the text more stand out?
> Or print it in a different color? Maybe red?
>
> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625

I considered using color for the first line, but then didn't do it,
because I didn't want to decide the color :)
Anyway, red is the general error/failure color, but this is neither.  It
could either be green, to match the color of the build job's result, or
something neutral like blue or yellow.

Gábor

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

On Thu, Dec 28, 2017 at 12:00 AM, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> On Wed, Dec 27, 2017 at 8:15 PM, Lars Schneider <larsxschneider@gmail.com> wrote:
>>> +             then
>>> +                     cat <<-EOF
>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>> +                     This commit has already been built and tested successfully by this build job.
>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>> +                     EOF
>>> +             else
>>> +                     cat <<-EOF
>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>> +                     This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
>>> +                     The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>> +                     EOF
>>
>> Maybe add a few newlines before and after EOF to make the text more stand out?
>> Or print it in a different color? Maybe red?
>>
>> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625
>
> I considered using color for the first line, but then didn't do it,
> because I didn't want to decide the color :)
> Anyway, red is the general error/failure color, but this is neither.  It
> could either be green, to match the color of the build job's result, or
> something neutral like blue or yellow.

OTOH, the message printed in skip_branch_tip_with_tag() is not
colorized, either.

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Lars Schneider]

> On 28 Dec 2017, at 00:00, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> On Wed, Dec 27, 2017 at 8:15 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> 
>>> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
>>> +# Skip the build job if the same tree has already been built and tested
>>> +# successfully before (e.g. because the branch got rebased, changing only
>>> +# the commit messages).
>>> +skip_good_tree () {
>>> +     if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
>>> +     then
>>> +             # haven't seen this tree yet; continue the build job
>>> +             return
>>> +     fi
>>> +
>>> +     echo "$good_tree_info" | {
>>> +             read tree prev_good_commit prev_good_job_number prev_good_job_id
>>> +
>>> +             if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
>> 
>> Under what circumstances would that be true?
> 
> When the user hits 'Restart job' on the Travis CI web interface,
> $TRAVI_JOB_NUMBER and _ID remain the same in the restarted build job as
> they were in the original.
> So the condition is true when the user hits 'Restart job' on a build job
> that was the first to successfully build and test the current tree.

I think I would prefer it if Travis would rerun all jobs if I hit the
"refresh" button. What is your intention here?


> 
>>> +             then
>>> +                     cat <<-EOF
>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>> +                     This commit has already been built and tested successfully by this build job.
>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>> +                     EOF
>>> +             else
>>> +                     cat <<-EOF
>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>> +                     This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
>>> +                     The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>> +                     EOF
>> 
>> Maybe add a few newlines before and after EOF to make the text more stand out?
>> Or print it in a different color? Maybe red?
>> 
>> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625
> 
> I considered using color for the first line, but then didn't do it,
> because I didn't want to decide the color :)
> Anyway, red is the general error/failure color, but this is neither.  It
> could either be green, to match the color of the build job's result, or
> something neutral like blue or yellow.

You are right about red. I think I like yellow to express "warning".
But this is just a nit.

"skip_branch_tip_with_tag" could print its output yellow, too.

- Lars

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

On Thu, Dec 28, 2017 at 12:16 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 28 Dec 2017, at 00:00, SZEDER Gábor <szeder.dev@gmail.com> wrote:
>>
>> On Wed, Dec 27, 2017 at 8:15 PM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>>
>>>> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
>>>> +# Skip the build job if the same tree has already been built and tested
>>>> +# successfully before (e.g. because the branch got rebased, changing only
>>>> +# the commit messages).
>>>> +skip_good_tree () {
>>>> +     if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
>>>> +     then
>>>> +             # haven't seen this tree yet; continue the build job
>>>> +             return
>>>> +     fi
>>>> +
>>>> +     echo "$good_tree_info" | {
>>>> +             read tree prev_good_commit prev_good_job_number prev_good_job_id
>>>> +
>>>> +             if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
>>>
>>> Under what circumstances would that be true?
>>
>> When the user hits 'Restart job' on the Travis CI web interface,
>> $TRAVI_JOB_NUMBER and _ID remain the same in the restarted build job as
>> they were in the original.
>> So the condition is true when the user hits 'Restart job' on a build job
>> that was the first to successfully build and test the current tree.
>
> I think I would prefer it if Travis would rerun all jobs if I hit the
> "refresh" button. What is your intention here?

I considered that and don't think it's worth the effort.

First, I think it's a rather rare use case.  I don't know what others
are doing, but I only hit 'Restart job' to restart timeouted OSX build
jobs (and to test this patch :), and that already works with this patch.
I don't really see any reason to restart old successful build jobs,
except perhaps when a new version of one of the dependencies becomes
available (e.g. P4 and Git LFS versions are not hardcoded on OSX, we use
whatever homebrew delivers), to see that an older version still works
with the new dependencies.  Has anyone ever done something like that? :)

Second, we need to know when a build job is run after the user hit
'Restart job'.  Unless I overlooked something, Travis CI doesn't
indicate this.  I'm not sure this is documented explicitly, but it seems
that a restarted build job gets the same $TRAVIS_JOB_{NUMBER,ID}
variables as the original.  We could use this to identify restarted
build jobs, but to do that we would have to save this information at the
end of every successful build, too, and add additional checks to this
function, of course.

>>>> +             then
>>>> +                     cat <<-EOF
>>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>>> +                     This commit has already been built and tested successfully by this build job.
>>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>>> +                     EOF
>>>> +             else
>>>> +                     cat <<-EOF
>>>> +                     Skipping build job for commit $TRAVIS_COMMIT.
>>>> +                     This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
>>>> +                     The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
>>>> +                     To force a re-build delete the branch's cache and then hit 'Restart job'.
>>>> +                     EOF
>>>
>>> Maybe add a few newlines before and after EOF to make the text more stand out?
>>> Or print it in a different color? Maybe red?
>>>
>>> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625
>>
>> I considered using color for the first line, but then didn't do it,
>> because I didn't want to decide the color :)
>> Anyway, red is the general error/failure color, but this is neither.  It
>> could either be green, to match the color of the build job's result, or
>> something neutral like blue or yellow.
>
> You are right about red. I think I like yellow to express "warning".
> But this is just a nit.

OK, yellow it will be then.


> "skip_branch_tip_with_tag" could print its output yellow, too.
>
> - Lars

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

On Fri, Dec 29, 2017 at 9:03 PM, SZEDER Gábor <szeder.dev@gmail.com> wrote:

>>>> Or print it in a different color? Maybe red?
>>>>
>>>> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625
>>>
>>> I considered using color for the first line, but then didn't do it,
>>> because I didn't want to decide the color :)
>>> Anyway, red is the general error/failure color, but this is neither.  It
>>> could either be green, to match the color of the build job's result, or
>>> something neutral like blue or yellow.
>>
>> You are right about red. I think I like yellow to express "warning".
>> But this is just a nit.
>
> OK, yellow it will be then.

Oh, hang on!  Have a look:

  https://travis-ci.org/szeder/git/jobs/322841285#L629

That yellow is barely different from default text color.
Bold stands out much better, but notice how Travis CI uses bold for all
its messages.  Therefore I think we should not use bold and leave it
exclusive to Travis CI's messages.

Green looks good:

  https://travis-ci.org/szeder/git/jobs/322372326#L629


Gábor

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Lars Schneider]

> On 29 Dec 2017, at 21:16, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> On Fri, Dec 29, 2017 at 9:03 PM, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
>>>>> Or print it in a different color? Maybe red?
>>>>> 
>>>>> See: https://travis-ci.org/szeder/git/jobs/322247836#L622-L625
>>>> 
>>>> I considered using color for the first line, but then didn't do it,
>>>> because I didn't want to decide the color :)
>>>> Anyway, red is the general error/failure color, but this is neither.  It
>>>> could either be green, to match the color of the build job's result, or
>>>> something neutral like blue or yellow.
>>> 
>>> You are right about red. I think I like yellow to express "warning".
>>> But this is just a nit.
>> 
>> OK, yellow it will be then.
> 
> Oh, hang on!  Have a look:
> 
>  https://travis-ci.org/szeder/git/jobs/322841285#L629
> 
> That yellow is barely different from default text color.
> Bold stands out much better, but notice how Travis CI uses bold for all
> its messages.  Therefore I think we should not use bold and leave it
> exclusive to Travis CI's messages.
> 
> Green looks good:
> 
>  https://travis-ci.org/szeder/git/jobs/322372326#L629

Agreed. Green looks good! :-)

- Lars

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Lars Schneider]

> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> Travis CI dutifully builds and tests each new branch tip, even if its
> tree has previously been successfully built and tested.  This happens
> often enough in contributors' workflows, when a work-in-progress
> branch is rebased changing e.g. only commit messages or the order or
> number of commits while leaving the resulting code intact, and is then
> pushed to a Travis CI-enabled GitHub fork.
> 
> This is wasting Travis CI's resources and is sometimes scary-annoying
> when the new tip commit with a tree identical to the previous,
> successfully tested one is suddenly reported in red, because one of
> the OSX build jobs happened to exceed the time limit yet again.
> 
> So extend our Travis CI build scripts to skip building commits whose
> trees have previously been successfully built and tested.  Use the
> Travis CI cache feature to keep a record of the object names of trees
> that tested successfully, in a plain and simple flat text file, one
> line per tree object name.  Append the current tree's object name at
> the end of every successful build job to this file, along with a bit
> of additional info about the build job (commit object name, Travis CI
> job number and id).  Check, using a simple grep invocation, in each
> build job whether the current commit's tree is already in there, and
> skip the build if it is.  Include a message in the skipped build job's
> trace log, containing the URL to the build job successfully testing
> that tree for the first time and instructions on how to force a
> re-build.  Catch the case when a build job, which successfully built
> and tested a particular tree for the first time, is restarted and omit
> the URL of the previous build job's trace log, as in this case it's
> the same build job and the trace log has just been overwritten.
> 
> Using an ever-growing flat text file might seem like asking for
> trouble on the long run, but it's perfectly adequate for this purpose.
> Contributors' topic branches are short-lived in general, so this file
> won't grow large enough to cause any issues.  Grepping through several
> tens of thousands such lines is sufficiently fast, so not even
> git/git's forever living integration branches will cause scalability
> issues with the current rate of ~1 push/day for a couple of decades.
> And even if we reach the point that this file grows too big, the
> caches can be deleted on Travis CI's web interface.

One more thing:
Maybe we could delete "$HOME/travis-cache/good-trees" if the file
has more than 1000 lines *before* we add a new tree?

Or we use something like this to cap the file:

  echo "$(tail -1000 $HOME/travis-cache/good-trees)" > $HOME/travis-cache/good-trees


I agree that the "always growing problem" is not a big one
but an approach like the one above would avoid any discussion
for sure.


Thanks,
Lars

> 
> Note: this won't kick in if two identical trees are on two different
> branches, because Travis CI caches are not shared between build jobs
> of different branches.
> 
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
> ci/lib-travisci.sh        | 43 +++++++++++++++++++++++++++++++++++++++++++
> ci/run-linux32-docker.sh  |  2 ++
> ci/run-static-analysis.sh |  2 ++
> ci/run-tests.sh           |  2 ++
> ci/run-windows-build.sh   |  2 ++
> ci/test-documentation.sh  |  2 ++
> 6 files changed, 53 insertions(+)
> 
> diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
> index 348fe3c3c..05e73123f 100755
> --- a/ci/lib-travisci.sh
> +++ b/ci/lib-travisci.sh
> @@ -21,6 +21,48 @@ skip_branch_tip_with_tag () {
> 	fi
> }
> 
> +good_trees_file="$HOME/travis-cache/good-trees"
> +
> +# Save some info about the current commit's tree, so we can skip the build
> +# job if we encounter the same tree again and can provide a useful info
> +# message.
> +save_good_tree () {
> +	echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
> +}
> +
> +# Skip the build job if the same tree has already been built and tested
> +# successfully before (e.g. because the branch got rebased, changing only
> +# the commit messages).
> +skip_good_tree () {
> +	if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
> +	then
> +		# haven't seen this tree yet; continue the build job
> +		return
> +	fi
> +
> +	echo "$good_tree_info" | {
> +		read tree prev_good_commit prev_good_job_number prev_good_job_id
> +
> +		if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
> +		then
> +			cat <<-EOF
> +			Skipping build job for commit $TRAVIS_COMMIT.
> +			This commit has already been built and tested successfully by this build job.
> +			To force a re-build delete the branch's cache and then hit 'Restart job'.
> +			EOF
> +		else
> +			cat <<-EOF
> +			Skipping build job for commit $TRAVIS_COMMIT.
> +			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
> +			The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
> +			To force a re-build delete the branch's cache and then hit 'Restart job'.
> +			EOF
> +		fi
> +	}
> +
> +	exit 0
> +}
> +
> # Set 'exit on error' for all CI scripts to let the caller know that
> # something went wrong.
> # Set tracing executed commands, primarily setting environment variables
> @@ -28,6 +70,7 @@ skip_branch_tip_with_tag () {
> set -ex
> 
> skip_branch_tip_with_tag
> +skip_good_tree
> 
> if test -z "$jobname"
> then
> diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
> index 3a8b2ba42..870a41246 100755
> --- a/ci/run-linux32-docker.sh
> +++ b/ci/run-linux32-docker.sh
> @@ -22,3 +22,5 @@ docker run \
> 	--volume "${HOME}/travis-cache:/tmp/travis-cache" \
> 	daald/ubuntu32:xenial \
> 	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
> +
> +save_good_tree
> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
> index 68dd0f080..fe4ee4e06 100755
> --- a/ci/run-static-analysis.sh
> +++ b/ci/run-static-analysis.sh
> @@ -6,3 +6,5 @@
> . ${0%/*}/lib-travisci.sh
> 
> make coccicheck
> +
> +save_good_tree
> diff --git a/ci/run-tests.sh b/ci/run-tests.sh
> index ccdfc2b9d..eb5ba4058 100755
> --- a/ci/run-tests.sh
> +++ b/ci/run-tests.sh
> @@ -7,3 +7,5 @@
> 
> ln -s $HOME/travis-cache/.prove t/.prove
> make --quiet test
> +
> +save_good_tree
> diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
> index 86999268a..d99a180e5 100755
> --- a/ci/run-windows-build.sh
> +++ b/ci/run-windows-build.sh
> @@ -99,3 +99,5 @@ gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
> 
> # Set exit code for TravisCI
> test "$RESULT" = "success"
> +
> +save_good_tree
> diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
> index 7a0a848e8..3d62e6c95 100755
> --- a/ci/test-documentation.sh
> +++ b/ci/test-documentation.sh
> @@ -25,3 +25,5 @@ sed '/^GIT_VERSION = / d' stderr.log
> ! test -s stderr.log
> test -s Documentation/git.html
> grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
> +
> +save_good_tree
> -- 
> 2.15.1.500.g54ea76cc4
> 

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

On Wed, Dec 27, 2017 at 8:35 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:

>> Using an ever-growing flat text file might seem like asking for
>> trouble on the long run, but it's perfectly adequate for this purpose.
>> Contributors' topic branches are short-lived in general, so this file
>> won't grow large enough to cause any issues.  Grepping through several
>> tens of thousands such lines is sufficiently fast, so not even
>> git/git's forever living integration branches will cause scalability
>> issues with the current rate of ~1 push/day for a couple of decades.
>> And even if we reach the point that this file grows too big, the
>> caches can be deleted on Travis CI's web interface.
>
> One more thing:
> Maybe we could delete "$HOME/travis-cache/good-trees" if the file
> has more than 1000 lines *before* we add a new tree?
>
> Or we use something like this to cap the file:
>
>   echo "$(tail -1000 $HOME/travis-cache/good-trees)" > $HOME/travis-cache/good-trees

Well, there is always something new to learn.
I was aware that things like 'cmd file >file' don't work, because the
shell opens and truncates 'file' before executing the command, so 'cmd'
will open the already empty file, but I didn't know that 'echo "$(cmd
file)" >file' works.  Thanks for letting me know.

However, this is subject to the portability issues of the shell's
'echo', i.e. try

  echo "$(cat git.c)" >git.c

with Bash and Dash.  Bash produces the exact same contents, but Dash
turns all '\n' in help and error strings to real newline characters.

Now, Git's object names will never contain such characters, and most
likely $TRAVIS_JOB_{NUMBER,ID} won't ever do that, either, so this is
not an issue for this 'good-trees' file.  Still, I think it'd be better
to stick to using a good old temporary file:

  tail -1000 good-trees >tmp
  mv tmp good-trees

> I agree that the "always growing problem" is not a big one
> but an approach like the one above would avoid any discussion
> for sure.

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Lars Schneider]

> On 28 Dec 2017, at 11:31, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> On Wed, Dec 27, 2017 at 8:35 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> 
>>> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
>>> Using an ever-growing flat text file might seem like asking for
>>> trouble on the long run, but it's perfectly adequate for this purpose.
>>> Contributors' topic branches are short-lived in general, so this file
>>> won't grow large enough to cause any issues.  Grepping through several
>>> tens of thousands such lines is sufficiently fast, so not even
>>> git/git's forever living integration branches will cause scalability
>>> issues with the current rate of ~1 push/day for a couple of decades.
>>> And even if we reach the point that this file grows too big, the
>>> caches can be deleted on Travis CI's web interface.
>> 
>> One more thing:
>> Maybe we could delete "$HOME/travis-cache/good-trees" if the file
>> has more than 1000 lines *before* we add a new tree?
>> 
>> Or we use something like this to cap the file:
>> 
>>  echo "$(tail -1000 $HOME/travis-cache/good-trees)" > $HOME/travis-cache/good-trees
> 
> Well, there is always something new to learn.
> I was aware that things like 'cmd file >file' don't work, because the
> shell opens and truncates 'file' before executing the command, so 'cmd'
> will open the already empty file, but I didn't know that 'echo "$(cmd
> file)" >file' works.  Thanks for letting me know.
> 
> However, this is subject to the portability issues of the shell's
> 'echo', i.e. try
> 
>  echo "$(cat git.c)" >git.c
> 
> with Bash and Dash.  Bash produces the exact same contents, but Dash
> turns all '\n' in help and error strings to real newline characters.
> 
> Now, Git's object names will never contain such characters, and most
> likely $TRAVIS_JOB_{NUMBER,ID} won't ever do that, either, so this is
> not an issue for this 'good-trees' file.  Still, I think it'd be better
> to stick to using a good old temporary file:
> 
>  tail -1000 good-trees >tmp
>  mv tmp good-trees

Agreed!

- Lars

Re: [PATCH 2/2] travis-ci: record and skip successfully built trees

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> I was aware that things like 'cmd file >file' don't work, because the
> shell opens and truncates 'file' before executing the command, so 'cmd'
> will open the already empty file, but I didn't know that 'echo "$(cmd
> file)" >file' works.  Thanks for letting me know.
>
> However, this is subject to the portability issues of the shell's
> 'echo', i.e. try
>
>   echo "$(cat git.c)" >git.c
>
> with Bash and Dash.  Bash produces the exact same contents, but Dash
> turns all '\n' in help and error strings to real newline characters.

Besides, that is a bad practice in general if you do not have a
guarantee that 'git.c' (or whatever file in question) is small
enough not to bust any platform's argv[] length limit.

> ...  Still, I think it'd be better
> to stick to using a good old temporary file:

Yes.

[PATCHv2 0/3] Travis CI: skip commits with successfully built and tested trees

[SZEDER Gábor]

This is the second iteration of 'sg/travis-skip-identical-test',
addressing the comments of Lars and Jonathan:

  - Colorize the "Tip of $TRAVIS_BRANCH is exactly at $TAG" message
    in the new patch 1/3.

  - Create the cache directory at the beginning of the build process
    (patch 2/3).

  - Limit the the cached good trees file size to 1000 records, to
    prevent it from growing too large for git/git's forever living
    integration branches (patch 3/3).

  - Colorize the first line of the "skip build job because this tree has
    been tested".  Green it is (3/3).

  - Removed stray whitespace (3/3).

  - Updated an in-code comment, to make clear which code path deals with
    a non-existing good trees file (3/3).

SZEDER Gábor (3):
  travis-ci: print the "tip of branch is exactly at tag" message in
    color
  travis-ci: create the cache directory early in the build process
  travis-ci: record and skip successfully built trees

 ci/lib-travisci.sh        | 51 ++++++++++++++++++++++++++++++++++++++++++++++-
 ci/run-linux32-docker.sh  |  2 ++
 ci/run-static-analysis.sh |  2 ++
 ci/run-tests.sh           |  3 ++-
 ci/run-windows-build.sh   |  2 ++
 ci/test-documentation.sh  |  2 ++
 6 files changed, 60 insertions(+), 2 deletions(-)

-- 
2.16.0.rc0.67.g3a46dbca7


diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 05e73123f..bade71617 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -16,7 +16,7 @@ skip_branch_tip_with_tag () {
 	if TAG=$(git describe --exact-match "$TRAVIS_BRANCH" 2>/dev/null) &&
 		test "$TAG" != "$TRAVIS_BRANCH"
 	then
-		echo "Tip of $TRAVIS_BRANCH is exactly at $TAG"
+		echo "$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)"
 		exit 0
 	fi
 }
@@ -28,6 +28,9 @@ good_trees_file="$HOME/travis-cache/good-trees"
 # message.
 save_good_tree () {
 	echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
+	# limit the file size
+	tail -1000 "$good_trees_file" >"$good_trees_file".tmp
+	mv "$good_trees_file".tmp "$good_trees_file"
 }
 
 # Skip the build job if the same tree has already been built and tested
@@ -36,23 +39,24 @@ save_good_tree () {
 skip_good_tree () {
 	if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
 	then
-		# haven't seen this tree yet; continue the build job
+		# Haven't seen this tree yet, or no cached good trees file yet.
+		# Continue the build job.
 		return
 	fi
 
 	echo "$good_tree_info" | {
 		read tree prev_good_commit prev_good_job_number prev_good_job_id
 
-		if test "$TRAVIS_JOB_ID" =  "$prev_good_job_id"
+		if test "$TRAVIS_JOB_ID" = "$prev_good_job_id"
 		then
 			cat <<-EOF
-			Skipping build job for commit $TRAVIS_COMMIT.
+			$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)
 			This commit has already been built and tested successfully by this build job.
 			To force a re-build delete the branch's cache and then hit 'Restart job'.
 			EOF
 		else
 			cat <<-EOF
-			Skipping build job for commit $TRAVIS_COMMIT.
+			$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)
 			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
 			The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
 			To force a re-build delete the branch's cache and then hit 'Restart job'.
@@ -69,6 +73,8 @@ skip_good_tree () {
 # and installing dependencies.
 set -ex
 
+mkdir -p "$HOME/travis-cache"
+
 skip_branch_tip_with_tag
 skip_good_tree
 

[PATCHv2 1/3] travis-ci: print the "tip of branch is exactly at tag" message in color

[SZEDER Gábor]

To make this info message stand out from the regular build job trace
output.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/lib-travisci.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 348fe3c3c..9d379db8a 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -16,7 +16,7 @@ skip_branch_tip_with_tag () {
 	if TAG=$(git describe --exact-match "$TRAVIS_BRANCH" 2>/dev/null) &&
 		test "$TAG" != "$TRAVIS_BRANCH"
 	then
-		echo "Tip of $TRAVIS_BRANCH is exactly at $TAG"
+		echo "$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)"
 		exit 0
 	fi
 }
-- 
2.16.0.rc0.67.g3a46dbca7

[PATCHv2 2/3] travis-ci: create the cache directory early in the build process

[SZEDER Gábor]

It seems that Travis CI creates the cache directory for us anyway,
even when a previous cache doesn't exist for the current build job.
Alas, this behavior is not explicitly documented, therefore we don't
rely on it and create the cache directory ourselves in those build
jobs that read/write cached data (currently only the prove state).

In the following commit we'll start to cache additional data in every
build job, and will access the cache much earlier in the build
process.

Therefore move creating the cache directory to 'ci/lib-travisci.sh' to
make sure that it exists at the very beginning of every build job.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/lib-travisci.sh | 2 ++
 ci/run-tests.sh    | 1 -
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 9d379db8a..197aa14c1 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -27,6 +27,8 @@ skip_branch_tip_with_tag () {
 # and installing dependencies.
 set -ex
 
+mkdir -p "$HOME/travis-cache"
+
 skip_branch_tip_with_tag
 
 if test -z "$jobname"
diff --git a/ci/run-tests.sh b/ci/run-tests.sh
index f0c743de9..ccdfc2b9d 100755
--- a/ci/run-tests.sh
+++ b/ci/run-tests.sh
@@ -5,6 +5,5 @@
 
 . ${0%/*}/lib-travisci.sh
 
-mkdir -p $HOME/travis-cache
 ln -s $HOME/travis-cache/.prove t/.prove
 make --quiet test
-- 
2.16.0.rc0.67.g3a46dbca7

[PATCHv2 3/3] travis-ci: record and skip successfully built trees

[SZEDER Gábor]

Travis CI dutifully builds and tests each new branch tip, even if its
tree has previously been successfully built and tested.  This happens
often enough in contributors' workflows, when a work-in-progress
branch is rebased changing e.g. only commit messages or the order or
number of commits while leaving the resulting code intact, and is then
pushed to a Travis CI-enabled GitHub fork.

This is wasting Travis CI's resources and is sometimes scary-annoying
when the new tip commit with a tree identical to the previous,
successfully tested one is suddenly reported in red, because one of
the OSX build jobs happened to exceed the time limit yet again.

So extend our Travis CI build scripts to skip building commits whose
trees have previously been successfully built and tested.  Use the
Travis CI cache feature to keep a record of the object names of trees
that tested successfully, in a plain and simple flat text file, one
line per tree object name.  Append the current tree's object name at
the end of every successful build job to this file, along with a bit
of additional info about the build job (commit object name, Travis CI
job number and id).  Limit the size of this file to 1000 records, to
prevent it from growing too large for git/git's forever living
integration branches.  Check, using a simple grep invocation, in each
build job whether the current commit's tree is already in there, and
skip the build if it is.  Include a message in the skipped build job's
trace log, containing the URL to the build job successfully testing
that tree for the first time and instructions on how to force a
re-build.  Catch the case when a build job, which successfully built
and tested a particular tree for the first time, is restarted and omit
the URL of the previous build job's trace log, as in this case it's
the same build job and the trace log has just been overwritten.

Note: this won't kick in if two identical trees are on two different
branches, because Travis CI caches are not shared between build jobs
of different branches.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/lib-travisci.sh        | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 ci/run-linux32-docker.sh  |  2 ++
 ci/run-static-analysis.sh |  2 ++
 ci/run-tests.sh           |  2 ++
 ci/run-windows-build.sh   |  2 ++
 ci/test-documentation.sh  |  2 ++
 6 files changed, 57 insertions(+)

diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 197aa14c1..bade71617 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -21,6 +21,52 @@ skip_branch_tip_with_tag () {
 	fi
 }
 
+good_trees_file="$HOME/travis-cache/good-trees"
+
+# Save some info about the current commit's tree, so we can skip the build
+# job if we encounter the same tree again and can provide a useful info
+# message.
+save_good_tree () {
+	echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
+	# limit the file size
+	tail -1000 "$good_trees_file" >"$good_trees_file".tmp
+	mv "$good_trees_file".tmp "$good_trees_file"
+}
+
+# Skip the build job if the same tree has already been built and tested
+# successfully before (e.g. because the branch got rebased, changing only
+# the commit messages).
+skip_good_tree () {
+	if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
+	then
+		# Haven't seen this tree yet, or no cached good trees file yet.
+		# Continue the build job.
+		return
+	fi
+
+	echo "$good_tree_info" | {
+		read tree prev_good_commit prev_good_job_number prev_good_job_id
+
+		if test "$TRAVIS_JOB_ID" = "$prev_good_job_id"
+		then
+			cat <<-EOF
+			$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)
+			This commit has already been built and tested successfully by this build job.
+			To force a re-build delete the branch's cache and then hit 'Restart job'.
+			EOF
+		else
+			cat <<-EOF
+			$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)
+			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
+			The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
+			To force a re-build delete the branch's cache and then hit 'Restart job'.
+			EOF
+		fi
+	}
+
+	exit 0
+}
+
 # Set 'exit on error' for all CI scripts to let the caller know that
 # something went wrong.
 # Set tracing executed commands, primarily setting environment variables
@@ -30,6 +76,7 @@ set -ex
 mkdir -p "$HOME/travis-cache"
 
 skip_branch_tip_with_tag
+skip_good_tree
 
 if test -z "$jobname"
 then
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index 3a8b2ba42..870a41246 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -22,3 +22,5 @@ docker run \
 	--volume "${HOME}/travis-cache:/tmp/travis-cache" \
 	daald/ubuntu32:xenial \
 	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+
+save_good_tree
diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
index 68dd0f080..fe4ee4e06 100755
--- a/ci/run-static-analysis.sh
+++ b/ci/run-static-analysis.sh
@@ -6,3 +6,5 @@
 . ${0%/*}/lib-travisci.sh
 
 make coccicheck
+
+save_good_tree
diff --git a/ci/run-tests.sh b/ci/run-tests.sh
index ccdfc2b9d..eb5ba4058 100755
--- a/ci/run-tests.sh
+++ b/ci/run-tests.sh
@@ -7,3 +7,5 @@
 
 ln -s $HOME/travis-cache/.prove t/.prove
 make --quiet test
+
+save_good_tree
diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
index 86999268a..d99a180e5 100755
--- a/ci/run-windows-build.sh
+++ b/ci/run-windows-build.sh
@@ -99,3 +99,5 @@ gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
 
 # Set exit code for TravisCI
 test "$RESULT" = "success"
+
+save_good_tree
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
index 7a0a848e8..3d62e6c95 100755
--- a/ci/test-documentation.sh
+++ b/ci/test-documentation.sh
@@ -25,3 +25,5 @@ sed '/^GIT_VERSION = / d' stderr.log
 ! test -s stderr.log
 test -s Documentation/git.html
 grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
+
+save_good_tree
-- 
2.16.0.rc0.67.g3a46dbca7

Re: [PATCHv2 0/3] Travis CI: skip commits with successfully built and tested trees

[Lars Schneider]

> On 31 Dec 2017, at 11:12, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> 
> This is the second iteration of 'sg/travis-skip-identical-test',
> addressing the comments of Lars and Jonathan:
> 
>  - Colorize the "Tip of $TRAVIS_BRANCH is exactly at $TAG" message
>    in the new patch 1/3.
> 
>  - Create the cache directory at the beginning of the build process
>    (patch 2/3).
> 
>  - Limit the the cached good trees file size to 1000 records, to
>    prevent it from growing too large for git/git's forever living
>    integration branches (patch 3/3).
> 
>  - Colorize the first line of the "skip build job because this tree has
>    been tested".  Green it is (3/3).
> 
>  - Removed stray whitespace (3/3).
> 
>  - Updated an in-code comment, to make clear which code path deals with
>    a non-existing good trees file (3/3).


This series looks good to me.
Nice work!

- Lars