[PATCH v3 0/7] Codespeed perf results

[Christian Couder <christian.couder@gmail.com>]

This patch series is built on top of cc/perf-run-config which recently
graduated to master.

It makes it possible to send perf results to a Codespeed server. See
https://github.com/tobami/codespeed/ and web sites like
http://speed.pypy.org/ which are using Codespeed.

The end goal would be to have such a server always available to track
how the different git commands perform over time on different kind of
repos (small, medium, large, ...) with different optimizations on and
off (split-index, libpcre2, BLK_SHA1, ...)

With this series and a config file like:

$ cat perf.conf
[perf]
        dirsOrRevs = v2.12.0 v2.13.0
        repeatCount = 10
        sendToCodespeed = http://localhost:8000
        repoName = Git repo
[perf "with libpcre"]
        makeOpts = "DEVELOPER=1 USE_LIBPCRE=YesPlease"
[perf "without libpcre"]
        makeOpts = "DEVELOPER=1"

One should be able to just launch:

$ ./run --config perf.conf p7810-grep.sh

and then get nice graphs in a Codespeed instance running on
http://localhost:8000.

Caveat
~~~~~~

For now one has to create the "Git repo" environment (in fact all the
values of the "environment" field sent to Codespeed) in the Codespeed
admin interface. (We send the perf.repoName config variable in the
"environment" Codespeed field.) This is because Codespeed requires the
environment fields to be created and does not provide a simple way to
create these fields programmatically.

There are discussions on a Codespeed issue
(https://github.com/tobami/codespeed/issues/232) about creating a
proper API for Codespeed that could address this problem in the
future.

Changes since previous version
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are very few changes compared to v2:

  - In patch 1/7 commit message has been improved following a comment
    by Junio.

  - In patch 3/7 some debugging comments were removed and 'use JSON;'
    was moved to the top of aggregate.perl as suggested by Junio.

The diff is the following:

diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl
index 34d74fc015..5c439f6bc2 100755
--- a/t/perf/aggregate.perl
+++ b/t/perf/aggregate.perl
@@ -3,6 +3,7 @@
 use lib '../../perl/blib/lib';
 use strict;
 use warnings;
+use JSON;
 use Git;
 
 sub get_times {
@@ -226,10 +227,6 @@ sub print_codespeed_results {
                }
        }
 
-       #use Data::Dumper qw/ Dumper /;
-       #print Dumper(\@data);
-
-       use JSON;
        print to_json(\@data, {utf8 => 1, pretty => 1}), "\n";
 }

Links
~~~~~

This patch series:

https://github.com/chriscool/git/commits/codespeed

Previous versions:

v1: https://github.com/chriscool/git/commits/codespeed1
v2: https://github.com/chriscool/git/commits/codespeed10

Discussions:

v1: https://public-inbox.org/git/CAP8UFD3Q4h-aYBDABSPOW948LQYVydWZ1hLPAD+kr9ZpXVZiaQ@mail.gmail.com/
v2: https://public-inbox.org/git/20171226215908.425-1-chriscool@tuxfamily.org/

Discussions about the cc/perf-run-config patch series:

v1: https://public-inbox.org/git/20170713065050.19215-1-chriscool@tuxfamily.org/
v2: https://public-inbox.org/git/CAP8UFD2j-UFh+9awz91gtZ-jusq7EUOExMgURO59vpf29jXS4A@mail.gmail.com/


Christian Couder (7):
  perf/aggregate: fix checking ENV{GIT_PERF_SUBSECTION}
  perf/aggregate: refactor printing results
  perf/aggregate: implement codespeed JSON output
  perf/run: add conf_opts argument to get_var_from_env_or_config()
  perf/run: learn about perf.codespeedOutput
  perf/run: learn to send output to codespeed server
  perf/run: read GIT_PERF_REPO_NAME from perf.repoName

 t/perf/aggregate.perl | 160 +++++++++++++++++++++++++++++++++++---------------
 t/perf/run            |  31 ++++++++--
 2 files changed, 137 insertions(+), 54 deletions(-)

-- 
2.16.0.rc0.40.gbe5e688583