git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: git <git@vger.kernel.org>
Cc: "Junio C Hamano" <gitster@pobox.com>, "Jeff King" <peff@peff.net>,
	"Thomas Rast" <tr@thomasrast.ch>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v2 0/9] Teach 'run' perf script to read config files
Date: Sat, 23 Sep 2017 21:39:02 +0200	[thread overview]
Message-ID: <CAP8UFD2j-UFh+9awz91gtZ-jusq7EUOExMgURO59vpf29jXS4A@mail.gmail.com> (raw)

(It looks like smtp.gmail.com isn't working anymore for me, so I am
trying to send this using Gmail for the cover letter and Submitgit for
the patches.)

Goal
~~~~

Using many long environment variables to give parameters to the 'run'
script is error prone and tiring.

We want to make it possible to store the parameters to the 'run'
script in a config file. This makes it easier to store, reuse,
share and compare parameters.

It also makes it easy to run series of tests.

Design
~~~~~~

We use the same config format as the ".git/config" file as Git users
and developers are familiar with this nice format and have great tools
to change, query and manage it.

We use values from the config file to set the environment variables
that are used by the scripts if they are not already set.

We want to make it possible to run series of tests by passing only a
config file to the 'run' script.

For example a config file like the following can be used to run perf
tests with Git compiled both with and without libpcre:

[perf]
        dirsOrRevs = v2.12.0 v2.13.0
        repeatCount = 10
[perf "with libpcre"]
        makeOpts = "DEVELOPER=1 USE_LIBPCRE=YesPlease"
[perf "without libpcre"]
        makeOpts = "DEVELOPER=1"

This makes it easy to see what changes between the different runs.

It's also possible (though maybe not so useful) to just separate tests
from different versions like this:

[perf]
        repeatCount = 2
        makeOpts = "DEVELOPER=1 USE_LIBPCRE=YesPlease"

[perf "with v2.12.0 and v2.13.1"]
        dirsOrRevs = v2.12.0 v2.13.1
[perf "with v2.11.0 and v2.12.1"]
        dirsOrRevs = v2.11.0 v2.12.1

Highlevel view of the patches in the series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - Patches 1/9 to 4/9 were already in v1 and haven't changed.

Patch 1/9 teaches the '--config <configfile>' option to the 'run'
script, but <configfile> is just put into the GIT_PERF_CONFIG_FILE
variable which is not used yet.

Patch 2/9 add the get_var_from_env_or_config() function to read config
options from the <configfile> and set values to some variables from
these config options or from default values.

Patch 3/9 and 4/9 use the get_var_from_env_or_config() function to
make it possible to set parameters used by the 'run' script.

  - Patches 5/9 to 9/9 are new.

Patch 5/9 introduce a function to deal with subsections in the config
file.

Patch 6/9 improves the get_var_from_env_or_config() function so that
it can handle subsections.

Patch 7/9 adds the run_subsection() function to run the tests for a
subsection.

Patch 8/9 improves the output when building a rev.

Patch 9/9 stores subsection results into subdirectories of test-results
so that results from previous subsections is not overwritten.

Future work
~~~~~~~~~~~

In the future I may work on the following:

  - improving aggregate.perl so that it can aggregates the results in
    different ways and formats, especially so that the results can be
    used by Codespeed (https://github.com/tobami/codespeed)
  - making it possible to configure more things in the config file
  - improving how GIT-BUILD-OPTIONS is handled

Though I think the series does not need the above improvements to be
already valuable.

Links
~~~~~

This patch series is also available here:

  https://github.com/chriscool/git/commits/perf-conf

Links to the previous version of this series are:

v1:
  https://github.com/chriscool/git/commits/perf-conf5
  https://public-inbox.org/git/20170713065050.19215-1-chriscool@tuxfamily.org/

Christian Couder (9):
  perf/run: add '--config' option to the 'run' script
  perf/run: add get_var_from_env_or_config()
  perf/run: add GIT_PERF_DIRS_OR_REVS
  perf/run: add calls to get_var_from_env_or_config()
  perf/run: add get_subsections()
  perf/run: update get_var_from_env_or_config() for subsections
  perf/run: add run_subsection()
  perf/run: show name of rev being built
  perf: store subsection results in "test-results/$GIT_PERF_SUBSECTION/"

 t/perf/aggregate.perl | 11 +++++--
 t/perf/perf-lib.sh    |  4 +--
 t/perf/run            | 89 +++++++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 15 deletions(-)

-- 
2.14.1.767.g2dbbf9317b

             reply	other threads:[~2017-09-23 19:39 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-09-23 19:39 Christian Couder [this message]
2017-09-24  7:59 ` [PATCH v2 0/9] Teach 'run' perf script to read config files Junio C Hamano
2017-09-26 15:40   ` Christian Couder
2017-09-26 19:33     ` Roberto Tyley
2017-09-26 19:38       ` Stefan Beller
2017-09-27  0:04       ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAP8UFD2j-UFh+9awz91gtZ-jusq7EUOExMgURO59vpf29jXS4A@mail.gmail.com \
    --to=christian.couder@gmail.com \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=peff@peff.net \
    --cc=tr@thomasrast.ch \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).