Re: [PATCH v1 0/4] Teach 'run' perf script to read config files

[Jeff King <peff@peff.net>]

On Fri, Jul 14, 2017 at 08:27:53AM +0200, Christian Couder wrote:

> The whole thing seems really complex to me though. And this makes me
> think that people might want to specify different GIT-BUILD-OPTIONS
> and config.mak files to be used when running perf tests, so that the
> results from perf tests can easily be reproduced later even if they
> have changed their build options in their development Git repo in the
> meantime.

I agree with the complexity. The general idea is that your currently
built HEAD is a snapshot in time of options. But that doesn't have to be
so, and laying out the options in a config file does seem like an
improvement.

There is another implicit dependency, though: the set of (and exact
content of) the tests depends on your HEAD, too. So if I do:

  git checkout v2.5.0
  cd t/perf
  ./run v2.0.0 v2.1.0

I might get different results if I replace "v2.5.0" in the first command
with some other version, because the content of the tests will be
different. I'm not sure how to account for that in storing results. Most
of the time the version of the tests you ran is not going to be
interesting. But it can be a source of confusing discrepancies if a test
subtly changed between two runs. It probably happens infrequently enough
that it's not worth worrying about.

> So perhaps the config file should make it possible to specify a
> directory where all the build files (GIT-BUILD-OPTIONS, config.mak,
> config.mak.autogen and config.status) that should be used should be
> taken. And then it could also let people change some variables to
> override what is in those files which is needed to run perf tests with
> different parameters.

That sounds reasonable. I think you could ditch GIT-BUILD-OPTIONS
entirely. It's only needed to pull in GIT_PERF variables that would be
better served by being in the config in the first place.

-Peff