Re: [PATCH v1 0/4] Teach 'run' perf script to read config files

[Christian Couder <christian.couder@gmail.com>]

On Thu, Jul 13, 2017 at 10:55 PM, Jeff King <peff@peff.net> wrote:
> On Thu, Jul 13, 2017 at 08:57:01PM +0200, Christian Couder wrote:
>
>> >> We want to make it possible to store the parameters to the 'run'
>> >> script in a config file. This will make it easier to store, reuse,
>> >> share and compare parameters.
>> >
>> > Because perf-lib is built on test-lib, it already reads
>> > GIT-BUILD-OPTIONS.
>>
>> Actually the 'run' script also sources GIT-BUILD-OPTIONS, so maybe
>> this is not necessary.
>
> Ah, right. The one that comes via perf-lib gets the variables into the
> test scripts themselves. But anything "run" would need itself would come
> from the source it does itself. And that's where GIT_PERF_MAKE_OPTS has
> an effect.
>
>> Also are the variables in GIT-BUILD-OPTIONS exported already?
>
> No, I don't think so. But because both "run" and the scripts themselves
> source them, they're available more or less everywhere, except for
> sub-processes inside the scripts.

Ok, I see.

>> > And the Makefile copies several perf-related values
>> > into it, including GIT_PERF_MAKE_OPTS and GIT_PERF_REPEAT_COUNT. So you
>> > can already do:
>> >
>> >   echo 'GIT_PERF_REPEAT_COUNT = 10' >>config.mak
>> >   echo 'GIT_PERF_MAKE_OPTS = CFLAGS="-O2" DEVELOPER=1' >>config.mak
>> >   make
>>
>> The "make" here might not even be needed as in the 'run' script
>> "config.mak" is copied into the "build/$rev" directory where "make" is
>> run to build the $rev version.
>
> You need it to bake the config into GIT-BUILD-OPTIONS, which is the only
> thing that gets read by "run" and the perf scripts. If you are
> just setting MAKE_OPTS to things that your config.mak already sets, then
> yes, you can skip that one.

Thanks for the explanations.

The whole thing seems really complex to me though. And this makes me
think that people might want to specify different GIT-BUILD-OPTIONS
and config.mak files to be used when running perf tests, so that the
results from perf tests can easily be reproduced later even if they
have changed their build options in their development Git repo in the
meantime.

So perhaps the config file should make it possible to specify a
directory where all the build files (GIT-BUILD-OPTIONS, config.mak,
config.mak.autogen and config.status) that should be used should be
taken. And then it could also let people change some variables to
override what is in those files which is needed to run perf tests with
different parameters.