Re: [PATCH] p5310: stop timing non-bitmap pack-to-disk

[Jeff King <peff@peff.net>]

On Thu, Mar 26, 2020 at 06:48:37AM -0400, Derrick Stolee wrote:

> >   - the point of the t/perf suite is to find performance regressions,
> >     and it won't help with that. We don't compare the numbers between
> >     two tests (which the perf suite has no idea are even related), and
> >     any change in its numbers would have nothing to do with bitmaps.
> 
> This does make me think if there is a way to adjust "./run" to test two
> different config settings or command-line options instead of just two
> different build versions. Perhaps something like
> 
>  ./run "HEAD -c core.commitGraph=true" "HEAD -c core.commitGraph=false" -- p4200-line-log.sh
> 
> But that's just musing on my part.

I think that's the right direction, but it would require tests knowing
how to make use of those extra parameters. What I'd _really_ like is a
way to parameterize a whole bunch of things:

 - test repo
 - git version
 - particular config settings

do runs with whatever combinations of parameters you choose, and then
create tables showing differences across various settings of a
parameter. Some parameters would be automatically set up, but test
scripts would have to advertise other parameters they know about.

I don't think it's conceptually hard, but the existing perf code is
pretty hacked-together and relies on the filesystem for storage. So the
notion that we'd only ever compare different versions runs deep; that's
how the on-disk storage is organized.

> > So let's just drop it. It's not useful and is adding minutes to perf
> > runs.
> 
> I agree with your reasoning. This is not a critical path for clients,
> and all servers should be using bitmaps.

Yeah, I should have noted that we're not actually testing the
performance of a stock repack anywhere else. It wouldn't belong in
p5310, but in theory it might be of interest in another script. But I
agree for the reasons you gave that it's not all that interesting a
timing.

-Peff