From: Derrick Stolee <stolee@gmail.com>
To: Ben Peart <peartben@gmail.com>, Git List <git@vger.kernel.org>
Subject: Re: Git Test Coverage Report (Tuesday, Sept 25)
Date: Wed, 26 Sep 2018 06:43:14 -0400 [thread overview]
Message-ID: <b46d6363-1709-968e-105a-3f4e8a77155e@gmail.com> (raw)
In-Reply-To: <d92f18e2-0bbc-e36b-123a-3ed3f44cf418@gmail.com>
On 9/25/2018 5:12 PM, Ben Peart wrote:
>
>
> On 9/25/2018 2:42 PM, Derrick Stolee wrote:
>> In an effort to ensure new code is reasonably covered by the test
>> suite, we now have contrib/coverage-diff.sh to combine the gcov
>> output from 'make coverage-test ; make coverage-report' with the
>> output from 'git diff A B' to discover _new_ lines of code that are
>> not covered.
>>
>> This report takes the output of these results after running on four
>> branches:
>>
>> pu: 80e728fa913dc3a1165b6dec9a7afa6052a86325
>>
>> jch: 0c10634844314ab89666ed0a1c7d36dde7ac9689
>>
>> next: 76f2f5c1e34c4dbef1029e2984c2892894c444ce
>>
>> master: fe8321ec057f9231c26c29b364721568e58040f7
>>
>> master@{1}: 2d3b1c576c85b7f5db1f418907af00ab88e0c303
>>
>> I ran the test suite on each of these branches on an Ubuntu Linux VM,
>> and I'm missing some dependencies (like apache, svn, and perforce) so
>> not all tests are run.
>>
>> I submit this output without comment. I'm taking a look especially at
>> my own lines to see where coverage can be improved.
>>
>
> Thanks for driving this. I think it provides an interesting view into
> new code and how well it is being tested. In an effort to make this
> as useful as possible, we should be looking to eliminate as much noise
> as possible otherwise people will stop looking at it.
Thanks for helping identifying the noise.
>
> I looked at the lines that came from my patches and most if not all of
> them are only going to be executed by the test suite if the correct
> "special setup" option is enabled. In my particular case, that is the
> option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.
>
> I suspect this will be the case for other code as well so I wonder if
> the tests should be run with each the GIT_TEST_* options that exist to
> exercise uncommon code paths with the test suite. This should prevent
> false positives on code paths that are actually covered by the test
> suite as long as it is run with the appropriate option set.
This is a bit tricky to do, but I will investigate. For some things, the
values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play
nicely with other index options, I think). For others, we don't have the
environment variables in all versions yet, as they are still merging down.
>
> I realize it would take a long time to run the entire test suite with
> all GIT_TEST_* variables so perhaps they can only be tested "as
> needed" (ie when patches add new variables in the "special setups"
> section of t/README). This should reduce the number of combinations
> that need to be run while still eliminating many of the false positive
> hits.
This is something to think about. For my own thoughts, I was thinking of
trying to run it when we see large blocks of code that are uncovered and
obviously because of environment variables. This is what I was thinking
when I saw your and Duy's commits in the output. I'll see if I can
re-run the suite using GIT_TEST_INDEX_THREADS=2 and
GIT_TEST_INDEX_VERSION=4.
Thanks,
-Stolee
next prev parent reply other threads:[~2018-09-26 10:43 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-25 18:42 Git Test Coverage Report (Tuesday, Sept 25) Derrick Stolee
2018-09-25 21:12 ` Ben Peart
2018-09-26 10:43 ` Derrick Stolee [this message]
2018-09-26 10:56 ` Jason Pyeron
2018-09-26 11:03 ` Derrick Stolee
2018-09-26 18:43 ` Thomas Gummerer
2018-09-26 18:54 ` Derrick Stolee
2018-09-27 15:21 ` Ben Peart
2018-09-27 15:28 ` Derrick Stolee
2018-09-27 15:38 ` Ævar Arnfjörð Bjarmason
2018-09-26 17:59 ` Junio C Hamano
2018-09-26 18:44 ` Derrick Stolee
2018-09-27 15:14 ` Ben Peart
2018-09-27 15:16 ` Derrick Stolee
2018-09-26 18:58 ` Elijah Newren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b46d6363-1709-968e-105a-3f4e8a77155e@gmail.com \
--to=stolee@gmail.com \
--cc=git@vger.kernel.org \
--cc=peartben@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).