git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Derrick Stolee <stolee@gmail.com>
To: Ben Peart <peartben@gmail.com>, Git List <git@vger.kernel.org>
Subject: Re: Git Test Coverage Report (Tuesday, Sept 25)
Date: Wed, 26 Sep 2018 07:03:20 -0400	[thread overview]
Message-ID: <39fbf022-a136-7d8a-a149-ef9aeeffc491@gmail.com> (raw)
In-Reply-To: <551b01d45587$90b27f30$b2177d90$@pdinc.us>

On 9/26/2018 6:56 AM, Jason Pyeron wrote:
>> -----Original Message-----
>> From: Derrick Stolee
>> Sent: Wednesday, September 26, 2018 6:43 AM
>>
>> On 9/25/2018 5:12 PM, Ben Peart wrote:
>>>
>>> On 9/25/2018 2:42 PM, Derrick Stolee wrote:
>>>> In an effort to ensure new code is reasonably covered by the test
>>>> suite, we now have contrib/coverage-diff.sh to combine the gcov
>>>> output from 'make coverage-test ; make coverage-report' with the
>>>> output from 'git diff A B' to discover _new_ lines of code that are
>>>> not covered.
>>>>
> <snip/>
>>> I looked at the lines that came from my patches and most if not all of
>>> them are only going to be executed by the test suite if the correct
>>> "special setup" option is enabled.  In my particular case, that is the
>>> option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.
>>>
>>> I suspect this will be the case for other code as well so I wonder if
>>> the tests should be run with each the GIT_TEST_* options that exist to
>>> exercise uncommon code paths with the test suite.  This should prevent
>>> false positives on code paths that are actually covered by the test
>>> suite as long as it is run with the appropriate option set.
>> This is a bit tricky to do, but I will investigate. For some things, the
>> values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play
>> nicely with other index options, I think). For others, we don't have the
>> environment variables in all versions yet, as they are still merging down.
> Remember that the code coverage is cumulative, on one of my projects where I have similar special cases the automated build will run through a sequence of different build options (including architectures) and executions - then generate the final report.
For Git, I'm just using "make coverage-test" and "make coverage-report". 
To make the results cumulative, I'd need to remove the 
"coverage-clean-results" as a dependency to "coverage-test". This may be 
a good thing to do, so I can run the test suite with multiple options. 
Perhaps I'll just add a new "coverage-test-with-options" step that just 
runs the tests in "default" mode and then "enhanced" mode.
> Another thought would be to weight the report for "new lines of code" - which is the same we do for our Fortify scans. We take the git blame for the change range and de-prioritize the findings for lines outside of the changed lines. This could give a tighter focus on the newly change code's test coverage.
This is what the contrib/coverage-diff.sh script does. I'd love your 
input, as it is still under review [1].

[1] 
https://public-inbox.org/git/21214cc321f80cf2e9eb0cdb1ec3ebb869ea496d.1537542952.git.gitgitgadget@gmail.com/
     [PATCH v3 1/1] contrib: add coverage-diff script

  reply	other threads:[~2018-09-26 11:03 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-25 18:42 Git Test Coverage Report (Tuesday, Sept 25) Derrick Stolee
2018-09-25 21:12 ` Ben Peart
2018-09-26 10:43   ` Derrick Stolee
2018-09-26 10:56     ` Jason Pyeron
2018-09-26 11:03       ` Derrick Stolee [this message]
2018-09-26 18:43     ` Thomas Gummerer
2018-09-26 18:54       ` Derrick Stolee
2018-09-27 15:21         ` Ben Peart
2018-09-27 15:28           ` Derrick Stolee
2018-09-27 15:38             ` Ævar Arnfjörð Bjarmason
2018-09-26 17:59 ` Junio C Hamano
2018-09-26 18:44   ` Derrick Stolee
2018-09-27 15:14     ` Ben Peart
2018-09-27 15:16       ` Derrick Stolee
2018-09-26 18:58 ` Elijah Newren

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=39fbf022-a136-7d8a-a149-ef9aeeffc491@gmail.com \
    --to=stolee@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=peartben@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).