git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Derrick Stolee <stolee@gmail.com>
To: Jeff King <peff@peff.net>
Cc: "git@vger.kernel.org" <git@vger.kernel.org>
Subject: Re: Git Test Coverage Report (Thursday, Aug 15)
Date: Tue, 20 Aug 2019 08:54:20 -0400	[thread overview]
Message-ID: <a081ccdb-bb0d-00a1-136a-c8fa54296639@gmail.com> (raw)
In-Reply-To: <20190816181103.GA13894@sigill.intra.peff.net>

On 8/16/2019 2:11 PM, Jeff King wrote:
> On Thu, Aug 15, 2019 at 10:01:04PM -0400, Derrick Stolee wrote:
> 
>> Here is today's test coverage report.
> 
> Are the scripts you use to generate these available somewhere?

The scripts I originally used are available in contrib/coverage-diff.sh.

However, to allow for some more advanced workflows, I created the following
C# project: https://github.com/derrickstolee/git-test-coverage/tree/master/TestCoverageReport

The runs are done via the following Azure Pipelines build:
https://dev.azure.com/git/git/_build/results?buildId=879

There, you can see how the code is built, tested, and the gcov output
is collected in the TestCoverageReport tool to generate the .txt and .htm
files.
 
> I think it's useful to look at uncovered code, but I often struggle to
> figure out whether the parts attached to my name are relevant. In
> particular, I think two changes to the report format might help:
> 
>   1. Include names alongside commit ids when listing uncovered lines. I
>      know that will end up with some overly-long lines, but it makes it
>      easy to grep for one's name to find relevant sections of the file
>      (as opposed to finding your name at the bottom and
>      cross-referencing with actual content lines).
> 
>      Seeing that an uncovered line is a BUG(), for example, makes it
>      easy to know that it's not really an interesting uncovered case in
>      the code.

The HTML reports [1] have the following feature: click on a commit in the
commit summary and it highlights the lines from that commit.

[1] https://derrickstolee.github.io/git-test-coverage/reports/2019-08-15.htm

However, the plain-text report _could_ add names. Maybe instead the report
should just group the output by commit instead of branch.

>   2. Include more context. Just taking a random example from this email:
> 
>> builtin/rebase.c
>> e191cc8b 129) strbuf_addstr(&buf, strategy_opts);
> 
> We know what the uncovered line was trying to do, but more interesting
> is likely the conditional that causes it to be uncovered. In this case
> the surrounding code is:
> 
>           if (opts->ignore_whitespace) {
>                   struct strbuf buf = STRBUF_INIT;
>   
>                   if (strategy_opts)
>                           strbuf_addstr(&buf, strategy_opts);
>   
>                   strbuf_addstr(&buf, " --ignore-space-change");
>                   free(strategy_opts);
>                   strategy_opts = strbuf_detach(&buf, NULL);
>           }
> 
> even the usual 3 lines of diff context would make it a lot quicker to
> understand what's going on (it only kicks in when multiple strategy
> options are used).

In this case, the additional context helps, but only if you expand by
several lines. In other cases, the necessary context could be dozens of
lines.

Perhaps a deeper report could include something like "git format-patch
--function-context" to naturally extend to the appropriate context. It
may be better to have a "block context" for these conditionals. In such
a case, it would be important to mark the "new" lines explicitly so the
context lines don't become too noisy.

> (As an aside, this code leaks the newly allocated buffer and leaves a
> dangling pointer in opts->strategy_opts, but that's all orthogonal to
> the uncovered line; I'll send a separate message to the original
> author).
> 
> 
> Anyway, I wonder if we could adjust the output of the script to make
> reading it that way a bit easier.

I'll think about this, for sure. When I have time to go deep into the
report (not just the code I wrote) I look for lines that seem to be
non-trivial, then go find their context by looking at the commit.
Here is another example of why the HTML report is helpful: it will
link to the commit diff view [2] or the exact line of the file [3].
(Note: the exact line link only works if we point to the branch that
was tested, not the commit that made the change.)

[2] https://github.com/git/git/commit/e191cc8b8080f63a0080031bf1276269c6bf42dd
[3] https://github.com/git/git/blob/6cf2e4a04610efe882eb663cd36436b80b343fa0/builtin/rebase.c#L129

Thanks,
-Stolee

      parent reply	other threads:[~2019-08-20 12:54 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-16  2:01 Git Test Coverage Report (Thursday, Aug 15) Derrick Stolee
2019-08-16 18:11 ` Jeff King
2019-08-16 18:15   ` Jeff King
2019-08-20 12:54   ` Derrick Stolee [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a081ccdb-bb0d-00a1-136a-c8fa54296639@gmail.com \
    --to=stolee@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).