From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: poffice@blade.nagaokaut.ac.jp Delivered-To: poffice@blade.nagaokaut.ac.jp Received: from kankan.nagaokaut.ac.jp (kankan.nagaokaut.ac.jp [133.44.2.24]) by blade.nagaokaut.ac.jp (Postfix) with ESMTP id 27A3B17D6049 for ; Thu, 6 Mar 2014 17:10:32 +0900 (JST) Received: from funfun.nagaokaut.ac.jp (smtp.nagaokaut.ac.jp [133.44.2.201]) by kankan.nagaokaut.ac.jp (Postfix) with ESMTP id 4C4FCB5D895 for ; Thu, 6 Mar 2014 17:05:29 +0900 (JST) Received: from funfun.nagaokaut.ac.jp (localhost.nagaokaut.ac.jp [127.0.0.1]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id 377BA97A858 for ; Thu, 6 Mar 2014 17:05:31 +0900 (JST) X-Virus-Scanned: amavisd-new at nagaokaut.ac.jp Authentication-Results: funfun.nagaokaut.ac.jp (amavisd-new); dkim=fail (1024-bit key) reason="fail (message has been altered)" header.d=sendgrid.me Received: from funfun.nagaokaut.ac.jp ([127.0.0.1]) by funfun.nagaokaut.ac.jp (funfun.nagaokaut.ac.jp [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hTXH3_nviuY3 for ; Thu, 6 Mar 2014 17:05:31 +0900 (JST) Received: from voscc.nagaokaut.ac.jp (voscc.nagaokaut.ac.jp [133.44.1.100]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id 1498997A857 for ; Thu, 6 Mar 2014 17:05:31 +0900 (JST) Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by voscc.nagaokaut.ac.jp (Postfix) with ESMTP id 0C0E995241A for ; Thu, 6 Mar 2014 17:05:28 +0900 (JST) Received: from [221.186.184.76] (unknown [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id E1AFA120556; Thu, 6 Mar 2014 17:05:27 +0900 (JST) X-Original-To: ruby-core@ruby-lang.org Delivered-To: ruby-core@ruby-lang.org Received: from o10.shared.sendgrid.net (o10.shared.sendgrid.net [173.193.132.135]) by neon.ruby-lang.org (Postfix) with SMTP id 28B451204D3 for ; Thu, 6 Mar 2014 17:05:24 +0900 (JST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.me; h=from:to:references:subject:mime-version:content-type:content-transfer-encoding:list-id; s=smtpapi; bh=KcJqTtbTlfwODBibbK4WIELhe8o=; b=Lbuup6Fyw8A00PHj2d IAb4wgeEnsqCbGALKzd44D+dCOiIJK8nUeeifx2IoCClGMUx47C7qTc1YB7Ra9s4 D9Yq49UDNbArEE0l1MDr/jK/fWYoZaN9f34O5oGxaqQt/BnOSbpZ4W5+DEctd72v 1my0EvXNVCfUXk115XjwxCmew= Received: by mf151.sendgrid.net with SMTP id mf151.30574.53182C433 Thu, 06 Mar 2014 08:05:23 +0000 (UTC) Received: from herokuapp.com (ec2-23-20-207-214.compute-1.amazonaws.com [23.20.207.214]) by ismtpd-010 (SG) with ESMTP id 144966ce712.6c89.adc730 Thu, 06 Mar 2014 08:05:23 +0000 (GMT) Date: Thu, 06 Mar 2014 08:05:23 +0000 From: sam.rawlins@gmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Redmine-MailingListIntegration-Message-Ids: 35153 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 9508 X-Redmine-Issue-Author: srawlins X-Redmine-Sender: srawlins X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: OOF Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS5KJOvVVKoUiOc7EPeBqczN38Rw7e9h2vaUMVslueu2k320fgt2zzl+/zmGGJfpCv21hIe5lc2/XMK6YVqbSwh4G9CUxKpbe9PLT67WjfdO0LeliMS/3C+9iiC4UmPEYGc= X-ML-Name: ruby-core X-Mail-Count: 61327 Subject: [ruby-core:61327] [ruby-trunk - Feature #9508] Add method coverage and branch coverage metrics X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #9508 has been updated by Sam Rawlins. Hi Eric, thanks for that test. It revealed my poor choice for `#define RUBY_EVENT_MCOVERAGE 0x040000` (`RUBY_INTERNAL_EVENT_SWITCH` is the same). I've updated my pull request with better choices in `include/ruby/ruby.h`: https://github.com/ruby/ruby/pull/511.patch Another design question: I think that block calls should probably also be considered part of "Method Coverage" (which would then be renamed to "Function Coverage", tracking both method and block coverage). One big design problem here is that it is common to chain several blocks in one line, like iterators. Should I put in the work to track block coverage now, or wait until after this gets merged? ---------------------------------------- Feature #9508: Add method coverage and branch coverage metrics https://bugs.ruby-lang.org/issues/9508#change-45652 * Author: Sam Rawlins * Status: Open * Priority: Normal * Assignee: * Category: * Target version: ---------------------------------------- Since the Coverage extension was introduced in Ruby 1.9, Ruby has had built-in line code coverage. Ruby should support more of the basic code coverage metrics [1]. I have a pull request on GitHub ( https://github.com/ruby/ruby/pull/511 ) to add Method Coverage (Function Coverage) and Branch Coverage. I'd love feedback to improve it. Currently, with this feature, Coverage.result would look like: {"/Users/sam/code/ruby/cov_method.rb" => { lines: [1, 2, 2, 20, nil, nil, 2, 2, 2, nil, 0, nil, nil, nil, 1, 0, nil, nil, 1, 1, nil, nil, 1], methods: {1=>2, 15=>0, 19=>1}, branches: {8=>2, 11=>0} }} which includes * the current Ruby line coverage report, * as well as a method report (The method defined on line 1 was called 2 times; the method on line 15 was called 0 times; ...), * and a branch report (the branch on line 8 was called 2 times; the branch on line 11 was called 0 times). Branches -------- Branches include the bodies of if, elsif, else, unless, and when statements, which are all tracked with this new feature. However, this feature is not aware of void bodies, for example: if foo :ok end will report that only one branch exists in the file. It would be better to declare that there is a branch body on line 2, and a void branch body on line 3, or perhaps line 1. This would require the keys of the [:branch] Hash to be something other than line numbers. Perhaps label_no? Perhaps nd_type(node) paired with line or label_no? More Coverage ------------- I think that Statement Coverage, and Condition Coverage could be added to this feature, using the same techniques. Caveats ------- I was not very clear on the bit-arrays used in ruby.h, and just used values for the new macros that seemed to work. Also, I would much rather use Ranges to identify a branch, so that a Coverage analyzer like SimpleCov won't need any kind of Ruby parser to identify and highlight a full chunk of code as a tested branch, or a not tested branch. I'm trying to find how that could be implemented... [1] Wikipedia has good definitions: http://en.wikipedia.org/wiki/Code_coverage ---Files-------------------------------- pull-request-511.patch (26.7 KB) pull-request-511.patch (38.5 KB) pull-request-511.patch (57 KB) -- http://bugs.ruby-lang.org/