[GSoC] [PATCH] travis-ci: added clang static analysis

[GSoC] [PATCH] travis-ci: added clang static analysis

[SiddharthaMishra]

Added a job to run clang static code analysis on the master and maint branch

Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
---
 .travis.yml               | 17 ++++++++++++++++-
 ci/run-static-analysis.sh |  9 ++++++++-
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 4684b3f4f..9b891d182 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -48,7 +48,7 @@ matrix:
       before_install:
       before_script:
       script: ci/run-linux32-docker.sh
-    - env: jobname=StaticAnalysis
+    - env: jobname=CocciStaticAnalysis
       os: linux
       compiler:
       addons:
@@ -59,6 +59,21 @@ matrix:
       before_script:
       script: ci/run-static-analysis.sh
       after_failure:
+    - if: branch IN (master, maint)
+      env: jobname=ClangStaticAnalysis
+      os: linux
+      compiler:
+      add_ons:
+        apt:
+          sources:
+          - ubuntu-toolchain-r-test
+          - llvm-toolchain-trusty
+          packages:
+          - clang
+      before_install:
+      before_script:
+      script: ci/run-static-analysis.sh
+      after_failure:
     - env: jobname=Documentation
       os: linux
       compiler:
diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
index fe4ee4e06..6ae032f54 100755
--- a/ci/run-static-analysis.sh
+++ b/ci/run-static-analysis.sh
@@ -5,6 +5,13 @@
 
 . ${0%/*}/lib-travisci.sh
 
-make coccicheck
+case "$jobname" in
+ClangStaticAnalysis)
+	scan-build -analyze-headers --status-bugs make
+	;;
+CocciStaticAnalysis)
+	make coccicheck
+	;;
+esac
 
 save_good_tree
-- 
2.16.2.248.ge2408a6f7.dirty

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Junio C Hamano]

SiddharthaMishra <sidm1999@gmail.com> writes:

> Added a job to run clang static code analysis on the master and maint branch
>
> Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
> ---

Why on 'master' and 'maint' and not others?  Quite frankly, I find
this choice of branches rather odd, as these two branches are not
where the real development happens.  If we do not want to increase
the number of jobs and limit the test only to a single branch, I
would probably pick 'next', and if we can afford two, probably
'pu' and 'next'.

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Siddhartha Mishra]

On Tue, Mar 6, 2018 at 6:57 AM, Junio C Hamano <gitster@pobox.com> wrote:
> SiddharthaMishra <sidm1999@gmail.com> writes:
>
>> Added a job to run clang static code analysis on the master and maint branch
>>
>> Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
>> ---
>
> Why on 'master' and 'maint' and not others?  Quite frankly, I find
> this choice of branches rather odd, as these two branches are not
> where the real development happens.  If we do not want to increase
> the number of jobs and limit the test only to a single branch, I
> would probably pick 'next', and if we can afford two, probably
> 'pu' and 'next'.

I might have been misinterpreting it, but I did so because that's what
it said in the microproject description. Thinking about it, I guess it
does makes more sense to work on the other two branches instead. Are
there any other glaring issues you see in the code?

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Johannes Schindelin]

Hi,

On Tue, 6 Mar 2018, Siddhartha Mishra wrote:

> Are there any other glaring issues you see in the code?

The rest looks good to me!

Ciao,
Johannes

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Lars Schneider]

Hi,

That looks interesting but I agree with Dscho that we should not limit
this to master/maint.

I assume you did run this on TravisCI already? Can you share a link?
I assume you did find errors? Can we fix them or are there too many?
If there are existing errors, how do we define a "successful" build?

Thanks for working on this,
Lars

> On 05 Mar 2018, at 21:04, SiddharthaMishra <sidm1999@gmail.com> wrote:
> 
> Added a job to run clang static code analysis on the master and maint branch
> 
> Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
> ---
> .travis.yml               | 17 ++++++++++++++++-
> ci/run-static-analysis.sh |  9 ++++++++-
> 2 files changed, 24 insertions(+), 2 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 4684b3f4f..9b891d182 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -48,7 +48,7 @@ matrix:
>       before_install:
>       before_script:
>       script: ci/run-linux32-docker.sh
> -    - env: jobname=StaticAnalysis
> +    - env: jobname=CocciStaticAnalysis
>       os: linux
>       compiler:
>       addons:
> @@ -59,6 +59,21 @@ matrix:
>       before_script:
>       script: ci/run-static-analysis.sh
>       after_failure:
> +    - if: branch IN (master, maint)
> +      env: jobname=ClangStaticAnalysis
> +      os: linux
> +      compiler:
> +      add_ons:
> +        apt:
> +          sources:
> +          - ubuntu-toolchain-r-test
> +          - llvm-toolchain-trusty
> +          packages:
> +          - clang
> +      before_install:
> +      before_script:
> +      script: ci/run-static-analysis.sh
> +      after_failure:
>     - env: jobname=Documentation
>       os: linux
>       compiler:
> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
> index fe4ee4e06..6ae032f54 100755
> --- a/ci/run-static-analysis.sh
> +++ b/ci/run-static-analysis.sh
> @@ -5,6 +5,13 @@
> 
> . ${0%/*}/lib-travisci.sh
> 
> -make coccicheck
> +case "$jobname" in
> +ClangStaticAnalysis)
> +	scan-build -analyze-headers --status-bugs make
> +	;;
> +CocciStaticAnalysis)
> +	make coccicheck
> +	;;
> +esac
> 
> save_good_tree
> -- 
> 2.16.2.248.ge2408a6f7.dirty
> 

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Siddhartha Mishra]

On Mon, Mar 12, 2018 at 3:49 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
> Hi,
>
> That looks interesting but I agree with Dscho that we should not limit
> this to master/maint.
>
> I assume you did run this on TravisCI already? Can you share a link?
> I assume you did find errors? Can we fix them or are there too many?
> If there are existing errors, how do we define a "successful" build?
>
> Thanks for working on this,
> Lars
>

Thanks for the reply,

I assume there will be false positives in the code which we can't fix
by making small modifications to the code as recommended in the FAQ
(https://clang-analyzer.llvm.org/faq.html). According to the FAQ,
there is no solid mechanism for suppressing a specific warning, so are
options are limited. Some of the things which might help reduce the
noise are:

1) To add specific tags in our source code to tell the analyzer to
ignore the code. This is probably a bad idea since it is intrusive and
forces changes to the actual source code which only affect one task.

2) Count the number of bugs in the previous pushed build and fail the
build if the number of bugs increases. It doesn't help remove the
noise from the error log but it does tell you if you've added more
bugs. However if you add a bug and remove one, it'll pass the job and
might mislead you into thinking that the code is correct.

3) Write a script to check the diff of the error log from that of the
previous pushed build(ignoring the line numbers). I haven't thought
about how exactly it would be implemented so I'm not commenting on it.

Is there a better solution that I'm missing or should I try coming up
with a script to come up the diff?

Thanks for the time,
Siddhartha

On Mon, Mar 12, 2018 at 3:49 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
> Hi,
>
> That looks interesting but I agree with Dscho that we should not limit
> this to master/maint.
>
> I assume you did run this on TravisCI already? Can you share a link?
> I assume you did find errors? Can we fix them or are there too many?
> If there are existing errors, how do we define a "successful" build?
>
> Thanks for working on this,
> Lars
>
>> On 05 Mar 2018, at 21:04, SiddharthaMishra <sidm1999@gmail.com> wrote:
>>
>> Added a job to run clang static code analysis on the master and maint branch
>>
>> Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
>> ---
>> .travis.yml               | 17 ++++++++++++++++-
>> ci/run-static-analysis.sh |  9 ++++++++-
>> 2 files changed, 24 insertions(+), 2 deletions(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 4684b3f4f..9b891d182 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -48,7 +48,7 @@ matrix:
>>       before_install:
>>       before_script:
>>       script: ci/run-linux32-docker.sh
>> -    - env: jobname=StaticAnalysis
>> +    - env: jobname=CocciStaticAnalysis
>>       os: linux
>>       compiler:
>>       addons:
>> @@ -59,6 +59,21 @@ matrix:
>>       before_script:
>>       script: ci/run-static-analysis.sh
>>       after_failure:
>> +    - if: branch IN (master, maint)
>> +      env: jobname=ClangStaticAnalysis
>> +      os: linux
>> +      compiler:
>> +      add_ons:
>> +        apt:
>> +          sources:
>> +          - ubuntu-toolchain-r-test
>> +          - llvm-toolchain-trusty
>> +          packages:
>> +          - clang
>> +      before_install:
>> +      before_script:
>> +      script: ci/run-static-analysis.sh
>> +      after_failure:
>>     - env: jobname=Documentation
>>       os: linux
>>       compiler:
>> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
>> index fe4ee4e06..6ae032f54 100755
>> --- a/ci/run-static-analysis.sh
>> +++ b/ci/run-static-analysis.sh
>> @@ -5,6 +5,13 @@
>>
>> . ${0%/*}/lib-travisci.sh
>>
>> -make coccicheck
>> +case "$jobname" in
>> +ClangStaticAnalysis)
>> +     scan-build -analyze-headers --status-bugs make
>> +     ;;
>> +CocciStaticAnalysis)
>> +     make coccicheck
>> +     ;;
>> +esac
>>
>> save_good_tree
>> --
>> 2.16.2.248.ge2408a6f7.dirty
>>
>

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Siddhartha Mishra]

I'm extremely sorry, I forgot to attach my travis-ci job build with my
previous message. Here it is :
https://travis-ci.org/SiddharthaMishra/git/jobs/349478746#L1204.

I also apologize for not trimming the email properly in the previous message.

Sorry for the inconvenience,
Siddhartha

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Lars Schneider]

> On 13 Mar 2018, at 18:45, Siddhartha Mishra <sidm1999@gmail.com> wrote:
> 
> On Mon, Mar 12, 2018 at 3:49 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>> 
>> That looks interesting but I agree with Dscho that we should not limit
>> this to master/maint.
>> 
>> I assume you did run this on TravisCI already? Can you share a link?
>> I assume you did find errors? Can we fix them or are there too many?
>> If there are existing errors, how do we define a "successful" build?
>> 
>> Thanks for working on this,
>> Lars
>> 
> 
> Thanks for the reply,
> 
> I assume there will be false positives in the code which we can't fix
> by making small modifications to the code as recommended in the FAQ
> (https://clang-analyzer.llvm.org/faq.html). According to the FAQ,
> there is no solid mechanism for suppressing a specific warning, so are
> options are limited. Some of the things which might help reduce the
> noise are:
> 
> 1) To add specific tags in our source code to tell the analyzer to
> ignore the code. This is probably a bad idea since it is intrusive and
> forces changes to the actual source code which only affect one task.
> 
> 2) Count the number of bugs in the previous pushed build and fail the
> build if the number of bugs increases. It doesn't help remove the
> noise from the error log but it does tell you if you've added more
> bugs. However if you add a bug and remove one, it'll pass the job and
> might mislead you into thinking that the code is correct.
> 
> 3) Write a script to check the diff of the error log from that of the
> previous pushed build(ignoring the line numbers). I haven't thought
> about how exactly it would be implemented so I'm not commenting on it.
> 
> Is there a better solution that I'm missing or should I try coming up
> with a script to come up the diff?

That's a good summary and I don't see a better solution. While (3)
sounds nice, I think (2) is the fastest/most pragmatic solution.

We already use the Travis cache [1]. You could use that mechanism to 
store a file with the latest number of bugs in the cache directory 
$HOME/travis-cache

If the "number of bugs" file does not exist, then create it and don't
complain. If the file exists and the previous number of bugs is higher
or equal, then don't complain either. If the file exists and the previous
number of bugs is lower, then let the build fail.

Do you think that could work?

Cheers,
Lars 

[1] https://docs.travis-ci.com/user/caching/


> 
> Thanks for the time,
> Siddhartha
> 
> On Mon, Mar 12, 2018 at 3:49 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>> 
>> That looks interesting but I agree with Dscho that we should not limit
>> this to master/maint.
>> 
>> I assume you did run this on TravisCI already? Can you share a link?
>> I assume you did find errors? Can we fix them or are there too many?
>> If there are existing errors, how do we define a "successful" build?
>> 
>> Thanks for working on this,
>> Lars
>> 
>>> On 05 Mar 2018, at 21:04, SiddharthaMishra <sidm1999@gmail.com> wrote:
>>> 
>>> Added a job to run clang static code analysis on the master and maint branch
>>> 
>>> Signed-off-by: SiddharthaMishra <sidm1999@gmail.com>
>>> ---
>>> .travis.yml               | 17 ++++++++++++++++-
>>> ci/run-static-analysis.sh |  9 ++++++++-
>>> 2 files changed, 24 insertions(+), 2 deletions(-)
>>> 
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 4684b3f4f..9b891d182 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -48,7 +48,7 @@ matrix:
>>>      before_install:
>>>      before_script:
>>>      script: ci/run-linux32-docker.sh
>>> -    - env: jobname=StaticAnalysis
>>> +    - env: jobname=CocciStaticAnalysis
>>>      os: linux
>>>      compiler:
>>>      addons:
>>> @@ -59,6 +59,21 @@ matrix:
>>>      before_script:
>>>      script: ci/run-static-analysis.sh
>>>      after_failure:
>>> +    - if: branch IN (master, maint)
>>> +      env: jobname=ClangStaticAnalysis
>>> +      os: linux
>>> +      compiler:
>>> +      add_ons:
>>> +        apt:
>>> +          sources:
>>> +          - ubuntu-toolchain-r-test
>>> +          - llvm-toolchain-trusty
>>> +          packages:
>>> +          - clang
>>> +      before_install:
>>> +      before_script:
>>> +      script: ci/run-static-analysis.sh
>>> +      after_failure:
>>>    - env: jobname=Documentation
>>>      os: linux
>>>      compiler:
>>> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
>>> index fe4ee4e06..6ae032f54 100755
>>> --- a/ci/run-static-analysis.sh
>>> +++ b/ci/run-static-analysis.sh
>>> @@ -5,6 +5,13 @@
>>> 
>>> . ${0%/*}/lib-travisci.sh
>>> 
>>> -make coccicheck
>>> +case "$jobname" in
>>> +ClangStaticAnalysis)
>>> +     scan-build -analyze-headers --status-bugs make
>>> +     ;;
>>> +CocciStaticAnalysis)
>>> +     make coccicheck
>>> +     ;;
>>> +esac
>>> 
>>> save_good_tree
>>> --
>>> 2.16.2.248.ge2408a6f7.dirty
>>> 
>> 

Re: [GSoC] [PATCH] travis-ci: added clang static analysis

[Siddhartha Mishra]

On Sun, Apr 1, 2018, 8:10 PM Lars Schneider <larsxschneider@gmail.com> wrote:

>
> That's a good summary and I don't see a better solution. While (3)
> sounds nice, I think (2) is the fastest/most pragmatic solution.
>
> We already use the Travis cache [1]. You could use that mechanism to
> store a file with the latest number of bugs in the cache directory
> $HOME/travis-cache
>
> If the "number of bugs" file does not exist, then create it and don't
> complain. If the file exists and the previous number of bugs is higher
> or equal, then don't complain either. If the file exists and the previous
> number of bugs is lower, then let the build fail.
>
> Do you think that could work?

Apologies for having sent this message twice, the last one wasn't in
plain text and were therefore not properly forwarded to the mail
group.

If we only consider the number of bugs in the last pushed commit, a
situation might happen that a commit creates a lot of bugs. If a
subsequent commit then fixes only a few of the previously created bugs
the build would still pass since the number of bugs from the last
commit has decreased.

Should this be the intended behaviour or should adding the number of
allowed bugs be a more deliberate process?

Thanks,
Siddhartha