Re: Covierty Integration / Improvement

[Derek Zimmer <derek@ostif.org>]

Hello all, (this is a resend, google mail arbitrarily decides to
switch out of plaintext whenever it likes)

Thank you for the conversations about Coverity. After some internal
discussions and negotiating with our security partners, we have
secured some engineers directly from Github who want to work on CodeQL
for Git. They will do the work of getting CodeQL working, do a scan,
and then evaluate how much work getting CodeQL into a usable state for
Git is by looking at the false positive rate and figuring out what can
be muted with rules, and the false negative rate vs Coverity / other
current tests and create some custom tests.

This should give us a good baseline on what is needed to get Git a
solid security scanner for the CI/CD pipeline. We are focusing on
making the results useful and removing nags to save as much developer
time as possible when using it, so that you get the security benefits
without significant drawbacks.

Do we have anyone here that is interested in helping the team set up
CodeQL? I'm sure the engineers will have some questions, especially
regarding the current Coverity mess and what needs to improve in order
to make this new setup more usable.

Derek Zimmer
Executive Director
Open Source Technology Improvement Fund

Derek Zimmer
Executive Director
Open Source Technology Improvement Fund


On Tue, May 10, 2022 at 12:43 PM Derek Zimmer <derek@ostif.org> wrote:
>
> Hello all,
>
> Thank you for the conversations about Coverity. After some internal discussions and negotiating with our security partners, we have secured some engineers directly from Github who want to work on CodeQL for Git. They will do the work of getting CodeQL working, do a scan, and then evaluate how much work getting CodeQL into a usable state for Git is by looking at the false positive rate and figuring out what can be muted with rules, and the false negative rate vs Coverity / other current tests and create some custom tests.
>
> This should give us a good baseline on what is needed to get Git a solid security scanner for the CI/CD pipeline. We are focusing on making the results useful and removing nags to save as much developer time as possible when using it, so that you get the security benefits without significant drawbacks.
>
> Do we have anyone here that is interested in helping the team set up CodeQL? I'm sure the engineers will have some questions, especially regarding the current Coverity mess and what needs to improve in order to make this new setup more usable.
>
> Derek Zimmer
> Executive Director
> Open Source Technology Improvement Fund
>
> On Mon, Apr 11, 2022 at 1:49 PM Derek Zimmer <derek@ostif.org> wrote:
>>
>> Hello all,
>>
>> Answers inline + more context
>>
>> > If OSTIF can help us get better support from Coverity (as you can see at
>> > https://github.com/git-for-windows/build-extra/commit/23eea104 I could
>> > have wished for a better experience there), I am all for it!
>>
>> We may be able to convince them to help based on the volume of work that we do with many open source projects. Not helping one open source project may seem like a small loss to them. Not getting recommended to hundreds of high profile projects because of lacking support is different. It is especially concerning that this particular bug likely affects a huge number of customers.
>>
>> > If not, have you considered if you could help us getting a comprehensive
>> > CodeQL coverage instead? Theoretically, CodeQL should be able to do the
>> > same as Coverity, while allowing us to tweak the analysis in a lot more
>> > powerful ways than Coverity (most notably, it should allow us to reduce
>> > the number of false positives rather dramatically).
>>
>> This is absolutely an option, although we may have to petition Google / OpenSSF / the Linux Foundation for a slight increase in funding, as setting up CodeQL from scratch is a much more laborious task than setting up rules for an existing Coverity setup. We absolutely can do this, but we'd have to split it into a second project with separate funding in order to keep the primary work moving forward while we work out the details.
>>
>> If you ultimately think that setting up CodeQL will yield better results overall for Git, I can get started on finding the resources to get it done immediately. (I have a meeting with the Linux Foundation tomorrow.)
>>
>> If we are going to go with CodeQL as a separate project, we can drop the Coverity work from the current SoW/Proposal and proceed with all of the other action items.
>>
>> Let me know your thoughts everyone on what best suits Git here. It sounds to me like CodeQL is the way to go but if there's a compelling argument for Coverity we can explore that.
>>
>> All the best,
>>
>> Derek Zimmer
>> Executive Director
>> Open Source Technology Improvement Fund
>>
>>
>> On Thu, Apr 7, 2022 at 6:58 AM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>>>
>>> Hi Markus,
>>>
>>> On Thu, 7 Apr 2022, Markus Vervier wrote:
>>>
>>> > On 4/6/22 00:17, Johannes Schindelin wrote:
>>> > > On Fri, 1 Apr 2022, Markus Vervier wrote:
>>> > > > X41 is processing the current RfP
>>> > > would you kindly provide a bit more context? This seems to come right out
>>> > > of left field. Is "RfP" a "Request for Proposals"? If so, I am not aware
>>> > > that the git developer team submitted one...
>>> >
>>> > thank you and everyone else for their comments. To clear up the context:
>>> >
>>> > The OSTIF (https://ostif.org) is organizing a security audit for git
>>> > and one of the questions was about Coverity and if the results it gave in the
>>> > past could be verified and/or improved.
>>>
>>> Thank you for the context!
>>>
>>> If OSTIF can help us get better support from Coverity (as you can see at
>>> https://github.com/git-for-windows/build-extra/commit/23eea104 I could
>>> have wished for a better experience there), I am all for it!
>>>
>>> Out of curiosity: are you (or is OSTIF) affiliated with Synopsys somehow?
>>>
>>> If not, have you considered if you could help us getting a comprehensive
>>> CodeQL coverage instead? Theoretically, CodeQL should be able to do the
>>> same as Coverity, while allowing us to tweak the analysis in a lot more
>>> powerful ways than Coverity (most notably, it should allow us to reduce
>>> the number of false positives rather dramatically).
>>>
>>> It is the number of knobs CodeQL allows that has looked too daunting for
>>> me to give it more than a cursory try [*1*].
>>>
>>> Thank you,
>>> Johannes
>>>
>>> Footnote *1*: I had played with CodeQL last year but was called away to a
>>> more pressing project, therefore this is woefully incomplete:
>>> https://github.com/git-for-windows/git/compare/main...dscho:codeql