Re: How secure is software X?

[Tim Newsham <newsham@xxxxxxxx>]

> At least as secure as Vulnerability Assessment Assurance Level P; or Q or R. 
> Well, that's what I think we should be able to say. What we need is an open 
> standard, that has been agreed upon by recognized experts, against which the 
> absence of software security vulnerability can be measured - something which 
> improves upon the failings of the Common Criteria.

What about a completely different approach, as chosen by the Sardonix
project?  Keep track of who has tested a particular product and what
they have found.  Keep track of the ability of testers to find things
and the number of things that are missed.  Combine these metrics into
some level of assurance and some security rating....

"5 very good security reviewers have done extensive testing of this 
product and found a small number of vulnerabilities."

"2 reviewers made a cursory pass over the code and identified a few 
issues"

"100 reviewers found many bugs in this product over the last 12 mos, and 
the number of vulns seems to be coming down very slowly with each new 
revision"

These sort of statements can be made more formal, and each carries a lot 
of useful information about security and confidence.  Of course its only 
as good as participation. I'm not sure the level of information sharing 
required to make this really work is present in the security community.

Tim Newsham
http://www.lava.net/~newsham/