How secure is software X?

["David Litchfield" <davidl@xxxxxxxxxxxxxxx>]

How secure is software X?

At least as secure as Vulnerability Assessment Assurance Level P; or Q or R. 
Well, that's what I think we should be able to say. What we need is an open 
standard, that has been agreed upon by recognized experts, against which the 
absence of software security vulnerability can be measured - something which 
improves upon the failings of the Common Criteria. Let's choose web server 
software as an example. When looking for flaws in a new piece of web server 
software there are a bunch of well known checks that one would throw at it 
first. Try directory traversal attacks and the several variations. Try 
overflowing the request method, the URI, the query string, the host header 
field and so on. Try cross site scripting attacks in server error pages and 
file not found messages. As I said, there's a bunch of checks and I've 
mentioned but a few. If these were all written down and labelled with as a 
"standard" then one could say that web server software X is at least as 
secure as the standard - providing of course the server stands up.

For products that are based upon RFCs it would be trivial to write a simple 
criteria that tests every aspect of the software as per the RFCs. This would 
be called Vulnerability Assessment Assurance Level: Protocol. If a bit of 
software was accredited at VAAL:Protocol  then it would given a level of 
assurance that it at least stood up to those attacks.

Not all products are RFC compliant however. Sticking with web servers, one 
bit of software might have a bespoke request method of "FOOBAR". This opens 
up a whole new attack surface that's not covered by the VAAL:Protocol 
standard. There are two aspects to this. Anyone with a firewall capable of 
blocking non-RFC compliant requests could configure it to do so - thus 
closing off the attack surface - from the outside at least. As far as the 
standards go however - you'd have to introduce criteria to cover that 
specific functionality. And what about different application environments 
running on top of the web server? And what about more complex products such 
as database servers? I suppose at a minimum for DB software you could at 
least have a standard that simply checks if the server falls to a long 
username or password buffer overflow attempt and then fuzz SQL-92 language 
elements. It certainly makes standardization much more difficult but I think 
by no means impossible.

Clearly, what is _easy_ is writing and agreeing upon a VAAL:Protocol 
standard for many different types of servers. You could then be assured that 
any server that passes is at least as secure as VAAL:Protocol and for those 
looking for more "comfort" then they can at least block non-RFC compliant 
traffic.

Having had a chat with Steve Christey about this earlier today I know there 
are other people thinking along the same lines and I bet there are more 
projects out there being worked on that are attempting to achieve the same 
thing. If anyone is currently working on this stuff or would like to get 
involved in thrashing out some ideas then please mail me - I'd love to hear 
from you.

Cheers,
David Litchfield
http://www.databasesecurity.com/
http://www.ngssoftware.com/