Re: [Full-disclosure] How secure is software X?

["Michael Silk" <michaelslists@xxxxxxxxx>]

On 5/12/06, David Litchfield <davidl@xxxxxxxxxxxxxxx> wrote:
> How secure is software X?
>
> At least as secure as Vulnerability Assessment Assurance Level P; or Q or R.
> Well, that's what I think we should be able to say. What we need is an open
> standard, that has been agreed upon by recognized experts, against which the
> absence of software security vulnerability can be measured - something which
> improves upon the failings of the Common Criteria. Let's choose web server
> software as an example. When looking for flaws in a new piece of web server
> software there are a bunch of well known checks that one would throw at it
> first. Try directory traversal attacks and the several variations. Try
> overflowing the request method, the URI, the query string, the host header
> field and so on. Try cross site scripting attacks in server error pages and
> file not found messages. As I said, there's a bunch of checks and I've
> mentioned but a few. If these were all written down and labelled with as a
> "standard" then one could say that web server software X is at least as
> secure as the standard - providing of course the server stands up.
>
> For products that are based upon RFCs it would be trivial to write a simple
> criteria that tests every aspect of the software as per the RFCs. This would
> be called Vulnerability Assessment Assurance Level: Protocol. If a bit of
> software was accredited at VAAL:Protocol  then it would given a level of
> assurance that it at least stood up to those attacks.
>
> Not all products are RFC compliant however. Sticking with web servers, one
> bit of software might have a bespoke request method of "FOOBAR". This opens
> up a whole new attack surface that's not covered by the VAAL:Protocol
> standard. There are two aspects to this. Anyone with a firewall capable of
> blocking non-RFC compliant requests could configure it to do so - thus
> closing off the attack surface - from the outside at least. As far as the
> standards go however - you'd have to introduce criteria to cover that
> specific functionality. And what about different application environments
> running on top of the web server? And what about more complex products such
> as database servers? I suppose at a minimum for DB software you could at
> least have a standard that simply checks if the server falls to a long
> username or password buffer overflow attempt and then fuzz SQL-92 language
> elements. It certainly makes standardization much more difficult but I think
> by no means impossible.
>
> Clearly, what is _easy_ is writing and agreeing upon a VAAL:Protocol
> standard for many different types of servers. You could then be assured that
> any server that passes is at least as secure as VAAL:Protocol and for those
> looking for more "comfort" then they can at least block non-RFC compliant
> traffic.
>
> Having had a chat with Steve Christey about this earlier today I know there
> are other people thinking along the same lines and I bet there are more
> projects out there being worked on that are attempting to achieve the same
> thing. If anyone is currently working on this stuff or would like to get
> involved in thrashing out some ideas then please mail me - I'd love to hear
> from you.

why do we need this?

you're referring to what already takes place commercially.

"hi i want a security assessment".

who's going to do these assessments for free? who confirms that the
people doing the assessment know what they are doing?

"Customer: I was hacked .." -> me: -> "David Litchfield told me it was
secure, blame him" -> "David Litchfield: Oh no, our VAAL is just a
guide." -> "Customer: So why the hell do I care about it then?"

Guides for people to use are okay (hello OWASP Guide, and others) but
all your trying to start is a non-commercial free security assessment
service.

... ?

-- Michael