After 16+ years in application security, one of the questions I have consistently heard asked is about false positives. Questions like “what is your false-positive percentage” or “how do you deal with so many false positives produced by this tool or that tool.”
False-positive is a term that has become the black eye of application security scanning technologies. But what false positive means is a very subjective conversation because every organization, team, or individual has their view of what risk is.
A very simplistic definition of a false positive is something that is reported as a security issue by an AST technology that is not a security issue. It seems black and white but let’s go in a bit deeper.
AST scanning technologies can only understand the context of what they are scanning. The target can be code, frameworks, a running application, plus many other things. AST scanning technologies are only as good as what they are configured to scan.
The technology cannot understand something like a security compensating control outside of the scan target, which would make an identified security risk a false positive.
AST Scanning technologies are all configurable to look for a specific list of security risks. You can turn every rule, checker, or query on or off based on your company’s security posture to find the things that concern you the most.
Configuration is a best practice as it narrows the result set to focus on the type of risk you are most concerned about, but this creates a problem. These configurations can be too granular and introduce the dreaded False Negative to limit the feared false positive.
Is it a False Positive?
Most of the issues reported by AST scanning technologies do have security risks. You just don’t care about the risk, or they do not fit your organization’s risk profile.
A few examples of issues being classified as false positives are a compensating control or security protection outside the scanned entity, like a WAF, or something specific to your organization, like a custom validation framework. The AST scanning tools do not have enough information to understand the whole picture.
These, in my opinion, are not false positives but are security issues that can be marked as a non-issue because of additional organizational tribal knowledge.
My Definition of a False Positive
My personal definition of a false positive is a bit different than the standard industry definition of a false positive. To me, a false positive is a bug in the AST scanning tool. The AST tools report an issue that has zero security risk.
A simplistic example would be an AST tool that reports a database connection is never programmatically closed, but in fact, there is an explicit call to close the database connection. This is a bug in the AST tool’s rule, checker, or query and a false positive and a bug.
I thought this blog was about False Negatives.
Ok, I hear you. Sorry, but I had to set the table before we talked about False Negatives. People are so hung up on false positive ratios and percentages that they become way too tunnel-visioned to limit them. They do not think about what they could be missing.
Organizations don’t know what they don’t know. In my opinion, false negatives are much worse than false positives. It is pretty much the same story if you read an article about a new hack or data breach.
The company that got hacked did not know about the issue and was shocked that it happened. The fact that they did not know about the problem is a false negative. False negatives cost organizations tons of money in regulatory fines, damaged reputations, and compliance sanctions.
Just look at what happened to TalkTalk after their data breach in 2015. They did not know certain web pages were still publicly facing, and a SQL Injection caused a massive loss of PII data and a significant fine.
They did not know about this false positive, which turned into a nightmare for them.
I can’t say this enough, but organizations need to understand their application architecture to be truly secure. Not what they think it looks like. There needs to be a systematic change from looking at application security from an out of context vulnerability lens to an architectural risk lens.
If this change does not happen, false negatives will continue to be a significant issue for all organizations.
Bionic has created a next-generation platform that allows organizations to understand their applications architecture from the deployed artifacts to address the problem of false negatives.