Distinguish false positives from true positives: Ask any security operations center professional and they’ll tell you it’s one of the most challenging aspects of developing a detection and response program.
As the volume of threats continues to increase, having an effective approach to measuring and analyzing this type of performance data has become increasingly critical to an organization’s detection and response program. At the Black Hat Asia conference in Singapore on Friday, Airbnb senior engineer Allyn Stott encouraged security professionals to reconsider how they use such metrics in their detection and response programs, a topic addressed last year Black Hat Europe.
“At the end of that talk, a lot of the feedback I got was, ‘This is great, but we really want to know how we can improve the metrics,’” Stott tells Dark Reading. “This is an area where I have seen a lot of struggle.”
The importance of metrics
Metrics are critical to evaluating the effectiveness of a detection and response program because they drive improvement, reduce the impact of threats, and validate investments by demonstrating how the program reduces risk to the business, Stott says.
“Metrics help us communicate what we do and why people should care,” Stott says. “This is especially important for detection and response because it is very difficult to understand from a business perspective.”
The most critical area for providing effective metrics is alert volume: “Every security operations center I’ve worked in or ever set foot in, that’s their primary metric,” Stott says.
Knowing how many alerts are coming is important but, in and of itself, is still not enough, he adds.
“The question is always, ‘How many alerts are we seeing?’” Stott says. “And that doesn’t tell you anything. I mean, it tells you how many alerts the organization gets. But it doesn’t really tell you whether your detection and response program is detecting more things.”
Leveraging metrics effectively can be complex and labor-intensive, which adds to the challenge of effectively measuring threat data, Stott says. He acknowledges that he has made his share of mistakes when it comes to engineering parameters for evaluating the effectiveness of security operations.
As an engineer, Stott regularly evaluates the effectiveness of the research he conducts and the tools he uses, striving to obtain accurate true and false positive rates for detected threats. The challenge for him and most security professionals is connecting that information to the business.
Correct implementation of frameworks is critical
One of his biggest mistakes was his approach of focusing too much on MITER ATT&CK framework. While Stott says he believes it provides critical details about threat actors’ different techniques and activities and that organizations should use it, that doesn’t mean they should apply it to everything.
“Each technique can have 10, 15, 20 or 100 different variations,” he says. “And so to have 100% coverage is kind of a crazy feat.”
In addition to MITER ATT&CK, Stott recommends using those from the SANS Institute Hunting Maturity Model (HMM)which helps describe an organization’s existing threat hunting capability and provides a blueprint for improving it.
“It gives you the ability to, as a metric, say where you are as far as your maturity today and how the investments you plan to make or the projects you plan to do will increase your maturity,” Stott says.
He also recommends using those from the Security Institute SABER frameworkwhich provides risk management and security performance metrics validated with third-party certifications.
“Rather than testing the entire MITER ATT&CK framework, you’re actually working through a list of prioritized techniques, which includes using MITER ATT&CK as a tool,” he says. “This way, you not only consider threat intelligence, but also security incidents and threats that could pose critical risks to your organization.”
Using these metrics guidelines requires buy-in from CISOs, as it means gaining organizational buy-in to these different maturity models. However, it tends to be driven by a bottom-up approach, where threat intelligence engineers are the prime movers.