Return to site

Introducing the Threat Hunting Automation Maturity Model

· threat hunting,Automation,Information Security

The discussion around “automation” in Threat Hunting is usually defined as something tautological. The definition of threat hunting on several circles accounts for “human activity to find badness that the automated products missed”. The issue with this definition is that according to it, as soon as something is automated, it ceases to be Threat Hunting.

This argument is rooted on the belief that the only kind of automation that exists is “signature-based” or similarly simplistic forms of rule-based automation. Under this view, humans would use intuition and knowledge to learn from the existing signatures and unusual markers in organization log data to supplement a static signature matching approach.

Now, it is obvious that a signature-based strategy does not scale with the number of threats available, so the idea of doing something “smarter” to complement that is a good one. However, computer science is long past the point in which it was believed that humans alone could learn from experience to make complex multivariate decisions with accuracy.

In order to inform this discussion, Niddel proposes a Hunting Automation Maturity Model with four distinct levels in which to classify automation activities:


  • First Order: Indicator Matching Automation -  The vast majority of hunting automation solutions work on this tier.  Consists mostly of a signature match, such as matching a list of file hashes to the processes running on a machine, or an IP address search on network logs. However, this is an incomplete strategy, both prone to extensive false positives present in badly vetted lists and false negatives because those records will naturally be incomplete.
  • Second Order: Higher Level Context Analysis and Enrichment - A solution on this tier is capable of calculating statistical summaries and other context-based enrichments to give additional information to a Threat Hunting analyst. One example of this would be to evaluate individual hunting pivoting points such as what datacenter an IP address is hosted at, or a domain's WHOIS information. By assessing them, you can assign a maliciousness level, based on how many malicious and benign samples the system came across aggregated by pivoting point. A system can then single out on all the entries that are related to the high maliciousness pivoting points, and even provide context information to what they are linked to based on the connections to known malicious samples.
  • Third Order: Multivariate Decision Making Engine - The challenge of this tier is around the aggregated experience from a human analyst. Out of the First Order and Second Order matches or evaluations associated with a group of logs or events, how do we prioritize which one of those are the most relevant? Most SIEMs and Security Analytics will try to achieve that via a scoring or weighting engine. However, these do not take into consideration particular input or conditions from customers and have little reconfigurability or transparency on how to tweak them. A system operating in the Third Order would be able to decide which variables described as First Order or Second Order are the most relevant to determine how to prioritize an incident. Purposefully designed supervised machine learning models are a natural fit for developing Third Order engines.
  • Fourth Order: Curiosity and New Techniques Development - For an automation at this Order, a system would evaluate failures or successes from human feedback to make the system decide to add new First Order matching capabilities or figure out new Second Order context or statistic analysis to aggregate new capabilities to a Third Order engine. The system would be actively looking for new kinds of data to analyze based on what it has available. That would be analogous to writing a new playbook for a Threat Hunting team in response to a newly uncovered threat, and we firmly believe that this tier is exclusive to the human domain as of now.

A lot of the frustration we perceive on the market comes from the fact that marketing materials advertise Artificial Intelligence capabilities that seem as magical as the Fourth Order described above, while they can barely deliver First Order results.

Niddel Magnet is the only system on the market today that implements the functionalities on the Third, Second and First Orders of this maturity model for end-to-end hunting automation.

You can learn more about this maturity model from the talk presented by our Chief Data Scientist, Mr. Alex Pinto, at the SANS Threat Hunting Summit of 2017.

All Posts

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OKSubscriptions powered by Strikingly