Endpoint Testing via MITRE ATT&CK

In many organizations, endpoints account for most of the attack surface; therefore, logically, endpoint security remains a priority. Choosing the right solution is paramount and the decision unique to the goals and objectives of an organization. However, for all, the focus is on obtaining the maximum value from the chosen endpoint security platform by operationalizing capabilities. Forward-thinking security organizations can boost the effectiveness of their security program by using endpoint telemetry data to improve detection and response, refine threat hunting, and by integrating sensor data feeds into complementary security solutions.

Background

Information security practitioners have often lamented that industry terms have been often left to individual interpretation as practitioners provided guidance without globally-accepted definitions.

This started to shift with Lockheed Martin’s 2011 release of the Cyber Kill Chain® -- the industry began to find common ground. Its terminology was subsequently widely adopted and started to help move the industry to a common lexicon. 

Then in 2013, MITRE released Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) to catalog post-exploit techniques on enterprise systems. The ATT&CK knowledge base consists of 12 tactical categories and provides a standardized method to describe the activities of threat actors. Since the release of ATT&CK, many security solution providers and internal security organizations have adopted the information framework. In fact, many solution providers are using the ATT&CK terminology to enrich their telemetry data. Security organizations are modeling components of their security program on ATT&CK and are looking for methods to locate gaps in cyber program control coverage.

ATT&CK enterprise tactics provide:

• Excellent post-exploit knowledge base that gets everyone speaking the same language
• Granularity that far exceeds kill-chain descriptions
• A knowledge base that excels at classifying malicious behaviors, but which should not be viewed as a coverage checklist for all security controls

Note: Not all possible techniques are documented, nor will they ever be.

For this year’s evaluation, nine of our partners provided their solution for hands-on testing. The evaluation emphasized detection of atomic ATT&CK techniques, in-platform threat hunting and API instrumentation. In contrast to MITRE’s own recent series of endpoint security evaluations, we placed a focus on varying technique execution verses modeling a threat actor. 

Our testing goals

• Identify which preventative or detective component of the solution would trigger on a given technique if any
• Determine the extent of telemetry enrichment the vendor had included
• Assess the usefulness of the product’s interface for threat hunting
• Validate API functionality for alerting, telemetry ingestion and host quarantine
• Test the effectiveness of any anti-tampering mechanisms used to protect endpoint agents

What we discovered

We grouped a series of techniques within the same tactical category together and staggered the automated execution of each with one-minute intervals. The tactics employed in the testing were:

• Execution
• Persistence
• Privilege Escalation
• Defense Evasion
• Credential Access
• Discovery 
• Lateral Movement
• Collection
• Command and Control
• Exfiltration 

Testing features

• For the efficacy testing phase of this year’s evaluation, each test case was mapped to a specific ATT&CK technique. 
• Each test focused on a single ATT&CK technique; tests were performed independently of each other with no dependencies on prior techniques. 
• Using Verodin’s Security Instrumentation Platform (recently acquired by FireEye), we executed 122 individual tests, incorporating 80 techniques across 10 of ATT&CK’s tactical categories to determine how the endpoint security products would behave. 

What we discovered

Endpoint security products will react differently to the same technique with the same objective when the technique is executed in an alternate manner. This was a common theme seen across all tactic categories and across all products during testing. 

Highlights:

• Persistence - Only two out of 12 test cases were prevented by more than half products tested.
• Discovery - Many techniques resembled everyday software and user behavior. Most endpoint products do not capture command arguments executed from a PowerShell terminal. 
  • Only one product natively captured all PowerShell terminal commands and seven out of nine products were blind to the commands executed with PowerShell.
• Defense Evasion – The vast majority of attacks were prevented or alerted on by five or more products.
• Lateral Movement – Remains a weak area for endpoint security solutions. 
• Collection - Many techniques resembled everyday software and user behavior.
• Command and control – A single solution detected the simulated malware communications and prevented execution of the tests. Detection rates fell off dramatically for the remainder of the solutions; most at least logged the IP and port used. 

Takeaways and things to remember

• When aligning an endpoint security program to MITRE’s ATT&CK framework, organizations need to have realistic expectations of endpoint product capabilities. 
• No product has coverage for all methods of executing each technique, nor will they ever, because of the myriad of ways that a technique could be executed. 
• Beware of percent-of-coverage claims, as that is a deceptive metric. This is due to the variety of ways a test could be constructed. 
• In our testing, several techniques were executed in two or more ways. Different outcomes always occurred for tests executed in multiple ways.

To get the full analysis, download our full whitepaper Endpoint Security Evaluation MITRE ATT&CK Edition.