Don’t Take MITRE ATT&CK Results as Gospel

MITRE is an unbiased and respected organization that performs a valuable service to the cybersecurity community. The MITRE ATT&CK evaluation is an industry standard, and the industry can use all the help it can get to identify the tactics and techniques employed by cybercriminals. (See The Cyber Threat Landscape for 2022 Darkens.) MITRE helps unite efforts by governmental organizations, academics, and vendors to develop strong defense mechanisms. Even so, should cybersecurity leaders take the results provided in the recent MITRE ATT&CK Engenuity tests as gospel?

My view is that while the tests have merit, they only offer part of the picture. Caution is warranted when evaluating each vendor’s interpretation of the results. Organizations seeking to improve their cybersecurity posture may well want to review the raw results, but using a vendor’s analysis as the sole basis for making a security solution purchase is likely unwise.

There’s one overriding reason for this, which I’ll get to. But let’s start by examining the raw results based on what MITRE tested, which vendors participated, and how they fared.

Details of the MITRE ATT&CK Engenuity Evaluation

The MITRE ATT&CK Engenuity tests for the Wizard Spider and Sandworm Edition evaluated the detection and prevention capabilities for thirty endpoint detection and response (EDR) vendors who agreed to have their products scrutinized. Wizard Spider is a criminal group that has been conducting ransomware campaigns since at least 2018. Sandworm is a Russian threat group active since 2009. Sandworm was linked to 2015 and 2016 attacks on Ukrainian electrical groups, as well as to the NotPetya attacks in 2017.

The MITRE ATT&CK evaluation emulated the ways these two groups use the Data Encrypted for Impact technique. They were intended to determine how well vendors detected malware these organizations used and prevented attacks from persisting. Evaluations were made within Microsoft Azure.

Vendor detection rate percentages were as high as 100 percent and as low as 57 percent. Prevention rates ranged from a high of almost 90 percent to as low as 3.67 percent. Eight of the vendors, or ~25 percent, opted not to participate in the prevention tests. This shows a very significant range of results where almost every participant claimed to perform at or near “the best.”

It's also important to note that many of the tests utilized several attack process steps. Some vendors gave themselves higher scores for stopping an attack earlier in the process. Other vendors ignored this viewpoint to obtain a better ranking. In all cases, many of these tests relied on known signature detection or prevention capabilities. But studies show 94 percent of ransomware attacks have moved beyond disk or file (signature)-based attacks and are now polymorphic. As such, they change tactics (morph) every 15 to 20 seconds. They are also mostly fileless and in-memory attacks. Nearly all EDR vendors claim to prevent these types of attacks. But MITRE and other test results validate that most are not very effective.  

Vendor Motivations

It’s understandable why a vendor would be quick to tout a MITRE ATT&CK evaluation result as evidence of their superiority over the competition. After all, wouldn’t you prefer to buy a tool that scores high on MITRE tests rather than one that had only mediocre results? And wouldn’t you expect the highest scoring solution to be the smartest choice? While this might simplify the process of purchasing cybersecurity solutions, it’s not a guarantee for effective protection. It takes a lot of filtering and a deep examination of vendor claims to understand what’s real—and what’s not. Thankfully, there are reliable sources that can provide more illumination.

Gartner Cautions

In Select Endpoint Security Products with Confidence, Gartner urges caution about taking the results of these tests too seriously:

“Conditions under which tests are conducted may not be the same as in a typical enterprise organization,” Gartner pointed out. “Equally, security vendors will have configured their solutions to adapt to the tests to get the best results possible and no tests evaluate solutions using all the attack methods likely to be encountered.”

Their warning applied to all tests, with specific guidance regarding MITRE ATT&CK evaluation: “Focus on those controls that relate to identifiable and prioritized risks or threats that are already familiar. Check a vendor’s test results for these specific controls and ask any vendors who have not submitted their products for such testing to give evidence of how they too provide capabilities.”

Real World Evaluations

Respected academic publication the Journal of Cybersecurity and Privacy published an even more skeptical analysis of how leading EDR solutions are failing to detect and prevent advanced attacks. In An Empirical Assessment of Endpoint Detection and Response Systems against Advanced Persistent Threats Attack Vectors, researchers published the results of comprehensive tests using typical Cobalt Strike MITRE tactics and techniques.

The researchers stated, “Quite alarmingly, we illustrate that no EDR can effectively detect and prevent the four attack vectors we deployed,” the authors wrote. “Despite the significant advances in cybersecurity, an organization needs to deploy a wide array of tools to remain secure and not solely depend on one solution.” 

The researchers also said it’s critical for organizations to recognize the valuable role humans have in effective cybersecurity, and to ensure adequate funding for cybersecurity. Note, in these more real-world tests, vendor solutions were tested using Standard/Moderate detection alert settings. In the MITRE Engenuity tests, all vendors used Ultra Aggressive settings. The researchers’ approach is more realistic. Few organizations can employ an aggressive EDR setting as it causes too much “noise.” Higher settings result in a flood of false positive alerts that require IT support tickets and analyst investigations. The cost and time requirements are far too high for almost every organization.

Evaluating the Mitre ATT&CK Evaluation

Thirty vendors participated in the MITRE Engenuity tests. Eight decided not to have their preventative abilities evaluated, only their detection capabilities. Of those that agreed to participate in prevention tests, many had mediocre scores, even if they had strong detection scores. For example, 83 percent on detection but only 34 percent on prevention.

You might question the value of strong detection scores as a noteworthy accomplishment if prevention scores are ignored. Vendors should prevent attacks. Malware typically creates over a hundred artifacts. So once an attack has been detected, these artifacts may have already spread if the attack was not prevented. It’s important to get an alert, but it’s far more important to not get an infection in the first place. 

Prevention is Better Than the Cure

That’s why Morphisec created its Moving Target Defense (MTD) technology. MTD creates a dynamic attack surface threat actors can’t penetrate, causing them to abort attacks before EDR might detect them. MTD can protect almost all endpoints. And its lightweight agents are especially optimal for virtual desktop infrastructure (VDI), and server environments where EDR agents cause performance issues. That’s why Gartner recently named MTD as a key technology for improving security:

“Moving Target Defense prevents unknown and zero-day attacks by using system polymorphism to hide application and operating system targets from adversaries in an unpredictable manner leading to a dramatically reduced attack surface and lower security operational costs.”

Virtual Patching = Critical Defense

MTD offers another crucial benefit no EDR solution does: virtual patching. More than 60 percent of ransomware attacks result from exploits against known vulnerabilities. Security breaches or audit failures could have been prevented at 80 percent of firms that were too slow to patch. (Related: How to Nail Your NIST Cybersecurity Audit.) This is an age-old problem that typical vulnerability management and EDR solutions have failed to resolve. For example, over a month passed before a patch was available for the Log4j vulnerability. And most firms took another month or longer to implement patches. MTD can provide a seamless, instant virtual patch that is more effective, efficient, and affordable than traditional web application firewall (WAF) or intrusion prevention system (IPS) solutions. This ensures you’re protected before a vulnerability is exposed and until a patch is provided and installed.  

Morphisec security experts can help you understand how MTD provides virtual patches and how it augments EDR. If you’re using Microsoft Defender for Endpoint, integrating with Morphisec prevents polymorphic advanced persistent threats (APTs), and gives you additional security intelligence to improve threat hunting. MTD guards against the sophisticated attacks MITRE tested by providing multilayer defense in depth. If you’d like to see MTD in action, schedule a demo today.