February 28 How to interpret MITER Engenuity ATT&CK® assessments: Enterprise
And why is this assessment essential for testing cybersecurity vendors?
– George Tubindirector of product strategy, Cynet
Boston, Massachusetts – February 28, 2024
In-depth, independent testing is a vital resource for cybersecurity leaders and their teams as they evaluate vendors’ ability to protect against increasingly sophisticated threats to their organizations. And perhaps no assessment is more widely trusted than the annual assessment MITER Engenuity ATT&CK Assessments: Enterprise.
This assessment is essential for testing vendors because it is virtually impossible to evaluate cybersecurity vendors based on their own performance. In addition to vendor reference checks and proof-of-value (POV) assessments – a live test – in their environment, MITER Engenuity results add additional objective information to comprehensively evaluate cybersecurity vendors.
In this article, we will introduce MITER Engenuity’s latest methodology for testing security vendors against real-world threats, offer our interpretation of the results, and identify key takeaways from evaluating Cynet’s all-in-one security solution.
How does MITER Engenuity test vendors during evaluation?
The MITER Engenuity ATT&CK assessment is performed by MITER Engenuity and tests endpoint protection solutions against a simulated attack sequence based on real-world approaches taken by well-known advanced persistent threat (APT) groups. The MITER Engenuity ATT&CK: Enterprise evaluations tested 29 vendor solutions by emulating attack sequences from Turla, a sophisticated Russia-based threat group known to have infected victims in more than 45 countries.
An important caveat is that MITER does not rank or rate vendor results. Instead, raw test data is published with some basic online comparison tools. Buyers then use this data to evaluate suppliers based on their organization’s unique priorities and needs. The participating providers’ interpretations of the results are just that: their interpretations.
So how do you interpret the results?
This is a great question that a lot of people are asking right now. THE MITER Engenuity ATT&CK Assessments: Business Results are not presented in a format that many of us are used to digesting (looking at you, magic graph with quadrants).
And independent researchers often declare “winners” to ease the cognitive load of determining which vendors perform best. In this case, identifying the “best” supplier is subjective. Which, if you don’t know what to look for, can seem complicated if you’re already frustrated trying to evaluate which security vendor is best for your organization.
With these disclaimers out of the way, let’s now look at the results themselves to compare how the participating providers performed against Turla.
Summary of MITER Engenuity ATT&CK Results
The following tables show Cynet’s analysis and calculation of all vendor MITER Engenuity ATT&CK assessments: Enterprise test results for the most important metrics: overall visibility, detection accuracy, and overall performance. There are many other ways to look at MITER results, but we consider these to be the most indicative of a solution’s ability to detect threats.
Overall visibility is the total number of attack steps detected in the 143 substeps. Cynet defines detection quality as the percentage of attack substeps that include “analytical detections – those that identify the tactic (why an activity may occur) or technique (both why and how the technique occurs ).
Additionally, it is important to review the performance of each solution before the vendor adjusts configuration settings due to the absence of a threat. MITER allows vendors to reconfigure their systems to attempt to detect threats they missed or to improve the information they provide for detection. In the real world, we don’t have the luxury of reconfiguring our systems due to missed or poor detection. The most realistic measure is therefore to detect before configuration changes are implemented.
How is Cynet?
Based on Cynet’s analysis, our team is proud of our performance against Turla in the MITER Engenuity ATT&CK 2023: Enterprise assessments, outperforming the majority of vendors in several key areas. Here are our key takeaways:
- Cynet delivered 100 percent detection (19 out of 19 attack steps) without CONFIGURATION CHANGE
- Cynet delivered 100 percent visibility (143 out of 143 attack sub-steps) without CONFIGURATION CHANGE
- Cynet delivered 100 per cent Analytical coverage (143 out of 143 detections) without any configuration changes
- Cynet delivered 100 per cent Real-time detections (0 delays out of 143 detections)
Let’s take a deeper look at Cynet’s analysis of some of the results.
Cynet’s all-in-one security solution performed well when evaluating visibility and detection quality. This analysis illustrates how effective a solution is in detecting threats and provides the context needed to make detections actionable. Missed detections are an invitation to a breach, while low-quality detections create unnecessary work for security analysts or may result in the alert being ignored, which, again, is an invitation to a breach .
Cynet provided 100% visibility and perfectly detected each of the 143 attack steps without any configuration changes. The following chart shows the percentage of detections in the 143 attack substeps before vendors implement configuration changes. Cynet performed as well as two very large, well-known security companies, despite being only a fraction of their size and much better than some of the biggest names in cybersecurity.
Cynet provided 100% analytical coverage of all 143 attack steps without any configuration changes. The following chart shows the percentage of detections containing important general, tactical, or technical information during the 143 substages of the attack, always before implementing configuration changes. Cynet performed as well as Palo Alto Networks, a $115 billion publicly traded company with 50 times more employees and far better than many established, publicly traded brands.
Still have questions ?
In this on-demand webinar, Aviad Hasnis, CTO of Cynet, and Tom Field, editorial vice president of ISMG, review the most recent results from MITER ATT&CK and share expert tips for cybersecurity leaders to find the right vendor best meets the specific needs of their organization. They also analyze Cynet’s performance during testing and identify opportunities to advance your team’s goals with the all-in-one security solution.
– George Tubin is the Director of Product Marketing at Cynet Security
About Cynet
Cynet has developed the first end-to-end, natively automated cybersecurity platform, backed by 24/7 service, purpose-built for smaller security teams, paving the way for a new era of cybersecurity, making protection of organizations easy and stress-free. With thousands of customers, Cynet enables organizations of all sizes to put their cybersecurity on autopilot and focus their limited resources on managing security rather than running it.