Is CTI a product without a process?
What's wrong with CTI and what we should focus on

Over the last years, cyber threat intelligence has become an essential component in security practice, with the promise of detecting emerging threats earlier, more effectively and more comprehensively based on customized and always up-to-date information. In response to this need, a flourishing market for cyber threat intelligence has emerged, which antivirus vendors to analysis firms fill with a plethora of dedicated offerings. Gartner estimates that by 2023 more than 12 billion USD will be spend on cyber threat intelligence.

CTI is still a very young field, and not all is rosy. As analysts we are often faced with the problem that intelligence delivered in feeds contains significant wrong information, and a quantitative analysis of 24 feeds we show here that the bulk of indicators of compromise is often outdated and would cause significant collateral damage. The road praising the many success stories we can share about CTI victories is also plastered with major failures, where cyber threat intelligence fell short and did not deliver.

What is however the root cause of these quality issues? In this position paper, available as open access here (PDF version), we take a look at the value propositions of CTI and the challenges the industry faces. This is a quick summary of the main points, for a more nuanced discussion and elaboration see the paper:



• CTI is lacking methodology. Many of the talks and training materials on cyber threat intelligence refer to methodologies such as Structure Analytic Techniques, or frequently cite from Heuer’s Psychology of Intelligence Analysis. While everyone talks about them, we find these however to be little use in practice. A sound methodology is however a prerequisite for consistent and high quality output.
  
  At the moment, much of day-to-day CTI work is driven by raw input data and alerts. We think that the lack of a formalized process in the analysis can lead to analysis paralysis. We as a community need to develop better processes for intelligence analysis, and especially new ones that will fit our needs, the solution is not add more technology.
  
  
• CTI is shared but hardly being used. One of the biggest value propositions of CTI is that sharing threat intelligence will give advance warning to and increase the security posture of the entire community. In practice however, we see that little intelligence is shared, and even less of the intelligence shared is actually applied with success. Indicators of compromise used in the Bundestag hack of 2015 and in the US DNC, as well as the World Anti-Doping Agency were readily available and identical, and thus could have prevented incidents if they had been used. Beyond that, the organizatorial aspects of sharing are even harder as we argue in the paper, such as ambiguities around the traffic light protocol or the issue that ISACs don’t scale well and maintain their quality.
  
  
• CTI is generally of low quality. Today, most cyber threat intelligence is entirely focused on sharing information that is located low on the pyramid of pain. We still mostly talk about IP addresses, domain names and hashes, which frequently lack good annotation to put these items into context. There are innovations such as MITRE ATT&CK that progress us up the ladder, but better techniques, methods and tools are necessarily to fully leverage all the benefits that can be reaped at the top of the pyramid. Otherwise, CTI could become yet a basic activity needed for compliance only.
  


  This move up the ladder is especially important as the data that we do consume today as IoCs is of comparatively low quality. Studies like this show that indicator lists contain too many false positives, the coverage is insufficient, data sources are heavily biased, and indicators are delivered with too high a delay. Part of the problem is that the methodology and sources by which CTI vendors get their data and produce results are intransparent, and developing solid business intelligence of top of this is difficult. The fact that different feeds which claim to assess similar components of the threat landscape have frequently little to no overlap, means that we likely have issues in our data coverage, and our view will be less comprehensive and reliably than we think.
  

 

• CTI vendors are intransparent in their supply. Even though it is a critical ingredient to good intelligence products, we know today relatively little where data obtained from CTI vendors comes from. Not only is the methodology of the vendors largely unknown, but also the way data is collected and processed. This makes it in practice difficult if not impossible for the analysis to judge the relevancy of the data, or account and balance for potential collection biases.
  


• CTI attribution is difficult. A proper evidence-based attribution of cyber threats is difficult. We make this even more difficult by based on our current naming strategy in the community. Fancy Bear, APT-28, Stronium, Sednet, Tsar Team, Grizzly Steppe - all of these refer to the same actor group, and every vendor uses different bloomy actor names in their reporting. This is largely an artifact of marketing, but it severely complicates relating information and drawing conclusions. Developing a common methodology for cyber threat intelligence also means that common naming schemes and taxonomies have to be in place.




While there is much to criticize with the current state of CTI, there are also great opportunities ahead of us. As we argue in our paper, solutions to many of the problems CTI currently faces have been developed in the intelligence community, and we could greatly advance cyber threat intelligence if we adopt tradecraft, best practices but also critical introspection. These are some of the aspects we are working on in the cyber threat intelligence lab; to stay tuned for more updates, please follow us on Twitter.

This was only a brief summary of a few key points, for a more nuanced and complete discussion on the state of CTI see the full article here.