Developing Safe AGI: Assessing the New Cybersecurity Features of Advanced AI

Artificial intelligence (AI) has long been a cornerstone of cybersecurity. From malware detection to network traffic analysis, predictive machine learning models and other narrow AI applications have been integral to cybersecurity for decades. As we move closer to artificial general intelligence (AGI), AI’s capacity to automate defenses and address vulnerabilities becomes even more significant. However, to fully leverage these advantages, it is crucial to understand and mitigate the risks associated with the potential misuse of advanced AI in cyberattacks.

Our new framework for evaluating the emerging offensive cyber capabilities of AI enables us to do just that. It represents the most comprehensive evaluation to date, covering every phase of the cyberattack lifecycle, addressing a wide array of threat types, and grounded in real-world data. This framework allows cybersecurity experts to discern which defenses are necessary and prioritize them effectively, ensuring preparedness against the sophisticated cyberattacks that could exploit AI.

Building a comprehensive benchmark

Our updated Frontier Safety Framework acknowledges that advanced AI models could automate and expedite cyberattacks, potentially making them more cost-effective for attackers. This shift increases the risks associated with large-scale attacks.

To remain proactive against the rising threat of AI-driven cyberattacks, we have adapted established cybersecurity evaluation frameworks to evaluate threats throughout the complete cyberattack lifecycle—from reconnaissance to execution. While these traditional frameworks have served us well, they weren’t designed to address AI’s role in facilitating attacks. Our approach fills this gap by identifying where AI could enhance the speed, affordability, or ease of an attack—such as enabling fully automated cyberattacks.

We analyzed over 12,000 real-world instances of AI usage in cyberattacks across 20 countries, utilizing data from various cybersecurity intelligence sources. This analysis provided insights into common patterns of attack. From this data, we curated a set of seven archetypical attack categories—including phishing, malware, and denial-of-service attacks—and pinpointed critical bottleneck stages in the cyberattack chain where AI could significantly alter the traditional costs associated with an attack. By concentrating evaluations on these bottlenecks, defenders can effectively allocate their security resources.

Lastly, we developed an offensive cyber capability benchmark to thoroughly assess the cybersecurity strengths and weaknesses of frontier AI models. This benchmark comprises 50 challenges that encompass the entire attack lifecycle, addressing aspects like intelligence gathering, vulnerability exploitation, and malware development. Our goal is to empower defenders to create targeted mitigations and facilitate simulations of AI-driven attacks as part of red teaming exercises.

Insights from early evaluations

Our preliminary evaluations using this benchmark indicate that current AI models, when considered in isolation, are unlikely to provide breakthrough capabilities for threat actors. However, as frontier AI continues to advance, the nature of possible cyberattacks will evolve, highlighting the need for ongoing enhancements in defensive strategies.

We also discovered that existing AI cybersecurity evaluations frequently overlook significant elements of cyberattacks—such as evasion techniques used by attackers to conceal their presence and persistence tactics that ensure long-term access to compromised systems. These areas are where AI-driven approaches can be particularly effective. Our framework addresses this concern by evaluating how AI might reduce the barriers to success in these critical attack components.

Empowering the cybersecurity community

As AI systems continue to grow, their ability to automate and enhance cybersecurity has the potential to transform how defenders anticipate and address threats. Our cybersecurity evaluation framework is designed to facilitate this transition by providing a clear view of how AI might also be misused, and where current cyber protections may be lacking. By emphasizing these emerging risks, this framework and benchmark will assist cybersecurity teams in fortifying their defenses and staying ahead of rapidly evolving threats.