Spy or Ally: Unpacking the Cybersecurity Risks of AI Agent-based Systems

In recent months AI innovators have rolled out both exciting and concerning AI capabilities. Early this year Microsoft Copilot released new capabilities that while helpful for workflow optimization, heightened the risk of sensitive data exposure and data privacy violations. 

Many companies are running the commercial AI development race. However, in the rush to create and bring new capabilities to market significant security risks are overlooked pre-release. 

Anthropic is an AI safety and research company that strives to build safer AI systems. This month the Amazon-backed company announced the release of a new version of its Claude 3.5 Sonnet model, which can do many tasks, including inputting keystrokes and mouse clicks that allow it to interact with virtually any application used on a machine. Its goal (according to Anthropic) is to evolve beyond AI assistants like Microsoft Copilot and introduce AI agents that can perform more complex tasks without human intervention. 

The Claude 3.5 Sonnet model embeds itself in one’s system, with the idea being that upon prompt, Claude can use multiple tools and applications to execute a command, like researching a trip or building a website. However, this model presents a host of security risks.  

Caution: unruly AI ahead 

GenAI systems adoption is rapidly increasing in corporate environments – whether permitted by an organization or not. Without appropriate processes, guidelines or formal AI governance in place employees may inadvertently introduce these systems on corporate devices without fully understanding the risk or whether the use of these systems could violate a company’s existing data security and data privacy guidelines.  

Security leaders can get ahead of GenAI risk by understanding how potential GenAI use maps across existing data security and data privacy use cases. This understanding can be applied to risk benchmarking and scoring calculations, which can then help teams identify whether their organization has appropriate protections in place to remediate GenAI risk, or conversely, if security gaps exist.  

What are the security risks of AI agent-based programs? 

All Gen-AI systems rely on large data transactions and data retrieval techniques that potentially raise data security and data privacy risks.  

Already we’re seeing attackers leverage GenAI systems to refine social engineering techniques. In the case of AI agent-based programs, attackers could potentially leverage model extraction techniques to reverse-engineer proprietary AI models like Claude. With an understanding of an AI model’s structure and behavior, attackers can create mimicked versions to launch attacks or gain unauthorized insights into an organization’s operations.  

Similarly, attackers can drive prompt injection attacks by manipulating an AI model’s prompt to produce unintended or harmful outputs. Advanced AI models (like AI agent-based programs) can craft adaptive prompts, generating multiple variations of injection prompts to see which ones succeed by bypassing filters.  

Attackers are harnessing sophisticated AI models and programs to accelerate the sophistication of ransomware attacks. Aside from faster reconnaissance, attackers are successfully using AI to continuously modify ransomware files on an ongoing basis, making ransomware attacks more difficult to detect with traditional cybersecurity solutions. 

Getting ahead of AI agent-based security risks with preemptive cybersecurity 

Automated Moving Target Defense (AMTD) technology provides a powerful approach to defending against AI-enabled attacks by continuously shifting the attack surface, making it harder for attackers to successfully target an organization. 

The preemptive cyber defense that Morphisec delivers puts organizations in a position of strength. A preemptive cyber defense approach that’s powered by AMTD anticipates and acts against potential attacks before they occur, which is ideal for AI-enabled attacks that can evade traditional detection and response technology. 

AMTD offers a unique, independent layer of cybersecurity by not relying on AI-driven mechanisms, making it resistant to AI manipulation and exploitation. Unlike AI-based security tools, which could be influenced or misled by AI attackers, Morphisec’s pioneering AMTD technology functions autonomously, providing a robust line of defense that remains unbiased and impervious to AI's influence.  

This ensures that companies maintain a secure, predictable protective layer without the potential vulnerabilities associated with AI algorithms or machine learning errors. 

In scenarios where AI-enabled systems may conflict or become compromised, AMTD operates as a stabilizing force, safeguarding the environment from cascading AI-related issues.  

It avoids the common pitfalls of AI-driven solutions, such as biases, misinterpretations, or unexpected errors that can arise in complex AI-versus-AI interactions. With AMTD, companies are shielded by a technology that provides security continuity and consistency, which is essential when AI systems themselves may present unpredictable behavior. 

Additionally, AMTD enhances visibility by monitoring AI-driven activities, acting as a "watcher of the watchers" to ensure that AI agents function safely within set parameters. This continuous observation allows AMTD to identify unusual behavior or signs of compromise that AI might miss or overlook, ensuring comprehensive threat detection and response.  

By using non-AI-based detection techniques, AMTD remains resilient against AI-enabled exploits, providing an essential, stable layer of security that safeguards against evolving threats while remaining untainted by the vulnerabilities inherent to AI systems. 

Learn how preemptive cyber defense from Morphisec can protect your organization from AI-enabled attacks.  .