AI agents don't have to be a security nightmare

AI agents are set to be the next key AI trend. After more than two years of all things generative AI, attention is beginning to turn to AI that can perform actions autonomously. 

However, they’re yet to see widespread adoption: a Capgemini survey of business executives found only 10% of respondents are currently using AI agents; 50% plan to implement them in 2025. Within three years, 80% plan to. Clearly, businesses are not only excited about the opportunity of bringing new levels of automation to their processes, they’re actively planning to do so. 

Yet while agentic AI offers significant opportunities for operational efficiency and accelerating workflows across the enterprise, there are also risks that need to be acknowledged and addressed. That makes this moment a particularly critical one. Over-enthusiastic and thoughtless adoption could undermine the long-term benefits of the technology. 

So, what are the risks? If AI agents have the potential to be a security nightmare, what can we do about it?

AI agents: what are they and how are they different from what’s come before?

To understand the risks, it’s worth first establishing what, exactly, we’re referring to; the term AI agent is used in different ways across the industry. 

The best way to understand them is as algorithmic systems that are able to both make a decision based on data and then take an action based on that decision. While there are similarities to generative AI, agents don’t produce outputs — they perform actions.

It’s worth noting that this kind of technology isn’t quite as new as the hype would have you believe. Algorithmic agents have been deployed in fields ranging from video games to robotic process automation for decades. What’s different now, though, is that agents have wider application. Although not quite general purpose, they can nevertheless perform a relatively sophisticated set of actions in response to a straightforward prompt.

This is, of course, one of the reasons that today’s AI agents have captured the imagination of technologists and business leaders alike. They seemingly possess an ability to find their way through a given problem or task.

What are the security risks of AI agents?

So, what are the risks of AI agents? There are a number of ways of thinking about it, but there are two that are primarily technical and one that’s social. Let’s take a look.

Data leaks

A potentially more serious risk is leaking data. AI agents depend on accessing different sources of information. They need to interface with various services and collect and share data where necessary in order to carry out the tasks with which it has been prompted. 

This seemingly fluid movement across a problem might be one of the benefits of AI agents, but fluidity is often antithetical to security. Think about it: even at a very basic level, security is really about setting limits about who or what has access to certain things.

An AI agent that is crossing organizational boundaries, or even operating inside and outside an organization is moving data in ways that aren’t easy to track. Even something as simple as booking travel could have some risk: what employee or organization information is being shared? And where?

A lack of accountability

Cutting across the key technical risks of AI agents, though, is their inherent lack of accountability. We can think about accountability in two distinct but important ways: first in managing work and second in a product liability sense. 

Task accountability

Accountability in tasks or managing work is relatively simple but also critical. If you ask a human to complete a task — a travel agent to book you a holiday, say, or a team mate to perform maintenance on a faltering part of a system — accountability is clear: they either did it or they didn’t. If they didn’t you can hold them to account: find out why they didn’t do it, work out what went wrong, escalate if necessary. While that can feel tricky from an interpersonal perspective, it’s still straightforward and clear.

The same can’t be said of AI agents — in some senses they ostensibly remove accountability from humans. In other words, there’s no regulating force to ensure that they perform the right actions and, just as importantly, there’s no easy way to make them accountable or find out what went wrong and why if a mistake is made. So, while employing agents might seem frictionless and seamless, without necessary mechanisms for accountability in place, there can be some difficult downstream consequences.

Product liability

A similar issue is the question of legal liability: when something goes wrong or an agent fails to accomplish a task as expected, who is at fault?  Unfortunately, at present there’s little legislation that really tackles this issue. While things like the EU AI Act attempt to make organizations accountable for the way AI and data is used, it's not clear how some of the latest regulation will tackle these questions when it comes to the latest changes in technology.

Indeed, at the moment, there’s a lot of legal energy being put into placing liability on human actors. That’s what’s happening in autonomous driving, an area where AI agents are making decisions that have a direct impact on human safety. 

While that might look like good news from a corporate perspective, this could have adverse mid-term consequences. Think about it this way: if liability when interacting with these systems falls to human users, that will undermine trust. In turn, that may make wider adoption much more difficult than the market currently expects it to be.

AI agents in action: the risks of unthinking overenthusiasm

The security risks of AI agents is arguably compounded by over-enthusiasm and a desire to move quickly. 

For instance, one of the most common ways they’re being used in organizations at the moment is to perform repetitive procedural tasks like moving data or widgets from one place to another. This is the sort of work that’s tedious for a human but could, with a little work, be improved by building better APIs or automations, which are easier to quality control and can be consistently managed. 

In other words, security risks are being introduced unnecessarily as teams attempt to tackle irritating problems quickly. 

So, how can we address the security risks of AI agents?

There are a number of things we can do to tackle the security risks of AI agents. 

First, it’s important that we think through where they’re being used. Are AI agents really necessary when a properly constructed and well-tested API would be more effective, maintainable and secure?

But if AI agents are necessary — and, for sure, organizations will identify use cases where they are relevant — it’s critical that we are mindful of well-established engineering practices around testing and security. 

This means ensuring things like risk analysis and threat modeling are embedded into your engineering workflows early on. At a deeper level, though, it will require a process of testing and probing agent behavior. Admittedly, this space hasn’t matured and coalesced into a standardized set of practices (which is arguably a security risk itself). However, what teams will need to do will be create test scenarios, trying to entrap or disrupt agents to see where they work and where they falter. That process can then inform further work to put in controls — not dissimilar to the guardrails that are today used in generative AI development — that help to provide a better defined space in which the agent can ‘act.’

The importance of holistic thinking and early testing

Ultimately, there are two key elements needed to ensure AI agents are not just secure but also impactful. 

The first is strategic: be holistic in how you view the opportunities for agentic AI. In other words, don’t have a narrow view whereby it’s the solution to every friction point and challenge across your organization. Think of it as one tool among many others that can improve and accelerate your teams — one that sits comfortably alongside, say, generative AI or, indeed, new APIs. 

Secondly, call upon good engineering practices — perform the necessary security analysis early in development pipelines and ensure you’re testing at every level. 

While this may add some friction it will pay off in the end, ensuring agentic AI delivers for your organization as the market and the current hype cycle is promising today.