AI Agents & Agency
Artificial intelligence agents are rapidly reshaping how the internet functions. For the first time, software systems with varying degrees of autonomy are operating across multiple systems, interacting directly with other agents, and even executing multi-step tasks without human oversight.
This evolution marks a structural shift from an internet primarily mediated by human decision making to one characterized by machine-to-machine interaction.
For the last two years, IST’s researchers have convened private roundtables, held workshops, and conducted interviews with industry practitioners, policymakers, and researchers to identify ways to help AI agents enhance, rather than erode, trust and stability in the digital ecosystem.
Today’s paper, AI Agents & Agency in the Internet Ecosystem, presents three core insights into the AI agent landscape to support activities by policymakers, industry leaders, and standards bodies. To learn more, I sat down with lead author IST Senior Associate for Cybersecurity and Emerging Technologies Jennifer Tang.
Q&A: From Capability to Trust
Looking ahead, how do the challenges you identify change as AI systems move toward more complex, multi-agent environments?
Jennifer Tang: “One area we don’t explore in depth in the report, but that is quickly rising in importance, is how the challenges identified in the paper evolve in multi-agent environments. Much of the current conversation assumes a single agent interacting with a user or system. We’re moving toward settings where multiple agents, often from different developers and models, interact, coordinate, and sometimes compete with one another.
This introduces a different class of risks. For example, information can degrade as it passes between agents. Errors can compound, and it becomes harder to trace where something went wrong. This also raises concerns around intellectual property. What information is being shared, what is retained, and how is proprietary or sensitive data protected when agents are effectively “talking” to each other?
We also don’t yet have a good grasp of the full spectrum of outcomes possible when agents interact with one another. This is definitely an area where technical evaluations, governance, and commercial incentives are all going to intersect, and where current approaches are still quite early.”
In your second recommendation, you address the fact that benchmarks specifically focused on an AI agent’s trustworthiness are still rare. How should the average consumer be evaluating whether or not to use a model?
Jennifer Tang: “Right now, most of the resources and tools we can use to identify ‘trustworthiness’ are still quite technical. In our report, we highlight in detail the role of technical evaluations, model cards, and agent cards, which are all great ways to better understand what models and agents can do and to what degree. And Hugging Face for example, as well as a number of researchers in this space, are really pushing to fill this gap. That said, placing the burden on individual users and average consumers is not sustainable for building trust. More durable guidance will need to come from a combination of third-party evaluators, think tanks, and other stakeholders who seek to distill this information to the public.
This will inevitably become more timely as we see instances of agent failure without any parallel progress in developing mechanisms for recourse.”
Your paper also argues that the authorization of AI agents into systems should be a live, revocable process that responds to both behavior and environmental context. How do we determine what constitutes revocable behavior?
Jennifer Tang: “If an agent behaves outside its expected bounds—or anomalously—it seems straightforward enough to then reduce or remove its access. However, as you might expect, there are several challenges associated with this premise. Two worth highlighting are 1) how one might detect anomalous behavior; and 2) knowing what to do after detecting it.
Both are more resource-intensive than they might appear. Detecting anomalous behavior from agents requires some baseline for “normal” behavior and the ability to monitor performance over time. For leading firms and companies, this baseline and monitoring capability is likely already in place, but these capabilities are far less accessible to small- and medium-sized enterprises (or, for that matter, the average user!). We touch on the role of evaluations and benchmarks in the report and explain why evals focused on trustworthiness and real-world reliability are incredibly important and should be a greater focus in the space.
Even when anomalous behavior is identified, it remains a challenge to figure out how to respond. Organizations need to determine what the agent has accessed, modified, or exposed. Identifying these points depends on a range of processes like logging, audit and traceability mechanisms, and system visibility that may not already be in place. This is, of course, a familiar problem echoed in the cybersecurity realm. Containment and remediation can be greatly constrained by incomplete visibility into the scope of impact. In that sense, revocation is only as meaningful as an organization’s ability to observe and respond to what has already occurred.”
