The Observability Gap in AI Agents

Key Takeaways: What our assessment reveals

1. Continuous monitoring is an essential safeguard, as a growing number of incidents show agents taking unauthorized actions that testing could not anticipate.
2. The infrastructure for monitoring agents does not yet reliably exist, though policymakers assume it does.
3. We found six gaps in the telemetry that agent frameworks emit. Taken together, they leave enterprises unable to reliably trace an agent’s authority, how it changed, and why the agent acted as it did.
4. Closing these gaps requires coordinated action across framework developers, model providers, enterprises, and regulators.

The Problem

As AI agents move into regulated industries like banking and healthcare, the tools that should be monitoring them have critical blind spots.

In July 2026 an AI agent being tested by OpenAI broke out of its testing environment and attacked Hugging Face’s systems. It was later reported that OpenAI did not detect the intrusion for nearly a week.

This was not an isolated case, and pre-launch testing cannot catch everything an agent will do. Continuous monitoring is a key safeguard, but it depends on observability: recording what an agent did, turning that into signals, and notifying those who can act.

The Observability Gap in AI Agents assesses four widely used agent frameworks against six monitoring goals. It examines whether these frameworks produce the records that enterprises need to monitor agents in real time and to review and adjust the agents’ decisions afterward.

How Current Observability Falls Short

Any system for monitoring agents is only as good as the data underneath it, and today’s agent frameworks often don’t record the data that matters:

Monitoring goals
Monitoring must go beyond whether agents stay within permissions, to whether agent decisions are sound and human review is meaningful.

Processing techniques
Agents emit more telemetry than human reviewers can analyze, so the signals enterprises need to monitor and act on must be computed with emerging techniques.

Telemetry gaps
Meeting these monitoring goals requires records of both what agents did and how these systems reached their decisions.

Each monitoring goal depends on particular processing techniques, each of which may be blocked by specific telemetry gaps.

depend on…

blocked by…

Six Telemetry Gaps

We found six categories of telemetry that the frameworks emit inconsistently or not at all.

Authors

The authors of this report span academia, industry, and civil society.