Your Agent Traces Are Lying to You: The Race to Standardize Agent Observability

Most teams running agents in production are debugging with logs that were designed for request-response APIs. A trace shows you a span that says "call tool," a duration, and maybe a status code. It does not show you why the agent picked that tool, what it ignored in context, or how the conversation state drifted three turns earlier. For the last year, every studio and platform team building agents has been improvising its own schema for this. That improvisation phase is ending, fast, and it's worth understanding what's landing before you re-architect around it.

The standard that just showed up

OpenTelemetry's GenAI Special Interest Group has been quietly defining semantic conventions for generative AI observability, and it recently expanded that scope specifically to cover agents, with instrumentation support landing in Python and other languages. This is not a vendor's opinionated SDK. It's a standards body process, with open SIG meetings and an active CNCF Slack channel, which means the spec is still moving but the direction is now set: agents get first-class span types instead of being shoehorned into generic RPC spans.

The spec is specific in ways that matter for anyone instrumenting a real system. It defines conventions for agent calls that extend and override the general GenAI span conventions, and it dictates naming: the span name should be "invoke_agent {gen_ai.agent.name}" when the agent's name is available. That sounds pedantic until you've tried to correlate traces across five different in-house agent frameworks, each naming spans differently, none of them joinable in a dashboard without custom regex. A shared naming convention is boring infrastructure work, and boring infrastructure work is exactly what agent tracing has been missing.

Why vendors are moving now

Datadog announced native support for the OTel GenAI semantic conventions in December 2025, which is a bigger deal than it sounds. Previously, getting agent telemetry into a platform like that meant instrumenting with a vendor-specific SDK and accepting lock-in on your instrumentation layer, not just your storage layer. Native OTel support means the same span data you emit can, in principle, land in multiple backends without rewriting your instrumentation code.

The pitch behind the convention is straightforward: establish a standard schema for prompts, model responses, token usage, tool and agent calls, and provider metadata, so that spans, metrics, and events use a consistent vocabulary across frameworks and vendors. If it holds, that's the difference between agent observability being measurable and comparable across systems versus every team building its own dialect that only its own dashboards understand. This is the same maturity curve HTTP tracing went through a decade ago: painful vendor fragmentation, then convergence on a shared model, then the tooling gets genuinely good because everyone is building against the same primitives.

The part everyone skips: content capture

Here's where teams actually get burned, and it's not glamorous. The spec explicitly recommends that instrumentations should not capture prompts, inputs, or outputs by default, and should only do so as an opt-in. Most homegrown agent tracing does the opposite: someone wires up a quick wrapper around the LLM call, dumps the full prompt and response into the trace payload because it's convenient for local debugging, ships it, and six months later that trace backend has become a de facto data lake full of customer PII, API keys pasted into chat, and internal system prompts, all sitting in a third-party observability tool with retention settings nobody configured.

Store content externally and record references on the spans. This pattern is recommended in production environments where telemetry volume is a concern or sensitive data needs to be handled securely.

That's the actual guidance, and it's the right default. The trace carries a reference, an ID, a pointer into a separate content store with its own access controls and retention policy. The trace backend carries structure: token counts, latencies, tool call arguments as typed fields, status. If you're instrumenting agents right now and you haven't drawn this line, draw it before you scale traffic, not after a security review finds it for you.

The counter-argument: traces might be the wrong shape entirely

Not everyone thinks bolting agent semantics onto the existing traces/logs/metrics model is enough. There's a real technical argument, gaining traction among people who've actually tried to debug production agents at scale, that the three-pillar model breaks down for this workload specifically.

• Stuffing prompts into logs loses structure and makes analysis across sessions nearly impossible.
• Forcing tool calls into rigid trace spans doesn't hold up against genuinely dynamic, branching agent behavior.
• Pre-aggregating token metrics into counters and histograms loses the exact context you need when something goes wrong.

The underlying claim is that agent execution produces something structurally different from a typical request trace: high-cardinality data, since you're generating something close to unique behavior per session at scale, high-dimensional data, with dozens of relevant fields per single execution, and context-rich events where the interesting information is the relationship between fields, not any single metric. The proposed alternative is often called "wide events," a single structured event per unit of agent work that carries every relevant dimension together, queryable after the fact, rather than pre-splitting that information into separate logs, metrics, and trace spans that then have to be stitched back together during an incident.

This isn't purely academic. If you've ever tried to answer "why did the agent choose tool B over tool A in this specific session, and was that decision correlated with a particular context window state," you already know that a trace waterfall and a metrics dashboard don't answer that question well. You end up grepping logs and cross-referencing timestamps by hand, which is exactly the failure mode observability tooling is supposed to eliminate.

Why this is landing under real governance, not just vendor blogs

On December 9, 2025, the Linux Foundation announced the formation of the Agentic AI Foundation, anchored by founding contributions that include the Model Context Protocol, the goose agent framework, and the AGENTS.md convention. This matters for observability specifically because it signals that the interop layer for agents, protocol, tool-calling, and by extension the telemetry that describes agent behavior, is consolidating under neutral governance rather than staying fragmented across competing vendor SDKs.

That's the pattern that makes standards durable. A spec backed by one vendor gets adopted until that vendor's incentives shift. A spec developed in an open SIG and then anchored under a foundation with multiple corporate and open-source stakeholders tends to survive longer and gets wider tooling support faster. If you're deciding whether to invest engineering time in adopting OTel's GenAI conventions now versus waiting, this is a real signal that the standard has institutional weight behind it, not just an early-adopter curve.

What to actually do with this right now

If you're instrumenting an agent system today, adopt the OTel GenAI semantic conventions for the parts of your system that map cleanly onto them: tool calls, model invocations, token usage, provider metadata. This gets you vendor portability and a shared vocabulary your future hires will already understand, since this is rapidly becoming the lingua franca rather than a niche choice.

Do not treat OTel spans as your only telemetry layer if you're running anything with real multi-step agent behavior, branching tool use, or long-running sessions. Emit a wide event per agent run or per significant decision point alongside your spans, even if the tooling to query it well is still maturing. You want that data captured now, structured and queryable, even if you're stuck grepping JSON blobs in the interim, because rebuilding historical context after the fact is not possible. You cannot retroactively instrument a session that already happened.

Enforce content externalization from day one. Reference IDs in spans, actual prompt and completion content in a separate store with its own retention and access policy. This is the single most common mistake in agent observability right now, and it's the one most likely to turn into an incident report instead of a postmortem note.

Where the standard still doesn't match how agents misbehave

Even with clean spans and wide events, there's a category of failure the current conventions don't capture well: multi-turn drift, where an agent's behavior degrades gradually across a long session due to context accumulation rather than any single bad call. A span tells you what happened at a point in time. It doesn't tell you that the agent's fifth tool call was subtly worse because the first four polluted its context window in a way no single span reveals. Wide events help here more than traces do, because you can carry cumulative session state as a dimension on the event rather than losing it between spans. But nobody has fully solved this yet, and it's worth treating the current standards as a floor, not a finished house. Instrument to the spec, but keep building the parts it doesn't cover yet, because that's where your actual incidents will come from.