AI agents are rapidly replacing tasks once delegated to search engines. As we shift toward sophisticated multi-agent and agent-to-agent architectures, observability becomes the cornerstone of reliability. Relying solely on standard request-response patterns is insufficient; when agents operate autonomously, they frequently drift into hallucinations or performance bottlenecks that remain hidden without granular visibility.
Let’s not assume everything will always go smoothly; AI agents can go rogue, start hallucinating, take more time than anticipated, or fail during a chain of LLM calls or tool invocations. The default approach is to examine logs, metrics, token usage, and the AI models involved in the execution to understand what went wrong.
A log tells you something happened. A trace tells you what happened, in what order, how long it took, and what caused what.
In this blog, we’ll explore how to configure OpenTelemetry traces for AI agents using a travel-planning application as a case study.
Travel Agent App
The travel orchestrator multi-agent app takes a simple prompt such as “Plan a 5-day trip from NYC to Tokyo in March” and does the needed tool calling to give you a complete itinerary. Under the hood, it’s a Strands multi-agent app where there is an orchestrator that invokes the specialist agents and tools (research, weather, flights and hotels) where each agent uses a model that is ideal for the task. This is deployed on Amazon Bedrock AgentCore Runtime that is a managed serverless runtime where the agent is hosted on.
Because of the environment constraints with AgentCore, when something goes wrong, you cannot SSH into the VM to access your logs or use a debugger. There are multiple agents and tools being invoked which makes it hard to pinpoint without the right telemetry information.
And explaining the behaviour for why the request is failing with a specific agent or tool OR even a scenario for a slow request given that AI is nondeterministic and hard to reproduce. And your only dependency is on traces.
Here is a snapshot of one of the traces which gives a complete breakdown of spans and dependent downstream AI tools invocations.
Understanding the instrumentation
Observability comes as a pre-baked configuration with the Strands SDK and supports the OpenTelemetry integration.
Strands observability
Strands SDK comes with `telemetry` and for enabling traces, ensure the agent has the below code snippet that sets up the OTLP exporter for traces. Strands follows GenAI semantic conventions, and it reads the standard `OTEL_*` environment variables.
from strands.telemetry import StrandsTelemetry
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_otlp_exporter()
strands_telemetry.setup_meter(enable_otlp_exporter=True)
from opentelemetry import trace, metrics
_tracer_provider = trace.get_tracer_provider()
_meter_provider = metrics.get_meter_provider()Configuring the exporter
Strands SDK’s `StrandsTelemetry` helper builds the traces and metrics from OTEL environment variables. Unfortunately, logs must be manually bridged from Python’s stdlib to OTLP.
# --- Exporter targets (New Relic OTLP by default) ---------------------
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"api-key={license_key}"
os.environ.setdefault("OTEL_EXPORTER_OTLP_ENDPOINT", "https://otlp.nr-data.net")
os.environ.setdefault("OTEL_SERVICE_NAME", service_name)
# New Relic prefers delta temporality for OTLP metrics.
os.environ.setdefault("OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE", "delta")
# Emit the current GenAI semantic conventions (model, tokens, tools…).
os.environ.setdefault("OTEL_SEMCONV_STABILITY_OPT_IN", "gen_ai_latest_experimental")
# 1) Traces + metrics via Strands' built-in OTel integration
from strands.telemetry import StrandsTelemetry
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_otlp_exporter() # spans -> OTLP
strands_telemetry.setup_meter(enable_otlp_exporter=True) # metrics -> OTLP
# 2) Bridge stdlib logging -> OTLP logs, correlated by trace/span id
provider = LoggerProvider(resource=Resource.create({"service.name": service_name}))
exporter = OTLPLogExporter(endpoint=f"{endpoint}/v1/logs", headers={"api-key": license_key})
provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
logging.getLogger().addHandler(LoggingHandler(logger_provider=provider))Now, all of your telemetry—logs, metrics, and traces—are exported over OTLP/HTTP.
OTEL GenAI instrumentation
To ensure the OTEL instrumentation is able to capture information about the AI invocation as part of the span attributes, enabling the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_*` environment variables.
for _flag_var in (
"OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT",
"OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_METADATA",
"OTEL_INSTRUMENTATION_GENAI_CAPTURE_TOOL_INPUT",
"OTEL_INSTRUMENTATION_GENAI_CAPTURE_TOOL_OUTPUT",
):
os.environ.setdefault(_flag_var, "true")And when the spans are available on New Relic with the gen_ai attributes as below
Filtering message content
The default configuration with Strands and its GenAI telemetry is that it captures the message content as part of the tool input/output irrespective of the configuration with `OTEL_INSTRUMENTATION_GENAI_CAPTURE_*` and to enforce the environment variable based configuration, ensuring the tracer filters the message content is crucial.
from strands.telemetry import tracer as strands_tracer_mod
class _GatedTracer(strands_tracer_mod.Tracer):
"""Drop content per the OTEL_..._CAPTURE_* flags, before it hits a span."""
def _filter_event_attributes(self, span, event_name, event_attributes):
if capture_message_content and capture_tool_input and capture_tool_output:
return event_attributes # nothing disabled ⇒ native behaviour
is_tool = self._operation_name(span) == "execute_tool"
filtered = dict(event_attributes)
if is_tool and not capture_tool_input:
filtered.pop("gen_ai.input.messages", None) # redact tool args
# …message content + tool output gated the same way
return filtered
# Install BEFORE any agent is built — agents grab the tracer singleton.
strands_tracer_mod._tracer_instance = _GatedTracer()Now if we flip the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_TOOL_INPUT=false*`, this would drop the message content at the source itself so sensitive data isn’t exported.
Span Events vs Span Attributes in New Relic
Strands writes all of the GenAI telemetry - prompt, completions, tools inputs, tool outputs into OTLP span events and the NRQL queries which search for `gen_ai_*` attributes is always returned NULL. To address this scenario, we would have to override the Strands’ Tracer’s `_add_event`.
from strands.telemetry import tracer as strands_tracer_mod
BaseTracer = strands_tracer_mod.Tracer
_SPAN_ATTR_PROMOTE_KEYS = (
"gen_ai.input.messages",
"gen_ai.output.messages",
"gen_ai.system_instructions",
"content",
"message",
)
class _GatedTracer(BaseTracer):
def _add_event(self, span, event_name, event_attributes, to_span_attributes=False):
try:
event_attributes = self._filter_event_attributes(span, event_name, event_attributes)
if (
content_on_span_attrs
and event_attributes
and any(k in event_attributes for k in _SPAN_ATTR_PROMOTE_KEYS)
):
to_span_attributes = True # promote to span attributes for NRQL
except Exception as exc:
logger.debug("GenAI capture gate: event filter error: %s", exc)
return super()._add_event(span, event_name, event_attributes, to_span_attributes)Now when you run the NRQL queries, you are able to find the Span with gen_ai content.
What do traces actually unlock?
With all this telemetry data, we want to know: what do traces specifically unlock for AI agents?
- Root-cause debugging: Trying to identify any specific “trip plan” was slow and the real cause of the slowness is because a `duckduckgo_search` tool is erroring out. And identifying the error and how that bubbles up to the complete agent app degradation.
- Latency: Identifying which spans of LLM chat invocation is taking time when the agents and tools are executing with the slow span identification.
Wrap up
It’s easier to build a multi-agent orchestration with Strands on a managed serverless environment such as AgentCore, but doing so without tracing and telemetry data makes troubleshooting significantly more difficult.
This complete application is using the OpenTelemetry spans which Strands emits out of the box and some of the custom instrumentation for filtering and dropping sensitive information along with enabling span events and attributes. You can clone the repository and deploy it to your own AWS Account with CDK and New Relic credentials.
Now build your Strands agents on AgentCore with the right instrumentation for traces and leverage the traces to pinpoint bottlenecks and errors.
이 블로그에 표현된 견해는 저자의 견해이며 반드시 New Relic의 견해를 반영하는 것은 아닙니다. 저자가 제공하는 모든 솔루션은 환경에 따라 다르며 New Relic에서 제공하는 상용 솔루션이나 지원의 일부가 아닙니다. 이 블로그 게시물과 관련된 질문 및 지원이 필요한 경우 Explorers Hub(support.newrelic.com)에서만 참여하십시오. 이 블로그에는 타사 사이트의 콘텐츠에 대한 링크가 포함될 수 있습니다. 이러한 링크를 제공함으로써 New Relic은 해당 사이트에서 사용할 수 있는 정보, 보기 또는 제품을 채택, 보증, 승인 또는 보증하지 않습니다.
