Key Takeaways

  • Choose from five core routes (A through E) to stream traces, metrics, and logs into New Relic based on where your AI agent runtime lives and your required level of control
  • Enable OpenTelemetry (OTel) GenAI standards to capture rich execution context across agents, application frameworks, and model deployments without unnecessary code complexity
  • Monitor Microsoft Foundry model performance, token usage, and provisioned utilization using direct New Relic Query Language (NRQL) queries and Event Hub log streaming to proactively avoid rate limits

Microsoft Foundry (Foundry) brings Azure's AI building blocks into one place: Deployments in Foundry Models, AI agents in Foundry Agent Service, and the projects that connect them to tools and data. When a Foundry app answers slowly, incorrectly, or at higher cost than expected, the cause could be the agent's reasoning, a tool call, a throttled model deployment, or the code around them. Finding it means seeing all of those layers together. Without it, an engineering team debugging a production Foundry app is left guessing across disconnected tools, slowing incident response and eroding confidence in the AI systems the business is building on. 

Foundry, its agent frameworks, and related Microsoft services (like Agent 365) support OTel’s GenAI semantic conventions. New Relic made OTel support for its AI Monitoring offerings generally available, so gen_ai.* spans fill AI Monitoring without a New Relic agent installed. The catch is that Foundry offers three ways to run an AI agent, and each exposes a different amount of the runtime.

In this blog, we'll walk through a reference architecture with five routes for getting Foundry telemetry into New Relic, the configuration for each, and how to confirm the data is arriving.

The three layers of Foundry observability

  • Agent: Traces of each invoke_agent call, with the model calls, tool calls, and planning steps inside it
  • Application Framework: Client apps that call Foundry agents, and services you run yourself on Foundry Models
  • Model: Foundry Model metrics such as requests, tokens, availability, and provisioned utilization, plus resource logs

Architecture overview

Microsoft Foundry observability with New Relic: Routes A to E by Foundry component.

The lanes on the left are Foundry components, from most control (services you host on Foundry Models) to the least (Foundry prompt agents), with Foundry Model deployments at the bottom. 

Each colored line is a route into New Relic. Badges show maturity: GA, Preview, or Inferred, which means each step is documented but the end-to-end flow isn't.

Foundry componentRouteWhat it takes

Self-hosted AI agents and apps on Foundry Models

A (OTel) or E (New Relic agent), optional B (Collector)

One function call or one agent install

Foundry hosted agents

A through agent.yaml

Three environment variables

Foundry prompt agents

A or E in the client app; D for server-side spans

Client instrumentation

Foundry Model deployments

C

Azure Native / Azure Monitor integration and a diagnostic setting

Before you start: The New Relic Open Telemetry Protocol (OTLP) endpoint

Routes A and B send to the same place, so standard Otel variables can be set once:

# US endpoint shown; EU accounts use https://otlp.eu01.nr-data.net
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.nr-data.net"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="api-key=${NEW_RELIC_LICENSE_KEY}"
export OTEL_EXPORTER_OTLP_COMPRESSION="gzip"
export OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT="4095"

TL;DR is to use an ingest license key, not a user key. New Relic recommends HTTP/protobuf over the public internet, rejects payloads over 1 MB, and cuts attribute values at 4,095 characters, so compressing and setting the same limit in your app keeps long prompts predictable.

Foundry-hosted agents: Three variables, zero code

A hosted agent is your code in a container that Foundry Agent Service runs. Its Responses or Invocations protocol library already includes the OTel distro and sends traces, logs, and metrics to the project's Application Insights. Add OTLP settings, and the same telemetry also gets pushed to New Relic:

environment_variables:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: https://otlp.nr-data.net
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
  - name: OTEL_EXPORTER_OTLP_HEADERS
    value: ${{connections.nr-otlp.credentials.otlp_headers}}

To keep the license key out of the file, create a Custom keys connection named nr-otlp in the Foundry project, with a field otlp_headers set to api-key=<your license key>. A few behaviors to know:

  • Create the connection before deploying, or the placeholder resolves to an empty value
  • service.name is fixed to the agent name, and OTEL_SERVICE_NAME is ignored
  • Environment variables are immutable per version, so changes mean a new agent version
  • Application Insights keeps receiving data unless you turn off monitoring for the project

Self-hosted AI agents on Foundry Models

When you run the AI agent or app yourself, choose OTel or the New Relic agent for each service, not both, or every model call is recorded twice.

Route A: The Microsoft OpenTelemetry Distro

The microsoft-opentelemetry package (1.5.0, released October 20, 2026; Python 3.10 or later) configures exporters and instrumentation in one call. With the variables above set, the OTLP export turns on automatically:

# pip install "microsoft-opentelemetry>=1.5.0"
from microsoft.opentelemetry import use_microsoft_opentelemetry
 
# OTLP export to New Relic turns on because OTEL_EXPORTER_OTLP_* is set.
# enable_azure_monitor also keeps Application Insights and Foundry's Traces view.
use_microsoft_opentelemetry(enable_azure_monitor=True)

It instruments OpenAI, the OpenAI Agents SDK, LangChain, Semantic Kernel, and Microsoft Agent Framework, and can export to Microsoft Agent 365 at the same time. It's a different package from azure-monitor-opentelemetry, which only exports to Application Insights.

Route B: Add a Collector for control

An OTel Collector gives you one place to batch, redact, filter, and fan out. Point OTEL_EXPORTER_OTLP_ENDPOINT at it, including from hosted agents if the Collector is reachable:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  otlphttp/newrelic:
    endpoint: https://otlp.nr-data.net
    compression: gzip
    headers:
      api-key: ${env:NEW_RELIC_LICENSE_KEY}
service:
  pipelines:
    traces:   # add metrics and logs pipelines the same way
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/newrelic]

Route E: The New Relic agent

If your teams already run New Relic agents, route E adds New Relic AI Monitoring data next to full APM context. In Python:

pip install "newrelic>=14.2.0"   # 14.2.0+ for Microsoft Agent Framework
export NEW_RELIC_LICENSE_KEY="<your license key>"
export NEW_RELIC_APP_NAME="support-agent"
export NEW_RELIC_AI_MONITORING_ENABLED=true   # off by default
newrelic-admin run-program python app.py

New Relic AI Monitoring requires Python agent 9.8.0 or later, but instrumentation for Microsoft Agent Framework tools and agents arrived in 14.2.0 (October 29, 2026); multi-agent configurations aren't covered yet. The .NET agent (10.37.0 or later) supports OpenAI and Azure.AI.OpenAI 2.0.0 or later.

Foundry prompt agents: Instrument the client

Foundry runs prompt agents entirely, and their server-side traces go to Application Insights automatically. What you can instrument is the client app that calls them, using Foundry's client-side tracing (preview, azure-ai-projects 2.0.0 or later):

import os
# Must be set before instrument() is called
os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true"
 
from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor
from azure.identity import DefaultAzureCredential
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
 
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
AIProjectInstrumentor().instrument()
 
with AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
                     credential=DefaultAzureCredential()) as project:
    with project.get_openai_client(agent_name="support-agent") as openai:
        conversation = openai.conversations.create()
        response = openai.responses.create(
            conversation=conversation.id, input="Where is my order 1234?")

The SDK adds W3C trace context to these calls by default, so client spans in New Relic share a trace ID with the prompt agent's server-side spans in Application Insights. If tracing produces nothing, check that the environment variable is set before instrument() runs.

Foundry Models deployments

Azure Monitor collects Foundry Models metrics automatically. Turn on the New Relic Azure Monitor integration, filter it to your Foundry resources, and it polls as often as every minute. The metrics arrive as raw dimensional metrics rather than entities, so query them directly:

FROM Metric
SELECT sum(azure.cognitiveservices.accounts.TotalTokens),
       max(azure.cognitiveservices.accounts.ProvisionedUtilization)
WHERE collector.name = 'azure-monitor'
FACET azure.resourceId TIMESERIES

ProvisionedUtilization makes a good first alert: At 100%, Azure throttles calls with HTTP 429. Use the Event timer streaming method for alerts on these metrics. TimeToResponse only exists for PTU and PTU-managed deployments, so get latency from traces on standard deployments.

For resource logs, deploy New Relic's Event Hub ARM template, then stream each Foundry resource's logs to it:

# Check category names first: az monitor diagnostic-settings categories list
az monitor diagnostic-settings create \
  --name foundry-to-newrelic \
  --resource "$FOUNDRY_RESOURCE_ID" \
  --event-hub "$EVENT_HUB_NAME" \
  --event-hub-rule "$EVENT_HUB_AUTH_RULE_ID" \
  --logs '[{"category": "RequestResponse", "enabled": true}]'

Foundry server-side traces through Application Insights

To bring a prompt agent's own steps into New Relic, export them from the Log Analytics workspace behind Application Insights to the same Event Hub:

az monitor log-analytics workspace data-export create \
  --resource-group "$RG" --workspace-name "$WORKSPACE" \
  --name foundry-traces-to-newrelic \
  --tables AppRequests AppDependencies AppTraces \
  --destination "$EVENT_HUB_NAMESPACE_ID" --event-hub-name "$EVENT_HUB_NAME"

The spans arrive as log records rather than traces, the Event Hub must be in the workspace's region, and data takes a few minutes. If that's more than you need, instrument the client and look up the trace ID in Application Insights.

Verify your data

After a few test requests, allow 2 to 5 minutes, then run:

SELECT count(*) FROM Span
WHERE gen_ai.system IS NOT NULL OR gen_ai.provider.name IS NOT NULL
SINCE 30 minutes ago FACET service.name, gen_ai.operation.name

If no rows come back, check the regional endpoint and that you used an ingest license key. Spans without New Relic AI Monitoring view mean missing GenAI attributes, so upgrade the instrumentation. New Relic validates OTLP data after accepting it, so also check SELECT * FROM NrIntegrationError SINCE 1 hour ago.

Security and cost considerations

  • Content: OTel and Foundry don't capture prompt text unless you opt in with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true; the New Relic agent does by default once AI Monitoring is on
  • Residency: Choose the US, EU, JP, or FedRAMP endpoint to match your account
  • Volume: Sample with OTEL_TRACES_SAMPLER, keep content capture off unless needed, and filter in a Collector

Five questions to help pick the right design

Use these five questions to match your Foundry setup to the right route:

  1. Who runs each AI agent: Your team, Foundry hosted agents, or Foundry prompt agents?
  2. Do you already run New Relic agents? If yes, route E fits self-hosted services
  3. Can prompts and responses leave your tenant? That sets content capture and the endpoint region
  4. Do you need one egress point or redaction before export? Then add route B
  5. Are your deployments provisioned (PTU) or standard? That decides which latency and utilization metrics exist

In summary

Observing Microsoft Foundry in New Relic comes down to one question per component: Who runs it? Foundry-hosted agents need three environment variables, self-hosted services need one distro call or a New Relic agent, prompt agents need client instrumentation, and Foundry Models need the Azure Monitor integration and a diagnostic setting. Because routes A and B are plain OpenTelemetry, your instrumentation stays portable. Start with the route that fits each component, run the verification query, and add model metrics so every layer is covered. From there, your team can ship AI-powered features with the same confidence they bring to any production service.

Get started

Sign up for a New Relic account, point your Foundry telemetry at the OTLP endpoint, and open New Relic AI Monitoring to see your first traces.