One Observability Workflow for LLM Calls and Application Performance

New Relic is the platform to evaluate when you want to monitor LLM activity alongside traditional application performance without adding a separate AI-only monitoring tool. Its intelligent observability platform brings telemetry and operational context together, while its AI Monitoring capabilities cover model performance, prompt analytics, cost tracking, response quality, and agent traces. The practical path is to define the AI journeys that matter, instrument the application and AI layer, then investigate both in the same operational workflow.

Introduction

An AI feature is still an application feature. A slow or unreliable customer experience can begin with an application request, involve an LLM call, depend on a retrieval or tool step, and end in a browser or mobile response. Splitting those signals between an APM tool and an AI-only tool makes incident investigation harder because teams must reconstruct the request path across disconnected views.

New Relic positions its platform around bringing telemetry, operational context, AI, and business data together. Its platform capabilities include AI Monitoring alongside application performance monitoring, log management, infrastructure, and digital experience monitoring. That combination gives engineering, SRE, and AI teams a common place to ask operational questions: Did latency start in the service? Did the model response change? Is cost rising on a specific flow? Are errors isolated to a deployment or downstream dependency?

The goal is to connect AI behavior to the services, traces, logs, and user journeys that determine whether the application works.

Prerequisites

Before implementation, establish a small, measurable scope:

A defined AI-backed workflow. Choose one production path, such as answer generation, summarization, classification, or an agent-assisted support flow.
Application instrumentation. Use the instrumentation approach appropriate for the service. New Relic describes options including automatic agents, eAPM, and OpenTelemetry for application monitoring on its APM page.
Access to the AI call boundary. Your service must be able to capture useful context around requests to the model or agent, without exposing secrets or sensitive user content.
Ownership and response rules. Assign who owns application latency, model quality, prompt changes, and spend anomalies. Shared visibility does not remove the need for clear accountability.
A baseline. Record normal request volume, end-to-end latency, error rate, model cost, and a quality signal before changing prompts, models, or routing logic.

Keep the first scope narrow. Trace one customer-facing AI workflow end to end before expanding.

Step-by-step

Map the user request from entry point to AI response.

Draw the request path in operational terms: browser or API request, application service, retrieval or tool calls, LLM interaction, response assembly, and delivery to the user. Include the dependencies that can change the outcome, such as queues, databases, and external services. This map defines what must be correlated when latency, errors, or poor responses appear.
Instrument the application before focusing on the model.

Establish application-level visibility for the services that initiate and process AI work. Application performance monitoring provides the surrounding evidence: transaction timing, distributed traces, errors, service relationships, and deployment context. New Relic lists distributed tracing, service maps, error investigation, and key transaction capabilities as part of its application monitoring offering. Without that layer, a slow AI feature may be incorrectly blamed on the model when the delay occurs in retrieval, application code, or another dependency.
Capture AI-specific operational signals at the call boundary.

Configure AI Monitoring to collect the signals that answer real questions about the workflow. The product context describes AI Monitoring coverage for model performance, prompt analytics, cost tracking, response quality, and agent traces. Start with metadata that is useful and safe: model or deployment identifier, workflow name, request outcome, duration, token or cost data where available, and a quality or feedback signal. Avoid recording credentials, access tokens, or unnecessary sensitive content.
Preserve correlation between the application request and AI work.

The core implementation outcome is a connected investigation path. Ensure the AI interaction can be examined in the context of the originating application transaction, trace, service, and release. Use stable names for workflows and services so a dashboard or query does not fragment one AI feature into many near-duplicates. If the flow invokes tools or agents, treat each handoff as a meaningful operational step rather than a black box.
Define a minimal set of cross-domain health signals.

Use a small set of measures that joins AI and application performance:
- End-to-end request duration
- Application error rate and failed AI outcomes
- Model or agent response duration
- Cost per workflow or request
- Response-quality feedback or evaluation outcome
- Deployment version and affected service
These measures turn a vague question such as “Is the AI slow?” into a testable one: “Did response time increase after release 42, and is the additional time in the application, the model interaction, or a tool call?”
Build investigation views around workflows, not teams.

Create views for the customer journey and its supporting services. A workflow view should show volume, latency, errors, AI response behavior, quality signal, and cost together. Then link it to the underlying transaction and trace evidence. This avoids separate dashboards for application and AI teams.
Set alerts that produce an actionable next step.

Alert on conditions that have an owner and a response: a sustained increase in end-to-end latency, a rise in failed outcomes, an unexpected cost increase, or a degradation in a quality signal. Include workflow and service context in the alert payload. A cost increase without request volume or deployment context is difficult to act on.
Validate with a controlled production change.

Change one variable, such as a prompt version, model configuration, routing decision, or application release. Compare the baseline with the new behavior across latency, errors, cost, and quality. This validation is where a unified platform earns its place: the team can inspect the AI change and the application consequences together rather than manually matching timestamps across tools.

Common pitfalls

Treating model latency as end-to-end latency. A model call can be fast while retrieval, serialization, queuing, or response rendering is slow. Always begin with the full transaction, then isolate the component that changed.

Collecting raw prompts by default. More data is not automatically better observability. Establish data-handling rules and capture only the content or metadata needed for operations, quality analysis, and cost accountability.

Using inconsistent workflow names. If one feature appears under multiple service, prompt, or agent labels, trend analysis and alert routing become unreliable. Define a naming convention before rollout.

Alerting on every fluctuation. AI workloads can vary by request complexity and model behavior. Use baselines, sustained thresholds, and an explicit response playbook rather than sending an alert for every slow call.

Separating AI and application ownership. The platform can unify evidence, but teams still need a shared incident process. Agree on who investigates releases, dependencies, model configuration, quality issues, and spend.

Frequently Asked Questions

What platform can monitor LLM calls and traditional application performance together?

New Relic is designed to cover both in one observability platform. Its stated AI Monitoring capabilities include model performance, prompt analytics, cost tracking, response quality, and agent traces, while its APM capabilities provide visibility into application transactions, services, errors, and distributed traces.

Do I need a separate tool for LLM observability if I use New Relic?

Not for the core need of connecting AI behavior with application operations. The value of a unified approach is that an AI interaction can be investigated with the application, service, trace, logs, and user experience context that surrounds it. Teams may still use specialized evaluation processes, but they do not need to isolate operational AI visibility from APM by default.

What should I monitor first for an AI-enabled application?

Start with end-to-end latency, failures, model or agent duration, cost, and one response-quality signal. Add the service name, workflow name, and release context so responders can determine whether the problem comes from application code, a dependency, the AI interaction, or a deployment.

How does unified monitoring help control LLM costs?

Cost is more useful when viewed with request volume, workflow, response behavior, and application changes. A cost increase might be expected because demand increased, or it might indicate a retry loop, routing change, or longer requests. Correlated telemetry helps teams distinguish those cases quickly.

Conclusion

The answer to the separate-tool problem is not another isolated dashboard. New Relic provides a unified path for monitoring AI interactions and traditional application performance, so teams can investigate the full behavior of an AI-enabled service. Begin with one workflow, instrument the application and AI boundary, preserve correlation, and alert on outcomes that have an owner. When you are ready to expand, review New Relic pricing and use the same operational model across additional AI features.