Instrumenting an LLM router: what New Relic sees, Prometheus doesn't

Routing prompts to different tiers of language model by complexity is a pattern with real, unglamorous economics behind it: send the easy questions to something small, fast, and cheap, save a bigger model for the requests that actually need it, and stop paying frontier-model prices for every message just because one message might be hard. It's the kind of infrastructure that pays for itself the day it ships.

Building the router is the easy part. The interesting engineering and the part that determines whether this pattern is safe to run in production starts the moment it goes live, because at that point a piece of infrastructure is deciding, autonomously and per request, which model answers which question. The operational question stops being "is the service up" and becomes "is the router making good decisions, and would anyone know the moment it started making bad ones." A generic metrics stack — request counters, JSON logs, a latency histogram — was built to answer the first question. It was never built to answer the second, and a striking amount of AI infrastructure ships with nothing else.

This post is about answering that second question with New Relic. We built a three-tier routing setup — entirely on Ollama, no cloud fallback — as a reference implementation, then instrumented it end to end: what New Relic captures automatically once you point it at the right place, what it can't capture because the concept doesn't exist in any AI observability schema yet, what it costs an organization to not have that visibility, and what happens when we wire in Workflow Automation and Autopilot so the system doesn't just get watched, it gets fixed.

What happens when nobody's watching the router

None of this is hypothetical. The specific failure modes an unmonitored LLM call opens up have already happened, publicly, independently of each other — which is the actual argument for building the visibility described in this post before needing it, not after.

Model behavior drifts, silently, between versions you didn't choose to change. A Stanford and UC Berkeley study published in mid-2023 measured GPT-4's accuracy on a simple prime-number-identification task fall from 84% to 51% across two model snapshots three months apart, with no version announcement a downstream team could have watched for. Without a continuous evaluation signal sitting on top of every call, the only way to notice a regression like that is a user noticing first.

A bad response stops being an embarrassment and becomes a liability the moment it reaches a customer unreviewed. In February 2024, Canada's Civil Resolution Tribunal held Air Canada responsible for bereavement-fare information its own website chatbot had given a customer — and explicitly rejected the airline's argument that the chatbot was "responsible for its own actions." There was no eval layer and no human checkpoint between the model's output and the customer who acted on it, and the legal system did not treat that as an excuse.

The visibility layer itself can become the leak if nobody's thought about what it's capturing. In May 2023, Samsung banned employee use of ChatGPT after engineers pasted proprietary source code and internal meeting notes directly into prompts — and it's worth being specific about why that risk doesn't go away just because you're now watching your own AI traffic instead of a third party's: several popular tracing conventions capture full prompt and completion content by default. Visibility and data governance have to be designed together, or the trace meant to protect you becomes the thing that needs protecting.

And the mundane version of the same blind spot doesn't need an incident at all. A token bill can climb for weeks before anyone traces the increase back to which tier, which prompt pattern, or which misbehaving retry loop is actually spending it — the least dramatic failure mode on this list, and the one every team building on LLM APIs eventually hits regardless of whether anything else on this list ever does.

None of this is a future concern regulators are only beginning to gesture at, either. The EU AI Act requires high-risk AI systems to support automatic logging of events across their operating lifetime, specifically to enable post-market monitoring — a legal floor, not a best practice, phasing in through 2027-2028. By the time it's mandatory, the honest question won't be whether to instrument an AI system this way; it'll be how far behind the teams are who waited.

The setup: one router, three tiers, all local

The shape is deliberately simple. A FastAPI gateway scores every incoming prompt with six additive, deterministic signals — prompt length, code content, multi-step language (“step by step,” “algorithm,” “trade-off”), explicit depth requests (“explain in detail,” “why does”), math density, and multiple questions — and maps the total (0-10) onto a tier by a fixed threshold. No embeddings, no classifier model: the whole decision is a handful of regex checks, which means routing overhead is effectively zero next to the model call it's about to make.

The gateway hands the request to a LiteLLM proxy, which is the one piece of glue doing real work in this stack: it exposes a single OpenAI-compatible endpoint, and its model_list maps each tier name to an actual Ollama model. All three tiers run on the same Ollama instance; there is no cloud fallback in this build, which turns out to matter later.

One line in that LiteLLM config is the whole reason this post exists:

That's LiteLLM's own native New Relic integration. It's easy to miss in the docs, and it's the bridge that makes everything in the next section true.

Layer one: what New Relic captures without being asked

New Relic's AI Monitoring feature ships with built-in auto-instrumentation for a specific set of providers — OpenAI's SDK, Bedrock, LangChain, Google's GenAI SDK. Ollama and LiteLLM aren't in that list, so you might expect a router built entirely on self-hosted, non-cloud-API models to fall outside what the feature can see.

It doesn't. LiteLLM ships its own New Relic callback, and that callback reports on every model call the router handles, regardless of which provider sits underneath it — wrapping each one as a real LlmChatCompletionSummary / LlmChatCompletionMessage event. Run the LiteLLM proxy itself under the New Relic Python agent (newrelic-admin run-program litellm ...) and every one of our Ollama-backed calls lands in New Relic's actual AI Monitoring UI, tagged with vendor: ollama_chat and the real model name, indistinguishable in the product from a hosted-API call.

The same agent wrapping gives us the other half of layer one for free: distributed tracing. The gateway and the LiteLLM proxy both run under the New Relic Python agent, so a single request's trace ID survives the hop between them — one waterfall spanning the gateway's chat handler, its outbound call to litellm-proxy, litellm's own middleware, and its outbound call to Ollama. Not three services each reporting in isolation; one continuous trace, the way it would look for any other microservice call chain.

That trace-group summary is the index card; the waterfall underneath it is the actual read. Opening a single real request shows the full chain in order: the gateway's own middleware, the outbound call to litellm-proxy, litellm's middleware stack running span by span (request-size limiting, security headers, in-flight-request tracking, billing metrics, its Prometheus auth check), and at the bottom, the call that actually left the building — ollama:11434 — with its own measured duration next to it. Every hop is a real span from a real request, not a diagram we drew to explain the architecture.

Layer two: what has no schema yet

AI Monitoring answers “what model got called and how long did it take.” It has no concept of why the router picked that model, or whether the response it got back was any good — because those are specific to this system, not something a vendor auto-instruments. That data still deserves to be first-class in New Relic; it just has to get there as a custom event instead of a native one.

So the gateway emits two: LlmRoutingDecision (complexity score, the specific signals that fired, the tier chosen) and LlmResponseEvaluation (a fast heuristic score — empty/refusal detection, latency against a per-tier budget, length sanity for the tier — explicitly not an LLM-as-judge). Both feed a purpose-built NerdGraph dashboard: request volume and tier mix, a complexity-score histogram, the most common routing reasons, eval-score trends, and a flags breakdown.

Building that dashboard surfaced a real NRQL quirk worth knowing about before anyone else hits it: FACETing on a continuous float attribute silently truncates to an integer bucket. A real eval_score of 0.8 shows up as facet "0" — no error, no warning, just quietly wrong data on the chart. histogram(eval_score, 1, 10) is the fix; FACET is for discrete values, not scores.

Two more pieces of this fall out almost for free, and both are the kind of thing that only matters the day someone asks for it. Every time the circuit breaker engages later in this post, it lands as a New Relic change-tracking event on the gateway's own entity — the same primitive a code deploy uses — so a routing regression is traceable back to "what changed" exactly the way a bad deploy is, instead of living only in a Slack thread someone has to remember existed. And because every tier in this build runs on the same local Ollama instance with no cloud fallback configured, the setup is single-vendor by construction — worth stating plainly rather than discovering during an outage, since a routing layer that can only ever reach one model provider has a business-continuity exposure baked in, whatever that provider is.

The part nobody puts in the diagram: local inference is slow, and that's worth showing

Running every tier on Ollama, with no cloud fallback, means the complex tier's cost is real and worth being honest about. A cold qwen2.5:7b — the very first request after the container starts, before anything is loaded into memory — measured 84 to 126 seconds on CPU-only inference. Warm, the same model answers in about a second for a short prompt. For a genuinely detailed answer, even warm, we measured around 125 seconds — token generation on a 7B model without a GPU is simply slow, and a longer answer means more tokens.

We mitigated the cold-start problem operationally — the Ollama container now sends one warm-up request per model immediately after pulling it, and OLLAMA_KEEP_ALIVE=60m keeps every tier loaded between requests instead of the 5-minute default — but we didn't mitigate the second number, and deliberately didn't tune the evaluator's tier-3 latency budget upward until the flag stopped firing. It's set to 90 seconds, calibrated to what we actually measured, not to what would look good on a dashboard. A detailed complex-tier answer still trips latency_over_budget in the eval events, and we think that's the right outcome: it's New Relic telling you, accurately, what running a bigger model fully local and offline actually costs.

Closing the loop: when the router itself needs an intervention

Everything so far is New Relic watching the system. The more interesting question — and the one that led to this blog being written up — is what happens when New Relic sees something worth acting on. Not “alert someone,” but actually change the system's behavior, with a human still approving the change.

This is also where most AI observability stops. A dashboard that shows token spend, a trace that shows which model answered, an eval score trending down — all genuinely useful, and all still just information sitting in a UI until a person notices it, opens a terminal, and does something. Very few tools in this space carry a signal any further than a notification. We wanted to see what it looks like when the same platform that noticed the problem is also the one holding the lever to fix it — with a human still in the loop, not cut out of it.

The trigger is a NRQL alert condition: CRITICAL when two or more LlmResponseEvaluation events pick up a quality flag within five minutes. That's deliberately easy to hit in a live demo; two complex-tier prompts back to back reliably trips it on this hardware.

Wiring the alert to an actual response took a specific, previously-broken path: New Relic's WORKFLOW_AUTOMATION-typed notification channel has a known channel-attach bug on this account, but a generic WEBHOOK-typed destination works cleanly. That destination points at a new endpoint on the gateway itself: POST /alert-webhook — rather than a separate Lambda, since the gateway is already a persistent server with nothing serverless to work around. The webhook receiver checks a bearer token, gates on CRITICAL priority, and calls workflowAutomationStartWorkflowRun.

That kicks off a Workflow Automation canvas that does three things in order. First, a real Autopilot investigation: a newrelic.agent.run call (agentId: sre_agent) against the gateway's APM entity, returning a genuine, per-run LLM-generated analysis — not canned text, and not guaranteed to succeed, so a live NRQL query pulling the actual flag breakdown runs alongside it as a fallback, and the Slack message that follows labels its source honestly either way. Second, the investigation gets posted to Slack with a request for a ✅ reaction, and the workflow genuinely waits — no reaction within the timeout, no change. Third, on approval, an http.post step calls the gateway's own POST /admin/routing-override, which forces every request onto tier-1-fast regardless of complexity score until someone clears it — a real circuit breaker, not a notification with extra steps.

The reaction-gate step isn't decoration. An Autopilot agent that can call an internal endpoint and change routing behavior is, by definition, an autonomous agent taking a real action against production — exactly the shape of risk OWASP's LLM Top 10 calls Excessive Agency (LLM08): a model doing something consequential with no checkpoint before it happens. The fix isn't clever; it's a single required human reaction on a Slack message, and it's the difference between an agent that recommends and an agent that just acts.

We ran this whole chain for real, not as a tabletop exercise: a synthetic alert payload, a real workflow run, a real Slack message, a real human reaction, a real HTTP call back into the gateway over a public tunnel, the circuit breaker actually engaging, and a real change-tracking marker landing on the entity afterward. Every hop in that chain is a real system call, not a simulated one.

Taking it off the laptop

Everything above was built and verified locally first, then deployed to a single EC2 instance to confirm it holds up somewhere that isn't a laptop with warm caches. A few real things broke in the move, worth naming because they're the kind of gap that only shows up under real network conditions: the base docker image's bundled tooling didn't include a new enough buildx to build multi-stage images at all, and nginx's default 60-second proxy timeout silently cut off a real complex-tier request that the gateway itself was still waiting on — local testing never caught it because most local testing went straight to the gateway's own port, bypassing nginx entirely.

The instance itself is deliberately narrow: no SSH key pair and no open port 22 at all — shell access is Session Manager only, through an IAM role scoped to exactly one managed policy. The security group opens a single port, the one nginx listens on; the gateway, litellm-proxy, and Ollama's own ports never leave the Docker network.

What we'd tell someone building this

A short list of the concrete, sometimes-surprising things this build turned up, in the order we hit them:

• The provider gap in AI Monitoring's supported list doesn't mean the provider can't show up there.  LiteLLM's own callback is the bridge — check whether the layer between you and the model has a native integration before assuming a self-hosted model is invisible to AI Monitoring.
• Application registration inside a New Relic integration can be asynchronous.  LiteLLM's newrelic callback registers the application without blocking; traffic sent within the first couple of seconds after startup can race ahead of that registration and silently vanish, no error, no warning unless verbose logging is on.
• FACET on a continuous float truncates to an integer bucket.  Use histogram() for anything that isn't naturally discrete.
• A generic WEBHOOK notification destination is a more reliable alert trigger than the type built for exactly this purpose.  The WORKFLOW_AUTOMATION channel type has a known channel-attach bug; a plain webhook into your own receiver sidesteps it entirely.
• Cold-start latency on CPU-only local inference is not a rounding error.  84-126 seconds for a first response from a 7B model is a real operational number, not a benchmark footnote — budget for it, or warm the model before anyone's watching.
• Local testing that bypasses your own reverse proxy will miss real reverse-proxy bugs.  The 60-second nginx timeout only showed up once requests went through the actual public path.
• Whatever captures your traces is also a data-governance decision, not just an observability one.  Several popular AI tracing conventions capture full prompt/response content by default — decide deliberately whether that content belongs in your telemetry before the first real request goes through, not after.

Where this leaves us

Model routing is a cost and latency optimization. On its own, it's also a new blind spot: a system making autonomous decisions about which model handles which request, with no obvious place to see those decisions, their quality, or their cost — the exact blind spot behind every failure mode earlier in this post, from a silent model regression to a chatbot's employer left holding legal responsibility for what it said. The point of this build wasn't to prove New Relic can watch an LLM call — that part was expected. It was to find out how much of the actually-interesting surface area (the routing logic, the response quality, the moment something needs to change) New Relic could reach without inventing a new product category to do it, and then to close the loop with an intervention that's real rather than illustrative.

The answer, concretely: native AI Monitoring reaches further than its supported-provider list suggests, distributed tracing doesn't care whether the model is self-hosted, custom events make the routing-specific gaps — and the governance ones, cost, vendor concentration, change history — first-class rather than invisible, and Workflow Automation plus Autopilot can turn a quality signal into an actual corrective action with a human still in the loop, at the exact point where most AI observability setups stop at a chart. All on a routing pattern that, left uninstrumented, would have shipped with nothing watching it at all.