How errors are represented in New Relic
Errors a client sends to New Relic will be accumulated into an error rate chart accessible on a given service summary page, along with other helpful information to monitor the health of that service.
New Relic generates this graph with error data derived from metrics with an `error.type` attribute. If there is an error in an OTel metric like `http.server.request.duration` or `rpc.server.call.duration`, it will have this attribute.
Errors sent over to New Relic via span data are marked with a status code of `ERROR`. These error spans drive the Errors Inbox experience, which assists users in detecting, triaging, and fixing errors.
New Relic Language Agent Error Filtering Functionality
New Relic offers several server-side configurations, including those related to error collection, as opposed to configurations set in a New Relic user’s language agent configuration file. NR language agents can therefore be configured to ignore certain errors by HTTP status code or error class. The result is that errors are still sent to New Relic, but are prevented from affecting error rate or appearing in errors inbox.
Why is Ignoring Errors Desirable?
In most applications, certain exceptions or HTTP responses—such as a 500 Internal Service Error—are expected behaviors rather than actual backend system failures. If these routine, non-actionable events are continuously recorded as critical errors, they can artificially inflate error rates, distort service level indicators (SLIs), and trigger false alarms for on-call engineers.
Ignoring these specific errors is important because it helps teams maintain visibility and prevent alert fatigue. However, instead of blindly discarding valuable debugging data, selectively updating the span status code and metric attributes can prevent the event from negatively impacting golden signals, while still preserving the underlying trace data for debugging when needed.
How to Configure the OTel Collector to Ignore Errors
Alternatively, New Relic users can set up the OpenTelemetry Transform Processor to modify error data before exporting it to New Relic. The transform processor supports more types of transformations than just the span status code, and can execute multiple transformations to each qualifying bit of telemetry data at once.
As mentioned before, changing span status codes from `ERROR` to `UNSET` effectively marks the error ignored so that it will not clutter the errors inbox.
transform/traces:
error_mode: ignore
trace_statements:
- context: span
conditions:
# Match spans from this service
- resource.attributes["service.name"] == "ignored-errors"
# With HTTP 500 status code
and attributes["http.response.status_code"] == 500
# That are server spans (span.kind == 2 is SERVER)
and span.kind == 2
statements:
# Change status from ERROR to UNSET (not an error)
- set(status.code, STATUS_CODE_UNSET)
# Remove error.type attribute
- delete_key(attributes, "error.type")Then, to remove metric error data from the error rate, the `error.type` attribute must be deleted.
transform/metrics: error_mode: ignore metric_statements: - context: datapoint conditions: - resource.attributes["service.name"] == "ignored-errors" and metric.name == "http.server.request.duration" and attributes["http.response.status_code"] == 500 statements: - delete_key(attributes, "error.type")
The Impact
As a proof of concept, I ran the `ignored-errors` example app in the newrelic-opentelemetry-examples repo and sent the following HTTP status codes: five 200s, three 404s, three 500s, and five 503s.
With the new configuration (where HTTP status code 500 is ignored), the summary page boasts a drop in error rate from 50% to 31.25%.
Additionally, 500 errors are no longer found in the Errors Inbox page.
Additional Resources:
For more information on
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
