Manage Costs and Sensitive Data: How to Drop Data in New Relic

When you send data to an observability platform, you have to carefully manage that data. For example, your company likely collects some form of sensitive information, governed by laws and regulations such as PCI or GDPR. In order to remain compliant, you need to ensure that you don’t pass on certain types of data or very specific attributes, like personally identifiable information (PII).

Or, let’s say you collect telemetry data from third-party sources, like Prometheus or DropWizard, but you only want to send certain parts of that data to your platform. Changing the behavior of those sources to send less data may require you to modify or manage multiple client-side configurations, and that only creates unnecessary toil.

As an open platform, New Relic can ingest high volumes of metrics, events, logs, and traces from New Relic agents and third-party data sources. This means you can send massive amounts of data to New Relic. But at the same time, you need to manage that data appropriately, as well as its associated costs. We don’t want to see data overages anymore than you do. Yet we want to give your teams the confidence to make the most of New Relic’s open platform while also allowing you to manage the data you send, whether for volume or compliance purposes.

How to drop data

Dropping data gives you control over the data that you send to New Relic. Data is dropped at ingest, which means it's never stored in the New Relic Database (NRDB). 

By dropping specific events or attributes from events, you determine the data that is ultimately stored in NRDB for you to later query, alert on, and analyze. More specifically, dropping data helps you:

1. Minimize costs. Drop low-value data that isn’t useful to analyze, so that you aren’t at risk of data overages.
2. Manage sensitive data. Drop personally identifiable information  (e.g., credit cards or social security numbers) and other types of data that you don’t want stored in NRDB.

To drop data, create a NerdGraph-format drop rule that includes an NRQL query that specifies what data types to drop at ingest. You can form and make the call in the New Relic GraphiQL explorer.

There are two ways to drop data:

1. Drop entire data types or a data subset (with optional filter).

   This uses the DROP_DATA action type and uses NRQL of the form:

   SELECT * FROM DATA_TYPE_1, DATA_TYPE_2 (WHERE OPTIONAL_FILTER)

2. Drop attributes from data types (with optional filter).

   This uses the DROP_ATTRIBUTES action type and uses NRQL of the form:

   SELECT dropAttr1, dropAttr2, FROM DATA_TYPE (WHERE OPTIONAL_FILTER)

To learn about requirements and how to configure rules for dropping data, check out the documentation.

To learn more about how New Relic manages your data, don't miss Inside NRDB: A Flexible, Unified Database Built to Scale.