When you send data to an observability platform, you have to carefully manage that data. For example, your company likely collects some form of sensitive information, governed by laws and regulations such as PCI or GDPR. In order to remain compliant, you need to ensure that you don’t pass on certain types of data or very specific attributes, like personally identifiable information (PII).
Or, let’s say you collect telemetry data from third-party sources, like Prometheus or DropWizard, but you only want to send certain parts of that data to your platform. Changing the behavior of those sources to send less data may require you to modify or manage multiple client-side configurations, and that only creates unnecessary toil.
As an open platform, New Relic can ingest high volumes of metrics, events, logs, and traces from New Relic agents and third-party data sources. This means you can send massive amounts of data to New Relic. But at the same time, you need to manage that data appropriately, as well as its associated costs. We don’t want to see data overages anymore than you do. Yet we want to give your teams the confidence to make the most of New Relic’s open platform while also allowing you to manage the data you send, whether for volume or compliance purposes.
How to drop data
Dropping data gives you control over the data that you send to New Relic. Data is dropped at ingest, which means it's never stored in the New Relic Database (NRDB).
By dropping specific events or attributes from events, you determine the data that is ultimately stored in NRDB for you to later query, alert on, and analyze. More specifically, dropping data helps you:
- Minimize costs. Drop low-value data that isn’t useful to analyze, so that you aren’t at risk of data overages.
- Manage sensitive data. Drop personally identifiable information (e.g., credit cards or social security numbers) and other types of data that you don’t want stored in NRDB.
There are two ways to drop data:
- Drop entire data types or a data subset (with optional filter).
This uses the
DROP_DATAaction type and uses NRQL of the form:
SELECT * FROM DATA_TYPE_1, DATA_TYPE_2 (WHERE OPTIONAL_FILTER)
- Drop attributes from data types (with optional filter).
This uses the
DROP_ATTRIBUTESaction type and uses NRQL of the form:
SELECT dropAttr1, dropAttr2, FROM DATA_TYPE (WHERE OPTIONAL_FILTER)
To learn about requirements and how to configure rules for dropping data, check out the documentation.
To learn more about how New Relic manages your data, don't miss Inside NRDB: A Flexible, Unified Database Built to Scale.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.