With New Relic Data Plus, you’re now able to export your historical and real-time streaming telemetry data directly to external sources for your compliance and data analysis needs. Historical data and streaming data exports allow you to explore data more deeply across related entities, events, and anomalies within your tech stack. You can also upload results to your external analytics, AI, or machine learning tools to maximize the value of your New Relic data.
In this blog, you'll learn how to export your data and help you learn when to use historical exports or streaming exports. But first, a quick overview of Data Plus, which is a requirement to access the data export capabilities.
Data Plus: Empowering your data through powerful observability enhancements
New Relic Data Plus gives you even more powerful observability capabilities to enhance your security, scale, and performance, supercharging your observability practice. Features in Data Plus include data export, extended data retention, logs obfuscation, FedRAMP for enabled accounts, longer-running queries, and more. To learn more about Data Plus, explore our announcement blog.
When to choose a historical or streaming data export
Historical data exports allow you to export data from more than 12 hours ago while streaming data exports allow you to export data from the last 12 hours. You can also use a streaming data export to continuously export data to another source.
Historical data export
A historical data export sends existing data past initial ingestion out of New Relic. This type of data export offers unlimited query durations and up to 200 million data point rows, overcoming the standard NRQL query limitations. You’ll build queries in NerdGraph with the querying data returned as a JSON file in an Amazon S3 bucket. The historical data export feature doesn't support querying live data or data from the preceding 12 hours. The time range must end at least 12 hours in the past.
These exports are helpful when you need to examine any data from the past. Maybe you have a new or updated security or compliance rule that requires saving or analyzing past data. Or maybe you need to provide data to someone who doesn’t have access to your New Relic account.
Streaming data export
A streaming data export sends out data exclusively at ingestion by New Relic, so any data from the past 12 hours since ingestion. Streaming data is exported via Kinesis Firehose. You can set up custom NRQL-based rules through NerdGraph that govern which types of New Relic data you’ll stream through New Relic’s streaming data export.
You’ll use a streaming data export when you want to look at data at the time of ingestion. For example, you may want a steady stream of metrics and log data, so it can populate your data lake for easier analysis. Or maybe you want a continuous data export for long-term retention for your compliance, legal, or security needs, so you don’t have to manually run historical data export queries each time.
How to export historical data
Let’s walk through an example of how to export historical data. This example assumes that you have already taken the NerdGraph explorer tutorial to learn how to retrieve data, use mutations, and run queries from the terminal.
Let’s create a historical data export of query results that show all logs of the entity during the incident so you can troubleshoot. To start, use the following command to run a query, which will output results you can export. Make sure to customize the code so it has your own account information, entity id, and times.
mutation {
historicalDataExportCreateExport (accountID: YOUR_ACCOUNT_ID, nrql: “FROM Log select * WHERE entity.guid = “YOUR_ENTITY_ID” SINCE ‘YOUR_BEGIN_DATE_TIME’ UNTIL ‘YOUR_END_DATE_TIME’”){
percentComplete
nrql
status
id
message
}This results in a query ID that will be used as the source and entity reference for the export.
historicalDataExport{export(id: "YOUR_QUERY_ID_HERE")
Next, export your historical data using the following command:
historicalDataExport {export(id: “YOUR_QUERY_ID_HERE”)
This process can take some time but it runs asynchronously without being subject to HTTP or browser timeout interruptions.
The JSON result files expire after a week after it's generated. Each URL is live for six hours before it expires. These links can be generated again after expiration by re-querying.
For more historical data export information, read the detailed documentation or watch this two-minute Nerd Byte.
How to export streaming data
Exporting streaming data in real-time requires you to set rules via NerdGraph about the specific type of data to be exported. You’ll need to create a mutation that makes a rule about streaming data, including the rule name, a description of what data the rule is specifying for export, and the actual query that returns the data being exported.
Here’s an example of a mutation:
mutation {
streamingExportCreateRule(
accountId: account id here,
ruleParameters: {
description: "rule description",
name: "rule name",
nrql: "SELECT * FROM NodeStatus"
},
awsParameters: {
awsAccountId: "aws account id here",
deliveryStreamName: "firehose-stream-name",
region: "us-east-1",
role: "firehose-role"
}
) {
id
status
}
}Enabling and disabling your streaming export rules
Enabling and disabling your rules is an important way to manage your real-time exports without having to constantly create or delete new rules. This can be useful in cases where you’re only interested in data during a certain time period or incident and otherwise wouldn’t need to export it. Note that rules can only be enabled after they’ve previously been disabled. Otherwise, you’ll get an error message.
To enable a rule, use the following mutation:
mutation {
streamingExportEnableRule(id: “query id here”) {
id
...
}
}To disable a rule, use the following pattern:
mutation {
streamingExportDisableRule(id: “query id here”) {
id
...
}
}For a deep dive into managing your streaming data export rules including updating rules, deleting rules, and other best practices, read the documentation about streaming data exports.
Next steps
- Learn what is included with Data Plus and upgrade today.
- Watch this Nerdbyte video on exporting historical data and this Nerdbyte video on exporting streaming data.
- Read additional documentation on historical data export and streaming data export.
If you’re not already using New Relic, get started with New Relic for free. Your free account includes 100 GB/month of free data ingest, one free full-access user, and unlimited free basic users.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
