Introducing live archives: Instant access to historical logs to reduce compliance risks

Today, we are unveiling the general availability of a game-changing, long-term log storage solution that’s cost-effective and queryable instantly. Now, you can easily manage compliance risks while meeting regulatory requirements using New Relic live archives.

Why do you need logs for compliance?

In today's tightly-regulated business environment, long-term log retention is not just a best practice but a legal imperative. This imperative stems from various regulatory compliance requirements such as the Health Insurance Portability and Accountability Act of 1996 (HIPAA), U.S. Executive Order 14028, Basel II Accord, Center for Internet Security (CIS) Critical Security Controls, Payment Card Industry Data Security Standard (PCI DSS), Sarbanes–Oxley Act (SOX), and Systems and Organization Controls 2 (SOC 2), each tailored to specific industries. For instance, in healthcare, HIPAA mandates a six-year retention period for records, while in the banking sector, the Basel II Accord requires log retention for three to seven years. The stakes of non-compliance are high, encompassing not just heavy financial penalties but also severe legal and reputational consequences. 

These logs aren’t just mere operational information; they’re the cornerstone of regulatory compliance, ensuring businesses meet the rigorous standards set by their respective industries. However, the challenge continues beyond just retaining these logs. Businesses face the daunting task of managing the significant costs and complexities associated with storing and accessing these vital records for compliance and audit purposes. This situation poses a pressing question: how can organizations efficiently and cost-effectively handle these essential yet challenging requirements?

The challenges of traditional long-term storage solutions

When it comes to long-term log storage, you confront a critical choice between hot and cold storage—each with its own benefits and drawbacks. Hot storage, preferred for its speed and accessibility, is ideal for frequently-accessed logs and real-time analytics. However, hot storage is costly over the long term and is often three times more expensive than cold storage. In contrast, cold storage is more affordable for infrequently accessed data but has slower retrieval times and requires complex data management. Accessing data in cold storage is toilsome and time-consuming, involving rehydration, reloading, re-indexing, and moving data between multiple locations or tiers, often resulting in hidden costs and surprise bills. 

In essence, while hot storage excels in ease of use, it becomes cost-prohibitive for long-term log storage. On the other hand, while initially more cost-effective, cold storage can become a maintenance and query nightmare, riddled with hidden costs and operational hassles. As a result, you face a painful tradeoff between costs and ease of use, which begs the question: how can we bridge the gap between cost efficiency and ease of use in long-term log storage?

New Relic live archives overcomes the challenges of traditional solutions

New Relic live archives is a game-changing solution that combines affordability, ease of use, and performance. It helps you comply with industry regulatory requirements by offering a unique storage capability, storing logs in an active, hydrated, and enriched state for up to seven years. As a result, you can analyze your historical logs more quickly and effectively without the hassles of rehydration, reloading, re-indexing, or moving data to multiple locations or tiers. Moreover, priced at just a quarter of other long-term storage solutions, live archives is currently the most cost-effective long-term log storage solution available in the market. 

Key benefits of live archives:

• Access historical logs instantly: Because your logs never leave New Relic Database (NRDB), you get seamless, hassle-free query access to your data without delay. Your historical logs are managed and accessed as quickly as any other logs using the same user interface that you do for other queries—no extra steps required.
• Eliminate toil: No need for manual and time-consuming steps of traditional solutions, such as rehydrating, reloading, re-indexing logs, or moving data between multiple locations or tiers for analysis. No need to bring your own storage solutions (such as S3 buckets) or pull historical logs separately.
• Reduce costs: Say goodbye to the additional costs associated with traditional long-term log storage solutions, such as the need to pay for ingress and egress, indexing, reformatting log data, or licenses for additional log tools.

How to set up and use live archives

New Relic live archives will revolutionize the way you approach long-term log storage and analysis, now streamlining the process into just a few simple steps: 

• Send logs: Send your logs to New Relic, as usual, giving you observability of your applications, infrastructure, and more.
• Set retention periods: Determine your desired retention period(s) to meet your compliance requirements. Your New Relic account administrator can adjust the retention settings for the “Logging live archives” namespace through the data management interface (see image below). You can choose a retention period from any number of days up to seven years. After your logs exceed the standard retention period (which could be 30 days or longer, depending on your plan), they automatically migrate to live archives storage, where they remain secure and easily accessible for the rest of your specified retention time.

• Choose which partitions to archive: You don't need to archive every log for compliance. Your New Relic administrator can select specific log partitions for archiving and set a default partition for this purpose. Utilizing a NRQL WHERE clause, you can define a subset of your logs to be stored for compliance in order to minimize storage costs. After selecting your desired partition, any new logs data in that partition will automatically transfer to live archives storage when it reaches the designated "age out" period.

• Query data in the logs UI: Live archives logs are queryable alongside your other logs and the rest of your telemetry data, eliminating the data silos you may have dealt with using other long term storage solutions.

• Manage user permissions: By default, we’ve disabled querying live archives for users because doing so incurs querying costs that exceed standard data queries. But when you decide which users you want to give query access to live archives, your New Relic administrators can easily grant query permissions by assigning custom roles.

To learn more about live archives, read the documentation, and watch the Data Bytes video provided at the end of this blog.