New Relic and Grafana Labs CEOs Discuss Partnership to Bring Observability to All

Editor’s note: Throughout this interview, references are made to recently added support for PromQL-style queries in New Relic’s Telemetry Data Platform. To clarify, we support PromQL-style queries by translating them to their closest approximation in NRQL. While we’re continuing to improve our level of support for PromQL, there are cases where translation between languages may lead to different query results. For more information, read about our supported PromQL features.

I’ll admit it, I’m an open source fanatic. I firmly believe that collaborating with the open source community is the best way to help developers and engineers understand their digital systems better. And I couldn’t be more excited about the New Relic-Grafana Labs partnership we announced today. I sat down with New Relic’s CEO Lew Cirne and Grafana Labs’ CEO Raj Dutt to find out more about what developers can expect and how they’ll benefit from the partnership. 

Joel Worrall: Raj, first I want to say congratulations to you, and to the entire Grafana Labs team, on building an incredible and popular open source tool for observability. Can you talk to us a little bit about Grafana and how developers end up encountering the open source project?

Raj Dutt: Sure. The Grafana project was created by our co-founder, Torkel Odegaard, in Stockholm, about six years ago. And it's really grown into the most popular visualization software for bringing together all your observability data, wherever it lives. So Grafana provides some really rich visualization capabilities and the ability to unify and contextualize all your observability data, whether that's metrics, logs, events, or traces. And Grafana's completely database-agnostic, so we believe that organizations get the best view of what's going on when they pull in their data from wherever it lives. And we're super excited that that data now includes New Relic for open source Grafana users.

Joel: And Lew, obviously you're very passionate about building fantastic software and enabling developers to do the same. Can you talk just a little bit about the announcements that New Relic made recently around open source and the platform and pricing model, and how that impacts developers who would be using tools like Grafana?

Lew Cirne: Sure. We're incredibly excited about, first of all, this increased level of partnership with Grafana. And that's all part of a strategy we rolled out over the last couple weeks. We reimagined our New Relic One platform, and the whole philosophy behind it is simplicity. We've gone from 11 products to just three. As the company that created APM, we are now ending it. You can't buy APM from New Relic anymore, believe it or not. You can buy one of our three products. Those three products are Telemetry Data Platform, Full-Stack Observability—which includes all the APM functionality as well as all the other monitoring products we've had—and the third is our Applied Intelligence product.

Our partnership with Grafana really centers on our Telemetry Data Platform. Think of that as a world-class petabyte-scale database-as-a-service that operates on and consumes all metrics, events, logs, and traces into a single database with a single query language. Most other databases are optimized for, say, logs or metrics. And so we've had a lot of customers say, "I'd love all that data in one database at petabyte scale. And I'd love for someone else to manage that instead of me having to manage that kind of massive amount of data."

And virtually every one of our customers love Grafana. It's everywhere, in our customer base and beyond. And so when we decided to make Telemetry Data Platform our first-class product, we knew that it had to work great with Grafana. We made that a core requirement for launch. So we reached out to Raj and the team. We want to make sure that the Grafana team felt like the integration and the capabilities we offer work well. And we're delighted that that's the case. And we're excited to be announcing it in partnership with Grafana this week.

Joel: Raj, can you talk a little bit about how this partnership actually impacts the experience in Grafana? And what sort of capabilities exist now for interacting with the Telemetry Data Platform in New Relic?

Raj: So there are really two levels, Joel. For the open source Grafana users, we're really excited that New Relic now supports a PromQL-like query language on its Telemetry Data Platform. We've seen tremendous growth of the Prometheus community within the Grafana community. In fact, Prometheus is now the most popular metrics database amongst Grafana users. And Grafana Labs is one of the main contributors to the Prometheus project. So we're really involved in that ecosystem.

Essentially, since New Relic now supports PromQL on its telemetry backend, New Relic joins the ranks of companies, including Grafana Labs, that natively support Prometheus on our backends. So essentially, developers have spoken and they've chosen Prometheus, particularly within the Kubernetes and cloud native ecosystems. And so I think this move by New Relic really kind of recognizes, in my opinion, the tremendous growth and popularity that Prometheus has achieved. And Grafana is the de facto visualization software for Prometheus. So one of the most exciting things is that New Relic is another really good Prometheus backend that gives developers choice in terms of where to store their Prometheus data, right? They can continue to use open source Prometheus, of course, or they can use the new Telemetry Data Platform from New Relic.

There are a variety of other open source tools that natively support Prometheus, such as Timescale, InfluxDB, and other cloud providers that support Prometheus, including our own cloud product. So Prometheus has really become a de facto standard, and I think this really bodes well for the Prometheus community because New Relic is now native Prometheus also. So that's what's really exciting.

Joel: Can we talk nuts and bolts for a bit? So if I'm a developer collecting my Prometheus metrics, how do I actually go ahead and get that information into New Relic’s Telemetry Data Platform? Is it just as simple as a couple of configurations, and then I've got the power of New Relic behind that data? Is that the case or is there more involved?

Lew: It is quite straightforward. Let me echo what Raj said. We love Prometheus. We see its ubiquity; we love how simple it is. Obviously, given how it's part and parcel of, or virtually bundled with every Kubernetes installation, it's going to be ubiquitous, and for good reason. And so we wanted to make it super easy for people who have Prometheus to also have that in a central place along with their events, logs, and traces, and have 13 months of retention, at petabyte scale, across many, many Prometheus installations. And two lines of YAML config can forward those Prometheus metrics into our Telemetry Data Platform, where it can sit alongside all that other data.

It's important to note also that when we launched this platform, we did it with a new viewpoint on how to think about this data. We want it to be as cost effective as possible for developers and engineers to collect all the telemetry data they can without worrying about the cost. So we launched with an offer of 100GB of ingest for free. And then after that, we only charge 25 cents per ingested gigabyte, with no additional costs for running queries or anything like that.

So super simple and very cost effective pricing so that people shouldn't have to worry about whether that data is both inside Prometheus and a copy is in our Telemetry Data Platform. At these economics, you can have a copy of that data and enjoy the benefit of querying it in whatever environment makes sense. We love the idea of Grafana being the front end for those queries. And you can write those queries in PromQL, or in our native NRQL, whatever is most comfortable for you.

Joel: I've got a question for you both about the growth of observability. We're all involved in work where we see it as central to the way good software is built. But it's definitely on a journey. It's not as ubiquitous as, say, source control at the moment. So I just wonder for you both, as people who are leading in this space, what kind of role do you think we're each playing in helping promote the idea of observability in your code as something that should be a first-class tool for a developer? What are the things we're doing, first of all, to promote that? And then second, what more could we be doing?

Lew: I'll start with that. I think Grafana has really led the way because a core element of that has to be open source. In order for observability or anything to be ubiquitous with developers, there has to be a strong open source story. That includes source code control and the rise of Git, for example. And that's why we've fully open sourced all our agents. Which would have been unthinkable, say, 10 years ago, as the agents used to be thought of as crown jewel technology. Now we're contributing them—all of our agents—to open source. And already, within hours of open sourcing our agents, we received PRs to make them even better. And we're fully embracing the leading open source projects and technologies such as Grafana, Prometheus, to make observability ubiquitous, as it needs to be.

The other thing we're doing is removing the second primary barrier to ubiquitous observability, which is cost. Prior to what we announced a couple of weeks ago, the way you got observability, from a commercial vendor at least, was by paying by the host. Every host you rolled out in production, you paid a tax on that host. If you double the number, you were doubling your observability bill. We felt like that's just not right. Our customers need that data, but that data shouldn't be priced at such a premium.

So with the rollout of our pennies-per-gigabyte approach, we want our platform to be cost effective. In fact, we'd like it to be even more cost effective than hosting your own free open source database at the scale that we deliver. If we're not doing that, then people will turn to a less expensive alternative.

With those economics, we think we’re making it more reachable to a broader set of developers who should be embracing observability just like they embraced source code control or bug database, or use an IDE or vi—for those of you who love vi and Emacs. But we think it deserves to be at that level of ubiquity.

Raj: First of all, I'm not going to touch the vi versus Emacs debate with a 10-foot pole. But other than that, I totally agree. Obviously at Grafana Labs, we're pretty biased on open source. We started our company as an open source company six years ago, and everything from Grafana to the high-scalability Prometheus backend that powers our cloud is fully open source, so we're all in on open source. And we really believe, like we said, that it's key to modern observability. Developers are “The New Kingmakers” as Stephen O’Grady said. And I think the action in the last few years within the open source observability community has been really phenomenal. It used to be, say, 10 years ago, that open source was kind of the cheap and cheerful alternative to a lot of the commercial vendors.

And now, and again, biased here, but with things like Grafana and Prometheus, that's where a lot of the cutting edge activity is happening. And Prometheus in particular has gotten really efficient to run. So pricing for service providers is definitely paramount, right? Because it's so efficient that it can be a pretty tall order for a company to pay expensive per-host pricing when they're already enjoying the efficiency of Prometheus open source. We've definitely had to think about that in our own cloud offering, and we really believe in open source. Developer mindshare is so important, in general, and open source and open communities are really the way to unlock and achieve that. For us, everything we do from Grafana itself to our own Prometheus cloud backend Cortex is fully open source. And we believe we're about connecting customers with communities more so than anything else.

Joel: I'm going to ask you both a question about the future. Over the last several years we've consolidated on what the data set looks like, the types of things that we need to measure and manage to get a complete picture of what's happening in our software. And we've standardized on some visualization tools. We've now got some offerings out there, particularly at scale with what New Relic's doing with the Telemetry Data Platform that makes it possible to manage this data at extremely high volumes. I wonder, what's the future look like? What's the next thing in observability that is relevant and important for developers to be successful?

Lew: This one almost sounds a little bit trite because it's been used in hype cycles for so long, but there are AI opportunities with this data so that developers can, with machine learning models, automate and predict things that right now humans are doing way too manually. I don't believe that in the next five years we're going to have AI replicate what a first-class SRE does in troubleshooting production systems. There's a reason why these engineers are so talented and so hard to find. I don’t want to overplay the hand with what AI will do, but there has been more and more that AI is and will be doing around reducing noise on alerts, correlating problems, accelerating time to diagnose and fix some root cause.

But the other thing being talked about less is there is so much value in the data being collected. I think there are opportunities for new classes of applications to be built on the data. We have a fondness for dashboards. We have our own, and dashboards are the primary way in which this data is being presented to people. I'm excited about how you make it easier for developers to build new applications on top of the data. If you can just write a React component, you can do more than present a visualization. You can actually write a rich application that might be able to show you new things about your digital business and/or interact with the digital business in a way that might cost more than 10 times to build if you're doing it from scratch. So we're focused on that as well.

Raj: I agree the hype around AI and machine learning is pretty high. We joke that every time someone mentions AI or ML, that person has to do a shot of vodka, which isn't a good rule at Grafana Labs because it greatly reduces our productivity. That being said, one of the things we're really focused on is to Lew's point. There's so much data coming out of these complex distributed systems that we're really focusing on the workflow and the troubleshooting experience and the journey from metrics to logs to traces, right? We released an open source project called Loki about a year and a half ago that's kind of like Prometheus but for logs. We're about to release another open source project in the tracing space.

We're really trying to connect a journey and make it a seamless experience for people to get to the root of an issue by transitioning very easily between metrics and logs and traces. We really want to optimize for the poor SRE who's gotten paged at 3 a.m. and she's trying to troubleshoot a production issue blurry-eyed and having to log in to all these different tools. The experience and the unified context is something we're really focused on: optimizing for 3 a.m. and preserving an optimal flow in that experience. That’s something we're really interested in a lot.

Joel: There's one more item in the partnership and what's being announced that I want to make sure we highlight, as well, in relationship to the enterprise and commercial offering that Grafana Labs has available. Raj, could you talk about that for a moment, please?

Raj: Absolutely. We've talked a lot about how open source users can use the built-in support for PromQL and Grafana to talk to the New Relic Telemetry Data Platform. But in addition to that, Grafana Labs has a native NRQL plugin that's available in the enterprise version of Grafana, and all New Relic customers get a 30-day free trial of Grafana Enterprise. Grafana Enterprise has data sources to New Relic and to dozens of other commercial vendors in addition to a lot of other features purpose-built for really large organizations around things like security and scalability.

Essentially, New Relic customers can spin up a 30-day free trial of Grafana Enterprise and then interface with New Relic not only over PromQL, but over NRQL, and also bring in all their other data from a variety of commercial vendors.

Joel: I love that idea. This is what a real partnership looks like: where we're focused on the person we're trying to serve in both cases—the developer. If they're in New Relic now, they can query data through PromQL, or use NRQL in the other direction with Grafana, to be able to engage with their information in a way that makes sense to them as developers. That's a great thing to be able to offer. So, I just want to close out by asking each of you, is there anything else you'd like to share with regard to trends or what’s happening in open source?

Lew: I am just incredibly excited about the partnership that we're announcing with Grafana Labs and how we're embracing Grafana as a first-class frontend to our Telemetry Data Platform. We believe our Telemetry Data Platform allows our customers to consider us as a single source of truth of all their operational data. And to do that at far cheaper cost and far better query speeds than they would do if they were scattering that across a bunch of backends.

But we realize that, in reality, customers have a lot of backends, many of them being Prometheus, and so we're not asking our customers to make a switch or all-at-once decision. They can adopt our Telemetry Data Platform at whatever pace they want, and they can continue to use Grafana as a fantastic frontend to our data store as well as others. We just want that to be a fantastic experience for every developer.

So Raj, it's been a real treat. Every interaction I've had with you and with the team at Grafana Labs has been great from our perspective, and we're just thrilled to be announcing this partnership.

Raj: We're also really psyched about the partnership overall. Grafana has always been about connecting your data, so unifying your data wherever it lives, whether that's in your own on-prem open source software. Whether that's in Grafana Labs' cloud offering. Whether that's in New Relic's offering or other vendors' backends.

And we're really excited about the New Relic partnership in particular, because now New Relic supports Prometheus and has a really innovative pricing model. So we think that they're going to be a great option for our users. And because they're a native Grafana data source that speaks PromQL, they're really easily combined with a customer's data wherever it lives. And that's really exciting.

The fact that New Relic has open sourced its agents and gotten behind Prometheus is both really smart and will be welcomed by the open source and Grafana communities. And it's really great to see this level of innovation.

In fact, I have a picture that I'll send you maybe later, Joel, of my co-founder Anthony and I standing outside the New York Stock Exchange the day New Relic went public. We've always admired New Relic as a company and it's really cool that six years later, we can announce this partnership between our organizations.

Joel: That's fantastic. Thank you both so much for taking a few minutes to talk about this.