Life at New Relic: More Than Monitoring (Part 2)

As part of our “Life at New Relic” blog series, we recently sat down with Josh Biggley, an Ops Strategist on the TechOps Strategy Consulting team. In this role, Josh and his team, with experience spanning telecoms to tech startups, healthcare to managed services, meet with customers and prospects to strengthen the adoption of fellow Ops SMEs. Josh joined New Relic in November 2019 from Cardinal Health, where he was a Senior Engineer on the Enterprise Monitoring team.

This two-part blog series (read part 1) explores why democratized access to data is essential, why moving from monitoring to observability gives you confidence, and more.

New Relic: Is there anything about this moment in time that makes observability particularly important?

Josh Biggley: I think we finally see a convergence of egos. Historically, we’ve seen the monitoring folks who have always said monitoring has been the right way. Developers would say, "Well, we've got our tools, and our tools are cooler." And the business people saying: "Those technical people, all they want to do is spend money."

Those egos have clashed in the past. Observability says we can bring together all of the data, regardless of where it comes from, and we can start to have conversations. Our CEO Lew Cirne refers to this common data set as a common source of truth. Business people talk about dollars and cents; developers talk about things like... I don't even know what developers talk about. And then the infrastructure people, we talk about things like CPU cycles, storage IO, and bandwidth. But now we can bring all that data together and have this discussion around, "Oh, that's weird. When we give you an extra $10,000 a year for bandwidth, it improves our frontend response time in exactly this way."

Or, "When we switch storage providers, it improves our Apdex scores on the frontend and that, in turn, means more customers come because they have a better experience. And a better customer experience is driving an increase in revenue."

Having those interconnected connections is what’s possible now with observability. And we haven't been able to do that before. There have been many attempts to do it, and observability is still in early days. But things are starting to coalesce in a very interesting way. One of our biggest dangers in the observability space is for engineers to declare that observability belongs to them, and nobody else can have it.

New Relic: When you talked about egos, I was thinking about how people tend to be very attached to whatever tools they use. And with the direction that New Relic is headed, it seems like we're positioned well because you can use whatever tools you want and bring them together in the same platform. Can you talk about that?

Josh: Absolutely. I often warn our account executives, when they're going out and talking with prospects, “If you run across SolarWinds admins, do not tell them that we're going to take away their tools.” Just don't do it. They will light their torches and grab their pitchforks and come after you because they love their platform. SolarWinds is a company that has done a fantastic job in the hearts and minds campaigns.

So our approach now is about inclusion. Instead of rip and replace or a displacement, we're talking about tool rationalization and data consolidation. We want your tools to do the very best thing that they can, collecting the data that they excel at collecting. But we want you to share that data across organizational silos. And if it means that New Relic can natively collect that data in our Telemetry Data Platform, then so be it.

If you're going to use an open source solution to send the data to us, so be it. If you're going to use a third-party solution, who cares? We can normalize all of the data to set the foundation for that observability-based discussion.

New Relic: From what we're hearing, it sounds like folks respond very well to this message.

Josh: Oh, yes. We met with a large customer just the other day. They're doing this full digital transformation and modernization. So we went in and did this tool rationalization and data consolidation pitch. They gave us a spreadsheet that included the tools in their environment, who owns them, what they're used for, and how much they're paying. They said, “Now help us rationalize the toolset." And then, the next part is, “OK, here’s our point of view on how to consolidate all your data in New Relic One.”

That ability to be a trusted partner is a very different approach for vendors. I'll often tell account executives before I engage with any of their opportunities, “If a customer has a solution that is working for them and is better than what we can deliver, I'm going to tell them to keep that solution.”

I will not say, "No. We should use this roundabout method of data collection because it's New Relic native." I'll tell them to keep their existing tool, and we'll focus on engaging with them on what we do best. Because for me—and for the longevity of our customer relationships—it's about being that trusted advisor versus increasing ARR by another $20,000 a year.

New Relic: I have heard other customers say New Relic is a trusted partner, and I think that by being open and not telling folks that they have to do this or retire that shows we can be your trusted partner and help you to provide better visibility, right?

Josh: Yeah. And that's one reason why I opted to come to New Relic from being a customer. As an enterprise customer, I had initially picked Datadog as the tool that I wanted for infrastructure monitoring. But as a customer, I needed to grab a whole bunch of data across the entire landscape—not just infrastructure—and I also needed my account team to be someone that I could trust, that I could reach out to. I mean, they gave it to me, no holds barred. My New Relic account rep famously said to me at one point, "Josh, you're doing monitoring wrong." He's pretty bold. I don't recommend all account executives saying that to your prospects. But he was right. And I remember being super angry at him like, "Who does this account executive think he is? I've been doing monitoring for decades..."

And then a couple of months later, I was like, "Oh, he’s right. I am doing monitoring wrong.” I was doing monitoring wrong because I was only doing rear-view mirror monitoring and not moving to the larger observability view. I could tell you all about when things did fail, but I couldn't tell you why they failed without engaging a human to ask additional questions.

New Relic: Interesting, so what is required for observability from a technical perspective?

Josh: I think it comes down to the open platform. I was thinking about this last night. Maybe I need a new hobby or something. But last night I was thinking, what does “open” mean to us now? As we reimagined and simplified our platform, open means we'll take any data from anywhere and bring it together.

The best thing about an observability platform is enabling the technical teams who need to go deep, but also allowing people who don't have a deep background in say, containerization or serverless, to review the surface-level data to take a pulse. The definition of observability from the control theory perspective is: Can I understand what’s happening within a black box by looking at the outputs from that black box?

And so I look at our observability platform as very much like that. I can look at a Kubernetes cluster even when I may not be the person with the expertise to fix what's going on, but I can tell you if something's broken, and to me, that's elevating the signals that matter.

Observability democratizes the access to data by allowing people to consume it the way that they want to and need to consume it. If that means going deep, go deep. If it's just to take the pulse of that environment, do that.

New Relic: What do you think of observability in the context of confidence to make changes whenever you want, whenever you need? Confidence, for example, in your systems’ resilience, and confidence in your ability to do your job.

Josh: [Some folks argue that] unless you are borderline reckless with your deploys—which is another way of saying you have the utmost confidence in your system and your ability to fix any issue before customer experience is degraded—then you're not doing true observability. And I don't believe that that's the case. Observability is not prescriptive. It doesn't tell you how you need to behave. It allows you to behave in a way that is right for your business, not somebody else's.

New Relic: What about false positives?

Josh: False positives are typically an outcome of your alerting notifications. As an ops engineer, false positives are the thing you despise because it violates the most important metric of all, mean time between sleeping, which we jokingly refer to as MTBS.

The big challenge with false positives is that it reduces the trust in the system. If you cry wolf because of your data, because of something you think you see in your data enough, when the wolf finally comes, no one will be there.

New Relic: Other than sleep, do you see a more significant connection for people who do your job around observability and overall well-being?

Josh: Oh yeah. Absolutely. And going back to that confidence question, when you have observability, it improves the confidence to step away from your environment because you trust it. You can come in and then quickly take the pulse with the confidence to say, “I know what's going on. I can probably anticipate what might go on,” and then I can step away for a weekend. I can step away for a week and have the confidence that it’s not only me who has the tools to solve problems as they arise...now my whole team has that power. So yeah, linking those two together is important.

I think you've touched on a fantastic thread that everyone sees, but nobody's picking up. I sit on the Employee Resource Group at New Relic for mental health and neurodiversity. And I would love to pull on that thread and highlight it. The IT culture can be all about, "Oh, I can work 70 or 80 hours a week. It's awesome. If I'm in a startup, I can kill myself for this thing, and I'm going to be a multimillionaire," where a lot of ops personas are like, "Hey, look, I want to put in my 40 or 50 hours a week. I want to be able to play ball with my kids. I want to be able to take my weekends." And observability is a way to get there.

New Relic: I think that's a wonderful theme. New Relic is often recognized for its culture of empathy.

Josh: I've heard people talk about their bottom lines and how much money they make, but nobody talks about the value of coming to an organization that doesn't just value its success as a company, but it values you as an individual.

To read more about observability, check out Observability in 2020: A Manifesto.