Life at New Relic: A Behind the Scenes Look at Building Better Observability (Part 2)

As part of our “Life at New Relic” blog series, we sat down with Tim Krajcar, a Senior Director of Engineering, Full-Stack Observability. Tim joined New Relic six years ago, four months before the company went public—an exciting time to join. Today, Tim’s teams deliver the end-user interfaces for APM, Browser, and Mobile, the products that combined to form Full-Stack Observability. This two-part post (read part one) explores New Relic’s engineering culture, leadership, and innovation.

New Relic: Let’s talk a little bit about observability since it’s in your title, and it’s what we're talking about at New Relic. First of all, how do you define observability?

Tim Krajcar: I was a New Relic customer before I started working here. So I was very familiar [with New Relic]. I have support tickets that I filed back then. And the funny thing is the person who answered those tickets in 2013 is still here.

I would define it as the ability to inspect the operation of a complex running system, and, crucially, without the need to preplan your inspection points. I think it's very easy for developers to think, "If I know that I need to keep an eye on a certain thing, I can add all sorts of output and logging and statistics collection around the thing."

For example, if your business has a checkout process, you're absolutely going to have good instrumentation on your checkout process because everyone needs it and cares about it. The business wants to know what the checkout is doing. Engineers know if there are no checkouts, they're not making money—it matters.

What you might not have crazy-good instrumentation around is the complex web of upstream services that cause problems. So, observability means I can go to that service, and the data is there when I need it. And I don't have to take it down and add a whole bunch of debugging information and re-deploy, and I can get the information that I need.

New Relic: Do you think engineers, in general, care about the term “observability?” I know people care about what it can do for them, but what about the term itself?

Tim: I think it's become a good way to summarize a whole collection of disciplines. I know in our industry, it's become passé to say monitoring. Monitoring is not a word that we like anymore; we want observability. And some of that is just the natural tendency of humans to define new language and find new slogans for T-shirts.

But the word encompasses multiple disciplines, which works for me. It feels like a broader definition because monitoring can, for some companies and some individuals, take on a very operational focus. And for companies that don't have a solid DevOps practice and don't have their developers doing operations work, you can have people who don't know anything about the software that they're running; they just know whether it's passing health checks. And that's not ideal; you want a much more holistic, comprehensive view.

You want everyone to have the same amount of knowledge about what's happening, and I think observability takes you closer to that.

New Relic: If you were talking to a customer, what would you say about why they need to have New Relic Full-Stack Observability?

Tim: This is not a theoretical question for me—I enjoy working at trade shows and getting to answer this question in real time. Developers are my people, and I enjoy understanding their problems and explaining how our solutions meet them. In my opinion, what we do that is better than anyone else is the depth and breadth of what we instrument. That combination is better. And I'm skewed here because my background is in data collection, so I know how good our agents are, and I know how detailed the data is that they collect. I know how much we have invested over 12 years in our data collection systems, and I believe they're outstanding.

We want your data, and we want to help you make sense of it. With the open source work that we've done in the last six months, we don't care what kind of data it is or what it's shaped like or where it came from, whether it's an app or a mobile app or something through OpenTelemetry that we don't even have any idea what it is; it's just spitting data at us. That's what we do well. Hopefully, if my group is doing their job well, and I'm doing my job well, we’ll help you make sense of the data you've collected, and it will surface to you in usable ways.

New Relic: What are the top benefits for customers who embrace Full-Stack Observability?

Tim: End-to-end tracing is up there—being able to follow a request through all the layers of the stack—that end-to-end visibility is incredibly important. Also, I think we are strong in the incident response use case. And I know we spend a lot of time thinking about, “OK, someone just got an email, it's 2:00 a.m. How do we provide them the most accurate information as quickly as possible to a root cause, figure out what's going on, and solve it? Whether it's something they personally can solve, or it means they have to page someone else, we want to get them back to sleep as soon as possible.”

New Relic: You talked about open source and embracing that community. Would you like to go down that path a little bit more?

Tim: This has been an interesting journey because some of our agents were open source in the beginning. I started here working on the Ruby agent, which has always been open source from day one, and that's a strongly-held principle by the Ruby community—they're used to working with all open source tools, so it made sense. But not all the agents were. I've watched that change over the last six years.

And now, all of our agents are open source—we are a completely open platform. You can send us data from all sorts of different tools. We've done the hard work in many cases to take in data in different data formats that aren't specific to us, or we'll put data out in formats that aren't specific to us. And I think that's an important change to help customers who are leery of lock-in.

New Relic: Is there anything that we've not covered?

Tim: Always have to work in a plug for team culture. I'm at a position now where I'm pretty well-known within engineering. People know my name, at least, maybe because it's unusual, or they've worked for me or worked with people who've worked for me. And I work hard to be a great leader and support the teams and people who work for me. And I love that I also have the support to be able to do so.

This is an interesting company. We give engineering managers a tremendous amount of latitude in how they want to run their teams. And that goes all the way up the stack. As a Senior Director now, I see opportunities to influence not just my group’s culture but also the culture of engineering and even the broader company. I spend a lot of time thinking about how to influence and improve that culture.

New Relic: That's great. In terms of New Relic values, is authentic, by any chance, the value you identify with most, or maybe it's accountable, bold, passionate, or connected?

Tim: All of them, to a degree. I would say authentic is absolutely the right one for me. And I'm very authentic about my identity, who I am, and where privilege exists in my life (which is all over the place). And I try to invest a lot in people who have different identities than mine and understand them as well—what it takes for others to be themselves at work, be authentic, challenge each other, challenge me all the time, and produce incredible software.

New Relic: I hear you are hiring.

Tim: Yes, we have some interesting positions as we continue to reimagine our products with less siloing between the different, what used to be our old SKUs, and we move more toward this full-stack model. We have some opportunities for cross-cutting engineering work, and I'm excited to bring in some new folks to fill those roles.

We are at an exciting time in the company and currently hiring across various geographies and departments. Check out our opportunities.