Life at New Relic: A Behind the Scenes Look at Building Better Observability (Part 1)

As part of our “Life at New Relic” blog series, we recently sat down with Tim Krajcar, a Senior Director of Engineering, Full-Stack Observability. Tim joined New Relic six years ago—four months before the company went public—an exciting time to join. Today, Tim’s teams deliver the end-user interfaces for APM, Browser, and Mobile, the products that combined to form Full-Stack Observability. This two-part post explores New Relic’s engineering culture, leadership, and innovation.

New Relic: What attracted you to join New Relic?

Tim Krajcar: [That was] a long time ago—I have to rewind. The answer has not changed as to why I'm still here, and it has to do with working with so many vastly intelligent, top-of-their-profession engineers in so many different engineering areas. This is a unique problem space. We have cutting edge UI work happening, which I'm very close to. I started my career here working in the data gathering/agents side of the world. So that was an incredibly unique field—working alongside brilliant engineers with very specific and esoteric knowledge of how to instrument customer applications in a very high-performance, low-impact way.

We have some of the best people in the world at this company; we have people who work on high-throughput data services that handle millions and billions of transactions per minute, people building cutting-edge data query technology, and engineers building visualization tech and high-density UIs that are unlike anything else out there. There are not many companies where that's true.

New Relic: That's awesome. What has it been like for you to grow your career at New Relic?

Tim: It's been really interesting. I started as the manager of a four-person team and went from that into managing multiple teams. I spent a good while managing three teams and used that to learn how to scale beyond managing just a single team and learned how to scale myself. Then, transitioning into a manager-of-managers role, I had a gentle curve with a great VP who invested in my development.

More recently, I've gotten to work with managing more and more senior folks, including having a director reporting to me, which was another chance to do some coaching at a higher level, which was very interesting and rewarding.

New Relic: Are you part of the Mini-M Groups?

Tim: I love the Mini-M program! And I used to be a facilitator, but I had to step out of facilitation a couple of months ago because I got too busy. But I’m still an enthusiastic fan of it.

Anytime I'm hiring managers, I talk the Mini-M program up because it’s so powerful to have a chance to connect with other peer managers that aren’t in the same reporting group you are. For a few years, I got the chance to run the program itself, which was incredibly fun—I got to have some manipulation of the social structure, and say "I know who would be a great partner for this person; I'm going to put them in that group." And then you get to see whether it works out or not. And sometimes it doesn't, but usually it does.

New Relic: You're an excellent communicator. Is having this type of high-level communication part of what makes the New Relic engineering teams succeed?

Tim: I've seen that change over time, for sure. When I first started here, we had more people who were the negative stereotype of an engineer, people who are antisocial or only out for themselves, and refused to communicate positively with their peers or team. They don't end up lasting in the culture here.

The culture here is way more interactive; it's more interpersonal. We thrive on our social relationships. It's a very complex organization, with a lot of people, a lot of teams.

The social currency here is your relationships with your team and outside of your team. I work hard to cultivate an environment in my group that doesn’t tolerate [jerks], and I've seen a steady improvement in the broader company.

New Relic: Talk to me about a project that you are especially proud of. If that's the recent New Relic One refresh, what were the challenges you faced, and what made it rewarding for you?

Tim: That would be my first answer. A couple of challenges: One is simply the scale of what we were trying to do. We were helped greatly by a clear vision that came straight from the top, from Bill Staples, from some product executives and Lew Cirne. We knew what we wanted to do, and we knew it was the unquestioned top priority. So we didn't have to deal with selling the project internally. We knew what we had to do, but we knew it was going to be very difficult.

We did an inventory and ended up with around 150 individual experiences as a part of the move into New Relic One, and each one of those might have been trivial or might have taken a month. And we had to continuously plan and re-plan and make sure that we were tracking along and make hard scope decisions. We dropped a ton of scope along the way, which I knew we would, and I'm glad we did.

We also had huge amounts of coordination with the platform teams in Barcelona, which was a new challenge for me. The total roster of people that had something to do with the New Relic One project was almost everyone at the company. The UI refresh and the “new” New Relic One that we launched ended up somewhere in the hundreds of engineers who contributed. So coordinating all of that was quite a challenge. I had an incredibly strong Program Manager, Andrea Beithon, who was instrumental in that success too.

We knew from the beginning the timeline was very short. The end of August timeline was set from the beginning. And then the pandemic hit, and everything got way harder. I invested a lot of time making sure that my people understood that they had space to take care of themselves and allowed them space to take care of the people in their lives. And the good news is we were able to make adjustments to about every other part of the project, and we hit the timeline, and it went great. The last couple of weeks were busy, but they were not all-nighters. And the launch was super smooth.

New Relic: Does the fact that we have part of the engineering team in Barcelona help from an execution point of view because you have more hands working more hours of the day?

Tim: We got into a cycle where during our workday, we would file a bunch of issues and say, "Hey, this isn't working quite right." And then the next morning, we would log in, and everything would be fixed. It was a great partnership.

We have some fantastic engineers there, and it was really fun. We’ve had great folks in Barcelona for a long time, but I hadn't had a chance to work much with them. And that's part of why this project took executive sponsorship to happen. The instinct is to work on projects within your scope, control, and not these messy cross-ocean things. That's not anyone's idea of, "Hey, let's go do this; this sounds fun." No, it's really hard. It takes a lot of work and coordination, and it's challenging, but that's why you have to do it because the results are worth it.

Read more about how New Relic engineers use New Relic One to deliver more perfect software.