What an Anthropologist Brings to a Software Company—The Modern Software Podcast

In this episode of the New Relic Modern Software Podcast, we welcome Ali Colleen Neff, PhD, a trained anthropologist and user experience researcher on New Relic's product team. She left academia and brought her skills to tech because she likes to, in her words, “watch hackers hack in their native habitats.” Her goal? To gather useful data about how New Relic users work, and then use that data to “build products that can help our customers resolve their incidents more quickly.”

My co-host, New Relic senior program manager Tori Wieldt, and I talk with Dr. Neff about how she uses her anthropology and media background to better understand how people engage with technology and how non-anthropologists can better understand the ways users engage with their products.

You can listen to the episode via the player below, or you can get all the episodes by subscribing to the Modern Software Podcast on Apple Podcasts, Libsyn, or wherever you get your podcasts. Or read on for a full transcript of our conversation with Dr. Neff (edited for clarity):

New Relic was the host of the attached forum presented in the embedded podcast. However, the content and views expressed are those of the participants and do not necessarily reflect the views of New Relic. By hosting the podcast, New Relic does not necessarily adopt, guarantee, approve, or endorse the information, views, or products referenced therein.

Fredric Paul: I'm fascinated by the role an anthropological approach can play in a software company. What does an anthropologist actually do for a software firm?

Ali Colleen Neff, PhD: I love this question because there are so many streams that meet to place me here at New Relic. What's particularly fascinating about working in this space is we're not just working with any end user, like if we were producing a salad spinner and wanted to know how any salad eater might use the spinner. Here at New Relic, we are working with highly technical users and highly specialized situations.

My job is to better understand the development worlds of those users. That takes a lot of work and a lot of nuance and a deep understanding of context. So they call in people like me—a PhD in anthropology who left academia to do this kind of work for tech—into the room. I say things like, “I watch hackers hack in their native habitats.” I'm really, really good at watching people do what they do and understanding those contexts.

Fredric: Is this becoming a common role at software companies?

Ali: It is. It's the big thing in Silicon Valley, particularly in the last one to two years. Just now, we're starting to hire anthropologists at B2B companies and at startups.

Fredric: What kind of training and background do you have? You mentioned academic training and you have a PhD and media experience: How do you get into applied anthropology in a technology company?

Ali: Well, as an anthropologist, it's always my fundamental job to ask first, “What makes us human?” That's what the field of anthropology is all about. And then secondly, to ask, “How do humans organize themselves into groups and how do those groups obtain and distribute the resources they need to survive?” And that's where the media studies kicks in. Media studies is all about the tools and practices that mediate people or groups of people and the resources they need to survive. I’m looking at the tools we need to build and maintain systems and structures, and then the practices and rituals that we use to assure us that they will stay up and running.

And that's how I see New Relic. It's both a tool that we use to detect and fix threats, possible breaches in our system, or reliability issues, but it's also a practice, a ritual. People keep New Relic dashboards up in their office on monitors everywhere. It's sort of a talisman that we return to in order to make sure that our systems are reliable and resilient.

User experience meets social science

Tori: Wow. I have a basic understanding of UX research, but what does it mean in your context? 

Ali: User experience at large is about the journey of the user. So rather than looking at the overall trends in the industry or some of the data that a marketer might be more likely to pay attention to, user experience researchers try to get into the mind of our end users and understand what they experience as they go through their tasks.

For New Relic users, often that involves being alerted, waking up in the middle of the night, and having to handle a giant reliability incident. Being able to study the nuances of what different users and different organizations do when that happens is the kind of thing an anthropologist can bring to the table.

Tori: Do you have a job description? And what groups do you primarily work with?

Ali: About a year ago, we hired a director of research and she hired eight new people. So basically, a brand new team of nine of us were brought here to resolve any questions we had remaining about the customer journey.

Those job descriptions said things like, "You will identify high-impact research opportunities, figure out what it means to answer the questions that these groups or people or stakeholders have, and then come up with a list of methods you can use to execute that research. Do the research, synthesize the research, and present it in a way that's useful not only to your stakeholders, but to the company at large and maybe even the digital field at large to understand the digital landscape better.”

The data that we're coming up with and the insights and understandings that we synthesize are useful not only to making our product better, but we’re also contributing to discourses that are going to make our field more powerful and make digital development that much more reliable.

My first couple of months at New Relic involved a discovery phase where I came to understand the organization I was working for—what makes it tick, who primary stakeholders are, and what problems we're trying to solve with our product. As I did so, I found the shape of my work really did change. What I needed to do with my skills as an anthropologist was come in and really understand the user journeys of folks who were experiencing incidents or who were standing at really critical parts of their organization where they're keeping things up and running. I'm really looking at a lot of fight-or-flight behavior in my day-to-day job activities.

A day in the life of a tech-company anthropologist

Tori: What's a typical day for you like? Walk us through your 9 to 5.

Ali: When I arrive at work, I usually grab a cup of coffee and a piece of chocolate. That seems immaterial but I mention it to explain that instead of going directly to my desk, I do what anthropologists do, and I go to the hearth.

Here at New Relic in Portland, we have a giant fire circle with pillows and a great view of the city. I like to sit there—in the social center of New Relic—and talk with people. I talk with them about what's going on in their groups; I talk with them about what kinds of projects they're working on, what kind of product we've got going, any feedback they've gotten from users. As people come to recognize me as an anthropologist in the company, they come to me with stories. That's really important to me, to get an understanding of what's going on at New Relic and what's going on in the reliability landscape at large.

And then I sit in a lot of meetings and consult as the social scientist in the room. For instance, I was called into the room to help the reliability team revamp their reliability retrospectives and think about how we can use social science's thinking to more deeply understand New Relic's own incidents. What's exciting about that is not only can I help them get the best kind of data, but we can take that data and see if we can build products that can help our customers resolve their incidents more quickly.

Fredric: That's an interesting group of processes. What's the end result? What are the deliverables that you're trying to create?

Ali: A lot of the insights I generate don't have product managers ready to hear them and turn them directly into products. Instead, what I can do is raise discourses about the questions and insights I turn up. I do a lot of company-wide share backs and a lot of writing and evangelizing for research across the company. That's what I do with big questions like, "What does it mean to be in a fight-or-flight situation when firefighting or resolving an incident?"

I also get to do more targeted partnering with groups where I get to say, "Okay, we're working on communication in this or that area during an incident. Let me go talk to 20 users about where their real pain points and struggles are when they're resolving incidents. How much of that has to do with communication?"

And as I turn up insights, I could turn up not only understandings of what they go through but also journey maps saying, "Okay, this is the process that you go through." I also discover pain points where they hit bottlenecks. For instance, they may have too many people giving them updates at once, and so they can't keep up with that while also resolving their incident. We'll turn those insights into recommendations that will become a part of the product development process.

Amazing real-world stories

Fredric: Can you share some of the most interesting observations that you've made in your research?

Ali: My friends who know of my anthropological work say, "Oh, it must be so much drier to work in tech.” I say, "Wait a minute. I got stories, y'all. Listen to this."

I'll build trust with some New Relic users over the course of a couple of months, and we'll become partners in research. When they have an incident where their system is going down, they will page me and call me into the room. I get to witness what goes on in that space.

The insights I get from that are incredible. In anthropology, we call that process “participant observation.” I get to observe and then later, I ask the participants to do some directed storytelling with me, and they tell their stories of how they experienced the incident in ways that helps New Relic understand much more deeply what their journey is and where we can help them.

One such story involves a dear friend of mine who is a site reliability engineer; he was tasked with maintaining a system that had a lot of impact on the world of global finance. He was the person most likely to get paged in the middle of the night to fix any potential issues. Sure enough, when his financial application went down in the middle of rush hour—meaning that folks didn't have access to the funds they needed to get gas—it was so bad that it made the front page in the New York Times.

The impact on his life was intense and the impact on his company was very, very deep. Thinking about that journey and understanding better what that experience was like for him helped me—and then helped New Relic—to build empathy for what our users rely on us for, which is to keep that from happening or to resolve those issues as quickly as possible when they do arise.

Another story involves a severity-zero incident I was able to witness. In this particular company, a severity zero means that the entire site is down or the product is down for all users—a really high-impact situation. In the midst of this situation, a very, very skilled engineer walked in to be an incident commander.

See also: Command Performance: 5 Keys to Creating Great Incident Commanders

As he goes to resolve this high-impact incident, he finds he needs updates from 25 different engineers who are fixing and researching various parts of the system. The longer they are delayed in getting this information, the longer the product is down.

Interestingly, having to pull everybody in the room and to begin to troubleshoot and find the root cause of the situation did not particularly stress out this person. When things got really bad, it was the tangle of information and communication during the incident that made it nearly impossible to coordinate a response to the crisis.

The reason I know this is because later, when talking with this incident commander about the situation, he pulled up on his iPhone a chart of his blood sugar. (He has Type I diabetes and monitors his blood sugar at all times.) We were able to look at that chart and coordinate it with our timeline of the incident—his adrenaline levels spiked about four hours into the incident, just when this communication situation happened.

It’s amazing to be able to go back to our product teams and say, "Look at this. Our users are doing perfectly fine during all phases of the incident, but their adrenaline is really spiking when they can't coordinate communication. Is there something we can do to resolve that? Let's drill down and do more research and see how widespread that problem is.” That's a really, really powerful way to understand our product and its users.

Different cultures, different problems

Fredric: What a great story.

Tori: In addition to that, do you see differences culturally across the world? Are expectations and goals different in different places in the world or are there different company cultures that jump out at you?

Ali: Yeah, absolutely. I see each potential user as situated in a unique number of cultures. There's the culture of the kind of product they make. There’s the culture of the place where they are, meaning the geographic location of where they are. There's the culture of their organization. I find that a lot of users' journey is impacted by how much agency they have to make decisions. I find that certain engineers have different kinds of backgrounds and that impacts their ability to resolve issues in particular ways or to understand their systems.

Globally, I think one of the most impactful differences I've seen so far has to do with government regulation. There are places where New Relic customers’ products are regulated by the government. If they have reliability issues, by law, they have to alert their government. That brings a dimension of oversight and impact to the work of site reliability engineers, for instance, that we might not have in other cases.

Qualitative data nerd

Tori: Ali, you call yourself a “qualitative data nerd.” What does that mean?

Ali: Any user experience research team is comprised of researchers with different skill sets. Our team has quantitative specialists who are really good at generating surveys and understanding huge data sets. My specialty is in the qualitative realm. I focus on understanding the quality of the user experience. That's a really tricky and highly specialized thing to understand.

It's not just that I collect stories. It's that I put a lot of structure to the ways that I approach getting those stories, understanding those stories, and then turning that into the kind of information we can synthesize with other user stories, and then tell a much larger story woven from those pieces.

It takes a lot of work to understand how to root out bias in a process like that and how to make sense of all of that in a rigorous way. When it gets into those really complicated questions of research strategy and design, I am here for that. In fact, I'm obsessive about it. I go to conferences five to six times a year to present and hear more about how to apply classic questions in the social sciences to the kind of complicated and wild data we encounter in the field of UX research. I'm a nerd for that.

Fredric: Given your qualitative focus, how do you measure the effect of what you do?

Ali: That's complicated. We're asking New Relic for a bit of a suspension of disbelief here. There isn't a mechanism for understanding the impact of this kind of work beyond what I would call a very powerful kind of data, which is stories. We want product managers to be able to go to leadership and say, "We weren't sure what direction to take on our product development and then we heard the story of this problem with Slack communication that deeply affected one of our customers. We looked into it, and it turns out this is a problem across the board that we hadn't gathered data on yet. Now that we have, we see that it's an industry-wide problem. No one else is doing this because no one else has invested in research of this kind.”

Those stories, I believe, have as much or more impact than numbers can.

Fredric: Is this something that other companies are doing? What's the future of applied anthropology in the technology world?

Ali: I think the future is bright. Many of my colleagues who do social sciences are moving to tech because we want to have an impact on the world of digital development. The opportunity is drawing some great minds away from the classrooms!

What I love about the fact that New Relic brought in this research team is that it places us on the bleeding edge. I don't know that anyone else working in this space is investing this much in user-centricity.

Best practices for becoming more user centric

Tori: Ali, that's fascinating. Are there tips or best practices for companies that want to bring in anthropologists or do something similar to what you're doing?

Ali: The first thing a company can do who wants to become more user-centric in the way that New Relic is, is to do a lot of reading up on what applied anthropology has done for industry thus far. Europe has been using this model for a long time in product development and so there are well-documented, high-impact case studies that folks can look at. And then Silicon Valley is evangelizing this approach right now, so there's a ton of amazing stuff out there about this kind of work.

I think a lot of consulting, a lot of collaboration and conversation, a lot of workshopping at the outset will set companies up for success. This contributes to a sure-footed product development path where we don't get halfway in a roadmap and say, "Oh my God, were we doing this for troubleshooters or were we doing this for system architects? We really don't know!" Instead, it allows product development to confidently say, "These are the users we're prioritizing right now. This is what their journey looks like." And we want to stay attuned to these users in this phase of our development.

So trust trust.

Anthropologists know how to do this. We've had to construct really complex research programs and strategies to get our degrees, and we're very well-trained in how to apply those to the world of tech. It's not always easy to onboard social scientists and it's not always easy to have that conversation. But when you do, the results are absolutely golden and very, very impactful and exciting.

Stay tuned!

If you like what you hear, be sure to subscribe to the New Relic Modern Software Podcast on iTunes, Libsyn, SoundCloud, Stitcher, or wherever you get your podcasts.

Note: The music for the Modern Software Podcast is courtesy of Audionautix.