3 Ways to Face Observability’s Change Management Challenges

A version of this post previously ran on Diginomica.

As someone who spends a lot of time talking with our IT prospects and customers, I’m painfully aware of the effort required to shift from traditional performance monitoring to achieving end-to-end observability. Companies arrive at the conclusion that they need to change for different reasons. Some want to move off a proprietary monitoring platform to one compatible with open standards, or consolidate multiple point solutions to unify on an observability solution. To complicate things, your apps may be mid-stream in their journey to the cloud, or you may have just implemented a microservices architecture. Regardless of the technology changes you need to make or accommodate, observability also requires IT leaders to manage organizational change. And a tautological reality of the world is that, by default, people will avoid change until the pain or the cost of not changing exceeds the pain or the cost of the change itself.

Frankly, we’ve reached the tipping point where making this change is now an imperative. Applications have become dramatically more complex in the last handful of years—the monolith-to-microservices story—so managing performance now involves a much broader surface area. Although open standards like OpenTelemetry have made it easier than ever to instrument new services, the sources and volumes of data are growing at a pace much faster than IT staff can keep up with. And finally, though perhaps least interesting, is the pain of contract management. Adding a new tool to monitor each new type of telemetry data has created a scenario where some IT teams spend as much time troubleshooting their monitoring bill as they do their actual applications.

When you think about it, the pain of not changing is the proverbial “stick,” and the relief from that pain is the “carrot.” You’ll need to wield both when managing the organizational change required to support observability. It’s highly likely none of your engineers are jumping out of bed each morning fired up at the prospect of changing out a bunch of monitoring tools. You’ll need to sweeten the deal for IT teams embarking on the shift. I’ll share three areas for you to think about for motivating your staff to get excited about updating traditional monitoring tools with an observability platform.

1. Leadership and accountability

Whether your business is facing new and increasing competitors, you’re undergoing a digital transformation, or customer demands on your digital experience have unexpectedly changed (a common theme lately due to COVID), business is a fluid dynamic. And so must be the technology it relies on. Regardless of your unique situation, your people have to understand why a change is required. What’s driving the need for observability? Is it to speed issue resolution? Lower costs? Allow developers to innovate and perform chaos testing with more confidence? Basic leadership involves trusting people to do the right things when they understand the rationale underneath. So make sure you communicate the “why” over and over, until your team can repeat it to one another on their own.

At the same time, connect the “why” to clear objectives and hold people accountable to them. The most successful companies align those objectives to customer-focused outcomes instead of internally focused ones. For example, instead of grading your engineers and product managers on the number of features they ship in a unit of time, hold them accountable for the number of monthly active users of their recently deployed app. Track net promoter scores (NPS) and hold them accountable for reaching a customer satisfaction metric.

With the evolution toward DevOps and the growth of the full-stack engineer role, it’s becoming routine that the people who write the code are also going to deploy it, monitor it, maintain it, and ensure it adds value to the end-user’s experience. When they’re responsible for the end-to-end experience and encounter an issue, suddenly they're chasing problems across the front end through the service tier, into the infrastructure, which is increasingly deployed on Kubernetes. When you break down the silos between the objectives software teams are held accountable to, they will naturally want to remove time-consuming silos getting in their way of achieving those objectives. And when those objectives are focused on your customers rather than internal metrics, customers win and so will your business.

So that’s one tip—start with the customer experience and work back to establish the metrics to which you hold your engineers accountable. Engineers naturally want people to use their software, so they’ll become more engaged when their success is anchored to customer outcomes.

2. The learning curve

Another responsibility for managers leading this type of change is to make it as easy as possible for staff to get and stay productive. Make sure that the observability platform you’re giving them is familiar and easy to use and learn. Having all the data about how a system is operating in one place won’t improve anything if your staff can’t figure out how to get answers to the questions they want to ask of this data when something goes wrong.

And as you consider the learning curve, pay particular attention to how well your chosen observability platform organizes all of the details of your system into a unified view of end-to-end health. The ability to visualize the health of your entire system at a glance is essential. Data visualization helps you comprehend trends, relationships, and patterns quickly and easily. According to Gartner, 90% of information transmitted to the brain is visual, and the brain processes visuals 60,000 times faster than text.

Be careful that you’re not replacing the toil of working across multiple tools with the toil of figuring out what one platform is trying to expose. Ease of use is a bigger issue than you might think when you’re asking your engineers to shift technology tools or platforms. It will shorten the time between an issue happening (or about to happen), mean time to detect (MTTD), and mean time to resolve (MTTR).

3. Acknowledge the trade-off

Another important thing to do is to explicitly acknowledge with your teams that change often means giving up something that was good in order to achieve something even better in return. For example, a common pattern we see is tool consolidation. In order to manage an increasingly complex and diverse software architecture, companies change from many “best of breed” point solutions to a single observability platform that stores everything about the system behavior in one place.

This is usually because being able to quickly navigate the connections between components enables IT teams to deliver the best possible experience to end users with the fewest and shortest possible interruptions in the quality of that experience. It’s not unusual though for a given point solution to have one or more “pet features” that teams have grown fond of. In order to embrace change, they need leaders to acknowledge the value of the things they are letting go of while making a convincing case for why the solution they are moving to will make their lives better in the long run.

People are people

Ultimately, as a manager facilitating change, the shift from monitoring to observability will be a lot smoother if you focus on the human beings you’ve charged with doing the work. Help them understand the need for change, connect their accountability to what matters most (usually the customer or end user’s experience), make it as easy as possible for them to effect the change, and ensure that what you’re asking them to give up is smaller than what they gain in return.

Don’t make the mistake of thinking a bag of monitoring tools will give you full-stack observability. Read the Three Essentials for Achieving Full-Stack Observability.