What Makes Observability a Priority

Executive summary

This white paper provides in-depth research about what makes observability a priority. It reviews how digital transformation is driving observability as organizations continue to shift workloads to multi- and hybrid-cloud environments and adopt open-source and cloud-native technologies, including containers, microservices, and serverless.

Organizations are managing a more diverse array of infrastructure than ever, which increases security, risk, and compliance concerns and affects service-level metrics. Monitoring and observability help address these concerns. However, monitoring is fragmented and significant data is unmonitored. Even so, as cloud, cloud-native, and open-source adoption, usage, and spending continue to increase, so do observability deployment and budget plans. Unfortunately, pricing and billing can be a barrier to achieving observability.

This white paper recommends that organizations understand the pricing and billing options used by observability vendors so that they can select the best solution and overcome any cost barrier to achieving full-stack, end-to-end observability.

Observability definition

Observability enables organizations to measure how a system performs and identify issues and errors based on its external outputs. These external outputs are called telemetry data and include metrics, events, logs, and traces (MELT). Observability is the practice of instrumenting systems to secure actionable data that details when and why an error occurs. 

Achieving observability brings a connected, real-time view of all data from different sources—ideally in one place—where teams can collaborate to troubleshoot and resolve issues faster, prevent issues from occurring, ensure operational efficiency, and produce high-quality software that promotes an optimal customer/user experience.

There are four fundamental components of observability:

• Open instrumentation to gather telemetry data from data sources, like services, hosts, applications, and containers.
• Data cardinality and correlation and topological context to help understand the bigger picture, including the current state and availability of highly distributed systems.
• Programmability to create tailored experiences with custom applications based on unique business objectives.
• AIOps (artificial intelligence for IT operations) capabilities to accelerate incident response and ensure services are available by proactively detecting issues and accelerating mean time to understanding (MTTU) and mean time to resolution (MTTR).

Software engineering, development, ​​site reliability engineering (SRE), operations, and other teams use observability to understand the behavior of complex digital systems and turn data into tailored insights. The primary consumers of observability tools are security, development, and infrastructure operations teams (SecOps, DevOps, and InfraOps).¹ Observability helps them pinpoint issues more quickly, understand root causes for faster, simpler incident response, and proactively align data with business outcomes.

According to the 2022 Observability Forecast report, information technology (IT) operations teams were the most likely to be responsible for observability, followed by network operations and DevOps teams; application development and SRE teams were more likely to be responsible for the implementation of observability than the maintenance or usage of it; and SecOps and DevSecOps teams were more likely to be responsible for the usage of observability than the implementation or maintenance of it.²

The rise of monitoring and observability

The concept of observability stems from the broader engineering principles of control theory.³ First coined in 1960 by engineer Rudolf E. Kálmán,⁴ the term itself “started to gain serious momentum in software engineering circles around 2018 as a natural evolution of monitoring practices.”⁵

A subset of observability, organizations use monitoring to identify problems in the environment based on prior experience that is expressed as a set of conditions (known unknowns). Monitoring enables organizations to react to these conditions and is sufficient to solve problems when the number and complexity of possible problems are limited.

Organizations use observability to determine why something unexpected happened (in addition to the what, when, and how), particularly in complex environments where the possible scope of problems and interactions between systems and services is significant. The key difference is that observability does not rely on prior experience to define the conditions used to solve all problems (unknown unknowns). Organizations also use observability proactively to optimize and improve environments. For example, they can use observability data and capabilities to reduce infrastructure costs through resource optimizations, improve customer experience through software optimizations, and so on.⁶

Monitoring tools alone can lead to data silos and data sampling. In contrast, an observability platform can instrument an entire technology stack and correlate the telemetry data drawn from it in a single location for one unified, actionable view. The ability to see everything in the tech stack that could affect the customer experience is known as full-stack observability⁷ or end-to-end observability.

Adopting a data-driven approach for end-to-end observability helps empower engineers and developers with a complete view of all telemetry data, so they don’t have to sample data, compromise their visibility into the tech stack, or waste time stitching together siloed data. Instead, they can focus on the higher-priority, business-impacting, and creative coding they love.

Many tools are purpose-built for observability and can include application performance monitoring (APM), infrastructure monitoring, log management, network performance monitoring, digital experience monitoring (synthetic monitoring and real user monitoring (RUM), including browser monitoring and mobile monitoring), serverless monitoring, machine learning (ML) model performance monitoring, security monitoring, and more.

Observability has risen in popularity because environments are now more complex, and the number and combination of problems organizations can experience grows exponentially. It’s no longer possible to know all the expected problems and conditions a complex system may trigger ahead of time. Hence, organizations need to observe systems 24x7 combined with good diagnostic capabilities, such as anomaly detection, query engine, dashboards, and so on. As systems become more complex and distributed and the rate of development and deployment continues to speed up, monitoring alone is no longer sufficient to predict, detect, and troubleshoot failures.

Digital transformation is driving cloud adoption

According to Gartner®, “Digital transformation can refer to anything from IT modernization (for example, cloud computing), to digital optimization, to the invention of new digital business models. The term is widely used in public-sector organizations to refer to modest initiatives such as putting services online or legacy modernization. Thus, the term is more like ‘digitization’ than ‘digital business transformation.’”⁸

Workloads are moving toward the cloud

As digital transformation continues, workloads are moving toward the cloud.⁹ The COVID-19 pandemic accelerated digital transformation and global cloud adoption.

According to nearly 75% of the respondents from a 2019 study commissioned by New Relic in partnership with Vanson Bourne, migration to the public cloud is at the core of any digital transformation. However, one in two respondents said managing and monitoring their digitally transformed organizations is a challenge, and 56% acknowledged that it’s humanly impossible to assess this data properly.¹⁰

A 2020 survey by Longitude Research found that, on average, software leaders run 66% of their software and systems in the cloud.¹¹

Respondents from a 2022 survey by Flexera ran half of their workloads in the cloud. They expected that number to increase in 2023 when they predicted more than half of respondents’ data will be in the public cloud (Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform lead the pack in public cloud providers). Cloud adoption continues to become more mainstream, with heavy users running more than 25% and as much as 63% of workloads in the cloud, an increase from 59% in 2021 and 53% in 2020. Optimizing existing use of the cloud and migrating more workloads to the cloud are top initiatives.¹²

Migration to multi- and hybrid-cloud environments is driving observability adoption

According to a 2021 report on the state of DevOps by Google Cloud, 56% of respondents said they use a public cloud, a 5% increase from 2019, and 29% use a private cloud. Multi-cloud and hybrid solutions are on the rise; 21% of respondents deploy to multiple public clouds, and 34% use a hybrid cloud. Only 21% said they aren’t using the cloud.¹³

A 2022 survey by Flexera found that all respondents use at least one public or private cloud, and 89% of organizations employed multi-cloud architectures, which are more complex and challenging to manage. Most are combining the use of both public and private clouds for a hybrid approach.¹⁴

When 2022 Observability Forecast respondents were asked which technology strategies and trends are driving the need for observability at their organizations, 42% cited migration to a multi-cloud backend environment.¹⁵

Cloud spend continues to increase

The increase in cloud adoption is reflected in spending as well. In 2021, Gartner expected cloud spend to represent about 20% of the US$842B overall infrastructure spend (approximately US$168B). It forecasted worldwide end-user spending on public cloud services to grow 23% to total US$332.3B in 2021.¹⁶ 

In the fiscal year 2021, Google Cloud Platform generated about US$19B in revenue (an increase of 47% since 2020),¹⁷ Microsoft Azure generated about US$34B in revenue (an increase of 49% since 2020),¹⁸ and AWS generated about US$62B in revenue (an increase of 37% since 2020).¹⁹ These three public cloud vendors alone were at a collective annualized run rate of US$135B and growing 42% year-on-year.²⁰ In the third quarter of 2022, the three vendors were at a collective annualized run rate of US$160B and growing about 32% year-on-year (compared to 36% in the first quarter).²¹

A 2022 survey by Flexera found that public cloud spend is now a significant line item in all IT budgets. Small and midsize businesses (SMBs) increased their cloud spend by 38% since 2020, with 53% of SMBs each spending more than US$1.2M in 2021.²²

Digital transformation increases the need for observability

Observability is applicable for cloud migrations, customer journey analytics, DevOps, and more. It gives organizations a real-time perspective of how their systems are operating and helps them perform real-time analysis of where they are compared to where they’d like to be in their digital transformation. Once they’ve completed the transformation, cloud observability can contribute to their operational, business, and IT awareness and help improve their daily metrics to drive the digital experience. Because it can highlight things that may not be obvious, it’s key to sustaining project goals, enabling innovation, and potentially providing new revenue opportunities or the ability to optimize the cloud build.

Observability is key for digital transformation and cloud adoption

Technology professionals see the value of observability for digital transformation and cloud adoption. The 2021 Observability Forecast found that 42% of respondents believe observability is important to support digital transformation efforts, including 37% who believe it optimizes cloud usage and spending, and 35% who believe it minimizes the risk of migrating core legacy applications to the cloud.²³ The 2022 Observability Forecast found that 31% of respondents apply observability to support digital transformation efforts and optimize cloud resource usage and spend. In addition, more than a quarter (26%) said that decreased cloud-hosting costs are a primary benefit enabled by their observability deployments.²⁴

A 2021 observability survey by Enterprise Technology Research (ETR) found that “as enterprises continue digital transformations, new workloads are aligned to the cloud and that observability ingestion is going to cloud-native vendors like Datadog and New Relic much faster than established vendors.”²⁵

Cloud-native technology adoption is now mainstream

Largely due to ongoing hybrid- and multi-cloud adoption, organizations are managing a more diverse array of infrastructure than ever, gathering metrics from on-premises (on-prem) infrastructure and private and public cloud services, including serverless and managed Kubernetes (also known as K8s) services. Modern systems increasingly involve open-source code and multiple cloud-native microservices running on containers and Kubernetes clusters.

The cloud-native age is here

The primary mode for customer engagement is now cloud-native.²⁶ In this cloud-native age, organizations purposely build applications running in the cloud and optimize them to take full advantage of cloud benefits such as elasticity and resiliency.²⁷ For example, the 2020 survey by Longitude Research found that 96% of software leaders have adopted a cloud-native approach to software development.²⁸

While a comparison study of title usage in 2019 versus 2020 for the O’Reilly online learning platform showed that usage of content about the cloud increased by 41%, microservices increased by 10%, containers increased by 99%, Kubernetes increased by 47%, and observability increased by 128%.²⁹ 

A 2021 survey by 351 Research found that 63% of respondents develop at least half of their DevOps apps using cloud-native technology and methodology, 65% have adopted some container technology, 53% have adopted some Kubernetes technology, and 55% have adopted some serverless technology.³⁰

The popularity of Kubernetes continues to explode 

The 2021 Observability Forecast found that 88% of respondents were exploring Kubernetes, 25% were conducting research, 25% were evaluating, 29% were in development, and 10% were in production.³¹

A 2021 survey by the Cloud Native Computing Foundation (CNCF) found that Kubernetes is mainstream in global technology, with 5.6M developers using Kubernetes—67% more than in 2020—representing 31% of all backend developers. Most organizations (96%) were either using or evaluating Kubernetes. New Relic observability platform user data supports these CNCF survey findings with a 37% year-on-year increase in Kubernetes adoption.³²

A 2022 developer survey by Stack Overflow found that 23% of developer respondents used Kubernetes. It also found that Kubernetes is the second-most loved (75%) and wanted (24%) tool.³³

According to a 2022 DevSecOps survey by GitLab, 33% of teams were using Kubernetes, 25% planned to implement it this year, and 29% planned to implement it in the next two to three years.³⁴

A 2022 survey by Flexera found that 42% of respondents use Kubernetes, and 25% plan to use it in the future (67% total).³⁵

In its 2022 report on the state of enterprise open source, Red Hat found that 70% of IT leaders said their organizations use Kubernetes.³⁶

The use of containers, microservices, and serverless is increasing

Developers and organizations are maturing in their approach to containerization and using microservices and serverless more intensively. The rise of Kubernetes is connected directly to the increasingly mature way organizations use containers. “This growth also brings challenges and gaps from the necessary cultural shift to technology trends and advancements. As the next wave of microservices and more stateful applications are deployed on Kubernetes and container-based platforms, there is a need for more visibility into operations, as well as tools for self-defense and self-healing against malicious applications (both intentional and unintentional).”³⁷

Most (93%) of the 2021 CNCF survey respondents used or planned to use containers in production. New Relic observability platform user data supports this CNCF survey finding with a 49% year-on-year increase in overall container adoption. In addition, 39% used serverless technology.³⁸

A 2022 report on the state of enterprise open source by Red Hat found that 68% of IT leaders run containers in their organizations, and nearly one-third planned to increase their container usage significantly in the next 12 months.³⁹

In addition, a 2022 DevSecOps survey by GitLab found that 36% of respondents use microservices, 28% planned to this year, and 29% planned to over the next two to three years.⁴⁰

“The forces that have helped propel Kubernetes (K8s) growth—such as containerization, application modernization, cloud-native development, and hybrid cloud infrastructure—are here to stay.”⁴¹

However, monitoring and optimizing applications running within containerized environments such as Kubernetes is a major challenge. According to a 2021 report by 451 Research, “Greater adoption of containers, Kubernetes, and serverless has created new challenges for enterprises embracing cloud-native and DevOps.”⁴²

Cloud-native technology is driving observability adoption

When 2022 Observability Forecast respondents were asked which technology strategies and trends are driving the need for observability at their organizations, they cited the development of cloud-native frontend application architectures (47%), adoption of open-source technologies such as OpenTelemetry (39%), adoption of serverless computing (36%), and containerization of applications and workloads (36%)—all trends where observability requires a unified approach.

About a quarter of respondents said they apply observability to manage containerized and serverless environments (29%) and for visibility to migrate from monolith to microservices (25%).

In addition, 88% expected to deploy Kubernetes monitoring by 2025, and 91% expected to deploy serverless monitoring by 2025.⁴³

Observability is mission-critical

The growing complexity of distributed applications and hybrid- and multi-cloud adoption has highlighted the need for observability capabilities like APM, infrastructure monitoring, and log management as organizations aim to maintain visibility, improve incident response, and gain a contextual understanding of their applications and infrastructure. As organizations modernize their applications and maintain visibility over their expanding and increasingly distributed IT environments while using a data-driven approach for their incident and threat response, observability has become more important than ever.⁴⁴

Observability can help eliminate the cloud complexity issues that hinder some cloud migrations and new development. Moving systems quickly to the cloud increases both complexity and risk, making observability more critical.⁴⁵

Security, governance, risk, and compliance are driving the need for observability

A subset of observability, security monitoring is also critical. In a 2018 report about developers by Stripe, 66% of C-suite executives said security/data breaches and 62% said increased regulation was threatening the success of their businesses.⁴⁶

In 2020, more than 22 billion records of confidential personal information or business data were exposed, according to a report on the threat landscape by Tenable.⁴⁷ According to a 2022 study by Gartner Peer Insights and Radiant Logic, 84% of organizations have experienced an identity-related data breach.⁴⁸ A 2022 survey report about the state of ransomware by Gigamon found that 95% of respondents experienced ransomware attacks in the last year, and 59% claimed the ransomware crisis worsened in 2022. The research also revealed that 89% think deep observability is an important element of cloud security.⁴⁹

According to a 2022 DevSecOps survey by GitLab, 57% of security team members said their organizations have shifted security left in the software development lifecycle (SDLC) or are planning to this year. About two-thirds of security professionals said they have a security plan for microservices (65%) and containers (64%). And 53% of teams said they had plans to secure cloud native and serverless. But while security scanning is increasing, access to data lags. In addition, almost 25% spent between half and three-quarters of their time dealing with audits and compliance.⁵⁰

Modern applications typically run in the cloud and depend on hundreds of components, introducing additional monitoring challenges and security risks. With cloud adoption, cloud-native application architectures, and cybersecurity threats on the rise, it’s not surprising that an increased focus on security, governance, risk, and compliance was the most frequently cited strategy or trend driving the need for observability at the organizations surveyed in the 2022 Observability Forecast (49%).⁵¹

Frequent outages are a reality

Survey results from the 2022 Observability Forecast showed that outages happen fairly frequently (up to 72% noted once per week or more). Alarmingly, 52% experienced high-business-impact outages once per week or more. More than half (53%) took more than 30 minutes and 22% took more than an hour to detect high-business-impact outages (MTTD), while 60% took more than 30 minutes and 29% took more than an hour to resolve them (MTTR).

Given the relative frequency of outages, the findings of how often manual effort and incident tickets are the sources of knowledge for these outages are noteworthy—33% still detected software and system interruptions with manual checks/tests or through incident tickets and complaints.⁵²

Observability improves service-level metrics

The good news is that, according to a 2020 survey by Longitude Research, of those using observability solutions, 83% experienced fewer than five outages per month, and 75% reported an average MTTR of less than 30 minutes.⁵³

According to GigaOm, “observability tools potentially can increase systems reliability 100-fold because issues are automatically discovered, analyzed, and corrected—without human intervention. Moreover, the system is able to learn from successes, so system reliability improves over time. Proper observability tools and optimized IT processes can reduce the MTTR metric between 50% to 90%.”⁵⁴ For example, developers and engineers who applied observability averaged less than half the time spent troubleshooting problems than those who didn’t (23% versus 46%).⁵⁵

And the 2022 Observability Forecast survey data supports a strong correlation between full-stack observability, fewer outages, a faster MTTD, and a faster MTTR. In other words, prioritizing/achieving full-stack observability can help improve service-level metrics. In addition, the data predicts a positive association between nine capabilities—including AIOps, distributed tracing, security monitoring, custom dashboards, synthetic monitoring, APM, database monitoring, alerts, and infrastructure monitoring—and a faster MTTD/MTTR (less than 30 minutes). Of those capabilities, AIOps is statistically significant within 10% significance levels.⁵⁶ 

Notably, a study by the International Data Corporation (IDC) found that New Relic customers experienced a 49% reduction in unplanned outages, 83% faster MTTD, and 27% faster MTTR.⁵⁷ And a Forrester study about the total economic impact of the New Relic observability platform found that it led to a 30% reduction in MTTR.⁵⁸

Observability has become a board-level imperative

A 2020 survey by Longitude Research found that 94% of software leaders believe observability is key to developing software.⁵⁹

According to a 2021 report about the state of DevOps by Google Cloud, monitoring and observability practices support continuous delivery, and teams with good observability practices spend more time coding instead of troubleshooting.⁶⁰

The 2021 Observability Forecast found that observability is mission-critical; 90% of respondents believe observability is important and strategic to their business, and 94% believe it is important to their role. In addition, observability delivers a clear, positive business impact as 91% of IT decision-makers (ITDMs) believed that observability is critical at every stage of the SDLC, with especially high importance on planning and operations. Most (83%) also noted that an observability platform is effective for Kubernetes monitoring.⁶¹

Nearly three-quarters of 2022 Observability Forecast respondents said C-suite executives in their organizations are advocates of observability, and 78% saw observability as a key enabler for achieving core business goals.

About a quarter of C-suite executives thought the most important outcomes of observability are:

1. Improving revenue retention by deepening understanding of customer behaviors (30%)
2. Mitigating service disruptions and business risk (28%)
3. Improving collaboration across teams to make decisions related to the software stack (28%)
4. Shifting developer time from reactive incident response towards proactive, higher-value work (25%)
5. Creating revenue-generating use cases (21%)

The report also found that observability continues to deliver a clear, positive business impact, with about a third of respondents citing:

1. Improved uptime and reliability (36%)
2. Increased operational efficiency (35%)
3. Improved customer experience (33%)
4. Accelerated innovation (30%)
5. Business/revenue growth (26%)

As far as how observability helps developers and engineers the most, at least 30% said it increases productivity and enables cross-team collaboration and less guesswork when managing complicated and distributed tech stacks, about three out of 10 said it makes developer/engineer lives easier and improves work/life balance and skillset/hireability, and roughly a quarter felt that it helps confirm/overcome assumptions, overcome opinions, and fill in gaps.⁶²

In addition, according to a 2022 survey about DevSecOps by GitLab, almost 60% of respondents said code is moving into production much faster—and the majority attributed the faster releases to observability.⁶³

Monitoring is fragmented

However, data, tools, and teams are fragmented—new tools emerge, and old ones fall out of favor, resulting in multiple tooling and fragmentation. 

A 2021 advisory report by 451 Research found that 96% of organizations prefer to buy monitoring and incident response tools from a single vendor when possible (up from 83% in 2020).⁶⁴ 

And a 2022 macro survey by ETR found that consolidating redundant vendors across the lines of business is the most common approach to decrease IT spending.⁶⁵ However, 49% of operations professionals said their teams use two to five monitoring tools, and a third use six to 10 tools, according to a 2022 survey about DevSecOps by GitLab. More than a third (35%) said it’s difficult to have consistent monitoring across so many tools and that developers aren’t happy with all of the context-switching. So, it’s not surprising that 69% wanted to consolidate their tools because of challenges with monitoring, development delays, and unhappy developers.⁶⁶

Most of the 2022 Observability Forecast respondents (94%) said they use two or more tools (84% said four or more), despite almost half (47%) who said they prefer a single, consolidated observability platform. Only 21% primarily learned about interruptions with one observability platform.

Furthermore, just 7% said their telemetry data is entirely unified (they unify telemetry data in one place), and only 13% said the visualization/dashboarding of their organizations’ telemetry data is entirely unified (they visualize telemetry data in a single dashboarding solution).

So, the state of observability today is most often multi-tool—and therefore fragmented—and likely inherently complex to handle. About a quarter of survey respondents noted that too many monitoring tools, un-instrumented systems, a disparate tech stack, and siloed data are primary challenges that prevent them from prioritizing/achieving full-stack observability.⁶⁷ Using multiple tools is a big challenge.

A siloed approach to IT management has become obsolete in a hybrid IT world.⁶⁸ As organizations switch to a single, consolidated observability platform, how that platform is priced and billed becomes even more important. 

Significant data is unmonitored

As organizations migrate workloads from on-prem servers to cloud virtual machines, move workloads from servers and virtual machines to Kubernetes and containers, re-architect applications from monolithic to microservices, and adopt serverless functions, the volume of telemetry data increases significantly. So, it’s not surprising a 2022 survey about DevSecOps by GitLab found that the most important monitoring category is metrics, followed by logging.⁶⁹ However, high data ingest costs can lead to organizations sampling their telemetry data.

The 2021 report by 451 Research found that APM, infrastructure monitoring, and log management are the largest observability sectors, with a combined market share of 67%. APM and infrastructure monitoring were the largest categories in the observability market, representing 26% of the market by revenue each. APM vendors Cisco AppDynamics, Dynatrace, and New Relic ranked among the top 10 vendors by revenue in the broader application and infrastructure performance market.

RUM, which can shed light on frontend systems and customer experience, garnered new interest due to the boost in e-commerce during the COVID-19 pandemic. Respondents considered synthetic monitoring nice to have. Event correlation interest has grown as more vendors embrace observability and pull together different data types to provide more context for root cause analysis. And 92% of organizations thought AIOps tools would enable them to manage more workloads with fewer employees; AIOps and machine learning operations (MLOps) in observability tooling have added value for organizations experiencing skills and personnel gaps in operations.⁷⁰

Most 2022 Observability Forecast respondents indicated their organizations deploy 17 different observability capabilities by as much as 57% (network monitoring) and as little as 34% (Kubernetes monitoring). Just 3% indicated that their organizations have all 17 observability capabilities deployed, while 61% had four to nine deployed.

For example, almost half (45%) had deployed APM, 51% had deployed infrastructure monitoring, and 50% had deployed log management. Just over half said they deploy environment monitoring capabilities, like database, infrastructure, network, and security monitoring, as well as log management. RUM capabilities, like browser and mobile monitoring, and services-monitoring capabilities, like APM, were in the 40% range. Monitoring capabilities for emerging technologies, like AIOps, Kubernetes monitoring, ML model performance monitoring, and serverless monitoring, were among the least deployed, with each hovering in the 30% range.⁷¹

The 2022 State of Logs Report by New Relic saw a 35% year-over-year increase in logging data. It also found that 56% of New Relic customers use logs with infrastructure monitoring. And approximately 14% use logs alongside APM, a 68% year-over-year increase, which it expects to rise.⁷²

While 2022 Observability Forecast respondents used some level of data-driven observability insights in all stages of the SDLC, only about a third of respondents used full observability in each stage.

Just 5% had a mature observability practice (by the report’s definition). And only 27% had achieved full-stack observability (by the report’s definition). An even smaller percentage—3%—said that their organizations have already prioritized/achieved full-stack observability.⁷³

In addition, just 30% of respondents from a 2022 survey about DevSecOps by GitLab said that observability/monitoring is part of their DevOps implementations.⁷⁴ And only 37% of developer respondents from a 2022 developer survey by Stack Overflow said that their organizations have observability tools.⁷⁵

These findings reveal that most organizations do not monitor their full tech stacks. However, this is changing.

The observability forecast

In the next few years, most organizations plan to ramp up cloud, cloud-native, open-source, and observability deployment and budgets.

Cloud usage and spending

Gartner forecasts that the revenue for cloud application services will grow by almost US$36B (approximately 34%) from 2020 to 2022.⁷⁶ And it forecasts worldwide end-user spending on public cloud services to grow 20.7% (higher than the 18.8% growth forecast for 2022) to total US$591.8 billion in 2023, up from US$490.3 billion in 2022.⁷⁷

A 2021 Barclays survey found that 94% of CIO respondents expected to increase public cloud-related spending, and 37% expected to increase private cloud-related spending in 2022. It also predicted that 41% of workloads would run in the public cloud (up 24% from 33% in 2021) and that 26% of total IT spend would go towards the public cloud (up 18% from 22% in 2021) in 2022.⁷⁸ A 2022 Barclays survey found that 66% of respondents expect to increase public cloud-related spending, and 48% expect to increase private cloud-related spending in 2023. It expects the percentage of workloads going toward the public cloud to increase from 34% in 2022 to 40% in 2023. And it expects the percentage of IT spend going toward the public cloud to increase from 22% in 2022 to 24% in 2023.⁷⁹

IDC expects IT budgets to shift away from in-house resources toward various cloud delivery models in 2022 and 2023, which means that “in-house IT operations organizations will get smaller and put greater emphasis on their ability to monitor and optimize business applications and services on an end-to-end basis.” IDC also expects revenue from cloud services to represent 54% of the software-as-a-service (SaaS)-based IT operations analytics (ITOA) management software market by 2025 (up 20% from 34% of the market in 2020).⁸⁰

Cloud-native and open-source adoption and spending

As cloud-native takes center stage in enterprise cloud in 2022, Forrester predicts that container adoption will hit 50%.⁸¹ Gartner predicts that by 2025, 85% of organizations will run containers in production (up from less than 30% in 2020), and the proportion of current enterprise applications that are containerized will rise to 15% (up from 5% in 2020).⁸²

A 2021 report by ISG Research found that observability is mainstream and predicts that the need for these tools will grow as more enterprises use containerized applications in production.⁸³ 

Gartner also predicts that, by 2025, 70% of new cloud-native application monitoring will use open-source instrumentation rather than vendor-specific agents for improved interoperability,⁸⁴ and 70% of new cloud-native applications will adopt OpenTelemetry for observability rather than vendor-specific agents and software development kits (SDKs).⁸⁵

IDC expects “open-source components to represent the core of many major big data solutions and to provide important enabling technology for ongoing IT operations analytics innovation,” which means enterprise IT buyers may prefer commercially-supported, open-source-powered ITOA solutions to gain affordable access to the latest in technology innovation.⁸⁶

The 2021 Observability Forecast found that, as open-source popularity expands and open-source software adoption becomes more commonplace (55% of IT departments allocated 5–10% of their budgets, and 29% allocated 10%+ for open-source tools in 2021), it expects observability for Kubernetes and containers to grow rapidly. Most (88%) respondents were exploring Kubernetes and containers, and 40% expected to be in production by 2024.⁸⁷

Additionally, 39% of the 2022 Observability Forecast respondents said the adoption of open-source technologies such as OpenTelemetry is driving the need for observability.⁸⁸

Observability deployment and budget plans

Because they consider observability mission-critical, 74% of the 2021 Observability Forecast respondents noted there’s room to grow their observability practice.⁸⁹

Gartner predicts that organizations will increase their adoption of observability tools by 30% per year by 2024,⁹⁰ and applied observability ranked second in its top 10 strategic technology trends for 2023.⁹¹

As IT and application environments increasingly move toward complex, cloud-based microservices, the 2022 Observability Forecast found technology professionals have bold plans to ramp up observability capabilities and proactively find and resolve issues before they affect customer experience and application security. By 2025, 88–97% of 17 different observability capabilities are expected to be deployed. This finding suggests that most organizations may have robust observability practices by 2025.

Most (69%) said they allocate more than 5% but less than 15% of their IT budgets for observability tools, with 14% allocating more than 15%. Just 3% allocated more than 20%, and only 16% allocated less than 5%. Organizations with more mature observability practices (by the report’s definition) and those with the most capabilities deployed tended to have the biggest observability budgets.

However, observability budgets are on the rise as a top priority—72% expected to increase or maintain their observability budgets next year (including 14% significantly or extensively).⁹²

A 2021 survey by 451 Research saw a compound annual growth rate of 11% for the application and infrastructure performance market and predicts that it will reach a total revenue of US$15.6B in 2025.⁹³

A barrier to achieving observability

Undoubtedly, observability is becoming a priority. So what’s preventing organizations from adopting it?

In a 2021 survey by 451 Research, 42% of respondents indicated that cost is a primary challenge of using cloud-native technology such as containers, Kubernetes, and serverless.⁹⁴

A 2021 report by 451 Research found that because organizations see APM as expensive tooling, they prioritize it for more critical apps only instead of applying it across their apps. In addition, highly distributed, microservices-based applications can generate huge amounts of telemetry data, and organizations are now managing more logs than ever. So organizations have to balance storing as many logs as possible for the most granular insights and prioritizing certain logs for longer-term storage to alleviate cost concerns.⁹⁵ This is known as data sampling.

The 2021 Observability Forecast found that the biggest barrier to achieving observability is a lack of resources—38% of respondents say that observability platforms are too expensive.⁹⁶

When asked about the primary challenges preventing them from prioritizing/achieving full-stack observability, 27% of the 2022 Observability Forecast respondents cited a lack of budget, while 22% said it was too expensive. And 36% said budget-friendly pricing is the most important pricing feature for their observability tools/platform.⁹⁷

Conclusion

Driven by digital transformation and cloud and cloud-native adoption, observability has become mission-critical. But, clearly, pricing and billing can be a barrier to achieving full-stack, end-to-end observability.

A 2021 observability survey by ETR found that pricing and total cost of ownership (TCO) were among the most significant weaknesses for most observability vendors.⁹⁸ Historically, most observability vendors have used host- and telemetry-based pricing and subscription-based billing. This begs the question of how observability platforms are priced and billed and whether the observability landscape needs a pricing overhaul.

References

Aase, Jan Erik, Blair Hanley Frank, and Prakash N. October 8, 2021. Executive Summary: ISG Provider Lens™ Container Services and Solutions - U.S. 2021. N.p.: ISG Research. https://research.isg-one.com/reportaction/Quadrant-ContainerServicesSolutions-US-2021/Marketing.

Baltazar, Henry. December 10, 2021. Voice of the Enterprise: Storage, Transformation 2021 - Advisory Report. N.p.: 451 Research S&P Global Market Intelligence. https://clients.451research.com/reportaction/103256/toc.

Barclays. 2022. “AWS v. Azure v. GCP - 4QCY21.” Barclays Live. https://live.barcap.com.

Basteri, Alicia, and Daren Brabham, Ph.D. September 14, 2022. 2022 Observability Forecast. N.p.: New Relic. https://newrelic.com/observability-forecast/2022/about-this-report.

Bhat, Manjunath, Padraig Byrne, Arun Chandrasekaran, Mark Horvath, Mark O'Neill, and Gregg Siegfried. December 3, 2021. Predicts 2022: Modernizing Software Development is Key to Digital Transformation. N.p.: Gartner. https://www.gartner.com/doc/4009060.

Brewer, Buddy. 2019. “Global Research Survey Results: Leveraging Digital Transformation to Enhance the Customer Experience.” New Relic. https://newrelic.com/blog/best-practices/global-research-survey-digital-transformation.

Carey, Scott. 2021. “What is observability? Software monitoring on steroids.” InfoWorld, February 18, 2021. https://www.infoworld.com/article/3607980/what-is-observability-software-monitoring-on-steroids.html.

Casey, Kevin. 2022. “Kubernetes by the numbers, in 2022: 11 stats to see.” The Enterprisers Project. https://enterprisersproject.com/article/2022/10/kubernetes-statistics-2022.

Caveza, Scott, Satnam Narang, and Rody Quinlan. 2020. 2020 Threat Landscape Retrospective. N.p.: Tenable. https://static.tenable.com/marketing/research-reports/Research-Report-Threat_Landscape_2020.pdf.

Cite Research and New Relic. September 14, 2021. 2021 Observability Forecast. N.p.: New Relic. https://newrelic.com/resources/report/2021-observability-forecast.

Cloud Native Computing Foundation. February 10, 2022. Cloud Native Computing Foundation Annual Survey 2021 Report. N.p.: Cloud Native Computing Foundation. https://www.cncf.io/reports/cncf-annual-survey-2021.

Dai, Charlie, Jeffrey Hammond, Sam Higgins, Paul Miller, Lee Sustar, and Tracy Woo. October 27, 2021. Predictions 2022: Cloud Computing. N.p.: Forrester Research. https://www.forrester.com/report/predictions-2022-cloud-computing/res176454.

Dayan, Danel, Chiraag Deora, Jack Mattei, Jason Mendel, and Dharmesh Thakker. November 2, 2021. State of the OpenCloud 2021. N.p.: Battery Ventures. https://www.scribd.com/document/536774580/battery-ventures-opencloud-report-2021.

Elliot, Stephen, and Harsh Singh. May 2022. The Business Value of the New Relic Observability Platform. N.p.: IDC and New Relic. https://newrelic.com/resources/white-papers/idc-business-value.

Fee, Nočnica. 2020. “How to Know if You're Faking Observability.” New Relic. https://newrelic.com/blog/best-practices/how-to-know-if-you-are-faking-observability.

Flexera. 2022. 2022 State of the Cloud Report. N.p.: Flexera. https://info.flexera.com/cm-report-state-of-the-cloud.

Fong-Jones, Liz, Charity Majors, and George Miranda. October 26, 2021. Observability Engineering. N.p.: O’Reilly Media. https://www.oreilly.com/library/view/observability-engineering/9781492076438/ch01.html.

Forrester. September 2018. The Total Economic Impact™ of the New Relic Platform. N.p.: New Relic. https://newrelic.com/analyst-reports/total-economic-impacttm-new-relic-platform.

Gartner. 2020. “Gartner Forecasts Worldwide Public Cloud Revenue to Grow 6.3% in 2020.” Gartner. https://www.gartner.com/en/newsroom/press-releases/2020-07-23-gartner-forecasts-worldwide-public-cloud-revenue-to-grow-6point3-percent-in-2020.

Gartner. 2021. “Gartner Forecasts Worldwide Public Cloud End-User Spending to Grow 23% in 2021.” Gartner. https://www.gartner.com/en/newsroom/press-releases/2021-04-21-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-grow-23-percent-in-2021.

Gartner. 2022. “Gartner Forecasts Worldwide Public Cloud End-User Spending to Reach Nearly $600 Billion in 2023.” Gartner. https://www.gartner.com/en/newsroom/press-releases/2022-10-31-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-reach-nearly-600-billion-in-2023.

Gartner. n.d. “Definition of Digital Transformation - IT Glossary.” Gartner. Accessed September 26, 2022. https://www.gartner.com/en/information-technology/glossary/digital-transformation.

Gigamon. 2022. State of Ransomware for 2022 and Beyond. N.p.: Gigamon. https://www.gigamon.com/content/dam/gated/wp-gigamon-report-state-of-ransomware.pdf.

GitLab. 2022. The GitLab 2022 Global DevSecOps Survey. N.p.: GitLab. https://about.gitlab.com/developer-survey.

Grieser, Tim. October 2021. Worldwide IT Operations Analytics Software Forecast, 2021–2025. N.p.: IDC. https://www.idc.com/getdoc.jsp?containerId=US48251621.

Groombridge, David. 2022. “Gartner Top 10 Strategic Technology Trends for 2023.” Gartner. https://www.gartner.com/en/articles/gartner-top-10-strategic-technology-trends-for-2023.

Haff, Gordon. February 22, 2022. The State of Enterprise Open Source. N.p.: Red Hat. https://www.redhat.com/en/resources/state-of-enterprise-open-source-report-2022.

Harvey, Nathan, Michelle Irvine, Dustin Smith, Dave Stanke, and Daniella Villalba. September 21, 2021. Accelerate State of DevOps 2021. N.p.: Google Cloud. https://services.google.com/fh/files/misc/state-of-devops-2021.pdf.

Katsurashima, Wataru, Daryl Plummer, and Michael Warrilow. October 12, 2021. Emerging Technologies: Kubernetes and the Battle for Cloud-Native Infrastructure. N.p.: Gartner. https://www.gartner.com/doc/4006796.

Lenschow, Raimo. October 13, 2022. “2H22 CIO Survey: Outlook Declines.” N.p.: Barclays. https://live.barcap.com/go/publications/link?contentPubID=FC2686494.

Lenschow, Raimo. 2021. “Technology: 2H21 CIO Survey: 2021 Outlook Inching Upwards; 2022 Steady.” Barclays Live. https://live.barcap.com/PRC/publication/FC_TEJ-bGFfMTY0NzI4MDYwMTk0NX4gfiB-IH4g_2619989.

Lenschow, Raimo. 2022. “Q4 Public Cloud Update; Strong Growth, Microsoft Takes Share.” Barclays Live. https://live.barcap.com.

Lenschow, Raimo. 2022. “CQ3 Public Cloud Update; Headwinds Visible.” Barclays Live. https://live.barcap.com/go/publications/link?contentPubID=FC2690083.

Linthicum, David S., and Andy Thurai. December 18, 2020. Key Criteria for Evaluating Cloud Observability. N.p.: GigaOm. https://gigaom.com/report/key-criteria-for-evaluating-observability.

Longitude Research and New Relic. June 20, 2020. Deeper Than Digital: Top-performing modern enterprises show why more perfect software is fundamental to success. N.p.: New Relic. https://newrelic.com/resources/ebooks/more-perfect-software-gated.

Loukides, Mike. January 25, 2021. Where Programming, Ops, AI, and the Cloud are Headed in 2021. N.p.: O’Reilly Media. https://www.oreilly.com/radar/where-programming-ops-ai-and-the-cloud-are-headed-in-2021.

Lyman, Jay. May 21, 2021. Voice of the Enterprise: DevOps, Workloads, & Key Projects. N.p.: S&P Global Market Intelligence. https://clients.451research.com/reportaction/102063/Toc.

New Relic. December 6, 2021. 2022 O11y Trends Report. N.p.: New Relic. https://newrelic.com/resources/report/observability-trends-report.

New Relic. 2021. Cloud Native Is the New Normal: Is Your Environment Optimized for Success? N.p.: New Relic. https://newrelic.com/resources/ebooks/cloud-native-is-new-normal.

New Relic. October 2022. 2022 State of Logs Report. N.p.: New Relic. https://newrelic.com/resources/report/2022-state-of-logs.

Ouillon, Grégory. 2022. “Observability and growth: six strong trends to boost your company's performance in 2022.” Forbes France, January 30, 2022. https://www.forbes.fr/business/observabilite-et-croissance-six-tendances-fortes-pour-booster-la-performance-de-votre-entreprise-en-2022.

Radiant Logic. 2022. “New Study Reveals Identity Sprawl Plagues Organizations.” Radiant Logic. https://www.radiantlogic.com/news/new-study-reveals-identity-sprawl-plagues-organizations-with-60-percent-reporting-over-21-disparate-identities-per-user.

Rogers, Liam. December 16, 2021. Application and Infrastructure Performance Monitoring (AIPM) Market Map™ 2021. N.p.: 451 Research. https://clients.451research.com/reportaction/103322/Toc.

Rogers, Liam. 2021. “Observability sees an uptick in funding alongside continued M&A by incumbents.” 451 Research S&P Global Market Intelligence. https://clients.451research.com/reportaction/103143/Toc.

Stack Overflow. 2022. 2022 Developer Survey. N.p.: Stack Overflow. https://survey.stackoverflow.co/2022.

Stripe. September 2018. The Developer Coefficient. N.p.: Stripe. https://stripe.com/files/reports/the-developer-coefficient.pdf.

United States Security and Exchange Commission. February 2, 2022. Alphabet, Inc. Form 10-K: Annual Report for Fiscal Year Ending December 31, 2021. N.p.: United States Security and Exchange Commission. https://www.sec.gov/ix?doc=/Archives/edgar/data/0001652044/000165204422000019/goog-20211231.htm.

United States Security and Exchange Commission. February 4, 2022. Amazon, Inc. Form 10-K: Annual Report for Fiscal Year Ending December 31, 2021. N.p.: United States Security and Exchange Commission. https://www.sec.gov/ix?doc=/Archives/edgar/data/0001018724/000101872422000005/amzn-20211231.htm.

Vellante, Dave. 2021. “Cutting through the noise of full-stack observability.” SiliconANGLE Media. https://siliconangle.com/2021/11/06/cutting-noise-full-stack-observability.

Vellante, David. 2022. “Breaking Analysis: Latest CIO Survey Shows Steady Deceleration in IT Spend.” Wikibon. https://wikibon.com/breaking-analysis-latest-cio-survey-shows-steady-deceleration-spend.