Table of contents
Let’s talk data. You love it, we love it. So indulge us for a moment, please. IDC predicts that by 2020, there will be more than 44 zettabytes (or 44 trillion gigabytes) of data in the ‘Digital Universe.’ That’s a lot of information. Already there’s enough that pretty much every company, in nearly every industry, has the potential to access simply astounding quantities of information. What remains to be seen—that is, what separates the good from the great companies—is how well they extract and use that data.
That’s what we at New Relic wanted to find out when it comes to data about software. For the second consecutive year, New Relic surveyed hundreds of companies in 22 different industries to check where they stand on their journey to becoming data driven in their pursuit of software success.
So what did we find out? Drumroll please… companies are getting better at using data, but many admit they still have a good deal of work ahead of them. Here are a few more highlights of our findings, including notable changes from last year’s results:
- Software development overtook marketing as the department that respondents believe use data most effectively in 2015, with the majority of those responses coming from marketers themselves.
- Compared to 2014 when most companies released code monthly or even less frequently, more than half of companies released code weekly or more frequently in 2015.
- More respondents reported in 2015 that they didn’t know whether their company was better at using data than the competition (27% in 2015 versus 13.7% in 2014).
- Software teams reported that they are best at tracking application and infrastructure metrics, but not nearly as good at tracking customer experience.
- More than a quarter of companies believe their software teams are good at experimenting with new features.
- For three-fourths of respondents, cultural, technical, and other blockers persist in hindering experimentation critical to software innovation.
That’s not all of course. Read on for more juicy survey results and the opportunity to compare your experience and your company’s to others in your industry and situation.
Who we surveyed
In December 2015, using Twitter and email to reach potential respondents, New Relic conducted its second annual survey to find out how companies are using data to effectively guide decisions about investment, product development, and operations. The target audience for the survey was people responsible for funding, designing, building, marketing, or operating customer-facing Web or mobile software.
Nearly 500 people from 490 companies responded to the survey, with an interesting mix of job types, industries, and company sizes:
- Customer-facing versus back-end/internal systems: The majority of respondents (72%) stated that they are responsible for success in delivering a mobile and/or Web customer experience. Another 16% of respondents work on back-end or internal systems and 12% were responsible for neither.
- All sizes of companies: Slightly more than half of the respondents (59%) work at companies with fewer than 100 employees, while 23% represented the mid-market where company size is between 100 and 1,000 employees. Large companies with more than 1,000 employees comprised 18% of respondents.
- Cross-section of industries: Overall, 39% of the survey respondents work for technology businesses. The single largest industry represented in the survey was B2B software at 13%, with Internet close behind at 12%.
- A variety of roles: More than a third of respondents were developers and software engineers (37%), while the second largest group was executives and managers in engineering and operations, with 23%. Rounding out the job areas were product management, marketing, sales, line of business, and security/compliance.
- Web-first versus historically off-line: Web-, mobile-, or software-native businesses represented 75% of the companies in the survey, while the remaining 25% self-identified as historically offline businesses with Web and/or mobile investments.
Finally, 65% of respondents identified themselves as New Relic customers, with 80% of the New Relic customers being mobile/Web-native companies and the remainder being traditional offline businesses.
On the road to data-driven software success
What does it take to become a data-driven company? Ask Etsy, the online marketplace for handmade or vintage items, which has made the transformation over the past five years. According to an article titled, "Why a data-driven transformation requires a cultural shift," in CIO magazine, data and metrics now support the entire company’s operations. The motto at Etsy is, “If it moves, graph it.” After starting with just five metrics in total, Etsy today has dozens of metrics for each of its services. Measuring and tracking metrics allows Etsy to give developers ownership of the success or failure of their process.
Is your company an Etsy in its use of data or is it still making its way toward creating a data culture? Here’s what our findings show about the state of data-driven software decisions.
Takeaway #1: Three-quarters of companies are on a data-driven journey
When asked how their company uses data to make decisions on how to invest, prioritize features, and run customer-facing Web and mobile applications, nearly half of our respondents (47%) acknowledged that they are still learning how to use data to drive important decisions. Another 24% indicated that their organizations are making good use of data in some places. Organizations making heavy use of data for software decisions came in at 9%. Not surprisingly, 95% of that group are Web- or mobile-native companies. To the dismay of 21% of respondents, their companies are not even close to being able to use data to drive investment or product decisions.
We also asked what would make the respondents’ departments more effective in using data in 2016. More than half (52%) said that dashboards that monitor key performance indicators would be the number one thing that would help their departments become more data-driven. The next three results on the data wish list included two technology-specific items and a staffing need. All three were nearly equal in importance for our respondents: an easier way to analyze data (42%), data that’s more accessible to those who need it (41%), and having more people with data analysis skills (39%).
Takeaway #2: Most businesses don’t feel they are better than the competition at using data
We also asked how respondents believed they stacked up against the competition when it comes to using data. Respondents were fairly evenly split between feeling superior to or about the same as their competitors (26% compared to 24%). Nearly a third (28%) didn’t know how their companies compared to the competition. Interestingly, nearly 84% of Web, mobile, or technology-native companies rated themselves ahead of the competition compared to 16% of historically offline companies.
Large companies (those with 5,000 employees or more) reported having more data nerds than competitors, while companies smaller than 20 employees were more likely to report that they didn’t know where they stand compared to the competition.
Industry also played a role in how companies felt about their own data usage versus the competition. Companies in education, telecommunication, real estate, and media and publishing were more likely to report that they are superior to the competition when it comes to using data. On the other hand, organizations in advertising and marketing, government, retail and consumer goods, and B2B technology were more likely to report that they were deficient in data usage compared to their competitors.
Speaking of competition, only 38% of the respondents believed that their company is superior to competitors in using mobile and Web software technologies to interact with customers. Less than half (40%) felt they were on par with the competition, while 14% believed they were inferior.
Suggested action: Use data to prioritize
What can your team do to improve its use of data to drive software decisions? One approach is to use data to prioritize projects and tasks, so that people work on the areas that are most important to your company’s goals. One way to do that is implement software analytics that show, for example, which parts of your application are most heavily used, which features are used by the customers who drive the most revenue, or which areas of code generate the most errors.
Takeaway #3: There’s a perception that the software development team uses data most effectively
When asked which department at their company uses data most effectively, 30% of respondents reported that software development is the most data-savvy department, followed by marketing at 21% and IT operations at 17%. Respondents identifying themselves as product management believe that both marketing and software development use data equally well.
There was a certain amount of bias towards a respondent’s own department, with people in IT operations, marketing, and software development all reporting that their own departments use data most effectively. For those respondents self-identified as marketers, 60% named their own department as the most effective user of data. Software development and IT operations were a bit more modest, with each group citing their own department as best data user 41% of the time.
Takeaway #4: Companies are confident about their use of application metrics.
This year we wanted to delve into specific areas of metrics to see how companies ranked the ability to track and use them. We started with the metrics we defined in our DevOps success framework (see our ebook on the topic at http://newrelic.com/measuring-devops) and grouped them into four categories:
- Customer experience
- Business results
We asked respondents to rate how well they measure each type of metric on a scale of 1 through 5, with 1 meaning they track it well with visibility and collaboration across the software team and 5 meaning they track it poorly. What we found was that of the four categories of metrics we asked about, respondents reported that the software team tracks application metrics more effectively than the other three types of metrics. The score for application metrics was 2.81, while infrastructure was a close second, with a score of 2.88. Business results and customer experience ranked third and fourth respectively, with scores of 3.09 and 3.18.
While not unusual, these findings definitely reflect the fact that most companies have not yet reached data Shangri-La, and are still growing their usage of data for customer experience and business decisions.
Similarly, we also asked respondents about data-driven feedback loops for software teams in the categories of application performance, customer experience, and business success. When asked which area would have the most positive impact in helping their organization deliver great customer-facing software in 2016, customer engagement (36%) and application performance (35%) topped the list, with business success close behind (29%).
For the purposes of the survey, customer engagement was defined as a function delivered by product owners/managers and developers, application performance by developers and IT operations, and business success by customer behavior and marketing or sales teams.
Suggested action: Enable self-service feedback
People want feedback on the quality and impact of their work. Giving engineers easy access to metrics that show performance, adoption, and business results of the software they create can help improve software quality. So start small with performance metrics such as uptime, application response time, SQL query performance, and resource usage. Then as your team gets used to a more data-driven culture, introduce additional metrics that can help determine the success of the team as a whole (for example, in proving the effectiveness of DevOps efforts).
Full speed ahead on innovation
One of the cornerstones of the agile development and DevOps movements has been to increase the speed of product delivery, providing faster value to customers and to the business. We were keen to see where companies are today with release frequency and the ability to experiment that helps create the groundwork for innovation.
Takeaway #5: One-quarter of software teams are good at experimenting with new features
Experimenting helps companies evolve the software experiences they deliver and enables the software development team to achieve better results for the business. In our survey, 73% of survey respondents indicated some type of difficulty that inhibits experimentation, while 27% said that their organizations are good at enabling experimentation. The majority (73%) of those who said their organizations are good at experimenting are New Relic customers.
More than a quarter of respondents (27%) reported that a limited ability to measure what works and what doesn’t hindered experimentation. Other reasons include: a culture that does not accept failures (13%), slow software release frequency (10%), too difficult to roll back features (9%), and lack of accountability between teams (8%).
Takeaway #6: More than half of companies release code at least weekly
According to Puppet Labs’ 2015 State of DevOps Report, high-performing IT organizations deploy 30 times more frequently with 200 times shorter lead times. Last year we found that a plurality of companies (nearly 25%) in our survey released code to production on a monthly basis. The next most common responses were quarterly and then weekly deployments. A surprising 13% of respondents said they didn’t release any code to production in 2014!
In 2015 however, more than half of the respondents (57%) said they released code into production weekly or more frequently. One-quarter of respondents reported releasing code monthly, with 10% releasing code quarterly, and 9% even less frequently than that.
For 2016, survey respondents are aiming to speed deployment even more, with 66% expecting to deploy weekly or more frequently, an increase of 11 percentage points compared to 2015.
Interestingly, 83% of New Relic customers said they deploy weekly, daily, or more frequently. Compare that to 17% of non-New Relic customers who said they deploy that often and you can see how access to data provides the insight and confidence that enables rapid iterations and deployment of new features.
Suggested action: Start tracking your speed
Teams adopting a DevOps approach typically move towards a continuous delivery goal. To help your organization track its speed of development and delivery, consider measuring your lead time for changes and the frequency of code releases. At the same time, quality is an equally important goal, so tracking mean time to resolution can help you balance speed and quality goals.
Job expectations now include a data culture
We were right when we said everyone loves data. The vast majority of respondents are excited about working in a data culture.
Takeaway #7: For nearly everyone, a data culture is a desire or expectation
We asked survey respondents how they would feel about working at a new job where the department they were going to join had a culture of using data to prioritize work, build and adjust feedback loops in a logical way, and hold team members and other groups accountable. A whopping 90% indicated that they’d like or expect to work in a data-driven environment. A small number (8%) were unsure, while 2% would find it distasteful (we’re not sure what to make of that).
Takeaway #8: Let the Wookie win
Finally, given last year’s release of the film, “Star Wars: The Force Awakens,” we wanted to revisit on the question of which classic sci-fi franchise our respondents preferred: Star Trek or Star Wars. The envelope please: The majority of respondents favored the saga of the Force (65%) compared to Star Trek’s final frontier. May the force (of data) be with you!
As our survey results make clear, companies are continuing on their journey to a data culture, making significant gains in areas such as time to market. Here at New Relic, we fully expect to see that and other data-driven trends continue, with even more companies empowering teams with meaningful data for more informed decisions, faster innovation, and greater responsiveness to customers.
For all the data nerds, wannabes, and even skeptics reading this report, we hope you enjoyed seeing this data as much as we enjoyed bringing it to you. After all, we’re the company that collects more than 690 billion metrics every day so you can go out and make informed decisions about your software.
To learn more about using data to make better software decisions, visit: http://newrelic.com/software-analytics.