Table of contents
State of the Mobile Stack
The mobile ecosystem is constantly changing. Between all the software updates, new devices, and expanding carrier networks, it can be hard to keep a finger on the pulse of what’s going on and how the mobile experience you’re delivering is measuring up—which is exactly why New Relic produced this report.
Our mobile application performance management (m-APM) product, New Relic Mobile™, collects performance data from thousands of applications every day, giving us unique and valuable insights into actual industry performance data. We thought it’d be both helpful to mobile developers and interesting to the general public to shine a light into the overall health and performance of today’s mobile environment.
Some of the key questions you’ll find answered in this report include:
- Which wireless carriers and countries have the
fastest response times based on throughput?
- Which mobile devices have the best performance
based on throughput?
- What typically causes a slow user interaction?
- Which operating systems and APIs are most widely
used by mobile developers worldwide?
The following findings are based on mobile data New Relic observed between January 27 and February 2, 2015, and are not meant to be a reflection of the market in its entirety. In charts where average response time is indicated, the measurement is based on average response time per application monitored. The median average response time is the value greater than or equal to 50 percent of the per-application average values.
As you read on to find out the answers to the above questions and more, we think you’ll find this benchmark data invaluable in planning and strategizing your own mobile app development.
Disclaimer: New Relic has not verified, validated, or audited the data used to prepare this benchmarking report. New Relic makes no representations or warranties with respect to the adequacy or accuracy of the information contained herein, and disclaims any and all liability for loss of any kind suffered by any party as a result of this benchmarking report.
The Need for Speed
Whether you’re a mobile developer or app owner, one of the first things you’re going to want to know about your customer experience is how fast your app isperforming in end-users’ hands. In this section we’ll examine both average response times and request volume by U.S. and global carriers, regions, and devices.
Top U.S. Carriers by Request Volume
The first area we explored was mobile carrier performance—specifically, which carriers in the United States have the busiest network?
Out of the top four U.S. carriers, it turns out that AT&T carries the most traffic, and has the slowest average response time of the four carriers. On the opposite end of the response time spectrum is Verizon, which has the second highest request volume.
Top Non-U.S. Carriers by Request Volume
In addition to finding out which U.S. carriers were the fastest, we also took a look at carrier performance on a global scale.
An interesting finding here: the Netherlands has the fastest wireless network performance in the world. As you can see in the charts, the carriers in the top three spots (T-Mobile NL, Vodafone Vodafone NL, and NL KPN) are all located in the Netherlands. While carriers in the Netherlands have a lower request volume than say, the U.S. or South Korea, the volume is in the top 20 worldwide, making placement for response time in the top three spots a pretty impressive result. T-Mobile Netherlands, in particular, shows solid performance.
App Responsiveness by Region
While we’re on the topic of global performance, let’s also take a look at median average response times by country. With the three fastest global carriers all being located in the Netherlands, one might assume that the country with the fastest median average response time would indeed be the Netherlands. But that’s actually not the case.
The country with the fastest mobile performance is Montenegro, which clocks in with a 0.25-second median average response time. The next two slots are taken by a couple of Montenegro neighbors: Bosnia and Herzegovina, and Slovenia, respectively.
From a request volume perspective, the United States takes a very strong lead with a whopping 33 billion requests per week. Yet the U.S. has a pretty slow 4.05-second median average response time. Canada has the second highest request volume, yet it has a much faster network than the U.S. — a zippy 1.85 seconds on average.
We were initially surprised to see the Netherlands come up as the country with the third highest request volume, but after taking a look back at the specific time frame this data was collected (January 27 – February 2, 2015), we found a possible correlation.
There was a big news story that broke on Jan. 29, 2015 about a 19-year-old man who forced his way into the studios of Dutch national television with a fake pistol while demanding airtime. Canada has the second highest request volume, with a much faster network, potentially driving up request volume dramatically.
App Responsiveness by Device
So how do mobile devices stack up? The Amazon Kindle Fire takes the top spot for app responsiveness, with a median average response time of 0.2 seconds. Tying for second is the iPod Touch 3 and the Lava Iris, a smartphone from Indian mobile handset company Lava. The HTC Desire HD clocks in at third, while Samsung and Apple devices make up the rest of the top 10 list.
Although Apple devices didn’t top the most responsive list, they definitely dominated on the popularity side, taking up all top 10 slots for maximum request volume. Based on our data, it appears the iPhone 5S is the most widely used Apple mobile device, with the highest request volume of almost 10 billion requests per week. Apple’s latest smartphone, the iPhone 6, had the second highest request volume, followed by the iPhone 5, 4S, and 5C taking up the rest of the top 5 slots.
When Things Go Wrong
Now that we’ve explored response times and request volume, let’s take a look at mobile performance from another perspective: when things go wrong. Specifically, we found out how often crashes occurred, and what’s typically the bottleneck when performance is degraded.
Crashes per user
Across more than 100 million devices, we observed 9.5 million app crashes in January 2015. That breaks down to an average of one crash per 11.22 users, which means nearly 10% of your hard-earned users are mad at your app.
Network vs. server failure rate
In addition to tracking crashes, we ran a comparison of average network failure rate versus average server error rates. The result? Servers encounter errors more frequently than networks fail. The average network failure rate (when the app didn’t receive a response from the server) across the apps we monitor was 0.9 percent. Meanwhile, the average server error rate (when the app received a response from the server, but the server could not complete the request) across apps was 1.6 percent.So if you’re responsible for building and maintaining a mobile app, be sure to keep a particularly close watch on how your backend app servers are affecting performance.
Anatomy of an Interaction
For mobile developers deep in the trenches building and shipping mobile apps, fully understanding the nuances of how smartphone APIs work can oftentimes present challenges. Processors, memory, OS versions, and manufacturer customizations all affect how code runs, which is why we took a look at the breakdown of code execution in a user interaction. (By user interaction, we mean any action performed in an app’s UI—presenting a new screen, refreshing a user’s account, displaying purchases made by users, etc.).
So how much time does a typical interaction spend running different types of code? Turns out, the most time (by far) is spent on network requests. Code execution of an interaction (in seconds) On average, your app might spend three times as long waiting for a network request compared to creating the UI in code. But in the worst cases, it could spend up to 10 times as long.
The key takeaway here: network response time (as well as view loading) can be quite variable, whereas SQLite/CoreData, image loading, and JSON parsing are much more uniform, showing consistent performance across devices and apps. If you’re a mobile developer, carefully tune your database access. And be sure to keep a close watch on network requests—they will likely be the source of slowness that is hard to reproduce.
Measuring Mobile Usage
Next up in our mobile analysis: when do users interact with mobile apps the most? Is it first thing in the morning when people are checking their phones shortly after waking up? Late at night before they go to bed? Or smack dab in the middle of the day? Below we track the total app sessions occurring by time of day and by day of the week.
App sessions by hour of day
According to our data, 6:00pm GMT (10:00am PST) is when mobile traffic is at its highest, with more than 30 million sessions taking place in that hour alone. The second busiest time of day for mobile usage is 3:00pm GMT (7:00am PST), while the least busy time of day for mobile usage is 8:00am GMT (12:00am PST), clocking in 16 million sessions.
The app sessions we observed are likely more reflective of user behavior in the U.S., but it’s interesting to note that heavier worldwide mobile usage takes place early in the day versus later in the evening.
App sessions by day of week
We also took a look at whether there was a particular day of the week that mobile usage happened most frequently, so we broke down the total number of mobile sessions by day of week as well. Our data shows that mobile usage is fairly consistent throughout the week, but Sunday has the highest number of total sessions (not surprising, considering most people have the day off from work to endlessly surf their phones), while the lowest number of sessions took place on Wednesday.
When building a mobile app, one of the first big decisions developers will have to make is: iOS, Android, or both? And the decision-making continues as they then get into figuring out which third-party APIs they’re going to use as well (e.g. Which social or payment services do I want my app to connect to—Facebook, Twitter, PayPal, etc.?) This section takes a closer look at where mobile developers’ preferences may lie when it comes to operating system and third-party APIs.
Most popular operating systems based on usage
Despite Apple iOS devices dominating the total request volume by device breakdown, the most popular operating system based on usage (actual session minutes) in our list is Android version 4.4.2. The second most popular OS is Apple’s latest and greatest iOS 8.1.2 (in January 2015), while the third is iOS 7.1.2, indicating there’s likely quite a few people who are still on older iPhone models not equipped for the most up-to-date iOS version.
Most popular APIs based on usage
Next we looked at which third-party services were connected to most often from the mobile apps we monitored. Facebook came out on top as the most popular API with the largest number of apps connecting to it (surprise, surprise!). Following Facebook are Google APIs.
When exploring the data from a higher level, most of the popular APIs fell into one of five categories: social, analytics, advertising, CDNs, and mapping.
Mobile apps have the power to bring customer satisfaction and competitive advantage to all modern businesses. With a massive end-user shift towards mobile, delivering phenomenal mobile-first experiences are critical to establishing this advantage and differentiating your place in the market.
Now that you know what kinds of experiences the average user is having on various networks and devices, and how the inner workings of your app may impact their interactions, we hope you’ll use this insight to go out and make your mobile apps even better. Just remember that to get five-star ratings and rave reviews, you’ll need more than a simple feedback loop of product/user input and user testing—you need a full-fledged m-APM solution to give you the above type of data and more on a regular basis.
To learn more about monitoring mobile app performance from end-to-end in real time, visit: www.newrelic.com/mobile.