Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon
eBook

Metrics-Driven Approach to Mobile

Table of contents

Introduction

People love their mobile devices and their apps, so much so that mobile app usage grew an incredible 76% in 2014.1 Users today spend 34 hours on average each month on their mobile devices.2 Better yet for app developers, mobile apps are on a path to reach $70 billion in annual revenue by 2017, with non-game apps expected to double their share of the total market from 26 to 51% by 2017.3

However, getting a share of the mobile app market is no easy feat. First you need a brilliant idea for an app. Then your company needs to invest the time, money, and effort into developing and marketing the app. If your team does everything up to this point well, you start getting users to download the app. And that’s where the proverbial rubber meets the road.

Your app must deliver an excellent user experience or it will not be used. If the performance is bad, it’s difficult to use, it takes too long to get what the user needs, or other experience-killing problems are present, your app will fail to grow its user base. It’ll be neglected and forgotten by the users who downloaded it because it didn’t deliver the experience the user was expecting. Put into this context, the user experience will make or break your app.

As a developer, you want all your hard work to result in an app that users will love. But how can you make sure your app is doing everything right to deliver a great user experience? Read on for insight from PierreLuc Simard, CTO of Mirego, a company that designs and develops innovative mobile strategies for iPhone, iPad, Android and other mobile devices.

1 “App Usage Grew 76% in 2014, With Shopping Apps Leading the Way,” Sarah Perez, TechCrunch, January 6, 2015.
2 “Mobile Apps Are Fueling Huge Growth in Time Spent on Digital Media,” Mark Hoelzel, Business Insider, September 5, 2014.
3 “Mobile Apps Could Hit $70B in Revenues by 2017,” Dean Takahashi, VentureBeat, April 29, 2014.

Defining Objectives and Measuring Success

One of the first things you want to ask yourself at the beginning of any mobile project is, “What user experience would make this app a success?” With that answer in mind, you can define clear objectives and prioritize them, which in turn, will help you make decisions about when the product is ready to ship and what it’ll take to get to that point. This goaldefining process doesn’t have to be overly complex or painful. Simply identify up to three key things the application must achieve once it’s in the hands of users. Getting from objectives to a solid first version, however, takes a lot of energy – in fact, often more energy than initially anticipated. To make sure that your work provides the highest return, mapping objectives to real measurements as soon as possible is very important. To illustrate this measurement-driven approach, let’s imagine we’re building an application for tracking performance runners and walk through a process of how to choose metrics to track task completion as if we were starting development today.

Tying Your Overall Mission to Specific Objectives

As we’re targeting runners, our mission would likely be something along the lines of, “to track the running performance for as many runners as possible.” Analyzing this mission, we can break it down into a few objectives.

First, the core of the mission is to track running performance. Without that, it’s not possible to fulfill the rest of the mission. The application must make it easy for users to start recording a run and to continue recording for the duration of said run. We can also deduce that the recording must be reliable. Thus, your primary objectives would be:

  • Make recording a new run the easiest function for a user to do.
  • Ensure that data recorded is as accurate as possible.

The secondary part of the mission focuses on attracting and retaining as many runners as possible. To attract users, the UI should be robust yet simple, (when it makes sense) allow rich integration with a user’s social networks, and find opportunity to drive user engagement within the application.

Sharing allows users to share their experiences through their social groups, so new users can try out your app, too. Once a user does decide to try your application, the first interaction he/she has with it must be as painless as possible. Retention of the user is then achieved in two stages. First, nudge the users to keep them engaged long enough for them to perceive value in of the app’s core features. If the user sees no value in the application, no amount of reminders, push notifications or other gimmicks will drive engagement. So let’s add a few secondary objectives to our list:

  • The run must be shareable to allow application discovery through
    social networks.
  • The time from first install to first run recorded should be as short
    as possible.
  • The data collected must provide value to the user.

Goals will usually include a revenue goal either directly tied to purchase or larger user engagement with ads or other revenue models.

Now for the Important Part: Measuring Success

After creating objectives for your application, you need to ask yourself how to measure its success when stacked up against its objectives. Typically one objective will require up to three metrics to accurately measure progress. To gather and report on those metrics, you need to install a software analytics solution such as New Relic to track how your app is performing out in the wild; an analytics engine such as Google Analytics or Mixpanel to track user-focused metrics and demographic information or analyze marketing funnels; as well as a crash reporter such as New Relic Mobile to capture and report on when the application failed. Together, each of these tools will provide a coherent view of the application’s performance.

Going back to the objectives we set out for our running app, we can start to map each one to the metrics that can be gathered by our measurement solutions.

Metrics to Collect for Objective #1

Make recording a new run the easiest function for a user to do.

The first thing to look for when measuring the success of an application is a timed event that starts when the application was installed and when it reaches an important moment in its “core function.” In the case of our application, that core function would be recording a new run. Calculating the time between application start and the beginning of a recording will tell you if the feature is easy to find and the steps involved to record (e.g. setting time caps, etc.) are user friendly and have good defaults. So the metric to collect here would be:

  • Time from when application was first installed to when core function was used
  • Wait time (i.e. screen render time) leading to recording a run

Screen rendering time is directly correlated with the user’s wait time. If we want the function to be easy to use, it should also be quick to use.

Metrics to Collect for Objective #2

Data recorded must be as accurate as possible.

The data being gathered by the app is important to the user. After all, it’s this data relating to the user’s recorded run that gives the person a sense of reward and accomplishment. Thus an application should measure how well it’s doing its job. In the case of a running app, tracking the quality of the GPS information gathered would be a logical metric to focus on. You’d also want to track:

  • Number of times the GPS accuracy was of low quality for a given recording
  • Number of recordings that began with low accuracy
  • Time to reach good accuracy at beginning of a recording Number of failed attempts to save the recording back to the server
  • Number of errors and incomplete transactions saved to the server

Metrics to Collect for Objective #3

Run must be shared to allow application discovery through social networks.

Metrics here are pretty obvious. If you want to measure how effectively your app is being shared via social media, you’d likely want to collect:

  • Number of sessions that published to a social network
  • Number of outside page views from the social share

Metrics to Collect for Objective #4

Time from first install to first run recorded should be as short as possible.

To track this objective, take a look at two key events: the date when the user installed your app and the moment they recorded their first run. If you notice that these two numbers don’t match up, that’ll tell you that x number of people dropped off without actually giving your app a try. You can also measure how long it typically takes between the time someone first installs your app and starts to actually use it.

  • Date/time when the user installed your app
  • Date/time when the user starting using your app (in this case, to record a run)
  • Amount of time spent in the application before recording a run (should be as short as possible)

Metrics to Collect for Objective #5

The data collected must provide value to the user.

The more time your users spend interacting with the data your app provides, the more value you know it’s delivering to them. For this, you’d want to track metrics like:

  • Number of times user looks at “distance ran” data
  • Number of repeat shares on social media

Start Measuring Early

Once you know which metrics you need to track so you can measure the user experience, the next step is to define good baseline numbers for the application. For most new apps, there won’t be much data available to know what those numbers should be. That’s why it’s often helpful to look at other comparable applications or potential competitors. You can conduct your own little user experience study of competing applications to help you understand what should be measured. After you’ve established your baseline, start recording as much data as you can, as early as you can. Getting the numbers from internal testers (first alpha, then beta users) will help inform what the numbers should be for launch and for subsequent releases. Also, gathering the data early will help determine if the data is being gathered accurately and will give you insight into what alpha/beta users tried to do in your app and what QA covered in the test plan.

Measurement tips

  1. Deploy measurement software: You’ll need an application performance monitoring solution, an analytics engine, and a crash reporter.
  2. Establish a baseline: Track and measure comparable applications (even potential competitors) to inform your target metrics.
  3. Start recording data early: Gathering data from internal testing and alpha/ beta users helps you establish production performance goals and helps you be sure that you are tracking correctly.
  4. Keep measuring throughout the lifecycle: As new versions are developed and released, you need to make sure the experience isn’t compromised by changes.

Shipping is only the first step

Getting an application out the door and into the hands of real users is only the first step toward reaching app success. You need to monitor your app’s metrics throughout the product lifecycle, ensuring that, as new versions are being developed, they meet the application’s core mission and objectives. But in addition, you should also be monitoring your app’s performance to make sure it’s not causing user frustration. After all, users may decide not to use some features simply because they are too slow and not necessarily because they are bad. Continuous measurement and monitoring will help you continuously improve your app as it progresses into more mature releases.

How to measure using both macro and micro metrics

We know that mapping real measurements to your overall objectives is a good way to ensure that all the work you put into your app delivers the highest return in the form of a good user experience. But you shouldn’t stop there. There are other metrics to collect that can help you and your company make informed development decisions throughout the lifecycle of your app. Let’s walk through the two major categories of metrics that all mobile developers should be collecting: macro and micro metrics.

Macro Metrics: Ideal for Understanding User Behavior

Macro metrics help you see the bigger picture. They help you understand group behavior relating to a specific area of your application (for example, the percentage of users in a given month that used feature X). Macro metrics don’t show you exactly what they did with the feature, but they show you how popular the feature is based on usage and help you understand navigation patterns within a given feature or between different sections of the application.

When adding or changing features of an application, being able to make informed decisions based on real usage patterns makes a huge difference. What you want to avoid is making decisions solely based on a few reviewers’ comments or only based on support calls. When you do that, you have only a portion of the story. Having metrics that confirm usage patterns instead allows you to make changes based on the entire user base experience.

View Count

The first step toward understanding user behavior is making sure you collect view count for all the screens in your application. Using an analytics SDK (such as Google Analytics, Mixpanel or Localytics), you should create an event to track when each screen is presented to a user. After that, you’ll want to add events that represent feature usage.

Usage Count

“Usage count” is different from “views,” as it should count the number of times a user completed a goal within that feature, and consequently, your application. “Views” will allow you to compare usage between screens, whereas a feature “usage count” will allow you to understand more closely what percentage of your users are interacting with a specific feature and correlate it to their exposure to it. One thing you want to look at is the number of times a feature is used compared to the number of views of a key screen. For example, in a social application, you could look at the number of times a user saw an activity feed versus the number of times she liked something in the feed. Once you have both “views” and “usage count” set up, you then want to gather some goal and timing-related information (for example, the time it took for a user to attain certain goals within the application). This helps us understand if assumptions we’re making about the use of the application are correct, and if users are actually getting to the part of the application you want them to reach.

Micro Metrics: Perfect for Tracking Performance

Unlike macro metrics, which look at a large portion of your application usage, micro metrics look at usage related to a small portion of your application (for example, the average rendering time of a specific screen). Micro metrics are extremely helpful for the operations side of the house, since they help you understand if there are performance problems in the application. And this is really where mobile APM (or m-APM) shines.

If you’re using an m-APM solution like New Relic Mobile, you immediately start collecting micro metrics without having to instrument your application. The dashboard will automatically show you the performance of your API calls and screen render times, giving you a good idea of the wait times of your users right off the bat. If you’re missing some metrics, you can create a custom event and measure pretty much anything you want.

Tracking Interactions

A good m-APM will allow you to track interactions on top of already tracking individual actions (API requests, page render time). Using interactions allows you to track a group of actions (simultaneous or sequential) and report on their performance. Let’s say that you had a running app, and storing all the recorded run data requires three API calls (one to create the run, followed by an upload locations sample, and a simultaneous API call to upload an image of the map).

From a performance standpoint you want to see how long the three calls took, and especially, the two simultaneous calls. For the user, the upload finished when all three calls finished. Interaction metrics are great for tracking that experience because they have a start and stop concept. Having individual-call and grouped-call data allows you to see the real impact of performance degradation (or optimization) on the overall user experience while keeping the data needed to perform a root cause analysis.

Thus, micro metrics help you get things right the first time around. If you do end up making changes to your app, you’re doing so with a full understanding of whether or not they are likely to make things better or worse for your end users. As many mobile application developers and product managers know, one of the biggest problems with user retention is that the attention span of a user on a mobile device is so short. People get emails, they get various app notifications—there are all types of distractions that can easily take the focus away from your app. But micro metrics make sure that everything gets delivered seamlessly, so that there’s no downtime in which their attention can wander.

Metrics Overall: Critical in Driving Business Success

Both macro and micro metrics serve their own unique purpose in your overall app development lifecycle. But the beauty of collecting these metrics together is that they’ll help you avoid shooting in the dark when it comes to further development on your app. They give you a much more reliable and comprehensive way to know how and where to improve your app than simply relying on app store ratings and reviews. When you do decide to make a major change in your app, as Foursquare did in 2014, you’ll be in a much better position to make that pivot without upsetting your core user base and features. And if a problem does happen to occur, you’ll be able to show your users that you’re in control of the situation and have a fix in place, which is something they’ll appreciate. Collecting metrics helps you make more targeted decisions that help steer the business—and its reputation—in the right direction.

Improving performance with caching

The less time a user spends waiting for your application to respond, the better that experience will be. So let’s discuss a few tips related to caching that can help you ensure the best possible performance of your mobile app. Before we get started, consider these stats:

The average mobile session is anywhere between 1 and 4 minutes. (On tablets, it’s typically 2.5 to 3 minutes; on mobile phones, it’s about 1.5 to 2 minutes.)
The average mobile session is anywhere between 1 and 4 minutes. (On tablets, it’s typically 2.5 to 3 minutes; on mobile phones, it’s about 1.5 to 2 minutes.)
The average response time for a mobile app is 1.24 seconds. (80% of that time is spent on the network; 20% of that time is spent communicating with the server.)
The average response time for a mobile app is 1.24 seconds. (80% of that time is spent on the network; 20% of that time is spent communicating with the server.)

If we were to plug those numbers into this basic formula, session duration ÷ request duration = requests per session, we’d find out that the average app makes 120 requests per session. We did a bit of informal testing to check this calculation and found that:

  • To check bank account balance, a bank’s mobile app sends 43 requests in 70 seconds
  • To check-in on Foursquare, the mobile app makes 46 requests in 30 seconds
  • To open a message in Basecamp, the mobile app makes 24 requests in 40 seconds

As you can see, even just one simple action often requires a number of different requests to be made to the server. But the more requests your app has to make in order to complete that action, the longer the response time, and in turn, the worse the experience for the user. So the goal is to make the fewest requests possible to complete an action. And that’s where caching comes in.

Avoid Unnecessary Calls with Caching

Many API developers tend to overlook caching. After all, they’re used to using web frameworks in which the browser will cache everything by default. But when you’re developing for mobile, the default setting is not to cache anything. That means your app has to continually make the same calls every time the user performs an action. To avoid these unnecessary calls that slow down your app’s performance, here are three tips:

  1. Be specific with HTTP headers. By default, a mobile phone will process a request, show the data on its screen, and throw all that information away as the request gets processed. So what you want to do is have the server tell the mobile app to keep this data, and you do that by being declarative in your HTTP headers, as shown in the example below.
  2. Set specific cache control editors. Forgetting to set cache control editors is something that can trip up many junior developers. They’ll include a profile pic to load, for example, but they won’t set the cache control editor. So every time the app is opened, it’ll have to send an API call to pull up the image. You really want that data to be stored. That’s why you should remember to set specific cache control editors for your objects when uploading them in the metadata.
  3. Enable caching on the device. Once you set your HTTP headers and have the server telling the device to keep your data for a specified amount of time, the device will then need to be told where to keep it. If you choose to go with an iOS or Android devices’ default settings, only a very small amount of data will be stored on the device (a major issue when dealing with large files like images). So make sure to configure your caches and allocate enough memory and disk capacity to store all the content you know will be reused frequently (e.g., a user profile image or your app’s logo). That way, your app will be able to process more requests in less time, since the data will already be readily available. Just make sure to never hard code your cache logic, as this can sometimes cause synching issues.

Measure, Tune, Repeat

Once you’ve made the caching suggestions above, you’re not finished yet. The next step of course is to confirm that the changes improve your app’s performance by tracking and measuring as discussed in chapters 1 and 2. Specifically, you’ll want to measure, adapt, and tune your cache editors accordingly for optimal performance, especially as load changes occur.

For more details on getting the most from your mobile APIs, check out this slide deck.

Conclusion

The average smartphone owner uses 26.8 apps each month.4 If you want your app to be one of those two dozen or so applications that loyal users turn to regularly, then you need to make sure your app always delivers an excellent user experience. Measuring usage and performance lets you see your app through your customers’ eyes and gives you the insight to continuously improve the experience.

As we’ve discussed in this eBook, measuring app performance and gaining insight into the user experience takes the right analytics and monitoring toolset. Mobile application performance monitoring tools such as New Relic Mobile let you easily understand performance in real-time and quickly diagnose problems to help you grow your customer base and user loyalty.

4 “Smartphones: So Many Apps, So Much Time,” Nielsen, July 1, 2014.

Back to top icon