How synthetic monitoring supports nonprofits with Angel Flight West

As with many charitable and non-profit organizations, volunteers are the lifeblood of our operations. They contribute their time, expertise, and skills. At Angel Flight West, we need a very specific volunteer: pilots, who typically have their own airplanes, to donate both their time and the cost of the flight. These pilots fly patients seeking life-saving or critical treatments that are not available locally, and often requiring multiple trips to complete treatment. Our volunteer pilots fly somewhere between 300 and 600 miles, dedicating a day of travel to a person in need. We arrange about 5,000 flights per year, and have hundreds of flights to coordinate at any given time. To make finding opportunities easier, we built an app that matches opportunities to volunteers. As we continue to grow, we need systems that can be managed by us—we’re a small tech team of two—that keep costs under control. 

Reliability, redundancy, uptime, and volunteer experience are critical to our reputation and our ability to serve our passengers and healthcare partners. Using New Relic to monitor our apps and to observe the user experience helps us maintain this reputation. We use synthetic monitoring to maintain system reliability as we scale in flights and volunteers.

Insights into problems before they happen

Synthetic monitoring allows us to set up scenarios to see application experience from our volunteers' point of view. Synthetics also help us check reliability. We run regular tests on our most important features to see how they're performing. This background vigilance helped us improve both our app and infrastructure systems before our volunteers see any sign of degradation or poor performance. It has also helped us benchmark our application. As a result, we were able to go to the board of directors and show where the application was not performing as it should be and make the case for more resources to beef up our infrastructure. New Relic made it easy to present that data and show that this was a priority area.

As we define product specifications for new features, we determine what our volunteers need and how to implement it. We also use synthetic monitoring when we release a new feature. We can measure the impact of a new feature on the performance of our application to refine the product even further, removing that risk and uncertainty from deployments.

Features vs performance: Monitoring impact on app users

We have a couple of crucial interaction points for both the volunteers and the passengers, so it's really important that those work reliably. For example, when the pilot goes to view volunteer opportunities, we want to make sure that not only is the list of opportunities up and running, but that it's responsive and creates a personalized user experience, particularly in that they can use filters to narrow the list down to opportunities based on their preferences. But we're always concerned that when we do that, it's going to slow the page down because it adds additional complexity to the queries. Synthetics are a really good way for us to get a barometer on that.

One of the filters we recently added is to allow volunteers to select flights with passengers that they have flown before. Many times, our volunteers establish relationships with passengers, and they want to keep flying and supporting that passenger. But that's now an additional part of the query logic that has to search another table, which can potentially slow that query down. With synthetics, we set up a monitor to measure the exact amount of time that each of those queries takes and we set repeated testing of that at random times in order to see how it is really affecting user experience on average. We compare this with our current data on response times so that we can see if this new filter feature slows down performance against our baseline.

This alone, however, is not enough. When we add these new feature filters, we also need to make sure that performance isn’t degrading over time. Our synthetic monitoring might show that there is a negligible impact on response times when we add the new filter, but is our baseline changing over time? Is the response time getting worse or getting better? It could be getting worse based on other features we've been adding or other things that are going on, like increased volume of usage with more volunteers and passengers using our app. So that's also something that we can pay attention to with synthetics.

Our tips on how nonprofits can start synthetic monitoring

1. Identify the things that you consider to be critical to your process. If it's a web form or a directory that is key to delivering a service, identify those. 
2. Once you know which critical parts of your application you want to monitor, there are a couple of synthetics that you can run out of the box. You can monitor page loading, for example, to make sure that an essential page loads quickly. 
3. Build more complex synthetics. For example, a synthetics test can fill out the whole form, check for error conditions, and check that it is submitted properly. 
4. Treat synthetics as a crawl, walk, run approach. The first thing is to make sure the page is running (crawl). Then, maybe you're testing the load time or you're testing how many times the form was submitted (walk). Then, you can test the process of filling out the form to ensure that all the fields are being captured correctly (run).

Synthetics and other observability tools with New Relic add to our team capacity. As a small non-profit, our focus is on expenditures that keep the planes in the air and the passengers arriving at their appointments. New Relic’s tooling has become like having another employee: we have ‘someone’ working on monitoring 24/7, allowing us to focus on our app building, new features, and product maintenance, For a small non-profit team, this is invaluable: There would just be no other way apart from waiting till something breaks and then fixing it.