Why New Relic
Simple, intuitive dashboards that both technical and nontechnical users can understand; no custom instrumentation required
- Increased conversion rates by optimizing marketing funnel
- Enhanced product to better suit user behavior and preferences
- Maintained superior performance and reliability of service
Simplifying analytics for both technical and nontechnical users
To build the very best study tools, Quizlet listens obsessively to its users, reading thousands of feedback emails a week and doing everything it can to improve the user experience. But in order to improve the user experience and keep the site running optimally, Quizlet needed visibility into what exactly was going on in its complex code base, and how potential issues might be impacting what real users were experiencing.
“Performance and reliability of the service is of paramount importance for us,” says Thompson Paine, general manager of Quizlet. “If a teacher is using Quizlet in the classroom, it is basically replacing their workbook. So if our service goes down in the middle of an exercise, then it’s like we just ripped the books out of the hands of all the kids in the classroom.”
In addition to ensuring that students and teachers are getting the uptime they need, Quizlet wanted an easier way to gain business intelligence from its software. “Previously, we had to go through this long cycle of even being able to pose a question,” says Sutherland, founder and CTO of Quizlet. “You’d have to instrument, deploy, and then wait for the data to come in.”
The company’s previous way of running analytics queries was not only time-consuming and tedious, but also proved difficult for its nontechnical users to grasp. “We’ve done what we can to help our nontechnical users learn SQL, but there have always been some configuration issues—they’re never quite figured it out all the way,” says Sutherland. What the Quizlet team needed was a more advanced and intuitive analytics platform thatcould provide actionable business intelligence to anyone in real time.
New Relic Insights provides immediate, out-of-the-box value
According to Sutherland, Quizlet has been a New Relic user for “so long we can’t remember what it was like before.” The company uses the New Relic Software Analytics Platform to find and fix performance bottlenecks with cross-system visibility. So when Sutherland first signed up as a New Relic Insights beta user, he was thrilled to find the missing piece to the puzzle. “When I first saw Insights,” he explains, “I realized, ‘Here’s a simple dashboard that our nontechnical users can look at to get the analytics they’d typically run in SQL, but in a much easier interface that’s also fully auto-complete enabled.”
Indeed, nontechnical users like Paine immediately saw its value. “As someone who doesn’t have a background writing SQL queries, I found Insights and the NRQL query language to be very easy to pick up,” he says. “If I want to set up a dashboard, I can take a NRQL query that’s already been written and easily doctor it to get the exact data I want. There’s no need to bother an engineer for these type of requests anymore.” The fact that New Relic Insights already had all of Quizlet’s transaction data stored made the solution all the more useful. “The most powerful thing about New Relic Insights is that you don’t have to custom instrument anything,” says Sutherland. “The data’s already there, so you can ask questions about it after the fact.”
Now anytime Sutherland is in a meeting and someone asks a question like ‘How many users are creating study sets every day?’, rather than guess or spend time going through a hierarchical menu of dashboards, “I can just pull it up in about 10 seconds without stopping the flow of the meeting,” he says. “We’ve come to rely on New Relic Insights so much that we’ve basically stopped using all of our other analytics tools. We really treat New Relic as our second database, one that stores all of the information about how people use Quizlet.”
Username and user ID are two key custom attributes that the company added to New Relic Insights. This way, anytime a user writes in feedback with an issue, the team can easily open up a list of every single page view and interaction that the user looked at, and better understand why he or she may have encountered the problem they did.
On the technical side, Quizlet uses New Relic Insights to see, in real time, which of its content delivery network’s data centers are performing the slowest, which ones are having network congestion, or which ones have been turned off or back on. “That helps us give much better debugging information when we have an issue with our CDN provider,” says Sutherland, “and it helps us evaluate their performance overall.”
“The most powerful thing about New Relic Insights is that you don’t have to custom instrument anything. The data’s already there, so you can ask questions about it after the fact.”
One platform, multiple businessenhancing use cases
Thanks to the New Relic Platform, Quizlet has been able to use its application data to enhance its business in various ways, from marketing and product enhancements to performance improvements and even revenue generation.
“We’re currently working on a big revenue project where we were trying to optimize the funnel of people converting to our premium service,” explains Sutherland. The Quizlet team set up a New Relic Insights dashboard that tracks the ages of people going to the different parts of the funnel. “We immediately had this surprising realization that a lot of our younger users— which are our core demographic—were going to the first couple of pages, but were not paying, and that the people who were paying for our premium stuff were mostly older, many of them teacher users.”
Armed with this newfound information, Quizlet has changed the way it markets its premium service, now better targeting adult users. “We increased the number of places, pages, and emails that we send to older users, and decreased what’s being sent to the teenage demographic,” says Sutherland. “As a result, we’ve already seenconversion rates increase dramatically.”
Quizlet’s engineering team also uses Insights to adapt its product to user behavior and preferences. “We noticed that something like a quarter of our logins come from Google now,” says David Chanin, a software engineer at Quizlet. “Seeing that, we decided to make the Google login button much more prominent to better serve and simplify the experience for our target audience.”
After breaking down users by operating system, Quizlet also realized that older versions of Windows were the most popular OS among its user base. “You have to realize that most U.S. schools don’t have the luxury of using Macs and Thunderbolts,” says Paine. “New Relic Insights helps us understand what we need to do to optimize our service in classrooms that are using different technologies than we’re using here in our office.” Both of Quizlet’s technical and business users find New Relic Insights an invaluable tool in continuing to achieve the company mission. “Quizlet really wants to be a service that is there for every kind of student, every kind of teacher, and every kind of educational environment,” says Paine. “And New Relic Insights gives us actionable data to better support all of those different users.”
“We’ve come to rely on New Relic Insights so much that we’ve basically stopped using all of our other analytics tools.”