Why New Relic
Real User Monitoring and single unified interface
- New Relic delivered as single integrated solution – alerts, error monitoring, transaction traces, statistics, etc.
- Run New Relic in production and QA environments
- New Relic specificity helps discovery of root causes for issues
CentOS, a distribution of Linux, Apache web server, PHP, Magento, HAProxy, Nginx for load balancing, Redis for caching, memcached, MariaDB, Stackable
“With New Relic we get everything we need from a single service – alerts, error monitoring, transaction traces, application statistics, etc., and, best of all, we got immediate results and insights that we just had not seen before.”
Starting in 2005, Skullcandy has experienced consistent rapid growth. When the company launched its eCommerce site in 2008, it took off immediately. Built on the Magento eCommerce platform with PHP and Linux, the online channel’s sales growth has challenged the interactive team to keep up with the accelerating demands. The team’s goal is to ensure that their online customers have an easy, quick purchase transaction every time they come to the site to browse or buy any of Skullcandy’s products. With the “Every revolution needs a soundtrack” mantra as a backdrop, buyers expect the site to be fun and the buying process fast and friction-free.
When Andrew Spencer joined Skullcandy’s interactive team, the site was experiencing growing pains with a number of performance issues. Unplanned outages were on the rise, usually occurring more than once a week and often for significant duration. Since outages equate to lost revenue, and potentially, even to lost customers, his first task was to identify the cause and fix it. With a limited tool set to work with, Andrew jumped right in.
Using the Unix programs top and ps to identify and analyze potential issues, Andrew determined there was a memory leak and wrote a script, ‘The Plummer’, to find and correct the leaks going forward. Although this was a major step in the right direction, the team was still learning about performance issues and problems from their customers. Without any proactive real time performance monitoring and management tools, they simply were not able to see what was going on real time. Andrew started looking around for a better performance monitoring and management solution. A recommendation from a colleague who had used New Relic with Magento led Andrew to try the SaaS offering.
The Skullcandy team was delighted with the ease of New Relic’s implementation. And even more delighted with how fast they had actionable data and information. Andrew attributes much of that success to the fact that New Relic is a SaaS-based solution.
“New Relic is really nice,” comments Andrew. “It only takes about five minutes to install the agent. Once we added it to our provisioning, all our servers get New Relic automatically with no additional configuration required. Compared to that, the enterprise products I’ve used in the past took at least 30 days to configure your implementation and data before you would have anything that was usable. With New Relic we get everything we need from a single service – alerts, error monitoring, transaction traces, application statistics, etc., and, best of all, we got immediate results and insights that we just had not seen before.”
“Prior to New Relic, we weren’t set up to track things very well,” he continues. “We often discovered we had a problem from our customers. It took a long time to track problems down. Today New Relic does the tracking and monitoring for us. Now we know when we have an issue and can usually fix it before the customer even knows. It’s considerably easier to keep both our internal partners and our customers happy.”
Skullcandy runs an agile development cycle and finds New Relic’s historical reports are critical to their success as they now have visibility into how changes impact the application using the comparative data that allows them to see the application before and after.
The company runs New Relic in both their production environment and their QA development environment. They’ve had some very big wins with New Relic and Andrew is thrilled with the progress they made in that 18 months.
“New Relic provides a level of specificity that helps us root out deeper problems.”
A few great stories highlight the results Skullcandy has achieved using New Relic.
Andrew and his team planned to release a new eCommerce site, Skullcandy 2.0, in 2011. The new site was a major overhaul with just about everything being new - look and feel, features, more robust architecture. Skullcandy 2.0 was ready later in the year than they had had originally planned so they were faced with a decision whether to launch so they could take advantage of the holiday sales cycle or delay until Q1. From a sales perspective, they knew the new site was significantly better and key to maximizing the holiday season opportunity.
To get ready for the launch of the new site, the team had server monitoring in place and were comfortable that they knew what was going on in that dimension. But the application level was still a black box to the team. Their load tests were configured to handle up to 10,000 concurrent users. Once they started using New Relic to monitor the application, their confidence level went up immediately. “Once we started using New Relic, we had immediate insight into the application, how it was performing, where issues were,” says Andrew. “We had the confidence we needed to move forward even knowing that late in the year is not the best time to launch a new online site.”
“The second we launched the site, we saw a significant performance problem that was not revealed by prior load tests,” he continues. “Within minutes, New Relic pinpointed the problem in our Best Sellers module, highlighting both slow queries at the database level as well as specific PHP bottlenecks. Using Transaction Traces and SQL statement errors we could immediately see precisely what was happening. It took less than 30 minutes for us to see and fully diagnose the problem, and to pinpoint the issues. We had fixes in place 15 minutes later. We’d gone through the whole process in less than an hour and, after that, the site just hummed. Had we not been able to discover, identify, fix, release, and monitor those issues that quickly, we would have had to roll back and wait until Q1 of the next year to implement the new site. Thanks to New Relic, Skullcandy 2.0 went live on time - our internal customers were pleased and so were we.”
Andrew also appreciates New Relic’s ability to get to the root cause of issues and that was another significant win in the process of launching the new eCommerce site. “Without having New Relic, we would only have found the database problem, and not all of the PHP problems,” comments Andrew. “New Relic provides a level of specificity that helps us root out deeper problems. In this situation it would have been easy to overreact and push for rolling back to the old site. Instead, we had the data and information we needed to operate very, very quickly, and ensured a successful launch of the new site. This raised the confidence and credibility of the team.”
Andrew notes that the Internet is a very complex system and bottlenecks can occur at various levels of the stack. As he points out, “Without the insight into our application that New Relic provides we would have to spend a lot of time eliminating the application as the cause. New Relic allows us to quickly focus on the network, systems, or other aspects that can cause performance problems once we know that the application is not the cause.”
Skullcandy’s use of the Application Topology map (App Map) feature is also interesting. The team wrote their own internal services to replace parts of Magento’s system as well as a Magento extension. The first was the coupon system which they rewrote as an external service. When the systems makes an HTTP call to the external service, New Relic monitors that performance via the App Map feature. Similarly, there was one Magento extension that was proving to be very challenging. After rewriting the extension as an external service that New Relic monitors, the eCommerce site performance improved by 25%.
Skullcandy has many load spikes – sometimes unplanned. Most of these come from coupon weeks that can be compounded when a coupon is leaked by a site like Slickdeals.net. In one such instance, a leak unexpectedly drove $20K of business to the site in just over two hours. New Relic alerted the team to the fact that site that not only was slightly degraded, but that the traffic was coming from SlickDeals.net, making it easy for the team to spin up more application servers to handle the spike in traffic.
The company has achieved some notable results using New Relic – both quantitative and qualitative. In addition to the 25% site performance improvement achieved through the coupon rewrite, New Relic has helped the interactive team to improve MTTR by 300%.
They are a small team and everyone holds multiple responsibilities. They have a single DevOps person that the team shared across skullcandy.com, astrogaming.com (acquired in 2011) and 2xl.com, their value headphone company. And as Andrew comments, “We’ve been able to make our staff’s lives easier. Being able to be more proactive in trouble shooting issues has resulted in fewer emergencies and fire drills, making us less frazzled. It contributes to a better team environment and state of mind.”
As Andrew concludes, “We have been so delighted with the New Relic service and support team and simply find the company easy to work with. I look forward to using the tool to help shape our technology going forward and know that our future development plans will be easier to achieve and more successful using New Relic.”