Practical Use Case of New Relic Synthetics to quickly detect a disaster with CDN

This is an English version translated from this original version in Japanese.

Get ready for future disaster with CDN

Nobody doubts that CDN is a key component to build a redundant and stable service on the internet. In this circumstance any temporal down of CDN results in customers not being able to  access your service, regardless how hard you work on your backend system.

Through my career at CDN provider, I have learnt and experienced that we correctly understand that CDN is not perfectly redundant and we have to get ready for any cases of disasters with CDN.

 

 In the case of CDN failure, you can have some remedies to mitigate such situation. For instance…

• (Back Up CDN) To turn the CDN with problem off and to tentatively turn an alternative CDN on  
• (Without CDN) To run the CDN off and to directly deliver your service contents to your customer (To serve directly through your origin)

 

When you face this situation, you have to immediately make a decision and to start the workaround operation. Therefore, New Relic ONe can constantly monitor CDN’s situation for your decision in such emergency situation, and provide the best-fit & specific information for you. This is one of our accountabilities as observability platform provider.

 

 

To detect the service down with New Relic One

We strongly recommend you to adopt Synthetics as one of the advantages on our platform in order to properly collect the condition of your service. If you already have New Relic One license and are granted with Full User, you are eligible to start Synthetics right now.

 

Synthetics supports various use cases and advanced settings for your service. When it comes to detecting the disaster with CDN, the following simple setting can sufficiently meet your purpose.

 

How to set up Synthetics for your service

After logging in New Relic One portal, click the Add more data button on the top right corner.

You see several tiles in the menu. Then, click the New Relic Synthetics tile at the Simulate traffic section in UI.

Then, you see several choices for Synthetics settings.

Select Availability in the choices.

Then, you see the setting UI to configure HTTP HEAD request via Synthetics.

• Account: Select a right account in the pull down menu if you have multiple accounts.
• Name: Enter your preferable string for your management
• URL: Enter a URL to send an HTTP Head request. For example, Top page in your service site or dummy/test object for polling purpose.Note: This URL is required that the request is sent through CDN to your origin. In other words, the FQDN in URL is a CNAME record to go through CDN.
• Period: Select your preferable polling frequency. From 1 minutes at minimum to 1 day at maximum

 

 

Skipping Tags and Advanced options as not needed here.

 

After this setting, you see that the Select locations button get active in the bottom of UI. Then, click the Select locations button, go to the next step.

Next, you see the UI where you select the location from which Synthetic monitors. If your service are running for Japan market, check [Tokyo, JP] for instance. The best practice is to select the location where the majority of your customers are based on.

After checking your preferable locations, the Save monitor button gets active in the bottom of UI. By clicking the button, the setting is done.

Through the setting steps above, you can run the periodical polling of HTTP requests through Synthetics. After waiting for the completion of a couple of attempts, you see the service availability and so on in New Relic One portal.

 

Note: Alert and notification are not configured yet. This article doesn’t cover the valuable settings.

 

What you gain by gaining the observability and visualizing the unexpected situation with CDN

If there is some unexpected issue with CDN, we see some failures of Synthetics monitoring and/or the sudden increase of 50X HTTP responses that the CDN returns. We disclose such symptoms through our observability. Let me introduce a couple of the facts.

 

We see the degraded Success rate, not 100%, recorded by Synthetics. For the purpose of CDN adoption, such degradation is not allowed. Something unexpected must happen. Therefore, in the case that there is a degradation of Success rate with your service with CDN, you should verify if any service impact is occurred by carefully checking your system.

 

Also, you must see the sudden increase of 50X HTTP responses. Only with this information, we cannot understand the 50X HTTP response is sent either from CDN or from your system. However, if you already adopt New Relic’s solution such as Infrastructure agent and APM agent, as well, you can easily verify by utilizing both of data through Synthetics and the agents.

< the amount of 50X HTTP error responses in Failure report >

50X HTTP response is very sensitive results and you must take care of them as daily operations. Therefore, it’s not hard for you to observe such increase of 50X HTTP responses suddenly by comparing the regular and daily trend.
Note: It’s very crucial to retain the daily activities to minimize the number of errors indicated into Failure report.

< degradation of uptime% in SLA report >

In the case of CDN’s disaster, you easily see the uptime get down to less than 100%. For instance, you see that the uptime numbers were 100% until yesterday but less than 100% today. It means that CDN cannot send responses back for Synthetics’ requests.

If you have already adopt CDN for your active service, we strongly recommend you to adopt Synthetics for your service in order to comprehensively monitor and visualize the condition of CDN as well as your backend observability. Also, New Relic One can collect log lines/data generated by CDN and visualize them for much better observability. Therefore, if you are interested in the CDN integration, please try it.

Finally, here is a sample visualization (a.k.a. dashboard) with the collected log data from CDN. you see a chasm in the graph and in the period, there was a large/global-grade service down with CDN.

We continue the consideration and will post what to gain through CDN integration.

Thanks