Gain model observability at scale with New Relic One and Superwise

In order to create effective machine learning models, it’s necessary to scrape vast amounts of data, but truly understanding your incoming data in terms of annotation and label distribution for every segment can be very challenging. It can also be difficult to maintain multiple dashboards of ML models in production. In order to update your models effectively, you need a system that can monitor and alert you when your models aren’t working as expected.

New Relic is partnering with Superwise to help data scientists, data engineers, and machine learning practitioners automate model monitoring at high scale and gain insights at the most granular levels so they can pinpoint issues and drive actions to maximize business outcomes.

In this tutorial, you’ll learn how to set up and integrate Superwise with New Relic Alerts and Applied Intelligence to correlate model and infrastructure alerts, and dive deeper to investigate model performance. 

The following video shows the functionality you’ll have once you integrate Superwise with New Relic One.

Why integrate Superwise with New Relic One?

Superwise is a model observability platform for leveraging machine learning at high scale. You can use the platform's automated KPIs, incidents, and model customization builder to deliver critical insights out-of-the-box. By integrating Superwise with New Relic One, you provide your data teams with full visibility and analysis of your models.

Integrating Superwise with New Relic One

To set up the integration you need to do the following:

1. Log in to one.newrelic.com, go to Explorer and select + Add more data.

2. Scroll down to the MLOps Integration section and select Superwise.

3. Select your New Relic account ID.
4. You need to create two access tokens. First, select Create a Key under Model metric data. Then select Create a Key under Incidents. Make sure the keys are available for future steps.

5. Log in to the Superwise portal and go to notification channel settings. Select New Relic and create a channel name.
6. Using the two tokens you created in step 4, paste the API key under Incident Intelligence API and the Insight key under Telemetry Key in the Superwise portal. 
7. Test both tokens for verification and select Save.
8. In the New Relic integration dashboard, select See your data. This will redirect you to an automatically generated New Relic dashboard powered by Superwise.

9. Superwise’s dashboard contains three charts: the Model Activity chart, the Model Input Drift detection chart, and the Incident Insights chart.

10. Set up New Relic policies and conditions based on your ML Model metrics by creating an alert condition. To do so, select the three dots on any metric chart in your dashboard and choose Create alert condition from the dropdown.

11. Next, set up your notification channels to receive alerts.

12.  You can also correlate your incidents to reduce noise and create custom decisions by training your ML model.

New Relic’s partnership with Superwise will provide your data and engineering team full visibility into ML-powered applications and the future of software by allowing MLOps and DevOps teams to seamlessly collaborate.