Monitoring model performance with New Relic and Algorithmia

Imagine you just deployed a new machine learning (ML) model into production that increases the accuracy of your algorithm. How do you track and detect model drift, data drift, and model bias? And how do you relay that information efficiently to teams developing and operating these models?

If you’re a data scientist or a DevOps engineer tasked with deploying ML models, you’ve probably faced this challenge more times than you would have liked. Monitoring your machine learning models after production is vital to understanding how effective your algorithm is and making sure your business is doing better than it was before production. After all, it would be unfortunate to believe your model is doing wonders for your business, only to find out the damage it might be doing because of a small error or inaccurate report.

In this tutorial, you’ll learn how you can monitor your model performance metrics with New Relic using the Insights feature of Algorithmia Enterprise so that you can track the success of your ML models in production and easily adapt to changes your team needs.

Algorithmia is an enterprise machine learning operations (MLOps) platform that manages all stages of the production ML life cycle within existing operational processes, so you can put models into production quickly, securely, and cost-effectively.

With New Relic’s new partnership with Algorithmia, you’ll now be able to streamline models into our observability platform, have a central location to create dashboards, and use alerts to analyze model performance metrics in production systems across your entire model catalog. This allows for seamless collaboration between your data science and DevOps teams, ultimately leading to innovative solutions through the creation of sophisticated ML models, continued development and testing, and consistent operation and monitoring of the software.

New Relic Alerts and Applied Intelligence form a flexible and centralized notification system that unlocks the operational potential of New Relic.

With New Relic Alerts, you can manage alert policies and conditions, letting you focus on the metrics you care about most. Applied Intelligence helps you detect, understand, and resolve problems faster. Specifically, it’s a hybrid machine learning engine that automatically detects anomalies, reduces alert noise by correlating related alerts and incidents, and provides automatic root cause analysis.

Algorithmia Insights is a feature of Algorithmia Enterprise that provides a metrics pipeline that can be used to instrument, measure, and monitor your machine learning models.

7. Get notified through a specific channel.

By sending your model performance metrics from Algorithmia Insights to New Relic and having real-time monitoring for your algorithms, you can explore your metrics data with user-friendly charts, and quickly learn the state of your algorithms for faster and more efficient troubleshooting. The following flow chart is how New Relic integrates with Algorithmia from the backend.

To set up this integration, you will need to:

1. Retrieve your API key for the Algorithmia integration.

2. Create an MLOps focused dashboard in New Relic Dashboards. An example of this dashboard can be found below.

3. Configure Algorithmia to send Insights to a Kafka broker and topic.

4. Create your New Relic connector algorithm on Algorithmia, and configure it with your New Relic API key. The source code for the algorithm can be found on our docs page here.

5. Set up Algorithmia Event Flows for your New Relic connector algorithm.

6. Set up New Relic policies and conditions based on your ML Model metrics by creating an alert condition. You can do this by clicking the three dots on any metric chart in your dashboard and selecting “create alert condition.”

8. Correlate your incidents to reduce noise and create a custom decision by training your ML model.

Once you complete your integration, you can say goodbye to lost ML models and hello to increased ML performance. New Relic’s partnership with Algorithmia will provide your team full observability into ML-powered applications and the future of software.