AI Monitoring Hero

While AI is propelling modern applications to new heights, it also brings forth unique challenges to engineers that build and run AI-powered applications. Unlike traditional applications, AI applications require a new technology stack that incorporates advanced components like large language models (LLMs) and vector data stores. Additionally, they generate additional telemetry data such as quality and cost that need to be considered to ensure that AI applications are safe, secure, and reliable. Addressing the complexities and optimizing these novel applications are essential for the future of AI, especially in light of the Biden administration's recent executive order to establish standards that ensure the safe, reliable, and ethical development and deployment of AI systems.

Today we’re announcing New Relic AI monitoring (AIM)–the industry’s first APM solution that provides end-to-end visibility for any AI-powered application. Now available for early access, AIM provides engineers unprecedented visibility and insights across the entire AI stack so they can build and run safe, secure, and responsible AI applications with confidence.

Before diving into the more technical aspects of AIM, let's take a look at why AI monitoring is important, and what components of the stack need to be monitored to ensure your AI applications are working properly

    Why do AI applications need to be monitored?

    There are several reasons AI applications need to be monitored:

    • Quality and accuracy: Monitor for bias, toxicity, and hallucinations in complex AI models to ensure fair and reliable outcomes.
    • Performance tuning: Identify and resolve computational bottlenecks to maintain responsive and efficient AI applications.
    • Cost management: Track token processing to manage AI model costs effectively and stay within budgetary limits.
    • Responsible use: Ensure AI responses are free from bias and toxicity that can cause harm.
    • Security: Monitor AI applications for vulnerabilities, taking corrective action to mitigate potential security attacks.

    New Relic AIM brings the power of observability to engineers working on AI by providing the necessary insights to debug, monitor, and improve AI applications, ensuring that they operate as intended, deliver accurate results, and meet emerging standards for  responsible use.

    Decoding the AI stack

    AI stacks are complex sets of tools and technologies used to develop and deploy AI applications. As mentioned earlier, AI stacks not only bring a new set of telemetry data, they often require more data, more computing resources, and more specialized tools and technologies than traditional tech stacks. 

    Key components of an AI tech stack include:

    • Infrastructure layer: Provides the foundation for AI development and deployment, including powerful GPUs and CPUs to train and deploy AI models along with cloud computing platforms such as AWS, Azure, and Google Cloud Platform (GCP) that provide a scalable way to deploy AI applications.
    • Data storage/vector datastores: AI applications need to store and access large amounts of data. Vector databases are specialized databases that are designed to store and query high-dimensional data that's often used in AI applications. 
    • Model layer: Contains the AI models that are used to make predictions or generate outputs. Some of the popular AI models for content generation include GPT-4, Anthropic, Cohere, LLama 2, and Amazon Bedrock.
    • Orchestration framework: Orchestration frameworks like LangChain provide a way to chain together different components of an AI application, such as data processing, model invocation, and post-processing. 
    • Application layer: Contains the user-facing applications that interact with AI models.

    New Relic AIM: APM for AI 

    New Relic AIM brings the power of observability to the entire stack. Similar to how engineers monitor their application stack with New Relic APM, AIM provides engineers with full visibility into all components of the AI stack so you can easily monitor, debug, and improve your AI applications for performance, quality, cost, and ensure compliance. 

    Quick and easy setup

    New Relic agents provide quick and easy setup for AI monitoring, with no additional instrumentation required. They provide built-in support for popular models such as OpenAI and AWS Bedrock, as well as orchestration frameworks like LangChain. This gives you complete end-to-end visibility and deep trace insights across your AI stack, enabling you to easily identify and analyze the performance of individual components, trace the flow of data, and pinpoint potential bottlenecks in your AI applications.

    Debug faster with complete visibility of your entire AI stack

    New Relic AIM integrates seamlessly with New Relic APM 360 to provide end-to-end visibility across your entire AI stack, from the service layer to infrastructure to the AI models. You can now correlate your AI application performance with upstream and downstream trends to understand how issues impact other parts of your application in real time. This eliminates guesswork and makes troubleshooting intuitive and efficient for all engineers. 

    Below is a screenshot of the New Relic APM 360 summary with the integrated AIM view. This unified view gives you instant insights into the AI layer’s key metrics, such as the total number of requests, average response time, token usage, user feedback, and response error rates alongside your APM golden signals, infrastructure insights, and logs. Now imagine you see a spike in application errors and also in the AI response errors integrated in the New Relic APM 360 summary view. You can quickly isolate the issue to the AI layer and drill down into the AI responses view to the root cause of the problem.

    Optimize AI application performance, quality, and cost with deep insights 

    New Relic AIM provides deep traces for every response that gives you the visibility you need to understand how your AI applications are working and make informed decisions about how to fix performance, address quality issues such as bias and toxicity hallucination, and manage costs. With the New Relic AIM Response UI you can:

    • Identify outliers and trends: AIM provides a consolidated, roll-up view of all AI responses. This makes it easy to identify outliers and trends in your responses.
    • Trace the entire lifecycle of every response: New Relic AIM allows you to see the entire end-to-end lifecycle of the response. Starting from the prompt through all the stages in the application components with an easily understandable waterfall view as seen in the screenshot below.

    Further, you can view the details, including the prompt, negative feedback, and metadata for each response so you can quickly spot and fix issues related to performance or quality.

    Compare performance and cost across models

    Model comparison is a key part of AI monitoring. It allows you to identify the best model for your needs, track performance over time, and optimize costs. New Relic AIM provides a single, easy-to-use view for troubleshooting, comparing, and optimizing different LLM prompts and responses for performance, cost, and quality issues, such as hallucination, bias, and toxicity across all models. 

    Optimizing AI application cost is one common use case for AIM's model comparison. By tracking the token usage across AI models, you can identify which models are the most expensive to run. You can then choose less expensive models to optimize your AI application architecture.

    Instantly monitor any AI ecosystem with the largest set of integrations

    New Relic AIM provides more than 50+ integrations for the AI ecosystem. This includes popular LLMs, machine learning (ML) libraries, vector databases, as well as frameworks that are not currently supported by New Relic agents. These integrations include pre-configured dashboards, alerts, and other observability building blocks that give you instant visibility into your AI application’s performance and health.

    New Relic: Leading the way with AI monitoring

    New Relic is leading the way in observability with the introduction of AIM. AIM gives you unprecedented visibility, seamless integration, and deep insights into the entire AI stack. With its integration with New Relic APM 360, you can easily identify performance, cost, and quality issues affecting AI applications. By taking a step into AI observability, AIM empowers organizations to confidently adopt AI in their applications, build trust with customers and partners, and get ahead of regulators in the ever-changing landscape of artificial intelligence.