Streamline Message Flow Observability with Queues and Streams Monitoring

Part 1: Introduction

Event-driven architecture enables applications to communicate with each other by producing and reacting to events. For instance, ride-sharing apps use event streams to handle ride requests, with events triggering services like mapping, driver assignment, and payment processing. However, as modern applications increasingly rely on message queues and streaming data, monitoring these complex systems can be challenging. Growing event volumes makes debugging issues like bottlenecks, throttling, and data latency harder, with issues often going unnoticed until they impact users.

Today, we’re excited to announce New Relic’s Queues and Streams Monitoring, the industry's first fully integrated monitoring solution delivering bi-directional insights across your entire event-driven architecture. This comprehensive capability uniquely offers real-time, granular insights into Kafka clusters,and  proactive monitoring with out-of-the-box alert conditions based on trending data, overcoming the typical 2-minute delay seen in other Kafka monitoring solutions..

Part 2: Supporting Details

Unified Cluster View: Quickly Assess Your Kafka Health

New Relic Queues and Streams provides context-rich and end-to-end visibility for message queues and streams. Its distinctive bi-directional drill-down capability connects topics to both producer and consumer services, enabling DevOps teams to quickly identify and resolve issues such as slow producers, overloaded topics, or struggling consumers. Key capabilities includes:

• Granular insights into Kafka health, down to the cluster, partition, broker, topic, producer, and consumer level.
• Fast root cause and performance analysis with bi-directional capabilities to drill down from service to topic and back.
• Proactively identify potential issues with out-of-the-box alert conditions and real-time anomaly detection.

Accelerate troubleshooting with bi-directional drill-downs 

When issues arise in message-driven applications, pinpointing root causes is often the biggest challenge. New Relic’s bi-directional drill-down capability provides seamless, real-time, correlated insights from Kafka clusters to producers and consumers(APM services)—and back again. This contextual visibility enables engineers to quickly determine if the issue originates from overactive producers, sluggish consumers, or broker resource constraints, to significantly reduce Mean Time To Detection (MTTD) and Mean Time To Resolution (MTTR), enhance system resilience, and ensure seamless data flow across distributed architectures.

Instant Anomaly Detection and Proactive Alerting

Monitoring Kafka streams often involves delays that impede immediate action. New Relic mitigates this with real-time Kafka metrics via our Java agent, empowering your teams to spot anomalies instantly across all Kafka client components—producers and consumers alike. Additionally, proactive alerting (coming soon) will further enhance your monitoring experience by mitigating Kafka’s inherent monitoring delay and promptly notifying your team of potential issues before they escalate.

Identify and Optimize Data Throughput

Efficient data flow is crucial for performance. With New Relic’s detailed insights into metrics such as latency, throttling rates, queue depth, and retry counts, your team can quickly pinpoint bottlenecks that impact throughput. Furthermore, our solution identifies uncompressed data streams, enabling you to optimize your system with effective compression strategies and improve overall performance.

Comprehensive Kafka Topic Insights

Understanding individual topic performance is essential. New Relic’s detailed topic view enables users to quickly access the top 20 topic entities in their clusters, complete with real-time metrics such as incoming and outgoing throughput and message rates. This granular data ensures teams can manage and optimize topic-specific performance and resource allocation effectively.

Part 3: Getting Started

Integrating Queues and Streams monitoring into your workflow is simple and straightforward:

Step 1: Set Up Integration

• Choose your Kafka provider (Amazon MSK or Confluent Cloud) and follow the guided installation steps.

Step 2: Navigate the New Relic UI

• Log in to New Relic, navigate to one.newrelic.com > All capabilities > Queues & Streams.
• Use intuitive filters and search functionality to quickly identify Kafka clusters.

Step 3: Monitor and Diagnose

• Utilize the visual Kafka Navigator to quickly identify healthy or problematic clusters.
• Drill down for deep metrics and correlation with your APM services.

For more detailed setup instructions, view our New Relic Queues and Streams documentation.