A Complete Introduction to Monitoring Kubernetes with New Relic

When working in a Kubernetes environment, it can be difficult to untangle the dependencies between applications and infrastructure, and navigate all of the entities—containers, pods, nodes, deployments, namespaces, and so on—that may be involved in a troubleshooting effort. You need to observe performance and dependencies across the entire Kubernetes environment.

You should be able to visualize key parts of your services, including:

• The structure of your application and its dependencies 

• The interactions between various microservices, even those that are intermingled across your machine cluster

How New Relic helps

The cluster explorer provides a multi-dimensional representation of a Kubernetes cluster that allows teams to drill down into Kubernetes data and metadata in a high-fidelity, curated UI that simplifies complex environments. Teams can use cluster explorer to more quickly troubleshoot failures, bottlenecks, and other abnormal behavior across their Kubernetes environments.

Suggested alerting

When deploying the New Relic Kubernetes integration for the first time in an account, a default set of alert conditions is deployed to the account. The alert policy is configured without a notification channel to avoid unwanted alerts.

You can customize the alert conditions' thresholds to your environment and update the alert policy to send notifications. For more, see the New Relic Infrastructure alerts documentation.

Kubernetes environments vary from deployment to deployment, but they have all a handful of key components, resources, and potential errors in common. The following sections introduce best practices, including tips for how to use New Relic and alerts, for monitoring the health and capacity of any Kubernetes environment: 

• Track cluster resource usage
• Monitor node resource consumption
• Monitor for missing pods
• Find pods that aren’t running
• Troubleshoot container restarts
• Track container resource usage
• Monitor storage volumes
• Monitor the control plane: etcd, the API server, the scheduler, and the controller manager

How New Relic helps

New Relic makes it easier to avoid this issue by knowing the resource limitations of the cluster. 

If you don’t have enough resources to schedule a pod, add more container instances to the cluster or exchange a container instance for one with the appropriate amount of resources. In general, you can use the New Relic Kubernetes integration to monitor for missing pods and immediately identify deployments that require attention. This often creates an opportunity to resolve resource or configuration issues before they affect application availability or performance.

Kubernetes dynamically schedules pods into the cluster, but if you have resource issues or configuration errors, scheduling will likely fail. If a pod isn’t running or even scheduled, that means there’s an issue with either the pod or the cluster, or with your entire Kubernetes deployment.

When you see that pods aren’t running, you’ll want to know:

•  If there are any pods in a restart loop
• How often are requests failing
• If there are resource issues or configuration errors

How New Relic helps

As noted, if you have resource issues or configuration errors, Kubernetes may not be able to schedule the pods. In such cases, you want to check the health of your deployments, and identify configuration errors or resource issues.

With the New Relic Infrastructure Kubernetes integration, you can use default deployment data to discover and track pods that may not be running and sort them by cluster and namespace.

Additionally, you can analyze further root causes of terminated pods, with the terminated pods metric. For example, if a pod is terminated because its application memory has reached the memory limit set on the containers, it will be killed by the out of memory (OOM) killer. In such cases, New Relic will expose the reason for pod termination.

Suggested alerting

Set alerts on the status of your pods. Alerts should trigger when a pod has a status of “Failed,” ”Pending,” or “Unknown” for the period of time you specify.

In normal conditions, containers should not restart. Container restarts are a sign that you’re likely hitting a memory limit in your containers. Restarts can also indicate an issue with either the container itself or its host. Additionally, because of the way Kubernetes schedules containers, it can be difficult to troubleshoot container resource issues since Kubernetes will restart—or kill—containers when they hit their limits.

Monitoring container restarts helps you understand:

• If any containers are in a restart loop
• How many container restarts occurred in X amount of time
• Why containers are restarting

How New Relic helps

A running count of container restarts is part of the default container data New Relic gathers with the Kubernetes integration.

This type of monitoring can help proactively resolve resource usage issues before they affect your application.

Suggested alerting

Set alerts on container CPU and memory usage and on limits for those metrics.

One thing you definitely want to avoid when running applications on Kubernetes is data loss or application crashes because you’ve run out of space on your storage volumes. 

In Kubernetes, storage volumes are allocated to pods and possess the same lifecycle as the pod; in other words, if a container is restarted, the volume is unaffected, but if a pod is terminated, the volume is destroyed with the pod. This works well for stateless applications or batch processing where the data doesn’t outlive a transaction.

Persistent volumes, on the other hand, are used for stateful applications and when data must be preserved beyond the lifespan of a pod. Persistent volumes are well suited for database instances or messaging queues.

To monitor Kubernetes volumes, you’ll want to: 

• Ensure your application has enough disk space so pods don’t run out of space

• View volume usage and adjust either the amount of data generated by the application or the size of the volume according to usage

• Identify persistent volumes and apply a different alert threshold or notification for these volumes, which likely hold important application data 

How New Relic Helps

You'll want to monitor and alert on disk volume issues, especially in the context of persistent volumes where data must be made available to stateful applications persistently, so that it’s not destroyed if a specific pod is rescheduled or recreated—like when a container image is updated to a new version, for example.  

With Kubernetes volume monitoring in New Relic, you can monitor your volumes, and set alerts on them so that you get informed as soon as a volume reaches a certain threshold—a proactive approach to limiting issues with application performance or availability.

Suggested alerting

Set alerts on available bytes, capacity, and node usage in your cluster.

The central RESTful HTTP API handles all requests coming from users, nodes, control plane components, and automation. The API server handles authentication, authorization, validation of all objects, and is responsible for storing said objects in etcd. It’s the only component that talks with etcd.

To monitor the API server, you’ll want to track:

• Rate and number of HTTP requests

• Rate and number of apiserver requests

Suggested alerting

Set alerts to trigger if the rate or number of HTTP requests crosses a desired threshold. 

The scheduler is responsible for assigning newly created pods to worker nodes capable of running said pods. To do so, the scheduler updates pod definitions through the API server.

The scheduler takes several factors into consideration when selecting a worker node, such as requested CPU/memory vs. what’s available on the node. 

To monitor the scheduler, you’ll want to track:

• Rate, number, and latency of HTTP requests

• Scheduling latency

• Scheduling attempts by result

• End-to-end scheduling latency (sum of scheduling) 

Suggested alerting

Set alerts to trigger if the rate or number of HTTP requests crosses a desired threshold. 

To speed troubleshooting and issue resolution, you can correlate the health status of your cluster and other objects with Kubernetes events. If you run complex Kubernetes environments or don't have command-line access to your cluster, Kubernetes events provide the insights you need to understand what’s happening inside your cluster.

For example, let’s say you have a pod that doesn’t get properly scheduled and won’t start because the node it’s assigned to wasn't allocated enough memory. In this case, the node can’t accommodate the pod, so the pod stays in pending status, but no other metrics or metadata provide deeper insight into the issue. With Kubernetes events, you’d get a clear message:

FailedScheduling [...]  0 nodes are available: Insufficient memory

If you’re managing a Kubernetes deployment, or developing on top of one, you need:

• Visibility into the Kubernetes events for a cluster

• Visibility into the Kubernetes events related to a specific object, such as a pod or a node

• Alerting on Kubernetes events

How New Relic helps

As shown in the example, Kubernetes events provide additional contextual information that is not provided by metrics and metadata. When using Kubernetes events alongside the cluster explorer, you get a holistic view of the health of your platform.

When troubleshooting an issue in a pod, Kubernetes events more readily point toward root causes with useful context. New Relic also layers each event with useful details, so you can determine if an event affects several pods or nodes in cluster, such as when a ReplicaSet is scaled or when a StatefulSet creates a new pod.   

You can query Kubernetes events with New Relic chart builder, or view them from the cluster explorer.

Suggested alerting

Set alerts for specific types of events on objects and resources in your cluster. For example, New Relic can send alerts if an expected autoscaling action doesn’t occur. 

Knowing the names of the pod and node where the error occurred can speed your troubleshooting. Visibility into transaction traces will quickly highlights any abnormalities in your Kubernetes-hosted application.

Additionally, distributed tracing allows you to inspect the distributed traces captured for any application running in your cluster. If you click on an individual span in a distributed trace, you can quickly see the relevant Kubernetes attributes for that application; for example, you can find out which pod, cluster, and deployment an individual span belongs to.

New Relic distributed tracing provides automated anomaly detection to identify slow spans and bottlenecks. You should also set alerts on key transactions and communications with third-party APIs.

To learn about how to gain visibility into transaction traces in Kubernetes, see the blog post, Monitoring Application Performance in Kubernetes.

Suggested alerting

You’ll want to set up alerts for all applications running in production. Specifically, you’ll want to alert on API service requests, transactions, service latencies, uptime, and throughput, sending alerts when any of these metrics fall below the thresholds you define.

If a single pod or particular pod IP starts failing or throwing errors, you need to troubleshoot before those errors harm your cluster or application. When something goes wrong, zero in on root causes as quickly as possible.

You’ll want to know:

• In which namespace/host/pod did a transaction fail
• If your app is performing as expected in all pods
• The performance of application X running on pod Y

How New Relic helps

With New Relic, you can get a code-centric view of the applications running inside your cluster, monitor your Kubernetes-hosted applications for performance outliers, and track down any individual errors.

The Monitoring Application Performance in Kubernetes blog post can help you pinpoint applications that might be causing performance issues in your cluster. 

Suggested alerting

Set alerts to track error rates for any applications running in production environments in Kubernetes.

There are any number of ways to use Prometheus data in New Relic, but consider the following use cases:

Monitoring etcd

Etcd is a key-value data store that’s essential for running Kubernetes clusters. Prometheus pulls metrics from etcd, so to ensure your clusters are healthy, you can use the Prometheus OpenMetrics Integration to monitor etcd server, disk, and network metrics such as:

• etcd_server_has_leader
• etcd_server_proposals_failed_total
• etcd_network_peer_sent_bytes_total
• etcd_disk_wal_fsync_duration_seconds

Kubernetes Horizontal Pod Autoscaler (HPA)

HPA automatically scales up a Kubernetes deployment based on user-configured limits. After installing the Prometheus OpenMetrics Integration, you can use the following query in the New Relic One chart builder (or New Relic Insights) to build a dashboard widget and monitor the remaining HPA capacity.

FROM Metric select latest(kube_hpa_status_current_replicas),latest(kube_hpa_spec_max_replicas) where clusterName = '<YOUR CLUSTER NAME>'  facet hpa 

Node readiness

Set alerts to track error rates for any applications running in production environments in Kubernetes.

In Kubernetes, a node is marked ready when it can accept workloads (pods). If a node is having issues, Kubernetes will label it as "not ready." To create an alert condition for this, using the integration, use the following query:

FROM Metric select latest(kube_node_status_condition) where condition='Ready' and status = 'true' and clusterName = '<YOUR CLUSTER NAME>' facet nodeName

If you order an item from a delivery service, and it arrives at your house broken or late, do you really care what part of the delivery process failed? Whether it was the fault of the manufacturer, distributor, or the delivery service, the end result is equally frustrating. 

The same logic applies to companies hosting apps in Kubernetes: If a customer navigates to their website and a page doesn't load or takes too long, the customer isn’t interested in the technical reasons why. That’s why it’s not enough to track your own systems’ performance—it’s also essential to monitor the front-end performance of your applications to understand what your customers are actually experiencing. 

Even though your application is running in Kubernetes, you can still identify and track the key indicators of customer experience, and clarify how the mobile or browser performance of their application is affecting business.

How New Relic Helps

For example, when developers first migrate to Kubernetes, they can set up a pre-migration baseline to compare their front-end application’s average load time before and after the migration. Developers can use the same strategies detailed in the Monitoring Application Performance in Kubernetes blog post to gain insight into key indicators such as response time and errors for mobile application and browser performance. It’s also imperative to monitor load time and availability to ensure customer satisfaction. New Relic Browser and New Relic Mobile are built to give you that crucial view into your users’ experiences.