How To Monitor Prometheus With The New Relic OpenMetrics Integration

If you use Prometheus to monitor your microservices and Kubernetes environment, you can now send those metrics directly to New Relic for storage, visualization, and correlation with your other telemetry data using our Prometheus OpenMetrics integration.

This integration collects telemetry data from the many services (such as Traefik, Envoy, and etcd) that expose metrics in a format compatible with Prometheus. In fact, with the integration you’ll be able to monitor key aspects of your Kubernetes environments, such as etcd performance and health metrics, Kubernetes horizontal pod autoscaler (HPA) capacity, and node readiness.

As part of the integration, New Relic stores all metrics from Prometheus, removing the overhead of managing storage and availability of the Prometheus server. And when troubleshooting issues in your Kubernetes clusters, the metrics collected by this integration are accessible alongside those gathered from New Relic APM and and the cluster explorer.

About Prometheus

Prometheus, part of the Cloud Native Computing Foundation (CNCF), is an open-source toolkit that provides monitoring and alerting for services and applications running in containers. It’s widely used to collect metrics data from Kubernetes environments.

Prometheus has done a significant amount of work with the open source community to standardize how Prometheus formats and exposes metrics, especially within Kubernetes clusters. In fact, Prometheus’ scheme for exposing metrics has become the de facto standard for Kubernetes.

How Prometheus works

Prometheus uses a pull-based system to pull multidimensional timeseries metrics from services over HTTP endpoints, instead of relying on services to push metrics out to Prometheus. Because of this pull-based system, third parties like New Relic, can build integrations that work with Prometheus’ metric exporters to gather valuable data for storage and visualization.

While deploying the Prometheus server is easy, managing Prometheus at scale can be a real challenge for organizations as they grow. New Relic takes care of scaling and managing the storage of metrics over time, and provides the tools to visualize and alert on these metrics.

Prometheus uses key-value tagging to organize metric data, which allows users to build strong queries to interrogate their data. For example, this is what a metric looks like when queried over HTTP:

myservice_requests_total{service="catalogue", env="production"} 100

Core components of Prometheus

The Prometheus architecture is created from a handful of components, the following of which are essential:

• Prometheus server: scrapes and stores time series metric data
• Pushgateway: caches metrics from short-lived jobs before sending to Prometheus server
• Exporters: pull metrics from third-party services, such as Redis, etcd, and Grafana
• Client libraries: used to instrument services and applications to be monitored with Prometheus

Prometheus and Kubernetes

Prometheus has seen a significant increase of usage within Kubernetes deployments in part due to its status within the CNCF. Prometheus also supports a large number of open source exporters, and projects like the Prometheus operator make it very easy to deploy within Kubernetes environments. It’s a proven and effective way to get metrics from Kubernetes hosts and processes.

Getting started with the New Relic Prometheus OpenMetrics integration in Kubernetes

New Relic’s Prometheus OpenMetrics integration supports both Docker and Kubernetes, using Prometheus version 2.

Installing the Prometheus OpenMetrics integration within a Kubernetes cluster, for instance, is as easy as changing two variables in a manifest and deploying it in the cluster:

1. Download the integration manifest YAML file:
   

   curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/nri-prometheus-latest.yaml

2. Edit the nri-prometheus-latest.yaml manifest file, and add a cluster name to identify your Kubernetes cluster (required) and your New Relic license key (required).
   

   env: - name: LICENSE_KEY
   
          value: "<YOUR_LICENSE_KEY>"
   
   [...]
   
   config.yaml: |
   
     cluster_name: "<YOUR_CLUSTER_NAME>"

3. Deploy the integration in your Kubernetes cluster:
   

   kubectl apply -f nri-prometheus-latest.yaml

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map, which shows an example configuration. You can use the example file to configure, for example, how endpoints are scraped and metrics are filtered.

Once you’ve installed the integration for Docker or Kubernetes, you can begin building queries to track and visualize your Prometheus data in New Relic.

See the New Relic docs for more on compatibility and requirements, installation options, data limits, configuration, metric queries, troubleshooting, metric transformation, and more.

Gathering, viewing, and querying Prometheus metrics in New Relic

When you have the integration in place, you’ll first gather metrics from Prometheus, and then query and view them.

Gathering metrics

The integration automatically discovers which targets to scrape. By default, metrics are queried in the /metrics path on port 8080. However, you can use the prometheus.io/port and prometheus.io/path annotations or labels in your Kubernetes pods and services to specify the port and endpoint path the integration should use when constructing the target (but note that annotations take precedence over labels).

For example, if you have a deployment in your cluster, and the pods expose Prometheus metrics on port 8080 in the path my-metrics, set the labels prometheus.io/port to "8080" and prometheus.io/path to "my-metrics" in the PodSpec metadata of the deployment manifest, as shown:

apiVersion: apps/v1

kind: Deployment

metadata:

  name: my-deployment

spec:

  replicas: 2

  selector:

    matchLabels:

      app: my-app

  template:

    metadata:

      labels:

        app: my-app

        prometheus.io/scrape: "true"

        prometheus.io/port: "8080"

        prometheus.io/path: "my-metrics"

The integration will retrieve metrics from your pods by sending a request to http://<pod-ip>:8080/my-metrics.

Viewing and querying your metrics

Once collected, you can query your Prometheus metrics using the New Relic Query Language (NRQL) in New Relic Insights or the New Relic One chart builder.

For example, to get metric names for a specific cluster, namespace, or pod, run one of the following queries:

• Cluster:
  

  FROM Metric SELECT uniques(metricName) WHERE clusterName='<cn>'

• Namespace:
  

  FROM Metric SELECT uniques(metricName) WHERE namespaceName='<ns>'

• Pod:
  

  FROM Metric SELECT uniques(metricName) WHERE podName='<pod>'

From there, you can get attributes and attribute values for a metric:

• FROM Metric SELECT keyset() WHERE metricName='<mn>'

• FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<pod>'

Using the metric name and attributes you retrieved, you can now query your Prometheus data.

For example:

• To get raw metric values:
  

  FROM Metric SELECT <metricname> WHERE <attribute>='<value>'

• To get a graph of the metric (possible aggregators are average, min, max, and sum):
  

  FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES

• To view average memory usage for all pods in a deployment:
  

   FROM Metric SELECT average(container_memory_usage_bytes) WHERE deploymentName='my-app-deployment' AND namespaceName='default'

Read more about viewing and querying metric data in the documentation.

Example usage of Prometheus data in New Relic

There are any number of ways to use Prometheus data in New Relic, but consider the following use cases:

Monitoring etcd

Etcd is key-value data store that’s essential for running Kubernetes clusters. Prometheus pulls metrics from etcd, so to ensure your clusters are healthy, you can use the Prometheus OpenMetrics integration to monitor etcd server, disk, and network metrics such as:

• etcd_server_has_leader
• etcd_server_proposals_failed_total
• etcd_network_peer_sent_bytes_total
• etcd_disk_wal_fsync_duration_seconds

Kubernetes Horizontal Pod Autoscaler (HPA)

HPA automatically scales up a Kubernetes deployment based on user-configured limits. After installing the Prometheus OpenMetrics integration, you can use the following query in the New Relic One chartbuilder (or New Relic Insights) to build a dashboard widget that monitors remaining HPA capacity.

FROM Metric select latest(kube_hpa_status_current_replicas),latest(kube_hpa_spec_max_replicas) where clusterName = '<YOUR CLUSTER NAME>'  facet hpa

Node readiness

In Kubernetes, a node is marked ready when it can accept workloads (pods). If a node is having issues, Kubernetes will label it as "not ready." To create an alert condition for this, using the integration, use the following query:

FROM Metric select latest(kube_node_status_condition) where condition='Ready' and status = 'true' and clusterName = '<YOUR CLUSTER NAME>' facet nodeName

See it in action:

Let New Relic manage and scale your Prometheus data

Whether you’re getting started with Prometheus, or you have been using it in conjunction with a dashboard tool like Grafana to monitor your Kubernetes environment, New Relic can help you get started and scale your data without the hassles of managing Prometheus and your dashboard tool. The New Relic Prometheus OpenMetrics integration allows you to store and visualize those crucial metrics on one platform. With New Relic One, you can more easily combine metrics data with events, traces, and log data (or, as we call them, M.E.L.T.) from all of your entities across your entire software stack—from your Kubernetes backend to your frontend browser UI—for fully connected view of the relationships between your data and applications.

Contributing to the Prometheus OpenMetrics integration

New Relic has contributed the Prometheus integration to the open source community under an Apache 2.0 license.

We welcome contributions to this integration or any of our open source exporters and adapters. If you’d like to contribute, please review our Contributors Guide. Keep in mind that when you submit your pull request, you'll need to sign our CLA. If you'd like to execute our corporate CLA, or if you have any questions, please drop us an email at opensource@newrelic.com.