How to monitor with Prometheus

Prometheus is a powerful, open source tool for real-time monitoring of applications, microservices, and networks, including service meshes and proxies. It’s especially useful for monitoring Kubernetes clusters. Originally developed by Soundcloud in 2012, it’s now part of the Cloud Native Computing Foundation.

Prometheus has a dimensional data model, which is ideal for recording time series data—in other words, your metrics. It also includes PromQL, a powerful query language that helps you drill down into those metrics. Finally, Prometheus is extremely reliable because its servers are standalone and continue to operate even when other parts of your system are down.

In this article, you’ll learn the basics of how Prometheus collects data and how you can see your Prometheus metrics.

How does Prometheus collect data?

A Prometheus server collects metrics data from its targets using a pull model over HTTP. But what exactly is a pull model, and how is it different from a push model?

Push versus pull model

On a basic level, it really is as simple as it sounds. A Prometheus server pulls telemetry data from somewhere else. The server uses jobs to scrape HTTP endpoints. The data from those endpoints is then pulled into the Prometheus database.

A push model collects data and then pushes it to a database. New Relic and other observability platforms use the push model. That means using instrumentation, the process of installing agents in your systems, to collect telemetry data. The agent, which is embedded in your software, collects the data and pushes it to another server.

Both push and pull methods work well for monitoring and observability, and OpenTelemetry (OTEL), which is open source and vendor-neutral, supports both methods.

Targets and jobs

A Prometheus server pulls its data by scraping HTTP endpoints. The endpoints provide a continuous stream, allowing the Prometheus server to collect real-time data. These endpoints are also known as targets or instances, and a collection of instances that have the same purpose is known as a job. Jobs are commonly used for scalability and reliability.

Exporters and native Prometheus endpoints

It may seem like a major limitation that Prometheus can only pull data from HTTP endpoints. However, sometimes you need to monitor data from things that don't have HTTP endpoints such as proxies, hardware, and databases. In those cases, you generally need to use exporters. Exporters act as an intermediary between services and your Prometheus servers, collecting data from the services and making it available to Prometheus for scraping. Because Prometheus has a robust open source community, there are a lot of third-party exporters that help you expose data to a Prometheus server.

Many open source projects also have native Prometheus endpoints. When you use projects with native Prometheus endpoints, you don’t need an exporter. These open source projects make it easier to integrate Prometheus without any additional work from engineers. Examples include:

• ArgoCD, which helps manage GitOps workflows
• CoreDNS, which is used to manage DNS for Kubernetes’ clusters
• Istio, an open source service mesh

Time series database and multidimensional model

The Prometheus server pulls data into a time series database. Time series data includes a timestamp and optional labels stored as key-value pairs, and it’s especially useful for tracking how data changes over time.

While other database systems such as Cassandra, PostgreSQL, and MongoDB provide support for time series data, Prometheus is specifically designed for this kind of data. That makes Prometheus ideal for real-time monitoring.

Prometheus also uses a multidimensional model. That means you can query your data in many different ways, such as by job, endpoint, or time period. The ability to slice and analyze your data is very important for getting full observability into the systems you’re monitoring. For instance, narrowing down a query to a specific endpoint might correspond to a microservice or piece of hardware. On the other hand, querying data over a time period provides metrics for dashboards, including average response times and a high-level view of how a system is performing. Ultimately, a multidimensional model gives you tremendous flexibility in how you query and analyze your telemetry data.

Now that you know how Prometheus scrapes, pulls, and stores data, how can you actually see that data?

How can you see your Prometheus metrics?

Prometheus metrics provide valuable insight that can help you monitor your services. Prometheus comes with a flexible query language known as PromQL (short for Prometheus Query Language) that you can use to query your metrics. But where do you query your data, and how do you visualize it? There are several different options:

• Use the built-in expression browser. Prometheus provides an expression browser that you can use to see your data. It’s easy to use but very basic.
• Use an analytics and visualization platform. The most popular option is Grafana, which is open source and provides powerful dashboard and visualization features.
• Use an observability platform. You can use an observability platform like New Relic, which provides dashboards and visualizations like Grafana. An observability platform also provides data storage and the ability to instrument services you aren’t monitoring with Prometheus, which platforms focused on just analytics and visualization don’t provide.
• Combine multiple platforms. You can also combine the benefits of both kinds of platforms as well.

Let’s take a look at each option. Stating the obvious: you need to have a Prometheus server deployed and scraping at least one target before you can look at any data. If you haven’t learned how to do that yet, check out the Prometheus Getting started documentation for a tutorial that shows you how to download, run, and set up a basic configuration.

Expression browser

Once you’ve set up at least a basic Prometheus server, you can use Prometheus’s built-in expression browser at localhost:9090/graph to view your metrics. If you haven’t set up any external targets yet, you can even have Prometheus scrape itself (generally at localhost:9090). While that’s not extremely useful in itself, it’s a good way to ensure that Prometheus is up and running.

While you can make PromQL queries directly to a Prometheus server, that’s not the most effective way to see your metrics in action. Prometheus doesn’t offer features like distributed tracing, real user monitoring, and application performance monitoring, and its dashboards aren’t very fancy. You’ll likely want to query and visualize your data somewhere else.

Grafana

Grafana is open source analytics and visualization software that you can use to query your Prometheus data. Note that Grafana does not store your Prometheus data. Grafana’s only job is to query and visualize, and it does both very well. Here’s an example of a Grafana dashboard visualizing Prometheus metrics.

If you want to build dashboards with your Prometheus data in Grafana, you just need to add Prometheus as a data source and then create a Prometheus graph.

Observability platform

An observability platform like New Relic gives you additional options for scaling and storing your data. New Relic provides up to 13 months of storage for your data and is compatible with PromQL.

To store Prometheus data in New Relic, you can set up the remote write integration or use the Prometheus OpenMetrics integration for Kubernetes or Docker. The steps for setting up the remote write integration are included in Harnessing the power of open source metrics in New Relic. You can also read the documentation to learn more about both integration options. 

Combining multiple platforms

Fortunately, you don’t need to choose between the benefits of Grafana and the benefits of an observability platform. Grafana is so popular and useful that many observability platforms include integrations so you can use both. With New Relic, you can use New Relic’s Prometheus remote write integration to combine Grafana with both Prometheus and New Relic. With the remote write integration, New Relic ingests Prometheus metrics, then passes that data on to Grafana. You get the enhanced scalability and features of New Relic along with Grafana’s dashboards. New Relic also offers a Prometheus OpenMetrics integration that allows you to scrape data from Prometheus endpoints and send it to New Relic.

We recommend trying both Grafana and New Relic. There are advantages to both platforms, and you can benefit from using them together.