Monitoring Jenkins pipelines using OpenTelemetry and New Relic

Jenkins is a popular open-source automation tool for implementing continuous integration (CI) and continuous development (CD) workflows called pipelines. Continuous integration and continuous deployment are development practices that can help you deploy your applications faster—and with fewer bugs.  But if errors or other issues come up in your pipelines, it can negatively impact deployment and other steps in the DevOps lifecycle. That's why it’s important to monitor not just your applications but also the automation processes around development, including your CI and CD tools. This can help you shift left, which means incorporating more testing on the left side of the DevOps lifecycle, as shown in the next image.

Shifting left isn't just about testing your code and automating the build and deployment process. It's about monitoring and testing your CI and CD pipelines, too.

In this post, you’ll learn how to monitor your Jenkins jobs and pipelines using the Jenkins OpenTelemetry plugin and a New Relic quickstart integration. After you are finished, you'll be able to:

• Visualize key metrics on your Jenkins pipelines, such as how long your jobs are taking and error rates.
• Use distributed tracing to pinpoint where issues are coming from in your pipelines.
• Create alerts on your Jenkins pipelines.

The next image shows a New Relic dashboard with some of the Jenkins metrics you’ll be able to visualize.

With this integration, you can shift left and get observability for the build and deploy phases of the DevOps lifecycle. Check out the next video or continue reading to get started.

Prerequisites

If you don’t have Jenkins yet, you need to install it. Alternatively, you can use Docker to follow along with this example.  If you’re using Docker, you can get an instance of Jenkins up and running with the following command: 

docker run --name jenkinsci -d -e OTEL_LOGS_EXPORTER="otlp" -p 8080:8080 jenkins/jenkins:lts

Setting up the OpenTelemetry plugin

1. Log into Jenkins (usually http://localhost:8080) and log in with your username and password.
2. From the Jenkins dashboard, select Manage Jenkins from the lefthand pane.
3. Select Manage Plugins.

4. Select the Available tab and type OpenTelemetry in the search bar.

5. Select the OpenTelemetry checkbox and choose Install without restart.

6. After the installation is complete, select Restart Jenkins when installation is complete and no jobs are running.

7. Log into Jenkins again and select Manage Jenkins.
8. Under System Configuration, select Configure System.

9. Scroll down to the OpenTelemetry section and change the following settings:

• OTLP Endpoint should be https://otlp.nr-data.net:4317.

• Authentication should be Header Authentication.
• The Header Name should be api-key.
• For the Header Value, select Add > Jenkins. You’ll need to create a secret text with the following fields:
  • For Kind, choose Secret text.
  • The Secret should be your New Relic license ingest key. 
  • ID can be left blank.
  • Add a description such as New Relic INGEST - LICENSE key. 
  • Select Add.

10. After you’ve added your credentials, you’ll be able to select New Relic INGEST - LICENSE Key from the Header Value dropdown menu.
11. Select Save.

Sending Jenkins data to New Relic

After you’ve set up your New Relic credentials in Jenkins, you’re ready to send data to New Relic. Let’s start by setting up a test pipeline that will automatically fail. By testing a pipeline, you’re making sure that the connection to New Relic is set up properly. You can set up a test pipeline with the following steps.

1. In the Jenkins dashboard, select New Item.

2. Enter an item name (the name in this example is test) and select Pipeline. Then select OK.

3. Scroll to the bottom, select Scripted Pipeline, and then select Save.

Some older versions of Jenkins don’t have sample pipelines that you can choose from the list. If this is the case for you, paste in the following script and select Save.

4. Select Build Now. This sample pipeline should fail, which is the expected behavior.

5. Log into New Relic, go to Services - OpenTelemetry, and select jenkins.

Now you can go to Transactions to see your data. You should see all of the pipeline build steps, including both successes and failures. The next image shows a Jenkins dashboard in New Relic. It includes information on top transactions, top 20 transactions, and throughput.

You can also go to Distributed Traces to see data from your Jenkins pipeline.  In this example, you can see exactly where the pipeline starts failing, which is helpful for creating alerts. The next image shows a New Relic dashboard with specific spans in a Jenkins pipeline distributed trace.

If you’ve set the environment variable to OTEL_LOGS_EXPORTER="otlp", you can see your Jenkins console logs in the Logs tab. The next image shows logs related to a Jenkins pipeline.

Visualize your Jenkins data

Once you have Jenkins data flowing into New Relic, you can add our quickstart dashboard to get an overview of your Jenkins environment. For example, you can see a summary of the number of Jenkins instances available, the number of successful jobs, the top errors, and the job queue status. To get this dashboard, simply follow these steps:

1. Go to the Jenkins quickstart in New Relic Instant Observability, and select Install now.
2. Select an account and select Begin installation.
3. If you've already completed the earlier steps in this tutorial, select Done to move on to the next step.
4. The quickstart deploys the resources to your account. Select See your data to get to the dashboard.

After you're done, you'll have complete visibility into your Jenkins pipelines, from high-level metrics to logs and distributed traces. You can also optionally choose to add alerts when certain thresholds are met, such as when jobs are taking too long or when the error rate gets too high. To do so, you can follow along with the docs to first create a policy and then set conditions on that policy. You can send alerts to the notification channels of your choice. Options include Slack, PagerDuty, email, and more.

Sending your Jenkins pipeline data to New Relic gives you observability into the building and deployment phases of the DevOps lifecycle, giving you more insights into your DevOps practices and helping you shift left.