How to Use the New Relic Kubernetes Operators for Parallel Deployment

The New Relic Developer Toolkit offers a suite of tools expressly built to reduce toil for developers. In Practicing Observability as Code: Defining New Relic Alert Policies with Terraform, we explained how to use the New Relic Terraform provider to deploy monitoring and alerting configurations alongside your infrastructure and application code. Now we're doing the same for Kubernetes.

In this post, we'll briefly go over Kubernetes Operators, why they're useful, and show how the New Relic Kubernetes Operator provides a seamless way to deploy New Relic resources alongside your Kubernetes deployment.

What is a Kubernetes Operator?

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Kubernetes is becoming the de-facto standard for orchestrating containerized services in the cloud. Kubernetes introduced the Operator pattern in version 1.7, giving you the ability to extend Kubernetes by defining custom Kubernetes objects that can perform domain-specific operations, such as configuring and provisioning third-party services.

About the New Relic Kubernetes Operator

The New Relic Kubernetes Operator provides the ability to configure your New Relic monitoring resources the same way you manage the rest of your Kubernetes configuration.

For example, once you've installed the operator in your cluster, you can create a New Relic alert policy via kubectl apply using New Relic's custom Kubernetes AlertsPolicy object. You can also configure the AlertsPolicy object using NRQL alert conditions. To configure a standalone New Relic NRQL alert condition and apply it to a pre-existing alert policy, you can use New Relic's custom AlertsNrqlCondition object.

Note: This post was written with the New Relic Kubernetes Operator version v0.0.2 and Kubernetes version 1.18.2.

Before you begin

This walkthrough assumes you’ve already deployed a Kubernetes cluster. You could even create a local cluster on your machine with kind. To complete the full exercise, you’ll need to:

1. Install kubectl, and point it at the correct cluster, as this will determine the cluster where you’ll install the New Relic operator.
2. Install kustomize.

Installing the operator on your Kubernetes cluster

Installing the New Relic operator is a two-step process.

First, install cert-manager, which automatically provisions and manages TLS certificates in Kubernetes:

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml

Next, install the Kubernetes operator:

kustomize build https://github.com/newrelic/newrelic-kubernetes-operator/configs/default | kubectl apply -f -

To confirm the installation was successful, run a few kubectl commands to check the status of the Kubernetes operator.

Ensure the Kubernetes operator's namespace, newrelic-kubernetes-operator-system, has been applied:

kubectl get namespaces

The output should be similar to the example below, which includes the Kubernetes operator's namespace, newrelic-kubernetes-operator-system:

NAME                                     STATUS   AGE

cert-manager                              Active   4m35s

default                                   Active   20m

kube-node-lease                           Active   20m

kube-public                               Active   20m

kube-system                               Active   20m

newrelic-kubernetes-operator-system       Active   3m48s

Now, make sure the Kubernetes operator's controller manager is running:

kubectl get pods --namespace newrelic-kubernetes-operator-system

Note: Don't forget to include the --namespace (shorthand -n) option when running kubectl get pods to ensure you're inspecting resources within the correct namespace.

You should see output similar to the following:

NAME                                                              

READY   STATUS    RESTARTS   AGE

newrelic-kubernetes-operator-controller-manager-7b9c64f58crwg9j   2/2     

Running   0          157m

If you see output similar to the above, you’re ready for the next step. If you don’t see a pod named newrelic-kubernetes-operator-controller-manager-<hash>, double check your Kubernetes configuration to ensure you’re within the correct context and pointing to the correct cluster.

Using the Kubernetes operator

Now that Kubernetes operator is deployed to your cluster, you can put it to work. You’ll write the alert policy and NRQL alert condition configuration the same way you write your other Kubernetes configurations.

Workflow overview

Generally, it’s a three-step process to create your first alert policy using kubectl apply.

1. Using a declarative approach, you’ll write an alert policy configuration file.
2. You’ll add your New Relic personal API key to the configuration.
3. You’ll run kubectl apply when your configuration is ready to go.

Creating your first alert policy

To kick things off, start small. First, create an alert policy with the minimum required configuration, then add a NRQL alert condition to the policy, which will add the condition to the policy in New Relic.

A minimal alert policy configuration is represented in the code below. For the sake of this walkthrough, name this file new_relic_alert_policy.yaml.

new_relic_alert_policy.yaml

apiVersion: nr.k8s.newrelic.com/v1

kind: AlertsPolicy

metadata:

  name: my-policy

spec:

  account_id: <your New Relic account ID>

  api_key: <your New Relic personal API key>

  name: "Alert Policy Created With k8s" # Feel free to rename

  region: "us"

Note: For help locating your personal API key, check out New Relic's personal API key documentation.

Now run the kubectl apply command to create your alert policy:

kubectl apply -f ./new_relic_alert_policy.yaml

You'll see output that reads similar to the following:

alertspolicy.nr.k8s.newrelic.com/my-policy created\

Confirm that your alert policy was created by viewing your policies at alerts.newrelic.com/accounts/{your account ID}/policies. You can search for your new policy by its name. In this case , you’d search for "Alert Policy Created With k8s."

You should see your new alert policy. Now it’s time to add a NRQL alert condition to the policy using the same configuration file.

Add NRQL alert conditions to your alert policy

Now that you’ve created an alert policy, you’ll want to add some alert conditions to the policy so you can trigger alerts when certain metrics are out of line.

In your new_relic_alert_policy.yaml file, add a NRQL alert condition to the policy that will alert you when an application's average overall response time is above five seconds for a three minute period.

new_relic_alert_policy.yaml

# The policy from the previous steps

apiVersion: nr.k8s.newrelic.com/v1

kind: AlertsPolicy

metadata:

  name: my-policy

spec:

  account_id: <your New Relic account ID>

  api_key: <your New Relic personal API key>

  name: "Alert Policy Created With k8s" # Feel free to rename

  region: "us"



  # Add a NRQL alert condition to the policy

  conditions:

    - spec:

        type: "NRQL"

      name: "NRQL Alert Condition Created With k8s"

      nrql:

        query: "SELECT average(duration) FROM Transaction WHERE appName = 'YOUR APP NAME'"

        evaluationOffset: 3

      enabled: true

      terms:

        - threshold: "5"

          threshold_occurrences: "ALL"

          threshold_duration: 180

          priority: "CRITICAL"

          operator: "ABOVE"

      violationTimeLimit: "ONE_HOUR"

      valueFunction: "SINGLE_VALUE"

Note: To receive notifications when an alert is triggered, you'll want to add notification channels to your alert policy.

With the alert condition added to the configuration,  you can apply the update, which will create a NRQL alert condition and add it to your policy.

kubectl apply -f ./new_relic_alert_policy.yaml

To confirm the successful creation of the NRQL alert condition, refresh your alert policy. If you see a new alert condition added to the alert policy, it was a success.

To finish things off,  create and add an alert channel to your alert policy. For example, maybe you want to send an email out to your team when your alert condition is triggered.

What’s next?

Nice work—now you can manage your New Relic alert policies and NRQL alert conditions with code that integrates seamlessly within your Kubernetes workflow. This allows you to configure and manage your alerts with a domain-specific pattern, providing consistency and maintainability. You also gain the benefits of code reviews for any potential changes moving forward.

As you and your team progress, you may need to adjust some of the configuration values to better fit your needs.

The New Relic Kubernetes Operator is just one of several tools in the New Relic Developer Toolkit aimed at facilitating observability as code.

Check out the New Relic Kubernetes Operator and other tools to help automate your workflows at New Relic Open Source.