How to Run Custom New Relic On-Host Integrations in Containers

On-host integrations for the New Relic Infrastructure agent are composable by design and allow multiple teams to add their own custom integrations to a host. However, if you have to run the Infrastructure agent in a container (for example, some workflows may require that you run the agent in CoreOS Container Linux), adding custom integrations becomes more difficult.

So how do you create effective hardware monitoring integrations, while still allowing hardware owners to add additional custom integrations that they may need to run on their hosts?

Let’s look at some potential approaches and an ideal a solution.

Notes:

• This post builds examples using the official newrelic/infrastructure image, available in the public Docker repo.
• Configuration and definition files related to custom integrations are added to hosts via the /etc/newrelic-infra/integrations.d and /var/db/newrelic-infra/custom-integrations directories.

Potential solution: build a new container image

An obvious solution to this problem would be to build a new container image, based on the official newrelic/infrastructure image. In this case, you could simply add the required custom integrations to the image using a Dockerfile. For example:

FROM newrelic/infrastructure:latest

RUN yum update -y && \

  yum install -y python && \

  yum clean all

ADD ./root/etc/newrelic-infra/integrations.d/* /etc/newrelic-infra/integrations.d/

ADD ./root/var/db/newrelic-infra/custom-integrations/* /var/db/newrelic-infra/custom-integrations/

If you’re part of a single team that has ownership over all the integrations you require, this approach could be workable. It becomes less desirable, though, when multiple teams supply integrations to the host. Also, if you update a single integration, you have to update the entire container and redeploy the Infrastructure agent.

Potential solution: volume mounts

Another option would be to use volume mounts to attach host directories to /etc/newrelic-infra/integrations.d and /var/db/newrelic-infra/custom-integrations, and then install your integrations into those directories using a configuration management tool like Ansible.

You could accomplish this by adding the following two arguments to your Docker run command:

-v "/etc/newrelic-infra/integrations.d:/etc/newrelic-infra/integrations.d" \

-v "/var/db/newrelic-infra/custom-integrations:/var/db/newrelic-infra/custom-integrations" \

Here’s the biggest problem with this approach: Unless your integration is a statically compiled binary written in a language like Go, than you must ensure that you add any supporting dependencies your integration might require (for example, Python and other library dependencies) to the container. One missed dependency could lead to a lot of needless troubleshooting.

The ideal solution: containerize each integration

Instead of building a new container image or using volume mounts, what if you containerized each custom on-host integration? Teams could write, build, and deploy their integration inside a container image, and you could then simply supply the Infrastructure agent with a managed integration config to tell it how to run the containerized integrations.

Install the Docker client in the Infrastructure agent container

To get the Infrastructure agent to natively monitor Docker, you are already required to mount the Docker unix socket into the Infrastructure container at /var/run/docker.sock. This means that you only need to install the Docker client into the Infrastructure container, to make it possible for the agent to launch Docker containers that contain your integrations. You can do this by building on top of the official Infrastructure agent image, as shown in the following code:

FROM newrelic/infrastructure:latest



# Install Docker CE



RUN yum install -y \

   yum-utils \

   device-mapper-persistent-data \

   lvm2 && \

 yum-config-manager \

   --add-repo \

   https://download.docker.com/linux/centos/docker-ce.repo && \

 yum install -y \

   docker-ce && \

   yum clean all

The resulting image includes the standard New Relic Infrastructure agent and Docker Community Edition (CE).

Using a Linux distribution based on systemd, you can start this container with a unit file that includes your New Relic license key and Docker tag for the custom image:

[Unit]

Description=New Relic Infrastructure Agent

After=docker.service

Requires=docker.service



[Service]

TimeoutSec=0

Restart=always

ExecStartPre=-/usr/bin/docker kill %p

ExecStartPre=-/usr/bin/docker rm %p

ExecStartPre=/usr/bin/docker pull myorg/nr-infrastructure:latest

ExecStart=/usr/bin/docker run \

   --net=host \

   --cap-add=SYS_PTRACE \

   --label service_name=%p \

   -e NRIA_IS_CONTAINERIZED=true \

   -e NRIA_OVERRIDE_HOST_ROOT=/host \

   -e NRIA_LICENSE_KEY=${LICENSE_KEY} \

   -v /:/host:ro \

   -v "/var/run/docker.sock:/var/run/docker.sock" \

   -v "/etc/newrelic-infra:/etc/newrelic-infra" \

   -v "/var/db/newrelic-infra/custom-integrations:/var/db/newrelic-infra/custom-integrations" \

   --name=%p \

   --memory="1g" \

   --cpus="2.0" \

   myorg/nr-infrastructure:latest

ExecStop=/usr/bin/docker stop %p

ExecStopPost=-/usr/bin/docker rm %p



[Install]

WantedBy=multi-user.target

Once this is running on all of your hosts, it’s possible to deploy and manage multiple containerized integrations on the host system with a tool like Ansible. So, what exactly would a custom on-host integration look like in this case?

Containerize the custom integration

Let’s assume you’ve built a standard on-host integration with the Integrations SDK in Go, and the resulting binary is called my-integration. Now you need to create a container for it, which you can do with the following Dockerfile:

FROM golang:1.10.1

# Build Image



ENV CGO_ENABLED=0

ENV GOOS=linux



COPY . /go/src/my-integration/

WORKDIR /go/src/my-integration

RUN make



FROM alpine:latest

# Deploy Image



ENV NRIA_CACHE_PATH=/tmp/my-integration.json

RUN apk --no-cache add \

ca-certificates

WORKDIR /

COPY --from=0 /go/src/my-integration/bin/my-integration .

ENTRYPOINT ["/my-integration"]

This Dockerfile utilizes Docker’s multi-stage build functionality (available in 17.05 and higher) to create a small Alpine Linux-based container with the my-integration binary in the root directory.

To build the container, use the following command:

docker build -t myorg/nr-infra-integrations-mycheck:production

Now give the integration a test run:

docker run --net=host --rm --tty myorg/nr-infra-integrations-mycheck:production

The JSON result should resemble the following:

{"name":"com.myorg.my-integration","protocol_version":"1","integration_version":"0.1.0","metrics":[{"ActiveUsers":"443","event_type":"CheckData"}],"inventory":{"EncryptionType":{"value":"bcrypt"}},"events":[]}

Configure and enable the integration

https://docs.newrelic.com/docs/infrastructure/host-integrations/infrastructure-integrations-sdk/get-started/introduction-infrastructure-integrations-sdk/">definition file and a configuration file (both in YAML) to your host system.

Create the configuration file in /etc/newrelic-infra/integrations.d/, and name it something like my-integration.yml. It should resemble the following:

integration_name: com.myorg.my-integration



instances:

 - name: my-integration-metrics

   command: metrics

   labels:

     env: production

     role: webserver



 - name: my-integration-inventory

   command: inventory

   labels:

     env: production

     role: webserver

The final file you need to create is where most of the magic actually happens—this is the definition file. Create the file in /var/db/newrelic-infra/custom-integrations/, and name it something like my-integration-definition.yml. It should resemble the following:

name: com.myorg.my-integration

description: Reports status and metrics for my service

protocol_version: 1

os: linux



commands:

 metrics:

   command:

     - /usr/bin/docker

     - run

     - --net=host

     - --rm

     - --tty

     - myorg/nr-infra-integrations-mycheck:production

     - -metrics

   interval: 15



 inventory:

   command:

     - /usr/bin/docker

     - run

     - --net=host

     - --rm

     - --tty

     - myorg/nr-infra-integrations-mycheck:production

     - -inventory

   prefix: config/check

   interval: 60

Once you’ve written these two files, ensure you have the most current Docker image on the host, and restart the Infrastructure agent.

systemctl restart newrelic-infra

Assuming everything is working, you should start seeing your data show up inside the New Relic platform almost immediately.

Note: If you have any custom on-host integration containers that are expensive to start up, like the hardware monitoring frameworks for Dell (OpenManage) or SuperMicro (SuperDocker) servers, you can run them permanently via a systemd unit file (or another init system configuration). Then you can configure the Infrastructure agent to use docker exec commands (instead of docker run commands) to execute the integration check inside the running container, and not have to launch a new container for each check cycle.

In conclusion

Containerizing each custom integration makes it easy to create, manage, and run multiple on-host integrations with a containerized Infrastructure agent. When multiple teams can add the integrations that they require onto a host, it greatly reduces the need for extensive, time-consuming cross-team coordination. Containerizing the checks often makes them easier to build, test, and deploy since they are essentially self-contained CLI commands and include all the dependencies that they require. Containerizing the integrations also makes them much easier to deploy onto platforms like CoreOS Container Linux, which is finely tuned for containerized workloads. To learn more about New Relic integrations, take the next step by reading about running infrastructure agents as a non-root user.

This approach should help you maintain a high level of flexibility throughout your platform, all the way down to the hardware. The isolation created by containers is ideal for this use case and should help give you other ideas for how you can separate workloads into more manageable components by leveraging containers with New Relic.

Learn more about New Relic integrations and our platforms like Infrastructure Monitoring today.