How to Implement New Relic on Linux Mainframe Hardware

We’re constantly testing the limits of our platform to deliver more value for our customers. In the process, we occasionally stumble upon something really unexpected, which is exactly what happened when we set our sights on applying New Relic’s capabilities to mainframe hardware.

Specifically, we wanted to see if we could address a potential customer need by getting New Relic Infrastructure and New Relic Logs up and running for Linux on mainframe hardware.

If you’re unfamiliar with mainframes, they’re basically high-performance, incredibly secure mega-computers with tons of memory and processing power. Given their powerful abilities, mainframes can act as servers, but they’re also used to run scalable software in industries where data security is paramount such as banking, healthcare, and insurance.

With the immense amount of data that mainframes’ can process (up to 1 trillion web transactions daily, according to IBM), we thought it would be useful if customers were able to gather logs from such a system and view them alongside their infrastructure data in the New Relic platform.

To tackle this, we used New Relic Logs, which simplifies log management. New Relic Logs reduces the complexity of gathering and visualizing your log data by using plugins that integrate with some of the most common open source logging tools, like Fluentd, Fluent Bit, and Logstash. For this mainframe application, we relied on Fluentd and another New Relic integration, Flex, which we’ll get to in a bit.

Best of all, we think there are more than enough customers who will find this discovery exciting.

Here are the four steps involved:

1. Create a VM running on mainframe hardware
2. Install New Relic Infrastructure
3. Set up Fluentd for mainframe logs
4. Add Flex for New Relic

Before you get started, it’s important to note that our process was tested with Red Hat Enterprise Linux on mainframe.

Step 1: Create a VM running on mainframe hardware

For starters, visit IBM LinuxOne Community Cloud, which is a platform provided by IBM where you can get a free virtual machine (VM) running on top of mainframe hardware. Register for a trial at LinuxOne and then:

1. Log in, go to Virtual Servers, create a new virtual machine, and select general purpose VM.
2. Select RHEL7.6.
3. Create a new SSH key pair and download to your computer.

Then, access your VM:

1. After a couple of minutes, the machine is in the ACTIVE state:

   

2. Change the permissions of your .pem key:
   

   chmod 600 <YOUR_KEY>.pem

3. Use the following command to ssh into the VM:
   

   ssh -i linuxone.pem <username>@<host>

   

4. Check the architecture of this machine—IBM/S390 (a.k.a. IBM Z or mainframe):
   

   cat /proc/cpuinfo

   

Step 2: Install New Relic Infrastructure

1. Download the New Relic Infrastructure agent binary, compiled for mainframe architecture (S390), to the VM and unpack the contents:
   

   wget 
   
   https://download.newrelic.com/infrastructure_agent/binaries/linux/s390x/newrelic-infra_linux_1.10.26_s390x.tar.gz
   
   tar xzf newrelic-infra_linux_1.10.26_s390x.tar.gz

2. Update your New Relic license key in config_default.sh:
   

   cd newrelic-infra
   
   vi config_default.sh

3. Run the installer to install the New Relic Infrastructure agent:
   

   sudo ./installer.sh

   

4. Now, check out the data—CPU, memory, and disk metrics, but also details about the processes running on the server—coming into New Relic Infrastructure:

   

   And we also have some Inventory data, which gives you a real-time, filterable, searchable view into each host's configuration:

   

Step 3: Set up Fluentd for mainframe logs

1. Download Fluentd—the open source data collector—for mainframe:
   

   wget -q 
   
   https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Fluentd/1.7.4/build_fluentd.sh
   
   bash build_fluentd.sh

2. Install the New Relic Fluentd plugin:
   

   export GEM_HOME=${HOME}/.gem/ruby
   
   export PATH=${HOME}/.gem/ruby/bin:$PATH
   
   fluent-gem install fluent-plugin-newrelic

3. Create a configuration directory for our Fluentd config:
   

   fluentd -s conf

4. Add the following to conf/fluent.conf (and don't forget to replace your license key):
   

   <match **>
   
   @type newrelic
   
   license_key <new_relic_license_key>
   
   </match>

5. Start Fluentd and let's test this:
   

   fluentd -c conf/fluent.conf &
   
   echo '{"message":"message to New Relic from my mainframe"}' | fluent-cat 
   
   debug.test

6. Go to New Relic and check out the Logs:

   

Step 4: Add New Relic with Flex

Getting New Relic Infrastructure and Logs up and running is great, and the combination instantly provides us with valuable data. Alongside logs and infrastructure metrics though, we can also easily collect metrics from a wide variety of services using New Relic Flex—an integration that complements Infrastructure’s out-of-the-box integrations.

Flex gives you the tools you need to build simple integrations for any service that exposes data over a standard protocol (such as HTTP) or in a standard format (such as JSON). This allows you to consume metrics from a large variety of services without the need to write any code, just by providing a configuration file. Many example configuration files can be found here.

Let's set up Flex.

1. Install Go:
   

   wget -q
   
   https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Go/1.13.5/build_go.sh bash build_go.sh wget https://raw.githubusercontent.com/golang/dep/master/install.sh bash install.sh sudo ln -s /home/linux1/go/bin/dep /usr/bin/dep sudo ln -s /home/linux1/go/bin/golangci-lint /usr/bin/golangci-lint

2. Compile Flex:
   

   go get github.com/newrelic/nri-flex
   
   cd go/src/github.com/newrelic/nri-flex
   
   make clean
   
   make compile-only

3. The nri-flex binary will be located in ./bin/linux/nri-flex. Copy the nri-flex binary to the custom-integrations folder:
   

   sudo cp bin/linux/nri-flex /var/db/newrelic-infra/custom-integrations/

4. Create a Flex configuration. We'll use the linux-file-count example. This example Flex config file will count the number of files in a specified directory (/etc/ in our example). So, create a file integrations.yaml in /etc/newrelic-infra/integrations.d/ and add the following contents:
   

   integrations:
   
   
   
     - name: nri-flex
   
       config:
   
         name: linuxFileCount
   
         apis:
   
          - name: linuxFileCount
   
            commands:
   
            - run: echo -e "{\"searchFilter\":\"/etc\",\"fileCount\":\"$(ls -1 /etc | wc -l)\"}"

   Notice how easy it is to build your own integrations for New Relic.

5. Check out the data streaming into New Relic:

Hopefully you found this use case as awesome as we did. If you are looking for other software to deploy on your favorite mainframe machine, check out some of these resources with corresponding documentation.Want to learn more? See how we achieved AWS Lambda ready designation.

And if you’re curious about the benefits of using a technology like New Relic to drive business innovation, Interested in what else we offer? Explore more about New Reflix by reading our study case with ZenHub.”