Level Up Observability for SolarWinds Orion with New Relic (Part 3)

This three-part blog series (read part 1 and part 2) explores why you’d want to improve your observability with SolarWinds, and how to do it.

When setting up your systems for observability, alerts can be a great starting point for data ingestion, as they’re typically an essential part of proactive troubleshooting workflows.

In a previous post, I showed how to use a PowerShell script to post alerts data from SolarWinds Orion to the New Relic Events API. Now you can take this one step further and add SolarWinds alerts as an additional source of data to New Relic Applied Intelligence (AI) to dynamically correlate incidents across all of your monitoring tools and data sources, reducing alert fatigue, engineer stress, and your mean time to resolution (MTTR).

In this post, I’ll show you how to post directly to both the REST API for New Relic AI and the Events API from the SolarWinds UI, reducing the script maintenance overhead.

Sending SolarWinds alerts to the Events API

In this first example, I’ll use SolarWinds’ send a GET or POST request to a web server alert action to send an alert trigger and a reset event to New Relic via the Events API. To do this, you’ll need an Events API Insert Key and a New Relic account ID.

Once you’ve identified an alert to begin with, navigate to the Trigger Actions tab, and add a new alert action, selecting the Send a GET or POST Request to a Web Server type.

In the form, edit the following fields:

• Name of action
  • Set a display name
• URL
  • Set to: https://insights-collector.newrelic.com/v1/accounts/ACCOUNT_ID/events
  • Replace ACCOUNT_ID with your New Relic Account ID
• Select Use HTTP/S POST
  • This will bring up the Body to POST field, which I’ll describe below
• ContentType
  • Change this to application/json
• Authentication
  • Select Token
• Header Name
  • Set to X-Insert-Key
• Header Value
  • Replace EVENT_API_INSERT_KEY with your Insert Key

For the Body to POST section, I’m going to use a very basic payload with common variables used in the previous blog post. This example is a very rudimentary starting point, and you should absolutely decorate your payload with as much metadata as you see fit. (In observability, cardinality matters.)

[

    {

       "eventType":"solarwinds_alerts",

       "swAlert.alertActionType":"Trigger",

       "swAlert.alertName":"${N=Alerting;M=AlertName}",

       "swAlert.alertMessage":"${N=Alerting;M=AlertMessage}",

       "swAlert.alertingEntity":"${N=SwisEntity;M=Caption}",

       "swAlert.alertingEntityType":"${N=Alerting;M=ObjectType}",

       "swAlert.alertingSeverity":"${N=Alerting;M=Severity}"

    }

]

For your alert reset action, you can generally follow the same settings, but set a more meaningful value, such as “Reset” or “Cleared” for the swAlert.alertActionType key. This extra step allows your teams on the receiving end of this data stream to understand the blast radius of your SolarWinds alerts and keep abreast of the current status of alerts in near-real time.

Sending alerts to the REST API for New Relic AI

In this next example, I’ll show you how to use a very similar pattern in SolarWinds to send your alerts telemetry into the New Relic AI Incident Intelligence feed, via REST API. To proceed, you’ll need your secure token.

Just like sending SolarWinds data to the Events API, you’ll start by navigating to the Trigger Actions tab, and add a new alert action, selecting the Send a GET or POST Request to a Web Server type.

In the form, edit the following fields:

• Name of action
  • Set a display name
• URL
  • Set to: https://collectors.signifai.io/v1/incidents
• Select Use HTTP/S POST
  • This will bring up the Body to POST field, which I’ll describe below
• ContentType
  • Change this to application/json
• Authentication
  • Select Token
• Header Name
  • Set to Authorization
• Header Value
  • Set to Bearer BEARER_TOKEN and replace BEARER_TOKEN with your secure token

For this example, I’m going to use a  slightly more advanced payload in the Body to POST section, by adding some custom properties from SolarWinds as additional attributes.

{

  "events": [{

      "attributes": {

           "alert/policy_name": "${N=Alerting;M=AlertName}",

           "alert/id": "${N=Alerting;M=AlertActiveID}",

           "state": "alarm",

           "label/environment": "${N=SwisEntity;M=CustomProperties.Environment}",

           "label/team": "${N=SwisEntity;M=CustomProperties.Team}",

           "label/object_type": "${N=Alerting;M=ObjectType}"

      },

      "event_description": "${N=Alerting;M=AlertMessage}",

      "event_source": "SolarWinds NPM",

      "host": "${N=SwisEntity;M=Caption}",

      "value": "${N=Alerting;M=Severity}"

  }]

}

Per the documentation, you’ll need to include the following required properties:

• event_source
  • Set a name that describes where this event comes from
• host
  • Specify which host, service, or application generated the event
• value
  • Set an incident priority (or severity)
• event_description
  • Provide a free text description of the event. (This is a good candidate for the Alert Message field in SolarWinds.)

The attributes block is a key-value map of additional metadata that you define as needed. Note that the data you include here helps New Relic perform better correlations, and annotating keys as significant attributes, such as labels/*, will directly impact the deduplication process. In essence, the more data you can provide, the easier it is for NR AI to create associated correlations.

Learn more about observability for SolarWinds Orion

Now you have a few more tools for your toolbox, and hopefully, some more ideas for applying observability best practices in your work. Don’t miss Part 1 and Part 2 of this series to learn more about why and how to use SolarWinds data in New Relic.

If you have questions about these techniques, observability in general, or tacos(!), come chat with me in the Explorers Hub. Or, if you're interested to see how New Relic works with partners, see our case study with ZenHub.

For an introduction to AIOps best practices with New Relic AI, check out Accelerate Incident Response with AIOps.