Australia Post, the country’s national postal service, gave New Relic a challenge—to find a way to minimize instability in their underlying systems caused by erratic weather and to help manage demand through these periods. They were interested in exploring ways to gather telemetry data from external sources, exposed only through an API layer.
With traditional tools, it might not be possible to instrument endpoints without a suitable agent. But New Relic Flex is an application-agnostic, all-in-one tool you can use to collect metric data from various services, available with the New Relic infrastructure agent. You can run HTTP/HTTPS requests, run shell commands, parse file contents, and extract data from databases using an agentless approach, then send it to New Relic. I’ll show you a step-by-step guide to getting started using the same example seen with Australia Post.
Get started
Let’s start with a simple configuration file to build on throughout this guide.
Prerequisites
Before you begin, you need to set up three things:
- A New Relic Account: If you don’t already have one, signing up is free and includes up to 100GB of free data ingest.
- The New Relic infrastructure agent (with Flex) or the agentless Flex binary (
nri-flexcan run in multiple modes and is bundled within the infrastructure agent by default, but you can download and run it as a standalone tool if needed) - The Weather API of your choice. We'll be using Open Meteo for this guide.
For other examples of New Relic Flex, take a look at the Flex Repository.
Step 1: Set up Flex (two options)
Use one of these two options, either the New Relic Flex binary or New Relic infrastructure agent.
Install the New Relic Flex binary
- Download the latest release of
nri-flexfrom our releases repository. - Extract
nri-flexand ensure it can be executed. Ensure Flex is an allowed application in your server security and privacy settings.
Install the New Relic infrastructure agent
- Log in to New Relic.
- Select Add your data.
- Search for New Relic Infrastructure Agent.
- Install the infrastructure agent for your operating system.
- To access the Flex binary from the infrastructure agent, find it located in either:
- Linux:
/var/db/newrelic-infra/newrelic-integrations/bin - Windows:
C:\Program Files\New Relic\newrelic-infra\newrelic-integrations
- Linux:
In case of any issues, these troubleshooting commands are useful for debugging:
- Linux:
/var/db/newrelic-infra/newrelic-integrations/bin/nri-flex -help - Windows:
C:\Program Files\New Relic\newrelic-infra\newrelic-integrations\nri-flex.exe -help
Step 2: Set up a weather configuration
Our examples here in this blog post use Flex from the infrastructure Linux agent as an example, but these instructions should transfer to whatever you are using.
1. Create a new file in a directory of your choice, called weather.yml. We’ll be testing the config from /var/db/newrelic-infra/newrelic-integrations/bin/.
2. Open weather.yml in your chosen code editor and copy the YAML below. We’ll use this as our base to build on.
---
integrations:
- name: nri-flex
interval: 30s
config:
name: weather
variable_store:
long: 151.21
lat: -33.87
custom_attributes:
city: Sydney
global:
base_url: https://api.open-meteo.com/v1/forecast?
apis:
- event_type: WeatherLocationExample
url: latitude=${var:lat}&longitude=${var:long}¤t_weather=true
add_attribute:
city: SydneyNote that you can change the longitude and latitude to a city near you. You can use What's my GPS to determine your location.
3. Next, run this command: ./nri-flex -config_path weather.yml -pretty -verbose.
(We're in the /var/db/newrelic-infra/newrelic-integrations/bin directory.)
This output is local to your machine and will not be sent to New Relic.
This configuration targets the current weather and records the city as a custom attribute.
Now let's build on this config and look at the hourly weather, precipitation, rain, and showers.
---
integrations:
- name: nri-flex
interval: 3600s
config:
name: weather
variable_store:
long: 151.21
lat: -33.87
custom_attributes:
city: Sydney
lat: ${var:lat}
long: ${var:long}
global:
base_url: https://api.open-meteo.com/v1/forecast?
apis:
- event_type: WeatherLocationExample
url: latitude=${var:lat}&longitude=${var:long}&hourly=temperature_2m&hourly=precipitation&hourly=rain&hourly=showers
jq: '.hourly | [."time", ."temperature_2m",."precipitation",."rain",."showers"] | transpose | map({timestamp: .[0], temp: .[1], precipitation_mm: .[2], rain_mm: .[3],showers_mm: .[4]})'
timestamp_conversion:
timestamp: TIMESTAMP::2006-01-02T15:04For more information on Flex functions, see the documentation in GitHub.
4. Finally, copy the new config and run the debug command: ./nri-flex -config_path weather.yml -pretty -verbose
Step 3: Sending Data to New Relic
When you’re happy with the config, you can move it to the integrations directory of the infrastructure agent.
1. Move the config to the integrations directory. The infrastructure agent will perform a hot reload and send data to New Relic.
-
Linux:
/etc/newrelic-infra/integrations.d -
Windows:
C:\Program Files\New Relic\newrelic-infra\integrations.d\
2. Log in to New Relic. Select Query Builder from the left menu bar and select the Query Builder tab. Enter this NRQL:
SELECT temp, rain_mm, precipitation_mm, showers_mm, timestamp FROM WeatherLocationExample SINCE 1 day ago LIMIT MAX
You should see this output:
As you can see, my selected location had no rain/precipitation/showers in the last 24 hours.
Once the custom telemetry is in New Relic, there are limitless ways to use the data. For example, Australia Post used precipitation amounts to predict parcel volumes. Their theory was that people are more likely to shop online while at home compared to being out in the real world. They also used weather metrics to predict traffic, or the volume of cars on the road, and make data-driven decisions to scale up their drivers and infrastructure services to accommodate an increase in shopping.
The next dashboard is an example that correlates their existing telemetry with the weather data collected by New Relic Flex.
Conclusion
New Relic Flex is a powerful tool for custom telemetry instrumentation. With its simple configuration and flexibility, the possibilities for gathering and analyzing telemetry data are endless. As seen with previous examples, Australia Post collected custom telemetry data in just a few minutes, highlighting the efficiency and effectiveness of this tool.
Next steps
By leveraging the power of New Relic and following the same steps as we used with Australia Post, you can gain a deeper understanding of how external factors impact your own applications. Whether you're a developer, system administrator, or business owner, custom telemetry can provide valuable insights to help you make informed decisions.
Check out the Flex repository on GitHub for ready-to-go configuration files
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
