Monitoring a home solar array with New Relic One

In 2020, we purchased an 8 kW Tesla solar system with 27 kWh of battery backup for our house. The system has an unofficial but widely used API, and I decided to try monitoring my solar array with New Relic One. Tesla also has an app that can display some of this, and it’s pretty slick.

I wanted to do things like monitor the battery level and send an alert if it got too low or too high, keep historical data, run liveness probes, and generally get my hands dirty with it. At first, I figured I’d have a script pull information from the gateway and dump it all into a CSV and dig into it with pandas or Excel. There’s also at least one project that does a lot of what I want right out of the box. But then I realized that this is infrastructure and I work for a company that monitors infrastructure. Plus, I’m a network guy, not a developer or DevOps engineer, so I wanted a service I didn’t have to worry about operating.

That’s when I decided to monitor the solar array with New Relic One. In this post, I’ll show you how I did it and provide an example you can use with your own array. While New Relic’s observability tools are designed for monitoring web applications, New Relic One can work with data from any source as long as that data looks like a metric, event, log, or trace.

There are four components to the solar system:

• solar panels
• inverter
• gateway
• batteries

The panels feed the house and the batteries when there is sunshine. When there’s not enough sun, the batteries feed the house, and when the batteries are empty, the house gets its power from the grid. The system will eventually be able to feed power back into the grid, which means I can sell the excess power I generate back to the utility company. Here’s a block diagram, more or less.

The wiring is more complicated than that, particularly the telephone pole to inverter to house flow. But that’s the basic idea.

When the sun’s shining, power flows to the inverter, which powers my house and banks the excess power in the batteries.

When the sun isn’t shining, the batteries feed the house.

The gateway is the brains of the operation, and it has an unsupported but widely used API with an unofficial Python library.

The examples below are iPython showing the bits that are most worth discussing. There’s a repo where this is all rolled together into a script which can be run as a systemd service or from the CLI.

Polling the API

Pulling data from the gateway API is simple enough.

This gives me a Powerwall object and a meters object. I can get things like the total system charge percentage:

Meters are how you look at power flow through the system. You can get them at four points: solar, load (consumed by the house), site (to and from the grid), and battery. Here’s one:

Let’s look at how to get this data into New Relic One and what to do with it.

Pushing data to New Relic One

New Relic One takes a number of data types: metrics, events, logs, and traces. There are a number of ways to send data into New Relic One, but the one I went with is a simple requests-based submission using the Metric API. For this script, I used the Metric API to report this data as metrics, which have 13-month retention. There’s also a Telemetry SDK with a Python library and maybe I’ll use that next time I write something.

The data took quite a bit of massaging but in the end here’s the block I send to New Relic One:

As you can see, this is a dictionary with two top-level items: common and metrics. common is data that is, well, common to each metric, and the metrics themselves are dictionaries of name/type/value.  My metric type is always gauge, but there are others.

Creating a custom dashboard in New Relic

Once I could push data to New Relic, I turned my attention to dashboards. I have three kinds of data to work with:

1. Data from the solar API. This is the bulk of what I send and is mostly real-time power generation, consumption data, and the battery charge level as well as some historical data on the total amount of power to and from various points in the system. The data isn’t terribly accurate so the numbers don’t always add up, but they’re pretty decent.
2. Data I get from the free API at OpenWeather: estimated cloud coverage percentage, sunrise and sunset times, temperature, and a string description of the weather so I can avoid looking out the window that’s literally right behind my monitor and directly in my line of sight.
3. Data I munge on my server before sending it over to New Relic. This is stuff like whether the sun is up, which is useful if I want to make any sort of predictions about how much power I should be drawing on a given day. If it’s before sunrise or after sunset, the answer is none.

I might also add Solcast to see if I can get daily solar radiation data. One big advantage of New Relic One over the Tesla app is that I can pull in data from other sources. Maybe I’ll get a sense monitor and see how that works.

Here’s what it all looks like.

This is cool.  Do you have any idea how long it would take me to make something like this out of python, pandas, and matplotlib? A while. And it would look like garbage. New Relic One is much, much better.

I find myself staring at the dashboard like it’s a new baby, watching it breathe and marveling that it exists.  It just works. It’s really nice to have this data and be able to get to it anywhere without having to run a container on a server in my house and port-forward to it or VPN into my home network. 

I’ve also got an alert when the batteries drop down below their reserve percentage, which would indicate that my house has lost power and the batteries are running low. New Relic has a whole lot of things I could fiddle with.

Conclusion

Now that I have this dashboard I can’t see doing without it. The power it gives me to query and analyze my own data, and to do it using a set of proper tools, is really refreshing. I have visions of trying to build predictions to see how much power I might draw given a particular time of year and amount of cloud coverage. I could set up alerts that trigger if I draw no power when I think I should, or if something goes weird with the gateway (like if it starts telling me I’m exporting power from my house or to the solar array).

I could see a future where I dump all the other stuff from my home devices into New Relic One and build myself a panopticon. I have a lot of IoT (Internet of Things) devices in my house, as I think most of us do, such as smoke detectors, a fairly robust Ubiquiti wi-fi setup, a Raspberry Pi that does DNS and DHCP and Pi-hole, a printer, a smobot, a few heat pumps, a TV, a cable modem, a NAS, a pile of Sonos gear, and probably some other stuff I’ve forgotten. Oh, and a dozen devices or more: laptops and phones and tablets for the family.  And I don’t have a dozen people in my family.  (Geez, that’s a whole lot of stuff. it’s stunning how IoT just invades your house.)  

I don’t think most of those have APIs, which of course means my ability to pull data from them is pretty much zero, but it would be really, really neat to build a whole-house dashboard I could monitor from anywhere. As long as I can get data from them, New Relic One would be a great place to send it.

I also have a couple of small personal websites that use New Relic synthetics (free account for the win!) to monitor and Cloudflare—also a free account—to run. And the most exciting to a network geek—I also have a  home router that can export IPFIX, so I can use New Relic network performance monitoring.  But that’ll be a whole separate blog post.