Monitoring Fantasy Football Using New Relic Insights and AWS Lambda Integration

The New Relic Digital Intelligence Platform helps modern IT, development, and operations teams harness the power of their data to improve the performance of their applications, infrastructure, and business. But what about using this power for more nefarious purposes, like monitoring a fantasy football league?

Having just recently started at New Relic, I wanted to play around with the tools on my own to get to know them better, so I decided to use New Relic Insights and AWS Lambda (paired with New Relic AWS Lambda monitoring integration) to gain some visibility into the various bits of data being pushed around in my fantasy football league. (Note: I’m new to fantasy football, too, but not as new as I may appear.)

Extracting the Yahoo data

Here’s a look at my fantasy roster in the Yahoo Fantasy Football app. This is some of the initial data that I wanted to extract and add to Insights.

To extract this data from the Yahoo Fantasy Sports API, available from their developer network, I wrote a pretty simple Python script to push the data directly to the Insights API. I just had to register an Insert Key for security and provide an Insights URL for where I could send the data.

Here’s the Python script I wrote:

import requests

import xml.etree.ElementTree

from requests_oauthlib import OAuth1



url_settings_in = 'https://fantasysports.yahooapis.com/fantasy/v2/league/nfl.l.261594/settings'

url_players_in = 'https://fantasysports.yahooapis.com/fantasy/v2/team/371.l.261594.t.12/roster/players/stats'

url_standings_in = 'https://fantasysports.yahooapis.com/fantasy/v2/league/nfl.l.261594/standings'



url_out = 'https://insights-collector.newrelic.com/v1/accounts/1652861/events'



auth_in = OAuth1('dg7tl….','66f47….')

auth_out = 'QA3K1n….'

headers_out = {"Content-Type": "application/json", "X-Insert-Key": auth_out}

ns = {'v2_ns': 'http://fantasysports.yahooapis.com/fantasy/v2/base.rng'}



def yahoo_to_newrelic(event,context):

    # Get the league settings data from yahoo

    req_settings = requests.get(url_settings_in, auth=auth_in) 

    settings_tree = xml.etree.ElementTree.fromstring(req_settings.content)

    settings_dict = {}

    for stat in settings_tree.findall('v2_ns:league/v2_ns:settings/v2_ns:stat_categories/v2_ns:stats/v2_ns:stat',ns):

        stat_id = stat.find('v2_ns:stat_id', ns).text

        display_name =  stat.find('v2_ns:display_name', ns).text

        settings_dict[stat_id] = display_name

    print settings_dict



    # Get the player data from yahoo

    req_players = requests.get(url_players_in, auth=auth_in)

    players_tree = xml.etree.ElementTree.fromstring(req_players.content)

    player_payload = []

    for player in players_tree.findall('*/v2_ns:roster/v2_ns:players/v2_ns:player', ns):

        name = player.find('v2_ns:name/v2_ns:full', ns).text

        bye_week = player.find('v2_ns:bye_weeks/v2_ns:week', ns).text

        display_position = player.find('v2_ns:display_position', ns).text

        week = player.find('v2_ns:player_points/v2_ns:week', ns).text

        total_points = player.find('v2_ns:player_points/v2_ns:total', ns).text

        temp_payload = {"eventType":"ff_player_update", "name": name, "bye_week": bye_week, "display_position": display_position, "week": week, "total_points": total_points}

        for stat in player.findall('v2_ns:player_stats/v2_ns:stats/v2_ns:stat',ns):

            stat_id = stat.find('v2_ns:stat_id',ns).text

            value = stat.find('v2_ns:value',ns).text

            temp_payload[settings_dict[stat_id]] = value

        player_payload.append(temp_payload)

    print player_payload

    

    # Get the standings data

    req_standings = requests.get(url_standings_in, auth=auth_in)

    standings_tree = xml.etree.ElementTree.fromstring(req_standings.content)

    for team in standings_tree.find('v2_ns:league/v2_ns:standings/v2_ns:teams',ns):

        if team[2].text == "Togha na Rogha":

            position = int(team.find('v2_ns:team_standings/v2_ns:rank',ns).text)

            points = float(team.find('v2_ns:team_standings/v2_ns:points_for',ns).text)

            wins = int(team.find('v2_ns:team_standings/v2_ns:outcome_totals/v2_ns:wins',ns).text)

            losses = int(team.find('v2_ns:team_standings/v2_ns:outcome_totals/v2_ns:losses',ns).text)    

     standings_payload = {"eventType":"ff_standings_update", "position": position, "points": points, "wins": wins, "losses": losses}

    print standings_payload

    # Send out player and standings data to New Relic

    r_player_data = requests.post(url_out, headers = headers_out, json = player_payload)     

    r_standings_data = requests.post(url_out, headers = headers_out, json = standings_payload)

if __name__ == "__main__":

    yahoo_to_newrelic("event","context")

Let me explain what’s happening:

1. The script defines the necessary URLs. There are three URLs from the Yahoo API: url_settings_in, url_standings_in, and url_players_in. The URL for the New Relic API is url_out. I also had to specify the authentication needed for both APIs, as well as the headers required for the New Relic Insights API.
2. I grab the  settings for my particular league in the Yahoo API. In my case, the important part of the settings is the stat_id and display_name, which I need to query the Yahoo API for player stat information. By default the stats are represented as a stat_id and a value. To get around this, I created  a Python dictionary to automatically convert the stat_id to the more viewer friendly display_name.
3. The script then queries the Yahoo API for the players on my team. First it gathers information like player name, bye week, position, and points, and then it loops through each stat available for each player. Each player's information is organized into a dictionary along with the “eventType”:”ff_player_update” key/value pair. This key/value pair determines where I’ll find the data in Insights, as it’s the name of the event, so to speak. Each player event is then organized into another dictionary as part of the overall payload that’s sent to Insights.
4. I also gather some general information about my team. The script cycles through each team until it finds mine (“Togha na Rogha”) and can gather information on my wins, losses, points, and overall standing.
5. Once all that data is gathered, I send it to New Relic. This happens in two separate POST requests. The first request sends the data for the payload for the ff_player_update, and the second request sends the payload for the ff_standings_update. Of course, I could send these two payloads in the same request, but I wanted to keep things separate in case I ever need to expand the script to keep an eye on rivals teams.

Triggering data extraction in the cloud

Next, I needed something to run my script, but a full-blown Amazon Elastic Cloud instance to run such a small script seemed like overkill. And that’s when I remembered AWS Lambda, which allows you to execute code or other applications without needing to first provision a server.

To set up the Lambda function, I had to specify some basic information, such as the runtime environment and any handlers or roles. In this case, I’m using the Python 2.7 runtime environment. The handler is app.yahoo_to_newrelic, and the method that Lambda executes within the file is yahoo_to_newrelic(). For the Identity and Access Management (IAM) role, I created a role specifically for the Lambda function that gives it permission to execute my script.

You can add your scripts to AWS Lambda via their text editor or by uploading them in ZIP files, which is what I did. However my script required some extra libraries that weren’t available in the AWS Lambda Python environment, so I had to create a deployment package to upload them to AWS Lambda.

Part of the revolution in serverless services that AWS Lambda offers is the ability to run your code based on triggers, set at whatever interval you want. I set a trigger with Amazon CloudWatch to run my script every 15 minutes.

New Relic Insights and AWS Lambda integration

Thanks to the New Relic AWS Lambda monitoring integration, the default dashboard straight away gave me insight into my Lambda function. It showed great information, like how long the function takes to execute, how many times it has run, and if the function encounters any errors.

Checking in on my team

Now that I have the script running, I can log into my Insights account and see the data coming in every 15 minutes. Currently I’m tracking my wins, losses, total points, position in the league, and (as shown below) various stats for individual players:

But the data for my team looks much nicer on this dashboard:

Now I can keep an eye on my team—with three losses and only two wins at the time of this writing, it’s not looking too good. But I'm confident that with the power of New Relic and AWS Lambda, I can turn this season around. There's plenty of football left to play.