Sportradar Transforms Engineering With a Holistic Observability Strategy

Whether it’s Formula 1 racing or world snooker championships, American baseball or European football, ice hockey, basketball, or one of more than dozens of other sports, the fan experience just wouldn’t be the same without Sportradar powering it. As the global leader in sports data intelligence and technology, Sportradar monitors and delivers insights from more than 400,000 matches annually across 60 sports categories. 

The company is the official partner of the NBA, NHL, and NASCAR, as well as FIFA and UEFA and is also the only provider entrusted to work with the U.S. sports leagues in an official capacity to distribute sports data (NBA and NHL) around the world for betting purposes. Sportradar’s sports betting, entertainment, and integrity products and services are relied on by more than 1,000 companies in 80 countries. 

‘Our technology is behind every major sporting federation and organisation around the world’, says Scott Dunbar, global head of DevOps engineering at Sportradar. ‘We are first and foremost a software development organisation, adopting bleeding-edge technology such as artificial intelligence and computer vision to deliver the best sports technology and products possible.’

When Dunbar and Sportradar’s chief technology officer Ben Burdsall joined the company, every engineering team was working completely independently. ‘It was the biggest start-up I’d ever seen, with teams siloed away from each other’, says Dunbar. `We had pockets of excellence, but there was very little sharing, collaboration, or alignment across teams.’ 

Burdsall’s vision was to transform the organisation of the engineering group into the Spotify scaled agile model. ‘We set out to create what we call the ‘Sportify’ model, where engineering capacity is organised into tribes, squads, chapters, and guilds’, says Dunbar. ‘The ultimate objective is to enable cross-functional teams, create transparency around roles and responsibilities, and align our engineering function around a clear strategy.’   

Today, that vision has been achieved, with more than 500 engineers organised into 17 tribes. Each tribe is a team of engineering capacity focused on one product portfolio. DevOps specialists are injected into tribes and squads to complete the cross-functional teams. 

‘As DevOps engineers, we focus on the software delivery lifecycle challenge of observability,’ says Dunbar. ‘We define observability as the creation of a system where we can debug unknown unknowns. Our team injects that skillset and passion for using data to understand the health of our applications within our tribes.’ 

One of the company’s engineering principles that remains in the new organisational structure is the autonomy of its teams. ‘We believe that our squads and tribes should be free to choose the technology or tooling they’re comfortable with and believe is best for what they need’, says Dunbar. ‘We’ve put a governance and alignment layer in place that enables their autonomy while ensuring consistency in exposing key metrics. Obviously, New Relic One plays a very large role in helping us do this.’  

When Dunbar first came to Sportradar, he found that teams were using a broad array of different monitoring tools and approaches to understanding system health. ‘There was very little consistency and sparse adoption of service level metrics, key performance indicators, and measurement of service level indicators against a service level objective’, he says.            

Now with New Relic One, teams can truly become autonomous while tracking and reporting on a consistent set of metrics, providing the company with a holistic view of its complex portfolio of products and services. ‘New Relic One’s flexibility and support of open source means that our teams can continue using the open telemetry tools that they know but surface key metrics to a single pane of glass’, says Dunbar. `We now have a consistent view of service health across our entire estate of more than 200 business services.´

For every technical service, Sportradar expects its teams to capture availability, reliability, and performance metrics against a service level objective (SLO). ‘We then bubble up technical services into a business service, compare service level indicators (SLIs) against our SLOs, and then guide the business on reliable service level agreements (SLAs) for that particular business service,’ says Dunbar. ‘We’re delivering against a 99.9% availability SLO for all our business services.’ 

Programmability in New Relic One gives Dunbar and the engineering teams the ability to customise how they track and calculate metrics. ‘For us, it was critical to get insight into time-based availability, where availability is determined by the duration of a specified condition in the technical service’, he says. ‘We created a custom observability app in New Relic One that pulls data from New Relic Synthetics and processes it to give us a complete view of time-based availability for every system.’

New Relic One has also helped the engineering organisation measure and improve release quality while increasing the frequency of releases by 15%. ‘New Relic One has helped us change the culture in terms of how we understand performance and test our systems’, says Dunbar. ‘With New Relic One as an integral part of the delivery pipeline for testing changes, it acts as a quality gate to help us capture issues before we ship to production.’

For Sportradar, low-latency data exchange between a massive web of interconnected systems is absolutely imperative. New Relic One is essential for helping the company understand and improve data latency using distributed tracing.   

‘We collect millions of data points from literally thousands of data sources in what I call the head of the snake’, says Dunbar. ‘The long tail of the snake is where a variety of our services such as anti-match-fixing and odds calculations consume the data.’ 

According to Dunbar, ‘New Relic One’s support for open telemetry and feature-rich APIs are what allows us to use distributed tracing to understand the latencies and ingest time from the data sources at the head of the snake all the way down to the tail. I don't think any other tool could adapt to the sheer variety of products and services within our portfolio and still be able to surface a trace, from the root of the span down to every consumer of that data point.’