The starting point for data driven sports graphics has to be to provide the viewer with information that is integral to the action. The most obvious and perhaps most ubiquitous example of this is time. Pretty much all sports have a requirement for either time elapsed or time remaining, and ever since the Innsbruck Winter Olympics of 1964, time has been a ‘screen’, a live graphics overlay on our TV screens. This is credited as the first public example of a data connection between the apparatus used by the officials at the event, being fed into the broadcast infrastructure of the day. Instantaneously sports broadcast production became more immersive for the viewer and sports production has never looked back.

Fast forward 61 years and sports graphics remains on a relentless innovation curve. On screen data essentials like time are joined by statistical data and graphics that enhance entertainment and analysis across the entire spectrum of sports. The vast majority draw upon data feeds from manual input and sports infrastructure and/or utilize video analysis & visualization tools powered by ML.

Player & Ball Tracking

The science of object tracking has applications in a huge array of industries so is an area of continuous technological advancement where broadcast benefits from the investment of others. Medical imaging, robotics and military applications are obvious key drivers of innovation.  In sports, investment in object tracking systems is fuelled more by coaching, injury prediction and adjudication than broadcast, but broadcast is making very good use of the data.

There are essentially three technical approaches used for object tracking in live sports graphics production: wearable sensor tracking, the use of depth camera arrays, and video analysis – these can be combined.

Body worn sensor tracking is fairly straightforward; the player or athlete wears one or more sensors mounted at strategic positions across the body. The position of these wearable wireless devices is captured by an RF or GPS based tracking system that charts the sensor movements in three-dimensional space. Often the devices are also able to monitor physiological data like heart rate and transmit this as a data feed. Coaching systems have been relying on this approach for many years to deliver extremely accurate performance analysis during training. Historically there have been some concerns that these devices can interfere with the performance of the athlete. Those concerns seem to have receded as newer, more discrete technologies have evolved, so in recent years almost all sports have started to allow the use of such devices during competitive matches or races – but they are there primarily for coaching and performance management. From a broadcast production perspective, data can be supplied from the coaching system as a data end point to a broadcast graphics system but it is not typical.

During training sessions, it is more feasible to mount multiple body worn sensors at strategic points across the body, usually at joint positions, to give a more detailed set of positional data – which is usually used to draw up a skeleton. This skeletal tracking allows coaches to help players improve body movements. This is where skeletal tracking evolved but it is not typically achieved in broadcast using sensors.

Depth cameras gauge the vector position and distance of an object relative to the camera(s). Simple stereo depth cameras are a commodity item and are widely used in AR/VR applications, gaming etc. Stereo depth cameras have two lenses in a single enclosure and compare the two images to calculate position. Using the same fundamental technique of comparing multiple images taken from alternate angles to triangulate a position in space - placing an array of high resolution, often high-speed cameras in a sports arena can provide effective ball and player tracking for the entire field of play. Software analyzes the multiple feeds to triangulate position and thus movement in real time.

The technology was introduced in 2000. It has steadily improved in sophistication and performance and has evolved to become a core element of adjudication in many sports. It is commonly stated that this array system can be accurate to within 1mm, so it is unsurprising that with the application of ML, this approach has become capable of producing extremely detailed motion tracking. The approach can require up to 10 cameras to achieve coverage of a football field so either needs to be permanently deployed in an arena (usually within the stadium roof) or requires considerable additional pre-game installation effort.

ML based software analysis of the various camera feeds to generate tracking data is an intrinsic part of camera array based tracking. It has been through several evolutionary iterations. First generation tools used only Center of Mass (CoM) analysis. In any given frame of video, pixel analysis can produce a single central point that would be the center of gravity for an object. As image resolution has improved, alongside more powerful real time ML based image analysis, it has become possible to track more positions across the body to achieve skeletal positioning purely based on video analysis, without the need for sensors.

Where video analysis may become unstuck is when players collide. Drawing together two data sources; sensors & video analysis, could help resolve that challenge.

Camera array tracking has become central to a number of broadcast workflows in various ways. The position and detailed motion data from such systems can be passed on as live or delayed data feeds via data endpoints available either locally or within public cloud servers. These data feeds are used by some live graphics production software systems to generate visualizations. Alongside ultra precise object tracking, camera array tracking can produce ultra precise multi-camera synchronization which some replay systems might use as a method of synchronization for multi angle review. Some stadium digital advertising hoardings use it to synchronize image replacement for localization of advertising.

Not all visualization systems take this approach however. Some graphics systems perform their own ML powered image analysis to track objects without the need for sensors or camera arrays. This area of video analysis is developing fast with some systems now able to track up to 29 body points in real time. Machine Learning has been pivotal in this evolution.

Not directly part of sports graphics but relevant in a discussion of image based object tracking is that various vendors have developed systems that use object tracking to control the position of PTZ cameras. Selection of an object within the frame allows the camera to stay framed and focused on the object as long as it remains within its field of view.

A conversation with Thom Stevens, Senior Director Of Solutions Engineering at Deltatre brings a little real world clarity with his observations of the market as to whether the object tracking theory works in practice across all sports. “Since we’ve had computer vision in any meaningful way, the idea for player tracking has been that you install specialist cameras near the field of play, up in the gantry etc, and you assign a value to the athletes.”

For some sports, Stevens confirms, “Player tracking itself is no longer the primary technical hurdle – in fact, soccer has benefitted from reliable tracking data for years, and cricket’s largely non-contact nature makes it even easier to implement. But when it comes to high-contact sports like American football or rugby, the implementation becomes more challenging.

Modern systems, Stevens goes on, report more than just position. “The technology has progressed beyond simple object tracking. Today’s systems use advanced skeletal and pose estimation techniques to capture detailed movement data for each individual athlete, unlocking a far richer layer of performance analytics. At the same time, the data pipeline itself is becoming more sophisticated – delivering greater volume, accuracy, and real-time insights.”

The current ambition we’re observing, Stevens says, involves minimizing the installation. “The real challenge now for the industry isn’t whether tracking works, but how to scale it without significant infrastructure requirements. Traditionally, the tracking vendors that we partner with have needed to install dedicated hardware at each venue – sometimes this means rigging in advance or keeping equipment in place all season. That approach can have significant costs across a league or season. The real commercial breakthrough lies in extracting rich, reliable data directly from the broadcast feed itself.”

Happily, required R&D work is happening across the board. “Broadcasting is a major endpoint for the technology and a driver of innovation in this market,” Stevens states, “while coaching and betting are also behind notable developments as they aim to improve team performance and evolve the overall data landscape.”

Machine Data

Time is our first example of machine data but it did not take long for it to be joined by data feeds from the adjudication and referee systems for various sports. Elapsed or remaining game or race time, scores, penalty notifications, and dozens more can be made available as data end points for graphics systems.

There is one global sport that has pioneered the use of machine data – Formula 1 motor racing. The sport itself started gathering telemetry data from cars during pit stops in the 1980s, it went real time and pioneered software performance modeling in the 1990’s. The first broadcasts to use some of this data for visualizations were in 2002, and in 2018 the sophistication of these visualizations took a leap forward with the introduction of new cloud-based processing systems that helped create the highly immersive viewer experience we see today, with real time race positions, intervals behind leader, track position markers, lap times per driver, tire data, pit timings, individual car speed data etc.

F1’s capture and modelling of this data is also cited by some as a pioneer in fuelling the gaming sector which drew heavily on it to create game play that closely resembled the viewing experience. It is a synergy between broadcast and gaming that is now visibile in a number of sports, with an ever deeper cross fertilization of very high resolution imaging, data driven graphics, and audio production creating unified immersive viewer experiences that have powered the rise of sporting super-brands. It is another area where ML holds significant promise to power new rich imaging experiences that blend cinematography, generative graphic imaging and VR/AR techniques to bring the viewer closer to the heart of the action. It is liable to be heavily reliant on secure, stable data feeds.

Human Data

In this machine driven age it is slightly counter-intuitive but much of the data in live sports production is still generated by humans. As we see later in this series, AI is catching up, but humans still have the edge when it comes to real time scoring - even when machine generated data and commercial data feeds are available, sports graphics operators are still faster and more reliable. Some of the leading commercial sports data feeds are still using highly skilled humans to log game event data.

Statistical Analysis

Machine Learning has also been pivotal in turning raw object tracking data, alongside machine data and human data into statistical and performance data to enrich the viewer experience across most sports. Each sport has evolved its own collection of expected statistic visualizations. Once the data sources are available it is custom statistical analysis software that converts it into meaningful data streams to feed broadcast graphics systems. Whether this happens as part of the data capture system or as an intermedia step before it hits the graphics system varies.