About the Live Sports Production series:

This is the fourth of a 12 article series which uses a round table style format to explore the technology of live sports production with some of the industry’s leading system designers. It is a fascinating insight into what is being done every day around the world.

This series is based on a set of interviews with six individuals who are the senior system architects within three of the most respected organizations in broadcast: Broadcast Solutions, Diversified and NEP. We cannot thank these individuals enough for giving us their time and sharing their immense wisdom and experience in the delicate art of designing and deploying broadcast production infrastructure. These interviews spanned many aspects of live sports production and their comments will be serialized across this whole series.

One of the most fascinating aspects of discussing where the industry is in its technological and workflow evolution is that there is no one single answer to most of the key talking points – each of the organizations supports a diverse portfolio of clients across a range of geographical locations – and the approach taken is always carefully tailored to specific rights holder and broadcaster requirements.

As we get into our conversation, it is important to understand that we have three quite different businesses represented; Broadcast Solutions build OB Vehicles and do Systems Integration primarily in Europe and the Middle East, NEP are a global production Service Provider, Diversified is a Systems Integrator with the representative taking part here based in the USA.

Bear in mind the obvious differences between designing and building individual systems for others to use and NEP’s capacity to build systems at scale for their own teams to use.

The transition from hardware centric infrastructure towards software centric production tools is enabled by IP. The scalability and versatility of IP production networks is the foundation stone of modern system design, but as we will see here, SDI is showing its mettle as a more suitable technology in some systems and hybrid approaches are a very popular way forward. We will also see that the journey into IP has not entirely been a smooth one.

Dan Turk. Chief Technology Officer, NEP Americas: “I would say we are on what I call Gen 3 of IP. Our IP journey started with needing to build a facility that had to fit within a truck in terms of weight and space, but to meet a demand that was bigger than we could achieve with SDI. At the time we built our first IP systems, it was about scale. We needed capacity that might have taken four SDI routers – which in a large studio facility would have been ok, but not in a mobile unit. Our initial IP approach needed proprietary SDI-IP gateways on everything, but it enabled us to meet the demands and make it fit in a mobile environment. This pushed us into IP which was good. That’s what I would call Gen 1. We got into that before SMPTE 2110 was ratified, so it was very much proprietary.”

“Then 2110 was ratified and there was more of a standard. That’s what I would refer to as Gen 2 IP. That was still about scale, but it was also about power saving and weight saving. For us in the mobile world, to take the system and put pieces where they are used and interconnect with cores, really helped distribute our weight amongst the trucks, and that truly makes a difference.”

“In Gen 2, a lot of times the SFP’s were the limitation. We had more FPGA processing capacity than we could use because we reached the maximum bandwidth of 4D SFP. Gradually, SFP bandwidth went from 1.5 gig to 3 gig and then 10 gig, so we’ve been able to evolve and it has really allowed us to be more flexible.”

“We’re now at what I call Gen 3, which is obviously a more mature 2110. In terms of capacity with 100Base-ZX SFP capable of 100 gig, you’re back to the networking having enough capacity to handle the processing requirements. Bandwidth has impacted redundancy at various stages on the journey. There were instances where, to keep within budgets, it was more practical to not do 2022-7 because we needed more processing power than the network infrastructure would allow. Now, a 100 gig network can do more than the FPGA and the processing, therefore dash-seven is almost free, although obviously you double the switch count.”

Broadcast Bridge: The capabilities and capacity of the network is only part of the story… interoperability and a change in engineering culture have been milestones too.

Rainer Kampe. CTO at Broadcast Solutions: “It’s been painful. It is still painful. But IP is not the only reason why transition is painful. The problem is that our industry is moving away from bespoke hardware, (even a BNC connector is bespoke hardware for transporting signals down a coax cable), to IP or IT driven equipment - and software is rarely ready. Software is usually 95% ready at release and then you get updates. You have that with windows and most other software as well. This is one of the big pain points at the moment for us as a system integrator.  It is the same with bespoke hardware, software and IT off the shelf equipment.  But the pain is much bigger with the IP world because fault finding is much more complicated. If you have a 100 gig interface, and 100 streams or more traveling on it and one of these streams is broken, where is the problem? Everybody says it’s the network. Typically, it’s not the network.”

Pierre Mestrez. Senior Director Software & Services at Broadcast Solutions: “Whilst there is SMPTE and NMOS and VSF and other standardization initiatives, all the vendors still read this standardization and implement it their own way. I don’t blame the standardization organizations at all, but standards are usually quite blurry in some details and it is really in these details that issues occur and we are needed most.”

Rainer Kampe. Broadcast Solutions: “In all these standardizations the pain points have been left out and the pain point occurs when the environment changes. For example, you are in a venue and you change signals, you change your workflow, etc.. This is not dealt with in the standards because the standards don’t deal with final setup of equipment or of signal flows. Standards do not define workflows. NMOS doesn’t tell you anything about what to do if somebody switches his camera from UHD to HD or the other way around. Before IP you had one signal, you plugged it into a router and you had it, you could see it because you routed it to ten monitors and 20 recorders and so to 30 destinations. If you do that in IP and unplug and change the signal, it won’t appear in these 30 destinations, you need to resubscribe for all 30 destinations. Mechanisms for this are not defined in NMOS or in SMPTE.”

Pierre Mestrez. Broadcast Solutions: “This IT IP involvement in the broadcast industry was supposed or is still supposed to add a level of flexibility, of repurpose-ability, and the reality is in this regard, this is where we struggled. It will serve and it does because we make it happen but where it’s supposed to bring new flexibility, it brings headaches as well.”

Antti Laurila. Chief Strategy Officer at Broadcast Solutions: “The broadcasters are not prepared for that. The decision makers look at how they can save and do more with the same kit and get this flexibility but then in reality the operators are not trained, they have good intentions to train themselves but there isn’t any training available from manufacturers or industry in general. They only learn by doing and then the first months are always painful because you get the new technology, new systems, you only learn from your mistakes and then those mistakes are kind of costly and causing a lot of pain for everyone involved in the project.”

Broadcast Bridge: The issues of standards and learning curves are echoed by others:

Dan Turk. NEP Americas.: “Although a lot of interoperability has been done over the years and there are standards, there are a lot of different ways people interpret the standard. There was also a huge learning curve for the guys on the road. For engineers that were used to being able to troubleshoot, IP is new and different and a lot of it is command line. It’s not something you can just figure out, it’s something you have to learn. That has been a large hurdle. But audio is the hardest part in 2110 because there are 10 different ways you can do it, for example with two-channel, four-channel, six-channel, or 16-channel. With different manufacturers, how you setup is different, and when you want to tie all this together and take advantage of 2110 then that becomes a challenge.”

Broadcast Bridge: Are you building hybrid or 100% IP – where are we right now in that transition?

Rainer Kampe. Broadcast Solutions: “We favor hybrid systems where you have the flexibility and scalability of IP in the core system. Here an IP core is unrivaled by SDI, because an SDI router is defined with inputs and outputs that you cannot exceed, but in IP you can and that’s the point where you can grow. Around that core we deploy islands. If it is a kind of static environment with skilled operators we go IP, if it’s an ever-changing environment where you have unskilled operators, we attach an SDI island via gateways to that IP core. That can also be done with NDI or Dante or whatever. The hybrid approach is a scalable IP core with a network of different islands around it. This is what we say to customers who have heard all the buzz words and want to go big IP.  Don’t always look to the huge IP thing. See it as clusters.”

John Guntenaar, Chief Technology Officer, NEP Europe: “The majority of our new installations are IP and the tipping point in price makes IP attractive. But there’s still a place for SDI. There are still smaller locations maybe with a small studio and a few cameras where SDI just makes it easy and makes sense. You can have less experienced engineers who aren’t familiar with how to configure switches, etc. Anyone can repair baseband cable fairly easily, so right now for that small fixed installation, it makes sense.”

“If we’re interfacing our systems with the outside world, very often, it’s still based on SDI, because we don’t just connect networks to each other for various reasons. We see that mostly the tail boards on trucks have SDI and if I want to feed a local monitor then very often it’s easier and cheaper to do that in SDI compared to IP, with all of the additional devices that you need in order to do that.”

Patrick Daly. VP Media Innovations at Diversified: “The IP protocols that are in play are really a foundational enablement. Whether we’re talking about 2110 or maybe IPMX in a mixed enterprise environment, or even NDI – we do a fair amount of NDI stuff for some of our smaller projects, the skills gap is closing. I’m seeing more and more expertise in my client base when it comes to network technology, IT technology. By and large Diversified is mostly 2110 deployments. It’s arguable in my mind that everything needs to be on a 2110 fabric. I think there are still plenty of use cases for islands of SDI. But it’s only becoming easier to build and operate IP centric plants. Doing so allows you to begin utilizing the capabilities of pure software on COTS without tying yourself by binding a particular application to a particular piece of hardware because it’s the one with the right vendor specific card and the SDI I/O.” 

“That’s extremely helpful. Especially in my high-cost real estate markets, I’m seeing a lot of desire just to move that tech center out of the facility, even if it’s not into a hyper scaler, even if it’s across the river into a data center somewhere. A lot of desire to free up the footprint of the data center and then the infrastructure requirements that come along with it, like a whole room for batteries and additional service from the grid. It’s an expense. And if you’re a large enough broadcaster you often have regional assets and it starts to make a whole lot of sense to put those all in the same space.”

Broadcast Bridge: Trucks are relatively isolated production environments, but have the challenges of ensuring separation and protection of assets within multi-studio production centers been resolved, especially with remote production at several sites simultaneously?

John Guntenaar, NEP Europe: “We designed our facilities so that Pepsi-Cola and Coca-Cola can work in the same building without knowing of each other, for example. If you look at the contribution side from the stadiums, that is very often point to point, and from there it lives inside the data center and inside the data center we typically use SDN, Software Defined Networking, in order to make sure that we can control what video and audio flows are going where. This is for security purposes, so that someone cannot just type in an IP address and get someone else’s signal. That’s where the SDN kicks in. We have a couple of other mechanisms, one example being where we randomize IP addresses to make sure that we aren’t leaking signals.”

Dan Turk, NEP Ameriacs: “That is a perfect example of where IT networking is merging with broadcast engineering. If you look at the way broadcast engineers worked in the past, everything was a big layer2 network. You could separate things with VLANs, but that was really only by device. It’s all just there, everyone can see everything, which makes life easier for discovery but is not really the proper way to do it and leaves you more vulnerable. In 2110, and the way the standards work, things are done with a layer3 network where everything is small, individual, and kind of port to device. This way gives you control because the device has to go into the appropriate port or it does not work. It’s significantly more work on the front end, but it gives you so much more control and protection for the system overall.”