It’s easy to see why broadcasters and other content providers are excited about software-based infrastructures. Their potential is so enormous that the future already looks brighter. But the problem with predicting the future is that it’s just so unpredictable.

Riedel’s CEO (Product Division) Jan Eveleens knows this all too well. He’s been here before.

“When it comes to software-based infrastructures, we are still early adopters,” he says. “We should not forget how long it takes for broadcasters to transition fully to IP. I remember saying it was going to take 10 years; I was wrong. It’s probably taking more like 20.”

Progress is not through want of trying. Many vendors already have a foot in this space, often through having initially created proprietary environments and progressively added more and more interoperability capabilities. The formation of the JT-DMF, a collaboration between the AMWA and the EBU to address broader questions around the Dynamic Media Facility (DMF), has only accelerated this, covering not only MXL but also orchestration and timing, as well as how it is all expected to work commercially. It’s new territory for everyone.

The Road To Adoption

But where are we really? Well, that depends on who you are and what you are trying to achieve.

“We’re still near the bottom of the ramp up, but I think there is much more adoption in lower tiers of production, where people need a more cost effective and flexible way of producing but don’t need the guaranteed performance of a high end production workflow,” says Andy Rayner, CTO, Appear. “I head up the music and the tech at my local church and we do a seven camera live broadcast every Sunday morning with a 32 channel, bespoke audio mix, graphics overlay on the lower third and logos. We do it almost entirely with volunteers on open source software and we do it for almost nothing.

“There is a real democratization of content creation at these lower tiers, but what most end users are paying for with the high end content is not just the nuanced technical capability; they are also paying to mitigate risk. If you look at the money that’s poured into high end broadcast, a lot of it is risk aversion. It needs to be on air, it needs support, and users need to know that the product is going to be onward-developed as they need it. All of those things are as critical as the bare capabilities of the equipment.”

Meanwhile, at the other end of the scale, broadcasters like the Canadian Broadcasting Corporation are already making use of initiatives like the DMF, including at this year’s Winter Olympic Games.

“We had the Canadian rights for the Olympics, and we built a Dynamic Media Facility – all software – with a simple web-based user interface for commentary workflows that a non-technical person doing commentary can operate themselves,” says CBC’s Director, Global Collaborations / Innovation Hub Felix Poulin.

“With the Olympics you have access to a huge volume of content running in parallel; some go to broadcast, some go online. So when there’s a non-prime-time sport competing against a bigger draw, the bigger draw will go to television, but we can put the other sport online with a much simpler infrastructure.  It’s a perfect model for software-based deployment, because you can pay for the licenses for the duration of the Games. If it were hardware, it would be sitting idle for the rest of the time.

“To be clear, this is running without MXL yet. We have MXL in the lab, but with the Olympics we had to limit risk, and it wasn’t sufficiently mature prior to the Games to feel safe deploying it at that scale. But next time, it will almost certainly be included, because several of the components are now MXL-capable.”

It may not be ready for prime time, and although all our panel agree that we’re still on the on-ramp, this kind of implementation is already possible because no-one is starting from scratch. Most vendors have already put the time into R&D.

“The software adoption part, especially for a company like Matrox Video, is already here and it has been for years,” says Matrox Video Product Manager, Daniel Robinson. “Most solutions on the market are now running software on computers; there are very few 100% ASIC or FPGA-based solutions.

“What we’re really talking about now is the next step: breaking down islands where there are boxes connected with SDI cables or via ST 2110 and moving to something more friendly like MXL. Moving from synchronous to asynchronous transport is the key phase we’re in now. It’s not about vendors starting to write software; they’ve been doing that for 20 years. It’s about connecting software applications from different vendors using the most efficient transport.”

In fact, there has never been a lack of desire within the vendor community to innovate – it’s kind of its thing – but according to Lawo CTO Phil Myers, it is the growth in infrastructure and compute power that is now able to support those innovations. It’s Moore’s Law in action.

“Since about 2016 there’s been an explosion in the IT space driven by the advancement of AI,” he says. “Network infrastructure has moved from 10/40G to 25/100G, and it is now moving faster than we can use it. We’ve got 100 and 200 gig in our products and we’re finishing work on 400 gig, but the IT industry is already at 800G and 1.6 terabit. I don’t need to worry about that curve because it’s moving faster than we can keep up with.

“The same is true for GPUs and CPUs. We’ve gone from 16 or 32 core servers to deploying 192 core servers, with 256 and 384 core systems on the way this year. I’m less worried about that curve because we’ve abstracted our software from the hardware and are able to leverage such benefits. We introduced Lawo’s HOME Apps a couple of years ago and they run identically regardless of the underlying server; it’s just that now there’s more compute available.”

The Barriers To Adoption

That’s not to say we don’t have any barriers. Beyond the headline benefits there’s a lot of orchestration and management going on, and as is often the case in broadcast architectures, it’s in the minutiae where the biggest challenges are.

“We still haven’t agreed timing in MXL,” says Robinson (Matrox Video). “We don’t have a way of controlling or discovering MXL flows; we don’t have metrics for things like writer latency and frame drops.

“I was on the front lines of debugging ST 2110 issues, and it was a nightmare. With SDI you could put a waveform monitor on the cable and know immediately whether it worked. With ST 2110 it was much more complex: is the server actually sending? Is it the switch port? Is the NMOS controller sending the right command? Is it the packets on the network, or is it how the receiver is processing them? There is enormous additional complexity in the chain and many of those problems will persist with MXL.

“Honestly, I think a lot of this isn’t really being thought about yet because these are the boring things, but these are the things you need when you’re in production.”

To return to Riedel’s Chief Executive Officer (Product Division) Jan Eveleens earlier admission, these things take time, with the transition to IP taking twice as long as he first anticipated. The adoption of software-based workflows and the move to Dynamic Media Facilities stretches across even more skillsets.

“There is a lot that we underestimated on the skill side when we started transitioning to work in an IP environment,” he admits. “It was a big challenge, and some clients still struggle with it because they just don’t have the in-house skillsets to do it. Acquiring these skills is also difficult because broadcasters need to hire IT people, and there are plenty of other IT companies going after the same talent. There is also a cultural challenge because traditionally IT and broadcast departments are not always the best marriage.

“The other big challenge is also about resource. There are very specific machines that are required for heavy video processing. Cloud companies like AWS only have a limited number of these machines and we are all competing with AI companies who run their models on these same machines. These resources are not infinite; as a solution customers can hire the compute long-term, but that is a significant investment, so achieving that flexibility can be an interesting challenge.”

Internal resources can also be a barrier, with teams who have historically operated in silos needing to work together in a more coherent way. Echoing Eveleens’ earlier comment, this too is a cultural adaptation, and cultures are infamously slow to adapt.

“The biggest challenge is procurement,” says Myers (Lawo). “If you want to build a data center with software-based orchestration and functions on top, in a business where a procurement department is integrating with lots of other internal departments, bringing that together as a single total cost of ownership is difficult.

“We do see broadcasters bringing IT and broadcast engineering together into one department and that’s a key step. But the network and security teams still tend to be separate. That’s where you have to educate and bring people together because they may have different remits. 

A security team might simply say ‘I don’t want anything going on my network’ – but the network is a critical component of all of this. You can’t build a network without security.”

All this before we even start thinking about ensuring that media applications from different vendors can coexist on the same servers and share resources without stepping on each other’s toes. Thankfully, all this is within scope.

“One example of an ongoing discussion within JT-DMF is this kind of compute resource management,” says Poulin (CBC). “We are essentially learning how the IT world handles this, using tools like Kubernetes and dynamic resource allocation, and testing whether those approaches apply cleanly to our scenarios. There are a lot of specialized questions at each layer and it will take some time to arrive at a complete model of interoperability across all of them.”

The Human Element

In any creative industry, it’s worth remembering that technical and human challenges are intertwined, and that is no different here. The other big barrier to adoption is people, and Myers (Lawo) thinks there are two ways to look at it; both as a technical challenge, but also as an opportunity.

“The biggest challenge is the same one we had with ST 2110, and it is around education, training and expertise,” he says. “If you’re going to put this technology into a market, you need to bring the market with you and support the industry to educate itself or else there will be a skills gap.

“But one of the benefits of this technology shift is that it makes broadcasting cool again. People want to come in and do DevOps and software development. If we want this to be sustainable, we have to create the demand for people to want to come and work in our industry.

“The other side of it is managing the transition because as more experienced people retire, a lot of traditional knowledge will leave the business. You either need to transfer that knowledge or build up new knowledge for the toolset you need going forward. My concern has always been that that transitional period is always left to the last minute, and it’s a big lift.”

Eveleens (Riedel) backs this up. “The broadcast industry is one of the most advanced industries in the world when it comes to redundancy, backup and risk,” he adds. “This is deep in the DNA of broadcasters, but cloud compute is more complex and again, we’re having to look at appropriate skill sets. People that understand how that works and what you need to do to make sure that you don’t get caught out on a single point of failure — that is a skill that you have to either acquire or learn.”

Here again is an opportunity to learn from the past and the switch to ST 2110 is a perfect example. The standard was published in December 2017, and we’re still training people on it today almost ten years later. “We should be doing the training as we implement, not a decade after introduction,” warns Myers (Lawo).

Robinson (Matrox Video) also makes the case for more practical operational skills. “DevOps-type skill sets are increasingly important,” he says. “In the past, you’d often send someone on site to set up a server and work through configuration screens until it worked, but there’s no change management in any of that. With DevOps principles, change management becomes essential and easier.

“FinOps is also a really interesting area. For a lot of people it’s not top of mind but leaving an Elastic Compute Cloud instance running because you’re not sure what’s running and don’t want to switch anything off is a real problem. By going software and having smarter orchestration tools, you get better utilization of your licensing and the ability to dynamically adjust.”

The End Result

It’s clear there are a lot of different elements at play, and a lot of development to do across a broad range of technologies and skillsets. Fundamentally, all our panel had different ideas about the rate of adoption. But the switch to software-based workflows is nevertheless inevitable.

“We’re probably another couple of years away from a concatenated software, multi-vendor workflow and we’re still very much in that proof of concept phase,” says Rayner (Appear). “There is definitely some ambition to adopt some elements of these software-based production workflows for the 2028 Olympics, but I think it’s probably going to be another five or six years before it becomes a technology which is considered to be business as usual.”

Similarly Robinson (Matrox Video) thinks adoption will be in the next few years, although with some very clear caveats.

“Realistically, I think we’re 18 months away from serious usage,” he says.  “And what that will probably look like initially is more like tie lines between vendors – not the North Star vision of every media function as an individual container, dynamically discoverable, where you can swap vendor A for vendor B and everything just works. That’s the long-term goal. The short-term goal is: can we remove the need for synchronous transport like ST 2110 and use something like MXL to do it more efficiently? That within 18 months is achievable.”

Meanwhile, Eveleens’ (Riedel) outlook is even more rational: “It will accelerate, that’s for sure, but will everything be in software and private or public clouds in five years? No. And I don’t think we’re going to be there in 10 years’ time either.

“I do think development is going to be faster than it was with IP, but there are still lots of elements around software-based infrastructures that we haven’t even discovered, let alone mastered. But I am confident we will get there. Speak to me in 10 years and see how far I was off!”