Pac-12 Networks Has Cloud Workflows Figured Out

One of Pac-12 Networks’ REMI control rooms, located in San Francisco's South of Market District.
While the emerging concepts of remote processing and storage and “At-Home” production might be new to some, the team at Pac-12 Networks, the sports media production arm of the collegiate Pac-12 Conference, has been doing it—and refining their IT-centric workflows—for years. When you are tasked with supplying content across multiple platforms generated from 100 venues across 12 campuses, it’s just not humanly possible to deploy and manage all of the resources required.
At the end of the day, for Mark Kramer, Vice President, Engineering & Technology at Pac-12 Networks, it’s about enhancing the fan experience and helping to generate revenue for the various Pac-12 Conference member universities. Kramer’s department recently reached an agreement with Amazon Web Services (AWS) to have all of its content uploaded, processed and stored in the AWS Cloud while also leveraging other AWS Elemental Media Services tools to prepare live streams, on-demand content and other services for distribution to the public via MVPD providers like Comcast, Charter, Cox and Dish.
“What’s exciting about what we’re doing with AWS is that we can take all of this content as its coming in and direct it to the AWS cloud and then you can take advantage of all of the incredible computing capabilities up there, which is something we are starting to do now,” he said. “The cloud allows our team to do so much more. For example, we haven’t had to increase our engineering team. We’re building skills within the existing team and giving them tools to do more.”
Using redundant 10 Gb connections with the Pac-12 schools, AWS IP-based services are now supporting the live broadcasts of 850 games for Pac-12 collegiate sports over the 2018-19 season. By moving the Pac-12 broadcasts onto a standard cloud platform, every aspect of the media workflow can be enhanced and made more flexible for production and syndication. Pac-12 Networks is connecting its 10Gb multi-venue contribution network to the cloud with a 1Gb link. Elemental's MediaLive services prepare all live streams for delivery by the MVPDs in Apple’s HLS format to IOS, Android, Web, Chromecast and Apple TV devices.

Taking the new cloud-centric approach, Pac-12 Networks worked closely with Comcast Technology Solutions and Amazon Web Services to create an efficient workflow.
I think automation and efficiency are very important in our business,” Kramer said. “As we’re really focused on trying to maximize revenue opportunities to distribute back to our athletics departments, we‘re always looking for new ways to drive efficiency. I think what is interesting about applying machine learning and automation to production is that we can do so much more with our resources. Automation and machine learning really works when you use it to augment the people you have. It does not replace them.”
Taking the new cloud-centric approach, Pac-12 Networks worked closely with Comcast Technology Solutions to deploy key parts of their master control operations in the cloud. It makes extensive use of Amazon Simple Storage Service (S3) and Amazon Glacier to support their petabyte-sized primary archives for all recorded content. An automated ingest workflow is also leveraged heavily to handle the large amount of content on any given weekend. The ability to scale up the amount of Amazon S3 storage as needed has freed up other aspects of the production workflow, enabling new services like a live-to-VOD capture feature.
Because of this cloud workflow, the Elemental MediaTailor now provides a straightforward option to perform server-side ad insertion for both live and on-demand content, creating a whole new basis for content monetization. Other cloud services used in the Pac-12 cloud infrastructure include Amazon's IAM, CloudFront, EC2 AutoScaling, Elastic Block Store and Elastic Load Balancing to provide a secure, highly-effective backbone for TV-everywhere service delivery.
“When you want to do highlight clipping or mark every single touchdown in a digital experience, that would work fine if we were only doing football games. But we are doing 850 sporting events a year and it’s hard to scale that,” Kramer said. “That’s the capability we get with AWS Elemental Media services. And by being able to supplement the staff with machine learning, we can automate some pretty impressive things. That makes our content accessible to a lot more people, which is what we’re all about here at Pac-12 Networks.”
In the future, automated processing of the same assets with machine learning services adds the potential for automated gameplay highlight clips. Another potential future application involves the company's Lambda, a serverless application model that uses less compute for processing clip and highlight generation. In this workflow, the second a game is over, Pac-12 Networks would have all assets, such as game highlights, directly accessible for streaming and syndication partners. As a result, a wide variety of highlight options could be made available to fans, offering a much richer post game experience.
Using the cloud, they have also launched business analytics to help Pac-12 universities promote their teams and sell tickets. And new capabilities for doing dynamic ad insertion have opened up new revenue opportunities of which they previously did not have access.
“As we standardize on AWS machine learning and media services, we’ll be able to usher in a new era of entertainment for collegiate sports enthusiasts,” Kramer said. “Our fans will benefit from highly reliable and personalized viewer experiences, even in times of rapid traffic spikes like conference championships or rivalry games, and our internal teams will be able to experiment with ease using AWS services to rapidly test new ideas.

Pac-12 Networks makes extensive use of Amazon's Simple Storage Service (S3) and Glacier to support its petabyte-sized primary archives for all recorded content. Click to enlarge.
One reason Kramer gives for why automated workflows are so successful at Pac-12 Networks is that the conference owns all of the content being produced, so content rights and other business rules don't hold back innovation.
“We have an advantage in that we are wholly owned by our universities,” he said. “It would be very hard to do some of the things we want to do otherwise. It would require a ton of resources and upfront investment.The reason why we like AWS Cloud technology is that it allows us to innovate and experiment. Because it’s a pay-as-you-go model, there’s not a lot of risk involved. This allows us to take chances that we might not otherwise be able to take.”
The cloud has also facilitated at-home productions that use minimal resources on site (also known as remote integration production, or "REMI"), which Pac-12 Networks has been employing since 2013. Kramer sees more automated production going forward.
“We're just at the beginning with what REMI production can do technically. I think we can do even more without adding operational overhead,” he said. “We have a lot of hardware that’s expensive to maintain. We do 850 live broadcasts and then the equipment sits there in the off-season over the summer. If I want new code or to update a piece of software, I have to direct an employee to do that, while it sits offline and is not usable. And the gear has an end of life. We’re working to find innovative ways to leverage the cloud to overcome these issues.”
“By moving to a cloud-based approach, we’re benefiting from a lot more flexibility,” Kramer added. “Our business is changing drastically. Media is not a simple business anymore. The volume of competitors has grown exponentially. So, being in a position where you have the flexibility to respond to that is what’s going to help organizations succeed in this space.”
You might also like...
OTT’s Unique Storage Requirements
Central storage systems for OTT are unique in the world of media storage. They combine a set of requirements which no other media use case must handle.
OTT Content Origination
Content Origination is in the midst of significant transformation, like all parts of the OTT video ecosystem. As OTT grows and new efficiencies are pursued, Origination must play its part as a fundamental element of the delivery chain. But Origination…
Scalable Dynamic Software For Broadcasters: Part 7 - Connecting Container And Microservice Apps
Messaging is an incredibly important concept in microservices and forms the backbone of their communication. But keeping systems coherent and resilient requires an understanding of how microservices communicate and why.
The Big Guide To OTT: Part 2 - Content Origination
Part 2 of The Big Guide To OTT is a set of three articles which dig into the key issues of OTT content origination, the unique demands of OTT content storage, and the role of CDN selection in achieving broadcast grade…
Machine Learning (ML) For Broadcasters: Part 11 - Generative AI In Content Generation
Machine Learning, under the banner of Generative AI, is encroaching on creative aspects of Audio and Visual content generation, even down to production of movie scripts and screenplays. This builds on established applications of ML lower down the production chain,…