Practical Broadcast Storage - Part 2
![Context aware data-sets, along with AI, can create scalable and dynamic storage systems.](/cache/uploads/content_images/data_rack_tony_HPE_part_2_purchased_canstockphoto22152380_789_522_70_s.jpg)
Context aware data-sets, along with AI, can create scalable and dynamic storage systems.
Broadcast systems are renowned for their high speed and high capacity data demands. Up to recently, they relied on bespoke hardware solutions to deliver the infrastructure required for live real-time uncompressed video. But new advances in IT data storage have now opened the doors for broadcasters to take advantage of this state-of-the-art IT innovation.
High-speed data and capacity requirements are now no longer exclusive to broadcasting, the proliferation of web sites and phone-apps has driven IT speed, capacity, reliability, and resilience way beyond the levels demanded by broadcasters. Big-data processing has further driven innovation and data storage systems capable of processing and streaming real-time video and audio is easily available using high-end IT solutions.
SDI broadcast infrastructures are highly tuned and optimized facilities that are difficult and expensive to maintain. Years of bolted-on tweaks and changes to facilitate new workflows add to the risk of outage and loss of transmission. Engineers are constantly chasing their tails trying to keep up with the changes to the system while maintaining high levels of reliability.
Web-site and phone-app service providers also encounter similar challenges to broadcasters, especially when trying to optimize data workflows, balance workloads, fine tune systems, and reduce the need for their team to constantly firefight.
IT Innovators Solutions
But IT innovators have been working hard to deliver high levels of intelligent automation and provide highly optimized infrastructures and workflows.
Although the first experimental AI systems go back to the 1940’s, recent advances in virtualized computation, networking, and storage has witnessed an explosion in the number of AI systems now available.
But what makes AI so important? Seemingly, we just analyze data collected from sensors and data-sets to help understand systems and provide a level of prediction. This method of statistical analysis has been used by engineers and scientists for hundreds of years.
![Diagram 1 – Although this statistical chart implies a high correlation between humidity and disc parity error, AI provides much greater levels of verification and prediction by parsing many more data sets](https://www.thebroadcastbridge.com/uploads/content_images/_Large/RealityOfStorage-P2-dia1.png)
Diagram 1 – Although this statistical chart implies a high correlation between humidity and disc parity error, AI provides much greater levels of verification and prediction by parsing many more data sets
By looking at the number of users logging onto a web-site, the designers can set a pre-programmed service to spin up new virtualized servers to meet the increased demand. If analysis indicates the number of users double for four hours on a Friday at 6:00pm, pre-emptive action can be taken to spin up the servers in anticipation of the increased demand.
Systems are Dynamic
For as long as businesses have been able to harness information, managers have analyzed data to help predict the future using ever increasingly complex statistical methods. But this form of analysis assumes a static or very slow-moving system.
Statisticians and Business Intelligence Analysts spend a great deal of time trying to find correlation between different data sets. In the example above, analysis indicated web-site users doubled at 6:00pm on a Friday. Although this is a simple example, it’s difficult to establish with much certainty whether this is a trend or an anomaly. The data used for analysis is out of date before the information can be adequately scrutinized.
Context Aware
To improve the prediction, more data-sets would be needed so better correlations can be established. The prediction can never be 100% accurate, but accuracy is directly proportional to the number of correlated data-sets that can be established. AI comes into its own when we look at context-aware dynamic and fast-changing systems.
AI is a generic term that covers four different disciplines; reasoning, natural language processing, planning, and machine learning. Split into four more categories, machine learning includes supervised learning, unsupervised learning, reinforcement learning, and deep learning (or neural learning).
Traditional computer programming requires a set of logical rules to be provided for the data to be analyzed. Usually providing a series of decision trees, the programmer will parse the data and then provide binary outcomes. For example, if it’s 6:00pm on a Friday, then spin up two more virtual web-servers.
Programming is Too Slow
Changing the data to variables, such as “6:00pm” and “Friday”, allows generic rules to be established so that the time and day can be referenced from a database and easily changed. However, this method soon becomes complex and difficult to administer when the rules need to be adapted.
![Diagram 2 – AI encapsulates four different disciplines and machine learning is further classified into four more classes](https://www.thebroadcastbridge.com/uploads/content_images/_Large/RealityOfStorage-P2-dia2.png)
Diagram 2 – AI encapsulates four different disciplines and machine learning is further classified into four more classes
It might become clear that Tuesdays also need an extra server at 6:00pm. This would require the programmer to add an addition to the rule or add a completely new rule. The example is trivial, but complexity of the program soon escalates, and efficient software maintenance is further compromised.
The method is slow and potentially problematic as the rules must be constantly adjusted to improve the efficiency of the system.
AI Generates Rules
Instead of manually programming rules for the data, AI takes the opposite approach and the data is used to automatically generate its own rules within the context of the data it’s working with in real-time. And that is the true power of AI.
Algorithms continuously monitor historic as well as real-time data and find accurate correlation between the data sets to help improve the future predictions for the system. This approach further improves efficiencies to take advantage of dynamic systems to constantly balance resource.
In the previous article in this series we looked at how advanced IT storage systems balance the user experience against the different storage resource available. An edit system would require instant low latency access to their edit masters and rushes, and this would be provided by the Solid-State-Device (SSD) and memory storage during the edit session.
Improving Efficiency
It would be uneconomical to continue to keep the media files in the high-speed storage when the edit session had finished so the files would need to be copied back to spinning disks. Although manual rules could be established to copy the files between the two storage mediums, AI provides a far more efficient method of providing this facility.
Algorithms can establish which files to copy based on previous sessions and the editor’s behavior. If the editor was previewing off-line rushes earlier in the day, the AI algorithm would associate these files with them and automatically copy the media files to SSD and memory storage associated with the edit suite for their edit session.
Real-Time Analysis
AI not only provides ever increasing efficient levels of automation, it does it without any human analysis, programming, or intervention. The algorithms learn the behavior of the storage dynamics and accurately predict which files need to be transferred between the different storage medium to improve the user experience and make the most of the resource available.
All this happens in near real-time making the traditional record-analyze-program methods now obsolete. In the next article in this series, we look at how AI has influenced and greatly improved the reliability and resilience of storage.
Part of a series supported by
You might also like...
Essential Guide: Delivering Intelligent Multicast Networks
This Essential Guide discusses the potential weaknesses of the ‘Protocol-Independent Multicast’ protocols that underpin multicast, and explores how a bandwidth aware infrastructure can maximize network capacity to reduce the risk of congestion.
Standards: Part 16 - About MP3 Audio Coding & ID3 Metadata
The MP3 audio format has been around for thirty years and has been superseded by several other codecs – so here we discuss why it still has a very strong position in broadcast. We also discuss ID3 metadata tags which often a…
HDR Picture Fundamentals: Brightness
This article describes one of the fundamental principles of broadcast - how humans perceive light, how this relates to the technology we use to capture and display images, and how this relates to HDR & Wide Color Gamut
Virtualization - Part 2
In part one, we saw how virtualization is nothing new and that we rely on it to understand and interact with the world. In this second part, we will see how new developments like the cloud and Video Over IP…
Standards: Part 15 - ST2110-2x - Video Coding Standards For Video Transport
SMPTE 2110 and its related standards help to construct workflows and broadcast systems. They coexist with standards from other organizations and incorporate them where necessary. In an earlier article we looked at the ST 2110 standard as a whole. This time we…