Applied Technology: Using Object Storage for M&E Workflows

​As today’s media workflows increase in size and speed, with Big Data analysis and Fast Data processing added to the mix, the need to better manage the entire lifecycle of content becomes ever more important. Building an efficient and effective workflow requires the use of a hybrid storage infrastructure, leveraging the speed benefits of on-premises storage with the distribution capabilities of cloud.

Digital content generated from production houses and studios is reaching new levels of growth, mostly propelled by higher image resolutions and faster frame rates that are pressuring legacy storage infrastructures and their associated budgets. The increase in collaborative pre- and post-production workflows in development are examples where more on-site data storage is required.

Traditional Storage Challenges

The need for such flexible infrastructures is driven by rapidly changing and arduous workflows that require productivity and collaborative improvements, as well as better utilization and management of large film and video production assets over time. With assets continually growing in size and density, flexible petabyte (PB)-scale architectures are required to address the creation, manipulation and transmission of this digital content.

A recent survey of 200 enterprise professionals and service providers revealed that more than 60 percent of the M&E survey respondents estimated that their storage capacities were 50PB or more (see Figure 1). They also reported that more than 50 percent of their data was unstructured.

Figure 1. Estimated on-site storage capacity for media and entertainment facilities. Source Western Digital survey.

Figure 1. Estimated on-site storage capacity for media and entertainment facilities. Source Western Digital survey.

Many studios and production houses still rely on RAID-based storage for data protection, and/or to edit, transcode, color correct, add audio, or collaborate in real-time (as part of a Fast Data application), and/or for back-up and archival (as part of a Big Data application). Unfortunately, regardless of whether SSDs, HDDs, or a combination of the two are used, RAID-based storage architectures do not scale well. And they are not optimal for some media applications, which is especially true for today’s petabyte-size media workflows.

In such RAID-based systems, if a hardware or data integrity issue occurs, it could take weeks to rebuild the data, negatively affecting system performance and workflow productivity. If a subsequent issue occurs, the risk of data loss increases significantly. Similarly, tape library systems used for cold, archived data can be slow or produce unsuccessful results that can also negatively impact productivity, and at times, data may even become unreadable and difficult to access.

Global production is now the norm

Today’s M&E workflows need to support collaboration at a global scale, requiring seamless access to all content. In the future, the content may be mined to hopefully find some hidden nuggets of importance or analyzed as part of a Big Data application to extract further value or intelligence.

Traditional storage systems tend to create silos of data as storage capacity grows. This makes it difficult to ensure a consistent global view to all data, and provide fast access to it as required by a Fast Data application at the edge. In this scenario, leveraging on-premises and/or cloud object storage technology can help eliminate these challenges and have become foundational systems for large production houses and studios around the world. 

As image sizes increase, so too does storage and bandwidth demands.

As image sizes increase, so too does storage and bandwidth demands.

In addition, today’s 4K imagery and digital cinema can produce approximately 6TB of data per hour, and emerging 8K TV can generate 100TB per hour. Each doubling in resolution creates a 4x increase in density and file size. To ingest and process this much content requires a different approach to storing data and object storage is a good solution.

Object Storage

Object storage is an architecture that stores data as objects, whether it’s a document, film, video, audio, image, photo, or some other piece of unstructured data. Each stored object includes metadata that provides descriptive information about the object and the data itself. Because object data and metadata can be placed in a flat address space, the need for a hierarchical structure is eliminated, simplifying data access.

The system uses a unique identifier that is assigned to each object making it easier to index and retrieve data, or find a specific object. Because users define the metadata, data analytics, discovery techniques, or other information can be enabled for large volumes of data at scale, and the objects can also be aggregated to provide more efficient capacity scaling.

Traditional storage architectures offer little information about stored data that can help simplify manageability, or even support data at scale. Unlike file-based NAS systems that manage data in a folder hierarchy, or a block-based SAN approach that manages disk sectors collectively as blocks, object storage manages data as objects providing the ability to store unstructured data at petabyte scale. 

ActiveScale Object Storage Systems

Having fast access to all content, from pre- and post-production, thru archiving, is the key to monetizing data.  Western Digital’s ActiveScale object storage technology addresses such media workflows while preserving the media assets for the long-term using a hybrid storage combination that includes on-premises and cloud-based storage. Key benefits of ActiveScale include:

  • Extreme Scalability: operating within a flat address space achieves massive scalability without the overhead associated with file system hierarchies, data look-up tables, or block re-assemblies. Instead of using indexes, both scale-up and scale-out capabilities are provided, as well as support for cloud-based storage analytics and petabyte-scale capacities.
  • Dynamic Data Placement:  by dynamically placing data within a three-tier hierarchy using ActiveScale BitSpread technology, performance degradation of traditional systems is avoided during rebalancing operations used to update static data placements.
  • Advanced Geo-Spreading:  Spreading data across three geographic locations requires only one-third of the object data stored in each location. This reduces network traffic and incurs about two-thirds of the overhead versus replicating, storing, and managing 100 percent of the data at three locations as part of the triple mirroring data replication model of legacy systems. The result is high data accessibility and resiliency, at a substantially lower cost in equipment and in management.
  • Advanced Data Durability:  reconstructing data from corrupted data sectors and detecting accidental changes to raw data using ActiveScale BitSpread technology, provides superior protection against sector-level bit errors. Additionally, ActiveScale BitDynamics helps to detect and self-heal errors in the background. These capabilities, in combination with erasure coding and data scrubbing technologies, can achieve up to 17 nines of data durability – and is capable of surviving a data center crash.

Hybrid Cloud Workflows

ActiveScale is a fully cloud-optimized system, in compliance with the Amazon Web Services (AWS) S3 API, enabling it to easily plug into an on-premises private cloud configuration or a hybrid-cloud design (see Figure 2). The hybrid design allows for efficient ingest and post-production activities, and provides an onsite content archive for all media. The public cloud can then be used to provide compute power to transcode and efficiently distribute final content.

Figure 2.  Media post-production workflow with ActiveScale object storage.

Figure 2. Media post-production workflow with ActiveScale object storage.

The main benefit of this hybrid cloud configuration is that it enables the most precious and valuable assets, the raw content, to be close and available on a high-speed internal network. The cloud is then used for distributing final content. Cloud-compatible MAM software can manage all of the data, metadata, workflows, and distribution, while the ActiveScale system provides all of the scalable storage required by a cloud-compatible environment.

The Solution

Production houses and studios have been facing the challenge of massive data growth for more than two decades, with no signs of a slow-down. The use of unstructured data presents multiple challenges when it comes to preserving, protecting, monitoring, and managing content throughout its life.As a result, M&E companies of all sizes can use object storage systems to cost-effectively capture, preserve, access, transform, and manage all footage and video, accelerate production workflows, increase productivity, while lowering TCO.

Western Digital’s ActiveScale object storage system provides scalability, dynamic data placement and advanced geo-spreading, which enables enterprises to easily leverage the public cloud in a hybrid configuration. This approach is economical and easy to manage at scale. 

Erik Weaver is the Director of Product Marketing for media and entertainment solutions within the Data Center Systems group of Western Digital Corporation.

Erik Weaver is the Director of Product Marketing for media and entertainment solutions within the Data Center Systems group of Western Digital Corporation.

Let us know what you think…

Log-in or Register for free to post comments…

You might also like...

NAB 2019 – Technology vs Consolidation

Giving his unique view on NAB2019, Gary Olson considers and scrutinizes the big moving trends of consolidations, and casts clarity on the cloud, ATSC 3.0, and AI.

Server-Based “At Home” Workflows Provide Efficiency For NASCAR Productions

NASCAR Productions, based in Charlotte NC, prides itself on maintaining one of the most technically advanced content creation organizations in the country. It’s responsible for providing content, graphics and other show elements to broadcasters (mainly Fox and NBC), as w…

Essential Guide:  Practical Broadcast Storage

Ground breaking advances in storage technology are paving the way to empower broadcasters to fully utilize IT storage systems. Taking advantage of state-of-the-art machine learning techniques, IT innovators now deliver storage systems that are more resilient, flexible, and reliable than…

Practical Broadcast Storage - Part 3

Artificial Intelligence (AI) has made its mark on IT and is rapidly advancing into mainstream broadcasting. By employing AI methodologies, specifically machine learning, broadcasters can benefit greatly from the advances in IT infrastructure innovation and advanced storage designs.

An Insider’s Guide to Object Storage

As the IP revolution continues to gain momentum and more broadcast facilities take advantage of the fantastic and unprecedented opportunities IT delivers, administrators and system designers must master the complex aspects of data storage.