As the size of image files increase, facility engineers must consider new types of storage solutions.
As today’s media workflows increase in size and speed, with Big Data analysis and Fast Data processing added to the mix, the need to better manage the entire lifecycle of content becomes ever more important. Building an efficient and effective workflow requires the use of a hybrid storage infrastructure, leveraging the speed benefits of on-premises storage with the distribution capabilities of cloud.
Digital content generated from production houses and studios is reaching new levels of growth, mostly propelled by higher image resolutions and faster frame rates that are pressuring legacy storage infrastructures and their associated budgets. The increase in collaborative pre- and post-production workflows in development are examples where more on-site data storage is required.
Traditional Storage Challenges
The need for such flexible infrastructures is driven by rapidly changing and arduous workflows that require productivity and collaborative improvements, as well as better utilization and management of large film and video production assets over time. With assets continually growing in size and density, flexible petabyte (PB)-scale architectures are required to address the creation, manipulation and transmission of this digital content.
A recent survey of 200 enterprise professionals and service providers revealed that more than 60 percent of the M&E survey respondents estimated that their storage capacities were 50PB or more (see Figure 1). They also reported that more than 50 percent of their data was unstructured.
Figure 1. Estimated on-site storage capacity for media and entertainment facilities. Source Western Digital survey.
Many studios and production houses still rely on RAID-based storage for data protection, and/or to edit, transcode, color correct, add audio, or collaborate in real-time (as part of a Fast Data application), and/or for back-up and archival (as part of a Big Data application). Unfortunately, regardless of whether SSDs, HDDs, or a combination of the two are used, RAID-based storage architectures do not scale well. And they are not optimal for some media applications, which is especially true for today’s petabyte-size media workflows.
In such RAID-based systems, if a hardware or data integrity issue occurs, it could take weeks to rebuild the data, negatively affecting system performance and workflow productivity. If a subsequent issue occurs, the risk of data loss increases significantly. Similarly, tape library systems used for cold, archived data can be slow or produce unsuccessful results that can also negatively impact productivity, and at times, data may even become unreadable and difficult to access.
Global production is now the norm
Today’s M&E workflows need to support collaboration at a global scale, requiring seamless access to all content. In the future, the content may be mined to hopefully find some hidden nuggets of importance or analyzed as part of a Big Data application to extract further value or intelligence.
Traditional storage systems tend to create silos of data as storage capacity grows. This makes it difficult to ensure a consistent global view to all data, and provide fast access to it as required by a Fast Data application at the edge. In this scenario, leveraging on-premises and/or cloud object storage technology can help eliminate these challenges and have become foundational systems for large production houses and studios around the world.
As image sizes increase, so too does storage and bandwidth demands.
In addition, today’s 4K imagery and digital cinema can produce approximately 6TB of data per hour, and emerging 8K TV can generate 100TB per hour. Each doubling in resolution creates a 4x increase in density and file size. To ingest and process this much content requires a different approach to storing data and object storage is a good solution.
Object storage is an architecture that stores data as objects, whether it’s a document, film, video, audio, image, photo, or some other piece of unstructured data. Each stored object includes metadata that provides descriptive information about the object and the data itself. Because object data and metadata can be placed in a flat address space, the need for a hierarchical structure is eliminated, simplifying data access.
The system uses a unique identifier that is assigned to each object making it easier to index and retrieve data, or find a specific object. Because users define the metadata, data analytics, discovery techniques, or other information can be enabled for large volumes of data at scale, and the objects can also be aggregated to provide more efficient capacity scaling.
Traditional storage architectures offer little information about stored data that can help simplify manageability, or even support data at scale. Unlike file-based NAS systems that manage data in a folder hierarchy, or a block-based SAN approach that manages disk sectors collectively as blocks, object storage manages data as objects providing the ability to store unstructured data at petabyte scale.
ActiveScale Object Storage Systems
Having fast access to all content, from pre- and post-production, thru archiving, is the key to monetizing data. Western Digital’s ActiveScale object storage technology addresses such media workflows while preserving the media assets for the long-term using a hybrid storage combination that includes on-premises and cloud-based storage. Key benefits of ActiveScale include:
- Extreme Scalability: operating within a flat address space achieves massive scalability without the overhead associated with file system hierarchies, data look-up tables, or block re-assemblies. Instead of using indexes, both scale-up and scale-out capabilities are provided, as well as support for cloud-based storage analytics and petabyte-scale capacities.
- Dynamic Data Placement: by dynamically placing data within a three-tier hierarchy using ActiveScale BitSpread technology, performance degradation of traditional systems is avoided during rebalancing operations used to update static data placements.
- Advanced Geo-Spreading: Spreading data across three geographic locations requires only one-third of the object data stored in each location. This reduces network traffic and incurs about two-thirds of the overhead versus replicating, storing, and managing 100 percent of the data at three locations as part of the triple mirroring data replication model of legacy systems. The result is high data accessibility and resiliency, at a substantially lower cost in equipment and in management.
- Advanced Data Durability: reconstructing data from corrupted data sectors and detecting accidental changes to raw data using ActiveScale BitSpread technology, provides superior protection against sector-level bit errors. Additionally, ActiveScale BitDynamics helps to detect and self-heal errors in the background. These capabilities, in combination with erasure coding and data scrubbing technologies, can achieve up to 17 nines of data durability – and is capable of surviving a data center crash.
Hybrid Cloud Workflows
ActiveScale is a fully cloud-optimized system, in compliance with the Amazon Web Services (AWS) S3 API, enabling it to easily plug into an on-premises private cloud configuration or a hybrid-cloud design (see Figure 2). The hybrid design allows for efficient ingest and post-production activities, and provides an onsite content archive for all media. The public cloud can then be used to provide compute power to transcode and efficiently distribute final content.
The main benefit of this hybrid cloud configuration is that it enables the most precious and valuable assets, the raw content, to be close and available on a high-speed internal network. The cloud is then used for distributing final content. Cloud-compatible MAM software can manage all of the data, metadata, workflows, and distribution, while the ActiveScale system provides all of the scalable storage required by a cloud-compatible environment.
Production houses and studios have been facing the challenge of massive data growth for more than two decades, with no signs of a slow-down. The use of unstructured data presents multiple challenges when it comes to preserving, protecting, monitoring, and managing content throughout its life.As a result, M&E companies of all sizes can use object storage systems to cost-effectively capture, preserve, access, transform, and manage all footage and video, accelerate production workflows, increase productivity, while lowering TCO.
Western Digital’s ActiveScale object storage system provides scalability, dynamic data placement and advanced geo-spreading, which enables enterprises to easily leverage the public cloud in a hybrid configuration. This approach is economical and easy to manage at scale.
Erik Weaver is the Director of Product Marketing for media and entertainment solutions within the Data Center Systems group of Western Digital Corporation.
You might also like...
The first commercially available helium-filled hard drives were introduced by HGST, a Western Digital subsidiary, in November, 2013. At the time, the six terabyte device was the highest capacity hard drive available. Backblaze, a major hard drive user, wanted to find…
A revolution in storytelling for TV, cinema, VR, and related forms of entertainment has just begun, enabled by artificial intelligence (AI). This computer-science and engineering-based technique, including machine learning, deep learning, language understanding, computer vision, and big data - is…
With near unfettered access to portable media players of all types and faster networks, consumers are increasingly migrating to video providers that serve them best. Quality and reliability are the key drivers for loyal and recurring engagement.
It was late 2007, after seeing a Drobo in action at a trade show, that I bought one. It could handle up to four 3.5-inch hard drives of any capacity and automatically backed up redundant data without the user having to…
The transformation of the media and entertainment workflow from discrete, server-based silos to software-based environments is well underway. As the industry makes this shift, media companies find that placing a scale-out storage solution at the heart of the IP workflow…