Hardware Infrastructure Global Viewpoint – October 2015

Storage Challenge: What’s your content worth?

At IBC virtually every camera relied on solid-state storage. No more tape, no more cabling back to recorders. Cameras today come with sufficient on-device storage for recording up to several hours of content.

This begs the question about storage. Can content producers afford to store everything on flash?

As a world, we’re closer to having everything/anything connected. It’s called “Internet of Things, IoT) where everything from your watch to refrigerator and thermostat are connected via the Internet. All that connectivity led to the shortage of IPv4 addresses, creating the need to move to IPv6, which supports 50 billion “things”.

Memory size shrinking

Microprocessor speeds double every two years and device prices get lower, so we can create more things, stuff faster/better to be stored. This year, we’ll pack away more than 3.5ZB (Zetabytes) of data. In five years, we’ll be producing almost thirteen times that amount of content--44ZB per year.

And the drive to create more, and store more, is increasing. More content has been created in the past two years than in the previous 5,000

Figure 1. More than Ever – We’ve created more content in the past two years than in the previous 5,000 and the volume of data, pictures, movies and things continues to grow at an accelerated rate.  No wonder you’re constantly buying additional storage devices.

Figure 1. More than Ever – We’ve created more content in the past two years than in the previous 5,000 and the volume of data, pictures, movies and things continues to grow at an accelerated rate. No wonder you’re constantly buying additional storage devices.

Broadcast leverages off of Cots

There are basically two kinds of application storage—consumer, including professional applications and IT enterprise storage.The latter includes the Cloud.One may quibble about this simplistic division, but hang with me.

Spinning disks have been around since the 1950s when IBM’s RAMAC 305 held 5MB of data at $10,000 a megabyte. The drive was as large as two refrigerators and required 50 24-inch platters.

Today, laptop drives are 2.5-in size and store several terabytes of data. With Moore’s Law, capacity doubles every two years and prices drop by half.The insane rate of change had reduced the drive industry from 200 plus producers down twenty years ago, to today’s three – Seagate, Toshiba, WD (Western Digital) and their subsidiaries.

They were doing great until 1984 when Toshiba introduced the first flash memory (electronic non-volatile computer storage) that can be electrically erased and reprogrammed.RAM (random access memory) SSD (solid state drives) started to appear in 1979 but you and I didn’t care because the cost was outta’ sight.

Figure 2. Up and Up – Businesses, industries, governments and individuals are producing/storing content everywhere.  With prices coming down for flash-based storage, sales have increased significantly.  But, according to IDC, as good as the numbers and features are, flash continues to be a fraction of the total storage market.  About 75 percent of our content is still stored on hard drives.

Figure 2. Up and Up – Businesses, industries, governments and individuals are producing/storing content everywhere. With prices coming down for flash-based storage, sales have increased significantly. But, according to IDC, as good as the numbers and features are, flash continues to be a fraction of the total storage market. About 75 percent of our content is still stored on hard drives.

Spinning drives are dead

Maybe, but It depends on who you ask.

By 2016,flash storage is the leading solution for short-term storage. It’s cheap (though not as cheap per GB as spinning drives or tape) but it’s fast and widely available.

The storage market is huge ($45B) and ultra-competitive.

With the increased capacity, lower prices and better performance, sales have increased significantly. Despite these changes, flash storage continues to represent a fraction of the total storage market--about 75 percent of content is still stored on hard drives.

Figure 3. Shrinking Difference – Historically, there has been a significant price difference between flash-based storage and hard drives; but thanks to engineering breakthroughs and production scale increases, the price differential continues to shrink.  The primary differential continues to be capacity and availability.

Figure 3. Shrinking Difference – Historically, there has been a significant price difference between flash-based storage and hard drives; but thanks to engineering breakthroughs and production scale increases, the price differential continues to shrink. The primary differential continues to be capacity and availability.

Rapid growth for flash

Flash storage is being designed into practically everything because of its requirement in producing new features; fast data transfer rates, ultra-low latency and fast access time. But flash manufacturers don’t just want to make more storage space, now they want to make it cheaper and tougher for rough environments. Producing flash for cameras represents a minuscule portion of the market. Solid-state storage for cameras would never be possible without other, much larger, applications.

With the IoT representing some 50 billion devices, there is a significant market yet to be addressed. Transportation applications will be a growing portion of that marketplace, with car makers saying that already an estimated 60 percent of today’s car cost is electronics. Add the billions of smartphones being used and built and it is clear that flash storage will only grow. Broadcast and content makers will benefit from those, not-related, applications.

Performance versus capacity

At a recent Flash Memory Show, an interviewer asked attendees, which technology fit which application; flash versus spinning disk. Most respondents said they preferred flash for performance and HD for capacity. Even so, some speakers at the show claimed hard discs would soon go the way of tape and optical disc.

They claimed the benefits of flash outweighed the larger capacities of HD.

Flash:

-is inherently fast, easy to manage

-compresses data more efficiently, removes dup files

-takes up less space in datacenters, requires less electricity/cooling

Perhaps equally important, all-flash arrays for enterprise applications have grown faster as prices decline. That’s probably why IDC projects that the all-flash market will reach $2.24B this year.


">

Figure 4. Outta’ Sight – Individual and organization content creation has become overwhelming for data centers to capture, identify/categorize and retain. It’s less a matter of the storage used, but will it provide sufficient capacity and ultimately, can you access and retrieve specific structured and unstructured data.

Flash can lower production costs

Historically, there has been a significant price difference between flash-based storage and hard drives; but thanks to engineering breakthroughs and production scale increases, the price differential has decreased.

Alex Grossman, of Quantum, which specializes in studio/station production solutions, noted during one of the convention M&E sessions that (with flash) their customers see tangible benefits with faster workflow and lower admin costs.

“These people are at the tipping point,” he commented.“They know what days of saved production/post time does for their bottom line.Until the project is wrapped, they don’t make money. So all-flash solutions are easy to justify.”

Beyond the entertainment market, he said that Quantum’s enterprise customers often use a two-tier approach with a mix of about 20 percent all-flash and 80 percent spinning disk storage.

Why two solutions?

According to a recent EMC report, 20 percent of the stored data is active and the other 80 percent is seldom, if ever, touched.

Wikibon* is wildly optimistic about flash sweeping HDs aside. Even so, the organisation noted that last year, about 1,000 times more HDs were sold than flash solutions.

While the flash suppliers love Wikibon’s projections, they gloss over two important facts – we’re producing data/content faster than we can store it and there isn’t enough flash storage production capacity –globally – to hold that content. HDs remain the primary storage technology.

Think before you save

Hard drives will continue to be used when price, capacity and availability are the primary considerations.

Flash will win where speed, ruggedness, form factor and fragmentation are important factors.

As hungry as people are for creating stuff and storing it, most in this industry just don’t believe the doomsday warning Seagate issued last year; “The world could run out of storage capacity in two years.”

If it gets even close, logic says prices would go up because the more in demand a technology is, the more expensive it becomes. At that point, folks might heed Juhana Venhalainen’s, University of East London, suggestions.

In a presentation he noted, “Our reluctance to throw away anything has made our storage infrastructure into something approximating a kitchen junk drawer.Instead of focusing narrowly on capacity allocation efficiency, planners need to consider capacity planning efficiency.”

I guess he means that just as we have a line at the end of our emails that says, “Think before you print,” we should have an image on our screens that say, “Think before you save.”

In response, I can just hear The Broadcast Bridge readers responding, “That’s not going to happen. We story everything!”

Editor note:

*Wikebon is an organization focused on “improving the adoption of technology and business systems through an open source sharing of free advisory knowledge.”

Let us know what you think…

Log-in or Register for free to post comments…

Related Editorial Content

The Perfect “Forever” Archival Storage Medium? Don’t Count On It!

Let’s say you or your company own a very valuable artistic work of media — a film like Casablanca, perhaps. How would you go about protecting the digital master file of Casablanca against data corruption, fire, theft or some other unf…

​A Producer’s Guide to Storage - part 1

As production company’s deliver more and more of their content to UK broadcasters as files, it is asking a great many questions about their company wide asset management policy. The Digital Production Partnership (DPP) is supporting indies with guidance…

A Producer’s Guide to Storage – part 2

Part 2 of the Digital Production Partnership’s guidelines for independent producers looking to refresh or create a file-based asset management policy examines where content should be kept and advice for organising a digital file.

LTO-6 tape storage cost drops further

The backers of LTO tape storage have hit back at disc-based alternatives claiming that LTO is the most reliable and cost-effective format. LTO Generation 6 tape has reached a cost-per-gigabyte milestone of below a cent and $8 per terabyte, according to the…

Archive Storage: DIY or Outsource?

Of all the crucial requirements often mentioned about storing data, permanence is the most important when it comes to archival storage.

​Exploding Storage Needs Solved

Today, everything is high definition, 4K and even 8K. Every piece of raw footage must be kept, securely stored, and always accessible. Every channel requires customised edits, formats, versions. This all leads to exploding storage needs.