Hardware Infrastructure Global Viewpoint – January 2018

Why Is Storage So Slow To Change?

In the realm of video technology, I — like many others — predicted one thing entirely wrong a few years ago. I assumed by 2018 the hard drive would be dead and gone and solid state flash memory would have totally replaced it. I was incorrect. It is interesting to understand why the predicted transition has yet to occur.

For one thing, my thinking was way over simplified. I didn’t consider the user limitations and high costs of flash memory or comprehend how much and how quickly hard drives would improve. With most flash memory, you get between 10,000 and 100,000 read-write cycles and then you operate on blind luck. That means solid state memory is fine, as long as you know this and switch out the memory after so many cycles.

Of course, therein lies the problem. Most of us don’t do that. We tend to pay top dollar for brand name flash memory for cameras and editing and use it too long (meaning until we have a problem.) Single SD and compact flash cards have been around a long time, but have remained pretty much in cameras and camcorders. We usually replace them when speedier and higher capacity models come out.

Editing and large scale video-related post applications use different types of memory, usually a combination of both flash and hard drives. One of the key uses of hard drives today is in cloud and online storage.The hard drives are cheap, reasonably fast and most of them just keep on working.

While hard drives are a relatively mature technology, that works to their benefit as manufacturers find new ways to improve on an already low-cost, high-performance product.

While hard drives are a relatively mature technology, that works to their benefit as manufacturers find new ways to improve on an already low-cost, high-performance product.

Hard drives, of course, don’t have the same limitations on read-write cycles as flash memory, but the question remains: how long do hard drives last? Funny you ask, because no one knows the exact answer. But, very slowly, we are beginning to get a pretty good idea.

Backblaze, the unlimited online backup company serving computer users, has about 82,516 standard computer-level hard drives spinning at all time. It has performance records going back about four years. Their inventory includes 17 hard drive models that average between three and and eight terabytes in size.

Running 24/7, Backblaze found that hard drives have three distinct phases for failure. In the first phase, which lasts about 1.5 years, hard drives have an annual failure rate of 5.1 percent. For the next 1.5 years, the annual failure rate drops to 1.4 percent. After three years, the failure rate explodes to 11.8 percent per year. In short, this means that around 92 percent of drives survive the first 18 months. Plus, almost all of those — 90 percent — then last three years.

This means just under 80 percent of all hard drives survive four years. Backbaze speculated that the failure rate will probably stay at about 12 percent per year. This means that 50 percent of hard drives survive six years.

During the first quarter of 2017, Backblaze noted a wide range of performance, with some drives achieving a zero percent annualized failure rate and one brand suffering more than 35 percent failure, albeit with a statistically insignificant 170 drives. Out of the tens of thousands of drives in the database, a total of 357 drives failed during the first quarter.

Click image to enlarge.

Click image to enlarge.

Backbaze started out using standard, consumer-drives. Now they use enterprise-level hard drives, specifically adding in almost 2,500 Seagate 8TB drives and is now upgrading to 10 and 12 TB models. In the third quarter of 2017, the company added more than 59 petabytes of data storage to bring its total storage capacity to 400 petabytes.

The company’s decision to implement the new drives was based on higher performance and additional features, but so far Backbaze is finding little difference between the annual failure rates of consumer and enterprise drives. The main difference is the feature set.

Click image to enlarge.

Click image to enlarge.

What causes hard drives to die? In the first year, it is normally manufacturing defects. Between 18 and 36 months, drive deaths are caused by random failures. As drives move into the fourth year, failure rates occur because the mechanical device simply wears out.

As we continuously generate and store more and more data, storage technology is a fast moving target. What is true today, will not be true tomorrow. In 2012, solid-state flash memory cost ten or more times that of hard disks in terms of price per gigabyte. However, as we have learned, cost is not everything in comparing types of media.

Also, most of the world’s flash memory does not perform professional applications.The bulk of flash is soldered into mobile phones, tablets, USB dongles and other consumer devices. Professional use is a small percentage of the output.

In video, while we enjoy the benefits of flash for recording with digital cameras, in post-production the applications are used differently and we’d prefer to save money for storing vast amounts of memory.

I still have no doubt that one day in the future flash memory will eventually win out over hard drives. But when that will happen, I will no longer try to guess.

Commenting is not available in this channel entry.

Related Editorial Content

​Quantum Add AI to Storage

Quantum has linked its storage platform StorNext with an artificial intelligence layer from Veritone.

Video Storage: It’s All About Loss Prevention

Losing media content is a disaster. Yet, there are multiple technologies designed to store content, some for short-term and others for long-term. In the expanding world of content development, everyone has the same question; “How best to store it?”

Digital Storage: Consider The Options

A recent report looked at four digital storage technologies; hard disk, solid-state, optical and tape. While each offers its own advantages, choosing the one (or more) best suited for your application requires careful consideration. The differences are significant and for…