Hardware Infrastructure Global Viewpoint – July 2016

Digital Data Storage: A Universe of Choices

Predicting the future is just setting oneself up for being made the fool. While everyone has an opinion, when it comes to predicting coming technology, there many more errors made than home runs. As Yogi Berra said, “It's tough to make predictions, especially about the future.”

Fortunately, people far smarter than me are looking into their crystal balls and some see a bright future for storage technology. Let’s see what those experts learned and how it may affect the storage of data.

Size of the digital universe

An IDC report, commissioned by EMC, predicts there will exist more than 40 zettabytes (ZB) of digital data by 2020. Let’s think about that for a minute. A zettabyte is one sextillion bytes. Another way to look at 1 ZB is that it equals 1 million petabytes = 1 billion terabytes = 1 trillion gigabytes. One source predicts a 4300-percent increase in annual data generation by 2020. Drivers include the switch from analog to digital, business storage, media production and storage and the latest technology to generate data, the internet of things, IoT.

While much of the data generated comes from media, individuals too are responsible for many of those bits. More than 70-percent of all data generated comes from individuals. Ask yourself how many photos and videos of your kids have you stored on your home computer or phone? But that is not where most of data is actually stored. Data storage enterprises are responsible for storing 80-percent of data.

While the IDC report took a top-down appraisal of the creation of all digital content, storage vendor, Spectra Logic, projects that much of that data will never actually be stored or will be retained for only a brief time. And, says the company, data stored for longer retention is frequently compressed. This means that the “stored” digital universe is therefore a smaller subset of the entire digital universe as projected by IDC.

Storage technology

Storing data costs money and users will seek the least expensive ways to keep their data. That means infrequently accessed data will be stored on the lowest cost storage technology. For interactive data, users will seek to store their data where the technology offers the required mix of access time versus cost of storage. Spectra Logic says there are five distinct storage tiers that will define data migration workflows and long-term storage.

● Solid-state disk (SSD) will completely own the portable, smartphone, and tablet sector, and will eventually replace the magnetic disk for laptop and desktop storage.

● The two-and-half-inch 15,000 RPM magnetic disk will be completely displaced by enterprise flash.

● The three-and-half-inch magnetic disk will remain the workhorse for enterprise and cloud storage applications that need online or nearline access. However, the disk industry will need to deploy new technologies that can continue to reduce the cost per capacity.

● Three-and-half-inch magnetic disk using shingled media recording (SMR) will be the basis for a new near online tier of storage. While we may further examine SMR in another article, for now suffice it to say the technology overlays data tracks upon one another much like shingles on a roof, thereby saving disk space.

● Expect 3D NAND to improve thereby providing lower costs and higher densities.

● Tape has the easiest commercialization and manufacturing path to higher capacity technologies, but will require continuing investment in drive and media development. Spectra Logic suggests that size of the tape market will result in further consolidation, perhaps leaving only one drive and two tape media suppliers.

● Optical disc will remain a niche technology, primarily in the media and entertainment sector, unless great strides can be made in the production cost of the new higher capacity discs.

● Spectra believes no new storage technologies will have a significant impact on the storage digital universe through 2025. One exception may be solid-state technologies whose characteristics straddle DRAM and flash.

Cost of storage

Let’s now look at some cost of storage numbers

*Access for SMR drives are equivalent to enterprise disk, but writes may take significantly longer.
** Marginal media and device cost for Spectra TFinity tape library with LTO technology.
***Human intervention required to mount tape. Four hours is based on cloud offerings. Likely response is 4-48hr.

This chart provides some comparisons between five different storage technologies for speed and price per TB. Source: Spectra Logic
*Access for SMR drives are equivalent to enterprise disk, but writes may take significantly longer.
** Marginal media and device cost for Spectra TFinity tape library with LTO technology.
***Human intervention required to mount tape. Four hours is based on cloud offerings. Likely response is 4-48hr.

Solid state storage

The fastest growing storage technology is NAND flash. It provides both durability and speed. It is the de facto technology for consumer products. Consumer-grade solid-state flash drives sell for about $0.30 per GB while consumer-grade magnetic disk drives are approximately $0.05 per GB. For these reasons, the contribution of flash to the digital universe will increase but at a slower rate than in previous years.

As a solid-state technology, flash is restricted by the same lithograph limitations present in the manufacturer of processors and DRAM. As the flash cells become smaller, they experience more leakage across cells and also contain fewer electrons per cell. To compensate for these effects at higher die densities, more cells have to be reserved for error correction. At some point it provides no value to increase the cell count as all of the additional cells are consumed by error correction. This point for NAND flash appears to be around 15nm.

3D NAND is one way to increase capacity without requiring more horizontal real estate. The solution is to stack storage layers vertically. A 48-layer, 256 GB device is available now.

Intel and Micron are proposing a new technology, 3D XPoint (3D Crosspoint). Basically it is a memory cell whose resistance level changes because of a bulk change in its atomic or molecular structure. The type of memory is known as Resistive RAM, also ReRAM or RRAM.

Proponents claim that Resistive RAM will have 1000 times the performance and 1000 times better wear characteristics than NAND flash, and 10 times the density of DRAM. Expect it to become available in the 2017-2018 time frame.

Magnetic disk

The familiar magnetic disk drive may be fading from its seat at the top of the storage pyramid. Today’s 2.5-inch HDs are being replaced by solid-state-drives (SSD). 3D Xpoint (discussed above) will improve storage performance perhaps by being hosted directly on the memory bus. This will blur the distinction between storage and system memory.

Drive makers will continue to bring new drive technology to market. Examples include helium-filled drives. Helium reduces the head-to-disk friction, generate less heat and run more efficiently, resulting in a higher capacity device. Shingled Magnetic Recording (SMR) drives write overlapping tracks. The result is a savings in guard space between tracks resulting in higher capacity.

One vendor is delivering variable capacity drives whereby each drive of a designated capacity arrives at a slightly different capacity point. For example, 6TB drives purchased might have capacities ranging from 6TB to 6.4TB. The manufacturer formats a disk to its full potential instead of downsizing it to a pre-specified capacity. Both of these solutions create challenges for RAID controllers.

To greatly improve HD performance and capacity a fundamentally new technology will have to be introduced. The technology with the most potential is called Heat Assisted Magnetic Recording (HAMR). The storage process increases the areal density of a disk platter by heating the target area with a laser. This heated area becomes more receptive to a change in magnetic properties (reduced coercivity), allowing a lower and more focused charge to “flip” a smaller bit. The development process has not been without its challenges. It may be 2018 before for a drive in the 20TB capacity range becomes available.


Despite all the new storage technologies, the need for tape in the long-term archive market continues to grow. Tape technology is well suited for this space as it provides the benefits of a low environmental footprint, primarily floor space and power; a high level of data integrity over a long period of time; and a much lower cost per GB then all other storage mediums.

While the cloud has become a long-term storage solution for many, typically smaller, applications. At the same time, large cloud providers are adopting tape—either for backing up their data farms or for providing an archive tier of storage to their customers.

Cloud providers and large scale-out systems provide high levels of data availability through replication and erasure coding. These methods have proven successful for storing and returning the data “as is.”

However, and this is key, if during the lifecycle of that data, it becomes corrupted, then these methods simply return the data in its corrupted form. For the tape segment to see large growth, a wide-spread realization that “genetic diversity” in data storage technologies is required to protect customers’ digital assets. (See Reader's Resource note at the end of this article for an additional resource on the importance of genetic diversity in digital preservation.)


In early 2014 Sony Corporation and Panasonic Corporation announced a new optical disc storage medium designed for long-term digital storage. Trademarked “Archival Disc” it was on display at NAB 2016.

The Sony ODSD280U Optical Disc Archive provides a 1 Gbps write (verify on) and 2 Gbps read using the ODC-3300R Generation 2 cartridge. Optical disc cartridges are available in 300 GB, 600 GB, 1.2 TB, 1.5 TB and 3.3 TB capacity sizes, in re-writable and write-once formats.The 3.3 TB ODC-3300R (Generation 2) cartridge is write-once. And the 3.3 TB cartridge is only readable/writable by the Sony ODS-D280U and the ODS-D280F drives.

Unlike LTO tape, there is no interchange guarantee between two drives. In other words, a disc written with one vendor’s drive may or may not be readable with a different vendor drive. Even if one vendor’s drive is able to read the disc written in by another drive, there might be a penalty of lower performance.

Future of storage is bright

Fortunately, storage solutions continue to be developed. Each iteration is faster and typically more efficient. And, when the technology can be leveraged against consumer applications, the professional versions may become more cost effective.

The challenge for both content producers and the technicians charged with safely storing that content is what is a “best” solution? There may be several, or only one. It is important to keep in mind that the data will eventually need to be migrated to another technology. Then consider, when that time comes—will your data be “good” or “corrupted”?

Editor’s note: The content in this editorial is based on a Spectra Logic white paper, “Digital Data Storage Outlook 2016.” The entire white paper is available at this link.

Reader resource: Spectra Logic CEO Nathan C. Thompson and colleagues Bob Cone and John Kranz have just release an excellent book on the importance of maintaining genetic diversity in digital preservation. The book, “Society’s Genome: Genetic Diversity’s Role in Digital Preservation,” is available at Amazon, Barnes & Noble, iTunes and numerous online bookstores. The Kindle and paperback versions are available for $7.99 and $16.95 respectively on Amazon here. The eBook is also available for download through Barnes and Noble and iTunes for $7.99.

Readers may wish to follow the discussion at www.societysgenome.com or on Twitter at #societysgenome

Let us know what you think…

Log-in or Register for free to post comments…

Related Editorial Content

NAB 2016 Shared Storage Solutions: Spectra Logic

Spectra Logic’s hybrid solution uses both LTO and TS tape for a solution that Chief Technology Officer Matt Starr claims can really drive cost down to just pennies per GB.

Spectra Logic Introduces Spectra TFinity ExaScale Tape Storage System

Spectra Logic, a specialist in deep data storage, has announced Spectra TFinity ExaScale Edition — the world’s largest tape storage system designed to address the needs of organizations with exponential data growth, such as those in media and entertainment.