CW@50: Data storage comes into its own

[ Editor's note: This is the second and last part of our story about the storage industry's evolution over the past 50 years. Part 1 is here.]

In 1973, CDC saw a chance to one-up IBM and produced the 9760 Storage Module Drive (SMC). It held 40MB of data and represented the first significant departure from IBM standards for disk storage media.

While increasing the capacity of its drives, CDC and others were also decreasing the gap between a read/write head and the magnetic platters on which data was stored, boosting read/write speeds.

Computerworld

By the mid-1970s, industry efforts accelerated to standardize on ways to connect hard drives to computers. Seemingly every few years, something emerged to push the storage market forward:

Seagate

• In 1977, CDC’s Storage Module Drive became a standard.
• In 1982, Shugart Associate’s Systems Interface was adopted for the Small Computer System Interface (SCSI).
• In 1984, Sun Microsystems released its Network File System, a distributed file system protocol that allowed servers to share storage space with computers on the same network.
• In 1985, CDC, Compaq Computer, and Western Digital collaborated to develop the Integrated Data Electronics specification -- a 40-pin connector for attaching Host Bus Adapters to disk drives.
• In 1998, the Fibre Channel specification was developed to give SCSI and other industry drive specifications a higher-performance, switched serial interface. Fibre Channel offered 100MBps block-level access to all types of storage – disk arrays, optical jukeboxes and tape libraries. Fibre Channel, in turn, enabled the development of storage area networks, a dedicated storage high-speed infrastructure separate from a corporation’s local area networks.

As interface standards evolved, IBM continued to lead in hardware development. In 1979, it created the first 8-in. disk drive, nicknamed Piccolo, which stored 64MB on six platters, for the IBM 62 PC. IBM also sold it as the 3310-attachable direct access storage device (DASD).

More importantly, IBM introduced the first drive to use thin-film heads, which allowed areal density to increase. In 1981, IBM used thin-film technology to ship a drive with four times the capacity of its predecessor.

While IBM's DASD continued to shrink in size and grow in capacity into the 1980s, the huge disk drives "were kind of the company's undoing" when it came to storage, said David Bennet, an engineer who worked on various storage projects for IBM from 1959 through 2014. Other companies, notably Seagate, had started working on smaller formats as IBM continued to build its 14-in. platters. "By the time IBM got into the smaller form-factor drives, there were competitors way ahead of them."

Storage rivals rise

The work of other vendors marked the beginning of an explosive era in hard drive development, beginning with the birth of the PC industry in the late 1970s and early 1980s. Among those companies pushing the industry forward: Microsoft and Apple Computer, which created the market for ubiquitous desktop computers; those in turn created the economies of scale for the hard drives that proliferated within them.

Computerworld / David Laws, 'The Storage Engine', Computer History Museum

"About the time IBM launched its PC is when Al Shugart developed 5-in. hard disk drive," said John Rydning, IDC's research vice president for hard disk drives (HDDs). "It was the 5-in drive that made the PC possible."

Seagate's 5.25-in hard disk drive -- the ST506 -- could hold up to 5MB and became the de facto standard for the PC industry.

"A lot of advanced technology happened on HDDs from that point forward," Rydning said. "Seagate launched the hard disk drive industry because you had the increased volume driven by the PC market... and that fueled the other corporations as well." After Seagate's 5.25-inch drive came competitors from other vendors: The 3.5-inch drive, then a 2.5 inch model and finally a 1-incher.

In 1990, IBM introduced the first disk drive with thin-film magnetoresistive (MR) heads, which alternated layers of thin magnetic and non-magnetic conductive materials, enabling the first multigigabyte drives.

While IBM had used MR heads in tape drives in the mid-1980s, the company's San Jose research lab adapted the technology for a hard disk code-named "Sawmill." It shipped as the IBM 9345 DASD system.

Western Digital

By 1994, other manufacturers followed suit as Fujitsu, Hitachi, and Hewlett Packard developed their own MR heads. By 1999, the technology was ubiquitous.

Drive capacity continued to grow, but areal density was running up against a wall at around 100Gbp square inch. It wasn't until 2005 that a technology breakthrough would smash past that limitation and launch a new era.

It was called perpendicular magnetic recording (PMR), and it took bits of data that had laid horizontally (or longitudinally) on the disk platter and stood them upright -- shoulder to shoulder -- so they could be packed closer together.

The development of PMR by Toshiba and Seagate increased magnetic media's capacity as much as 10-fold, which meant microdrives that held 3,000 songs now had room for about 30,000 songs.

When the industry again faced capacity limits in 2013, Seagate overlapped data tracks like roof shingles, increasing capacity by 25%; then in 2014, HGST introduced helium-filled drives, boosting capacity by another 50%.

"There were just so many brilliant people working on data storage that have taken so many innovations and commercialized them so quickly," Rydning said.

Faster than Moore's Law

Since the 1980s, hard drive makers have also competed against a different kind of storage rival: NAND flash memory, where cost and density reductions have been no less spectacular. In the early 1980s, Toshiba developed the first programmable non-volatile memory -- NOR flash (so named because the device could erase a large amount of memory).

By 1987, Toshiba had unveiled the first NAND flash memory, which stored data directly to silicon, rather than on spinning platters. Four years later, Toshiba debuted its first 4Mb NAND-type Electrically Erasable and Programmable Read-only Memory.

At about the same time, three-year-old SunDisk shipped the world's first NAND flash-based Solid State Drive (SSD). The 20MB SSD came in a 2.5-inch form factor and was aimed as a replacement for the hard drive in an IBM ThinkPad computer. The retail price was $1,000, or $50 per MB. (SunDisk was later renamed SanDisk.)

SanDisk

Just as PMR technology had created drives with greater density, flash memory was about to get its first big density boost when, in 1997, manufacturers began storing two bits of data instead of one per transistor -- ushering in multi-level cell (MLC) products. And, in 2008, they began packing three bits per cell, which further increased chip storage capacity and reduced cost.

The rise of smartphones and other mobile devices that use NAND flash for onboard storage also lowered prices through economies of scale.

The effect was dramatic.

In 2010, a gigabyte of NAND flash capacity retailed for $1. Today, a gigabyte of flash memory costs about 26 cents.

Other technologies have swept the storage industry in recent decades.

In 1999, SanDisk, Matsushita, and Toshiba used the MultiMediaCard (MMC) specification to create the Secure Digital (SD) MemoryCard. The first SD card held 2GB and provided digital rights management based on the Secure Digital Music Initiative (SDMI) standard.

Today, companies such as WD/SanDisk sell 512GB microSD cards, which have 256 times the capacity of the original card and that can store 128,000 MP3s encoded at 128Kbps.

Indeed, NAND flash had surpassed Moore's Law in terms of the time it took to grow in density and capacity.

Take, for example, SSDs. In 2012, SSDs maxed out at about 250MBps throughput and a top capacity of 512GB. SanDisk and others now offer SSDs with more than 1,700MBps data transfer rates and with 4TB of capacity. Last year, Samsung began shipping a 15TB SSD.

IDG

Innovation continues

Manufacturers over time have also been able to shrink the geometric size of the circuitry that makes up NAND flash technology -- from 90 nanometers (nm) a few years ago to 34nm and finally to 10nm. But as circuitry shrank, problems arose. Electrons leaked from cell to cell, creating data errors. Manufacturers, in turn, designed sophisticated error correction code to obviate the problem.

Ultimately, however, NAND flash makers faced a wall: They had little room to shrink the density of the circuitry further.

IDG

So, just as PMR marked a new era for hard drives in the mid-2000s, vertical NAND (also called 3D NAND) has recently allowed memory makers to break through the capacity limitations of planar (single layer) NAND flash.

Samsung, Intel, Micron and others began working on stacking NAND flash dies one atop the other.

The 3D NAND technology used a technique called Through Silicon Via (TSV) created micron-sized holes through the chip silicon vertically instead of just horizontally to allow data transfers. The result was a NAND flash device that used 45% less power and had twice the performance and endurance of planar NAND. What was once a single layer of transistors became a microscopic skyscraper of flash cells up to 64 layers high.

"Silicon surface area over time has become most expensive real estate around; it would be $3.5 billion an acre," said Siva Sivaram, executive vice president of memory technology at Western Digital (WD), which owns SSD maker SanDisk. "So when it costs you billions of dollars per acre to make, you don't buy California ranch-style homes. You go Manhattan-style.

"We introduced our first 3D flash products in 2005, with four layers. Now we're at 64 layers. I can tell you that there is no physical limitation on how high we can go," Sivaram said. "On all three axis, we don't have any immediate limitations approaching us, which is a powerful place to be."

This year, Intel and Micron announced plans to begin shipping the latest innovation in non-volatile memory -- 3D Xpoint, a new class of memory they say will fill a niche between NAND flash and DRAM. It is purported to have 1,000 times the performance of typical flash drives.

Micron will market its 3D XPoint memory under the name QuantX, and Intel under the name Optane.

From IBM to today

Over the past five decades, what began as a single hard drive manufacturer -- IBM -- grew into an industry that at one point had 262 hard-drive companies. Today, after years of consolidations and mergers, there are three hard-drive makers.

As the technology advances and morphs, prices continue to fall. From 2000 through 2016, the price of hard drive storage has declined 28% per year, while prices of NAND flash memory have declined 48% per year, according to IDC's Rydning.

In the next decade, the industry may see breakthroughs involving life sciences and the use of DNA to archive limitless amounts of data.

The foundation that storage has created for the tech industry is "just mind boggling," Rydning says. It has laid the ground work for the cloud, the apps we run on smartphones, Facebook, and digital music, he adds. "All that was made possible by advancements in disk drive technology."

This story, "CW@50: Data storage comes into its own" was originally published by Computerworld.