HPC Storage Grows Cloudier, Flashier

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Organizations running high performance computing (HPC) workloads are increasingly seeking out cloud-based storage solutions and speedy flash-enabled systems to help them cope with growing complexity and the sheer amounts of data they are managing nowadays, according to new research from DataDirect Networks (DDN).

For starters, organizations are making use of more data, the company found in its survey of over 100 HPC professionals. Eighty-five percent of reported that they are using or managing more than one petabyte (PB) of storage, a 12-percent increase compared to last year’s results. Nearly 30 percent said they in charge of over 10PB of storage.

Cloud Storage and Backup Benefits

Protecting your company’s data is critical. Cloud storage with automated backup is scalable, flexible and provides peace of mind. Cobalt Iron’s enterprise-grade backup and recovery solution is known for its hands-free automation and reliability, at a lower cost. Cloud backup that just works.

SCHEDULE FREE CONSULT/DEMO

To help them cope, HPC specialists are turning to the cloud, albeit cautiously.

Nearly half (48 percent) of all respondents said they planned to stash at least some of their data on a public or private cloud, an 11 percent jump compared to 2016. Yet, only five percent of those polled said they expect to place more than 30 percent of their data in the cloud.

HPC, cloud computing

Regardless, cloud usage is on the upswing. Forty percent of respondents said they planned to place at least some of their data on the cloud, compared to only 20 percent a year ago.

Flash storage is another technology organizations are using in their complex and resource-intensive HPC workloads.

While flash usage in HPC environments remains steady at 90 percent in basic data center deployments, the amount of data being placed on solid-state drives (SSDs) and other forms of flash storage is on the rise. Currently, most organizations (76 percent) have less than 20 percent of their data on flash but many more organizations expect more of their data to get soaked up by flash-enabled storage systems.

Twenty-five percent of respondents said they expect to pump 20 to 30 percent of their data into flash storage in 2018, while another 10 percent are estimating that they will place 20 to 40 percent of their data on flash storage. Accelerating file system metadata (54 percent) and application-specific data (54 percent) are the top flash use cases.

In terms of challenges, I/O (input/output) bottlenecks remain a big worry among 76 percent of organizations using running complex analytics downloads. Despite flash’s ability to transfer data at a brisk pace, at least compared to traditional hard disk drives, it’s not the cure-all for HPC environments, according to DDN.

“I/O performance is a huge bottleneck to unlocking the power of HPC applications in use today, and customers are beginning to realize that simply adding flash to the storage infrastructure isn’t delivering the anticipated application level improvements,” said Kurt Kuckein, director of marketing for DDN, in a statement. “Suppliers are starting to offer architectures that include flash tiers optimized specifically for NVM [non-volatile memory] and customers are actively pursuing implementations utilizing these technologies.”

Pedro Hernandez is a contributing editor at Enterprise Storage Forum. Follow him on Twitter @ecoINSITE.