Analyst company Statista estimates that in 2018, data center storage capacity stands at 1,450 exabytes, growing to a massive 2,300 exabytes by 2021. (That’s 1.45 and 2.3 zettabytes respectively.)
To say the least, this is a lot of data – and it's growing fast. In business storage, data growth spurs development of capacity and performance across all storage media. It also spurs development in software tools meant to manage all that data, including gleaning business value from unstructured data and staying in compliance.
What's Happening in Enterprise Data Storage?
To understand where the enterprise data storage market is today, let’s look at the major trends impacting data storage.
1. Tiering is Still King
Storage tiering is been around for decades as businesses stored their primary data on hard disk drives (HDD) and backed up to tape. Tiering has become far more sophisticated, but the principle remains the same.
Businesses ingest active data on SDD tiers and process them on SSD and/or high-performance HDD, then move to nearline storage using high capacity online HDD. This tier may be on the same storage system or within the data center. Archive the aging but still active data onto active archive tape or warm data cloud tiers, and store backup and long-term archival onto off-site tape or cloud-based cold storage.
2. SSD is Growing in the Business Sector
In the business market, NAND flash offers high durability and performance, and is especially popular for high transaction, low latency business applications.
SSDs populate all-flash storage arrays and Tier 0 and Tier 1 in hybrid arrays. IDC reported that the enterprise all-flash array market grew 75.7 percent in the first quarter of 2017 and hit $1.4 billion in revenue. Hybrid flash sales did even better at $2 billion; these sales continue to grow in late 2017 and 2018.
Flash is still more expensive than HDD, but this distinction is unraveling fast. The price gap has been narrowing for some time, flash is capable of higher capacity than HDD development, and it delivers the highest performance-to-capacity value of the two. (The world’s largest capacity media today is not HDD but SSD.) Flash developers remain focused on less cost and more capacity per unit, and advances in storage class memory (SCM) non-volatile storage significantly lower application latency.
3. HDDs are Not in Fact Dying
Despite the SSD success story, HDDs are not going away. HDD vendors are also making capacity gains by increasing areal density and packing more heads and disk into single drives.
And although SSDs are working to close the price gap, they still average 6.6 times higher cost than HDDs. And although SSDs are lowering their prices across the board, IDC forecasts that HDDs should cost 2.2 times less than SSDs by 2021.
Storage tiering is optimized for a mix of media, and HDDs make up a huge sector of media between flash and tape. HDD’s high capacity, reliability, fast performance, and economy will continue to be a part of this mix.
4. Tape Isn’t Dying Either
Tape is no longer general-purpose storage, but it serves an important purpose for active archives and long-term backup retention and archival.
Cloud cold storage tiers are gaining in popularity, but many active archive environments stay on-premise to assure sufficient bandwidth. Many tape deployments are also more efficient then disk-based storage systems. Behemoth tape libraries can take up a lot of room and energy, but most libraries and drives are smaller and take up less floorspace, cooling, and power. And tape media costs a fraction of the cost per gigabyte of SSDs and HDDs.
Speaking of tape media, the Linear Tape Open (LTO) consortium has done good work on LTO development roadmaps and keeping LTO drives and media interoperable between manufacturers. Capacity increases with each generation, and next-gen tape drives from IBM are using TMR (tunneling magnetoresistance) to grow capacity even further.
5. The Cloud Makes Giant Inroads
The cloud is high growth and high opportunity for data storage, but different models serve different storage needs. Organizations should know the difference when they decide to place an application or data in the cloud.
· Private clouds are a sole-tenant cloud environment located on-premise, usually in the primary data center. This environment runs cloud orchestration services and allows companies to use cloud architecture behind their firewall.
· Hybrid clouds and multi-cloud. These two are different animals: hybrid clouds orchestrate services between multiple clouds, while multi-cloud strategies simply mean a business keeps more than one cloud in its portfolio. Hybrid clouds practice some level of integration and orchestration between different clouds, such as integrating a customer’s private cloud with a vendor-owned cloud. Large hybrid clouds include clouds owned by SAP and IBM, who extend on-premise services from their customers’ private clouds to their own.
· Public clouds are multi-tenant clouds that serve a variety of workloads and use cases. I subdivide public clouds into two types: the hyperscaled public cloud and customized cloud service providers (CSPs). Hyperscaled public clouds are the generalist giants AWS, Azure, and Google. They are famous for their global reach and scalability, and their support of multiple computing environments including long-term data storage and SaaS, PaaS, and IaaS deployments.
· Customized CSPs are specialized public clouds that serve distinct customer use cases. These cases can be geographical locations, industry verticals, data workloads, and more. Although they are a type of public cloud, they are distinct from hyperscaled public clouds by their specialized focus. Their competitive differentiator is usually their specialty focus, customer service, and flexible SLAs with their customers.
6. Unstructured Data Management and Analysis
Unstructured data represents most of stored data in the enterprise to the tune of 85% and up. Since this data rapidly ages, within 1 to 3 months most of this unstructured data is on secondary systems and cold storage.
The issue is that much of that data holds business value, but it can be very difficult to locate and extract it. Developers are responding with data management and analytics tools that allow business users to gain actionable intelligence from unstructured data.
Unstructured data management tools discover and intelligently acts on unstructured data. For example, classification software discovers unstructured data on a variety of storage systems and analyzes it by metadata, content, and context. The software then carries out policy-driven actions like defensible deletions and moving data into storage that supports unstructured data analysis.
7. Governance and Compliance
Governance and compliance requirements deeply impact storage decisions. It is IT’s responsibility to work closely with business units and the legal department to understand data compliance requirements for regulations like SOX and HIPPA, and to adequately secure personally identifiable information (PII).
In addition, companies who do business in Europe are scrambling to comply with the EU’s General Data Protection Regulation (GDPR). The new regulation impacts data retention, archiving, and storage architectures; and packs a mean punch for companies in violation. These companies are looking to their IT departments, internal compliance processes, and their cloud providers and business partners to stay in compliance as much as possible.
And it’s not just external regulations that are putting on the pressure. Public companies are obligated to investigate fraud and corporate espionage; and even private companies need to remain alert to bad actors. These companies are investing in compliance software tools that act on pattern recognition and messaging cues in email and on social media.
Many Trends in Data Storage...
There are many more trends in data storage that are impacting businesses, like artificial intelligence and machine learning, the Internet of Things, software-defined storage, object storage, and hyperconvergence.
These trends make for exciting times in the business world. But keep this in mind: nothing happens overnight. If you keep up with data storage trends and understand your business’s technology needs, you can adopt new technologies when your business is ready for them.