Overview

The need for economic and operational efficiencies transcends all organizations. Commercial businesses, non-profits, government departments, and large and small enterprises are all striving to make the most, and more, of what they have. Economic conditions don’t matter either. In economically challenging times, tight budgets dictate efficiency in order to get tasks accomplished with limited funds. But in boom times, efficiency is equally important to maximizing value in terms of profit, time, or resources while executing on objectives. This is borne out by ESG research: ESG asked senior IT professionals in enterprise and midmarket organizations in North America and Western Europe to identify the most important considerations for justifying IT investments in 2009-2011. As Figure 1 shows, the top two priorities continue to be reducing operational expenditures and business process improvement.[1] Clearly, investments are made in efficiency-focused endeavors.

Efficiency and Hypervisors

The evolution of IT is reaching at least one true pinnacle of efficiency with server virtualization: the transformation from the “one application, one server” paradigm to running multiple applications on different operating systems on a single physical machine. This is possible because of a specialized software layer that separates functionality from specific hardware: the hypervisor. The term was coined to indicate a software layer that resides at a higher level than a simple “supervisor” that controls hardware. It is a layer of abstraction between the physical hardware and guest operating systems. It can do the heavy lifting that makes it possible to run UNIX on a Windows machine. In a purely physical server environment, the operating system is intimately aware of underlying hardware such as processors, drivers, bios, etc. In a virtualized server environment, hypervisors such as VMware vSphere, Microsoft Hyper-V, Citrix XenServer, and IBM PowerVM get in between these physical realities and the virtual machines (VMs) to provide a consistent platform regardless of the VM operating system. Since this is so accepted for servers and provides proven value to users, mightn’t the same hold true for storage?

The Storage Hypervisor

A storage hypervisor provides a similar layer of abstraction between the physical storage resources and the applications using them. Here’s how Wikipedia defines it:

The storage hypervisor, a centrally managed supervisory software program, provides a comprehensive set of storage control and monitoring functions that operate as a transparent virtual layer across consolidated [storage hardware] pools to improve their availability, speed, and utilization.[2]

This sounds so straightforward, almost innocuous, but it subsumes some very important attributes and is dramatically different from the way that most storage is managed today. For instance, a storage hypervisor is designed to be agnostic to, and to accommodate arrays from, different manufacturers that may be using different disk tiers (SSD, SAS, SATA) and storage network protocols (Fibre Channel, FCoE, iSCSI). Arrays that would be incompatible in a traditional physical environment are suddenly able to work together.

However, just because a definition exists doesn’t mean the complete reality exists (although a few vendors are beginning to name storage virtualization products this way, and there are green shoots of a storage hypervisor spring appearing). Some early storage virtualization products that deliver some of this functionality are array-based and don’t support multi-vendor storage; some are appliance-based and may support some other vendor’s storage. Some storage virtualization products are software-only, some are software built into an external controller, and a few are network appliances. These are definitely a start, and given the resounding success of hypervisors delivering server virtualization, doesn’t it make sense to take storage virtualization in the same direction?

What Do Users Need?

The challenges that IT managers face today are daunting and well documented, and as such will not be repeated in depth here. First, there is a constantly increasing need for additional resources to support growing data volumes based on both natural application growth and new workloads (think social media and big data). In many cases, organizations try to handle this by simply throwing more hardware at the problem, resulting in massively under-utilized assets. Next, operational processes have not caught up with technology innovations, so as IT service delivery becomes more agile (and cloud enters the fray, for instance), administrators struggle to manage with the same old, inflexible processes. And, of course, budget constraints are always an issue, even more so in recent years.

In addition, as virtual servers and cloud computing are improving provisioning and providing a higher level of services, users are beginning to expect “instant IT.” In the old days (let’s say 5 years ago—or now for those that have not taken the server virtualization plunge!), if a user wanted to launch a new application to support a business process, he/she had an expectation that it would take a while—weeks or months—to get through the normal channels. But with fast, easy provisioning made possible by virtualization, IT can spin up a new VM in minutes.

The New Normal

Server virtualization is revolutionizing IT. It has enabled levels of efficiency (for which, read cost savings) never before dreamed of: reducing physical server needs by 50%, 60%, or even 80% and freeing up staff members from many management tasks. In the past, many business decisions were made based on what IT was practically able to do. Now, IT is more able to adjust to serve the business.

Of particular note is the fact that virtualization benefits actually increase as virtual deployments grow and mature. This was documented in ESG research which led to the creation of ESG’s Server Virtualization Maturity Model.[3] ESG was able to separate respondents into three groups based on the extent of their virtualization deployments: advanced (25% of those surveyed), progressing (53%), and basic (22%). The criteria used were:

• Scope of deployment, measured by the percent of servers virtualized
• Virtual production ratio, measured by the percent of VMs running in the production environment
• Efficiency, measured by the virtual-to-physical server consolidation ratio
• Workload penetration, measured by the deployment of multiple virtual workloads, particularly mission critical ones

ESG research indicates that while lower capital and operational costs and greater IT efficiency accompany all deployments, only the advanced implementations are actually becoming dynamic IT environments and vastly improving application provisioning, maintenance, availability, and backup/recovery processes. The benefits increase as virtualization experience expands. This reality will soon become the new normal in IT.

Storage Hinders Progress

Sounds good, does it not? But it doesn’t look quite so rosy from the storage end of things. While new server strategies have transformed operations, storage implementations can often hinder progress. This is not wholly surprising. Storage architectures, implementations, and management techniques originally came from the monolithic mainframe era and, while huge improvements have been made, the underlying concepts upon which the storage architecture was originally designed are cracking under the weight of progress. Scale-up silos of proprietary disk were designed to be physically managed and mapped to individual servers, but that is no longer how the processing side of IT works.

Server virtualization users know this already. As Figure 2 shows, when asked which storage developments would enable wider server virtualization usage, at least 25% of respondents mentioned each of the following aspects: faster storage provisioning, more scalable storage infrastructure to support rapid VM growth, and increased storage virtualization.

Server Virtualization Pressures Storage

The virtualization of servers impacts storage and storage decisions; and not in a good way. First, equipment needs: servers with direct-attached storage can’t move VMs to other servers because the data is only available on that server. What good is the ability to move your virtual workloads if the data can’t come, too? Networked storage is required to make that happen, and consequently virtual implementations require investments in more and better storage. In addition, storage (and networks) must handle higher IO and throughput densities because now multiple applications are sharing and better utilizing a single server’s capabilities. And when it comes to backup, that problem gets worse. The upshot, which is not often mentioned, is that the cost of upgrading storage to handle server virtualization can negate a significant portion of the savings that virtualization enables. Oops!

Servers are basically cheap and interoperable today, and the hypervisor provides the functionality to make applications work on any server. Server virtualization is hardware agnostic and, except for some speeds and feeds, servers aren’t really differentiated by functionality. On the other hand, storage is expensive, complex, proprietary, and incompatible; vendors sell a lot of it based not just on capacity, but on built-in features like snapshots, replication capability, storage tiering, etc. that is software-based but built into the array. As a result, buying and deploying servers is a pretty easy process, while buying and deploying storage is not. It’s a mismatch of virtual capabilities on the server side and primarily physical capabilities on the storage side. Storage can be a ball and chain keeping IT shops in the 20^th century instead of accommodating the 21^st century.

Storage needs to improve its résumé to match the job openings in server virtualization. Does it make sense to make these improvements within arrays? Or, having seen the impact that server hypervisor functionality has made, would a storage hypervisor make more sense? Clearly, it makes conceptual sense—both operational and financial—to complete the abstraction; to separate the storage functionality from the physical storage system by creating a storage hypervisor. Then, IT organizations could use whatever arrays they like, even supposedly incompatible ones. Put snapshots, tiering, load balancing, and the like in the hypervisor, and IT can easily move data between arrays, scale up or down to accommodate business needs, and deliver high service levels to users while minimizing cost and waste. It would make virtual storage as easy to provision and manage as virtual servers.

A Storage Hypervisor Vision

So, the concept of a storage hypervisor is generically good and appealing. But what should it actually look like, and what would the specific benefits be? It should start with the same kind of core components that server virtualization uses: a secure, scalable storage virtualization platform (like VMware vSphere and others in the server space) to consolidate workloads, and a management platform (VMware vCenter, etc.) to simplify and automate infrastructure tasks, provisioning, service delivery, compliance, data protection, etc. As with server virtualization, this would enable IT administrators without specific storage expertise to manage storage tasks. Below are some of the key capabilities that a storage hypervisor would provide:

• A common management platform to aggregate capacity from any hardware platform. This would be regardless of vendor or disk type and could be shared out to different servers. This capability would let IT select storage based on price and service regardless of vendor without worrying about the choice’s impact on processes and functionality.
• Storage provisioning and management automation. This relieves administrators of manually creating array groups, allocating LUNs, partitioning volumes, creating RAID sets, and performing other complicated tasks designed to provide the capacity required for each application at the agreed performance and availability levels. Today, provisioning is often little more than an educated guessing game that requires estimating how much capacity an application will need today, next week, and next year. Over-provisioning is a common strategy for ensuring that applications don’t run out of capacity, but it is extremely inefficient as wasted resources sit idle. In contrast, a storage hypervisor would use built-in intelligence to provision storage for applications and servers, automatically selecting a combination of disks to achieve performance, availability, and cost objectives. It would also rearrange volumes to maximize performance and efficiency.
• Transparent data mobility across storage tiers and arrays without shifting or losing functionality. Like a server hypervisor, a storage hypervisor must provide its own functionality, not count on what the underlying arrays can do, in order to be vendor-neutral. For example, a storage hypervisor would free administrators from having to map virtual volumes to physical volumes to retain vendor-specific snapshots. In addition, this would reduce the total cost of ownership by eliminating array-specific software licenses to provide features such as multi-pathing, thin provisioning, automated tiering, etc.
• The ability to pool storage resources from various arrays and vendors, and across data centers, to be accessed from anywhere. This would enable the creation of a single virtual data center made up of geographically distributed locations. If not yet quite a data center without walls, at least the walls can be over the horizon! Most storage virtualization solutions are restricted to the data center by physical array limitations. Server clustering allows distributed servers to depend on each other for high availability and transparent movement of workloads, but storage at the site is the boundary. If that boundary is eliminated and data centers 50 km apart can serve that function for each other, then multi-data-center failover is possible without expensive add-on mirroring and automation products. This makes disaster recovery, failover, and high availability much simpler to achieve. Now you can think of having a highly available, virtual storage cloud that is stretched across geographic boundaries.
• Similar capital and operational cost benefits as the server hypervisor. This would minimize the number of arrays required, allowing IT to leverage price competition, and simplify operations. IT would benefit from “array meritocracy.”

What to Look For

Some key capabilities are required in any viable storage hypervisor. IT departments depend on storage for more than the high level functions described above; the actual methods of achieving these objectives currently lie in array-based software features, so they would need to be included within the hypervisor. These include:

• Application integration with snapshots for backup and cloning, with options for when to snap, how long to retain snapshots, and when to back up offsite.
• Integrated snapshot recovery management.
• Automated storage tiering to balance performance and capacity requirements.
• Thin provisioning for greater efficiency.
• Deep integration with server hypervisor capabilities. This would include virtual array integration and data protection (such as integration with tools like VMware vStorage APIs for array integration) and management platform plug-ins. This would let the server hypervisor speed operations while consuming less processing power, memory, and storage network bandwidth.
• Synchronous or asynchronous mirroring across sites, using virtualization to allow primary site tier-1 storage from vendor A to, for instance, mirror at a secondary site to tier-2 storage from vendor B.
• Integration with site switching automation (examples are VMware Site Recovery Manager and IBM Tivoli System Automation) for complete server, storage, and network failover. Taken to its logical conclusion, this enables virtually any location, regardless of distance or infrastructure type, to provide high availability and failover services in case of disaster.
• Intuitive management features such as providing visibility across the entire SAN topology with drill-down on individual components and integrated virtual volume performance analysis to speed problem resolution.

A few vendors have already begun to start thinking in these terms; there are the industry giants—IBM with its SAN Volume Controller, EMC with VPLEX, HDS with its Universal Storage Platform-V—as well as some smaller software-only players such as DataCore. As the capabilities of these various approaches not only expand but become known and understood better, the end-user opportunity for improving efficiency and simplifying operations is simply monumental.

The Bigger Truth

The concept of a storage hypervisor is not just semantics. It is not just another way to market something that already exists or to ride the wave of a currently trendy IT term. A storage hypervisor has substantial, actual operational and business value. While some storage virtualization capabilities are available currently, they typically have limitations. Organizations have now experienced a good taste of the benefits of server virtualization with its hypervisor-based architecture and, in many cases, the results have been truly impressive: dramatic savings in both CAPEX and OPEX, vastly improved flexibility and mobility, faster provisioning of resources and ultimately of services delivered to the business, and advances in data protection. The storage hypervisor is a natural next step and it can provide a similar leap forward.

The combination of benefits from server and storage hypervisors can provide the sort of truly efficient infrastructure utility that has been promised for more than a decade. Both capabilities are needed to achieve flexible IT pools. As long as IT organizations have to manually intervene to ensure accurate, optimized, and smooth operations, then the “promised land” has definitely not been reached. Task automation is the foundation for any infrastructure that is truly in service to the business. Storage virtualization itself is a start—and has become endemic because of the inherent value and functionalities it provides—but it is ultimately only virtualizing a component and not a full system. By comparison, a competent storage hypervisor is the route to a fully flexible storage infrastructure that can help create a cross-site “‘storage cloud” with pooled resources accessible from anywhere, while providing a new approach to high-availability and workload flexibility. It can extend freedom of choice regarding storage and enable easy re-purposing of arrays for investment protection. This is one of those areas where technological possibility has now caught up with logical business desire. Storage hypervisors are logical, vital, and just plain sensible. Naturally, their impact will be earliest and greatest in large, complex IT environments, but, as with server hypervisors, the benefits will cascade broadly across the industry over time.

It’s somewhat ironic that the first virtual operating systems (SVS and MVS for IBM’s System/370 in 1972) were for mainframes; four decades later, it’s possible to argue that server virtualization is creating “software mainframes.” As we put IT back together again, it is crucial that we consolidate the management of storage just as much as anything else.

[2] http://en.wikipedia.org/wiki/Storage_hypervisor

[3] Source: ESG Research Report, The Evolution of Server Virtualization, November 2010. ]]> http://www.enterprisestrategygroup.com/2011/10/the-relevance-and-value-of-a-%e2%80%9cstorage-hypervisor%e2%80%9d-virtualized-management-for-more-than-just-servers/feed/ 1 http://www.enterprisestrategygroup.com/2011/09/classifying-database-data/ http://www.enterprisestrategygroup.com/2011/09/classifying-database-data/#comments Mon, 26 Sep 2011 16:44:34 +0000 Julie Lockner http://www.enterprisestrategygroup.com/?p=25252 Organizations are also looking to gain control of the sheer volume of data flooding their data centers. Facing stringent compliance regulations, organizations in many industries are retaining massive amounts of data, and are becoming acutely aware of the problems they will encounter as they attempt to manage such vast quantities of information. Until an organization knows exactly what kind of data it has stored in its database and where the data resides, it cannot manage the data or apply cost-cutting technologies such as virtualization and tiered storage solutions to it. Data classification can help.

Introduction

For years, an organization’s storage management strategy consisted of nothing more than using data’s age as its primary means of classification. Older data was summarily deleted or transferred to alternative methods of storage (e.g., tapes), regardless of its business value. As a result, large stores of corporate knowledge were misplaced or lost. However, facing both internal and external pressures to increase data security, and retain data for longer periods of time to meet compliance requirements, more and more organizations are rethinking their existing data classification strategies.

One of the key drivers in this evolution of data classification is the desire to spend IT dollars wisely, optimize storage management, and minimize storage costs wherever possible. Storing all data at the same level or tier provides little value as unused/underused data consumes valuable storage space. Identifying the various “types” of data stored sets an organization up to store that data in the most cost-effective way possible.

Organizations are also looking to gain control of the sheer volume of data flooding their data centers. Facing stringent compliance regulations, organizations in many industries are retaining massive amounts of data, and are becoming acutely aware of the problems they will encounter as they attempt to manage such vast quantities of information. In response to a recent ESG survey, respondents stated that managing data growth and more efficiently managing the data that already exists in their databases were among their top challenges, pointing to the immense size of the databases they now manage.[1]

Unfortunately, while the amount of data continues to multiply by astounding amounts, few organizations have sound data classification processes in place to help them fully identify and optimize this data. Such growth is a double-edged sword; while the additional data may, in some cases, provide enormous opportunity for activities such as data mining, until an organization knows exactly what kind of data it has stored in its database and where the data resides, it cannot manage the data or apply cost-cutting technologies such as virtualization and tiered storage solutions to it. Therefore, the first step in any data optimization process is classification.

Why Classify Corporate Data?

First and foremost, a data classification initiative enables an organization to understand its data. This, in turn, enables activities such as identifying and deleting unnecessary data, and implementing the most cost-effective storage hierarchy for data that is of value to the organization. It also ensures that important data remains available to its users.

For most organizations, the main factor driving a data classification initiative is the need to optimize data storage to its fullest. Companies don’t want to spend their IT budgets on storing data that no longer serves any purpose, and they are well aware that it is cost-prohibitive to store all data under the auspices of a “tier-1 only” infrastructure.  Higher performing tier- 1 storage systems cost more per gigabyte. Companies want to be sure that these high performers are leveraged optimally. Organizations also want to ensure that their valuable intellectual assets are well-protected, easily accessible, and stored in accordance with all compliance requirements.

Initiating a Data Classification Program

There are a number of ways to initiate a data classification program. Classifying the entire database is one approach; however, as organizations consolidate multiple business processes into the same database application, they may discover that same databases store sensitive information in one table and public data in another. This could result in unnecessary, inappropriate, or excessive action taken against data in the future. In such situations, organizations must resort to other means for classifying their data.

For those seeking to manage data growth, identifying a classification schema that collects retention and end-user access requirements will help IT deploy an ILM technical strategy and architecture.[2] For organizations interested in improving information security compliance, a data classification schema that maps out sensitivity to data fields in a database will help focus their database security solutions and initiatives on the highest risk elements.[3]

Regardless of the goal, the first step in the data classification process should always be to position the task as a companywide effort, with participation from every unit (e.g., business owners, IT, audit, DBAs, developers, legal, and records management) and a common understanding of the various classification levels to be implemented.

When it comes to the actual classification of data, it is also vital that those classifying it can identify all uses for, and users of, that data, and can pinpoint any possible dependencies the data may have with other stored information. Those involved with data classification must also be aware of business policies, and be able to classify the data under the agreed-upon schema.

One byproduct of the data classification process should be the development of a metadata repository that lists the databases, the applications that the database supports, the tables and business objects, retention requirements and information security policies should be established. This metadata repository can serve as the baseline set of information for the implementation of a data governance program.[4]

Key Steps in the Data Classification Process

Ideally, data classification should be done proactively during application deployment or development, and include a process that enables end-users to easily classify new data as they create it. If data classification occurs reactively, the process should target costly or at-risk applications and databases first.

Regardless of the approach taken, a few key steps must be followed:

• For each database, list tables should be sorted by schema and by size, including all indexes. The resulting list should highlight where the bulk of the data resides. For retroactive initiatives, a good approach is to initially focus only on tables that are 5 GB in size or larger, with the exception of tables that are known to contain regulated data.
• Next, assign a business object and information owner to the various components of the filtered list. For example, general ledger will be assigned to Finance, while sales orders would be owned by Sales. Information owners can then be assigned the task of identifying the various types of data, who (if anyone) will be using it, and any specific SLAs or other requirements that pertain to it. This will enable a valid classification to be assigned. Information owners should also take into account that, for data they must keep, end-user access requirements may change. Data may be heavily accessed during peak times only. Such changes in usage patterns allow IT teams to take advantage of tiered storage options using mixed price/performance storage media.
• If necessary, refer to documentation from developers or the source application’s ISV to gather a complete list of the tables that comprise the business object. For packaged applications, such as Oracle E-Business Suite, PeopleSoft, or Siebel, leveraging a tool can significantly speed up the business object-to-database table classification.
• After all tables are grouped by business object, it should be checked against the organization’s record retention schedule (RRS) to determine how long that business object needs to be retained. In some cases, large volumes of data may reside in a table that does not map to any business object in the RRS. This data could be working data or interface data, transient in nature, or log data. In this case, ask the respective business teams whether the data is used for any valid purpose. If so, define operational retention periods for this information as well.
• Once the bulk of the data has been classified and assigned both a legal and operational retention period, an appropriate storage solution can be designed according to the classifications now assigned to the data.

The Bigger Truth

Organizations are continuously seeking new ways to spend their storage management dollars wisely. Data classification, which enables them to “layer” data in cost-appropriate storage and service “tiers,” is an ideal way to both cut costs and regain control of the massive amounts of data under management. A carefully-planned, structured approach to data classification is vital for maximum leverage of such an initiative.

With the right team comprised of representatives from all disciplines, a comprehensive viewpoint can be incorporated into the data classification taxonomy. This is needed to effectively set up categories with regard to value and relevance aligned with the business while getting a sense of the quantity of data in each category. With this base information, enterprise architects can work more effectively with capacity planners in defining and scoping the technology needed while more accurately forecasting and budgeting technology spend by data category.

A corporate data governance framework can provide the communication guidelines, tools, processes, and organizational structure to facilitate proper management of data in accordance with company policy to result in more optimized investments in information technology. Data classification is the foundational component that ensures data governance can be implemented and executed for enduring success.

[2] See: ESG Market Landscape Report, Managing Database Growth – Optimizing Database Application Lifecycles, April 2011.

[3] See: ESG Market Landscape Report, Database Security: Keeping Data Safe, September 2011.

[4] See: ESG Market Report, Structured Data Reference Model: Managing Databases in the Efficient Data Center, June 2011.

See also: ESG Brief, Oracle ASM: A Frontrunner in Storage Management Solutions, September 2011. ]]> http://www.enterprisestrategygroup.com/2011/09/classifying-database-data/feed/ 0 http://www.enterprisestrategygroup.com/2011/08/the-rise-and-logic-of-the-all-flash-storage-array-solid-state-takes-center-stage/ http://www.enterprisestrategygroup.com/2011/08/the-rise-and-logic-of-the-all-flash-storage-array-solid-state-takes-center-stage/#comments Tue, 23 Aug 2011 13:05:30 +0000 Mark Peters http://www.enterprisestrategygroup.com/?p=24301 There’s an old saying about having your cake and eating it too. It’s something to which we all aspire or gravitate: to avoid compromise. If you can get more goodness rather than less, why wouldn’t you? That’s the essence of the proposition of the all-flash array, a storage system based entirely on solid state storage.

Introduction: Putting the Storage World into a Non-spin

The Underlying Idea

There’s an old saying about having your cake and eating it too. It’s something to which we all aspire or gravitate: to avoid compromise. If you can get more goodness rather than less, why wouldn’t you? That’s the essence of the proposition of the all-flash array, a storage system based entirely on solid state storage. Of course, many IT buyers might immediately say, “Sure, I’d love to have solid state for all my storage, but it’d be way too expensive.” That is understandable, but also a position that is about to be turned upside-down. It’s also the inspiration behind this paper: the storage world is on the brink of a large and significant change. It will not be a complete overnight change, but the reality of all-flash storage is here. These systems can bestow all the well-known value of solid state storage without added complexity, lack of functionality, or increased expenditure. A natural human tendency toward skepticism is to be expected, but if all the above were possible—that is, fully-solid-state storage arrays that are easy to use, fully featured, and do not carry a price premium over current performance spinning disk arrays—why wouldn’t you at least consider it? That is the magnitude of what is happening: IT managers are about to be able to have their storage cake and eat it too!

Some Perspective

Spinning disk technology has been with us since IBM introduced the RAMAC in 1956 and it is essentially unchanged today; sure, the platter sizes, rotational speeds, and materials, amongst others things, have changed; but the basic technology has not been altered for over five decades. Consequently, as the IT world has developed, the storage industry and users have worked hard to make hard disk drives (HDDs) manageable and useful. These efforts have essentially been attempts to mask the two key pains associated with HDD-based systems: their inability to deliver sufficient raw performance, and their inability to do more (in terms of working with faster CPUs and integrating with virtualized systems) as the IT world progresses. The impacts of these pains are seen in a range of operational issues that motivate change:

• “Ponzi” Storage: HDDs have many inherent restrictions and limitations; industry attempts to ameliorate these have created what might be called “Ponzi” storage where complexity is layered upon complexity in order to try to squeeze more blood out of the [HDD] stone. But this requires many unnatural acts as well.
• Unnatural Acts: HDDs are mechanical devices, each head and spindle can only do a set amount of work. Unfortunately, this means that things like short-stroking (only using a small percentage of a disk’s capacity in order to gain the IO that an extra spindle can serve) are necessary. Poor resource utilization is, of course, just another way of saying wasted money. Let’s say, for instance, that raw disk costs $10/GB; if you only use 25% of the available space in order to optimize performance, then the cost per GB is actually $40! And this is before the additional costs associated with wasted space, power, and cooling.
• Increasing Pains: From both a capacity and a performance viewpoint, the pain of HDDs is only increasing. Hence, the imperative for change is growing, too:
  • The rate at which storage capacity demand is increasing has accelerated over the last decade, and, contrary to the first few decades of computing, it is rising faster than the price-per-capacity is declining, which is not sustainable. Bigger disks help capacity, but not performance.
  • Random IO needs are actually increasing, driven by such things as multi-core processors, virtualization, and cloud implementations. This, in turn, only increases disk contention issues and therefore exacerbates the need for more over-provisioning and short-stroking. All of this leads to more money wasted on disks, power and space. Using less of a disk’s capacity helps performance but not economic value.
  • Putting these things together creates enormous strains in storage infrastructures: “access-ability” has not even kept close to the increase in HDD capacities. It’s been a bit like adding seats in a theater or bus without adding extra doors: it will inevitably lead to a slower audience or passenger arrival and departure process! Indeed, queuing dynamics will ensure that the impact is commensurately worse than linear as the crush (whether audience, passenger or data) produces deteriorating operational impacts. The immutable laws of physics mean that contention – even perhaps a scuffle that only introduces extra delays! – is inevitable.

Against this backdrop, the industry has developed a number of “band-aid” innovations that range from thin provisioning and deduplication (aimed at the capacity utilization issue) to automated tiering and solid-state caching (to alleviate performance woes). These are steps in the right direction, but they do not represent the quantum leap that all-flash storage arrays do.

Flash storage is the current iteration of a technology, solid state, that has been around for decades, although its earlier limited scale and extremely high cost restricted it to supporting and niche roles. That looks to be about to change. Ironically, although solid state has been around in some manner for decades, many users are perceiving it as new and vendors spent the early year or two of this new era of solid state deployment (kicked off by its integration into enterprise storage systems in 2007) having to defend some of its attributes such the need to manage endurance and differing read and write capabilities. Yet it’s worth remembering that a range of things we do with HDDs today (just think of RAID, for instance) are only there to ameliorate and cover up the inadequacies of spinning disks. All that said, most IT managers are well-disposed to flash (people essentially “get it”) but simply think it’s too expensive and therefore out of reach. It’s been put in the Ferrari bracket: something many of us would love to have, but we know that very high performance comes with a very high price and severe operational restrictions (like only two seats and no trunk space). Many vendors are now focused on bringing the Ferrari of solid state into the mainframe storage world, making it affordable with room for the family and shopping and still offering high performance.

This brings us back to the earlier question: what if this really could be? What if the obvious goodness of solid state (especially performance and utilization) came without all the downsides, especially cost? What if flash really could be a complete storage system rather than an HDD turbo-boost? To many, this may seem simultaneously preposterous and astoundingly appealing. As we investigate the reality and the IT values of this new approach, we suggest checking any preconceptions and prejudices at the door. As Oscar Wilde said “An idea that is not a little dangerous is unworthy of being called an idea at all.” Perhaps a storage system without anything that spins is an idea whose time has arrived? We have already seen signs of all-flash storage, but implementations have been mainly limited to relatively small capacity PCIe cards or flash appliances. The big and dramatic change this paper examines is the arrival of the all-flash enterprise storage array.

One final point should be made before getting to the details: this paper is not at all suggesting that we’re even close to, or even realistically contemplating, the day when the Smithsonian is the only place to see spinning disks. However, it is both feasible and logical to believe that the purposes for which spinning disks are used are going to change, and that we’re rapidly going to move to a situation where leading edge organizations are increasingly likely to serve most of their active IO from solid state storage (perhaps solid state tiers and caches today, but moving to all-flash storage arrays tomorrow). We are already accustomed to the differences between performance HDDs and capacity HDDs, and the former seem to have a terminal disease even if the actual expiration date is not precisely known yet. It is in that space that all-flash arrays make sense, whereas capacity disks look to have many years left to live.

How and Why All-flash Storage Arrays Make Sense

Prior Barriers

Far and away the most significant barrier to all-flash storage arrays being deployed before now has been the financial cost, both real and perceived. Because of its combination of high storage value (in terms of positive performance impact) and high price, flash has always been seen to be about fixing particular issues and/or addressing point applications. More recently, the technology has been employed more horizontally for its generic infrastructure value as a tiering and caching tool. And yet, logically, if the industry could go further, with the caveat that such an implementation would only ever gain market traction IF it can be made affordable (or, at least if the delta is small, worth affording for other reasons), that would be preferable. Other areas of early concern for flash vendors have been issues with reliability and longevity or endurance. Some of the concerns were valid, but there was also misunderstanding and FUD around them because these are issues that can be managed around (via wear leveling, over provisioning, cell management, etc.). And, as mentioned already, spinning disks have plenty of pains and drawbacks themselves, not least of which is that HDDs have actually deteriorated in terms of effective performance (IO per GB) as disk capacities have grown (as illustrated in the theater/bus analogies).

Another significant restriction on the adoption of all-flash storage arrays has been a lack of all the necessary enterprise storage functions that mid- and higher-end IT organizations look for in their storage subsystems, which would typically include:

• High Availability/Resilience: via clustered controllers and redundant components with automated failover
• Virtualized storage software functions: such things as thin provisioning, snapshots, etc.
• Scalability: preferably this should be non-disruptive, and capable of ranging from tens to hundreds of terabytes
• Integration: standard storage protocols like FC, iSCSI, NFS, etc.

Generally, these barriers have been, or are being, overcome. Although prospective users of all-flash storage arrays will no doubt demand proof points (both in an operational and business sense), the work that ESG has seen from vendors in this emerging space shows that all-flash storage arrays are [going to be] coming to the market fully loaded with the necessary functions and can also make financial sense when compared to high performance HDD-based systems.

Earlier and Alternative Approaches

The prima facie attractions of all-flash storage arrays have not precluded other less extensive implementations of solid state. From the first niche-application-focused solid state drives back in the 1980s and onwards, there has been some form of solid state appliance on the market invariably aimed at addressing very specific application-performance challenges.

A broader implementation model that is gaining popularity lately, and that can provide real benefits, is to use solid state as a cache and/or tiering tool. The Achilles Heel for such implementations is the ability to predict what data should be moved where and when; which has been why most HSM type implementations like this have either not worked well or have not been able to deliver optimal results. While there are improved software tools available, the approach is still clearly a compromise because it is inherently an odds game: users (or their software!) are trying to “guesstimate” the optimum placement of all data at all times. Since there are bound to be sub-optimal decisions made at some point (you can’t always know before it’s too late) applications often have to run on the assumption that the guesstimate might be incorrect and thus at times only provide application(s) with disk response/latency rather than flash response/latency (hence slowing applications because you don’t know which sort of response you’ll get and therefore you need to design for disk). To use another analogy, think of betting on a craps table: you can guarantee winning if you cover every bet, but then you’ve ruined the payback and original purpose. While performance can be improved, the implementation ratios advocated by vendors (5% flash and 95% disk is typical) can never achieve performance as good as an all-flash solution. No matter how generously you calculate the cache hit ratio, it’s likely that 25-50% of IO will be coming from disk, and that means that your worst-case latency will be the same, slowing the application.[1]

The latest alternative approach is the use of PCIe-based solid state in servers. As with all solid state, this can definitely provide good benefits although one way or another, it is still usually a matter of cherry picking the data or applications that are deemed worthy of the solid state boosts. Far more restrictive is the fact that this approach is not physically shareable across servers, which not only severely restricts or precludes the delivery of high availability, but makes all sorts of standard enterprise operations, such as backup, very challenging. Clearly, this approach doesn’t help for write-centric workloads as all IOs have to go to disk. Many users have spent years trying to escape DAS and the need to manage multiple islands of storage; this can obviate such progress.

This all brings us back to the recurring question: if an all-flash approach can do everything that these other alternatives can and do it economically (which means at or below the cost of a tiered methodology), then why would users want to bother with the other, inherently more restrictive, approaches? Tiering is popular right now, and it has relevance and value, but it is clearly (and by definition) also a halfway house. It acknowledges that flash is better while restricting its usage because of the price. If you could just jump to the logical end-game without having to juggle and balance the cost-benefit-analysis, why wouldn’t you?[2]

Technological Enablers

Aside from the price aspect itself, it is also apparent that a range of technology changes and improvements must have occurred or been optimized in order for all-flash storage arrays to be a realistic proposition. Some of the areas that make this new approach possible, and which different all-flash vendors [will] have to some degree are:

• Flash media improvements
  • Sheer data handling and processing speed of all aspects of the flash media and controller.Higher flash media capacities and lower prices (the combination permits significant scalability).
• Flash management improvements
  • Enhanced ability to manage longevity, endurance, and reliability.
  • On flash, the location of data does not matter. With HDDs, where you put data matters a lot.
  • Application usability is crucial. In other words, users can’t be expected to tweak their existing applications or to need an army of “white coats” to make everything work.
• Flash system attributes
  • Full storage virtualization for an all-flash storage array is, of course, a prerequisite since it is the key to providing many crucial and valuable storage services.
  • Software advances are important, both for optimizing the amount of data that is actually stored (minimizing it via compression and deduplication for example) as well as for ensuring ease of adoption and use (maximizing this via integrated management tools).
    • Compression, deduplication, and advanced thin provisioning are crucial elements for all-flash arrays to hit the price point that enables them to overcome the economic adoption hurdle. Even although thin provisioning may seem old hat, it is often underutilized in traditional HDD systems (whether by the manufacturer or user) because of understandable concerns over performance: more data on a drive is great, but as you increase capacity utilization, you strain performance and may need to either restrict its use and/or reserve spindles for IOPS (again, this can be best illustrated by the theater analogy).

Pricing Possibilities

The price/value for an all-flash array still has to make sense. However, it is important to learn to analyze the prices as more than just cost per gigabyte or at least to understand that raw GBs alone do not necessarily matter. After all, if a user can (for instance) use inline compression and deduplication in an all-flash array, then less actual back-end physical storage will be needed to store a given amount of front-end capacity. As mentioned earlier, if a user is paying $10 for a gigabyte of regular HDD and short stroking to use only 25% of the available capacity, then the real price per gigabyte is actually $40. The logic of this is clear, yet users are so used to evaluating storage purchases simply based on cost per gigabyte that such easy, and significant, calculations are sometimes overlooked or hard to accept. It’s a bit like having someone say they would not (“no way … never”) pay $25 for a gallon of a new fuel that’s fully compatible with their existing vehicle because “it’s just too much” without finding out how many miles per gallon per dollar their vehicle could achieve on that new fuel. Years and decades of learned behavior are hard to set aside, but the fact is that old tools and old thoughts cannot be used to measure new approaches. The relevant measure is effective cost for a given workload or performance, and the all-flash-array innovators have the ability here to send some shock-waves around the industry incumbents.

Beyond straightforward storage costs, there are a couple of important additional TCO advantages for flash:

• Power and space consumption: This is an order of magnitude or more less than for HDD arrays. Since it is common to find that it costs as much to rack, power, and cool an HDD array over three years as it does to buy it in the first place, this is a huge consideration in favor of flash systems.
• Server consolidation: If (as is possible) using flash produces enough performance that enables enough consolidation to reduce, say, Oracle nodes by 50% (both hardware and licenses), then the financial savings are such that they can make the math on the storage itself almost insignificant.

While it’s tempting to think that technology rules the IT space, far more often it is the financial aspects that really dominate. Certainly, every once in a while, a game changing technology emerges that addresses a critical business issue such as time or cost. Think of the virtualization of servers as a good example of delivering massive cost improvements, or data deduplication as a good example of beating time. In truth, while time was certainly the IT driver for the value of backup deduplication, the underlying business driver was financial. Why? Because prior to deduplicating backups, it wasn’t as if backups could not be completed in time; it was simply that the technology needed to complete them was so outlandishly expensive. Technologies can be IT game-changers, but better economics are invariably the root value.[3]

New Needs

New IT approaches also drive the move to flash. Take server virtualization, for example, which severely stresses the performance of storage systems. And, as Figure 1 shows, the move to virtualization has a long way to go yet.

Figure 1 shows that the number of users with more than 50% of their X86 servers virtualized is expected to jump from 14% in 2010 to 38% in 2012. Such consolidated compute capacity inherently drives increased performance demands into storage systems, and is exacerbated because the VMs also tend to drive randomization of IO. Put starkly, even with most organizations having well under 50% of their servers virtualized, storage has already become a critical choke point for VM scalability. And, to make matters worse (unless you are an all-flash array vendor), the odds are that those early-virtualized servers support some of the less important applications since most users have taken a conservative approach to the prioritization of what gets virtualized. So we might take an educated guess that perhaps less than 20% of the IO in an organization is virtualized to date and that is already causing a storage issue! If the remaining server virtualization represents the other 80% of IO, then it is hard to imagine serving that successfully without flash.

Buyer Advice: What’s Possible & What to Look For

Prospective users of new all-flash storage arrays should remember that this is not just about a nicer, “washes whiter” solid state disk drive. This is an ALL-flash storage system that replaces rather than enhances a performance HDD-based system. Performance is important, but this is about more than just performance.

A good way to think of things is in terms of means and ends. Previous SSD incarnations tended to view performance as an end in itself, but with all-flash storage arrays it is important to think of the performance as also being a means to achieve other ends. As we move along the natural continuum to using more and more silicon-based storage, leading edge IT operations will naturally want to consider maximizing their competitive advantage by moving into the new systems relatively early. For those users considering the purchase of an all-flash storage array as more become available through 2011 and 2012, what sort of things should they consider?

Operational and Financial Reality

Costs and Performance

There has always been a group of users and applications that are ready to pay for performance. Some 13 years after IBM introduced the RAMAC drive at just under $8K per megabyte, it came out with the high-performance 2301 drum which commanded nearly three times the cost per megabyte because it could offer a then-startling average access time of less than nine milliseconds. A decade later, StorageTek offered the first commercial DRAM-based solid state drive, which, more than 20 years after the RAMAC, could still command nearly $9k per megabyte. EMC, quickly followed by others, ushered in a new era of solid state use in 2007 when it started to add a limited number of flash drives as turbo boosts to its HDD systems.

All-flash is the next step, where solid state performance is not just an end, but also a means to deliver other things—including good economic value. Flash can, of course, easily compete with HDDs in terms of IO throughput; what’s intriguing is that flash can also compete in terms of TCO because it can be utilized better, has dramatically better OPEX structures, and because its performance characteristics are delivered without requiring the operational inefficiencies that are endemic to do so with spinning disks. All-flash storage arrays can also help to improve ROI, notably by driving the necessary performance to preclude storage from being an obstacle to server consolidation; such consolidation in turn leads to fewer CPU and server software licenses.

Real World Practicalities

With flash still on a steep product roadmap, there is a greater likelihood that compared to HDDs, it can deliver regular and dramatic cost/capacity improvements while at least maintaining performance levels (something HDDs are physically incapable of doing). And maintaining performance is a significant point: in all-flash implementations, consistent high speed is really more relevant and important than pure top speed since the products will be targeted at replacing high performance regular disk systems. In other words, one might think of comparing a racing car (point product solid state) with an everyday car (all-flash array): race cars are very good at what that do, but that’s all. They are not particularly cost effective when it comes to transporting people over distance and they are certainly not designed for everyday use. For flash to be a success as a mainstream “vehicle,” it needs to fit into the sports sedan rather than the Formula One class where the target is not top speed or performance at any cost, but is instead consistent high speed cruising, in town agility, safety, and plenty of room—all at an affordable price. Such long term, consistent high performance is itself demanded by many of the major drivers and initiatives within IT as we saw earlier in the paper.

The Argument for Solid State

It is interesting to step back to get a broader perspective; after all, nowhere is there any rule or law that says that storage must spin. It is simply that Winchester drives were the best option until now. Indeed, many in the industry will explain that the HDD was viewed as something of a stopgap and not expected to have a particularly long life. For instance, “They talked about spinning disks going away in three years when I worked on IBM’s 62GV in 1973.” [5]

Ironically, it is the biggest vendors of enterprise storage (all of whom have embraced solid state in some form or fashion) that are making the argument for solid state in general and by extension for all-flash storage arrays specifically. They are all showing by their actions that the best place from which to serve IO is silicon. In other words, all the major IT vendors are implicitly saying that all-flash is the way to go for storage: this is because their use of some flash shows they know that there is ultimately a better place for [some] data to go. They are, however, suggesting that not all data should go there because it’s too expensive (a cynic might also say it has something to do with protecting their margins elsewhere with spinning disks). Some, EMC and Oracle for instance, have made big plays and statements about their long term commitment to enhancing solid state and increasing its role in storage.

But, what if you could get all your data to that better place without spending extra or losing anything? Perhaps even gaining operational value? What if you could have your storage cake and eat it too?

The Bigger Truth

All-flash storage arrays are here, and they are going to impact the market; the only question is how fast it happens. That is a function of the price point. Why? Beyond that price hurdle, the move to flash is simply too logical and attractive. If the all-flash price is under or even just close to that of performance spinning disk, the question simply becomes “Why wouldn’t you make the change?” All else being equal, why wouldn’t you want a MacBook Air (another revolutionary product enabled by flash) rather than a regular laptop?

There’s more to the enterprise use of all-flash storage arrays than just TCO and OPEX, however; making the financial aspect work simply removes a reason to not consider the change. There are also positive reasons to consider these all-flash storage systems as their availability increases. Most notably, consider the bifurcation of performance and capacity (which current technology has forced together). Think of two storage hierarchies replacing the one that we typically think of: there should be an IO-, or work-, focused hierarchy (solid state) and a pure capacity, or long-term-retention, hierarchy (fat cheap disk, or tape). Moreover, there will be a broader change coming to IT if solid state takes over storage: databases, file systems, and virtualization infrastructure will all evolve to take better advantage of flash. Although that’s a long term view, it is made more likely because of the short term ease of adopting all-flash arrays, which can be plug-compatible with current infrastructures, as well as being equally or more reliable, with better OPEX and, best and most eye-opening of all, competitive CAPEX to traditional disk.

Let’s face it: the end game for storage is not spinning rust, however clever we’ve gotten at doing it! The way we’ve been running disk farms is a sheer waste of space both on the disks themselves and in the physical data center at large. It would be laughable or criminal if it were not for the fact that it’s been the only practical option for decades. Churchill once said that “Democracy is the worst form of government there is … except for all the others.” And that’s how disk storage has been, too. Of course, things won’t change overnight; it will still take time for many IT users to get their heads around the fact that these all-flash devices are full mainstream, general –purpose storage arrays (and not an appliance or an add-on). But, assuming the promised financial points can be achieved, the emotional attachments to HDDs and simplistic $/raw GB analyses will fade.

It looks like there will be a number of new players in this space pretty quickly. So what of the traditional storage incumbents?  Disk-centric vendors could soon face an innovators’ dilemma in that they have centuries of man-hours of engineering invested in spinning disk, and markets and margins to protect. History shows that the smarter ones will already be working on their [contingency] plans and will adapt to all-flash on the basis that, unpleasant as it might be, it is better to do so rather than wait to be relegated to storage also-rans. It is easy to get into hyperbole, but all-flash arrays genuinely look to have paradigm shift potential. As such, it is clear that IT purchasers should absolutely consider an all-flash array in their next performance storage decision.

[2] Without too much depth on this, there are always challenges with hybrid approaches; for instance, because of cache efficiency, disks favor large chunk sizes (keep the map in DRAM, contiguous access on disk) but solid-state favors small chunk sizes (don’t waste the flash for cold data adjacent to hot data). By trying to balance the needs of disk and flash, you end up with an implementation that is not ideal for either.

[3] Source: This paragraph is adapted from a companion ESG Market Report, How Economics Alter the Storage Landscape, May 2011.

[4] To add some technical detail, look for things such as a data layout that does not update in place, auto-refreshes (for data aging) and has efficient garbage collection and wear leveling. A data layout that aligns with flash page boundaries will avoid write amplification issues.

[5] Source: Chris Pollard in a comment on The Business of Storage Blog (“Storage…with an emphasis on age”), February 19, 2011. ]]> http://www.enterprisestrategygroup.com/2011/08/the-rise-and-logic-of-the-all-flash-storage-array-solid-state-takes-center-stage/feed/ 0 http://www.enterprisestrategygroup.com/2011/08/defining-tier-1-storage-in-the-modern-data-center/ http://www.enterprisestrategygroup.com/2011/08/defining-tier-1-storage-in-the-modern-data-center/#comments Fri, 19 Aug 2011 13:03:55 +0000 Mark Peters http://www.enterprisestrategygroup.com/?p=24139 The purpose of this paper is to define what constitutes “tier-1” storage in the modern IT world and in the data centers and services that support it. The term seems to be implicitly understood when discussing traditional environments, but what constitutes the norm in data centers has evolved rapidly over the last few years. What was a simple environment just a few years ago with mainframes or a few large servers to be supported has evolved into a complex web of virtual machines, clouds, and expanding user expectations. These three factors demand, and also create, flexibility, but they do so in a way that pushes a lack of predictability upon the storage infrastructure.

The Purpose & Nature of this Paper Storage in the New IT World The purpose of this paper is to define what constitutes “tier-1” storage in the modern IT world and in the data centers and services that support it. The term seems to be implicitly understood when discussing traditional environments, but what constitutes the norm in data centers has evolved rapidly over the last few years. What was a simple environment just a few years ago with mainframes or a few large servers to be supported has evolved into a complex web of virtual machines, clouds, and expanding user expectations. These three factors demand, and also create, flexibility, but they do so in a way that pushes a lack of predictability upon the storage infrastructure. Part of a Broader Categorization Initiative Of course, the impacts of the structural, application, and environmental changes are not just felt in the storage infrastructure. This paper is just one of a series in which ESG will attempt to define high-level criteria, attributes, and categorization for the infrastructure required to operate a “next-generation data center” be it on premises, in the cloud, or as some hybrid implementation.[1] These categorization exercises are necessary to help delineate those vendors and technologies that are only suited for past/current data center requirements from those that have what it takes to support a true next-generation data center. Partake in the Effort ESG’s objective is to run the overall undertaking with the support of the industry as these categorizations are certainly not specific to ESG and as such input from other professionals—users, vendors, commentators—can only be beneficial. Each document will frame the debate and, for lack of anything better, outline a proposed methodology and approach to the definitional conundrum.ESG will act as a clearing house and compositor and will update this paper (and the others as they become available) regularly. Please address comments and suggestions to the author (contact information available on the ESG website), as dialogue is encouraged to continually improve the definitions. A Line in the Sand Clearly, having tier-1 abilities in anything should be a function of delivering the requisite product functions that meet necessary attributes. This paper is a “line in the sand” for what those criteria should be for tier-1 storage. It does not purport to be the answer, although it may be! It is, however, an answer; a “straw horse” of definition based upon numerous industry conversations. Within each criterion, there will also be some variation of completeness of delivery and ESG has also provided guidelines for the levels of maturity within each criterion where possible: basic, moderate, and advanced. The intent is to use both the general criteria and the specific maturity/completeness as an objective gauge by which the whole industry can measure the suitability of products for the next-generation data center.

Tier-1 Storage in the New Data Center

The Beauty of Storage Tiers

Human beauty, so the saying goes, is in the eye of the beholder. It defies description, but we know it when we see it. Is that really so? Recent scientific investigation has attempted to deconstruct what defines beauty and there are rigorous mathematical ratios (such as the dimension of facial features, length of nose compared to width of eyebrows, etc.) that can be shown to be standard across subjects that multiple beholders perceive as beautiful. Now, of course this work and these relationships do not mean that all beautiful people look alike. But it does show that they have certain attributes that could be considered to be a “core beauty foundation” upon which their own individual facial nuances are layered.

What on earth does this have to do with tier-1 storage? The parallels are extremely pertinent.Tier-1 storage is something that people nod sagely about, and they know what it is when they are confronted by it. But, when asked for a definition, things get somewhat hazier. After all, one user’s tier-1 storage system may be another’s data dump depending on situations, budgets, applications, and industries. But, as with beauty, does this have to be the case? Surely there must be some absolutes for which nothing but the best will do? Is there a set of core tier-1 storage system attributes? As with the beautiful people, it would not mean that all tier-1 storage systems are the same, but rather it would mean that all such storage has certain prescribed attributes.

As we enter a new era of virtualized and “clouded” IT (with its burgeoning complexity, yet strident focus on flexibility and economic value) a term like “tier-1 storage” would be useful, provided it had actual meaning (as opposed to just a positive marketing nicety) attached to it. This is for two reasons:

1. There is a wave of recent, emerging, and brand-new storage providers (themselves built on waves of storage virtualization, scale up- and out- architectures, and [relatively] new technologies such as solid state) that might well be tier-1 storage, but which are implicitly precluded from joining the self-anointed “tier-1 club” today by assumption rather than fact.
2. It is very easy to throw around a descriptor such as “tier-1” without a definition. It is hard for tier-1 providers to push back on those who would unfairly seek their mantle.

A description allows everyone, existing and new vendors alike, to have a level semantic and technical field upon which to be measured, replacing glib assumption with specific assertion. Now, that would be a thing of beauty, wouldn’t it?

The Storage “Tier-archy:” What and Why?

Describing some storage as tier-1 is by, self-definition, to acknowledge that there are at least two tiers of storage. In reality, there are more, of course. The goal of this paper is not to focus to any great extent on the merits of those various tiers or of tiering in general, but rather to define the criteria necessary for tier-1 storage in a modern data center. Some brief references are necessary as background, but that is all.

For the purpose of this report, tier-1 does not necessarily mean the highest tier that a user needs or can afford since their particular needs might be met by some other technology even if they refer to it as “tier-1.” Instead, tier-1 storage is storage designed for mission-critical applications in extremely high performing, extremely highly available, and extremely well-protected data environments. Put another way, the tier-1 storage that this paper seeks to define is about quality tiering (which is desired or dogmatically required for an application) as opposed to forced or economic tiering (born of lesser needs or pragmatic choice).

New IT World = New Storage Needs

Traditionally, the top tier of storage has been very much about performance and reliability: IOPS and “five nines.” Of course, these requirements do not go away in a new world of virtualization and clouds! What makes the “new tier-1” for the “new data center” such a tough challenge is that it requires vendors to combine traditional attributes with exceptional agility and efficiency. As we’ve moved through the various eras of IT, from mainframe, to distributed, to internet-focus, to virtualized/cloud, the storage infrastructure to support each type of IT had to change, too. Just adding capacity to a “string” of tier-1 mainframe disk in the 1980s or 1990s could require weeks of planning and multiple visits to the change management meetings. Today, that sort of thing had better happen automatically and non-disruptively. We used to employ armies of specialists who did capacity planning and implementation. There were even guidelines as to how many terabytes each individual might conceivably manage. Today, we manage petabyte systems and expect dynamic provisioning tools and automated Quality of Service (QoS) software to manage things for us. Today, predictability and order are replaced with flexibility and the unexpected, and yet somehow the storage infrastructure has to flex and adapt accordingly. Virtualization, flexible business needs, and all sorts of clouds have made for extremely active, unpredictable, and variable demands upon storage. With so many changes in terms of what’s expected from a tier-1 storage system, we cannot afford to assume modern capabilities based on old criteria.

High Level Storage Needs[2]

How We Got Here

There are clearly some immediate areas within storage that need to be addressed for it to be a compliant and valuable contributor to this “new IT.” In order to understand exactly why, a little history is needed.

Commercial computing took hold when one single infrastructure stack executed one specific application for one specific purpose. The original mainframe was a glorified calculator. Centralized computing was predictable and controllable, albeit expensive. But it could be managed: one processor system and one IO subsystem.

Decentralized (or distributed) computing was developed largely to try to solve the economic challenges of centralized computing (essentially CAPEX) and yielded low-cost, commodity servers which we promptly plugged into proprietary, large, expensive, monolithic storage boxes. Servers became cheaper and more interoperable while storage has remained proprietary and expensive. In the old days, the server was the thing that cost all the money. You picked your server by your OS. You picked your OS by your application. Storage was a “peripheral.”

Today, servers are cheap and interoperable while storage is outlandishly expensive, complex, incompatible, and difficult. In many respects, it is the last bastion of IT awkwardness: the peripheral tail wagging the purposeful dog!

Where to Next?

Let’s take for granted that we want to virtualize (which could include clouds as an element) in general because it can bring efficiencies in asset utilization, take advantage of the commoditization of hardware, leverage common infrastructures, provide seamless mobility options, etc. If we can do all this, then we are set up to drive the next (higher) level of value where we can then aspire to provide infrastructure that:

• Self-optimizes: boxes that tune/reconfigure themselves for the workloads that are presented and change as those requirements change.
• Self-heals: infrastructure deals with fault scenarios autonomously, remapping/rebuilding itself so that the application is not affected.
• Scales dynamically: up or down, in or out; infrastructure that extends—virtually—to whatever requirements the workload(s) presents.
• Self-manages: adapts to changing scenarios based on policy and enforces those policies via automation.

A Note on Finances

One thing that has not changed, and indeed which is becoming more crucial by the day because of the well known gap between the demand for IT and the resources to supply it, is the need for all storage to make financial sense. We may couch our conversation in terms of the “right data” and the “right place,” but those are really symptoms; the underlying cause is the simple fact that no one can afford to have everything on main memory, which is what would occur (putting things like back up, replication, and volatility to one side) if storage were free. A tier or level of storage can be built from a wide variety of storage products and the number of choices has never been greater. Choices range from ultra-high capacity, low cost, lower performance storage devices to subsystems with advanced data management functionality, scalable capacity, and very high levels of performance and data protection.

The need for financial efficiency is clear. At the same time, the needs of the “new data center” (that is, some mix of clouds, virtualization, maybe ITaaS, and definitely burgeoning needs and expectations) are for extreme flexibility and agility. Balancing the two (and, of course, doing so with high levels of reliability and availability) is the job of the IT organization and ESG research (see Figure 1) shows that the needs for efficiency (cost control) and agility (business process improvement) are indeed the top two business initiatives that impact IT spending decisions.

New Data Center Tier-1 Storage Criteria Defined

For the purposes of brevity, we shall assume that the general concepts of tiers and tiering are understood. The trade-off decisions among price, performance, availability and so on for various applications and users have been made; data classification has been done (where it can be in today, since automated and policy-based solutions that tier based on age and access patterns are taking over in the new data center); and now it’s time to evaluate which tier-1 storage to use for your mission critical needs.

As a side note, the next-generation storage infrastructure model will eventually need to be folded into the historical tiering model, or indeed supplant it. We need to take the manual labor out of managing storage. An effective policy-based classification and tiering requires a system that can accommodate the automation and orchestration of resources—you can’t effectively tier with stove-piped, fixed systems of old. As we shall see, this calls for tiers that can expand dynamically, absorb and return capacity, etc., in order to be truly effective in the new world.

Here are the criteria that should exist for a tier-1 next-generation storage infrastructure designed to address the demands of a virtualized, clouded, variable world, with flexible and unpredictable storage demands. A few general notes apply to the criteria:

1. Basic (traditional) attributes  such as high performance, high reliability and an easy to use management GUI are taken as givens.
2. Here, the focus is on the criteria that separate and distinguish the “new” tier-1 storage (whether block or file at the system view) from the traditional.
3. The sub-bullets shown under each criterion are the maturity levels (where applicable and where they have been defined to date).
4. There is a deliberately pedantic aspect to the terminology and, at times, a very specific delineation of functions. This is simply so that aspects that might all “hang together” automatically in one vendor’s implementation are not inadvertently assumed to do the same in all products. For instance configuration and provisioning might be integrated into a set up wizard, but that is not always the case; plus it helps to highlight the distinction of the new criteria from the traditional.
5. A less technically precise, but far more colloquially appealing, description of the main impact/benefit follows each of the criteria; a kind of “Cliffs Notes” version to glean the key points.

• Self-Configuration
  • This is about various levels of automated set up, and is designed to speed and simplify system adoption and growth. No more spreadsheets and whiteboards for system administrators to figure things out!

1. Automatic initialization.
2. Automatic configuration of new disk/controller/cache resources.
3. Auto load balancing and optimization of new resources.

• Dynamic Volume/File Provisioning
  • Flexible provisioning (by attribute and/or over time) accelerates and simplifies one of the most common storage administration tasks. The lower maturity levels are very common while the true full dynamic ability drives enhanced optimization and is operationally very valuable.

1. Can configure and present basic LUNs/volumes/files.
2. Ability to provision resources dynamically, without pre-planning, based upon policies/SLAs which can have differing performance and/or economic attributes.
3. Ability to dynamically and non-disruptively reconfigure/reprovision storage system resources to address new service level requirements.

• Self-optimizing and Tuning
  • Think of this as an engine management system that automatically helps to balance $/IOPS, $/GB and service levels in a user environment.

1. At the basic level, all things are treated identically.
2. Automatic tiering is based on pre-determined schedules which may include some on-the-fly reaction and some meritocracy such as time-based auto-tiering.
3. Can dynamically and perpetually configure/reconfigure the storage system to optimize the use of available resources based on QoS, performance and/or economic parameters.

• Sub-LUN Optimization (of Capacity and Other Resources)
  • An enhanced, more granular management technique that improves the ability to balance $/IOPS, service levels, and $/GB with the aim of delivering satisfactory service levels at the lowest cost. Multiple workloads with varying QoS levels can thereby reside on one storage system while retaining optimum efficiency.

1. Multiple [sub] LUNs can share a single disk.
2. Multiple [sub] LUNs of varying QoS can share a single disk, reducing the need to provision and manage separate pools for each QoS.
3. Sub-LUN/volume elements are perpetually optimized at granularly, each with QoS, capabilities, and autonomous attributes derived from the master volume. In other words, multiple sub-LUN elements that have different QoS needs can share common disk resources without prescribed pools or reservations.

• Secure Multi-tenancy
  • This capability is about securing the visibility to, and interaction with, data belonging to any particular user on a shared system.

1. Basic level is just physical domains.
2. Permission-based access to [virtual] independent storage domains securely implemented on shared physical assets.
3. Dynamic support for varying QoS across virtual domains.

• Scale-out Infrastructure
  • This is the ability to deploy new resources fast and as needed, and thereby to grow flexibly. A scale-out infrastructure is obviously one of the keys to supporting new, dynamic, and unpredictable capacity demands.

1. Basic level is limited to two controllers.
2. Ability to add controllers, cache, and storage devices horizontally.
3. Ability to add performance and capacity resources independently and dynamically, and for the system to automatically absorb and reconfigure new assets as part of the resource pool in real-time.

• Scalable Multi-tenancy
  • The combination of scaling and multi-tenancy provides additional benefits in terms of greater consolidation per physical asset, which can reduce hardware sprawl (and hence both CAPEX and associated OPEX) and management

1. Basic level only supports two controllers per system
2. Ability to scale transactions or workloads (themselves being transactional and/or sequential) by adding device, controller, cache and/or IO capabilities. This can be characterized as predictable scaling.
3. Dynamically integrate new assets into the overall resource pool and redistribute/reconfigure to automatically optimize the performance/SLAs of diverse and high-service-level workloads based on QoS policies. This can be characterized as the dynamic ability to address and manage unpredictable workload changes.

• High Availability (HA) and Planned Resilience
  • The benefit here is clear: it is about having tools to preclude the escalating negative impacts of prolonged downtime which is especially harmful to heavily-trafficked shared storage such as that which is found in highly virtualized and cloud environments.

1. Failure resilience, which maintains QoS even if major system components (cache, controller, etc.) fail.
2. Multi-site (anything more than two) remote data replication is crucial to ensuring a rapid return to service should an entire array or site fail (and even tier-1 systems can do so).
3. “Planned Resilience” concerns minimizing the real operational impact of issues. A “graceful” failure and a rapid return to service are more important than the number of “9s.” Whether it is a cloud or a “regular” data center operation, the idea of staying down for days is untenable.

• Optimized Asset Utilization
  • The main intent and benefit here is financial impact, including TCO and ROI; better utilization means buying less hardware and also aligns with a pay-as-go model which suits many modern budget cycles well.

1. Thin provisioning, so that real capacity is only consumed when data is actually written.
2. Ability to convert under-utilized (“fat”) assets to optimized (“thin”) utilization; other capacity reduction tools such as deduplication and compression can also apply.
3. Reclamation, dynamically and perpetually, releases under-utilized assets back to the overall pool.

• Virtualized  and Federated  Resource Pooling
  • This extends the ability to pool resources and serve workloads/applications across systems and distance, enabling additional economic improvement, flexibility, and even security.

1. Simply applies to LUNs/volumes
2. All LUNs or volumes can span all resources within the system
3. Virtual storage instances can move transparently and non-disruptively across physical systems within the broader/federated pool, and do so while retaining their QoS/policy/SLA attributes.

The goal is to achieve automated operational flexibility and scalability (a.k.a., IT agility) that is combined with the optimum use and re-use of all the resources (a.k.a., business efficiency). When used for mission-critical applications (those demanding top-notch performance and RAS), the criteria above constitute the range of elements that can be combined to produce varying levels of tier-1 storage in new world data centers.

The Bigger Truth

Often a paragon of progress in general, the IT industry can be surprisingly conservative or assumptive at times. A set of clear criteria for the prerequisite attributes of tier-1 storage in the new era of computing is a prime example. We don’t have an agreed set of such criteria even though this would help vendors and users alike. Instead, we typically simply assert “tier-1ness” (if you’re a vendor) or you’ll “know it when you see it” (if you’re an IT user).

With so much change occurring in IT and data centers—virtualization, clouds, and a necessary fixation on economic efficiency—this is neither a sensible nor a sustainable state of affairs. You would never think to measure the attributes of a car by the expectations or design specs of the mid 1970s, nor would you seek to procure wireless service for your iPhone or BlackBerry based on the dial-up Internet standards of even a decade ago. Things have changed. And yet those sorts of approaches are how we still tend to judge the uber-premium, absolutely positively, high end, tier-1 storage. We not only need criteria, we need the appropriate criteria for the emerging data center where uncertainty reigns supreme for storage demands and tier-1 storage had better be able to cope and respond so that users can meet their IT and business needs. Hopefully this paper will be cause for the industry to stop and to think. Hopefully, it provides at least some of the answer.

[2] Note: this section is an extract from the ESG Market Report, The Future of Storage in a Virtualized Data Center, January 2011. ]]> http://www.enterprisestrategygroup.com/2011/08/defining-tier-1-storage-in-the-modern-data-center/feed/ 0 http://www.enterprisestrategygroup.com/2011/08/application-retirement-options/ http://www.enterprisestrategygroup.com/2011/08/application-retirement-options/#comments Thu, 18 Aug 2011 15:51:44 +0000 Julie Lockner http://www.enterprisestrategygroup.com/?p=24144 As new applications and technologies flood the marketplace, organizations are hard-pressed to keep up with the rapidly-changing technological landscape.  Companies often find themselves adding newer, more sophisticated applications to their growing arsenal of tools, while continuing to retain redundant or unused ( legacy) applications for fear of losing access to the data associated with them.  While these applications have long since outlived their original purpose, they continue to absorb valuable time and resources, impact overall organizational agility, and add unnecessary financial burdens to already-strained IT budgets.

For most organizations, the solution to the unnecessary financial drain on data centers is obvious—retire unused applications and implement a solution that retains access to the data held within these applications.  But, before such an effort can be undertaken, it is necessary to identify those applications that are suitable for retirement, analyze the needs of various entities who must access the data stored in those applications, and develop a comprehensive plan for ensuring that the data in the applications can be accessed and leveraged as necessary, while retaining the original data in order to meet compliance standards.

Introduction

IT is About More than Technology

While it’s tempting to think that technology rules the IT space, the truth is that far more often, it is the financial aspects that really dominate. Sometimes, that’s obvious, other times it’s less so. Certainly, every once in a while, a “game changing” technology emerges that addresses a critical business issue such as time or cost. Think of the virtualization of servers as a good example of delivering massive cost improvements or data deduplication in the backup market as a good example of beating time. In truth, while time was certainly the IT driver for the value of backup deduplication, the underlying business driver was financial. Why? Because—prior to deduplicating backups—it wasn’t as if backups could not be completed in time; the technology needed to complete them was so outlandishly expensive. Technologies can be IT game-changers, but better economics are invariably the root value.

Today, we are facing another major challenge that is contextually similar to the server sprawl or incomplete backups that, respectively, spawned the success of server virtualization and backup deduplication. Storage demand continues to grow rapidly at a rate that exceeds its relative price decline. It is not a sustainable model: data growth trends are driving the need for another game-changing event. Data growth is outstripping IT CAPEX budgets, available IT space, management capabilities, and OPEX thresholds and the pressures to manage more data in cloud environments only makes the challenge more acute. We are approaching—some users are even at—a breaking point. As a result, data storage must undergo a massive leap in efficiency or the business advantages that could be realized from all that data will be lost.

A sea change is about to occur in primary storage, driven by the obvious mismatch between demand for capacity and the ability to supply that demand at a reasonable cost. There are only two variables here: since the price of storage isn’t declining sufficiently fast, the only other option is for the amount of data that actually gets stored to decrease. Doing this while still serving demand for up-front capacity means that the necessary sea change for storage is (once again) spelled D-E-D-U-P-L-I-C-A-T-I-O-N. However, this is not the technology that was designed for backup. Deduplication for primary storage has a different set of challenges—mainly mitigating or precluding performance impacts, and dealing with massive data sets. It also offers different opportunities, not only reducing traditional data center needs, but also adding an operational and economic enablement to cloud implementations. The storage vendor that gets this right could be the next game changer that will most probably gain similarly dominant market shares, much like other recent market game changers such as VMware and Data Domain.

Market Functions

Why were those companies game changers? Because of the problems they solved, markets that were previously unavailable—or simply did not exist before—suddenly became valid with significant opportunities. For example, not being able to complete a backup within a 24-hour period is a great example of a time problem as are the server deployment costs and subsequent underutilization that results in out of control expenses. In both examples, users/buyers invariably looked to current suppliers to provide a “fix” to the issues. The typical quick fix is to make incremental improvements to core products, providing the bare minimum incremental “feature.” Vendors continue to support and extend that bare minimum feature set until such time that the solution simply doesn’t work anymore—which is when the user/buyer will first truly look for an alternative.

Unfortunately, history demonstrates that, regardless of how much better (cheaper/faster) an alternative company’s product/technology/solution is versus the incumbent supplier’s, most buyers will almost always stick with the incumbent until the pain is too great or they outright fail. It’s the “Devil you know” applied to business, the “no-one ever got fired for buying vendor X” mantra. That is the power of incumbency and it’s why massive cost savings are so important: only overwhelming improvement will overcome natural inertia and conservatism. Huge, “slap in the face” economic improvements demand attention.

Indeed, ESG research has found that users report that cost reduction is the prime business initiative that impacts their storage spending, as Figure 1 shows.[1]

It’s also worth noting that the same research has, for the last few years, shown a greater relative emphasis on bringing down OPEX as opposed to CAPEX; no doubt as a realization that the lifetime costs of storage (space, power, and management) can be much greater than the initial purchase price.

Remember, also, that we only have storage tiers because storage isn’t free. Different types of storage exist only because of cost differentials; after all, if storage were free, we would all naturally put everything on the fastest devices possible. Only because storage has a price—indeed, a range of prices—do we think about where to put different data. We often disguise that as a conversation about performance, but that’s a symptom rather than a cause. The cause is money. By the way, just to extend this to its logical conclusion, even if storage were free, it still would not be free to operate. As mentioned, reducing OPEX is the most important aspect users mention in terms of overall cost reduction. As such, we need to be driving down operational costs as much, or more than, the capital costs.

The industry has done many clever things to address the escalating cost of storing data: dynamic tiering, thin provisioning, and the like. These methods provide only incremental cost improvements. What’s needed is a much more effective tool for addressing the storage growth problem.

So, how are we to address this challenge? What’s needed is what might be called “data optimization economics,” a potential kill shot that has the ability to address the root cause—the massive growth of data that need to be physically stored—and thereby produce the next great leap forward for IT economics.

“The Law of 10X”

For whatever reason, history tells us that when we are in a “good enough” market state, the only vendor combatants to successfully alter the status quo (and thereby steal appreciable share from the incumbents) are those that offer solutions that are 10X “better” than the competition. You can argue the degree of improvement—it may be 7X or 12X—but the point is that it is an immediate and extreme change that results in improved financial value; way more than could have been expected under the standard regime of incremental change . These are the only ways that market tectonics truly shift. While being able to provide such a leap forward does not guarantee success, not doing so does seem to relegate those would-be players to a life in the niches of the market, rarely providing them an opportunity for dramatic market penetration or riches.

Furthermore, all “10Xs” are not created equal. Simply having a 10X “something” is by no means a guarantee that the market will adopt an offer on a broad scale. For the last 20+ years in storage, most systems have performed at a level that’s “good enough” and, as such, simply having a system that is 10X faster would only be interesting to whichever niche within the market craves additional performance. But, where the “something” is a financial play that yields a 10X improvement, more of the market will listen. After all, most of the market is focused on cost reduction to some extent. Simply said, a 10X improvement does guarantee widespread adoption when it is driving hard capital cost and operational cost (both hard and soft) savings. It cannot be ignored.

Recent Proof Points

Let’s look at a couple of examples of the “Law of 10X” in the real world. Each of these is an example of the “data optimization economics” already mentioned and each also achieved first-mover-advantage. That means reaping massive rewards (market share, selling price, etc.) for the vendor in lock-step with the massive benefits enjoyed by IT users.

• Data Domain: Data Domain saw that the issue was not any intrinsic need for deduplication for better backup per se; instead, it solved the business issues of time and cost. Users could not complete backups in 24 hours (at least not without ludicrous costs) and so Data Domain delivered backup to disk, solving the time issue. It delivered on the “Law of 10X” and solved the broader economic problem with deduplication that drove the effective cost of disk much closer to that of tape. A huge percentage of users now buy deduplication technologies for their financial value—they save them money! As a first-mover, Data Domain took 90% or more of the value generated while it was independent and now, post acquisition, EMC probably still takes 90% of that value.
• VMware: Server virtualization is undoubtedly the biggest wave in IT at present. Yet it was not an overnight success; initially, it was something of a “cool toy” used by a small cadre of propeller heads, QA teams, and the like who could avoid buying 10X the equipment to get their projects and work done. The epiphany moment came when the economic—as opposed to technological—value of VMware was realized to be a mainstream market solution because the footprint and cost issues of server sprawl had become paramount in data centers. Virtual machines have no additional footprint and cost a great deal less than physical servers: 10X utilization with no footprint gain was an economic home run.

What Will Happen?

The Current Situation

Despite all the consolidation and marketing noise, the enterprise storage market has been settled, some might say stagnant, for a while. In this status quo, a “big picture view” reveals that:

• Everything is fast enough for the most part and where it’s not, there is investment in very expensive, albeit cost-effective, flash/solid state storage devices to at least address the niche need.
• Enterprise storage remains up around 100X the cost of consumer storage; there are many valid reasons for this, but it is a significant gating factor on businesses ability to keep up with growing capacity demands.
• As the world heads into the zettabyte era, relentless data growth—and the commensurate cost growth that accompanies it—is the major challenge in IT.

The collective impact of these points is that data optimization—and the massively favorable economic impact of such optimization—is hugely important in storage today. As such, it is a massive business opportunity. However, it’s not about deduplication technology (wonderful as it may be): it’s about the ability to deal with rampant data growth economically. The technology is the enabler, but the real business issue is to be able to keep up with data growth demand at an affordable cost, not just in terms of CAPEX but also from an OPEX perspective, too. Given that a range of high level industry projections show annual data creation being in the tens of zettabytes over the next decade and with the knowledge that typically only 20% or 30% of that data is unique, the impact of data optimization could preclude the necessity to physically store tens of zettabytes of duplicate data. It’s a huge number, a huge potential impact, and a huge market opportunity.

Data Domain (and its followers) taught the market how deduplication (data optimization) technology could be applied to disk to deliver huge (and impossible to ignore) financial leverage; but to this point (other than a couple of relatively immature, although impactful, offerings by NetApp and Oracle), it has only been applied to the very “end tier” of the data lifecycle (i.e., backup) which represents only a tiny portion of the total available disk-based storage market. The same thing needs to happen across all the other storage tiers and when it does, the 10X criteria will be readily met and the market impact will be enormous.

What Has to Happen

It’s simple: someone will figure out how to apply technologies to gain 10X economic advantage, without performance impact and at scale across the breadth of the primary storage market. Taking a page from the Data Domain playbook, someone (or someone else!) will do the same thing to the higher levels of the primary storage market (tier 0, and beyond), thus making the most expensive parts of the primary storage market far more attainable for those who need it by, for instance, even applying deduplication or compression to solid state and thereby extending its value. SSD and flash are today’s hot topics in terms of helping address the highest performance needs of users, but they remain priced in the stratosphere. Compression and deduplication technologies that can be applied to this part of the market could alter the economic dynamics of performance by more than 10X—maybe even an order of magnitude more.

As ever, first-mover advantage is available to any company that understands the “Law of 10X” and makes the land-grab while other market players are talking about science projects and roadmaps or simply trying to maintain the status quo.

Who Will Do It … and Why?

The “who” is hard to know. Given the dynamics of the primary storage market, it could be an existing big player (the likes of HP, EMC, IBM, Dell, and now even Oracle), a large pure-play (NetApp or HDS) or possibly one of the mid/smaller vendors (such as Nexsan, Xiotech, BlueArc, Coraid, or nascent “next-gen” technology providers). Market coverage and traction favors the first two categories. It will come down to who wants to merely defend their market position compared to who wishes to aggressively grow their market position at their competitors’ expense. The first-mover advantage is available to be claimed; what’s most likely is for disruptive primary data optimization on a broad scale to be claimed by a “major” in combination with a “next-gen” technology player, with others then scrambling to catch up as the market adopts and embraces this next “aha” technology and financial “no-brainer.”

This is, after all, how markets have invariably operated. Once a market ignites, the first-mover, rarely facing stiff price competition, tends to take 90% of the value generated in that market until such time that the market morphs (i.e., the problem that was solved is changed or disappears) or matures to a point where the solution is so commoditized that only a price leader can sustain a presence.

The opportunity for whichever vendor decides to grab the reins in order to “Data Domain/VMware” the primary data optimization market is enormous. The storage business represents tens of billions of dollars in annual spend—and it is growing. With the demand for data storage capacities soaring and yet with cost control paramount for users, there is a genuine opportunity for primary data optimization to cause a significant shift in storage vendor market shares. Even a few percentage points translate to multi-billion dollar annual revenues and almost certainly significant market valuation.

The Bigger Truth

This opportunity—to optimize the enormous amounts of data being stored—is based on a problem whose time has clearly come. It has massive financial implications (The Law of 10X) and it is relevant and unavoidable across the entire storage marketplace. It is a logical IT need with an abundantly clear financial requirement. It will happen. The questions are: who will be the big winner, who will lose, and when will it happen? The economics are compelling and the business impact is even more seductive, which means that the “when” is likely to be very soon.

The obvious need, the financial value to users, and the impact on the market are all nearly impossible to overstate. Economics always wins and, in this case, the winning economics have a realistic chance to alter the market landscape, awarding the first-mover huge top line and bottom line gains. It will change everything and upset (or at least re-arrange!) a $20B or more annual spend. It will definitely create net new value distribution in the market and it’s entirely conceivable that some of the biggest incumbents—those that look unassailable today—will get hurt (in terms of sales, market share, and valuation) unless they move very quickly for leadership.

Optimizing data volumes for financial benefit is not some passing need; for sure, it is categorically relevant to all the key IT moves today including virtualization, cloud, “big-data,” and next generation OSs, which are all driving increased data storage needs. Further, it will remain relevant in all conceivable future data storage and management scenarios. There is no forecast on earth that has IT and data demands shrinking; we have to find ways to manage and contain that growth, which is why this is such a crucial move. As we increase sharing, mobility, functionality, raw demands, and general IT expectations, the underlying and continuing truth is that money matters.  And with data demands growing exponentially, the extent to which money matters in the storage world is growing exponentially, too.

There’s a multi-billion dollar win that will accrue to whoever addresses it first—and well.