Everyone is trying to figure what will happen next. Just for fun I want to exercise my contrarian gene and look at a few things that might not happen.

First, we might not get to $2BN for the final sales price – that’s an awfully fat number for the thin 3PAR. Yeah, we’re close, but even in these stratospheric realms, surely we can’t keep moving in hundreds of millions of dollars…and another billion mark is a mental block as much as a financial hurdle.

Second, we might not have a rapid and immediate domino-effect as the world’s few monster IT organizations scramble to gobble up the few remaining small to mid sized storage players. There are not that many waiting to be asked to the dance – Compellent, Isilon, Xiotech, Pillar, BlueArc, Nexsan and the OEM suppliers – LSI, Dot Hill, Xyratex – spring to mind (apologies to any I missed). The high 3PAR price, at whatever final point, might make the market consolidation slower rather than faster, while everyone goes to check their coffers….everyone that is except (a) the folks in Freemont who will no doubt be busy counting their gains and whose Christmas lists and wills I’d now like to appear in! and (b) either Dell or HP, because one of them will presumably have a couple of billion (roughly) burning a hole in its corporate pockets, and also presumably a publicly avowed need that should be fulfilled – this might not be a bad thing for some of the folks already mentioned, although meeting the high-end capabilities of 3PAR will be hard for most to match. Maybe the loser can find a few [zillion] extra nickels and buy EMC?! 

Third, we might not see Pillar almost-automatically rolled into Oracle, as many pundits have proclaimed. Much as Larry Ellison (Pillar’s bank-roller) has plenty ‘in’ at Pillar, he’s not always taken his side investments to their ‘logical’ Oracle home; some that he backed, such as Salesforce and NetSuite, have done quite nicely for themselves – and Larry – elsewhere.

Fourth, we might not still be in the dark about 3PAR’s winning suitor for much longer – the bids have been pretty fast of late, so you have to figure we’re close to done.

Fifth, whomever wins out, this might not change the entire face of IT quite as much as the acres of print about it would suggest. It’ll be a good acquisition for either party I’m (pretty) sure – whether providing strength and its own IP for HP in a space it already occupies, or whether it’s allowing Dell to enter a space it does not yet properly compete in – but the world might not stop on its axis as a result.

And - yes – this is all speculation; so sixth and last, I might not of course be correct on any of it….

You can read Mark’s other blog entries at The Business of Storage.

via 3Par turns to HP for “superior” deal, snubbed Dell may raise bid – International Business Times.

via HP or Dell — which is the better suitor for 3Par? – Computerworld.

While I did not hear all of this morning’s call (I was dashing to catch a plane to get to the Flash Memory Summit) it was  – as is to be expected in the situation – pretty much motherhood and brochure-pie. But that doesn’t mean this isn’t an exceedingly interesting change, potentially good for all parties, including current and prospective users. Much has already been said about the product fit….about Dell being a full storage company….much has been speculated about the future of the Dell relationship with EMC….much has been quizzed about the channel route to market that will now [presumably]  be [more] used.  However, I think there are a couple of other interesting angles that actually have not been covered much and that deserve a little more virtual ink.

1. While this is a very interesting piece of the storage jigsaw, and with large potential upside for Dell, let’s keep some perspective. 3PAR represents only $200M or so of revenue right now. 3PAR has done well to hit this level of adoption and performance of course, and it has solid genuine high-end technology. Yeah, I realize Dell is buying it to grow it (as they did with EqualLogic) but that will inevitably take some time. The way some commentary has been today, you’d think that Dell had bought someone like NetApp or EMC! In other words, while this has significant interest and potential, its immediate direct storage market impact is less earth-shattering. However, tectonic shifts are possibly afoot….
2. With so much of the commentary being - no surprise – focused on the intricacies of the storage world, I think the bigger picture (the one where Dell wants to be a legitimate IBM or HP competitor across all IT including the ‘serious’ high end enterprise stuff)  is the more interesting one. Dell wants to be an IT ‘grown-up’ rather than something of a teenage ingenue – this may seem odd for such a successful multi-billion dollar organization, but there’s still plenty for Dell to do to reach this ‘ultra level’. And it wants to do it – things like buying Perot (services capability was a big tactical need) and now 3PAR (smaller today, but filling a key strategic hole) show that Dell is serious. It has the desire and it has the dough.

Overall, all acquisition journeys are tough through the integration phase, but this one seems to be worth the effort and looks good for both parties.

Read Mark’s other blog entries at The Business of Storage.

Is Data Replication Enough?

When asked about HA/DR planning, many IT managers immediately think about data replication tactics such as tape backup, off-site vaulting, and remote data replication. However, as important as an organization’s data is, it is only a part of the broader requirement for business continuity.  Even if the data is recovered, it is virtually useless if the appropriate application is not up and running. Similarly, applications need to be protected from system failures, human error, and natural or man-made disasters.  This paper examines strategies that help organizations measure the importance of applications (or the impact of their lack of availability) to their business, followed by a discussion of common HA/DR configuration options.

Business Needs

Successfully aligning an IT infrastructure to business needs in the context of HA/DR planning requires far more than executive-level oversight.  Business metrics for success and failure must be specifically linked with the details of both an infrastructure’s current and desired states. It’s the archetypal “do more with less” dilemma, only now with the added challenge to “do it better” as well.  Looking for economic and operational advantage must be balanced with the need to adequately protect data and applications. The highest priority applications to be protected are shown in Figure 1 and can clearly be seen to be at the very heart of any organization.

Protection Balanced Against Business Value

As the saying goes, you cannot manage what you cannot measure, so two things are important before anything else. The first is for a user to know the hardware and software required to manage and deliver IT services; and the second is to know the requirements to protect the business.  Both require financial linkage through measurable objectives to serve the business optimally.  One measure serves to enumerate the value that an application returns to the business when it is running and the other enumerates the expense to the business when it is not. Their values aren’t the same. Generally, the cost of application downtime over time is far greater than the potential revenue the application can create when running.  Therefore, the technologies used for business continuity can be thought of as “protection architectures.”

The first objective in setting the goals of an investment strategy for these protection architectures is to develop a cost justification model for the expenses required to protect each application. If the expense to protect the application is greater than the value the application provides to the business, plus the cost to recover it, then optimizing the protection architecture to reduce the expense associated with protection is in order.

From Value to Architecture

To build appropriate protection architectures, IT managers must know the business value of the applications they are trying to protect and align that with technologies that results in a cost justified level of protection. It is important to make an implicit point explicit here: not every application deserves the highest level of protection money can buy. That seems reasonable, even obvious. However, in countless data centers, there is a desire to deploy a HA/DR plan with a “best that money can buy” mentality, even though that can exceed the practical needs of many applications. Often—whether because of a lack of personnel or simply too many demands placed on the personnel that exist—there just isn’t enough time to think sufficiently about efficiency, which results in many data centers adopting a “one size fits all” strategy.  And not that many vendors are going to complain about users buying more than they could, or should, have!

So, if you wanted to get it right, where would you start?

Pragmatic Analysis and Planning

Step one, without a doubt, is to have a scheme to characterize and balance the value of the applications to be protected against the cost required to protect them. Different applications will have different values, but as they are analyzed, trends will begin to emerge.  That is the time to assign applications a “protection class of service” justifying the protection cost, linked to a reference architecture that contains cost to a known technological approach.  Think of it as a blueprint.

The best way to put some structure around the blueprint is to set some parameters for how you intend to protect each application; setting recovery objectives can do that. Recovery objectives include two major measurements that are used as the foundation for building protection architectures:

1. The Recovery Time Objective (RTO) is the duration of time and a service level within which a business process must be restored after a disaster (or disruption) to avoid unacceptable consequences associated with a break in business continuity.  For example, the RTO for a payroll function may be two days, whereas the RTO for sales order processing may be two hours.
2. The Recovery Point Objective (RPO) is the place in time (relative to the disaster) to which you plan to recover your data.  Different business functions may have different recovery point objectives.  RPO is expressed backward in time from the point of failure.  Once defined, it specifies the minimum frequency with which backup copies must be made.

As the data in Table 1 shows, RPO and RTO objectives can be used as a guideline by a system administrator in order to ensure that an organization is employing the appropriate protection tiers to meet the various application objectives.  The use of protection architectures defines how HA/DR solutions are used to physically and logically protect the applications to meet service levels defined by the RPOs and RTOs.

What to Look for in Optimum HA and DR Implementations

Once a business impact analysis has been completed, it will be apparent that not all applications are equal and that different applications should be mapped to different protection tiers. Of course, if resources were fully abundant and/or free, then everything could have the best possible protection: since that is never the case, the intent of all the planning is to ensure that the available resources are allocated in an optimum fashion to the various applications (from the invariably revenue-impacting “tier-1” through to less vital materials on tiers 3 and 4). The constraints that limit the world from being perfect range from the pragmatic (such as cost, number of sites, and data growth) to the more prosaic (such as process errors, virtualization, and the need for testing).

Even when the business impact analysis has been done well and the resources allocated optimally, there is unfortunately still plenty of room for things to go wrong and throw a spanner in the works of all the apparently wonderful RTO and RPO planning. And users know that there’s not a lot of room for error. ESG research shows the very tight parameters that businesses are working with—53% of the respondents to a recent ESG survey said their organization would suffer significant revenue loss or other adverse business impact if their Tier 1 applications were unavailable for anything from no time up to one hour.[1] And even a slight deterioration in that RTO objective would rapidly make things worse; some small issue that took downtime over one hour (but still less than three) would cause the “significant negative impact” to apply to an additional 21% of tier-1 applications. Not to mention that, by this stage, nearly half (47%) of tier-2 applications would also produce significant adverse impacts for their organizations. And what might the “small issues” be that could cause deterioration in the recovery time?  The range covers awareness of the issue; not having some mix of the right people, configurations, or patches available; and actual error diagnosis.

All of this leads to the realization that effective HA/DR needs to be supported by solutions that can help avoid the negative impacts of small things destroying a plan that was based on good diagnosis and looked great on the whiteboard! Such solutions which can dramatically enhance and protect application availability should do the following:

• Automate the process: while most users will want to be notified that the system is being recovered per whatever plan is in place, automation is a good way to ensure that pressure and mistakes don’t become a cocktail that causes errors to creep in to the HA/DR execution; automation reduces errors and will usually speed the process as well as reducing the reliance on specific individuals and teams.
• Accelerate the recovery: anything that can improve the speed of failover is invariably going to be a good thing. This will, of course, involve automation, but should also include solutions that mean failover is less of a move to something else and more of a switch over to something else that’s already there—this, if you will, is like having the spare tire already fitted rather than having to pull over, jack up the car, and fit it (even though it could be argued that having an available and suitable spare at all is at least some level of DR!). Put another way, the best form of reconnection to a new application and storage is no actual reconnection at all; merely a switch to another available connection.  
• Ensure advance and regular testing: both of the readiness and appropriateness of the response; readiness is testing that the system is functional, while appropriateness is testing that the system is fit for purpose.   The above solutions—in combination with a thorough business impact analysis—will provide an overall “best practice” approach to HA/DR … and help many an IT manager sleep better!

Symantec Offerings

While the purpose of this paper is mainly to inform users of what to look for with HA and DR, Symantec does have an impressive set of solutions to address the needs that have been outlined. While the key components are briefly summarized here, there is considerably more explanation of how they all function in the appendix to this paper:

• Veritas Cluster Server (VCS): this solution adds useful functionality to the basic automated failover requirements—it can move applications to the node that is most able to accept the new workload and can do that at local, synchronous (up to 100km) or asynchronous distances, utilizing either a spare node approach or rebalancing across all available nodes. Because Symantec tends to agnosticism whenever it can, the actual data can be replicated using just about any tool the user wants.
• Veritas Cluster File System: this is what provides the accelerated recovery; the clustering of file systems avoids many of the recovery steps involved in “classic” recovery processes (such as unmounting a file system, deporting and importing disk groups, and remounting the new file system) by, to continue the analogy, having the virtual “spare tire” already mounted.
• Fire Drill (this is in VCS) / Disaster Recovery Advisor: these are the testing solutions. The former can simulate a disaster to ensure that the clustering and failovers will work properly when needed; the latter supports VCS, but is separate and is all about whether the HA/DR configurations are fit for purpose. To use an analogy, rather than testing whether all the fire extinguishers and escape routes work (that’s Fire Drill), it calculates whether the available extinguishers and escape routes are sufficient to cope with a potential fire and the number of people to evacuate.
• Veritas Volume Replicator (VVR): which provides application and storage hardware independent long distance data replication. Volume Replicator offers flexibility to choose any mix of SAN based storage architectures and replicate data over existing IP networks; this enables the implementation of lower cost storage at the DR location, which can result in significant savings.

The Bigger Truth

As is so often in IT, laborious work is required to achieve the payoff of an optimum HA and DR strategy. Knowledge of the applications and their business needs and impacts is first and foremost. And then the art of possibility and pragmatism must be applied to the science of the facts. Choices must be made, especially if the resulting plans are to be done in an “economically optimum” manner. The solutions that Symantec offers can certainly help, especially since the company is agnostic in terms of the applications and server/storage hardware with which it has to work. Having essentially one tool that can cover HA and DR across a wide range of applications and scenarios and distances is an advantage that many users would value and do well to investigate.

However, the most important thing for users is to realize that planning and thoughtfulness are key to this process; the title of this paper refers not only to the “right” HA/DR, but also to “implementing” it. Like any insurance policy, it is not always fun to discuss or to pay for, but also like any insurance policy, it is equally important not just to have one, but to ensure that the one you have is adequate for your needs and that it is going to work properly when needed.

ADDITIONAL NOTES: COMMONLY USED HA/DR CONFIGURATIONS

The following information is designed to add additional technical detail and insight to the business aspects covered in the preceding ESG Brief. The material discusses various clustering and data replication architecture options.

Application recovery solutions are designed to protect mission-critical applications running at the primary data center from a disaster that no longer allows those applications to run at that location. This could be a disaster as simple as a backhoe cutting all of the communications cables outside the primary data center or disconnecting external users from the applications. It could also be as severe as a major natural disaster (earthquake, tsunami, hurricane, tornado, pandemic, etc.) or terrorist activity. Application recovery solutions are designed to automate the process of recovering from a disaster and to ensure that not only is the mission-critical data protected by using replication, but also that mission-critical applications using that data are highly available locally and remotely in the event of a disaster.

Supported Application Recovery Configurations

There are basically two types of configurations in the application recovery model: Metro Clusters and Global Clusters. The difference is in how a failover is treated when going from one site to another. Is there a degree of control involved or is it fully automated? Both methods support local failover within the site.

Metro Clusters extend a single cluster between multiple sites and act at all times like a single cluster. The operator can configure failover ordering to always failover between systems located within the primary site before failing over to systems located at the disaster recovery site. However, when an outage affects all servers at one site, a failover to the second site will occur automatically. The single cluster solution is typically deployed in a metropolitan area, with full synchronous mirroring or replication and very reliable communications links between sites. In such cases, the company is essentially expanding the concept of high availability to include more than one data center. Assuming that all the infrastructure components are solid and that a business need exists to have full automated failover to a remote site, this is a very viable solution.

The following sections provide more detailed information on various cluster configurations.

Metro Clusters

Metro Clusters are single clusters that have been extended to more than one site. A Metro Cluster behaves exactly the same way as a cluster in a single site in terms of failover behavior. The underlying data transport mechanism between the sites provides for slightly different configurations in terms of the storage and how data from the primary site is “copied” to the disaster recovery site. The data can be replicated using application-based, host-based, or array-based replication. Metro Clusters can provide metropolitan area disaster recovery.

Metro Cluster with Replication

A Metro Cluster with Replication is a single cluster spread across two physical locations. This configuration eliminates the single point of failure represented by a single data center as it assumes independent power and communications at each facility. The Metro Cluster provides extended area high availability, which gives it the capability to provide disaster recovery automation at metropolitan distances. The extent of separation between the two data centers depends on the risks that the company wants to protect its environment from, but is limited at the margin by the requirement for synchronous replication. This typically is metropolitan in nature—less than 80 kilometers (50 miles). The nodes within each data center share storage at that data center. There is no shared storage between data centers, nor is there an extended SAN between the data centers. A Metro Cluster uses data replication to assure data access to all nodes at each data center. In a Metro Cluster configuration, if an application or a system fails, the application is restarted on another system within the current primary site or zone. If the entire primary site fails, the application is automatically restarted on a system at the remote secondary site (which then becomes the primary). In the event of a storage failure at the primary site, the cluster will detect that there has been a failure and will perform the operations necessary to prepare the storage at the disaster recovery site for production use and then restart the application using that storage.

Synchronous data replication keeps the copies of data at the two data centers synchronized. Asynchronous replication cannot be used for a Metro Cluster due to the potential for data loss during an automatic failover between sites.  Replication can take place at the application, host, and storage levels. Application-level replication products, such as Oracle Data Guard, maintain consistent copies of data between systems at the SQL or database levels. Host-based replication products, such as Veritas Volume Replicator, maintain consistent storage at the logical volume level. Storage or array-based replication products such as EMC SRDF or Hitachi Data Systems TrueCopy maintain consistent copies of data at the disk or RAID LUN level. Supported distances between data centers for synchronous replication are approximately 80 kilometers (50 miles) or less. The Metro Cluster configuration provides both local high availability and disaster recovery functionality in a single cluster.

Notes Figure 1: Metro Cluster with Replication

A Metro Cluster configuration is appropriate in situations where dual dedicated cluster interconnects are available between the primary site and the disaster recovery secondary site, but there is no shared storage or SAN interconnect between the primary and secondary data centers. To understand how a Metro Cluster configuration works, consider the example of an Oracle database configured in a Veritas Cluster Server HA/DR Metro Cluster. The configuration has two system zones:

• Primary zone (zone 0) comprising nodes located at the primary site and attached to the primary storage
• Secondary zone (zone 1) comprising nodes located at the secondary site and attached to the secondary storage

Oracle is installed and configured on all nodes in the cluster. Oracle data is located on shared disks within each Metro Cluster zone and is synchronously replicated across Metro Cluster zones to ensure data concurrency. The Oracle service group (a service group within Veritas Cluster Server is the smallest unit of failover—it contains all of the resources that a mission-critical application needs to come online) is online on a system in the current primary zone and is configured to fail over in the cluster.

Notes Figure 2:  Metro Cluster with Replication Failover Example

In the event of a system or application failure, Veritas Cluster Server HA/DR attempts to fail over the Oracle service group to another system within the same Metro Cluster zone. However, in the event that Veritas Cluster Server HA/DR fails to find a failover target node within the primary Metro Cluster zone, Veritas Cluster Server HA/DR automatically switches the service group to a node in the current secondary Metro Cluster zone (zone 1). Veritas Cluster Server HA/DR also redirects clients by bringing up the application IP address(es) at the zone 1 data center. The Metro Cluster with Replication configuration requires Veritas Cluster Server HA/DR to be installed on each server at each location and supports the use of VCS HA/DR Fire Drill for automated testing.

Metro Cluster with Mirroring

The Metro Cluster with Mirroring is a single cluster that spans two or more physical locations, similar to the Metro Cluster. The storage at one location is mirrored to the other using a data replication technology. Mirroring is a synchronous process. Unlike Metro Cluster with Replication, the storage at both data centers is connected using an extended SAN. Effectively, the user is configuring a RAID 1 mirror between two arrays at different sites. All data is written simultaneously to both mirrors. Reads can be configured to be serviced by the site where the application is currently running. The separation between the data centers has a practical limit in terms of performance of about 80 kilometers (50 miles). A Metro Cluster with Mirroring can also be configured using the concept of zones to set failover ordering to always attempt failover locally first.

Notes Figure 3: Metro Cluster with Mirroring

A Metro Cluster configuration is appropriate in situations where dual dedicated cluster interconnects are available between the primary site and the disaster recovery secondary site and there is an extended SAN connecting the shared storage between the two sites. The storage at one site is configured so that it will be mirrored to the second site.

For failures not involving storage, a Metro Cluster with Mirroring will have the exact same failover behavior as the Metro Cluster with Replication.

Global Clusters link two (or more) independent local clusters to form a global failover relationship for specific applications. Failover between local systems in either cluster is fully automatic. Failover between sites (i.e., clusters) requires operator confirmation. Enabling confirmation gives operators the ability to ask “what should I do?” in the event that local failover protection can no longer protect the mission-critical applications, and a decision has to be made around whether to perform an application recovery at the disaster recovery location. An operator can thus indicate:

• “Yes,” we will incur a site failover. The local disaster is not expected to be resolved in the near future—or more specifically within our RTO limits—and it makes sense from a business perspective to bring all of the mission-critical applications up as quickly as possible at the disaster recovery site. In cases where an asynchronous replication technology is employed, this decision essentially accepts that the company will be coming up on somewhat out-of-date data.
• “No,” we believe that the disaster will be resolved shortly, within our RTO limits, and therefore we will not incur a site failover. It is less disruptive to stay at the primary site, even though the site is down, than it is to incur a site failover to the disaster recovery site, and then another one to come back to the primary data center once the disaster has been resolved.

The decision to place an operator in the decision path with global clustering is usually recommended as it allows the business to assess the severity of the disaster with respect to the RTO goals of the mission-critical applications and then act accordingly. Manual disaster recovery failover control also eliminates the risk of faulty communications between sites, triggering an automated response. In many situations, site-to-site communications are provided by an outside vendor and may not always be as reliable as desired.

Global Clusters

A Global Cluster is a collection of 2-4 clusters in separate locations linked together to enable intelligent application recovery over any distance, i.e., globally. There is no shared storage between the clusters in a Global Cluster. Each cluster within the Global Cluster is connected to its own shared storage. A Global Cluster has a single primary cluster (i.e., site), and up to three secondary clusters (sites). The storage within the Global Cluster is replicated, either synchronously or asynchronously, from the primary cluster to each of the other secondary clusters. Typically, asynchronous replication over the wide area network (WAN) connecting the data centers is used, but synchronous replication can be used for shorter distances (less than 80 kilometers/50 miles). Local clustering provides local failover for each site or building. Metro Cluster configurations offer protection against disasters affecting very small geographic regions. Large-scale disasters such as major floods, hurricanes, earthquakes, and acts of terrorism can cause outages for an entire city or region. In such situations, a company can ensure global availability by migrating applications to sites located a considerable distance apart. Over the past few years, the best practice regarding the minimum distance separating the primary and disaster recovery data centers has grown from 50 to over 200 miles. In a Global Cluster, if an application or a system fails, the application is migrated to another system within the same cluster. If the entire cluster fails, the enterprise can make a business decision as to whether or not to move operations to one of the alternate disaster recovery sites. If the decision is to move to a specific disaster recovery site, the application is automatically restarted on a system in the cluster at that disaster recovery site(s).

Notes Figure 4: Global Cluster

Now consider the example of an Oracle database configured in a Veritas Cluster Server HA/DR Global Cluster that connects two clusters together. Oracle is installed and configured in both clusters. Oracle data is located on shared disks within each cluster and replicated across clusters to ensure data concurrency. The Oracle service group is online on a system in cluster A and is configured to fail over globally, on clusters A and B. Veritas Cluster Server HA/DR continuously monitors and communicates events between clusters. Inter-cluster communication (ICMP-based) ensures that the Global Cluster is aware of the state of the global service group at all times.

Notes Figure 5: Global Cluster Failover Example

In the event of a system or application failure, Veritas Cluster Server HA/DR fails over the Oracle service group to another system in the same cluster. If the entire cluster fails, Veritas Cluster Server HA/DR alerts the operator and provides an opportunity for action. The operator may declare a “disaster,” which indicates that the primary data center has been lost, (at least as far as the application is concerned—this might perhaps be the result of a localized power outage) in which case operations are migrated to the disaster recovery data center automatically. Or the operator may declare an “outage” and decline to allow failover in cases where local restoration of service will happen in a short (or at least acceptable) period of time. In either case, a business decision must be made regarding whether or not a failover is in the enterprise’s best interests, considering RTO limits and what is known about the particular disaster. The Global Cluster configuration requires Veritas Cluster Server HA/DR installed on each server at each location and supports the use of VCS HA/DR Fire Drill for automated testing.

Replication Support in Metro and Global Clusters

Clustering technologies need to support a wide variety of replication products to completely automate the process of replication management and application startup at the remote site without the need for complicated manual recovery procedures involving storage and application administrators. The clustering solution should provide all the necessary logic to completely control the underlying replication configuration, whether that replication operates:

• At the storage array level (e.g., EMC SRDF)
• At the database level (e.g., Oracle Data Guard)
• Synchronously or asynchronously

Depending on the type of failure, this control may involve reversing the direction of the replication (otherwise known as role reversal, role swap, dynamic swap, or personality swap) or simply moving the data and applications back when the original site comes back online. This solution should also include the capability to select automatic or operator-confirmed site-to-site failover.

The EMC Guarantee

For many years, the total amount of storage users needed to support application and file environments wasn’t often a prime IT concern. The amount of raw storage needed to generate a given amount of usable storage was just math; utilization was known, but it didn’t really drive buying behavior. You just bought more when you needed it because getting the job done was all that mattered. The sheer growth of data and recent economic concerns have combined to change that; while getting the job done—effectiveness—is still, of course, crucial, doing it as efficiently as possible is now equally crucial in most IT organizations. EMC has just made a noticeable move in this arena.

The Unified Storage Guarantee: In a nutshell, EMC has announced that users can achieve any given level of usable storage space, yet use 20% less raw storage capacity on its unified storage platform (file and/or block) than with a competitive platform. How does it work? Users don’t need to run bake-offs on site—instead, the program is based on a capacity calculator tool EMC has developed, which itself is based on published specifications and—where possible—both its own and competitive best practices. If the tool doesn’t show a 20% advantage over the competitive solution, EMC will make up the difference at no charge to the customer.  In terms of details (the “nitty gritty” as EMC calls it), the guarantee applies to all EMC unified storage arrays and is aimed at “out of the box” configurations, with no caveats on the types of data customers will store.

Evaluating the Guarantee: EMC certainly has a plethora of analysis and background available to support its claim. The claim is based on having an architectural approach that avoids some of the capacity reservation requirements of its competitors for common functions such as snapshots and deduplication.  As one might expect in such a competitive market, EMC’s language is typically assertive. Its web page states:

“Efficiency is one of EMC’s fundamental strengths. Even though our competitors try to match it by altering their systems, turning off options, changing defaults, or tweaking configurations – no amount of adjustments can counter the EMC unified storage advantage.”

Although ESG has not formally verified the approach, the logic looks sound; however, of course, there is no point beating one’s head against a wall on the mathematical niceties. Since this is a guarantee, and since it is backed by EMC, a customer can’t lose. Unless EMC is embarking on a path to deliberately give away additional capacity, we can assume that there is indeed an achievable value here. There is little more to say about the guarantee itself; EMC is making a strong market play and is putting its money where its mouth is. Two rather more interesting questions are these:

• How relevant is EMC’s position in today’s market environment?
• Is this guarantee just a one-off positioning battle or is it an early indication of a bigger war to come?

User and Market Relevance

The market relevance of using 20% less capacity is extremely easy to determine and it goes to the heart of the efficiency need highlighted in the introduction to this paper. While using 20% less capacity is inherently good (as long as effectiveness is at least maintained) and while it would mean less equipment has to be bought (an obviously good thing for users), it is also a means to an end. That end is to reduce IT operational expenditures. Specifically, 20% less disk capacity means less power consumption, less cooling needed, less floor space required, less maintenance, and even potentially less management overhead; in other words, less operational expense to get the same work done. And that matters a lot: as ESG’s latest 2010 Spending Intentions survey data in Figure 1 shows, OPEX reduction is the current number one consideration in justifying IT investments, ahead of even CAPEX reduction.[1]

There are other insights as to why using 20% less capacity is not only inherently good, but also particularly relevant:

• When asked about their greatest challenge with respect to their storage environments, users’ responses were surprisingly basic, even arcane: the top challenges were not things like the growth of unstructured data, performance, or e-discovery—all these were important and made the list—the number one challenge was “storage system costs,” number two was “keeping pace with overall data growth,” and number four was “running out of physical space.”
• Responding to a question about areas for the most significant data storage investments in 2010-11, respondents had “more SAN implementations” at number one, followed by “data replication for offsite DR” and “storage virtualization” (probably no huge surprises and good for EMC in any case); however “more power efficient storage hardware” was a strong number four on the long list of responses and was ahead of such things as data reduction technologies, tiering, and improved storage management tools.

Clearly, using 20% less unified storage capacity is an extremely relevant proposition for EMC to bring to the marketplace.

Broader Implications

Following on from some of EMC’s announcements at EMC World (Boston, May 2010), it looks extremely likely that the relevance of this new guarantee is more likely the opening salvo in a wider campaign than it is a one-off marketing package. The overbearing importance of economic efficiency is shown in Figure 2. It shows that “cost reduction initiatives” are not just an aim within IT itself, but are also the most significant business driver impacting IT spending this year and next.

This makes the timing and intent of EMC’s unified guarantee rather intriguing as much for what’s not included as for what is. To be specific, the new guarantee is all about “out of the box” value, and it does not take into account the  incremental efficiency gains that will be the result of several features EMC is scheduled to make available this summer. The features include:

• Sub-LUN FAST: EMC’s Fully Automated Storage Tiering that allows sub-LUN/volume automated, real-time data migration and placement. This translates as the automated movement of data to the right place rather than trying to employ either post-event reaction (which will be too late to have the economic benefit that was desired) or pre-event prediction (which is impossible in dynamic environments). In most IT environments, perhaps 90% of the user data is “cool” or “cold” and only 10% is “warm” or “hot”—FAST makes tiering automated and simple. It can reduce overall costs by enabling customers to use more SATA drives in production workloads.
• FASTCache: The ability to flexibly use all or a percentage of a user’s EFD (Enterprise Flash Drive, EMC’s term for SSD) capacity as an extended cache pool. Flash drives can, in other words, be employed—flexibly—as either an extremely high performance “tier” of storage for a relatively limited capacity or as a tool to provide a turbo-boost of performance across an entire storage system.  The expectations for this capability are not trivial; although still in formal testing, relatively small FASTCache usage is expected to be able to reduce backend disk IO by over 60% in medium-to-large VMware View environments, while a medium-to-large Oracle OLTP environment could reduce its storage footprint by 50% and double performance.
• Compression: This will be a tool for primary data and it will be an ability that is new in terms of mainstream use by a major storage vendor. An often overlooked function, data compression can significantly further reduce storage capacity needs.

While the potential efficiency gains of these new features are themselves dramatic, they will be coupled with other existing EMC capabilities, including Virtual (thin) Provisioning and deduplication (using Data Domain and Avamar). Looked at in total, it is clear that EMC is driving efficiency as a core value for its mid-tier storage customers. Although efficiency will be promoted, it will not be at the cost of effectiveness. For example, the use of FASTCache and Sub-LUN FAST promises improved performance; this is especially useful for demanding and unpredictable environments, exemplified by virtualized server operations. Combining greater efficiency with improved performance fits exactly with what IT practitioners—squeezed as they are from all sides—both need and want.

None of the capabilities of these tools are calculated into—or demanded for—the 20% capacity guarantee. This leads to two main realizations:

• Assuming that all the new functionality works as advertised (and there’s no reason to believe otherwise), then it all translates to even more efficiency (value) for users; and that EMC is priming for a bigger market battle in the months to come.
• Rather obviously—whether for new or extended EMC Unified storage users—this makes the current 20% less capacity guarantee even more alluring because it’s almost certainly going to be exceeded in practice.

The Bigger Truth

EMC’s new guarantee can be viewed on two levels. Looked at simply for what it is, it is attractive (backed as it is by EMC) and also very relevant to help address a common user need: reducing operational expenses. However, taken in consideration with the efficiency themes and functions coming in July 2010, it is hard to see anything other than the start of an all-out market battle around simplicity and economic value that will benefit users.

This new guarantee talks specifically about today’s “unified storage.” The larger EMC manifestation of unified storage is EMC’s recently announced unified management system: Unisphere. As announced, Unisphere enables both current and new EMC customers to manage block and file storage as one, and we would not be surprised to see object management following reasonably soon. Given that the new 20% guarantee appears conservative in light of the new functions arriving this summer, this all looks like the beginnings of a much broader market campaign.

Of course, EMC’s competition will not take this lying down. We can expect furious exchanges and challenges in terms of which solution is best, but extensive vendor debates and offers around improved efficiency can ultimately only be good for customers. EMC’s “20% less” move is almost certainly just the first salvo in a much larger engagement focused more than ever on customer needs. This is, of course, going to make for an extremely interesting storage world over the next year or two—guaranteed!

Online Digital Archiving, the Market Backdrop

Challenges, Realities, and why Archive?

ESG estimates that organizations will retain nearly 63,000 petabytes of unstructured data in digital archives over the next three years in order to meet litigation and regulatory mandates (e.g., Federal Rules of Civil Procedure (FRCP), SOX, HIPAA, GLBA, and PCI DSS) and to support business intelligence initiatives.[1] To put this in perspective, it was only in 2009, that Eric Schmidt, Google’s CEO, estimated the total capacity of data on the internet at 5,000 petabytes. [2]

ESG research indicates that the vast majority of corporate digital assets are stored as unstructured data. Unstructured file data—which includes office documents, digital images, audio, and video files—accounted for 74% of global digital archive capacity in 2009 and is expected to constitute the bulk of digital assets for the foreseeable future.  Growing at a compound annual rate of 79%, the worldwide capacity for file-based unstructured archive data clearly dwarfs that of database and e-mail data, as shown in Figure 1. The situation cries out for organization, management, and easy access.

It’s no surprise then that the majority of organizations recognize the need for file-based archiving solutions: 65% of organizations surveyed by ESG currently have some form of formal process and/or technology in place to manage the archiving of file-based content. An additional 29% of organizations surveyed expect to deploy file archiving solutions over the next few years.  In the same survey, improved search, performance, and security were high on customers’ archive solution wish-lists. Indeed, full content index and search is the number one “must have” feature among current and prospective customers of archiving solutions.

Nexsan’s Assureon: Secure, Online Archive

Assureon is a self managing content addressable storage (CAS) platform that extends Nexsan’s family of highly efficient storage systems. Assureon is a disk-based online archive platform designed to be a simple, plug-and-play solution that enables organizations of all sizes to protect, retain, search, and retrieve unstructured digital assets. Applications producing fixed content can point to an Assureon located on the network as a shared drive and begin writing data to the device immediately. This simple process enables IT generalists in small enterprises and mid-market companies to quickly deploy the system with confidence.

Assureon is engineered with the goal of providing a secure, online archive for high value content in a high performance platform that is power-, space-, and cost-efficient. Windows 2003 and newer servers use a lightweight client that provides direct archiving to Assureon of any desired drives or folders while any platform that can mount an NFS or CIFS share can access Assureon through an integrated NAS gateway, completely transparently.

Assureon provides these benefits in a flexible, scalable package that allows customers to deploy their own self-managing archive solution or to engage a service provider to provide secure hosted archive services, as shown in Figure 3.

Hands-on: Assureon in Action

Assureon online disk storage archive provides numerous features to guarantee that files are safe from prying eyes, have not been tampered with, and are not corrupted over time. ESG Lab recently tested Assureon’s online archive capabilities in a simulated content management environment, focusing on data privacy, integrity, and longevity.

Privacy

Assureon uses the following capabilities to provide secured, private access to data, both within organizations and between multiple organizations sharing the same infrastructure.

• Data Separation and Security: Individual users, departments, or customers can store their files in their own virtual archive. Data is logically and physically separated, ensuring privacy and security.
• Access Audit Trail: Assureon establishes an unalterable audit trail for the life of an archived file. Every time a file is accessed, a record is kept of who accessed it and when it was accessed.
• Encryption: Assureon protects information privacy and security with AES256 encryption, ensuring the highest levels of file privacy and security.

To provide scalability and data separation, Assureon can be divided into separate CAS archives, providing physical data separation inside an enterprise or between customers of a cloud computing SaaS provider.  Each CAS archive can house multiple virtual archives, each consisting of a separate database and CAS object store, as seen in Figure 4.

Assureon’s support for multiple virtual archives enables Assureon to scale, have no CAS object limits, and to take advantage of Nexsan’s AutoMAID reductions in power and cooling costs for infrequently accessed virtual archives. Each classification within a virtual archive can be further divided into subclasses, such as departments, for more granular control. All CAS archives and virtual archives and data classifications can be managed from a single pane of glass as a single system.

ESG Lab Testing

ESG Lab used a standard x86 PC running Windows Server 2008 with the Assureon client installed.  Using Windows Explorer, a folder named DATA was created and its properties were accessed by right-clicking on the folder name.

As seen in Figure 5, the Assureon client adds a tab to the properties page to access and set Assureon archive policies. Administrators and users will only see the Assureon tab if they have permission to set policies.

Two folders were created: Finance and Marketing. For both folders, ESG Lab set the organization name, retention rule, and action to be performed on all files created in or copied to the folders.  Archiving can be set to real time and/or sync mode. Real time causes all files to be processed immediately upon creation and close or modification and close, while sync processes files manually or according to a schedule. For the purposes of this test, the action was set to “Replace with shortcuts” to immediately replace all archived files with shortcuts. Users have the option to leave the original files in place while still protecting the files in the archive, or set a threshold to have files replaced with shortcuts when space is needed. The Assureon Web GUI can be used to create more advanced or detailed archive policies, including setting flexible or compliant retention, setting the number of days before an archived file is replaced with a shortcut, or the specific file name pattern to match, if only certain files in a folder need to be archived.

ESG Lab tested data privacy by logging in as two different users, one from the Finance department, and one from the Marketing department, then examining their archived folders using Windows Explorer, Assureon Explorer, and the content search feature in the Assureon GUI. Only the folders and files each user had permission to see were visible.

Integrity

Assureon ensures data integrity using multiple integrated techniques and technologies.

• Fingerprint and Integrity: When data is archived, it is assigned its own unique fingerprint that stays with the file through its lifecycle. If a single bit changes, the fingerprint will change.
• Self-Auditing and Self-Healing: Assureon continually monitors files for fingerprint discrepancies protecting against tampering, viruses, corruption, accidental or deliberate deletion ,and theft. If discrepancies are discovered, Assureon notifies administrators and automatically self-heals the file.
• Independent Date and Time Stamp: Assureon’s independent time source prevents modifications to the system’s time clock ensuring the integrity of retention periods. With Assureon, tampering with the system clock or the date stamp on the file itself is prevented.

To test data integrity, a text file was created and copied into the folder created in the previous step. The copy operation completed instantly, with all the archiving and housekeeping functions, such as calculating the digital fingerprint, completing automatically and transparently. The file was examined and verified that the text was exactly as entered.  Next, the file was intentionally “corrupted” by changing the contents of only one copy. ESG Lab examined the integrity log, shown in Figure 7. Self Auditing and Self Healing after the system completed an automatic integrity audit. The integrity log shows that the changes to the object were detected, corrected, and the corrupted copy of the file was copied to a quarantine area to preserve it for investigative purposes.

ESG lab examined the contents of each version of the file using Assureon explorer, confirming that the contents of the file had been restored to the correct state.

It’s important to note that Assureon explorer enables administrators to restore shortcuts, or full files, to a production server by right-clicking on the file and selecting the appropriate action.

Longevity

Longevity is the ability to guarantee that all versions of a file will remain immutable and uncorrupted over time. Assureon uses multiple techniques to provide file longevity.

• File Availability Audit: Assureon serializes each file and continually checks to ensure all files are present in each store. If a file is missing, Assureon notifies administrators and automatically returns it to its original state.
• Automated Retention and Deletion: To satisfy regulatory compliance or corporate governance, organizations need to guarantee the retention and deletion of files. With automated integrity management and file immutability technology, Assureon protects against accidental or unauthorized file deletion and ensures files can be retained for compliant or flexible time periods as well as securely deleted when their retention period expires.
• Legal Holds: Legal holds can be placed on files to protect against file deletion when retention periods have expired. This ensures availability of files in cases of litigation or potential litigation.

ESG Lab examined multiple aspects of file longevity.  The retention rules page is shown in Figure 9. Assureon has the ability to create separate sets of retention rules for every organization created in the system. Each organization can create multiple rules within each organization to provide appropriate retention for any data type.

A retention rule was created with a retention period of seven years and data encryption enabled. Next, the audit logs were examined and it was confirmed that Assureon was running both file availability and integrity audits daily, restoring corrupted and lost files as necessary.

The Bigger Truth

ESG research has found that unstructured file data accounted for 74% of global digital archive capacity in 2009 and is expanding at a compound annual growth rate of 79%. The worldwide capacity for file-based unstructured data dwarfs that of database and e-mail data combined—and growth is accelerating. Organizations must deal with all this growth while meeting compliance and governance standards and controlling the cost of search and discovery. In this scenario especially, additional complexity is not desirable.

Nexsan’s Assureon is an excellent fit for businesses of any size that require intelligent archiving with CAS. The low cost of entry makes this solution available to a whole new set of IT departments that until now simply couldn’t afford to acquire or manage such technology. Companies can implement robust data security and protection with limited IT resources, and still utilize enhanced search capabilities to quickly produce pertinent data for internal investigation and litigation support without having to incur additional expenses from service providers culling through vast archives of data located on different media types.

Assureon delivers more business value by minimizing the power, space, and cost for given capacity and performance workloads. In addition to its powerful archiving tools, Assureon allows users to move away from a traditional tiered storage implementation and to gain the added IT and business benefits of Nexsan’s highly efficient storage.

ESG Lab was quite impressed with Assureon’s transparent and simple integration into a standard server environment, integrating archive policies, classification, search, reporting, and migration into one intuitive package.  ESG Lab testing revealed that Assureon provides excellent data privacy, integrity, and longevity in a package that is as easy for small businesses to deploy and manage as it is for large enterprises or service providers. The platform scales performance and capacity independently, leveraging Nexsan’s family of highly efficient RAID enclosures and industry standard servers.  It is clear that Nexsan is continuing to execute on its vision to provide users with a highly integrated storage infrastructure.

The company does face a couple of notable challenges. First, the CAS market has never exploded in quite the way that some imagined it would and second, the CAS market that does exist also has a very dominant player (EMC). However, like so much in life, the flip side to challenge is opportunity: EMC and others have shown that a decent market does exist and it’s all open to Assureon if its channel can find a way to the table; Assureon brings a level of simplicity and functionality that could well both expand the realistic market potential and certainly give prospective users more than a moment’s pause before leaping with the default solution. If Nexsan were a brand new player, its task might be very difficult indeed—however, it is a proven, relatively small, but high quality—provider with a base of “raving-fan” users. Nexsan already has several large medical OEMs supplying Assureon to medical and PACs archive end users.

ESG Lab found the system to be extremely easy to manage, providing policy based archiving while preserving privacy, guaranteeing data integrity, and protecting data longevity from a single, simple interface.  In ESG’s opinion, Assureon will enable organizations to reduce their costs and simplify their infrastructures while providing enterprise class archiving services at the speed of online disk.

[2] Chang, Fay Dean, Jeffrey Ghemawat, Sanjay Hsieh, Wilson C. Wallach, Deborah A. Burroughs, Mike Tushar, Chandra Fikes, Andrew and Gruber Robert E. “Bigtable: A Distributed Storage System for Structured Data”, 2006, Retrieved from http://labs.google.com/papers/bigtable.html on 10/7/2008

Why Thin Provisioning is Important

The Big Picture

The recent market contractions and contortions have pushed the economic pressure for IT performance to new extremes; to re-coin an old phrase, we in IT have never seen so much pressure to do so much with so little. This partly explains the rapid adoption of virtualization to help allow more to be done with less.  Even so, in storage, it is common to find a vast amount of capacity effectively underutilized and beneath the management radar. It is a best practice for organizations to optimize their existing storage infrastructure before procuring new storage—this is where “thin provisioning” comes in. Thin provisioning is an elegant solution that increases storage allocation efficiency and thereby more effectively leverages a shared pool of storage capacity. In doing so, there is further opportunity to do more with less and to answer the call of economic necessity from business leadership.

From Fat Storage to Thin Provisioning

In recent years, studies have shown that as much as 70% of physical storage capacity is often wasted by what can only be described as mismanagement. This happens partly as a result of disconnected storage management tools and processes; the problem is sometimes aggravated by a lack of compatibility and interoperability between storage management solutions. From a management efficiency point of view, this can create chaos, leaving allocation and utilization efficiency measurements low. Rather than fixing the larger issue of the storage management solution jumble by using a product (or products) that is/are well integrated with internal IT processes for managing capacity, administrators often buy cavernous amounts of new capacity to meet the never-ending demands for storage capacity. Thin provisioning is invariably a great efficiency enhancer, but used alone or out of management context, it can only help, but not solve, the broader need for improved storage efficiency. IT management is complex and so new technologies must be easy to operate and integrate, just as much as they are financially attractive.

Thin provisioning is comprised of two major capabilities. It aggregates multiple volumes, which can span arrays, into what appears to be a single reserve or free pool of capacity. This allows for a singular view of a defined storage pool spanning multiple arrays. Secondly, thin provisioning uses a technical approach known as “allocate-on-write” to only allocate capacity if blocks are actually written.  If, for example, a volume is created for 500 GB, but the application only writes and uses 100 GB, then the true physical storage allocated is just the 100 GB.  The remaining 400 GB are free to be shared and used among other operating systems tasks, applications, or users. Aside from the efficient “just-in-time” usage of actual capacity, storage administrators can over-allocate “just-in-case” capacity as a buffer for applications to have room to grow into. Each application, user, or system administrator can have (at least conceptually) as much storage allocated as is desired—the storage manager needs to monitor only the real physical consumption of storage across the entire system.

The “Allocated, but Unused” Problem

Put directly, the objective of thin provisioning is to reduce the total amount of storage necessary and it achieves this by allocating and using shared storage capacity on demand.  In the scope of mismanaged capacity, a standard provisioning scenario typically has a very significant percentage of the total available capacity (often 25% or even more) that is allocated but unused or allocated but orphaned—either way, it is stranded.  This can happen from a request for storage capacity from a user or application owner who never actually creates a file system or stores a single piece of data. The challenge here is that traditional storage reclamation methods are invariably cumbersome, expensive, and sub-optimal. Multiple teams—such as the application, database administration, system administration, and storage administration teams—have to cooperate to perform a series of time-intensive tasks, such as shrinking the applications, file systems and host volumes; reducing the relevant LUN space; and finally returning the freed space to the shared pool. In a nutshell, the reclamation process is highly disruptive to applications, requires cross-team communication, and often ends up being sub-optimal even if it is completed successfully (because of the greater cost and complexity of this reclamation method, it often simply does not get done).

Rather than solving the tough problem of identifying and reclaiming space that has been allocated  but is unused or stranded, administrators will often continue to over-purchase (or over-provision) storage capacity as a way of masking the limitations of existing storage management tools or processes, and believe that they will generally not get caught running out of capacity. Indeed, some storage administrators will even refer to the allocated, but unused, space as their “emergency free pool.”  If they run out of capacity, they’ll claim that they can simply “find” this space and return it to the free pool of storage.  All of this is a very ad-hoc and unstructured methodology that can eventually lead to application issues from unexpectedly running out of storage capacity and crashing or costly emergency storage hardware purchases. Once again, this is not a blame-game—the technology to do things better did not always exist. Today it does, but it would be naive to assume that simply purchasing new hardware will solve all problems.

Not all Thin Provisioning is Created Equal

Symantec’s Veritas Storage Foundation: a Broad View

Certainly, even a marginal use of thin provisioning is invariably a good thing, but leveraging the technology can be done even more efficiently when the thin provisioning enabled hardware is integrated with the host —specifically, the file system and volume manager.  Symantec’s Veritas Storage Foundation file system can gaze across multiple arrays, regardless of brand, and offers unique capabilities when it comes to leveraging thin provisioning. The primary benefit of these capabilities is that an organization can address the two major elements involved in the successful long term adoption of thin provisioning: as with any form of diet (this being a capacity diet!) organizations first need to get thin and then to stay thin. And they need to do both of these in a manner that is non-disruptive and seamless to the business.

Getting Thin. The first challenge for IT organizations is how to migrate to a thin environment while eliminating waste. Unfortunately, most online storage migration tools perform block-level copy, which means that every block on the destination is identical to the source. The reason for this limitation is that most migration tools are unable to differentiate between a used and an unused block of data. As a result, these tools treat all blocks equally and copy all blocks during a migration. This means that even in the process of getting into a thin environment, there can be a lot of waste generated in the new environment.

Veritas Storage Foundation solves this problem by uniquely identifying allocated, but unused, blocks and copies only used data blocks to the target storage device.  A few other unique capabilities of Veritas Storage Foundation are that in addition to migrating only the blocks that matter, the migration is hardware independent (that is, from any vendor to any vendor) and is performed via a simple web-based interface while applications stay online. This facilitates an easy, hardware-independent, non-disruptive transition to a thin environment combined with ongoing bandwidth and storage savings at any other time that a migration is performed.

Staying Thin. The second challenge IT organizations face is how to keep an environment thin over time. While this may seem like a rational expectation, hardware vendors often don’t make it easy to stay thin. Specifically, blocks of data which are erased by an application or user are typically non-reclaimable even by the most advanced thin provisioning enabled array. The reason for this is simple: when applications or users delete data, the respective blocks of data are not “zeroed” out. This means that the array is not informed that those blocks are available for reuse.

Veritas Storage Foundation provides the host side intelligence to notify the array and actively trigger reclamation of previously used but now available data blocks, returning them to the free pool. In this way, the reserve free pool can remain as capacious as possible, providing optimum value to the user by elongating the amount of time the available (real) storage capacity can stretch its effective use before new storage capacity must be purchased.  This capability is important.  Other file systems and volume managers use a “write-once, allocate-forever” approach that precludes the ability to repurpose unused blocks.

The Broader View. When thin provisioning is implemented along with a powerful suite of storage management tools (such as Symantec’s Veritas Storage Foundation), not only can 40-50%—or sometimes even more—efficiency in usable storage capacity be gained, there are also other immediate and measurable improvements:

• Centralized management of data centers can result in reduced management complexity.
• Higher availability and performance of applications, databases, and data center assets.
• Significant reduction in the capital and operating expenses for storage infrastructure.

The point of the link between storage management and overall storage costs is an important one—simply stated, inefficient storage management is a platform from which storage vendors can oversell capacity. Certainly, if storage were free of cost, any issues with the growing complexity of managing resources could be hidden behind vast amounts of capacity.  However, not only is this expensive from a CAPEX point of view, it is disastrous from an OPEX point of view; neither of which is consistent with the issues of tight[ened] budgets and management objectives for bottom line efficiency in IT.

Optimization Leads to Value

There are some interesting lessons here.  The first is that the combination of orphaned, unused, and inappropriately used space can easily represent some 30% of total storage capacity; in other words, it is storage capacity—money—that is wasted.  Perhaps another 40% of the capacity is previously used data blocks and thus ripe for migration. The recovery of all this “wasted” storage could provide a year of data growth—free!  There is value to be had, but how should users get at and release it? The issue is often not addressed at all and yet these are IT manifestations of low-hanging fruit.  The best way to really identify the opportunities is to avoid the use of non-integrated storage management solutions that transfer the burden of storage usage accounting to the storage array.

Instead, best practice is to use an integrated storage management platform, such as Veritas Storage Foundation, which offers simplified management, higher performance and higher availability, and works in a heterogeneous fashion with existing and new storage assets. When combined with thin provisioning enabled hardware, storage administrators can effectively provision and manage their physical storage use. Using Veritas Storage Foundation, storage administrators can painlessly move to a thin environment and, moreover, stay thin over time. This combination of effortless migration and automated identification and reclamation of wasted storage is what sets Storage Foundation apart in effectively addressing what can otherwise—and all too commonly—represent around 60% or 70% of wasted and/or wrongly used storage capacity.

The Bigger Truth

The success of any business and its IT department—in good times or bad—is dependent on a nimble and efficient infrastructure.  An efficient infrastructure is, in turn, dependent on IT professionals getting to the root of waste and minimizing or eliminating it. Tools that can identify and automatically reclaim waste are therefore critical. Veritas Storage Foundation offers an integrated solution that facilitates efficient storage management. Its storage virtualization and management platform enables storage administrators to truly leverage the power of thin provisioning by allowing them to effortlessly migrate to a thin storage infrastructure. Storage Foundation can then continually optimize the environment by reclaiming wasted storage in an automated fashion. This can actually be a majority of the installed storage capacity in many sites! It is an uncomfortable truth, but also an opportunity for IT managers to be heroes by returning bottom line value to their organizations.

Smaller amounts of well managed capacity can deliver the same value —and at a significantly lower cost—as larger amounts of poorly managed storage. Putting all of an organization’s wasted capacity back to work has frequently proven to provide millions of dollars in net present value. Sure, there is an upfront cost to buying any software, along with the effort to have IT administrators trained. But, the amount saved by efficient storage management is almost always much greater than the cost. Veritas Storage Foundation has proven for over a decade that it can be the cornerstone of a sound storage management practice. Although optimizing thinly provisioned storage is just one part—one might say a thin slice—of the much larger value that Veritas Storage Foundation offers, it is a pragmatic solution that any business-oriented IT manager should consider implementing.

Market Overview

The demands on and for storage are increasing and, commensurately, the challenge to effectively control the way vast amounts of data are created, stored, and accessed is intensifying. To address data growth without interrupting business operations, rapid deployment of storage and IT resources to meet increasing demand becomes a function of scalability. The primary objective of any type of scaling, therefore, is to create a dynamic, non-disruptive environment that supports data growth in such a way that system capabilities remain as balanced and productive as possible.

Scale-out Introduced

As computer and storage prices drop and performance and capacities continue to increase, low cost “commodity” systems are being used for high performance and high capacity workloads that previously could only be handled by scaling up to much faster storage devices or even to high-performance computers. Scale-out storage allows many small, low cost computers and commodity storage components to be combined and configured to either create an aggregate storage pool or to increase computing power and exceed that of a single traditional storage array or high-performance computer. This commodity-based scale-out model is offered by most vendors and is becoming popular, increasing demand for storage virtualization, shared data storage, and improved data protection services. Clusters and grids are examples of scale-out systems, but “scale-out” can apply just as well to SANs as it does to the NAS environments with which it is more commonly associated.

Scale-out Analysis

Scaling-out has the potential to change the face of data center operations by displacing large, costly enterprise storage equipment with a pool of less expensive devices. It differs from traditional architectures in that it adds new computing, networking, or storage resources to an environment; this is in contrast to scaling up, which can either consolidate several smaller devices into one larger device or simply expand that one device.

Scaling out is accomplished by adding additional nodes to a system—such as a server, a network switch, or a storage device. When you upgrade your PC to a faster model with higher capacity disk drives, you are scaling-up, not out. Scaling out would be expanding a two server system to four servers in order to increase overall compute power. Adding more storage devices to create a larger storage pool is therefore an example of scale-out storage just as creating more network ports by adding more switches is an example scale-out networking.

Essentially, scale-out solutions let users create larger shared pools of compute power and storage capacity to grow well beyond the limits of an individual node. However, they also have the potential to simultaneously create a more complex management environment simply as a result of all the interconnected pieces; generally, therefore, scale-out storage is usually packaged with a corresponding software management and virtualization/abstraction layer that can make hundreds or thousands of nodes behave like a single system (using a global namespace) and that allows the storage resources to be deployed as needed.

Scaling Storage Topologies

Although scale-out is often assumed to be synonymous with NAS and scale-up is, in turn, often assumed to be synonymous with SANs, the truth is that scalability is not a direct function of storage type; scale-out can therefore apply to all three major storage topologies:

DAS: As the first widely popular storage model, DAS (direct attached storage) implementations still comprise a large proportion of the installed storage capacity on all major operating systems. DAS is simpler to manage and can scale out—at least to a limited degree—for added performance and capacity by adding more, larger, or faster drives or by cascading additional control units; however, DAS doesn’t provide many of the efficiencies and advanced functionalities that simplify management efforts and add operational advantages as the storage pool grows.

NAS: NAS (network attached storage) is a special purpose architectural approach comprised of servers, disks, and management software dedicated to serving files over a network. This is why scale-out storage is frequently associated with NAS systems that scale to become a cluster of NAS nodes: these nodes allow for capacity and performance to scale either in combination or independently as needed and yet—this is a crucial point—still maintain a single system image. Unstructured data, normally in file format, is already a huge proportion—probably the majority—of all digital storage and is expected to significantly outpace the growth of structured data and hence increase demand for low-cost scale-out storage. Much of this data resides on NAS.

SAN: Scale-out storage is also an important aspect of SANs (storage area networks) and is usually achieved by adding more SAN switches and creating a scale-out SAN fabric. Users can still achieve linear scale by increasing performance “horsepower” and capacity in step. SANs provide excellent scale-out capability for large enterprises that anticipate significant growth in information storage requirements and, unlike direct attached storage, any excess capacity in SANs can be pooled to provide higher utilization of resources as well as advanced functionalities. SAN topologies can be implemented to meet different needs for SAN scaling: they can scale-out to connect SAN islands built around individual switches into larger single fabrics or can use routers to physically connect the switches while logically isolating the fabrics. Virtualization is also used heavily with SANs.

Making a Choice

The key for any organization searching for an optimum solution is to understand which of its goals are most important. As with so much in IT, the answer is “it depends.” Table 1 is a quick summary of the factors a user might consider in terms of the available topologies and their associations with the different scaling approaches. The eventual decision will almost certainly take other factors into account; these might include functionality, price, TCO, and skill/comfort levels as well as predicted changes and growth across the organization. Scale-out and -up vendors will battle over their relative values in terms of overall economic efficiency and “green-ness.”

What’s best will ultimately depend on the nature of the application: both scaling approaches are valuable tools for IT in general and storage in particular, and many organizations will continue to employ both (separately or in combination) for the foreseeable future as they offer useful, though varying, attributes. For instance, scaling up an existing system often results in simpler storage management than with a scale-out approach as the complexity of the underlying environment is reduced or at least known. That said, scale-up storage systems can only scale as far as the performance and capacity limits of individual storage resources permit.

The Bigger Truth

Putting things simply, to support growth, a user can either add infrastructure capabilities to the current devices (scale-up) or add more devices to the environment (scale-out). Scale-out storage can improve IT management’s ability to provide timely provisioning, greater resource utilization, required performance, and higher levels of data and system availability non-disruptively. The price differential between the two models is currently favoring scale-out computing for those applications that can embrace it, thanks largely to its use of commodity components.  In many environments, it is worth considering the two approaches in combination; what is certain is that scaling up and out will continue to play ever larger roles in IT as the demands to service business growth in a flexible manner climb ever higher.

Market Overview

Business demand on storage is increasing and the challenge to effectively control the way that vast amounts of data are created, stored, and accessed is intensifying as a result. To address data growth without interrupting business operations, the ability to rapidly deploy storage and IT resources to meet that increasing demand becomes a function of scalability. The primary objective of any type of scaling is therefore to create a dynamic, non-disruptive environment that supports data growth in such a way that system capabilities remain as balanced as possible.

Scale-up Introduced

To support growth, putting it very simply, infrastructure capabilities can be added to the current devices (scale-up) or more devices can be added into the environment (scale-out). For many IT users, scaling up their existing storage infrastructure to much faster storage devices, or even higher-performance computing platforms, can meet their needs perfectly adequately when compared to expanding infrastructure by adding more nodes or moving to more complex cluster or grid topologies. Storage resources continue to get faster, become cheaper, and offer larger capacities, which increases the appeal for both scale-up and scale-out architectures. Scale-up solutions let users increase compute power, add storage (performance or capacity), or create more network ports by implementing bigger and faster (often proprietary) resources rather than adding more (often commodity) resources.

Scale-up Analysis

Scaling up (vertical scaling) is accomplished by replacing or upgrading existing components with more powerful resources; as such, it can be applied to all major elements of computing infrastructure—leading to, for example, faster servers, higher capacity storage devices, or switches with higher port counts. Scaling up has historically been most tightly associated with regard to servers, though any IT resource can be scaled up or upgraded to a newer faster model. A server can be scaled up by upgrading from several small servers to a larger, high performance or multi-core server to increase overall compute power and reduce the number of management touch points.

Scaling storage up can be done by making existing storage nodes in a system larger and/or faster and/or more powerful. A scale-up storage example could be adding higher capacity storage devices to replace older, lower capacity devices in order to increase the overall capacity of the storage pool. Replacing a HDD (hard disk drive) with a faster flash-based drive or a solid state disk is another example of scaling storage up—in this case, to improve IO performance.

In the simplest terms, scale-up storage usually means either adding more to the existing storage and/or consolidating several smaller devices into a larger device, thus reducing touch points and simplifying management efforts. Fewer devices usually also mean less floor space and less energy consumed. A consideration for scale-up architectures is that they can’t span multiple locations or easily scale beyond the maximum capacity of any single storage node. Another consideration with scaling up is the potential lack of overall resiliency (without specialized constructs to add this) compared to the added redundancy that scale-out architectures more typically and easily provide.

Scaling Storage Topologies

Although scale-up is often assumed to be synonymous with SANs and scale-out, in turn, is assumed to be synonymous with NAS, the truth is that scaling is not a direct function of the storage type; scale-up can therefore apply to all three major storage topologies:

DAS: As the first widely popular storage model, DAS (direct attached storage) implementations still comprise a large proportion of the storage capacity on all major operating systems. It is simpler to manage and can scale up for added performance and capacity by adding larger control units and faster drives or cascading additional control units; however, DAS doesn’t provide many of the efficiencies and advanced functionalities that simplify management efforts and add operational advantages as the storage pool increases.

NAS: Network attached storage (NAS) is a special purpose device comprised of servers, disks, and management software dedicated to serving files over a network. Scale-up NAS can be associated with adding larger and faster—as opposed to more—NAS nodes that allow capacity and performance to scale as needed.

SAN: Scaling up is also an important aspect of SANs (storage area networks) and is often used to consolidate smaller SAN “islands” with individual switches into larger, single fabrics or it uses routers to physically connect the switches while logically isolating the fabrics. By offloading many traditional storage management functions from the host, SANs provide an effective scale-up capability for large enterprises that anticipate significant storage growth. Unlike DAS, excess capacity in SANs can be pooled and more easily shared, resulting in higher utilization of resources.

Making a Choice

The key for any organization selecting an optimum solution is to understand which of its goals are most important. Table 1 is a quick summary guide of the factors to consider in terms of topologies and scaling, although the eventual decision will also need to take other factors (such as functionality, price, TCO, and skill/comfort levels) into account.

The Bigger Truth

Scale-up and scale-out are both valuable storage tools and many organizations will continue to employ both for the foreseeable future as both have useful, though varying, attributes. For example, scaling up an existing system can often keep storage management simpler as compared to scale-out storage as the complexity of the underlying environment is reduced. That said, scale-up storage systems can only scale as far the performance and capacity limits of the individual storage resources permit. Conversely, a well implemented scale-up storage model (whether SAN, NAS, or DAS) can provide very timely provisioning, often with greater resource utilization, and deliver known levels of data and system availability. Perhaps it is best to consider the two approaches in combination; what is certain is that scaling up and out will continue to play ever larger roles in IT as demands to service business growth in a flexible manner climb ever higher.