We’re still in the early phases of deduplication adoption, with less than 40% of respondents to ESG data protection research having deployed the technology in backup environments.  Of these, ESG’s research found that adopters of deduplication selected a mix of hardware and software deployment approaches: 40% implement it in backup software, 26% in backup hardware (appliance or disk storage system), and 34% in a combination of hardware and software.

Deduplication as a feature of software solutions may be less expensive, but there could be more time and technical acumen required for installation, configuration, and performance tuning, and any hardware necessary to create a whole solution will need to be acquired. Appliance-based solutions are pre-assembled with all of the required components, offering end-users a more plug-and-play installation and configuration experience, and since they are purpose-built they may deliver better performance. One drawback to hardware appliances is that they may underutilize system resources since purchases are typically made to accommodate future growth.

Organizations with fewer production servers and less capacity tend to favor software-only approaches while those with more production servers and capacity take a hybrid approach, applying hardware to some workloads and software to others.

Both adopter and planned adopter respondents ranked cost, ease of installation/use, and impact on backup/recovery performance as the top three purchase criteria for deduplication.  Interestingly, dedupe buyers using hardware-only approaches were influenced most in their decision by a relationship with an existing vendor.  I wonder if that will be the case with Symantec’s appliance approach.

What’s your preference and why?

Read more of Lauren’s blog posts at Data Protection Perspectives.

ESG has been saying that primary data reduction would come to a storage environment near you.  We even pointed at NetApp as proof.  The response was “one vendor does not make a trend.”  EMC talked about it quietly, but no one gave them any credit.  So, let’s say it again.  “Data reduction is coming to PRIMARY storage ; Data reduction is coming to PRIMARY storage.”  Dell now owns Ocarina and IBM bought up compression-technology leader StoreWize.   Someone will be smart enough to own / partner with Permabit soon.   We have three concrete examples and more likely on the way.  Convinced?  One big bullet from a musket in honor of Mr. Revere:

• Storage solution vendors are going to try and help you save less data.   Please listen to us this time and try it.   It will save you money.  If not, ESG may have to stop blogging and tweeting and buy a horse.

Read Brian’s other blog entries at IT BULLETins.

via eChannelLine USA – IBM buys data-compression vendor Storwize.

So how does the acquisition benefit Dell? Dell sells storage systems for primary and backup data, and wants to provide features that lower data management costs for end-users. But where does the Ocarina technology fit considering Dell’s existing partnerships with CommVault (Simpana on PowerVault DL systems), EMC (resell Data Domain) and Symantec (Backup Exec on PowerVault DL systems)? What’s the relationship between data deduplicated on primary storage and data deduplicated in backup processes?

In an earlier post, I discussed the benefits of deduplicating “upstream”—as close to the source of data as possible with solutions from the likes of Exar, GreenBytes, NetApp, Permabit, and StorWize. One of the theories is that data can be backed up in its optimized state—that is, without having to be “rehydrated” or “reinflated” to its non-optimized format for backup processes. This would enable an end-to-end deduplication strategy for data. Not quite sure how this plays out with some of the primary deduplication solutions, including Ocarina.

With this gotcha in mind, it would, therefore, make sense that Dell would need deduplication for primary data and other solutions (CommVault, EMC and Symantec) for backup data. The technologies from these different vendors could be complementary. I don’t know how efficient a story that is for Dell customers … it’s definitely a wrinkle Dell will have to iron out as it brings Ocarina-optimized storage to market and positions it versus other capacity optimization technologies in its portfolio.

Read more of Lauren’s blog entries at Data Protection Perspectives.

What ESG’s research reveals is a mixed outlook for VTLs: there is widespread use of and ongoing interest in VTL technology; however, the technology may simply fill niche requirements for select organizations. Results of surveying over 500 IT professionals revealed VTL deployments are a function of data volume and organizations’ reliance on physical tape. Companies with multi-terabytes of data to back up daily require high-performance solutions—such as VTL—in order to meet backup windows, and VTL has higher appeal to those organizations leveraging both disk and tape.

Most IT organizations (80%) are using disk as the initial on-site backup target. Retention time on disk in D2D2T strategies has been extended from one week (66%) in 2008 to one month or more (68%) in 2010. The latter number jumps to 80% for VTL users in 2010. This is likely fueled by the availability of deduplication: more VTL users employ deduplication than those organizations that back up to other forms of disk targets.

VTLs will likely face mounting pressure from competing disk backup targets that are perceived as more cost-effective. However, organizations with retention mandates requiring physical tape, and data volume that prevents IT from meeting backup windows are likely to find relief in VTL technology.

Read more of Lauren’s blog entries at Data Protection Perspectives.

Background

While deduplication can reduce the cost of the raw storage required to store and replicate backup data on disk, integration with the existing ecosystem is crucial. As shown in Figure 1, recently completed ESG research indicates that ease of implementation, performance impacts, and integration with existing backup processes are key concerns.[1] Robust management, edge to core replication, tape integration, and deduplication options are important considerations as well, especially within large enterprise-class organizations. The diverse family of Hewlett Packard data protection solutions is ideally suited to address these, and other, concerns.

Deduplication

Choosing a deduplication strategy should include a discussion on how and where deduplication should occur. There are two basic locations deduplication can occur: at the source, typically accomplished via a software agent runing on the client machine, or at the target, which involves either writing directly to the device  or running a software agent on the media server to perform deduplication.  All deduplication includes some level of overhead. If it occurs at the source, that overhead occurs on the client machine or media server and may have an impact on backup performance due to the software required on client systems, which can consume processing, storage, and/or network resources to deduplicate data.  When deduplication is perfomed at the target, the overhead is incurred in the device where data is being written.

In addition, it is important to differentiate between a pure software approach — in which the software  runs on an industry standard platform, — and an appliance-based (or storage hardware-based) solution. With a  software solution users have flexibility in their choice of physical storage, but must ensure that the system has enough I/O bandwidth to support their performance needs, sufficient system resources to support desired de-duplication rates, and the right storage security to prevent potential data loss. An appliance-based  solution has the ability to address many of these concerns, but locks users into a particular hardware platform.

HP’s D2D and VLS are target-side deduplication appliances which are designed to provide cost-effective, easy-to-deploy deduplication and are tuned for optimal performance on each hardware platform.

HP Data Protection Solutions

Hewlett Packard offers diverse data protection hardware and software to address the concerns of enterprises from the core data center to the smallest remote office. Figure 2 shows the HP family of data protection solutions as they might be deployed in a typical distributed enterprise.

HP StorageWorks VLS virtual tape libraries provide a high performance primary backup target with optional deduplication in the FC SAN enterprise data center while HP StorageWorks D2D serves as an easy to manage data protection appliance in mid market data centers and remote offices with deduplication and WAN-efficient replication across sites. HP Data Protector software provides a solution to manage the D2D and the VLS not only in sngle-site backup environments but also in replicated solutions. While this report focuses on the benefits of a total HP solution, it’s important to note that HP products integrate well with third party products. Data Protector works with any inline de-duplication solution and the D2D and VLS can be used with third party backup software. However the end to end, single vendor solution including replication enablement and integration is a unique HP offering.

Benefits of HP StorageWorks Data Protection Solutions

VLS:

• Highly scalable capacity and performance: Up to 4800 MB/sec of throughput and 1280 TB of usable storage.
• The entire capacity of the system can be presented as a single virtual library target: The system can scale without time consuming reconfiguration and rebalancing of backup software and backup jobs.
• Accelerated deduplication: Fast hardware compression, in combination with post process deduplication, provides capacity efficiency without impact to backup windows.
• WAN efficient replication: Fast, cost-effective disaster recovery capability between data centers and remote sites.

D2D:

• Distributed capacity and performance: The D2D Backup Systems offer high performance multi-streaming backup speeds of up to 720 GB/hour.
• HP Dynamic deduplication: Enables longer term data retention on disk and WAN efficient replication.
• WAN efficient replication: Fast, cost-effective disaster recovery capability between multiple remote sites.
• Ease of use and deployment: HP’s D2D Backup Systems are designed for easy installation and deployment for mid-sized business environments—it is ready to deploy right out of the box.

As of the publication of this report, HP has refreshed the entire D2D product family and introduced a new high-end model, the D2D 4312. The D2D 4312 has more processing power and offers higher capacity than previous generation D2D models.  The new D2D product family runs a new 64bit data deduplication technology called HP StoreOnce.   HP’s vision is to port StoreOnce deduplication technology to several HP platforms – including HP Data Protector.   While ESG tested the previous generation D2D appliances, the scenarios depicted and conclusions drawn in this report still apply.

Data Protector:

• Advanced Backup to Disk – 24/7 information access and quick disaster recovery.
• Multiple Recovery Point/Recovery Time Objectives – Achieve business-driven recovery objectives.
• Manage data effectively – within existing budgets and infrastructure, even as the quantity of data grows.
• Centralized Data Protection – Protect data on distributed physical and virtual infrastructures.
• Broad Interoperability – Integrates with partner solutions including NetApp, Data Domain, IBM ProtecTIER, and supports any third party inline deduplication target appliance.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of HP’s data protection solutions at an HP facility in Fort Collins, CO. Testing was designed to demonstrate the scalability, performance, and ease of management  of HP’s solutions from the point of view of a typical enterprise systems administrator integrating HP’s disk-based solutions into an existing tape environment.

Ease of Deployment and Integration

The test environment, shown in Figure 3, was used throughout testing. Testing began with a one-node HP StorageWorks VLS9000 array with deduplication, physically installed and powered up as it would be by HP professional services for an enterprise customer, in a data center environment with HP Data Protector software installed. Other elements typical of an enterprise environment, such as physical tape libraries and  HP StorageWorks D2D appliances, were also present in the test bed.[2]

ESG Lab Testing

ESG Lab initially logged in to the Command View VLS console by pointing Internet Explorer at the administrative IP address on the VLS9000 library and entering the administrator username and password. Next, the Create Virtual Library Wizard was launched. The Create Virtual Library Wizard walks the administrator through the steps to create a virtual tape library, asking for such details as library type,  and is shown in Figure 4.

ESG Lab selected an HP ESL E-Series to match the physical library type already installed in the test environment. Next, and more important, the virtual tape drive type and quantity, as well as the capacity and quantity of virtual tape cartridges, were set, as seen in Figure 5.

In less than two minutes, the Create Virtual Library Wizard was completed and the virtual tape library was configured and presented on the SAN.  Next, ESG Lab started the HP Data Protector Autoconfigure Wizard. The HP Data Protector Autoconfigure Wizard, as seen in Figure 6, discovers new backup target devices and prepares Data Protector to use them.

The Data Protector Autoconfigure Wizard took about two minutes to discover and add the Virtual Tape Library and its virtual tape drives to Data Protector. As Figure 7 shows, within five minutes of sitting down at the keyboard, ESG Lab was running a full backup of the first server to the HP VLS9000.

Scalability and Performance

One of the fundamental advantages of VTL backup is the ability to run many backup streams concurrently using multiple virtual tape drives. A single tape drive can only perform one backup at a time.  To get more than one backup job running at the same time, more tape drives must be added and run in parallel. A disk-based backup and recovery solution with many random access disk drives emulating many virtual tape drives can run many backup jobs simultaneously.  The random access nature of disk also provides improved performance when locating individual files to be restored.

ESG Lab Testing

ESG Lab performed backups using a single node VLS9000 as a target first and then repeated the test after upgrading the VLS to two nodes in order to examine its relative performance as storage capacity is scaled.  A full backup of mulitple servers was simulated using the HP tapeperf utility set to generate data with 2:1 compressibility.[4] The first iteration ran with two servers running 10 backup streams; the second iteration was performed with three servers running sixteen streams. Performance scaled linearly when the second node was added. The screen capture in Figure 8 shows the VLS Command View Console during the two node test, running at 1,164 MB/sec.

Figure 9 shows actual performance results obtained in ESG Lab testing projected out to a fully populated, eight-node system. Detailed results for each test run are shown in Table 1.

What the Numbers Mean

• When a node was added to the VLS, performance scaled nearly linearly with no degradation.
• An eight-node system should be able to acheive 4,656 MB/sec of sustained backup throughput, representing the ability to protect nearly 128 TB of data in an eight-hour backup window.

Remote Office Protection

The family of HP disk and tape data protection solutions, combined with HP Data Protector software, can be configured to create an automated, edge-to-core data protection topology (see Figure 10) that spans multiple sites and provides disk-to-disk-to-tape (D2D2T) functionality. HP StorageWorks D2D appliances provide WAN-efficient movement of data between sites and storage tiers while Data Protector provides the single point of management and catalog for backup data—regardless of where it resides (remote office or corporate data center), what type of media it is stored on (disk or tape), or its age (recent backup or long term archive). D2D disk-based backup and replication appliances support data deduplication to reduce the resources required to store backup images on disk and replicate backup images over a WAN.

ESG Lab Testing

ESG Lab used HP Data Protector software to configure, automate, and track the migration of backup data residing on HP D2D data protection appliances, as shown previously in Figure 3. An edge-to-core D2D2T data protection strategy was implemented using an HP D2D appliance located in a simulated remote office. Remote office backup data was replicated over a simulated WAN to an HP D2D in a corporate data center with data movement carried out by the D2D systems. The object copy capabilities of the Data Protector software were used to write a copy of the data to removable media in a Fibre Channel SAN-attached HP StorageWorks MSL tape library. First, ESG Lab logged in to the D2D web management console, shown in Figure 11.

The first backup was ‘seeded’ or replicated locally over a gigabit ethernet LAN connection. Seeding is often employed by users who have large data sets at remote offices as it allows the first bulk transfer of data to complete very quickly. The D2D device is then shipped to the central data center and from then on, updates require much less bandwidth thanks to deduplication.  The first bulk replication is illustrated in Figure 12.

Replication of the first backup transferred 53 GB of data in 10 minutes and 40 seconds over an unrestricted Gigabit Ethernet connection. Once the first full backup was completely replicated to the target D2D appliance, ESG Lab inserted the ‘Network Nightmare’ WAN simulator and restricted throughput to 2 megabits per second to simulate a nearby WAN connection between the remote office and data center.

The first incremental backup was performed and automatically replicated over the 2 Mbit/sec simulated WAN connection. Replication of the incremental backup transferred 5 GB of deduplicated data in 1 hour, 17 minutes and 49 seconds.  This means the D2D transferred at the equivalent of 11 Mbit/sec over the 2 Mbit/sec WAN connection. By comparison, the second full backup resulted in the equivalent of 22.9 Mbit/sec over the same link. The higher virtual throughput is due to the greater volume of duplicate data in a full backup.

Figure 14 shows actual and projected deduplication capacity savings over 30 days of backups on a weekly full, daily incremental backup schedule. The capacity savings over 30 days was projected at 92%—an 11.94:1 deduplication ratio.

Finally, ESG Lab copied the latest full backup to tape the copy to tape capabilities of Data Protector with the D2D, which can be used for archiving. D2D frees up space by expiring source media (virtual tapes) rather than deleting the source data (files). This ensures that data that exists in multiple backups is not deleted until all references to it are deleted.

In 20 minutes, the copy to tape was complete.

Cost-Efficient Protection

Organizations of all sizes are struggling to meet the conflicting challenges associated with macro-level global financial uncertainty and micro-level information storage growth and complexity. A growing number of IT managers are turning to virtualization and consolidation technologies to meet these challenges. With a focus on scalability, automated management using rich software tools, and capacity-efficient pricing, HP’s data protection solutions are an excellent example of solutions that are purpose-built to address these issues.

ESG Lab created a total cost of ownership (TCO) model based on a hypothetical backup environment with multiple remote offices, a major data center, and remote replication for disaster protection.  The scenario examines cost-savings associated with moving from a tape-based to a VLS and D2D-based backup and recovery strategy, although it still assumes use of tape for long-term archive of backups. Costs were broken down by category:  capital expenditures, administrative costs, tape costs, maintenance costs, power and cooling costs, and total floorspace costs.  The cumulative costs for both tape- and disk-based backups were calculated annually over a five year period.  A number of assumptions were made and included in the calculations based on what a current IT organization might have in place for equipment, WAN connectivity, backup and restore policies, and capacity and performance requirements.[6] Comparisons were made between the total cost of ownership of a traditional tape infrastructure with no replication or deduplication and an HP Data Protection environment with disk-based backup, deduplication, and replication.

Figure 17 shows the total cost a hypothetical end-user would incur over five years when comparing a traditional tape environment to a backup to disk environment with replication and deduplication. The inflection point, where the disk environment becomes less costly than the tape environment, occurs just before the end of year two.

What the Numbers Mean

• The total cost of ownership of tape alone is roughly 69% higher than HP disk-based data protection with deduplication and replication.
• Data Protector software provides significant savings due to licensing based on data stored as opposed to data protected. Deduplication reduces licensing costs.
• The tape solution is more expensive in part due to the cost of acquiring tape media and the added complexity of managing the distributed tape infrastructure.
• While eliminating daily off-siting of tapes represents significant savings, tape is still the most cost-efficient method for long term archive of backups, and most organizations will replicate deduplicated data to a remote site for copy to tape for this purpose.

ESG Lab Validation Highlights

• Within five minutes of sitting down at the keyboard, ESG Lab was running a full backup of the first server to the HP VLS9000.
• The VLS 9000 demonstrated near linear performance scalability, achieving an impressive 4 TB/hour with a two-node configuration.
• The HP D2D Backup System achieved 81% bandwidth efficiency, transferring a 5 GB incremental backup across a 2 Mbit/sec simulated WAN link in just under 1 hour and 18 minutes for an effective throughput of more than 11 Mbit/sec. Replicating a second full backup yielded an even more impressive 90% bandwidth efficiency, transferring a 54 GB incremental backup across the same 2 Mbit/sec simulated WAN link in just over 6 hours and 42 minutes, for an effective throughput of more than 22.9 Mbit/sec.
• The HP Data Protection solution suite demonstrated faster, more reliable backups and restores at a significantly lower total cost of ownership than a tape environment.

Issues to Consider

• As with all VTLs today, when a cartridge is deleted or expired in a backup application, space on the VLS is not reclaimed until the cartridge is deleted, expired, or overwritten via the VLS management application. Integration with backup applications to automatically trigger a delete or expire in the VLS when a cartridge is expired in the backup application would be a useful enhancement.
• While the VLS, D2D, and Data Protector all have easy to use management interfaces, a single “manager of managers” that integrated all three products and provided an overall view of an entire enterprise environment would be of great value to administrators.
• While ESG is confident that one or more HP StorageWorks D2D Backup Systems can be used to meet the performance needs of a mid-sized organization,  D2D systems with more capacity and horsepower could reduce cost and complexity within larger mid-sized organizations.  HP has advised ESG that the new line of D2D Backup Systems, announced in June 2010, has been designed with these considerations in mind.

The Bigger Truth

ESG Lab conducted its first hands-on testing of the Hewlett Packard’s enterprise VTL, the VLS 6000, in 2006 and then validated the VLS 9000 in 2008. Testing and discussions with end-users have confirmed that HP’s disk-based backup solutions fit seamlessly into existing backup environments while providing dramatic performance and capacity reduction benefits compared to legacy tape-based methods. The HP StorageWorks D2D, aimed at delivering deduplication and WAN efficient replication to smaller, remote offices, completes a comprehensive, enterprise wide edge-to-core data protection architecture, managed by HP Data Protector, that goes beyond disk-based backup.

During this independent lab validation, ESG Lab confirmed HP’s edge to core capabilities in support of large enterprises as well as deduplication support across disk, replication, and tape throughout the product line. HP’s comprehensive offering with capacity efficient pricing provides high performance data deduplication technology to deliver dramatic disk capacity savings while offering scalable, predictable performance.

A modest two-node VLS configuration tested by ESG Lab was able to back up at a sustained 4 TB/hour.  Based on the nearly linear scalability observed by ESG,  an eight-node system should be able to protect data at 16 TB/hour. Easy to navigate, web-based management enabled ESG Lab to manage backups for a remote office from creation, through replication, and finally to tape—all using HP Data Protector software—giving rise to an edge-to-core data protection strategy covering remote and branch offices as well as multiple data centers. Direct attach to tape capability enables enterprises to meet offsite and deep archive requirements using familiar tools and techniques while keeping tape copy traffic off the SAN.

HP Data Protector software exemplifies the company’s depth and breadth of end-to-end solutions for backup and recovery encompassing disk and tape. It should bring significant value to customers grappling with the challenges associated with cost-effective management of their data protection resources.

In essence, deduplication has become a crucial component of disk to disk backups, but when considering competing methods for implementation, customers should consider the tradeoffs and what’s best for their organization: ease of implementation, cost, and bandwidth all play an important role.

ESG Lab believes that the combination of enterprise class performance and scalability—along with comprehensive storage management software and services—provides a unique approach for optimizing data protection and recovery strategy in the enterprise. Hewlett Packard customers can now retain more data for fast and reliable restores and longer retention periods while minimizing impact on backups with accelerated deduplication. Combined with the consolidated data management provided by Data Protector, customers have a wide choice of configurations which can be used to dramatically increase the role of disk in the protection of critical data.

Appendix

[2] Detailed configuration information can be found in the Appendix.

[3] Source: ESG Research Report, Data Protection Survey, April 2010.

[4] Full testing configuration is described in detail in the Appendix.

[5] Source: ESG Research Report, Data Protection Survey, April 2010.

[6] Assumptions and parameters can be found in the Appendix.

via eChannelLine USA – ChannelBIZ: Easing deduplication fear factors for customers.

Introduction

ESG Lab conducted this Validation Report in November 2009 prior to the public announcement of Albireo. At the time, we evaluated a Beta version of Albireo as Permabit was in early discussions under NDA with storage OEMs and beginning initial implementations. Permabit publicly announced Albireo High Performance Data Optimization Software on June 7, 2010.

Background

Back in 2001, when Data Domain was founded and EMC purchased Belgian startup FilePool for its Centera product line, few in the industry had heard of data deduplication.  Fewer still were aware that Permabit, founded in 2000, was hard at work developing scalable data deduplication technology.   Today, it’s hard to find anyone in the storage industry that hasn’t heard about data deduplication.

As shown in Figure 1, a recently completed ESG survey of 398 North American IT organizations decision makers indicates extreme interest in using data deduplication technology for data protection.  In fact, 43% of respondents currently utilize or plan to use some type of data deduplication technology to eliminate redundant data.[1] Within the data protection market, ESG expects that interest and adoption in deduplication will increase significantly over the next five years as it reaches mainstream market adoption.

Interest outside of the backup and recovery market is growing as well.  Solutions with built-in deduplication support are being used for disk-based archival (e.g., EMC Centera, Permabit Enterprise Archive).  In the primary storage market, data deduplication has been added to network attached storage (NAS) systems (e.g., NetApp FAS).   As a matter of fact, ESG’s research indicates that space efficiency and data reduction rank second and third, respectively, in the most important features and attributes when considering a NAS solution.  This survey of decision makers within enterprise-class organizations indicates that 30% would not purchase a NAS system without data reduction and 46% would strongly prefer a solution with this attribute.[2] These are surprisingly strong results since the majority of NAS solutions don’t support data deduplication.   Given the dramatic benefits that have been realized with existing data deduplication technologies,   ESG believes that it’s only a matter of time before policy-based deduplication support is added to primary block-based storage systems as well.

Introducing Permabit Deduplication Technology

Permabit’s field-proven data deduplication engine is now available as a software development kit (SDK) that provides unified data deduplication advisory services.    The Albireo SDK is unique in its ability to provide a wide variety of deduplication services.  The Albireo SDK can be used to:

• Reduce capacity within file- or block-based storage systems (e.g., NAS appliances or FC disk arrays)
• Process objects at the sub-file or single instance storage (SIS) level
• Remove duplicates in real-time (a.k.a., inline) or later after data has arrived (post-process)
• Eliminate application performance concerns
• Scale from a single storage controller to a cluster of storage controllers or a cluster of deduplication appliances communicating with a storage system over an industry standard Ethernet interface

Permabit was founded in 2000 by MIT engineers with a goal of developing  a scalable enterprise-class archive product with built-in data deduplication.   That product, now known as the Permabit Enterprise Archive, is shown towards the left in Figure 2. Industry standard servers are arranged in a grid, with each server acting as either an access or storage node. The storage nodes are packed full of high capacity SATA drives and presented as a network attached file system (NAS).   Servers can be added to the grid for increased performance and capacity.

In the Permabit Enterprise Archive product, a global pool with deduplication technology is implemented within a grid of servers.   Permabit has extracted this core deduplication technology to create the Albireo software development kit.   Albireo deduplication advisory services are accessed by software running within a storage system through an application programming interface (API) provided by Permabit.   A storage system with Albireo running within a single controller attached to a number of drive enclosures is shown towards the right in Figure 2.

Albireo indexes chunks of data and provides advisory notifications when duplication is detected. The software running in the storage system decides whether to take the advice. If the advice is taken, it is the responsibility of the storage system software to update references to duplicate data.  The Permabit Albireo API supports both block and stream-based access methods. The stream-based API provides content-aware segmentation to optimize deduplication based on the file type. Duplicate advisory services can be provided synchronously, or asynchronously using a registered callback.  Taken together, the Albireo SDK provides a flexible and powerful array of deduplication services.

ESG Lab Validation

ESG Lab evaluated the Albireo SDK during two days of hands-on testing at Permabit headquarters in Cambridge, Massachusetts.  The evaluation began with an overview of how Albireo deduplication advisory services are used within an existing storage system.  As shown in Figure 3, the software running within a storage system uses an API call to send new data and addressing information to Albireo.   Albireo ingests the data and performs a SHA-256 hash. A two stage lookup (memory and if needed disk) is performed to see if the data has already been stored. If the data is a duplicate, the API returns the location of the pre-existing data.  If the data is new, Albireo remembers the SHA-256 hash and the chunk’s location so that it can detect duplicate data in the future.

ESG Lab evaluated the Albireo SDK using a pair of programs developed by Permabit:

PBSCAN: This utility uses Albireo advisory services to determine the savings that can be achieved with deduplication. The program is routinely used at prospective customer sites to determine the benefits of deduplication with real-world data sets.

DD2FS : An open-source user-space file system was modified to illustrate the ease of integrating Albireo into an existing file system.

Each test program was implemented as a single process with the Albireo engine running as a separate service.  The pbscan utility was implemented with multi-threads working in parallel.  As shown in Figure 4, the test bed used for this phase of the evaluation consisted of a single quad core Intel Xeon 3 GHz processor with 4 GB of RAM.  For test purposes only and to more easily discern the resource utilization by Albireo, all but one of the processor cores were disabled in BIOS.

While ESG Lab testing was performed with a single processor core, it should be noted that the extreme scalability of the Albireo engine has been proven in production customer environments.  Later in this report, we’ll take a look at how Permabit’s underlying deduplication technology is routinely deployed within clusters of multi-core servers.

Capacity Savings

ESG Lab used the Albireo-enabled pbscan utility and real-world application data collected from Permabit’s production IT servers to evaluate the capacity savings that can be achieved with Permabit deduplication advisory services.   The results for two data sets are summarized in Figure 5 and Table 1.

What the Numbers Mean

• A 43 GB set of common office productivity files (e.g. documents, spreadsheets, presentations) was reduced by 32.67%.
• Four VMware virtual server images with a total capacity of 157 GB were reduced to only 4.3 GB. VMware virtual server images are often highly redundant—especially when each of the virtual machines is running the same guest operating system as was the case for this test.    In this example, 97% of the capacity required for the VMware images can be saved with Albireo enabled deduplication.

The experiment was repeated for a pair of Microsoft Hyper-V images and a week’s worth of Microsoft Exchange backup images.  One of the backup images was a full backup; the other images were incremental.  The results for all of the data types evaluated by ESG Lab are summarized in Figure 6 and Table 2.  Note that the savings are shown as a deduplication ratio instead of a percentage of capacity saved.

What the Numbers Mean

• A pair of Microsoft Hyper-V images was reduced by a factor of 2.1:1
• A single week’s worth of Microsoft Exchange backups (weekly full, daily incremental) was reduced from 2,203 GB to 501 GB, providing an excellent deduplication ratio of 7.4:1
• A deduplication ratio of 36.2:1 was recorded for four VMware images.

Ease of Integration

ESG Lab evaluated the ease of adding deduplication to an existing storage solution using the FUSE open-source user-space file system framework.[3] The FUSE file system is built over the native ext3 Linux file system.  There’s no need to patch or recompile the kernel.   Permabit created a file system using FUSE named dd2fs. The dd2fs file system was modified to use Albireo to identify and eliminate duplicates.   Six Albireo API calls and 52 lines of supporting code were added to the 1,563 line FUSE demonstration program.  Synchronous (inline) and asynchronous (post-processing) deduplication modes were demonstrated.

As shown in Figure 7, the update_block function passes data to be written to the Albireo uds_index_block API.  A return value of 1 indicates that the block is a duplicate and that it can share storage with the canonical chunk. After a bit of bookkeeping and error checking, the storage associated with the duplicate block is freed.  Other than this key function call, the majority of the Albireo-related code changes were isolated to initialization, shutdown, and handling callbacks when operating in asynch/post-process mode.

The FUSE demo was run to see Albireo-enabled deduplication in action.  As shown in Figure 8, a 64 KB file full of random data was written to an empty Albireo-enabled file system. A copy of the file with a new name was added.  The Linux df and du utilities were used to verify that the user’s view of the file system included the space consumed by both files, yet only a single file’s worth of disk capacity was consumed.  The Linux md5sum utility was used to verify that the files were the same.  As each file was deleted, the file system capacity and underlying disk capacity were checked.

Resource Utilization

Identifying duplicate data is a resource intensive operation.     First, data needs to be fed into a deduplication engine.  Moving a lot of data can consume a lot of bandwidth.  Next, the engine needs an algorithm which can be used to quickly, and accurately, identify duplicates.    Most deduplication solutions use cryptographic hashing functions to identify duplicates, but hashing a lot of data can consume a lot of CPU horsepower.  Last, but not least, the deduplication solution needs to maintain an index of previously processed data to find and keep track of duplicates.  Most deduplication solutions use a two stage lookup, with the first lookup occurring in memory and the second occurring on disk.  Indexing lots of possibly duplicate data can consume a significant amount of memory.   Each of these resource issues can have an effect on the overall performance of a storage solution.

ESG Lab’s analysis of CPU, memory, and performance efficiency began with a review of Permabit’s patented two stage deduplication detection algorithm. [4] The patent describes a highly efficient two stage index residing in memory and on disk.  This novel approach uses a combination of bit sampling and byte differencing to provide a first stage memory lookup that executes very quickly, consumes very little memory, and has a very low probability of false positives.  ESG Lab has confirmed that the Permabit deduplication advisory services engine:

• Delivers fast in-memory deduplication lookups that take between 9 and 17 microseconds[5]
• Requires less than 3.5 bytes of memory to index each chunk of deduplicated  data[6]
• Allows a single server to deduplicate up to 48 TB of data with under 2.5 GB of RAM[7]

A 42 GB set of office productivity files (documents, spreadsheets, presentations, PDFs, etc.) was processed by the Albireo-enabled scan utility.  The scan utility was used to quantify the processor, memory, and performance impact of Permabit deduplication advisory services. The results of three tests were compared:

1. Scan only: opened the file system and read 42 GB of office productivity files from beginning to end. The 42 GB was spread over 26,094 files.
2. Scan and index (64 KB chunk): opened and passed all of the data within each file to the Albireo API. The Albireo API was used to identify and keep track of 64 KB chunks of data in a Permabit index of duplicate candidates.
3. Scan and index (4 KB chunk):  repeated the second test with a chunk size of 4 KB.

Linux utilities were used to record CPU and memory utilization. Trace output was used to record the average latency for each Albireo deduplication lookup. In particular, trace data recorded all of the latency associated with a Permabit deduplication advisory services call, including the SHA-256 ingest, the first pass memory index, and a second order index operation for likely duplicate candidates.  The elapsed timed needed to process the 42 GB file set was used to calculate throughput.   The results are summarized in Table 3 and Figure 9.

What the Numbers Mean

• The scan only test spent most of the time reading from disk and consumed very little CPU (less than 1%).
• Permabit deduplication advisory services consumed approximately half (28% to 59%) of a 3 GHz Xeon processor core.
• Scanning and indexing with a larger chunk size consumes slightly more CPU. This is due to the fact that bigger chunks of data were passing through the CPU intensive SHA-256 algorithm.
• A SHA-256 scan and index with a 4 KB chunk size incurred only 43 microseconds of latency.  That’s less than 1% of overhead compared to a typical disk-based latency of 5 milliseconds.
• A single 3 GHz CPU core running at less than 75% CPU utilization was able to sustain 124 MB/sec of throughput.   ESG Lab is confident that Albireo can deliver significantly more aggregate throughput by scaling the number of processes, processor cores, and servers working in parallel.

Maturity

ESG Lab performed a high level assessment of Permabit’s software development processes to understand the maturity of the Albireo SDK.  The bulk of the code within the Albireo SDK has been used within Permabit’s shipping products for more than six years.  Permabit has been using agile software development processes for more than seven years.

Agile software development refers to a group of software development methodologies based on iterative development where requirements and solutions evolve through collaboration between self-organizing cross-functional teams. The term was coined in the year 2001 with the formulation of the Agile Manifesto. [8]Agile methods generally promote a disciplined project management process that encourages frequent inspection and adaptation; a leadership philosophy that encourages teamwork, self-organization, and accountability; a set of engineering best practices that allow for rapid delivery of high-quality software; and a business approach that aligns development with customer needs and company goals.

The code at the core of Albireo SDK is continuously integrated and tested on a two week iteration cycle. Stories captured in an online Wiki are used to manage requirements.  Unit, functional, and stress tests are highly automated and run continuously.  Developers write the majority of unit and system level tests. All changes are peer reviewed. Face-to-face cross functional interaction is embraced.  Reducing code complexity is valued and recognized.

Scalability and Fault Tolerance

Permabit’s global deduplication algorithm is designed to run within a grid for maximum scalability and fault tolerance.   Permabit builds systems that can scale to 4.6 petabytes today (that’s 4,600 terabytes).  Its Enterprise Archive product is continuously tested with deduplication services running on a grid of three or more servers.  A typical entry-level grid at a customer site, comprised of access and storage nodes, has deduplication services running on 11 nodes. The largest grid deployed in production at a customer site is 38 nodes.

ESG Lab performed a series of tests on a nine node Permabit Enterprise Archive system to determine whether deduplication advisory services continue running after multiple hardware failures.  A long running directory level file copy operation was started. As shown in Figure 10, a node was powered off and a drive was removed on a different node. The system remained available and the copy operation completed without error.

ESG Lab Validation Highlights

• An open-source user-space file system was modified to use Albireo deduplication advisory services using six API calls and only 52 lines of code.
• Synchronous and asynchronous APIs were used to implement inline and post-process data deduplication, respectively.
• The Permabit SDK identified potential capacity savings ranging from 33% to 97% for real-world applications including office productivity files, virtual server images, and e-mail backups.
• Fast and resource efficient deduplication indexing was confirmed.  Less than 3.5 bytes of memory per index entry was recorded.  Ingest and deduplication detection running at speeds of up to 124 MB/sec was recorded using approximately half of a 3GHz Intel Xeon processor core.
• Permabit’s agile software development processes were audited.
• Systems in the lab and in the field confirm that Permabit global deduplication advisory services have been deployed on grids of up to 38 servers.
• An error injection test confirmed that the Albireo deduplication services running within a field-proven Permabit Enterprise Archive solution remain available after both a drive and a server failure.

Issues to Consider

• The Permabit Albireo SDK detects duplicate data, but it does not actually remove it.  Removing duplicates and maintaining pointers to duplicate data is implemented within the storage system using the Permabit SDK.  Data structures, which map and keep track of duplicate data references, are needed to take advantage of the Permabit SDK. This is a trivial consideration for NAS systems which use an inode map to keep track of data on disk. For modern block-based disk arrays, this service is often available, but it may be an issue that may have an impact on the complexity, and resources, associated with Albireo integration.
• An Albireo-enabled storage solution continues operating even if Albireo becomes unavailable.  If and when Albireo becomes unavailable, the system is unable to detect new duplicates but data access is unaffected.  This is due to the fact that Albireo provides deduplication advice and the storage system uses that advice to maintain data integrity.
• While the Permabit architecture has been designed to use any hashing algorithm, Permabit has been using the SHA-256 algorithm for years.   While the 256 bit SHA hashing algorithm virtually eliminates the risk of deduplication induced data corruption due to a hash collision, it does add CPU overhead compared to less rigorous 64 bit algorithms (e.g., MD5).
• The API integration and capacity savings results presented in this report were collected using relatively simple test programs running on a single server.    Estimating the effort required to evaluate, architect, and implement a solution using a production storage system is beyond the scope of this report. Similarly, estimating the savings that can be achieved with your customer’s data is beyond the scope of this report.  Testing in your lab, with your storage system, and with your data is strongly recommended.

The Bigger Truth

One of ESG Lab’s first projects was a 2004 validation of a disk-based backup appliance with built-in data deduplication from Data Domain.[9] Since then, data deduplication has evolved into the hottest, most paradigm shifting technology to hit the storage industry since the UC Berkeley RAID papers were published in 1989.  Like RAID, data deduplication quickly permeated the storage market due to its outstanding value proposition.  Storage administrators struggling to finish backups within shrinking windows were able to reduce the capacity required to retain backups on disk by 90% or more.   The value of this new technology was clearly compelling: data deduplication reduced the cost of disk-based backups, putting it on equal, or better, footing than tape.   Backups that finish within a shrinking window and quick ad-hoc restores from disk had suddenly become economically feasible.

In recent years, data deduplication has begun to permeate the storage industry.  A number of startups, including Diligent, Sepaton, and Exagrid, followed Data Domain into the disk-based backup appliance market.    Since then, all of the major systems vendors have added deduplicating backup appliances to their portfolios.  More recently, all of the major backup software vendors have added deduplication to their offerings.  Content addressable disk-based solutions with embedded data deduplication technology were introduced in the archive market.  Permabit was among the first vendors to enter this growing market.  Deduplication has been used to reduce WAN traffic within primary and secondary storage replication solutions. And last, but not least, data deduplication is beginning to take hold in the primary storage market, with NetApp and Microsoft leading the way (Deduplication for FAS and Single Instance Store within Windows Storage Server, respectively).  In ESG’s opinion, it’s simply a matter of time before data deduplication gains wide market acceptance in the nearline archive and online primary storage markets.

Data deduplication technology has driven a number of strategic acquisitions.  ADIC purchased deduplication technology from Rocksoft for $63M (ADIC has since been acquired by Quantum).  IBM acquired Diligent in a deal rumored to be worth between $160M and $200M.  Those acquisitions were dwarfed by EMC’s recent acquisition of Data Domain, where a bidding war with NetApp drove the value of the deal up to $2.1B in cash.

So what’s the big deal with deduplication?  It’s actually rather simple: deduplication reduces storage capacity requirements up to 99% for primary data and backups stored on disk.  In other words, IT managers can squeeze up to a hundred times more out of each dollar they spend on disk capacity.  Even as data deduplication becomes more of a feature than a product, the value is clearly compelling.  While one could argue that the hype and valuations of deduplication solutions in the backup arena have gotten a bit ahead of the market, it’s clear to ESG that we haven’t seen the peak in the archive and primary storage markets yet.

The deduplication market has begun to mature in recent years.  As the feature becomes more of a check-off item within backup solutions, vendors are leveraging differences in architectures and implementations to grow market share.  Aside from the usual delineations based on price and performance, vendors are competing based on the finer differences between deduplication solutions:  object vs. block-based, inline vs. post-process, fixed vs. variable length, and global vs. islands of deduplication.  Permabit has extracted the core deduplication technology from a field-proven archiving solution with a goal of delivering a deduplication algorithm that can be used to architect solutions with any of these attributes in mind. In other words, instead of arguing the merits of one implementation vs. another, Permabit enables a vendor to implement multiple alternatives and just say yes.

ESG Lab has confirmed that the Permabit Albireo deduplication advisory services work as advertised. Inline and post-process deduplication support was added to a user space file system with only six Albireo function calls and 52 lines of code.  The capacity of real-world data sets were reduced between 33% and 97%. The patented deduplication lookup and indexing algorithm was fast and efficient.   Permabit deduplication was observed running on more than one server for a scalable global pool of deduplication and fault tolerance.  ESG Lab saw no interruption in access when errors were injected on a nine node Permabit Enterprise Archiving cluster.

ESG Lab’s experience with nearly all of the vendors that have brought data deduplication solutions to the market indicates that correctly implementing data deduplication is a difficult task that requires man years of effort.   Performance, resource efficiency, and scalability have proven to be particularly challenging for a number of vendors.  On top of the technical challenges, this relatively new and valuable technology has a growing number of patent portfolios that need to be navigated.

Speaking of patents, Permabit has been awarded a total of 16 patents covering diverse areas in data protection and archive and many more filings are pending in similar areas. The growing portfolio includes patents in the areas of hash–based deduplication for scalable file and object data storage, encrypted deduplication, memory based snap shots, and many other features of Permabit’s product line.  ESG Lab was particularly impressed with the resource efficient two-stage indexing method described in Patent 7457813, Storage System for Random Blocks of Data. Highlights of that well-claimed patent are summarized in the resource efficiency section of this report.

ESG Lab is confident that the flexibility provided by the Alberio SDK is unique in the industry.  ESG Lab has confirmed that Alberio can be used with file or block storage. It provides deduplication services at the object or sub-file level. It supports an inline and post-processing programming model with minimal performance and resource impact.  Running over a grid, it can be used to create a global pool of deduplication with predictably scalable performance and rock solid reliability.  It provides deduplication capacity savings that are far greater than can be achieved with compression. Stream support can be used to provide content aware data deduplication for objects that are misaligned with the block boundries of an underlying file system.  This allows data stored in container formats (e.g., TAR and ZIP files) to be intelligently deduplicated.

Last, but not least, the Permabit Albireo SDK was designed with quick and easy integration in mind. Based on hands-on experience with the Permabit Albireo SDK, ESG Lab believes that Permabit deduplication can be tested within an existing storage system in a matter of weeks.  Given the growing size of the market for capacity reduction and the high cost of developing a deduplication solution, organizations considering the merits of adding data deduplication to an existing storage system should seriously consider a test drive of Permabit’s field proven, patent protected deduplication algorithm.

Appendix

[2] Source: ESG Research Report, Enterprise Storage System Survey, November 2008.

[3] fuse.sourceforge.net

[4] US Patent 7457813, Storage System for Random Blocks of Data, Nov 25, 2008.

[5] Confirmed via a review of traces of code execution through the in memory index code path.

[6] Confirmed via memory usage comparisons presented later in this section.

[7] Depending on the size of the chunks that are used for deduplication and whether Permabit memory constrained mode is in use.  Memory constrained mode was not tested by ESG Lab.

[8] Agilemanifesto.org

[9] See: ESG Lab Report, The Data Domain DD200 Restorer, February 2004.

I spend a majority of my time researching the information management software market–solutions that deliver search, access, and business process management / automation  (including electronic discovery and retention management) capabilities.  As I was driving home from SFO the other night I saw a billboard for Box.net – a “cloud” content management provider and alternative to SharePoint.  The billboard highlighted “two software updates per month versus two years”–a direct shot at Microsoft which has pretty much standardized on bi-annual product releases.

Let me connect my two disparate thoughts.  One of the primary reasons that “cloud-based” software or “Software-as-a-Service” threatens major “on-premise” applications is the agility without risk.  Upgrades are delivered automatically and may be executed a few times per month as Box.net proclaims.  With on-premise software, a single upgrade can take months and, if anything goes wrong, it could be much longer and more expensive.   Now, there really isn’t an equivalent in the systems business–especially in storage.   However, one could argue that Permabit’s approach–delivering embedded software that runs on x86 architecture / standard operating systems that will ultimately serve as a single feature of the system–could be the next best thing.  Let’s say you build primary storage systems and you want to add dedupe, but your next product release is scheduled 12 months from now.  The issue is that you have already completed this future system design and, unless you make substantial changes, the next possible opportunity to add dedupe is a product that will be delivered three years from now (2 years from the next scheduled release).  Alternatively, you could leverage embedded software to minimize the R&D investment and get the capability to market in the next product cycle.  In other words, getting new features into a system using a third party-embedded technology developer is the way hardware systems become more “cloud-esque”.   The bullets explain the benefits:

• Users can still buy tangible assets–but can get upgrades via code loads from their favorite vendors.
• System manufacturers can add features more rapidly in the middle of planned product cycles.
• System manufacturers can focus R&D resources on getting the next solution to market quicker, potentially accelerating product cycles.

The bottom line is, while there are new R&D methods (agile development methodologies), there are other ways for developers to get features into hardware solutions much quicker and thus new capabilities into users’ hands faster.  Sometimes those other means may not be that apparent unless you study the software marketplace.

Read Brian’s other blog entries at IT BULLETins.

It’s definitely been interesting to watch capacity optimization technology (deduplication and compression) evolve.  The most notable trend is the move upstream.  Just last month, EMC announced its “Boost” option for Data Domain, which enabled Data Domain to move a portion of its deduplication processing up to the media server in NetBackup OST environments—thereby distributing the workload and gaining some network and performance efficiency in the process.  In a similar move, Symantec extended its deduplication processing to client systems (from the media server) to gain end-to-end efficiency in the backup process.

These shifts shouldn’t come as a surprise.  Vendors (and end-users) realize that there are big benefits to “stopping the avalanche at the top of the mountain.”  They’re also probably drooling over NetApp’s success with its storage efficiency efforts—headlined with NetApp deduplication in FAS arrays.  Just last month, NetApp posted seriously good financial results; its focus on server virtualization initiatives, where redundancy and proliferation create an optimal use case for deduplication, is probably a contributor to these results.

Permabit, long a provider of deduplication technology through its archive solution, has packaged its technology up for wide consumption.  Offering its Albireo High Performance Data Optimization Software via an SDK allows the primary dedupe “have nots” to embed Permabit’s time-tested deduplication technology into their primary storage offerings.

The objection every vendor moving the process upstream has to manage will be performance vs. cost.  Implementing deduplication to save bandwidth and capacity makes sense … but not if the added processing impacts application performance.

Read Lauren’s other blog entries at Data Protection Perspectives.