“All we can afford is tape.”

“We’re small – so we can’t afford enterprise stuff.”

“We only have 40 people. If we have to recreate something, it isn’t too bad.”

Respectfully, none of these are reasons — they are all excuses — and they are mostly inaccurate.

“All we can afford is tape.”

WRONG — no one, of any size, can only afford tape. For most crises, ranging from regional weather catastrophes to single hard-drive failures, you don’t need last month’s or last year’s data from tape. What you need is yesterday.  You need it online, now! By definition, that is not tape — that is disk. The reality is that what you cannot afford is “just” tape. Whether you start with an external disk drive or engage with a cloud-backup provider, you can afford something (anything) better than tape.

“We’re small — so we can’t afford enterprise stuff.”

PARTIALLY TRUE – you can’t afford “enterprise stuff.” But hey, you don’t need enterprise stuff since you aren’t an enterprise. If you did run an enterprise infrastructure, with all of its scale and complexity, then you could afford enterprise stuff. But you don’t, because you are an SMB. As an SMB, you’ll likely have a more straightforward collection of IT assets, such as file serving devices, some applications which are likely based on some SQL platform, perhaps a mail platform or two (unless you run from a cloud-provider) and perhaps a custom application platform. The great news is that these kinds of platforms are very cost-effectively protected. File serving can replicate, either within the file system (e.g., Windows Server DFS) or the storage layer, applications like databases and email can often replicate between multiple instances — often yielding not only resiliency but higher performance due to load balancing. Any server platform that isn’t natively resilient can usually be encapsulated into a virtual machine and then replicated and re-booted from a mirrored virtualization host.

The punchline: you don’t need “enterprise stuff” to protect your SMB IT infrastructure. In fact, in many cases, resiliency features are already built into what you already own.

“If we have to recreate something, it wouldn’t be too bad.”

Almost definitely WRONG, because most SMB data is unique. This means that if something took you 60 minutes to do the first time, it will likely take you 45-50 minutes to recreate something that was done recently and potentially 90 minutes to recreate older content that you still need. You’ll potentially gain some efficiency because you already have the intended outcome in your head — but it still will take the majority of time to repeat. But if you lose several days of data, can your SMB workforce tolerate the several days of recreating work that you’ve already done and likely already gotten paid for?

Limited Cashflow

One of the key differences between how enterprises and SMBs tolerate crises is cash flow. Based on the kinds of business that has recurring and/or pipelined revenue or just because their bank accounts are bigger, they are often able to suffer cash flow issues while they reconstruct. SMBs typically lack those deep coffers or assured income, so it is even more important that SMBs proactively deal with business continuity since they won’t be able to roll up their sleeves and bear through their recovery efforts

Limited Expertise

Another reality is that most SMBs don’t have deep IT veterans on-staff and waiting to throw on their superhero capes and jump into action during a crisis. This isn’t meant as a disparagement, as I know several IT Pros that work in SMBs who have built near-enterprise quality and services into their smaller IT infrastructures. But as is much more common, I have met many brilliant IT implementers working for systems integrators and consultants, who provide their services to SMBs. In this way, SMBs get deep IT expertise when they need it (billable), but don’t have to pay for it when they don’t — often as a project-level supplement to their in-house part-time IT person. And like most things in IT, that works great when everything is working … and is painful when something breaks. First, because when an SMB has an urgent outage, the right guy may not be available for immediate resolution. And secondly, while the SMB is down and likely losing revenue, they have to do the one thing that they don’t want to — pay for something expensive, a billable IT expert.

So, what is an SMB to do?

Invest pennies now … instead of dollars later. Today, most key workloads that an SMB relies upon are natively resilient — if you turn the features on:

• SQL Server and Exchange both offer replication with transparent failover through Database Mirroring and DAG, respectively.
• Windows infrastructure services (AD/DNS) are natively resilient and easily solvable with a simple secondary instance (running in a VM).
• Windows Server (file services) has DFS to transparently replicate/load-balance and resume service between servers and sites.
• Even virtualization platforms can be made resilient for far less money than you might think.

The reality is that SMBs arguably need resiliency as much, if not more than, some enterprises — and the tools that they need are more often than not already owned by them.  I am so passionate about this (using the built-in HA/DR technologies that you’ve already paid for) … that I wrote a whole book on it = Data Protection for Virtual Data Centers … also viewable at DataProtectionBible.com.

Whether you read the book or not, if you are an SMB, look at the core workloads that you are running in your shops – and then look in their product documentation for “availability” or “replication.”  You may be happily surprised with what you find – and already own.

In my earlier post, I talked about the 10 professional practices of Business Continuity Planning.

Practices 2 and 3 include “Risk Evaluation and Control” and “Business Impact Analysis.” They are essentially about gathering an understanding of what each of your core business processes are, the IT infrastructure that supports them, and a qualitative and quantitative assessment of what would happen if those operations were impacted. Folks, this is a very healthy thing that isn’t done enough in corporations. Too often, companies grow their business processes and the supporting IT functions over time and incrementally–but fail to step back and re-assess where they are.

Some of the quantitative math for a BIA and RA, as well as TCO/ROI, RPO/RTO and metrics, are available in a sample chapter from my book, which is available as a free download — Chapter 2 of Data Protection for Virtual Data Centers.

When you start BC/DR planning, here is what you’ll get:

Along with reassessing where you are, BCP process 4, “Business Continuity Strategies” starts to look at what kinds of mitigating technologies could be applied to resolve the potential IT faults in your environment:

• Invariably, you will end up putting in replication and failover technologies, where standalone servers used to be.
• You’ll discover new protection and recovery methods, such as cloud-based backup, that originally seemed unattainable and now appear more cost-effective and reasonable.
• You’ll almost definitely become much more aware of the business processes that can be improved.
• And, you’ll discover aspects of your infrastructure that could be consolidated or optimized.

Read those words again — optimized, improved, more cost-effective. Of course, you’ll also be ‘prepared,‘ but the lesson is that while BC/DR preparedness may be the cause, business optimization is the effect.

The server and storage virtualization that enables booting up an alternative data center during a crisis will also reduce your hardware, power, cooling, and space costs now.

The replicated data and resilient applications that ensures continuous operations of your business during a crisis will also assure productivity during a server-level issue or planned migration — and in many cases, improve performance through load balancing throughout the rest of the day and during peak usage hours.

The conversations that you’ll have with your business stakeholders will not only prepare you to be ensure their service during a crisis, they will also almost definitely create conversations where some of those operational processes and procedures can be optimized now. They will also create new dialogues of understanding between the business stakeholders and their IT counterparts which will yield big benefits of empathy and cooperation.

Some of these BC/DR benefits are covered in Chapter 12 of my book — but hopefully, this gets you thinking about BC/DR planning in a new way. While your preparedness and company survival may be the destination, you’ll be amazed how much that you’ll gain during the journey.

As always, thanks for reading.

First, let’s talk through the layers from a technology perspective:

Backup & Restoration (B/R)

B/R is about previous versions. On disk, there is an expectation around either rapid recovery–or end-user initiated without helpdesk interaction. Each are good, both are better. On tape, it is all about long-term retention (only). If you are still using tape as a primary means for data restoration or whole-server recovery, then …

High Availability (HA)

HA is about staying running. It used to be about ‘failover,’ where warm resources would come online when the primary failed – but these days, HA scenarios rarely miss a beat, because most have figured out how to combine load-balancing methods with HA, whereby when everything is fine, it is shared across devices so that load is optimized. When something fails, the other nodes of the configuration take up the extra slack and the user is never the wiser. An example being Exchange DAG or SQL Mirroring within the application or a web-farm.

Disaster Recovery (DR)

DR is about data survivability over distance. In olden days, this would then be followed with the IT and personnel processes to bring the data and services online again from the remote location. But the key, then and now, is that DR from a technology perspective is about data survivability–achievable through the application (e.g., SQL replication), the host (e.g., Windows DFS-R), from a backup application (Microsoft DPM), or the storage array.

Business Continuity (BC)

Simple math: BC = DR + HA … +/- B/R

Said another way: BC is remote (DR) resumption (HA) of services, ideally either transparently resilient or automated such that the decision is manual but the execution is programmatic, with the option of optimized backups (B/R) from the remote site.

Your Remote Replication feature is not my Disaster Recovery

All of the above is from a technology perspective, which is nice and probably worth more detail in another blog post as to what makes offerings in each area desirable, but … there is at least one flaw in the above descriptors. The ‘Remote Replication’ feature in your technology does NOT really equate to DR or BC.

Marketing guys are taught to use action verbs. So while your remote replication feature may genuinely ‘help you with your disaster recovery goals,’ many marketing guys (including me in my past roles) shorten it as ‘does disaster recovery.’ And somewhere along the way, product developers listen to the marketing guys and product managers, and then the feature or button in the product becomes ‘Disaster Recovery.’ But it isn’t! Because real BC/DR isn’t about technology–it’s about processes and people.

For all of the IT Pros who are certified in this technology or that, there is a certification for BC/DR – so let’s look at their defined ‘Professional Practices’ for Business Continuity Planning at www.drii.org/certification/professionalprac.php:

1. Program Initiation and Management
2. Risk Evaluation and Control
3. Business Impact Analysis
4. Business Continuity Strategies
5. Emergency Response and Operations
6. Business Continuity Plans
7. Awareness and Training Programs
8. Business Continuity Plan Exercise, Audit & Maintenance
9. Crisis Communications
10. Coordination with External Agencies

The punch-line here is that all of the IT technologies and methodologies are applicable to roughly 4 of the 10 aspects of real Business Continuity Planning.

Putting it all together

Are the ‘Disaster Recovery’ features in backup and storage technologies bad? NO, but take them for what they are. Storage that is replicated to a secondary site delivers disaster recovery on its own unless its only function is a file server and it offers built-in CIFS/NFS. Unless you also have the ability to bring the services back online, and reconnect the users, and report its restored availability, then it isn’t BC or DR, its RR–Remote Replication.

I won’t say “it is only remote replication” because RR is the secret sauce for ensuring data survivability. And without your data, most of the 10 phases of “real” business continuity planning won’t matter.

The first words and the last words in my book are “Get Your Data Out of the Building”–and remote replication is how you do it. In fact, chapter 12 of the book is around 40 fun-filled pages on how to make BC/DR work within a BCP framework and also how all of it applies to regulatory requirements.

So, embrace Remote Replication (RR)–just don’t call it ‘Disaster Recovery’ or expect it to entirely solve your Business Continuity goals by itself.

Disclaimers:

• Yes, as a former (and recovering) marketing guy, I have shortened out the ‘helps’ and ‘may’ phrases for BC/DR technologies.
• Yes, I have been a Certified Business Continuity Planner (CBCP), as well as an IT Pro (MCSE and Novell MCNE).
• Yes, these are simply my opinions and yours may be different.

Thanks for reading.

via Replication technology for business continuity and disaster recovery.

Introduction

Background

ESG recently asked IT professionals in North America and Western Europe about their top storage spending priorities over the next 12-18 months. Their responses make clear the importance of data protection: the top two priorities were backup and recovery solutions (36%) and data replication for off-site disaster recovery (24%). Also high on the list was improving storage management software tools (21%). Given the financial damage that can occur in a data loss event, the focus on data protection makes sense.[1]

EMC VNX Total Protection Pack

The EMC VNX Total Protection Pack provides local, remote, and application data protection for EMC VNX arrays. With EMC RecoverPoint/SE serving as the core replication engine, this data protection suite integrates continuous data protection (CDP) and continuous remote replication (CRR) with compression, bandwidth reduction, and support for both physical and virtual servers. Block-based data and file system remote replication are currently supported with RecoverPoint/SE on VNX Series arrays. Integration with EMC Unisphere and VMware vCenter enable unified management from a single location. The Total Protection Pack’s key capabilities include:

• Continuous local and remote data protection and recovery with DVR-like rollback
• Point-in-time recovery of individual or multiple virtual machines with a single click
• The ability to define and manage recovery point objectives and SLAs across the infrastructure
• Automated failover and failback through integration with VMware Site Recovery Manager

The VNX Total Protection Pack consists of:

• Local Protection Suite: RecoverPoint/SE CDP, SnapView, and SnapSure for local protection and data re-purposing
• Remote Protection Suite: RecoverPoint/SE CRR, Celerra Replicator, and MirrorView A/S  for remote protection against failures and disasters
• Application Protection Suite: EMC Replication Manager and Data Protection Advisor for Replication Analysis for application protection and automated replication management

ESG Lab tested RecoverPoint/SE CDP and CRR, Replication Manager, and Data Protection Advisor for Replication Analysis. Figure 2 shows the RecoverPoint/SE GUI within the EMC Unisphere management interface, with the System tab displaying the replication topology. Note that RecoverPoint/SE, Data Protection Advisor, and Replication Manager (as well as other management wizards) can be launched directly from the Unisphere management screen (right side navigation).

RecoverPoint/SE

As part of the RecoverPoint product family, RecoverPoint/SE is replication software that runs on SAN-attached RecoverPoint Appliances (RPAs) that protect physical and virtual hosts, providing unified SAN/NAS replication for EMC VNX arrays. Also in the RecoverPoint family are RecoverPoint/EX for replication within VMAXe and VNX arrays and RecoverPoint/CL which adds support for EMC VMAX and also non-EMC storage environments. They provide synchronous local replication with CDP and synchronous and asynchronous CRR, as well as concurrent local and remote replication. RecoverPoint tracks changes to data at the block level and journals those changes using CDP technology; this journal-based architecture enables rollback to any point in time and simultaneously reduces storage costs. WAN deduplication and compression features reduce TCO by reducing overall WAN bandwidth requirements. Management is integrated with both EMC Unisphere and VMware vCenter; in addition, RecoverPoint/SE supports all features of VMware vSphere 5.0 including Site Recovery Manager (SRM) automated failover and failback. The easy-to-manage RecoverPoint/SE product can be used to replace multiple application-specific point solutions.

Replication Manager

Replication Manager (RM) automates the creation and management of application-consistent replicas without impacting production performance for tasks such as instant restore, backup, and creating test/dev data sets in physical and virtual server environments. It integrates with the EMC RecoverPoint family as well as with EMC TimeFinder, SnapView, SnapSure, SAN Copy, and Celerra Replicator. It automates discovery of the environment at initial implementation and when changes occur. The point-and-click interface, pull-down menus, and wizards replace complex scripting and manual mapping between replication technologies, storage, and applications/file systems. Built-in intelligence places applications in the proper state to create application-consistent replicas, which can then be mounted to any host. RM’s simplicity, policy-based management, and definable user roles make it usable by most IT staff, not just storage or replication experts.

Data Protection Advisor

Data Protection Advisor (DPA) provides unified monitoring, analysis, alerting, and reporting across the data protection environment. It collects information about data protection automatically to inform IT decisions and help administrators correct problems and meet SLAs; the single, integrated view brings simplicity to a complex environment and helps IT work more effectively. DPA takes volumes of disparate data and turns it into actionable knowledge, enabling organizations to reduce costs by more efficiently managing people, processes, and equipment. Improved visibility helps to reduce risk and ensure compliance by alerting IT about bottlenecks that can impact replication, recoverability exposure, policy enforcement status, and more. The bottom line? Problems can be resolved with less effort and cost.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of components of the VNX Total Protection Pack at EMC’s lab in Hopkinton, MA. Testing of the VNX Total Protection Pack was focused around RecoverPoint/SE, Replication Manager, and Data Protection Advisor for Replication, designed to demonstrate the integration of local, remote, and application protection along with consolidated management through Unisphere.

Test Bed

Figure 3 illustrates the test configuration. Local and remote servers were configured with VMware ESX 4.1. Both environments were connected with 4 Gb iSCSI SANs to EMC VNX 5500 arrays. The RecoverPoint/SE configuration consisted of two Dell R610 servers at the local and remote sites with RecoverPoint/SE software version 3.4.1.1. These were connected via SAN and 1 Gb LAN. At the remote site, RM and DPA were run on VMs.

Ease of Management

ESG Lab tested the ability to execute replication configuration and management from within EMC Unisphere, the VNX array management interface.

ESG Lab Testing

ESG Lab was able to view and manage replication from a consolidated EMC Unisphere GUI. Figure 4 shows the dashboard view. Graphical summaries of storage capacity for the local and remote VNX arrays are shown, as are array status and details. Alerts would be displayed here and a link to the virtual server is available in the lower left window. To add other types of information to the dashboard, the administrator simply drags icons from the dashboard options. Different systems can be viewed by selecting them from the top navigation.

Management tasks are menu- and wizard-driven. Figure 5 shows the RecoverPoint wizard for configuring and creating replication consistency groups to a RecoverPoint Appliance. Settings such as bandwidth reduction, protection policy, resource allocation, clustering, and VMware SRM support are defined here. At least one journal LUN must be created for each consistency group copy; during testing, journal LUNs were created for the production data, local copy, and remote copy to ensure transactional consistency for recovery. Consolidated management makes this simple: because the RecoverPoint GUI is available within Unisphere, ESG Lab was able switch easily to the Unisphere GUI to create the new LUNs instead of having to exit one application/management station and start another.

ESG Lab was able to view the status of different portions of the environment through the RecoverPoint GUI. Figure 6 shows the view from the VMware vCenter API integration, including IP address, consistency group protection status, and replication set details. RecoverPoint’s integration with VMware vCenter provides management visibility into the virtual servers being managed within a replicated environment, allowing for real-time management of the production, local, and remote replicas within a single management interface. This allows administrators to manage the protection of their virtual server environment from a single console.

VMware 5.0 SRM integration is also part of the package; in the RecoverPoint tab, the administrator can choose to let SRM manage recovery or allow the administrator to manage failover within the RecoverPoint GUI.

Continuous Protection

The VNX Total Protection Pack includes RecoverPoint/SE, which provides simultaneous local and remote replication and disaster recovery for physical and virtual environments. Continuous protection is provided by the creation of sub-second snapshot copies that provide a set of time-stamped images that can be restored if needed.

ESG Lab Testing

ESG Lab launched the RecoverPoint/SE GUI from within Unisphere. After creating a consistency group, both local and remote replication processes were started. Figure 7 shows the RecoverPoint/SE GUI; the large graphic shows the view from the defined consistency group. The production site is creating local and remote copies simultaneously (represented by the dotted green lines). Local and site-to-site system traffic is charted in the upper right corner, while the upper left graphic shows the replication topology.

Next, ESG Lab created a failure by disconnecting the LAN. Figure 8 shows the topology view indicating a failure in the continuous protection process. The system traffic chart shows no local or remote replication traffic.

Next, ESG Lab tested remote recovery. After creating a bookmark to identify a replica at a particular point in time, ESG Lab deleted one file and edited another. The target site view (see Figure 9) shows the time-stamped CDP images available for recovery to another VM.

ESG Lab then selected “Enable Image Access” to confirm the viability of remote copies and journals to ensure proper restore (see Figure 10). Once the image was selected, a simple right click restored it to production; the journal was used to restore changes that occurred after the snapshot was created. Because replication is continuous, the local copy continues to journal changes during the recovery activity. Production resumed, completing the full failover and failback process.

Application Protection and Replication with Replication Manager

Replication Manager provides a single interface for automating all EMC replication applications, and is tightly integrated with RecoverPoint. Automation of scheduling, application-consistent replication creation, and workflow can reduce administrative effort and ensure that protection policies are enforced. In addition, Replication Manager enables self-service replication with its selectable user roles and privileges.

ESG Lab Testing

ESG Lab launched RM directly from the Unisphere management window. Figure 11 shows the view of replication jobs by application set. RM Server, storage pool, and storage service views are also available. The “Getting Started” pane lists the four steps required to launch RM: registering agents on servers to be protected and on hosts on which replicas will be mounted, adding replication storage, creating an application set to be protected, and creating a policy-based replication job.

Figure 12 shows the RM Job Wizard. From this window, ESG Lab selected RecoverPoint Concurrent Local and Remote (CLR) Replication. Other RecoverPoint/SE options are available, as would be any installed EMC replication software. Workflow can also be automated; for example, RM can be configured to launch another replication task immediately following completion of the current task to eliminate complex scripting.

Next, ESG Lab selected and restored the point-in-time replica from the Restore Wizard.

RM offers five levels of user-defined, role-based management (see Figure 14). This enables administrators to offer different levels of management according to user type, including self-service. Certain roles may be able to manage only particular storage pools or hosts, which can reduce IT’s administrative burden while ensuring that the infrastructure is managed properly.

Solution and Application Analysis with Data Protection Advisor

DPA can be used validate configurations and to analyze and report on replication activities; it enables IT to view replication and storage from the application perspective. The analysis engine monitors, discovers, and maps the environment. In addition, it automatically alerts IT to protection gaps via e-mail or SNMP; for example, if a new VM is created or a table space is added to an Oracle database, DPA will highlight that these additions are unprotected. DPA will also launch scripted actions and provide a view of systems trends. For example, if over two weeks a file system is filling up, DPA will give IT an idea of when action should be taken. While DPA discovers all details, role-based views can be created to show only what an administrator has access to. Integration with LDAP and Active Directory ensures that views are properly restricted.

ESG Lab Testing

ESG Lab confirmed that DPA maps connections across the replication environment. Figure 15 shows the EMC VNX view of replication connectivity. The arrows indicate which device is replicating to which. Storage topology, storage-to-server connections, exposure summary, and exclusion details can all be viewed.

DPA also reports on and provides visibility to successful and unsuccessful replications. As shown in Figure 16, DPA generated a replication gap alert, as well as alert icons when ESG Lab disabled the simulated WAN connection. The alert icons are displayed in the navigation and topology views with gap alert details shown at the bottom of the view.

ESG Lab Validation Highlights

• ESG Lab confirmed that VNX Total Protection Pack applications can be launched and easily managed from the EMC Unisphere and VMware vCenter management interfaces for visibility across the replication environment.
• ESG Lab validated the VNX Total Protection Pack’s ability to provide simultaneous local and remote replication, including application-consistent replicas, for virtual and physical servers.
• The DVR-like rollback to any point in time, combined with journal-based architecture, enables fast and easy recovery with no data loss.
• ESG Lab confirmed the benefits of RM’s task and workflow automation and DPA’s enhanced visibility, trending, and analysis for managing large scale virtual environments.

Issues to Consider

• It should be noted that RecoverPoint volumes at two sites do not have to be the same size. However, if this approach is used, RecoverPoint will not be able to utilize the extra capacity of the larger volume.
• Once the Total Protection Pack solution is in production, best practices should be followed when creating application sets. Initialization of protection may cause unacceptable performance degradation in the production environment until the process completes. Consider creating application sets during off-peak hours.
• Analysis jobs that are shipped as part of Data Protection Advisor cannot be edited. Selecting edit grants access only to a view of the job’s properties. To modify these jobs, a user must create a copy and edit that version.

The Bigger Truth

Traditional disaster recovery solutions often come up short when dealing with both physical and virtual infrastructures. Some require multiple, application-focused point solutions that must be individually managed and are difficult (if not impossible) to integrate with other tools. Others create challenges in virtual server environments as their recovery processes are manual, complex, and hard to execute—the kiss of death for dynamic virtual environments management.

WAN improvements have made replication and data protection much more viable for mid-range organizations. Still, these IT shops are often short-staffed and operate on limited budgets. Complex protection environments with multiple disparate applications and management stations are not conducive to improving their levels of protection.

ESG Lab tested EMC VNX Total Protection Pack components including RecoverPoint/SE, Replication Manager, and Data Protection Advisor in a virtual server environment connected to an EMC VNX SAN and validated the ability to perform CDP and CRR, and to restore quickly from point-in-time snapshots and application-consistent bookmarks. ESG Lab also validated RM’s automation of replica creation and management, and DPA’s visualization, analysis, and gap alerting capabilities. All of this was easily viewed and managed centrally via EMC Unisphere, which provided link and launch capability for the individual applications.

ESG Lab commends EMC for its commitment to simplifying its data protection portfolio for the mid-range customer. While the process is in transition, ultimately the functionality of several current products (such as MirrorView and Celerra Replicator) will be folded into RecoverPoint. Other products such as Replication Manager and Data Protection Advisor are already significantly integrated despite remaining discrete products. At its end state, this consolidation will make deploying, managing, and licensing data protection simpler for users—and selling it simpler for EMC.

Appendix

“I had backups, but nowhere to go to recover after the flood”

THAT is not a DR plan.

Lessons learned: in the event your facility is inaccessible for whatever reason, make sure you have a plan to recover the most critical servers, applications, and data at a recovery center.

This customer, according to my contact, had done everything right with regards to how they were handling their data protection strategy with the exception of having access to a recovery location.  No one had ever thought they wouldn’t have access to the facility.  Sounds very similar to many of the businesses in Chicago who thought they were not at high risk of losing access to their building or facilities.  However, when the Chicago River flooded the underground tunnels due to a construction mishap–that’s exactly what happened.

Thinking beyond backup and more about the what-if scenarios that make up the worst disasters possible will quickly and clearly expose your recovery plan if you do not consider your options for a recovery center.  Better yet, think about how replication can play a part when it comes to disaster recovery and even how it can play a part in maintaining continuity of your business.

There are a number of solutions, both host-based and array-based, supporting replication in a variety of flavors such as leveraging a journal-based replication like CDP type technology, snapshot technology, and even some synchronous replication solutions to support some high availability options.

Don’t ever get caught like this customer was caught.

Play out disaster scenarios and see how successful you will be if your DR plan is executed.  Remember, in this case, failure is a good thing–if you learn from it and adjust your plan.  Failure in the midst of an actual disaster is not.

-chapa signing off

You can read Dave’s other blog entries at Navigating the BaR Scene.

The Challenges

Organizations have enthusiastically adopted server virtualization and continue to expand their implementations beyond test and development into production and mission-critical applications. But consolidating multiple workloads onto individual physical servers does add some risk; a single server failure now impacts more than a single application. Virtualizing tier-1 applications magnifies the importance of enabling quick recovery of virtual machines (VMs). In fact, when ESG recently asked IT professionals to identify where they expect to make the most significant data protection investments, 30% indicated backup of their virtual environment, followed closely by improved application backup.[1] Business managers have little tolerance for application downtime or lack of data availability, and IT managers need assurance that backed-up systems are recoverable.

Virtualization helps to increase operational agility, facilitates higher availability, and reduces equipment and operational costs through consolidation. However, many existing backup solutions have not kept pace with these new environments and often negate the resource reduction benefits realized through virtualization. Traditional backup methods with agents in each VM can often increase the load on hosts and the time required to complete backups. A backup solution that can leverage the core attributes of virtualization—including cost, agility, and availability benefits—will accelerate the virtualization of tier-1 mission-critical application workloads, which must be assured of high availability and full protection. This report examines an innovative VM backup and replication solution that speeds recovery time, streamlines the recovery process, and alleviates the fear of non-viable backups.

The Solution: Veeam Backup & Replication

Veeam Backup & Replication was built specifically for virtual environments to provide fast disk-based backup and recovery of VMs. It offers a 2-in-1 solution of image-based backup and replication, enabling organizations to meet different recovery time and recovery point objectives (RTOs and RPOs) for different VMs. Organizations can address specific budgetary and VM availability requirements by replicating on-site for high availability, replicating off-site for disaster recovery, or backing up using cost-effective storage and techniques. Other features include instant file-level recovery and the ability to quickly provision a backup as an on-demand sandbox. It uses minimal resources and can scale to any size environment through the use of multiple Veeam Backup Servers, which can be centrally managed via Veeam Backup Enterprise Manager.

The latest version of Veeam Backup & Replication includes a Windows service that enables the Veeam Backup Server to act as an NFS server and provide ESX(i) hosts with transparent access to backed-up VM images. This vPower technology redirects changes that take place when a VM is running to a separate location, so backups remain unchanged. In essence, vPower enables a VM to run directly from a compressed and deduplicated backup from any restore point (incremental or full).

The VM does not need to be extracted, updated, and copied to production storage; instead, it is started directly from the backup in the backup repository, either in the production environment or in an isolated Virtual Lab that vPower creates and manages. This enables Veeam Backup & Replication to offer several new capabilities (with multiple patents pending):

• Instant VM Recovery – An entire VM can be restored from backup in minutes. Through Veeam vPower NFS, a vSphere host can connect to the native virtual machine files in the backup; by starting the VM from the backup, a failed VM can be quickly restarted on any host. Complete recovery is done by migrating the VM to production storage non-disruptively using Storage vMotion or by replicating or copying it to the production datastore.
• SureBackup Recovery Verification – Automatic verification of the recoverability of every backup for every VM. Many organizations don’t test backups because it requires additional dedicated resources and time; instead, those responsible for disaster recovery simply hope that if needed, their backups will be recoverable. SureBackup automates verification using available resources in the production or test environment. Using Veeam vPower NFS and the Virtual Lab concept, the VM is run in an isolated environment directly from the disk-based backup; Veeam Backup & Replication starts the VM, boots
  the OS, and confirms that the VM is functioning properly.
• Universal Application-Item Recovery (U-AIR) – Recovery of individual objects from any virtualized application. U-AIR is object-level recovery that needs no special agents, backups, or tools; it provides an elegant and inexpensive solution to problems such as accidental data deletion or corruption. With Veeam
  U-AIR, an application can be started on a recovery VM in a Virtual Lab from the desired restore point by leveraging Veeam vPower NFS. Once powered up, application objects can be copied to the production instance using a Veeam-provided proxy appliance.

Getting Started with Veeam Backup & Replication

ESG Lab found that getting started with Veeam Backup & Replication is an easy process. The application was available on the Veeam Web site for download, and a 30-day evaluation key was provided. Once the application was downloaded, it took about 15 minutes to install (including installing Microsoft SQL Server Express) and get to the login screen of the Veeam Backup Console (see Figure 2).

The Veeam Backup Console is used to configure and manage backup and replication attributes such as schedules, retention, targets, deduplication, and compression. Many restore options are also managed with this console.

As shown in Figure 3, Veeam Backup Enterprise Manager is a Web-based management and reporting application that comes with Veeam Backup & Replication. It allows management of multiple instances of Veeam Backup & Replication from a single console.

Veeam Backup & Replication offers three distinct data transfer modes to provide flexible support for different VMware environments. All modes leverage the vStorage APIs for Data Protection (VADP). Direct SAN access mode retrieves data directly from the SAN and can be used when the Veeam Backup Server has direct SAN access to VMware storage.

The Virtual Appliance mode requires that the Veeam Backup Server be installed in a VM running on an ESX(i) host with access to the datastores of the VM(s) to be backed up. With this mode, data is accessed via the ESX(i) I/O stack.

Lastly, Network mode can be used for VMware environments that use locally attached storage. With Network mode, VM data is accessed over the LAN through the ESX(i) host using VADP and Network Block Device (NBD) protocol. In Network mode, disk data can be transferred over an encrypted SSL connection.

ESG Lab confirmed that Veeam Backup & Replication is a flexible solution that provides tightly integrated data protection for any type of VMware configuration. Its intuitive user interface is designed to walk you effortlessly through the configuration process; wizards help streamline the recovery process and the multiple deployment options enable greater use of existing resources requiring less dedicated backup components than the traditional backup solutions.

Instant VM Recovery

Instant VM Recovery enables recovery of a VM by presenting it to an ESX(i) host directly from a deduplicated and compressed backup. Instant VM Recovery is made possible by exposing the virtual machine files in the backup repository through an NFS mount, as shown in Figure 5. Veeam vPower NFS enables the Windows-based Veeam Backup Server to function as an NFS server. Instant VM Recovery reads data directly from the backup through the NFS mount point. The full recovery runs in the background while the VM is accessible to end users. The NFS mount point also provides temporary write space to hold changes that would otherwise be made to the VMDK while the background recovery is in progress.

Figure 6 shows the actual backups stored on the Veeam backup disk target. This is the data that is read during the Instant VM Recovery process.

The Instant VM Recovery process is managed through the Veeam Restore Wizard, which intuitively walks the user through the restore process. As shown in Figure 7, the wizard is used for Instant VM Recovery as well as many other restore options.

ESG Lab tested Instant VM Recovery of a 200 GB virtual machine compared to a standard full-image recovery using a competing product, as shown in Figure 8. ESG Lab confirmed Veeam Instant VM Recovery was 77 times faster than the standard full-image recovery tested.

The testing compared Veeam Instant VM Recovery against a standard recovery from an image-based backup using a competing backup application. In the standard recovery, the VM was restored from a deduplicated backup stored on a disk-based backup target. The detailed results for the 200 GB VM recoveries shown in Figure 8—as well as the results of a 16 GB VM recovery—are documented in Table 1 and the accompanying bullet points.

What the Numbers Mean

• Veeam Instant VM Recovery was 77 times faster than a standard full-image recovery for a 200 GB VM.
• Veeam Instant VM Recovery took less than two minutes no matter how large the VM.
• The time savings become even greater as the VM to be recovered gets larger.
• Recovery from a traditional file-based backup of a VM would show an even greater performance disparity.

SureBackup Recovery Verification

SureBackup Recovery Verification is the ability to verify a backup by using that backup to power up and test the VM the backup was intended to protect. As with Instant VM Recovery, SureBackup Recovery Verification leverages Veeam vPower NFS to run a VM from a backup, but instead of running it in the production environment, it runs it in a Virtual Lab, as shown in Figure 9. With SureBackup Recovery Verification, the VM is powered up and tested in a Virtual Lab isolated from the production environment.

Veeam provides a proxy appliance with Veeam Backup & Replication to enable access to the Virtual Lab. This proxy appliance acts as a gateway between the production environment and the Virtual Lab. Figure 10 show the actual proxy appliance created as part of the ESG testing process.

ESG Lab validated SureBackup Recovery Verification by recovering an Oracle VM and a Microsoft Active Directory VM in a Virtual Lab environment. Virtual Lab options that enable granular resource configuration and allow application dependencies to be maintained during verification were explored.

Figure 11 shows the memory allocation setup option. This option allows memory to be throttled back for SureBackup Recovery Verification VMs so as not to impact the production environment during verification testing (in the event that production resources are used for Recovery Verification). The memory allocation option is set within the Application Group configuration. With Application Groups, VMs with application dependencies (e.g., Microsoft Exchange and Active Directory) can be grouped and tested together to ensure complete application functionality.

ESG Lab confirmed that SureBackup Recovery Verification provides a flexible and comprehensive recovery test solution that can be run independently from scheduled backup operations.

Universal Application-Item Recovery

Universal Application-Item Recovery (U-AIR) enables recovery of individual objects (e-mail messages, database records, directory objects, etc.) from applications running on VMs. U-AIR uses the same process detailed in the SureBackup Recovery Verification section of this document (see Figure 9). Once the VM is powered up, an application administrator can recover objects from the VM running in the Virtual Lab and copy them to the production VM through the Veeam-provided proxy appliance, which has visibility into both the production and Virtual Lab environments.

Like Instant VM Recovery, the U-AIR process is also wizard-driven. Figure 12 shows the Microsoft Active Directory recovery wizard; a Universal recovery wizard, Microsoft SQL Server recovery wizard, and Microsoft Exchange recovery wizard are also available.

ESG Lab leveraged the Universal recovery wizard to facilitate the recovery of an Oracle table space for an Oracle instance running on a VM. The JOB_HISTORY table was deleted from the production version of the Oracle database. A recovery version of the Oracle VM was then powered up in a Virtual Lab. Oracle SQL Developer was launched to view the point-in-time backup of the database that contained the JOB_HISTORY table space, as shown in Figure 13.

The red box in Figure 13 highlights the JOB_HISTORY table space from the point-in-time view of the Oracle Recovery instance. Note that the JOB_HISTORY table space is missing from the Production Oracle instance (shown at the bottom of the directory tree in the left pane).

Oracle SQL Developer was then used to copy the missing JOB_HISTORY table space from the recovery instance of Oracle back to the production instance of Oracle. As shown in Figure 14, the CREATE TABLE command was successfully used to do a hot insert of the missing table space into the running Oracle production instance.

The Bigger Truth

Server virtualization has been widely implemented by organizations of all sizes as they recognize not only the economic benefits of consolidating workloads, but also greater flexibility, ability to quickly provision new applications, and opportunity to increase data and application availability. In recent ESG research, 93% of respondents reported that they are either currently using server virtualization or are in the evaluation and planning phases. In addition, these survey respondents plan to expand their virtualization implementations over the next 24 months, with 58% planning to have virtualized more than 40% of their servers by that time. They also plan to increase the percentage of virtualized production applications from 39% to 58%.[2]

This demonstrates the continued infiltration of virtualization into tier-1 data and explains why backup of VMs is such a critical topic. Like other IT processes, backup methods were built for the one-application-per-server paradigm. Backup solutions presumed that only a single workload at a time would be interrupted by backing up a server; that is no longer true with server virtualization and as a result, backing up multiple VMs can clog networks and interrupt operations. Trying to make physically-based backup work in the virtual domain has proven difficult and has left IT spending significant effort without much confidence that backups have been done properly and will be recoverable. When you add these challenges to a generally low tolerance for downtime, particularly of tier-1 data, it is clear that a fast, reliable, non-disruptive backup solution purpose-built for VMs is urgently needed.

ESG Lab has verified that Veeam Backup & Replication offers an innovative “virtualization-powered” solution to VM backup that matches virtualization’s key benefits: reducing costs and improving service levels. By creating a way for a Veeam Backup Server to act as an NFS server, the Veeam solution can run a VM directly from a backup on disk, enabling instant recovery of VMs, the ability to automatically and non-disruptively verify backup recoverability, and object-level recovery from any virtualized application. ESG Lab found Veeam Backup & Replication easy and fast to install, with an option for each backup job to be processed via the SAN directly, via the ESX(i) I/O stack, or via the network. ESG Lab tested Instant VM Recovery of VMs in various sizes; no matter what size VM was tested, the Veeam method took just under two minutes. In contrast, standard recovery time grew from approximately 11 minutes to nearly 2.5 hours as the VM size grew from 16 GB to 200 GB. ESG Lab also verified a backup, including starting the VM and booting the OS. Finally, the U-AIR feature was tested and functioned as advertised; a deleted a Microsoft Active Directory user account was recovered and a deleted table space from an Oracle database was quickly restored.

The features we tested demonstrate clearly that Veeam Backup & Replication addresses the key challenges of IT managers and the needs of users. Instant VM Recovery returns users to productivity by enabling VMs to be restarted on any host in minutes, while relieving IT of emergency full restores. The SureBackup Recovery Verification feature relieves IT of worrying that backups are not viable by providing a fast and non-disruptive method of testing every VM backup, and the U-AIR feature provides object-level recovery quickly and inexpensively. In ESG’s opinion, the multiple awards recently garnered by Veeam Backup & Replication are well deserved.

[2] Source: ESG Research Report, The Evolution of Server Virtualization, November 2010.

Data efficiency is just as it sounds–making the data we need more efficient from an access, usability, and manageability perspective.  Why?  So we can drive more value out of it–which I would argue is the entire raison d’être for IT to begin with.

We layer on server virtualization technologies to make our commodity server hardware more valuable –more readily usable by the various data elements we posses.

In the storage world, deduplication of data has been one of the hot efficiency “enablers,” along with thin provisioning, snapshots, virtualization, multi-tenancy, and compression.  Some of them are new, some have been around forever, it seems.  All of them are important.

When it comes to making data more efficient, it’s important to consider “Why” and not just how.  For example, most deduplication solutions are designed for backup–not primary use data environments.  I’m all for making backup more efficient, but that only represents a small fraction of the value potential for IT.  I would argue that primary storage/data use cases represent an order of magnitude (or more) value to an organization potentially.

We have done a decent job, over the last five or so years, at making our infrastructure systems that store/manage data far more efficient.  We can thin provision (virtualize) physical storage assets such that we get the most usage out of them.  We virtualize data (from a presentation perspective) with the use of snapshots.  With multi-tenancy, we can optimize the utilization of our physical assets across multiple constituents.  All that is good, but new technologies are allowing us to take this much further.

Compression has been around for a long time, but this is one technology going through a renaissance period.  Primary data compression (aka Real-Time) is going to change the fundamental efficiency and overall value proposition users will derive–because the closer to the point of data “creation” that you create efficiencies, the more value you get.

Think of it this way:  If you start with 100GB of primary data, over time you will back it up X times, so you’ll end up with 100GB of primary, and 100GBX of backup–or secondary–data.  Backup deduplication players such as Data Domain spend their time on the 100X problem (which is a good problem to spend time on, don’t misunderstand my point).  There are probably a lot of other uses/duplicates of the originating data throughout the organization between creation and backup–like in test/development, data warehouses, etc.

Optimizing data as early as possible is the key. From that point on, all the downstream benefits are magnified.  Less to move, less to manage, less to back up, less to copy, less to replicate, less to store, less to break.  Less is the new more.

I’m no genius but it seems to me that the way one does this most effectively is to leverage all of the tools at hand.  First, compress the data as much as you can.  We’ve proven that you can squish 50% + of the primary footprint out of almost any kind of data–including databases.  Second, deduplicate it.  Whatever is left post compression that you want deduped should be. People don’t want everything deduped, but that’s ok.  Eliminate what you want, and start clean. That gives you the perfect baseline.

From there snap it, thin provision it, copy it (virtually), and do whatever else you would like to do with it, but at least you begin the journey with an optimized footprint–which can only make everything else you do with that data far more efficient.

The trick is to compress in real-time, without suffering the performance penalties that you remember from 20 years ago.  It can be done today.  We do some amazing things at wire speed that weren’t possible even a few years ago.  There will be a lot of money and R&D spent in this area, as it seems clear to me that it simply has to happen.

Moving the value needle of storage/data optimization from the “dead” end of the wire to the “live” end (where data is created) is inevitably the best way to drive value all the way through the entire lifecycle of that data.  Everywhere that data sits, every time that data is manipulated or used, there is value that can be optimized.

Since I am always right eventually (seriously, it’s uncanny), this means there is a lot MORE money to made in this area.  Data Domain was just the beginning.

You can read Steve’s other blog entries at The Bigger Truth.

On November 10, 2010, EMC announced it had acquired privately held, Massachusetts-based Bus-Tech, Inc.  Bus-Tech provides VTL (Virtual Tape Library) solutions for mainframes using open systems disk storage to support mainframe tape for data protection through FICON or ESCON interconnects.  Essentially if you have a mainframe and you want to use an open systems storage solution to back up to, then Bus-Tech was the company you called.  Bus-Tech would front end the storage and talk to the mainframe, present tape devices and VOLSERs to be written to/read from and the open storage solution on the back end would be the receiver.  Suddenly you have VTL backup for the mainframe and if the open systems disk solution on the back end supports deduplication, you have Mainframe VTL with dedupe–not a bad deal.

Why Mainframe Data Protection?

Mainframe is big business still–just ask IBM.  The EMC press release cites IDC estimates of $2.5B between 2010 and 2014 for mainframe tape storage and media revenues.  Bus-Tech has always been the “go to” solution for those open systems data protection solutions providers who at the 11^th hour found themselves with a deal that suddenly included mainframe and if they could provide THAT as part of their solution they would surely be in the running to win.

While to some this may not seem like a strategic acquisition, I believe it represents another incremental step for EMC to provide data protection solutions for the entire enterprise, not just the “open systems enterprise.”  If you have a good memory, you’ll recall EMC once owned CopyCross, a mainframe solution which redirects tape writes/reads to disk–essentially virtual tape for mainframe–which they sold off back in 2002.  They also  acquired the assets of NearTek in 2006, which if I’m not mistaken had a solution that worked with mainframe and even emulated the 3590–but as far as I know EMC didn’t do anything with those assets. Perhaps that was a defensive acquisition at the time.  EMC was a very different company back then and many, myself included, were a bit critical of their buying and selling sprees back in the late 90s – early 00s. From the outside looking in there didn’t seem to be a plan, but you always felt there was something there that never developed.

To quote Yogi Berra, “you can observe a lot by just watching.”

Watching EMC is what I have been doing for the last 14 years of my 25 year career.  EMC was a very different company back in the late 90s and early ‘00s compared to the EMC of today.  While they have always been a powerhouse, EMC seemed to lack the finesse to enter adjacent markets they wanted to expand into.  Well, those days seem to be long gone.  They have finessed their way from BURA to the EMC BRS Division with all of their data protection products under one group and one executive leader, Frank Slootman, former CEO of Data Domain.  When this consolidation occurred, a strategic vision began to emerge–something I hadn’t seen in the past–and I knew only good things could come from that kind of reorganization.

Look at some of the steps EMC BRS has taken over the last several quarters.  Acquired Data Domain; reorganized BURA; consolidated all data protection products into the EMC BRS Division led by Frank Slootman; announced DD Global Deduplication Array in April; announced the DLm960 which is built upon the DD880 paired with Bus-Tech in July; DD Boost for Networker; Networker integration with Avamar and Data Domain; and now mainframe support for VTL.  Some would say that EMC should have bought Bus-Tech a long time ago, but I say I don’t believe the timing would have been right–as I said, EMC was very different in early 2000.  Fortunately for them–no harm, no foul–they have them now.

There is a definite cadence to the activity we are witnessing at the EMC BRS Division.  Frank and team have aptly secured a very nice corner suite with a view of a $2.5B market thanks to the Bus-Tech acquisition that they now can attack with precision.  Only question remaining is what will happen to the open systems solution vendors who partnered with Bus-Tech previously.  One thing is for sure, Frank isn’t done yet.  All I can say is, nice job.

-Chapa signing off

You can read David’s other blog entries at Navigating the BaR Scene.