via Data center outsourcing dilemma: CIOs weigh in on owning vs. renting.

ESG sees a giant opportunity for companies to help match users with technology, choose the ideal desktop virtualization delivery model (BTW it won’t always be VDI), choose the technology vendor(s), and then monitor the results (which include the end-user experience). Companies such as Liquidware Labs have already established a relationship with Dell Services to help address this exact scenario. These companies also see the value of being engaged early in the game and continuing to provide value as users transition and desktop virtualization becomes embraced by end-users and IT operations.

Tactical VDI deployments should not be considered successful desktop virtualization implementations. It takes a clear understanding of the current end-user community, their jobs and responsibilities, application usage, endpoint choices, and the right technology to successfully build desktop virtualization into your company’s desktop strategy.

Read more of Mark’s blog entries at Liquefying IT.

Introduction

Organizations of all sizes are struggling to meet the conflicting challenges associated with information storage growth and complexity juxtaposed with global financial uncertainty. A growing number of IT managers are turning to virtualization and consolidation technologies to meet these challenges.

Background

ESG research indicates that a number of factors are driving IT decision makers toward more cost efficient storage solutions.  As shown in Figure 1, accelerating data growth, storage system costs, and increasing complexity are cited as significant challenges by IT managers.[1]

In addition to the storage challenges listed in Figure 1, ESG research indicates that reduced operational costs and reductions in capital expenditures are also top priorities when making purchasing decisions.[2] Put it all together and it’s clear that IT managers are looking for modular, cost effective storage solutions that are both efficient and scalable.

Coraid EtherDrive SAN

Coraid EtherDrive products combine commodity hardware, lightweight Ethernet networking, and a scale-out virtual storage architecture that can grow from a single appliance to multi-petabyte installations. As seen in Figure 2, Coraid provides both cost/capacity optimized and performance optimized storage appliances supporting SATA, SAS, and SSD drives. Coraid systems support all standard RAID types including RAID 0, 1, 5, 6, and 10.

To make the offering as turnkey and simple to deploy as possible, Coraid also offers HBAs, servers, and replication appliances. All Coraid products communicate using the lightweight AoE (ATA over Ethernet) protocol and standard Ethernet switches, which provides secure storage networking for industry standard x86 servers.

Coraid EtherDrive SAN promises an impressive list of capabilities, including:

• Price-performance: Higher performance than comparable Fibre Channel configurations, at approximately 20% of the cost.
• Massive throughput: More than 1200 MB/sec of throughput per Coraid EtherDrive SRX-Series storage array shelf for large-block sequential workloads.
• Simple scalability: Ease of implementation and management of Coraid EtherDrive storage compared to Fibre Channel and iSCSI.
• Optimized for virtualization: VMware and Hyper-v see Coraid storage as local-attached disks, with no need for switch configuration or multi-pathing software.

ESG Lab’s testing was designed to explore Coraid’s EtherDrive SAN and the AoE protocol, paying special attention to ease of use and management, capacity and performance scalability, and integration and operation in virtualized environments.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of Coraid’s EtherDrive SAN at Coraid’s Redwood Shores, CA headquarters. Testing was designed to demonstrate the ease of installing and configuring an EtherDrive SAN as well as the cost-effective performance and capacity scalability of the platform.

Background: Ethernet SAN

Coraid’s EtherDrive SAN utilizes the AoE protocol to present disk storage to servers across a standard Ethernet network. AoE is an extremely simple method for sharing disk drives through a network. The communication that would normally take place between a motherboard and an IDE disk drive is arranged into data packets and sent across the Ethernet.  As can be seen in Figure 3, AoE is a simpler and more direct protocol than either iSCSI or Fibre Channel. AoE is not built on IP, TCP, or SCSI; packets are addressed to devices using their Ethernet MAC addresses and sent across the network with a minimum of overhead.

Fibre Channel and iSCSI are both based on SCSI, which is a complex protocol designed for a variety of devices (scanners, printers, etc.), in addition to disk drives. Because of this, they incur significant overhead when processing each packet. Both Fibre Channel and iSCSI run SCSI over high level networking protocols on top of a physical network infrastructure, consuming additional overhead and processing compared to AoE, which connects servers and storage directly across the physical Ethernet layer. The typical AoE packet contains just 48 bytes, plus the data payload, enabling “bare metal” performance and native Layer 2 multi-pathing. Fibre Channel and iSCSI first encapsulate the data in the SCSI command set and then wrap SCSI in a transport protocol.

Because they do not run over high level networking protocols like IP, AoE packets (like Fibre Channel) are non-routable. While they can travel across the switches that make up an Ethernet LAN, routers cannot send them to another network and devices outside of the AOE devices local network cannot communicate with them.  This makes AoE packets intrinsically secure. Coraid enables remote access to EtherDrive SANs for administration via AoE tunneling, which is similar to VPN access to a corporate network over the internet.

Getting Started

ESG Lab testing was conducted on a pre-wired, rack-mounted environment consisting of multiple SR2421 and SRX3500 EtherDrive SAN disk shelves. The ESG Lab test bed, as presented in Figure 4, consisted of multiple industry-standard x86 servers with both 20Gbps Coraid HBAs and 1Gbps Ethernet NICs installed. Servers were running VMware ESX server with Red Hat Linux and Windows 2008 installed as guest operating systems as well as physical Linux and Windows 2008 installations. An industry standard Ethernet switch was used for SAN connectivity.[3]

ESG Lab Testing

ESG Lab testing began by powering on an SRX3500 EtherDrive SAN shelf, then logging into a Linux server. Coraid’s cec utility was used to scan for the new chassis using the AoE protocol. In less than a minute, the shelf was visible.

The next step was to name the shelf to make it easier to identify it in a large deployment. Shelf 3 was chosen as the name for these tests. Next, using just three commands, RAID groups were created (Coraid automatically creates one LUN per RAID group), hot spares were assigned, and the LUNs were brought online, as seen in Figure 5.

LUN masking, the means by which servers are given exclusive access to volumes in a SAN environment, is done by Ethernet MAC address using the “mask” command. ESG Lab did not use LUN masking in these tests.

On the Linux server, ls /dev/etherd showed all AoE devices on the network. The storage administrator has nothing else to do—no iSCSI mount, no NFS mount.  The AoE LUNs look like local storage. Next, ESG Lab used mkfs to create and format a file system on each of the AoE LUNs.

Creating LUNs and presenting them for use on the network took less than one minute, while creating the file systems for use by the server took about another minute. In less than two minutes and just four simple commands, ESG Lab configured, provisioned, and was using Coraid EtherDrive storage.

Disruptive Price-Performance

Coraid EtherDrive SAN storage is a modular disk storage system providing massive scale-out capacity and performance with granular, just-in-time scalability to industry standard, open systems environments. The Coraid solution scales by simply installing additional disks and shelves, allowing organizations to start small and scale capacity to petabytes. Using 2 TB SATA drives, users can scale to a petabyte of capacity and 100 GB/sec of raw storage bandwidth in just two racks.

Performance in a storage environment is best measured with the metrics used by the applications organizations actually run.  For an e-mail application, that measurement is the number of users or mailboxes a given system can support.  For a streaming media application, the number of objects served concurrently that can be sustained during peak periods of activity is the measurement that matters most.

ESG Lab Testing

Performance was tested using the IOMETER workload generator via simulated application workloads based on Microsoft Exchange and streaming media services.  Tests were performed to verify a Coraid platform’s ability to deliver predictably scalable performance in a clustered scale-out environment over a standard Ethernet network. The Exchange workload is random in nature and very disk intensive.

Microsoft guidelines recommend a maximum of 1,000 Exchange users per core and less for a server performing multiple roles. This means that a quad-core server, doing nothing but Exchange, should support about 4,000 users.

Microsoft’s IOPS per mailbox guidance for Exchange 2007 is calculated based on the number of messages per mailbox, the user memory profile, in what Outlook mode the mailboxes are operating, and whether any third party mobile devices are used. The baseline value provided by Microsoft is .32 IOPS per mailbox.[4] This means that a quad core Exchange server with 4,000 exchange users will, on average, drive 1,280 IOPS to the Exchange Datastore.  As can be seen in Figure 6, a single SRX3500 LUN was able to support enough transactional IO to support more than 4,500 Exchange users using just 12 SAS drives and scaled linearly to just over 9,000 users with 24 SAS drives.

Next, streaming media performance was examined. This type of traffic is sequential in nature and uses larger block sizes than transactional workloads, putting more of a load on the storage network.

As Figure 7 shows, streaming media performance was excellent, delivering 826 MB/sec from just 6 SSD drives and more than 1,200 MB/sec from 24 SATA drives. Put into perspective, a single shelf was able to drive enough bandwidth to saturate a 10Gbps interface.

The maximum throughput recorded (1200+ MB/sec) was used to calculate the number of streams that could be delivered for a couple of well-known content types including standard definition and high definition broadcast video.  Bit stream rates of 3.75 Mbps for standard definition broadcast video and 80 Mbps for high definition video were used to determine that a single SRX3500 has the bandwidth required to simultaneously stream 120 high definition broadcast videos or 2,560 standard definition broadcast videos as shown in Figure 8.

What the Numbers Mean

• The system showed excellent disk response times for both random and sequential IO. The simulated Exchange disk IO response time was 20ms, while streaming media requests from SATA disk were satisfied in just 1ms.
• Microsoft stresses that, to ensure a positive user experience, the Exchange database LUN requires read and write response times of 20 milliseconds or less so that Exchange can service users’ client software quickly and efficiently. In this context, the SRX 3500’s performance is right on target.
• A single SRX3500 has the raw bandwidth required to service 2,560 concurrent standard definition, broadcast-quality video streams.

Next, ESG lab examined cost of acquisition for a petabyte of storage and SAN connectivity for various technologies. Each storage technology was configured to support the same class and quantities of storage, and SAN connectivity was calculated to support 200 physical servers with redundant connections. Table 2 summarizes the configuration built for each technology.

The cost of storage and SAN connectivity hardware was obtained from a combination of publically available sources, including reseller websites, GSA pricing schedules, and online pricing available directly from vendors.

The cost was calculated for modular dual controller Fibre Channel SAN arrays from three major vendors. The cost of dual controller multi-protocol arrays from two major vendors and the cost of direct attached storage (DAS) solutions from two major vendors were also calculated. The solution with the lowest overall price in each category was used for the comparisons presented in this report.

The bottom line results are summarized in Figure 9. Note that the costs of iSCSI, multi-protocol, and FC SAN solutions are significantly higher than a comparable Coraid EtherDrive SAN system and that the base costs of a Coraid SAN solution are lower even than DAS.

Calculated costs are detailed in Table 3.

What the Numbers Mean

• Coraid EtherDrive SAN has the lowest cost of acquisition, by a wide margin.
• The relative cost of acquisition of alternative technologies ranges from roughly 1.4x for DAS to more than 5x for FC SAN.
• The FC SAN solution is so much more expensive in part due to the cost of acquiring FC SAN connectivity.
• DAS technology has a number of limitations that were not considered in this analysis. First and foremost, it is a dead-end when it comes to server virtualization. SAN attached storage is needed to take full advantage of the benefits of server virtualization. Storage capacity held captive within, or directly attached to, a server can’t be moved non-disruptively to another server for maintenance or better quality of service. SAN attached storage is also needed to achieve valuable disaster recovery capabilities that have recently become available from server virtualization vendors (e.g., VMware Site Recovery Manager). And finally, islands of DAS capacity typically lead to poor storage utilization. Poor storage utilization dramatically increases the overall cost of ownership.
• In addition to CAPEX, ESG Lab believes it is likely that Coraid EtherDrive’s simplified architecture and management would also yield OPEX savings over alternate technologies.

ESG Lab also compared price-performance for the Coraid EtherDrive SAN systems tested to publically available results published for DAS and traditional Fibre Channel SAN systems. Price-performance was determined using a simple calculation of cost in dollars for a specific configuration divided by the number of MB/sec supported by that platform.

Virtualization Optimized

Coraid EtherDrive SAN storage systems integrate with VMware using a simple driver that enables VMware to mount EtherDrive storage arrays as if they were local drives. A VMware administrator can provision and manage virtual machine storage without the need for FC SAN administration or iSCSI client configuration.

ESG Lab Testing

ESG Lab performed virtualization tests on a VMware ESX 4.0 environment with two physical servers and six virtual machines.

First, ESG Lab logged into the vSphere client and clicked on server 192.168.0.214. As seen in Figure 10, the Coraid EtherDrive HBA was visible in the list of storage adapters and volume 10, created using the steps in Figure 5, was visible and ready for use.

The volume was formatted and made available to virtual machines using the Add Storage wizard, shown in Figure 11.

Next, the volume was assigned to a virtual machine using the native VMware Add Hardware wizard. Once the addition was complete, the volume was visible to the Windows operating system on the virtual machine. Figure 12 shows the Windows Disk Administrator tool with the new drive circled in green.

Finally, ESG lab examined availability, testing the synchronous mirroring capability of the Coraid EMX EtherDrive Mirror Appliance as well as the ability to physically move disk drives between chassis without disruption.

The availability test bed, depicted in Figure 13, consisted of three Coraid EtherDrive SR2421 shelves, one EMX Mirror Appliance, and one vSphere server, with one virtual machine running Windows Server 2008.

Two 12-disk RAID5 LUNs were created on two separate shelves and synchronously mirrored through the EMX appliance. Mirroring two volumes using the EMX appliance could not have been simpler. The mkmir command was used to select the source and target volumes to be mirrored. This single command pairs the volumes and starts the synchronization.

Next, the volume was assigned to a Windows server 2008 VM on the vSphere server. Once the volumes were fully synchronized, an IOmeter workload was started on the server, performing a mixed read/write workload against the volume, set to continue indefinitely. Power to the primary SR shelf hosting one side of the mirror was killed. Iometer continued reading and writing to the volume with no errors.

Finally, a single eight-disk RAID5 LUN in a single chassis was used to test the online drive relocation capability of the Coraid architecture. The LUN was assigned to a Windows 2008 VM and an IOmeter workload was started on the server, again performing a mixed read/write workload against the volume, set to continue indefinitely.

Power was killed to the chassis housing the eight-drive RAID 5 LUN. All eight disks were then physically relocated from the primary chassis to a spare chassis. The spare chassis was then renamed to have the same shelf number as the original chassis and the eight-disk LUN was placed online.

Total time for this physical failover was approximately three minutes. After the LUN was placed back online, the IOMeter transactions resumed successfully with no further service interruption. Most, if not all, other architectures, including highly available Fibre Channel and iSCSI SANs, simply cannot take LUNs offline in a VMware environment while machines are running without bringing the server to a crashing halt.

ESG Lab Validation Highlights

• ESG Lab configured, provisioned, and was utilizing Coraid storage in less than two minutes from power on.
• The SRX3500 demonstrated the ability to support thousands of Exchange users using just 12 SAS drives.
• Coraid EtherDrive SAN was able to drive more than 1200MB/sec from a single appliance, enough to stream 2,560 broadcast quality video streams simultaneously.
• Commodity hardware and cost-efficient AoE connectivity enable a cost of acquisition far less than Fibre Channel, iSCSI, and even DAS.
• Coraid proved well-suited to virtualized environments, providing simple to provision SAN storage that looks to a VMware cluster like direct attached disk.
• The EMX Mirroring appliance provided synchronous data protection for volumes across shelves with no disruption to service.
• ESG Lab was able to remove drives that were actively being accessed and move them to a different chassis with only a momentary pause in IO and no errors.

Issues to Consider

• Coraid’s EtherDrive SAN is currently managed through a command line with no GUI. The system is incredibly simple to use and manage, with all necessary functions controlled through a few simple commands and logical, human readable addressing of shelves, disks, and LUNs. Coraid indicated plans to ship an upgraded management system in Q3 2010 with a GUI and REST API support.
• The Coraid EtherDrive SAN solution does not yet offer advanced storage virtualization functionality such as thin provisioning or storage tiering. The driving factor behind these features, reducing the cost of storage, does not necessarily affect Coraid as it does traditional SAN architectures, which typically sell for many multiples of Coraid’s acquisition cost. In addition, these features are increasingly available in software at the hypervisor or file system layer, further obviating the need for them as array-based features.

The Bigger Truth

With storage costs consuming at least 28% of IT budgets,[6] companies are under constant pressure to find ways to reduce costs. Taking a long hard look at reducing capital and operational costs in the storage environment makes sense and so today, more than ever, IT is investing in new technology with a clear focus on reducing storage costs.

The high capacity and performance requirements of scale-out applications including backup to disk, content delivery, server and desktop virtualization, clustered computing, rich media, and un-structured bulk storage are taxing the budgets and infrastructure of IT organizations. Traditional storage network infrastructure can provide the capacity, agility, and performance these applications need, albeit at a high cost of entry and daunting complexity.  Rows of equipment are often needed to provide a petabyte of capacity and gigabytes per second of throughput. Data center managers are being pushed to the limit as administrators spend more and more time managing an ever-expanding SAN infrastructure.

ESG Lab found that the Coraid EtherDrive SAN storage system delivers shockingly simple deployment and management, with complete functionality delivered via a handful of easy to use commands and rock solid Ethernet SAN connectivity delivered via the extremely lightweight AoE protocol. The ease of management, deep scalability, and performance required for bandwidth-intensive scale-out applications are seamlessly extended to VMware environments as well.

ESG Lab testing has confirmed that Coraid’s architecture provides consistent levels of throughput—even during hardware faults.  Sustained throughput in excess of 1,200 MB/sec was observed for large block sequential reads. Cost-efficiency was impressive, with acquisition costs as low as 20% of the costs of traditional SAN attached storage. ESG Lab also verified a very interesting recoverability and resiliency feature, whereby drives can be moved to a spare chassis while an application is running.

With EtherDrive SAN storage, Coraid has dramatically simplified storage for consolidated and virtualized environments while enhancing performance and providing incredible cost efficiency. While the speeds and feeds are impressive, ESG Lab is most impressed by the shocking simplicity of both the AoE protocol and the Coraid architecture, making management of petabytes a reasonable task. If your organization is struggling to keep up with exponential data growth while providing ever higher levels of performance and availability, ESG Lab recommends that you consider Coraid EtherDrive SAN storage as the foundation for your virtualized data center.

Appendix

[2] Source: ESG Research Report, 2010 IT Spending Intentions Survey, January 2010.

[3] Configuration details can be found in the appendix.

[4] http://msexchangeteam.com/archive/2007/01/15/432207.aspx

[5] Source: ESG Research report, The Impact of Server Virtualization on Storage, December 2007.

[6] Source: ESG Research Report, Enterprise Storage Survey, November 2008.

Appropriately called HP Flexible Data Center or Flex DC, as I have seen it referred to, it leverages a modular design of what HP refers to as a “Butterfly,” with a core building in the center and four quadrants of data center space. Scale is achieved by adding quadrants or creating multiple “Butterflies,” essentially creating a campus of data centers. HP claims significant cost savings and a turn time of less than a year. Not too bad when you consider these are real brick and mortar data centers and not containers or pods. Will be interesting to see if this is the approach HP will take with its ES (EDS) data center consolidation effort and how customers respond to this standardized approach. Would be great to hear from any customers that are going down this path.

Read Bob’s other blog entries at Data Center Continuum.

The Cloud

Cloud computing—everyone wants it, but no one knows what it is. Primarily driven by its ability to provide improved operational efficiencies, IT agility, and risk mitigation, cloud computing as a platform choice that can be applied across multiple IT initiatives will be top of mind for IT executives. The interesting dynamic that will evolve over time is the perceived risk it poses to current IT operations. ESG speaks with IT operations professionals who clearly see the benefits of cloud computing, but are fearful of the potential security risk it poses and even its impact on headcount reduction, shifting application responsibility, and automating current IT responsibilities. Expect to see IT “cloud teams” that are tasked to define and identify initial opportunities, and map cloud computing to IT initiatives. Some CIOs are already being compensated for migrations to cloud.

As businesses plan to transition virtualization investments to a cloud computing platform, they will be focused on:

• Consumption and delivery. Cloud computing is a self-service consumption model where workloads are deployed and transparently executed either internally or over the Internet and delivered to businesses that only pay for what they consume. IT must work to develop a deployment strategy or application owners and developers will turn directly to the public cloud and risk security breaches, violation of compliance mandates, and data loss.
• A cloud readiness checklist that includes virtualizing X86 based workloads, automating routine IT tasks, and delivering IT as a service on a highly optimized platform. Companies will also need to include deployment of a consumer interface, a chargeback model, and favorable economic consumption models for application and line of business owners. The virtualization platform will need to be highly secure with guaranteed service levels for all deployed workloads.
• An integrated platform that includes servers, networking, and storage.  The consumption model is changing as intercompany alliances are founded to build shared services layers. Cisco, EMC, and VMware have made significant movement in the market with VBlocks, same for HP with its Converged Infrastructure and IBM Cloudburst, and this trend should continue. Cloud maturity’s will rely on a stable, reliable, and scalable platform that is well integrated and ready to quickly adapt to business requirements. The modularity of the platforms will also be important for independent scaling of compute, network, and storage capacity.
• Third party capacity that can be leveraged on demand for additional compute or storage requirements. By no means is IT about to move entire data centers to Amazon or Google, but they will begin to leverage service providers as an extension of their cloud computing infrastructures. Savvis, Terremark, and Bluelock are all going to market with IaaS (infrastructure-as-a-service) offerings. Expect to see the service provider market heat up with additional IaaS opportunities focused on data protection, test and development, elastic capacity, and hosted desktop virtualization. The biggest barrier to adoption? Security and auditability in multi-tenant environments.
• Applications. The majority of focus to date has been on keeping existing applications up and running on an improved platform. This is where server virtualization has made giant strides. The next area of focus is new application design and development. Application owners and development teams are very interested in designing new applications to take advantage of HTML 5 as well as Java and .NET platforms that deliver design and deployment choices far superior to those that exist today. VMware has jumped in the game with the acquisition of SpringSource, Microsoft continues to evolve AZURE, and Amazon and Google are rounding out their services.

Cloud computing is an excellent way to capture the attention of IT professionals, but technology vendors, resellers, and service providers have to be armed with clear strategic roadmaps outlining where they are today and how to ultimately deliver IT as a service. A new skill set and a new language will need to be established for vendors and service providers to learn, become efficient at communicating, and quickly demonstrate the value cloud computing.

The Bigger Truth

The buildup around cloud today is primarily being driven by large technology vendors sharing their strategic directions regarding the future of IT. As vendors and the media build hype around the cloud, there will certainly be benefits to IT, but we may not even end up calling it “cloud.” It doesn’t matter. What does matter is that IT will begin consuming a more modular infrastructure that is likely to be delivered pre-integrated and the consumption of IT will be greatly simplified and economically favorable to the business. Server virtualization will lead IT consolidation efforts, but it won’t stretch much beyond helping with existing x86 workloads. IT will still have to maintain mainframe, UNIX, and other proprietary platforms.

IT operations and application owners will view cloud differently. IT operations will look at cloud as the potential to provide previously unavailable capacity; application owners and developers will look at the cloud as a development and potential deployment platform. Both paths lead to success if managed much in the same way we have always managed IT: by applying the required security policies, maintaining application availability, and driving application productivity. Cost, risk, and time to delivery will always be factors and IT will need to manage these variables to make informed decisions that drive value to the business. No one is suggesting going in on Monday morning and moving an entire data center to the cloud, but the opportunity to move pieces, certain workloads, and even desktop is very real and available.

The federal government is now catching on as well and recently (June 10, 2010) President Obama wrote a memo titled “Disposing of Unneeded Federal Real Estate” which called for a moratorium on new data centers and urged that consolidation plans commence almost immediately. The memo reads,

In addition, in order to address the growth of data centers across the Federal Government, agencies shall immediately adopt a policy against expanding data centers beyond current levels, and shall develop plans to consolidate and significantly reduce data centers within 5 years. Agencies shall submit their plans to OMB for review by August 30, 2010.

The complete memo can be found here.

The Obama administration has also been a vocal supporter of “cloud” computing as well. While many of the large, private enterprises that have embarked on consolidation strategies have retained control, others are looking to outsource their data center or leverage software-as-a-service (SaaS) models. Indeed, the EDS consolidation effort is specifically targeted at enabling more efficient and flexible computing options and ESG research indicates that those organizations that are focused on consolidating data centers are almost twice as likely to be exploring outsourced IT or SaaS models. This should be good news for those selling to the federal market and services organizations with substantial IT outsource businesses.

Read more of Bob’s blog entries at Data Center Continuum.

Overview

Computer hardware used to be an expensive resource that had to be centralized and carefully managed. Historically, there was only one way to meet the growing demand for more resources and performance, and that was to scale up; the old mainframe, network switch, storage controller, or UNIX server got traded in for a newer, bigger, faster one.

That has all changed. Hardware is now a relatively low cost commodity, which has created a growing trend that moves organizations away from expensive, scale-up architectures toward commodity, scale-out architectures.

Scale-out architectures drive network flow in an edge-to-edge (from the server farms to the storage farms), rather than edge-to-core pattern (between enterprise platforms), fashion. This has a profound impact on network architectures, where the edge carries most of the traffic and the core manages lower volume transitive traffic volumes.

Transformational Directions

Figure 1 shows the main directional movements in networking and storage technologies.

• Structure: Market share has moved away from block to file structured storage due to its flexibility and ease of use. ESG predicts future storage solutions will be based on object stores with drivers that provide block and file presentation options for legacy systems.
• SAN: SAN ports are moving strongly in the direction iSCSI due to its lower TCO and management simplicity. Legacy Fibre Channel users will deploy FCoE as a bridging technology driven by the benefits of network convergence.
• Architecture: Scale-out has been a market leading approach for computing for many years, with significant activity in scale-out storage from HP, IBM, Isilon, and others. Most networks are still designed in a three-tier scale-up architecture, which is becoming increasingly expensive and inflexible, which in turn explains the trend toward two-tier networks.  With the new phenomenon of virtual machine “motion” occurring more regularly, network technologies are now required to adapt to these dynamic change conditions.  BLADE Network Technologies’s VMready is one example of this coming to fruition.
• Network: As intelligence and functionality, driven by the deployment of scale-out storage and computers, move to the edge and away from the core, networks need to adopt a more pragmatic edge-to-edge architecture.
• Protocols: The most prevalent LAN protocol will continue to be IP over Ethernet. The SAN will converge, initially into FCoE protocols to support legacy Fibre Channel deployments. Economics will drive a full migration toward IP protocols on the converged network.

Scale-out Architectures

The unbelievable levels of infrastructure performance today are still not enough for the largest Internet-scale tasks such as hosting Twitter, Facebook, or LinkedIn or managing search indexes at Yahoo or Google. The problem is that scaling up infrastructure just doesn’t meet the capacity demands of these enterprises, so they don’t do it anymore. Instead, they scale out at every level: compute, storage, network, application architecture, and even down to the database.

Soon, almost all devices will be connected to a network. Mass consumer trends in mobile telephony and personal digital assistants (PDA)—combined with integration of video, still cameras, GPS systems, and smart analytics—offer rich opportunities for adding value to all technology areas from social networking to national security. Trillions of electrically connected devices will be added to the network; electronic direction signs, traffic lights, parking meters, toll booths, surveillance cameras, and number plate recognition systems are already revolutionizing traffic management in cities such as Seoul, South Korea.

This Internet-scale problem is becoming more common; more organizations are processing huge volumes of data in near real-time. Developers are beginning to create scale-out solutions and applications. They are adopting a MapReduce style approach to coding where a set of master processes splits the problem into a number of smaller parts and then farms them out to a large number of processes that derive the answer. The master processes then combine the answers to deliver a single consolidated output. For the largest scale computational problems, this is often the only way to get to an answer in a meaningful time frame.

A Scale-out Approach to the Problem

Keeping Data and Computing Independent, but Close

These Internet-scale solutions need to separate compute and storage nodes to enable massive parallelism while maintaining high performance, low latency connectivity between active processes and the persistent data sets they need to manipulate. Scale-out or distributed file systems (such as Apache Hadoop Distributed File System (HDFS) and Google File System (GFS)) provide support for rack awareness—ensuring that the compute element is in the same locale as the storage element. Edge-to-core networking has no place here as almost all traffic flows from edge to edge.

Network Convergence

For some time, the networks (LANs) carrying data flows from server to server and from server to consumer have been separated from the storage area networks (SANs) carrying traffic from server to data storage. The reasons for this are fairly straightforward: LAN flows expect (and are designed to manage) congestion and variable latency as well as occasional data loss, while SAN flows do not. SAN latency and data loss typically cause poor application performance and data consistency problems.

High capacity (10GbE, 40GbE, and eventually 100GbE) lossless Converged Enhanced Ethernet (CEE), also known as Data Center Bridging (DCB), changes the rules for storage traffic. CEE can deliver data storage flows reliably and without latency or loss, eliminating the need to maintain a separate storage network. Firms like BLADE Network Technologies are well positioned to support these demands for a combination of network convergence and edge-to-edge connectivity with an open standards approach that enables interworking with Cisco, Brocade, and other vendors using their BLADE Unified Fabric Architecture (UFA).

The Economics

There are very strong economic drivers for converged SAN and LAN connectivity at the edge. The cost per GB of data transferred is twice as much for 8GB Fibre Channel as for 10GB Converged Enhanced Ethernet. So, by converging the networks into CEE, total capital spending is reduced by as much as two-thirds and cabling cost and complexity are reduced by as much as half. Converged networks enable wire-once-and-forget approaches to the data center.

What is the Right Protocol for Storage Networks?

Use cases are demanding more granular control of data. For example, does the data need to be geographically replicated, need to have enhanced protection, or need to deliver higher performance? Block structured data offers limited capability in this area; NAS and object stores provide the promise of rich metadata and significantly more granular control. Scale-out architectures overcome the perceived performance penalty except for the most extreme RDBMS requirements.

As data moves from block to file storage and onto object structures with rich metadata, block protocols like FCoE or iSCSI are replaced by NAS protocols such as NFS and CIFS and then by object protocols such as HTTP or URI. All, with the exception of FCoE (which is still rooted in the legacy of Fibre Channel) are IP-based and can be routed, isolated, and managed in the same way as normal LAN traffic.

What Does a Scale-out Network Look Like?

In a scale-out network, racks are filled with high-density compute nodes, each with a large memory footprint and large numbers of cores and sockets all interconnected via 10G CEE top of rack switches. LAN and SAN protocols are converged, with most systems choosing IP-based storage connectivity (iSCSI, NAS, HTTP). Storage platforms are placed close by and also interconnected via 10G CEE top of rack switches. Each switch is connected to a pair of end of row switches via 40GbE uplinks.

In these large 1000+ port network domains, most traffic flow is edge-to-edge between the computing elements and the storage elements. Congestion control is managed from within the domain and is visible and manageable at the edge.

Scale-out networking follows the logic that most network traffic in a scale-out world is edge-to-edge, so why bother with a massive capacity, core network? Instead, converge on 10G CEE using top of rack switches supporting iSCSI, NAS, and HTTP protocols to converge the SAN and LAN into a common routable IP system.

These networks are incrementally additive; as new workloads are introduced and new compute and storage racks deployed, top of rack switches are added at the same time. This approach avoids the conventional waterfall investment strategy where all network equipment needs to be installed and wired before the first server is installed. In scale-out networks, the core is much diminished in importance. The differentiator is the adoption of open standards, low power, low latency, and low cost modular switches that can be added incrementally as the compute and storage capacity of the data center grows.

Scale-out Storage

Scale-out storage is also a growth industry with vendors like NetApp, HP (IBRIX), and IBM gaining traction. Object storage is also gaining popularity, with URI or HTTP protocols such as Amazon S3 becoming commonplace. Scale-out storage lends itself to Ethernet connectivity leveraging the lower costs of CEE adapters and switches.

Scale-out Databases

Scale-out databases are now commonly referred to as NOSQL databases that go back in time to pre-relational designs that do not provide atomicity, consistency, isolation, durability (ACID), or consistency guarantees but allow sharing to split the data sets over multiple nodes to improve parallelism and scaling of the overall system.

Analysis

The computer industry changes only gradually, building on the legacy of past. We are guided to the future by gradual changes in thinking and behavior driven by global business, governmental, and economic trends. Scale-out architectures will be a significant part of our future.

• Scale-out is becoming a common approach, is gathering momentum in storage, and is beginning to catch on in networking.
• There are a number of highly visible use cases for analyzing the data thrown up by myriad network-connected devices in near real time.
• The number of organizations needing massive scale analytic support is expanding rapidly.
• Scale-up architectures are unable to deliver the raw power and throughput needed.
• Application architectures are changing to enable use of scale-out approaches. Progress is being made in the open source world, with vendors following closely behind.
• The cost differential between commodity components and specialist scale-up components is growing particularly in the networking space driving procurement choices.

Is Scale-up Finished?

Not all applications will lend themselves to a scale-out architecture—either because they are strongly transactional and require referential integrity between transactions or need to be run sequentially, such as mainframe batch processing. Some of these applications and use cases could benefit from an alternative approach that may well fit with a scale-out, or hybrid, architecture.

Certain high performance computing (HPC) use cases demand ultra high scale-up performance. Conventional relational databases with atomicity, consistency, isolation, and durability (ACID) guarantees can be partially scaled-out via clustering; in the end, they demand high clock speeds, high performance networking, and high performance disk subsystems.  In these limited cases, a scale-out architecture may not be the right answer.

The Bigger Truth

Organizations are demanding previously impossible levels of performance and throughput to support large-scale analytics. Vendors are leveraging the economics of commodity hardware combined with smart software to deliver scale-out products that not only deliver massive aggregate throughput but also can provide cost effective and flexible service to ordinary workloads.

Buyers love the economics and flexibility of scale-out computing where increases in demand do not lead to a complete re-architecture of IT infrastructure. Scale-out offers the promise of public cloud economics in the private cloud.

The last dominions of scale-up thinking remain in networking and database designs. NOSQL approaches are challenging the very largest scale database architectures while top of rack edge-to-edge network designs such as BLADE’s UFA are providing practical, cost effective scale-out network solutions in the data center.

via SAS and SATA, solid-state storage lower data center power consumption.

Overview

In 2010, 52% of mid-size (i.e., less than 1,000 employees) and enterprise (i.e., more than 1,000 employees) organizations will increase IT spending.  This is a significant increase from 2009, when only 43% of organizations planned to increase IT spending during the global recession.  While these organizations have a long list of priorities, ESG’s research indicates that many will increase their use of server virtualization, upgrade their network infrastructure, continue to consolidate data centers, invest in BC/DR programs, and implement desktop virtualization (see Figure 1).[1]

While these initiatives may seem independent, ESG sees a pattern. Many organizations are:

• Centralizing services and applications. This long-standing trend is only gaining momentum. Business applications, collaboration systems, and IT services continue to migrate from secondary data centers and remote offices to large central data centers.  Why?  Many organizations have realized tremendous cost savings, uptime improvements, greater hardware optimization, more automated administration, and reduced power/cooling/space requirements.
• Doubling down on virtualization technology. IT centralization and data center consolidation efforts continue to accelerate as server and desktop technologies from vendors such as Citrix, Microsoft, Oracle, and VMware mature.  Many organizations now run production and mission-critical applications as virtual machines, physical appliances are transforming to virtual appliances, and virtual desktop initiatives are expanding beyond the test and pilot stage; according to ESG research, 21% of large organizations have already deployed a VDI solution. 
• Relying on networks to weave everything together.  While network connectivity isn’t new, enterprises have never been more reliant of high performance/low latency connections for several reasons.  N-tier applications now rely on a tremendous amount of server-to-server communications across physical systems and virtual machines.  Remote office workers who used to connect to local servers now depend upon central services and WAN links.  The growing numbers of mobile workers face a more difficult situation: they have to depend upon VPN connections across public networks.

In total, ESG’s data indicates a steady “race to the center” with a strong dependency on data centers, server virtualization, and networking.  Organizations clearly believe that this can help them get their arms around spiraling IT cost and complexity.  These investments will reverberate on both sides of the WAN, in data centers and branch offices.

The Potential Hazards of Ubiquitous Centralization

When organizations pursue simultaneous data center consolidation, application/services centralization, and virtualization initiatives, they tend to buy a lot of IT equipment and services.  Centralizing and virtualizing applications and services requires new servers and storage systems.  All of these new servers need to be connected together through Ethernet switches.  Finally, providing central services means more bandwidth between data centers and remote users.

Certainly, IT investments provide a strong foundation, but technology upgrades are not always enough.  In spite of new equipment and services, many “data center transformation” projects can lead to application performance problems that can make the user and IT experiences into nightmares.  When productivity plummets and the help desk is flooded with angry calls from the east coast branch, the Boston-based line-of-business manager won’t care a bit about all the money the company is saving on power and cooling costs.

With investments in new data center equipment and increased bandwidth, what causes these performance problems?  They are often related to a lack of:

• Understanding on IT dependencies.  In the rush to consolidate, data center operations staff find it difficult to determine which applications run on specific servers or to map out server to server connectivity dependencies.  Knowing these dependencies is complicated by server virtualization, where workloads move from server to server.  Typically, IT relies on spreadsheets, a slew of configuration databases, or direct user feedback for this information.  Manual tasks and haphazard efforts are a good start, but not the depth of information and visibility necessary for effective enterprise planning and documentation.
• Business impact analysis. Servers and applications are simply tools for employee productivity—and all applications and services are not created equal.  Without analysis, IT can mistakenly make changes to IT services that impact user productivity and employee morale.  When the CRM application consolidation project is scheduled just before the qualification deadline for a sales contest, IT heads will roll.
• Baseline metrics about network traffic and application performance. Oftentimes, IT services and business applications are consolidated sequentially.  This can help avoid problems with each consolidation project, but it ignores the impact that new consolidation project can have on aggregate network traffic.  To get this right, IT needs visibility into network traffic—from source to destination—at all times in order to understand the impact of each change on everything else.

These issues can affect all applications, IT services, and users, but the impact is usually compounded by the WAN.  Why?  WAN connections support a variety of critical and discretionary traffic flows—through a much smaller pipe than the corporate LAN. If corporate employees suffer when it takes 45 seconds to open a file, branch office workers will be rendered completely unproductive.

The Riverbed Alternative

Does IT simply neglect these issues?  No.  Most IT managers understand the importance of IT dependencies, business impact analysis, and network traffic/application performance baselining, but they depend upon either a multitude of management tools or manual processes to adequately address these requirements.

To overcome these obstacles, IT managers need a baseline of IT knowledge and visibility tools that provide real-time granular data about the impact that any IT change has on everything else.  With this knowledge in hand, IT also needs the ability to identify and fix problems immediately—especially as they relate to the WAN.

Many technology solutions are available, but a number are sold as point tools without the integration needed to respond rapidly and reduce costs.  Riverbed Technology provides a suite of products that may be one of the only exceptions to this rule.  With its 2009 acquisition of Mazu Networks, Riverbed added advanced network and application performance and visibility to its portfolio of leading WAN optimization products.  With its Cascade, Steelhead appliances, and Riverbed Services Platform (RSP), Riverbed provides the tools for a consolidation cycle with three phases:

• Discover. Before throwing money at IT products and services, large organizations should discover all assets and map out connectivity, normal behavior, and application performance.  These are the exact capabilities provided by Cascade.  Armed with this information, CIOs can improve consolidation project planning through automation, reducing the costs associated with manual processes or the risks related to the unanticipated problems described above.
• Enable. By mapping out typical application and network performance, IT can choose an appropriate timeframe for consolidation projects without unnecessary business interruption.  This is true of the consolidation project as well as additional phases like bringing a new branch online.  Network and performance administrators can monitor the impact of each step of consolidation projects and adjust project plans accordingly.  When IT staff is asked to evaluate a disaster recovery plan, they can assess traffic on the WAN links to ensure that there is enough bandwidth for backup operations.
• Consolidate. These discovery, planning, and enablement processes can help accelerate overall IT consolidation because changes are scheduled and monitored continuously.  In addition, problems that do arise are quickly discovered and addressed.  When e-mail consolidation impacts the performance of latency-sensitive IP-telephony, IT can pinpoint the problem and take action immediately.  Riverbed can help in two ways here:  Steelhead appliances can help ensure adequate performance when branch office applications are moved to central data centers and the Riverbed Services Platform (RSP) can help consolidate remaining branch office services onto a single platform.

In this way, Cascade anchors IT consolidation and data center transformation with rich contextual information about application performance and network utilization.  Once problems are discovered, however, it is also important to fix them as soon as possible.  This is where Steelhead appliances come into play; minor LAN problems can truly interrupt business processes for remote workers and mobile users connecting over WAN links or the public Internet.  When Cascade discovers a performance problem or network anomaly, Steelhead appliances can be fine-tuned to accelerate business-critical traffic and throttle back less important packets.

The Bigger Truth

ESG’s research data on IT spending is good news.  Large organizations realize the benefits of data center transformation and IT consolidation—and are investing accordingly.  Unfortunately, IT consolidation benefits don’t come for free; without the right knowledge, planning, implementation, and visibility, consolidation can cause more problems than it solves, especially for remote and mobile users.

To overcome these problems, IT managers can’t rely on guesswork or an assortment of point tools.  It is important to proceed with knowledge, data, metrics, and visibility throughout EVERY IT consolidation project.  This is exactly what Riverbed’s Cascade advanced network and application performance and visibility provides.  With Cascade, IT can baseline the existing environment in order to streamline projects, lower costs, and minimize the risk associated with making a consolidation faux pas.  Cascade information can also work hand-in-hand with Steelhead appliances to fine-tune WAN performance, accelerate critical traffic, and throttle back everything else.  As such, CIOs would be prudent to put Riverbed, with its Cascade, Steelhead appliances, and RSP solutions, on their list of potential IT consolidation and WAN optimization partners.

The survey was designed to answer the following questions:

• How many data centers do organizations operate today? How do these numbers vary by company size and industry?
• To what extent are organizations planning to reduce/consolidate their overall number of data centers?
• How many organizations currently have new data center construction projects underway?
• Where do data center consolidation and new data center construction initiatives rank on organizations’ current list of IT priorities?
• How do data center consolidation and construction plans vary by company size class and industry?
• How do the number of existing data centers and spending mode impact data center consolidation?
• What role will outsourcing and software-as-a-service (SaaS) models play in data center consolidation plans?
• What is the impact of consolidation efforts and corporate risk tolerance on new data center construction?
• What are the greatest factors influencing new data center construction?

Survey participants represented a wide range of industries including manufacturing, financial services, communications and media, health care, retail, government, and business services.

For more information on the contents and findings of this 24-page report, please download the executive summary below.

ESG Research Report Data Center Consolidation and Construction Trends Executive Summary