via EMC releases upgraded Clariion, Celerra storage units – Page 1 – Enterprise Infrastructure.

Things have changed, however. DNS security has been called to question many times through cache poisoning attacks and the infamous Kaminsky vulnerability. To address these security weaknesses, DNSSEC efforts are underway. The DNS root servers have all been signed, as have the .gov and .edu Top Level Domains (TLDs). The other TLDs will be signed soon. These efforts will eventually establish a root/chain of trust for all sub-level DNS servers.

Yes, DNSSEC will take years before it is fully deployed, but the foundation is nearly in place. The U.S. federal government is leading the transition to DNSSEC, which means that federal system integrators and leading technology vendors will follow suit. In terms of the market at large, ESG believes that the transition to DNSSEC means:

1. Lots of DNS server turnover. Most DNS server implementations are pretty basic, anchored by either Windows DNS or BIND. These will need to be upgraded or replaced. Windows 2008 DNS and BIND 9.0 support DNSSEC.
2. The DNSSEC appliance market should grow. Many organizations understand the value of DNS appliances, but never had a compelling reason to swap out software-based DNS for an appliance alternative. DNSSEC creates this opportunity. Good news for appliance vendors like Bluecat, BT, and Infoblox.
3. Managed DNSSEC services become a viable alternative. DNSSEC may improve security, but it also demands certificate and key management, adding cryptographic complexity to DNS operations. Rather than learn new skills, many organizations will decide to punt and outsource DNSSEC to cloud providers like Neustar and Verisign.

This migration will mostly fly under the radar, but it will be a lucrative opportunity for smart vendors with the right products and services at the right time.

Read more of Jon’s blog entries at Insecure About Security.

Overview

Merely uttering the phrase “data migration” can strike fear into the hearts of IT administrators. Migrations are often lengthy and painful processes; they are also a necessary evil that supports business initiatives like outsourcing, data center moves, and cloud computing.

ESG recently conducted a survey of 515 senior IT professionals concerning their organizations’ data center plans and priorities for 2010 and beyond. [1] Among other findings, ESG uncovered that one in three companies are aggressively consolidating their data centers: more than one-third (35%) of respondents currently have data center reduction or consolidation activities underway. Of that group, enterprise-class organizations are more likely than midmarket firms to either be in the process of reducing or consolidating data center space.  In fact, data center consolidation ranks as the third most important overall IT initiative over the next 12-18 months for enterprise-class firms.

The same survey found that 20% of respondents plan to increase use of IT outsourcing as a cost containment strategy and 17% plan to increase use of cloud computing services as an alternative to in-house applications and/or infrastructure. Whether it’s data center moves, outsourcing, or subscribing to cloud services, all of these initiatives often have one thing in common: they depend on a successful migration to get new operations up and running.

A number of issues must be taken into consideration when planning a large scale data migration:

• Time. How long the migration takes will vary extensively based on the approach. With some methods, users also need to consider whether any additional time will be required to roll forward changes that happen between when a data copy happens and when it is up and running at the new site.
• Data availability. Some methods can require extensive downtime, during which data cannot be accessed.
• The cost of bandwidth. Depending on the amount of data and distance to the new site, implementations can become prohibitively expensive: the greater the distance or larger the data set, the more costs incurred.
• Portability between service providers. This concern is especially valid when leveraging cloud services, which may require programming to proprietary APIs to access data.
• Data reduction/transport optimization. Users need to understand the available options that reduce the overall amount of data that needs to be migrated, which can reduce both costs (media or bandwidth) and time to complete the migration.
• Security. Each method comes with associated risks; users must be able to ensure data is secure while in transit.

Options for Large Scale Data Migrations

Options for moving a large amount of data en masse from one location to another are very limited.

Backup and Restore

Data is backed up from one system and then restored on another. The process takes applications down, is typically slow, and consumes significant server resources.  With frequent delays, it is not at all unusual for a user to get only some percentage of the way through (and whether that is 1% or 99% makes no difference), exceed the operational window, and then have to start all over again. Just about the only good news is that backup operations sometimes have their own network (although that can be an expensive proposition), so network clogging can be avoided.

The biggest challenge with tape backup is that a user doesn’t really know whether a job succeeded or failed until it is complete. One bad tape renders a backup useless—the process needs to be restarted.  Restores from tape can also be lengthy and time consuming processes.  Data reduction is available through deduplication software, which is becoming ubiquitous in the backup process, but extra time may need to be added to re-inflate the data upon restore if required.  Security can be attained through encryption.

Copy to and Ship a Disk Subsystem

Users do have the option of loading the data onto a subsystem and shipping it to the new or remote site. The process is similar to regular backup in many respects, but it is also faster than backup and restore since disk-to-disk copy is faster than disk-to-tape and once the system is at the new site, it just needs to be powered up—there is no need to wait for tape restores at the remote or new site.  Encryption technology is available to secure the data.

This process is also risky: disks could be damaged or lost, which could cause data loss or corruption that may not be immediately evident.  Depending on distance, shipping time needs to be factored in, as well as data scrubbing and integrity checking, which could take a long time depending on the size of the data sets involved.

Network Copy

In this scenario, the server or storage array is called in to copy data across the network. Depending on the available network bandwidth, the amount of data, and the distance, this can be an expensive and lengthy process.  Data is sometimes inaccessible during the copy or access is slowed significantly due to resource contention. Since migration can consume significant network resources and other applications can be affected as well, the transfer should happen during off-peak hours. The upside is that users don’t have to worry about tapes or disks getting lost or damaged and risk of data loss is virtually nonexistent as writes are acknowledged at the remote site.  The main downsides of network copy are time, security, and bandwidth costs, all of which can be mitigated by leveraging WAN optimization technology.

Enter WAN Optimization

WAN optimization technologies let companies do more with less, enabling them to transfer more data over a smaller network connection. Such efficiency could deliver significant improvements in both existing and new disaster recovery environments. WAN optimization leverages a host of technologies like protocol optimization, encryption, deduplication, and compression, which combine to provide consistent and secure high performance connectivity over the WAN.  Additionally, policy-based quality of service (QoS) and load balancing guarantee that specific applications can be given priority for available bandwidth, minimizing the impact of the migration on other network traffic.

Leveraging WAN optimization for data migration is:

• Faster and more efficient. Data reduction technologies like compression and deduplication mean less data needs to be transferred, allowing the process to finish sooner. More efficient data transfer also means more data can be moved in smaller windows.
• Safer than shipping a disk subsystem or tapes. There is no physical piece of equipment containing data being shipped from place to place.
• Cost effective. Users can push more data over a smaller network connection, minimizing network costs.  WAN optimization solutions can improve network traffic management and efficiency, which could reduce total network consumption needs.
• Secure. WAN optimization technologies provide secure communications by encrypting data in flight and authenticating users accessing applications and data. It can also help reduce the load on back-end servers by offloading processor-intensive security operations for SSL/TLS and network encryption.
• Less disruptive than backup or network copy. Leveraging technologies like load balancing and policy-based QoS, critical applications can receive priority bandwidth allocation to avoid disruption by the migration during the day—the migration can then take priority during slower network periods, optimizing available bandwidth to meet business needs. Additionally, protocol optimization is used to reduce the impact of chatty protocols, streamlining communication between sites.

WAN optimization technologies with multi-Gbps throughput offering deduplication, compression, and high availability configurations can create efficiencies that enable the use of standard service provider networks.  This could potentially save millions in capital and operating costs that would have been spent relocating a data center due to insufficient network connectivity. At the very least, there is the potential to save tens to hundreds of thousands of dollars in reduced network costs between sites.

The Bigger Truth

Migrations are migrations. The driving force behind a migration is ultimately not important: the challenges associated with migrating data to the cloud are not all that different from the challenges many enterprises have already faced with data center moves and consolidation. There are a number of ways to tackle the challenge.  Trade-offs will need to be made regarding cost, speed, and risk.  Shipping equipment around can be risky from a security standpoint and restoring from tape can be a long and laborious process that does not guarantee success. Moving data across the network can take some risk out of the equation and has a much better success rate, but it can also be time consuming and expensive without technology to give it a boost.  WAN optimization provides that boost.  It is already a proven technology and has been used broadly for data center consolidation, remote office application acceleration, and backup/disaster recovery initiatives.  WAN optimization technology can help reduce the pain of migrations so they are completed faster and cost less.  Using WAN optimization, users can finish migrations faster and return to productivity sooner.

Introduction

Organizations of all sizes are struggling to meet the conflicting challenges associated with information storage growth and complexity juxtaposed with global financial uncertainty. A growing number of IT managers are turning to virtualization and consolidation technologies to meet these challenges.

Background

ESG research indicates that a number of factors are driving IT decision makers toward more cost efficient storage solutions.  As shown in Figure 1, accelerating data growth, storage system costs, and increasing complexity are cited as significant challenges by IT managers.[1]

In addition to the storage challenges listed in Figure 1, ESG research indicates that reduced operational costs and reductions in capital expenditures are also top priorities when making purchasing decisions.[2] Put it all together and it’s clear that IT managers are looking for modular, cost effective storage solutions that are both efficient and scalable.

Coraid EtherDrive SAN

Coraid EtherDrive products combine commodity hardware, lightweight Ethernet networking, and a scale-out virtual storage architecture that can grow from a single appliance to multi-petabyte installations. As seen in Figure 2, Coraid provides both cost/capacity optimized and performance optimized storage appliances supporting SATA, SAS, and SSD drives. Coraid systems support all standard RAID types including RAID 0, 1, 5, 6, and 10.

To make the offering as turnkey and simple to deploy as possible, Coraid also offers HBAs, servers, and replication appliances. All Coraid products communicate using the lightweight AoE (ATA over Ethernet) protocol and standard Ethernet switches, which provides secure storage networking for industry standard x86 servers.

Coraid EtherDrive SAN promises an impressive list of capabilities, including:

• Price-performance: Higher performance than comparable Fibre Channel configurations, at approximately 20% of the cost.
• Massive throughput: More than 1200 MB/sec of throughput per Coraid EtherDrive SRX-Series storage array shelf for large-block sequential workloads.
• Simple scalability: Ease of implementation and management of Coraid EtherDrive storage compared to Fibre Channel and iSCSI.
• Optimized for virtualization: VMware and Hyper-v see Coraid storage as local-attached disks, with no need for switch configuration or multi-pathing software.

ESG Lab’s testing was designed to explore Coraid’s EtherDrive SAN and the AoE protocol, paying special attention to ease of use and management, capacity and performance scalability, and integration and operation in virtualized environments.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of Coraid’s EtherDrive SAN at Coraid’s Redwood Shores, CA headquarters. Testing was designed to demonstrate the ease of installing and configuring an EtherDrive SAN as well as the cost-effective performance and capacity scalability of the platform.

Background: Ethernet SAN

Coraid’s EtherDrive SAN utilizes the AoE protocol to present disk storage to servers across a standard Ethernet network. AoE is an extremely simple method for sharing disk drives through a network. The communication that would normally take place between a motherboard and an IDE disk drive is arranged into data packets and sent across the Ethernet.  As can be seen in Figure 3, AoE is a simpler and more direct protocol than either iSCSI or Fibre Channel. AoE is not built on IP, TCP, or SCSI; packets are addressed to devices using their Ethernet MAC addresses and sent across the network with a minimum of overhead.

Fibre Channel and iSCSI are both based on SCSI, which is a complex protocol designed for a variety of devices (scanners, printers, etc.), in addition to disk drives. Because of this, they incur significant overhead when processing each packet. Both Fibre Channel and iSCSI run SCSI over high level networking protocols on top of a physical network infrastructure, consuming additional overhead and processing compared to AoE, which connects servers and storage directly across the physical Ethernet layer. The typical AoE packet contains just 48 bytes, plus the data payload, enabling “bare metal” performance and native Layer 2 multi-pathing. Fibre Channel and iSCSI first encapsulate the data in the SCSI command set and then wrap SCSI in a transport protocol.

Because they do not run over high level networking protocols like IP, AoE packets (like Fibre Channel) are non-routable. While they can travel across the switches that make up an Ethernet LAN, routers cannot send them to another network and devices outside of the AOE devices local network cannot communicate with them.  This makes AoE packets intrinsically secure. Coraid enables remote access to EtherDrive SANs for administration via AoE tunneling, which is similar to VPN access to a corporate network over the internet.

Getting Started

ESG Lab testing was conducted on a pre-wired, rack-mounted environment consisting of multiple SR2421 and SRX3500 EtherDrive SAN disk shelves. The ESG Lab test bed, as presented in Figure 4, consisted of multiple industry-standard x86 servers with both 20Gbps Coraid HBAs and 1Gbps Ethernet NICs installed. Servers were running VMware ESX server with Red Hat Linux and Windows 2008 installed as guest operating systems as well as physical Linux and Windows 2008 installations. An industry standard Ethernet switch was used for SAN connectivity.[3]

ESG Lab Testing

ESG Lab testing began by powering on an SRX3500 EtherDrive SAN shelf, then logging into a Linux server. Coraid’s cec utility was used to scan for the new chassis using the AoE protocol. In less than a minute, the shelf was visible.

The next step was to name the shelf to make it easier to identify it in a large deployment. Shelf 3 was chosen as the name for these tests. Next, using just three commands, RAID groups were created (Coraid automatically creates one LUN per RAID group), hot spares were assigned, and the LUNs were brought online, as seen in Figure 5.

LUN masking, the means by which servers are given exclusive access to volumes in a SAN environment, is done by Ethernet MAC address using the “mask” command. ESG Lab did not use LUN masking in these tests.

On the Linux server, ls /dev/etherd showed all AoE devices on the network. The storage administrator has nothing else to do—no iSCSI mount, no NFS mount.  The AoE LUNs look like local storage. Next, ESG Lab used mkfs to create and format a file system on each of the AoE LUNs.

Creating LUNs and presenting them for use on the network took less than one minute, while creating the file systems for use by the server took about another minute. In less than two minutes and just four simple commands, ESG Lab configured, provisioned, and was using Coraid EtherDrive storage.

Disruptive Price-Performance

Coraid EtherDrive SAN storage is a modular disk storage system providing massive scale-out capacity and performance with granular, just-in-time scalability to industry standard, open systems environments. The Coraid solution scales by simply installing additional disks and shelves, allowing organizations to start small and scale capacity to petabytes. Using 2 TB SATA drives, users can scale to a petabyte of capacity and 100 GB/sec of raw storage bandwidth in just two racks.

Performance in a storage environment is best measured with the metrics used by the applications organizations actually run.  For an e-mail application, that measurement is the number of users or mailboxes a given system can support.  For a streaming media application, the number of objects served concurrently that can be sustained during peak periods of activity is the measurement that matters most.

ESG Lab Testing

Performance was tested using the IOMETER workload generator via simulated application workloads based on Microsoft Exchange and streaming media services.  Tests were performed to verify a Coraid platform’s ability to deliver predictably scalable performance in a clustered scale-out environment over a standard Ethernet network. The Exchange workload is random in nature and very disk intensive.

Microsoft guidelines recommend a maximum of 1,000 Exchange users per core and less for a server performing multiple roles. This means that a quad-core server, doing nothing but Exchange, should support about 4,000 users.

Microsoft’s IOPS per mailbox guidance for Exchange 2007 is calculated based on the number of messages per mailbox, the user memory profile, in what Outlook mode the mailboxes are operating, and whether any third party mobile devices are used. The baseline value provided by Microsoft is .32 IOPS per mailbox.[4] This means that a quad core Exchange server with 4,000 exchange users will, on average, drive 1,280 IOPS to the Exchange Datastore.  As can be seen in Figure 6, a single SRX3500 LUN was able to support enough transactional IO to support more than 4,500 Exchange users using just 12 SAS drives and scaled linearly to just over 9,000 users with 24 SAS drives.

Next, streaming media performance was examined. This type of traffic is sequential in nature and uses larger block sizes than transactional workloads, putting more of a load on the storage network.

As Figure 7 shows, streaming media performance was excellent, delivering 826 MB/sec from just 6 SSD drives and more than 1,200 MB/sec from 24 SATA drives. Put into perspective, a single shelf was able to drive enough bandwidth to saturate a 10Gbps interface.

The maximum throughput recorded (1200+ MB/sec) was used to calculate the number of streams that could be delivered for a couple of well-known content types including standard definition and high definition broadcast video.  Bit stream rates of 3.75 Mbps for standard definition broadcast video and 80 Mbps for high definition video were used to determine that a single SRX3500 has the bandwidth required to simultaneously stream 120 high definition broadcast videos or 2,560 standard definition broadcast videos as shown in Figure 8.

What the Numbers Mean

• The system showed excellent disk response times for both random and sequential IO. The simulated Exchange disk IO response time was 20ms, while streaming media requests from SATA disk were satisfied in just 1ms.
• Microsoft stresses that, to ensure a positive user experience, the Exchange database LUN requires read and write response times of 20 milliseconds or less so that Exchange can service users’ client software quickly and efficiently. In this context, the SRX 3500’s performance is right on target.
• A single SRX3500 has the raw bandwidth required to service 2,560 concurrent standard definition, broadcast-quality video streams.

Next, ESG lab examined cost of acquisition for a petabyte of storage and SAN connectivity for various technologies. Each storage technology was configured to support the same class and quantities of storage, and SAN connectivity was calculated to support 200 physical servers with redundant connections. Table 2 summarizes the configuration built for each technology.

The cost of storage and SAN connectivity hardware was obtained from a combination of publically available sources, including reseller websites, GSA pricing schedules, and online pricing available directly from vendors.

The cost was calculated for modular dual controller Fibre Channel SAN arrays from three major vendors. The cost of dual controller multi-protocol arrays from two major vendors and the cost of direct attached storage (DAS) solutions from two major vendors were also calculated. The solution with the lowest overall price in each category was used for the comparisons presented in this report.

The bottom line results are summarized in Figure 9. Note that the costs of iSCSI, multi-protocol, and FC SAN solutions are significantly higher than a comparable Coraid EtherDrive SAN system and that the base costs of a Coraid SAN solution are lower even than DAS.

Calculated costs are detailed in Table 3.

What the Numbers Mean

• Coraid EtherDrive SAN has the lowest cost of acquisition, by a wide margin.
• The relative cost of acquisition of alternative technologies ranges from roughly 1.4x for DAS to more than 5x for FC SAN.
• The FC SAN solution is so much more expensive in part due to the cost of acquiring FC SAN connectivity.
• DAS technology has a number of limitations that were not considered in this analysis. First and foremost, it is a dead-end when it comes to server virtualization. SAN attached storage is needed to take full advantage of the benefits of server virtualization. Storage capacity held captive within, or directly attached to, a server can’t be moved non-disruptively to another server for maintenance or better quality of service. SAN attached storage is also needed to achieve valuable disaster recovery capabilities that have recently become available from server virtualization vendors (e.g., VMware Site Recovery Manager). And finally, islands of DAS capacity typically lead to poor storage utilization. Poor storage utilization dramatically increases the overall cost of ownership.
• In addition to CAPEX, ESG Lab believes it is likely that Coraid EtherDrive’s simplified architecture and management would also yield OPEX savings over alternate technologies.

ESG Lab also compared price-performance for the Coraid EtherDrive SAN systems tested to publically available results published for DAS and traditional Fibre Channel SAN systems. Price-performance was determined using a simple calculation of cost in dollars for a specific configuration divided by the number of MB/sec supported by that platform.

Virtualization Optimized

Coraid EtherDrive SAN storage systems integrate with VMware using a simple driver that enables VMware to mount EtherDrive storage arrays as if they were local drives. A VMware administrator can provision and manage virtual machine storage without the need for FC SAN administration or iSCSI client configuration.

ESG Lab Testing

ESG Lab performed virtualization tests on a VMware ESX 4.0 environment with two physical servers and six virtual machines.

First, ESG Lab logged into the vSphere client and clicked on server 192.168.0.214. As seen in Figure 10, the Coraid EtherDrive HBA was visible in the list of storage adapters and volume 10, created using the steps in Figure 5, was visible and ready for use.

The volume was formatted and made available to virtual machines using the Add Storage wizard, shown in Figure 11.

Next, the volume was assigned to a virtual machine using the native VMware Add Hardware wizard. Once the addition was complete, the volume was visible to the Windows operating system on the virtual machine. Figure 12 shows the Windows Disk Administrator tool with the new drive circled in green.

Finally, ESG lab examined availability, testing the synchronous mirroring capability of the Coraid EMX EtherDrive Mirror Appliance as well as the ability to physically move disk drives between chassis without disruption.

The availability test bed, depicted in Figure 13, consisted of three Coraid EtherDrive SR2421 shelves, one EMX Mirror Appliance, and one vSphere server, with one virtual machine running Windows Server 2008.

Two 12-disk RAID5 LUNs were created on two separate shelves and synchronously mirrored through the EMX appliance. Mirroring two volumes using the EMX appliance could not have been simpler. The mkmir command was used to select the source and target volumes to be mirrored. This single command pairs the volumes and starts the synchronization.

Next, the volume was assigned to a Windows server 2008 VM on the vSphere server. Once the volumes were fully synchronized, an IOmeter workload was started on the server, performing a mixed read/write workload against the volume, set to continue indefinitely. Power to the primary SR shelf hosting one side of the mirror was killed. Iometer continued reading and writing to the volume with no errors.

Finally, a single eight-disk RAID5 LUN in a single chassis was used to test the online drive relocation capability of the Coraid architecture. The LUN was assigned to a Windows 2008 VM and an IOmeter workload was started on the server, again performing a mixed read/write workload against the volume, set to continue indefinitely.

Power was killed to the chassis housing the eight-drive RAID 5 LUN. All eight disks were then physically relocated from the primary chassis to a spare chassis. The spare chassis was then renamed to have the same shelf number as the original chassis and the eight-disk LUN was placed online.

Total time for this physical failover was approximately three minutes. After the LUN was placed back online, the IOMeter transactions resumed successfully with no further service interruption. Most, if not all, other architectures, including highly available Fibre Channel and iSCSI SANs, simply cannot take LUNs offline in a VMware environment while machines are running without bringing the server to a crashing halt.

ESG Lab Validation Highlights

• ESG Lab configured, provisioned, and was utilizing Coraid storage in less than two minutes from power on.
• The SRX3500 demonstrated the ability to support thousands of Exchange users using just 12 SAS drives.
• Coraid EtherDrive SAN was able to drive more than 1200MB/sec from a single appliance, enough to stream 2,560 broadcast quality video streams simultaneously.
• Commodity hardware and cost-efficient AoE connectivity enable a cost of acquisition far less than Fibre Channel, iSCSI, and even DAS.
• Coraid proved well-suited to virtualized environments, providing simple to provision SAN storage that looks to a VMware cluster like direct attached disk.
• The EMX Mirroring appliance provided synchronous data protection for volumes across shelves with no disruption to service.
• ESG Lab was able to remove drives that were actively being accessed and move them to a different chassis with only a momentary pause in IO and no errors.

Issues to Consider

• Coraid’s EtherDrive SAN is currently managed through a command line with no GUI. The system is incredibly simple to use and manage, with all necessary functions controlled through a few simple commands and logical, human readable addressing of shelves, disks, and LUNs. Coraid indicated plans to ship an upgraded management system in Q3 2010 with a GUI and REST API support.
• The Coraid EtherDrive SAN solution does not yet offer advanced storage virtualization functionality such as thin provisioning or storage tiering. The driving factor behind these features, reducing the cost of storage, does not necessarily affect Coraid as it does traditional SAN architectures, which typically sell for many multiples of Coraid’s acquisition cost. In addition, these features are increasingly available in software at the hypervisor or file system layer, further obviating the need for them as array-based features.

The Bigger Truth

With storage costs consuming at least 28% of IT budgets,[6] companies are under constant pressure to find ways to reduce costs. Taking a long hard look at reducing capital and operational costs in the storage environment makes sense and so today, more than ever, IT is investing in new technology with a clear focus on reducing storage costs.

The high capacity and performance requirements of scale-out applications including backup to disk, content delivery, server and desktop virtualization, clustered computing, rich media, and un-structured bulk storage are taxing the budgets and infrastructure of IT organizations. Traditional storage network infrastructure can provide the capacity, agility, and performance these applications need, albeit at a high cost of entry and daunting complexity.  Rows of equipment are often needed to provide a petabyte of capacity and gigabytes per second of throughput. Data center managers are being pushed to the limit as administrators spend more and more time managing an ever-expanding SAN infrastructure.

ESG Lab found that the Coraid EtherDrive SAN storage system delivers shockingly simple deployment and management, with complete functionality delivered via a handful of easy to use commands and rock solid Ethernet SAN connectivity delivered via the extremely lightweight AoE protocol. The ease of management, deep scalability, and performance required for bandwidth-intensive scale-out applications are seamlessly extended to VMware environments as well.

ESG Lab testing has confirmed that Coraid’s architecture provides consistent levels of throughput—even during hardware faults.  Sustained throughput in excess of 1,200 MB/sec was observed for large block sequential reads. Cost-efficiency was impressive, with acquisition costs as low as 20% of the costs of traditional SAN attached storage. ESG Lab also verified a very interesting recoverability and resiliency feature, whereby drives can be moved to a spare chassis while an application is running.

With EtherDrive SAN storage, Coraid has dramatically simplified storage for consolidated and virtualized environments while enhancing performance and providing incredible cost efficiency. While the speeds and feeds are impressive, ESG Lab is most impressed by the shocking simplicity of both the AoE protocol and the Coraid architecture, making management of petabytes a reasonable task. If your organization is struggling to keep up with exponential data growth while providing ever higher levels of performance and availability, ESG Lab recommends that you consider Coraid EtherDrive SAN storage as the foundation for your virtualized data center.

Appendix

[2] Source: ESG Research Report, 2010 IT Spending Intentions Survey, January 2010.

[3] Configuration details can be found in the appendix.

[4] http://msexchangeteam.com/archive/2007/01/15/432207.aspx

[5] Source: ESG Research report, The Impact of Server Virtualization on Storage, December 2007.

[6] Source: ESG Research Report, Enterprise Storage Survey, November 2008.

Overview

There is certainly a lot of hype around cloud computing as the technology industry positions itself for the “next big thing.” Unfortunately, this rhetoric masks an important inflection point: cloud computing is real and will impact IT in unprecedented ways.

ESG envisions the cloud’s evolution as a three-stage process (see Figure 1).  On the enterprise front, cloud computing will follow ongoing technology initiatives like data center consolidation and server virtualization as large organizations turn their IT infrastructures into their own private clouds. At the same time, public cloud options of all types (i.e., IaaS, PaaS, SaaS) will also mature, driven by new standards, development tools, and services offerings.  Enterprises will typically consume these services as an alternative to internal IT resources.  Finally, as standards emerge, large companies will bridge internal private clouds and external public cloud services into a federated hybrid cloud architecture. When this happens, traditional IT walls will disappear as the cloud brings ubiquity to computing (i.e., application processing, storage capacity, etc.) just as the Internet and IP protocols delivered universal connectivity for networks.

CIOs Should Start Preparing for the Cloud

At present, the entire IT industry is pouring massive amounts of R&D dollars into cloud computing. This will likely accelerate the technology maturation process, making cloud computing an increasingly attractive enterprise option.  Smart CIOs will quickly come to this realization and begin to adopt cloud computing by:

• Betting on virtualization technologies. Virtualization will be a foundational technology for cloud computing.  Why?  It disaggregates IT workloads from the underlying hardware. When combined with tools that enable workload mobility and flexibility, workloads can be moved from server to server, data center to data center, or private to public cloud services depending upon hardware contention, network traffic, IT resource limitations, or cost considerations.
• Modernizing the network. In the past, network traffic tended to flow in a “north and south” pattern from data centers to core, distribution, and then access networks.  With the explosion of Web applications and server virtualization, more traffic flows “east and west” between the servers themselves.  Make sure that data center networks are designed to handle this traffic, especially as the population of Web applications and virtual servers continues to explode.  Don’t forget to include data center to data center network capabilities in these plans as well.
• Embracing flexibility. With cloud computing, there is no “one-size-fits-all” model.  Rather, cloud computing is an entirely new mindset where CIOs have unprecedented flexibility. For the first time, IT managers have the true luxury of placing IT workloads (i.e., compute, application, storage) in the most advantageous and effective places.  As a result, future IT decisions should be based upon TCO, service quality, and time-to-market as much as existing IT infrastructure and organizational considerations.

What About Performance?

In spite of the promise of cloud computing, many IT professionals first conceive of two major hurdles: security and availability.  Yes, addressing these important issues is essential for cloud’s progress, but ESG believes that there is another key obstacle: high performance.  In fact, high performance connectivity to cloud services is crucial for:

• Connecting private and public clouds. High performance and responsive connectivity is a “must have” in cloud computing use cases for distributing applications, running processor-intensive scientific computing systems, mirroring transactions, and leveraging vast pools of cloud storage. Even high bandwidth “best effort” Internet connectivity could still be victimized by an unrelated traffic spike, possibly interrupting cloud computing systems and business operations.
• Providing user services. Private or public cloud ROI won’t matter if users experience reduced productivity due to unacceptable response time to access files, applications, or their virtual desktops.
• Future flexibility. Let’s face it—without high performance in the cloud, computing alternatives will remain suspect.  A few early performance problems could persuade risk-averse CIOs to abort their otherwise sound cloud initiatives and instead keep critical applications and services in-house, failing to realize the potential benefits.  High performance data migration also allows CIOs to adapt their approaches to new offerings in the future.

WAN Optimization will Bridge the Performance Gap

The performance issues above are prospective “show stoppers” for cloud computing, but there is a potential solution.  WAN optimization is usually equated with accelerating network traffic between enterprise data centers and branch offices, but ESG believes that it will also play a crucial role in cloud computing enablement.  Why?  Leading WAN optimization platforms already handle lots of diverse workloads to connect data centers to other data centers and a range of IT applications and services to end-users.

ESG sees cloud computing use cases as a new superset of WAN optimization functionality. That said, all WAN optimization solutions are not created equally.

To meet the challenges of cloud computing, WAN optimization offerings must provide high performance and:

• Support for multiple services. To connect cloud data centers, WAN optimization must accelerate network traffic, business system protocols and linkages, and remote mirroring application like EMC’s SRDF for disaster recovery. At the other end of the spectrum, users benefit from WAN optimization for high performance connections to business unit applications, cloud services, and file services. Finally, WAN optimization solutions can help enable greater use of cloud-based storage services for backup, archival, and near-line capacity by accelerating specific block based storage protocols (like iSCSI or REST.)  Each layer of IT resources, applications, servers, and storage can then be placed where it has the best mix of quality and cost, independent of other components’ locations.
• Multiple form factor choices. As workloads and users become more mobile, WAN optimization must find its way into lots of locations in the cloud and across the enterprise.  This will require additional form factors to traditional WAN optimization hardware appliances.  ESG sees the need for virtual WAN optimization appliances that can be easily deployed or moved with workloads around both public and private cloud environments. In addition, WAN optimization agents will accelerate cloud service performance for mobile worker endpoints across the globe.  Again, storage-specific cloud accelerators will also provide huge value.
• Visibility and manageability. Since the main goal is high performance, IT managers will want to monitor and adjust the network to ensure that business-critical traffic meets or exceeds SLAs, and plan for future moves of IT resources as desired.  This will require WAN optimization systems to act as a network service, providing consolidated visibility of real-time application traffic. With this information in hand, network engineers need the ability to fine-tune network traffic based upon changes in the application mix, cloud service adoption, or workload location.

With these characteristics, WAN optimization could go beyond high performance delivery alone.  ESG believes that these attributes could help large organizations take better advantage of cloud computing flexibility.  Why?  The combination of high performance, multiple form factors, and granular visibility will allow more choices for CIOs.  Enterprises can easily experiment and rapidly adapt their infrastructure by moving workloads around and assessing which location delivers the “biggest bang for the buck.” In this way, cloud-centric WAN optimization can also help improve the business’s top and bottom lines.

The Bigger Truth

Cloud computing is on the horizon. It will be arriving far sooner than many people think. It’s time that the cloud computing discussion moves beyond rhetoric and focuses on key issues. Beyond security and availability, ESG believes that high performance networking needs more attention.

WAN optimization is an obvious enabling technology as it is designed to accelerate network traffic between source and destination.  With cloud, however, the sources and destinations will multiply and change rapidly. This requires a new type of WAN optimization built with the cloud in mind. Cloud-centric WAN optimization will ultimately support lots of services, come in an assortment of form factors, and provide central visibility and management regardless of cloud service location.  Applications, servers, and storage can each be dynamically deployed and activated wherever most appropriate for the organization’s changing needs.

ESG believes that cloud-centric WAN optimization is an enabling technology that will play a major role in the success or failure of enterprise cloud computing initiatives. The sooner CIOs realize this, the more they can take advantage of available and emerging cloud computing options.

There are a few common themes here. Each vendor is targeting consumers with whiz-bang functionality and lots of applications. Video capabilities are highlighted in all cases.

Given this focus, you would think that mobile devices = consumer devices but this is not the case. Enterprises are also running to and jumping on the mobile device bandwagon in a big way.

ESG Research surveyed 174 IT professionals about their organizations’ adoption and use of mobile devices. Here are a few data points that illustrate growing mobile device usage in the enterprise.

Question 1. What are your organization’s spending plans for mobile devices and mobile device support?

37% spending will increase significantly
45% spending will increase moderately
14% spending will stay flat
3% spending will decrease
1% don’t know

Question 2. How important are mobile devices to your organization’s business processes and productivity?

38% critical
48% important
11% somewhat important
1% not important today but will be important in the future
1% not important today or in the future
1% don’t know

Question 3: Does your organization develop, or plan to develop, specific applications for mobile devices?

28% already develop applications for mobile devices
34% plan to develop applications for mobile devices
26% no plans at this time but interested in developing apps.
11% no plans or interest in developing apps.
1% don’t know

In summary, enterprises are spending more on mobile devices and device support, they believe these devices are “critical” or “important” for the business, and most already develop mobile device applications or plan to do so.

Sounds to me like every IT vendor in the endpoint (PC, laptop, mobile device), network, security, management, and application markets should have a mobile device strategy. Those that either haven’t developed or articulated their strategies are way behind.

Read Jon’s other blog entries at Insecure About Security.

Overview

Computer hardware used to be an expensive resource that had to be centralized and carefully managed. Historically, there was only one way to meet the growing demand for more resources and performance, and that was to scale up; the old mainframe, network switch, storage controller, or UNIX server got traded in for a newer, bigger, faster one.

That has all changed. Hardware is now a relatively low cost commodity, which has created a growing trend that moves organizations away from expensive, scale-up architectures toward commodity, scale-out architectures.

Scale-out architectures drive network flow in an edge-to-edge (from the server farms to the storage farms), rather than edge-to-core pattern (between enterprise platforms), fashion. This has a profound impact on network architectures, where the edge carries most of the traffic and the core manages lower volume transitive traffic volumes.

Transformational Directions

Figure 1 shows the main directional movements in networking and storage technologies.

• Structure: Market share has moved away from block to file structured storage due to its flexibility and ease of use. ESG predicts future storage solutions will be based on object stores with drivers that provide block and file presentation options for legacy systems.
• SAN: SAN ports are moving strongly in the direction iSCSI due to its lower TCO and management simplicity. Legacy Fibre Channel users will deploy FCoE as a bridging technology driven by the benefits of network convergence.
• Architecture: Scale-out has been a market leading approach for computing for many years, with significant activity in scale-out storage from HP, IBM, Isilon, and others. Most networks are still designed in a three-tier scale-up architecture, which is becoming increasingly expensive and inflexible, which in turn explains the trend toward two-tier networks.  With the new phenomenon of virtual machine “motion” occurring more regularly, network technologies are now required to adapt to these dynamic change conditions.  BLADE Network Technologies’s VMready is one example of this coming to fruition.
• Network: As intelligence and functionality, driven by the deployment of scale-out storage and computers, move to the edge and away from the core, networks need to adopt a more pragmatic edge-to-edge architecture.
• Protocols: The most prevalent LAN protocol will continue to be IP over Ethernet. The SAN will converge, initially into FCoE protocols to support legacy Fibre Channel deployments. Economics will drive a full migration toward IP protocols on the converged network.

Scale-out Architectures

The unbelievable levels of infrastructure performance today are still not enough for the largest Internet-scale tasks such as hosting Twitter, Facebook, or LinkedIn or managing search indexes at Yahoo or Google. The problem is that scaling up infrastructure just doesn’t meet the capacity demands of these enterprises, so they don’t do it anymore. Instead, they scale out at every level: compute, storage, network, application architecture, and even down to the database.

Soon, almost all devices will be connected to a network. Mass consumer trends in mobile telephony and personal digital assistants (PDA)—combined with integration of video, still cameras, GPS systems, and smart analytics—offer rich opportunities for adding value to all technology areas from social networking to national security. Trillions of electrically connected devices will be added to the network; electronic direction signs, traffic lights, parking meters, toll booths, surveillance cameras, and number plate recognition systems are already revolutionizing traffic management in cities such as Seoul, South Korea.

This Internet-scale problem is becoming more common; more organizations are processing huge volumes of data in near real-time. Developers are beginning to create scale-out solutions and applications. They are adopting a MapReduce style approach to coding where a set of master processes splits the problem into a number of smaller parts and then farms them out to a large number of processes that derive the answer. The master processes then combine the answers to deliver a single consolidated output. For the largest scale computational problems, this is often the only way to get to an answer in a meaningful time frame.

A Scale-out Approach to the Problem

Keeping Data and Computing Independent, but Close

These Internet-scale solutions need to separate compute and storage nodes to enable massive parallelism while maintaining high performance, low latency connectivity between active processes and the persistent data sets they need to manipulate. Scale-out or distributed file systems (such as Apache Hadoop Distributed File System (HDFS) and Google File System (GFS)) provide support for rack awareness—ensuring that the compute element is in the same locale as the storage element. Edge-to-core networking has no place here as almost all traffic flows from edge to edge.

Network Convergence

For some time, the networks (LANs) carrying data flows from server to server and from server to consumer have been separated from the storage area networks (SANs) carrying traffic from server to data storage. The reasons for this are fairly straightforward: LAN flows expect (and are designed to manage) congestion and variable latency as well as occasional data loss, while SAN flows do not. SAN latency and data loss typically cause poor application performance and data consistency problems.

High capacity (10GbE, 40GbE, and eventually 100GbE) lossless Converged Enhanced Ethernet (CEE), also known as Data Center Bridging (DCB), changes the rules for storage traffic. CEE can deliver data storage flows reliably and without latency or loss, eliminating the need to maintain a separate storage network. Firms like BLADE Network Technologies are well positioned to support these demands for a combination of network convergence and edge-to-edge connectivity with an open standards approach that enables interworking with Cisco, Brocade, and other vendors using their BLADE Unified Fabric Architecture (UFA).

The Economics

There are very strong economic drivers for converged SAN and LAN connectivity at the edge. The cost per GB of data transferred is twice as much for 8GB Fibre Channel as for 10GB Converged Enhanced Ethernet. So, by converging the networks into CEE, total capital spending is reduced by as much as two-thirds and cabling cost and complexity are reduced by as much as half. Converged networks enable wire-once-and-forget approaches to the data center.

What is the Right Protocol for Storage Networks?

Use cases are demanding more granular control of data. For example, does the data need to be geographically replicated, need to have enhanced protection, or need to deliver higher performance? Block structured data offers limited capability in this area; NAS and object stores provide the promise of rich metadata and significantly more granular control. Scale-out architectures overcome the perceived performance penalty except for the most extreme RDBMS requirements.

As data moves from block to file storage and onto object structures with rich metadata, block protocols like FCoE or iSCSI are replaced by NAS protocols such as NFS and CIFS and then by object protocols such as HTTP or URI. All, with the exception of FCoE (which is still rooted in the legacy of Fibre Channel) are IP-based and can be routed, isolated, and managed in the same way as normal LAN traffic.

What Does a Scale-out Network Look Like?

In a scale-out network, racks are filled with high-density compute nodes, each with a large memory footprint and large numbers of cores and sockets all interconnected via 10G CEE top of rack switches. LAN and SAN protocols are converged, with most systems choosing IP-based storage connectivity (iSCSI, NAS, HTTP). Storage platforms are placed close by and also interconnected via 10G CEE top of rack switches. Each switch is connected to a pair of end of row switches via 40GbE uplinks.

In these large 1000+ port network domains, most traffic flow is edge-to-edge between the computing elements and the storage elements. Congestion control is managed from within the domain and is visible and manageable at the edge.

Scale-out networking follows the logic that most network traffic in a scale-out world is edge-to-edge, so why bother with a massive capacity, core network? Instead, converge on 10G CEE using top of rack switches supporting iSCSI, NAS, and HTTP protocols to converge the SAN and LAN into a common routable IP system.

These networks are incrementally additive; as new workloads are introduced and new compute and storage racks deployed, top of rack switches are added at the same time. This approach avoids the conventional waterfall investment strategy where all network equipment needs to be installed and wired before the first server is installed. In scale-out networks, the core is much diminished in importance. The differentiator is the adoption of open standards, low power, low latency, and low cost modular switches that can be added incrementally as the compute and storage capacity of the data center grows.

Scale-out Storage

Scale-out storage is also a growth industry with vendors like NetApp, HP (IBRIX), and IBM gaining traction. Object storage is also gaining popularity, with URI or HTTP protocols such as Amazon S3 becoming commonplace. Scale-out storage lends itself to Ethernet connectivity leveraging the lower costs of CEE adapters and switches.

Scale-out Databases

Scale-out databases are now commonly referred to as NOSQL databases that go back in time to pre-relational designs that do not provide atomicity, consistency, isolation, durability (ACID), or consistency guarantees but allow sharing to split the data sets over multiple nodes to improve parallelism and scaling of the overall system.

Analysis

The computer industry changes only gradually, building on the legacy of past. We are guided to the future by gradual changes in thinking and behavior driven by global business, governmental, and economic trends. Scale-out architectures will be a significant part of our future.

• Scale-out is becoming a common approach, is gathering momentum in storage, and is beginning to catch on in networking.
• There are a number of highly visible use cases for analyzing the data thrown up by myriad network-connected devices in near real time.
• The number of organizations needing massive scale analytic support is expanding rapidly.
• Scale-up architectures are unable to deliver the raw power and throughput needed.
• Application architectures are changing to enable use of scale-out approaches. Progress is being made in the open source world, with vendors following closely behind.
• The cost differential between commodity components and specialist scale-up components is growing particularly in the networking space driving procurement choices.

Is Scale-up Finished?

Not all applications will lend themselves to a scale-out architecture—either because they are strongly transactional and require referential integrity between transactions or need to be run sequentially, such as mainframe batch processing. Some of these applications and use cases could benefit from an alternative approach that may well fit with a scale-out, or hybrid, architecture.

Certain high performance computing (HPC) use cases demand ultra high scale-up performance. Conventional relational databases with atomicity, consistency, isolation, and durability (ACID) guarantees can be partially scaled-out via clustering; in the end, they demand high clock speeds, high performance networking, and high performance disk subsystems.  In these limited cases, a scale-out architecture may not be the right answer.

The Bigger Truth

Organizations are demanding previously impossible levels of performance and throughput to support large-scale analytics. Vendors are leveraging the economics of commodity hardware combined with smart software to deliver scale-out products that not only deliver massive aggregate throughput but also can provide cost effective and flexible service to ordinary workloads.

Buyers love the economics and flexibility of scale-out computing where increases in demand do not lead to a complete re-architecture of IT infrastructure. Scale-out offers the promise of public cloud economics in the private cloud.

The last dominions of scale-up thinking remain in networking and database designs. NOSQL approaches are challenging the very largest scale database architectures while top of rack edge-to-edge network designs such as BLADE’s UFA are providing practical, cost effective scale-out network solutions in the data center.

Market Overview

The demands on and for storage are increasing and, commensurately, the challenge to effectively control the way vast amounts of data are created, stored, and accessed is intensifying. To address data growth without interrupting business operations, rapid deployment of storage and IT resources to meet increasing demand becomes a function of scalability. The primary objective of any type of scaling, therefore, is to create a dynamic, non-disruptive environment that supports data growth in such a way that system capabilities remain as balanced and productive as possible.

Scale-out Introduced

As computer and storage prices drop and performance and capacities continue to increase, low cost “commodity” systems are being used for high performance and high capacity workloads that previously could only be handled by scaling up to much faster storage devices or even to high-performance computers. Scale-out storage allows many small, low cost computers and commodity storage components to be combined and configured to either create an aggregate storage pool or to increase computing power and exceed that of a single traditional storage array or high-performance computer. This commodity-based scale-out model is offered by most vendors and is becoming popular, increasing demand for storage virtualization, shared data storage, and improved data protection services. Clusters and grids are examples of scale-out systems, but “scale-out” can apply just as well to SANs as it does to the NAS environments with which it is more commonly associated.

Scale-out Analysis

Scaling-out has the potential to change the face of data center operations by displacing large, costly enterprise storage equipment with a pool of less expensive devices. It differs from traditional architectures in that it adds new computing, networking, or storage resources to an environment; this is in contrast to scaling up, which can either consolidate several smaller devices into one larger device or simply expand that one device.

Scaling out is accomplished by adding additional nodes to a system—such as a server, a network switch, or a storage device. When you upgrade your PC to a faster model with higher capacity disk drives, you are scaling-up, not out. Scaling out would be expanding a two server system to four servers in order to increase overall compute power. Adding more storage devices to create a larger storage pool is therefore an example of scale-out storage just as creating more network ports by adding more switches is an example scale-out networking.

Essentially, scale-out solutions let users create larger shared pools of compute power and storage capacity to grow well beyond the limits of an individual node. However, they also have the potential to simultaneously create a more complex management environment simply as a result of all the interconnected pieces; generally, therefore, scale-out storage is usually packaged with a corresponding software management and virtualization/abstraction layer that can make hundreds or thousands of nodes behave like a single system (using a global namespace) and that allows the storage resources to be deployed as needed.

Scaling Storage Topologies

Although scale-out is often assumed to be synonymous with NAS and scale-up is, in turn, often assumed to be synonymous with SANs, the truth is that scalability is not a direct function of storage type; scale-out can therefore apply to all three major storage topologies:

DAS: As the first widely popular storage model, DAS (direct attached storage) implementations still comprise a large proportion of the installed storage capacity on all major operating systems. DAS is simpler to manage and can scale out—at least to a limited degree—for added performance and capacity by adding more, larger, or faster drives or by cascading additional control units; however, DAS doesn’t provide many of the efficiencies and advanced functionalities that simplify management efforts and add operational advantages as the storage pool grows.

NAS: NAS (network attached storage) is a special purpose architectural approach comprised of servers, disks, and management software dedicated to serving files over a network. This is why scale-out storage is frequently associated with NAS systems that scale to become a cluster of NAS nodes: these nodes allow for capacity and performance to scale either in combination or independently as needed and yet—this is a crucial point—still maintain a single system image. Unstructured data, normally in file format, is already a huge proportion—probably the majority—of all digital storage and is expected to significantly outpace the growth of structured data and hence increase demand for low-cost scale-out storage. Much of this data resides on NAS.

SAN: Scale-out storage is also an important aspect of SANs (storage area networks) and is usually achieved by adding more SAN switches and creating a scale-out SAN fabric. Users can still achieve linear scale by increasing performance “horsepower” and capacity in step. SANs provide excellent scale-out capability for large enterprises that anticipate significant growth in information storage requirements and, unlike direct attached storage, any excess capacity in SANs can be pooled to provide higher utilization of resources as well as advanced functionalities. SAN topologies can be implemented to meet different needs for SAN scaling: they can scale-out to connect SAN islands built around individual switches into larger single fabrics or can use routers to physically connect the switches while logically isolating the fabrics. Virtualization is also used heavily with SANs.

Making a Choice

The key for any organization searching for an optimum solution is to understand which of its goals are most important. As with so much in IT, the answer is “it depends.” Table 1 is a quick summary of the factors a user might consider in terms of the available topologies and their associations with the different scaling approaches. The eventual decision will almost certainly take other factors into account; these might include functionality, price, TCO, and skill/comfort levels as well as predicted changes and growth across the organization. Scale-out and -up vendors will battle over their relative values in terms of overall economic efficiency and “green-ness.”

What’s best will ultimately depend on the nature of the application: both scaling approaches are valuable tools for IT in general and storage in particular, and many organizations will continue to employ both (separately or in combination) for the foreseeable future as they offer useful, though varying, attributes. For instance, scaling up an existing system often results in simpler storage management than with a scale-out approach as the complexity of the underlying environment is reduced or at least known. That said, scale-up storage systems can only scale as far as the performance and capacity limits of individual storage resources permit.

The Bigger Truth

Putting things simply, to support growth, a user can either add infrastructure capabilities to the current devices (scale-up) or add more devices to the environment (scale-out). Scale-out storage can improve IT management’s ability to provide timely provisioning, greater resource utilization, required performance, and higher levels of data and system availability non-disruptively. The price differential between the two models is currently favoring scale-out computing for those applications that can embrace it, thanks largely to its use of commodity components.  In many environments, it is worth considering the two approaches in combination; what is certain is that scaling up and out will continue to play ever larger roles in IT as the demands to service business growth in a flexible manner climb ever higher.

IBM–has been too quiet during the downturn economically but now seems intent on changing that.  They are diligently working to develop messaging that works–to tell the world what they are and what they want to be.  They do great at high-end esoteric junk like “Smart Planet” but have been terrible at arming their soldiers and partners and customers with product and solution rationalization.  Granted, it’s a hard problem when you have a billion products, but they haven’t been able to do it at the macro level (between storage, networking, servers, services)–or micro level (when do I sell/buy XIV vs. DS5000 vs. DS8000?).  Until this is fixed, IBM garners no efficiency in the selling motion, and will continue to sub-optimize.  The good news is they are massive and entrenched, and if/when they do fix this, they will see immediate benefit.  IBM’s biggest battle near term will be fought amongst itself.

Oracle–the biggest threat to the majority of the big players is Oracle now that they have a whole stack approach.  We’re watching it in the “level 2 big iron” world–those app environments that use the same stuff as the level 1 transaction systems–but in much greater quantity.  Exadata is just the first instantiation of this for the big O.  What happens is companies run big iron mainframes or mega-huge Unix boxes, Cisco, Symm/HDS/DS8000s and Oracle in as their transaction systems–at a huge cost.  They then build data warehouses, BI systems, decision support systems, etc., by replicating those same transaction systems–only much bigger (10X is normal).  This level 2 business is HUGE for those who sell stuff into it.  Now Oracle is screwing up a cash cow by coming into your company, finding you out of compliance on your Oracle Dbase licensing, and making the whole problem go away by ripping out what you have and replacing it with a $7m all Oracle stack–hardware and software.  And, it’s working.  If you are EMC or NetApp or even IBM, this is bad news.  IBM can play the game as they have the pieces, but I’m not sure they have the sales muscle or focus.  HP is in the same boat.  There are BILLIONS at stake just in the level 2 big iron world.  Look for those under threat to be forced to partner or buy in order to negate the threat–namely the Dbase function itself.  You won’t beat Oracle as long as they control the rules–and the Dbase controls the rules.  You’ll need to find a better/cheaper way – Vertica, Greenplum, etc. are suddenly looking very appealing.  This is not lost on VMware either, as Oracle will try to do their own virtualization thing and keep VMware out.  SpringSource looks like a brilliant buy suddenly.  The BI/Analytics guys are going to have to find HW partners to play as well.  Oracle can wipe out an entire ecosystem worth 30B clams if they aren’t challenged.

Make no mistake about it – Oracle has the potential to be the most disruptive force in this IT universe, bar none.

HP v. Cisco.  This one is going to be awesome.  It boils down to this:  Cisco walked into HP’s bread and butter by hopping into the server space.  I’m not sure they really thought this through all the way.

HP countered by buying 3Com–wanting to become #2 (for now anyway) in the enterprise networking space.

HP is going to win this battle.  Here’s why:

It’s all about the margin structure.  Cisco has enjoyed roughly 70% margins in core networking forever, because they have had no real competition for the last 15 years.  They have not pushed the commodity envelope and passed on savings to the market–because they didn’t have to.  Now, they are addicted to that contribution margin–it funds everything else they do.

IF (note the big IF) they become successful in servers, they will only do so by changing their margin profile.  They will not ultimately be able to sustain margins in excess of HP or IBM or Dell–because they simply can’t buy at anywhere near the levels of the big server guys. Nifty packaging is just that–packaging.  It’s not a sustainable value proposition that justifies a huge margin profile.  Cisco has sold about 1,000 UCS systems over the last year, I think.  HP ships 8,500 servers a DAY.

To get to scale, Cisco is going to have to buy someone.  Dell?  That would be awesome and give them scale–but awfully expensively.  Who else?  They can’t buy IBM or HP, or Intel, so who else is there?  That means they have to do it organically–which will cost a fortune and drag down earnings in my estimation–let alone the distraction it will cause.

Best guess is Cisco tries for another year then suddenly gets very quiet.  They will package up bigger and bigger packages for fewer and fewer customers, and eventually be out of the business in any real sense.  VCE (VMware, EMC, Cisco) packages at the top of the pyramid could be a sustainable ecosystem, but not at volume scale.  I can’t see a continual R&D investment at a high level if it’s only going to be a boutique business.

Now for HP and networking, the opposite is true.  Networking margins for HP will be GREAT! They will be closer to 50%–twice that of servers–even if they price at 50% of Cisco.  HP already has 5,000 people trained on the products and is aggressively hiring salespeople.  HP will find no real trouble convincing shops to let them start as a legitimate number 2–their brand value alone pretty much guarantees it.  Once they are in, who knows?  No matter what, Cisco will be forced to either A: lower their pricing and eat margin or B: cede market share.  I suspect they will be forced to cede share, as once they drop pricing their whole model breaks–and the street will slaughter them.

1. HP only has to show minor, consistent share gains to win.
2. HP has a much deeper overall portfolio versus Cisco.  Cisco has networking and telepresence.  HP has everything and the kitchen sink.
3. Cisco can be outsold.  Their dominance has allowed them to move from hunters to farmers over the years, to the point where they are a salesforce of order takers/account managers versus deal assassins.  They can rectify this, but it will be costly.
4. If Cisco is committed to the server space long-term, they almost are forced to pick up the last big piece–storage.

Final thought here–HP has always pushed the commodity envelope.  They always push cost reductions all the way to the buyer.  They never really have been the ones who “gouge” the market when they are seated.  Their servers, storage, and networking have a long history of this–whereas Cisco (and many others addicted to the margin crack simply don’t operate that way).  The only way to avoid this is to augment that margin structure–similar to what EMC did when HDS came on strong and forced pricing corrections in the high-end storage market.  EMC reacted by being forced to price lower but made up for it with CLARiiON and ultimately lots of other things.  What does Cisco add to their mix?  It will be bad enough that HP gets a foothold in core networking and forces a one time price correction–but they won’t stop there.  They will keep on forcing it year over year.  At least that is what history tells me.  Even after they dominated the printer market, they didn’t try to artificially float margins–they forced costs to the buyer down.  Hard to do if it’s not part of your DNA. Maybe Telepresence becomes so huge it can offset those margin hits, but that’s a pretty big gamble.

Bring on the battles!

Read Steve’s other blog entries at The Bigger Truth.

The Challenges

The use of server virtualization technology is on the rise among organizations of all sizes and in all industries around the world. In a recent ESG survey of current and planned users of the technology, 52% of organizations had already deployed, while 48% plan to do so.[1] Given the impressive economic benefits of server virtualization, the glut of affordable and under-utilized processing power, and growing power and cooling issues in the data center, ESG predicts that the brisk adoption of server virtualization will continue for the foreseeable future.

ESG research indicates that the vast majority (87%) of organizations that have deployed server virtualization have done so in conjunction with networked storage. Compared to islands of direct attached hard drives, utilization is greatly increased when applications share a pool of networked storage. Applications deployed on virtual machines sharing a pool of storage are more mobile and available than those deployed on direct attached hard drives.

While the benefits of server virtualization and networked storage are clearly compelling, IT managers are faced with a number of challenges as they try to manage a consolidated mix of real-world applications running on a virtualized infrastructure. As shown in Figure 1, the top two concerns are performance and a general lack of information and best practices. This holds true across organizations of all sizes, regardless of the number of virtual servers deployed. That users would be so concerned with the performance of their infrastructures makes sense given the fact that 46% of virtualization users report that they currently run “Tier 1” applications on virtual machines and 33% plan to in the future.

The Solution

This ESG Lab report examines the performance of real-world application workloads running in a virtualized and consolidated IT environment that leverages the following technologies:

• IBM System Storage DS5020 Express and DS3950 Express storage systems: With high performance that is optimized for mixed workloads, the DS5020 Express and the DS3950 Express were designed for modular scalability (capacity and/or performance), high availability, and advanced functionality including copy services and remote replication.

• IBM BladeCenter HS22 servers: The IBM BladeCenter HS22 is a highly efficient server with extraordinary scalability that delivers the ability to add more processing power, memory, or IO needed in virtualized environments.
• QLogic 8 Gb SAN Switch Module and CFFh and CIOv form factor FC Expansion cards for IBM BladeCenter: Providing up to 20 ports of SAN connectivity, the QLogic 8 Gb Fibre Channel switch and expansion card infrastructure is designed to deliver sustained high throughput and reliability in highly available virtual environments.

• VMware vSphere: Building on the power of VMware Infrastructure, VMware vSphere transforms IT infrastructures into a private cloud which enables the automated delivery of IT infrastructure as a service.

The capabilities of the IBM servers and storage that were used during this evaluation are summarized in Figure 2. The IBM BladeCenter H supports up to 14 blade servers, each populated with up to eight Intel Xeon 5500 processor cores and 98 GB of DDR-3 RAM. The IBM System Storage DS5020 Express and the DS3950 Express support up to 112 drives (FC, SATA, or mixed configurations) and are equipped with up to 4 GB of cache and 8 GB/sec of internal bandwidth. The DS3950 Express is a variant of the DS5020 Express that is available in two pre-configured models. The DS5020 Express can be custom configured and adds support for full disk encryption (FDE). The DS5020 Express supports up to eight FC host interfaces and the DS3950 Express supports up to four.

The Results

This report examines the performance capabilities of IBM System Storage DS5020 Express and DS3950 Express storage systems running a mix of real-world applications in a VMware vSphere-enabled virtual server environment powered by a pair of IBM BladeCenter HS22 servers. In particular, this report explores how:

• A single BladeCenter HS22 achieved an excellent VMmark mixed workload score of 24.05@17 Tiles.
• A single DS5020 attached to a pair of BladeCenter HS22 servers running a mix of real-world application workloads in 16 virtual machines supports up to:
  • 8,680 mailboxes using the Microsoft Exchange Jetstress utility
  • and 3,593 small database IOs per second using the Oracle Orion utility
  • and 433 MB/sec of throughput for large OLAP Oracle Orion operations
  • and 3,317 simulated web server IOPs
  • and 374 MB/sec of throughput for simulated backup/scan/index jobs
  • with the predictably fast response times and scalability
  • Within a vSphere enabled infrastructure, the DS5020 Express achieved a maximum aggregate throughput of 3.1 GB/sec during bandwidth intensive throughput testing and 1.21 GB/sec during mixed application workload testing.

The predictably fast, mixed workload performance scalability of the virtualized environment tested by ESG Lab is summarized in Figure 3. The results will be explored in detail later in this report, but for now it should be noted that the performance of the DS5020 Express scaled extremely well as a mix of real-world application workloads ran in parallel on up to 16 virtual machines.

The balance of this report explores how mixed workload testing was accomplished, what the results mean, and why they matter to your business.

ESG Lab Validation

The real-world performance capabilities of the IBM DS5020 Express were assessed by ESG Lab at an IBM facility located in Tuscon, Arizona. The methodology presented in this report was designed to assess the performance capabilities of a single IBM DS5020 Express storage system shared by multiple virtual servers running a mix of real-world application workloads. The cooperation of VMware, IBM and QLogic was key to the success of this project. In particular, this project benefitted from VMware’s expertise in helping customers plan for the deployment of business-critical applications in virtual server environments and IBM’s long heritage of success in the modular storage systems market.

VMmark

Conventional server benchmarks were designed to measure the performance of a single application running on a single operating system inside a single physical computer. SPEC CPU2000 and CPU2006 are well known examples of this type of server benchmarking tool. Much like traditional server benchmarks, conventional storage system benchmarks were designed to measure the performance of a single storage system running a single application workload. The SPC-1 benchmark, developed and managed by the Storage Performance Council with IBM playing a key role, is a great example. SPC-1 was designed to assess the performance capabilities of a single storage system as it services an online interactive database application.

Traditional benchmarks running a single application workload can’t help IT managers understand what happens when a mix of applications are deployed together in a virtual server environment. To overcome these limitations, VMware created a mixed workload benchmark called VMmark. VMmark uses a tile-based scheme for measuring application performance and provides a consistent methodology that captures both the overall scalability and individual application performance of a virtual server solution. As shown in Figure 4, compared to a traditional benchmark, which tests a single application running on a single physical server, VMmark measures performance as a mix of application workloads are run in parallel within virtual machines deployed on the same physical server.

The novel VMmark tile concept is simple, yet elegant. A tile is defined as a mix of industry standard benchmarks that emulate common business applications (e.g., e-mail, database, web server). The number of tiles running on a single machine is increased until the server runs out of performance. A score is derived so that IT managers can compare servers with a focus on their performance capabilities when running virtualized applications.

The IBM BladeCenter HS22 used during this ESG Lab Validation has an excellent published VMmark score of 24.05@ 17 Tiles.[2]At a high level, this means that the IBM BladeCenter HS22 did 24.05 times more work than the dual processor, single core server that VMware used as a reference when VMmark was first released in 2007.

A Mixed Real-world Storage Benchmark Methodology

While VMmark is well suited for understanding the performance of a mix of applications running on a single server, it was not designed to assess what happens when a mix of applications is run on multiple servers sharing a single storage system. VMmark tends to stress server internals more than it does the storage system. The methodology presented in the balance of this report was designed to stress the storage system more than the servers. Taking a cue from the VMmark methodology, a tile-based concept was used. As shown in Figure 5, each tile is composed of a mixture of four application workloads. Two physical servers, each configured with eight virtual machines, were used to measure performance as the number of active tiles was increased from one to four.

The difference between the server-focused VMmark benchmarking and storage-focused ESG Lab benchmarking is shown in Figure 6. Note how VMmark testing is performed with a single server, often attached to multiple storage systems. As a matter of fact, the IBM BladeCenter HS22 VMmark results presented earlier in this report were achieved with a pair of IBM System Storage DS4700 arrays.[3] In other words, when vendors publish VMmark results, they make sure there is plenty of storage available so they can record the highest VMmark server score. This provides IT managers with a fair comparison of the performance capabilities of competitive server technologies.

ESG Lab storage-focused benchmarking uses a different approach. Instead of testing with a single server and more than enough storage, multiple servers are attached to a single storage system. Rather than running application level benchmarks which stress the CPU and memory of the server, lower level industry standard benchmarks are used with a goal of measuring the maximum mixed workload capabilities of a single storage system.

Mixed Workloads

Industry standard benchmarks were used to emulate the IO activity of four common business application workloads:

• E-Mail: The Microsoft Jetstress utility was used to generate e-mail traffic. Similar to the Microsoft LoadSimm utility used in the VMmark benchmark, Jetstress simulates the activity of typical Microsoft Exchange users as they send and read e-mails, make appointments, and manage to-do lists. The Jetstress utility is however a more light-weight utility than LoadSimm. Using the underlying Jet Engine database, Jetstress was designed to focus on storage performance.
• Database: The Orion utility from Oracle was used to generate database traffic. Much like Jetstress, Orion is a lightweight tool that is ideally suited for measuring storage performance. Orion was designed to help administrators understand the performance capabilities of a storage system, either to uncover performance issues or to size a new database installation without having to create and run an Oracle database. Orion is typically used to measure two types of database activity: response-time sensitive online transaction processing (OLTP) and bandwidth sensitive online analytic processing (OLAP).
• Web Server: The industry standard Iometer utility was used to generate web server traffic. The IO definition was composed of random reads of various block sizes. The web server Iometer profile used for this test was originally distributed by Intel, the author of Iometer. Iometer has since become an open source project.[4] Iometer tests were performed on Windows physical drives running over VMware raw mapped devices.
• Scan/read: The Iometer utility was used to generate a single stream of read traffic. Operations that tend to generate this type of large block sequential traffic include scan and index operations, long running data base queries, backup operations, bulk data uploads, and copies. One 256 KB sequential read workload was included in each tile to add a throughput intensive component to the predominantly random IO profile of interactive e-mail, database, and web server applications. As most experienced database and storage administrators have learned, a throughput intensive burst in IO traffic can drag down the performance for interactive applications, causing performance problems for end-users. Adding a few streams of throughput intensive scan/read traffic was used to determine whether interactive performance would remain predictably responsive as the amount of mixed IO utilization increased.

Each of the four workloads ran in parallel, with the Jetstress e-mail test taking the longest to complete (approximately three hours). The settings for each of the industry standard benchmarks are documented in the appendix.

Test Bed

VMware vSphere version 4.0 was installed on a pair of powerful IBM BladeCenter HS22 blades, each with a pair of quad-core processors and a QLogic 8 Gb Fibre Channel CFFh and CIOv expansion card providing four ports of connectivity per blade. A DS5020 Express with 112 15K RPM FC drives was connected to the servers through four QLogic 8 Gb SAN Switch Module for IBM BladeCenter as shown in Figure 7.

Drive Layout

The DS5020 drive configuration is summarized in Table 1. Four Exchange database volumes were configured. Each of the Exchange database volumes was configured with an eight drive RAID-10 database volume and a four drive RAID-10 log volume. The Oracle, web server and scan/read workloads ran against four drive RAID-10 volumes. The operating system volumes (Vmdk/OS) were configured using a 3+1 RAID-5 layout. Volume ownership was balanced across the dual controllers within the DS5020 Express and distributed evenly over the eight host interfaces. The volumes were spread evenly over two VMware host groups with a multipath policy of most recently used (MRU). [5]

Configuring Virtual Machines

Each of the Microsoft Exchange machines was configured with four virtual CPU cores, 32 GB of RAM, a virtual disk over VMFS for the operating system, and two mapped raw LUNs. DS5020 Express disk capacity was used for all storage capacity including VMware virtual disk files (VMDK), Windows Server 2008 operating system images, application executables, and application data. All of the application data volumes under test were configured as mapped raw LUNs (also known as raw device mapped, or RDM volumes). The configuration of one of the four virtual machines used for e-mail testing is shown in Figure 8. Note how three hard disks have been configured. One virtual disk for the operating system and two mapped raw LUNs for the Exchange database and Logs.

The Results

In a way, storage system benchmark testing is like an analysis of the performance of a car. Specifications including horsepower and acceleration from 0 to 60 are a good first pass indicator of a car’s performance. But while specifications provide a good starting point, there are a variety of other factors that should be taken into consideration including the condition of the road, the skill of the driver, and gas mileage ratings. Much like buying a car, a test drive with real-world application traffic is the best way to determine how a storage system will perform in real-world conditions.

Characterization

Performance analysis began with an examination of the low level aggregate throughput capabilities of the test bed. This testing was performed using the Iometer utility running within the eight virtual machines that were used later during mixed workload testing. The eight virtual machines accessed DS5020 Express storage through eight 8 Gbps FC interfaces.

Iometer access definitions, which measured the maximum throughput from disk, were used for this first pass analysis of the underlying capabilities of the DS5020 Express.[6] Similar to a dynamometer horsepower rating for a car, maximum throughput was used to quantify the power of a turbo-charged DS5020 Express storage engine. As shown in Figure 9, ESG Lab recorded a maximum throughput of 3.1 GB/sec.

What the Numbers Mean

• Much like the horsepower rating of a car, the throughput rating of a storage system is a good indicator of the power of a storage system’s engine.
• Storage throughput is a measure of the bandwidth available to the system. Throughput can be measured on a stream or aggregate basis. A stream is represented by one application or user communicating through one IO interface to one device. Aggregate throughput is a measure of how much data the storage system can move on a whole for all applications and users.
• ESG Lab throughput characterization was performed using the industry standard Iometer utility as 32 streams performed large sequential reads from eight logical devices through eight FC interfaces.[7]
• ESG Lab recorded a peak aggregate throughput of 3.1 GB/sec in a VMware vSphere environment.
• Forty-two percent of the throughput was delivered from DS5020 Express cache; the balance was serviced from disk.
• When comparing the performance capabilities of two servers in a virtual server environment, the server with more cache tends to perform better. ESG Lab is confident that a similar pattern holds true for storage systems. A storage system with more cache—and better caching algorithms—should perform better in a virtual server environment.
• ESG Lab characterization testing indicates that the DS5020 Express has more than enough cache and front end bandwidth to meet the needs of virtualized applications requiring up to 112 disk drives for capacity.
• ESG Lab is convinced that the patented caching algorithms of the DS5020 Express provide a significant performance boost during mixed application virtualized application testing.

Virtual Machine Utilization

Mixed application testing began with a quick analysis of server memory and CPU utilization to make sure that there were no bottlenecks between virtualized applications and the DS5020 Express. Memory and CPU utilization as reported by the VMware Infrastructure Manager are shown in Figure 10.

These screenshots were taken during the peak activity phase of the four tile test. With memory and CPU utilization at less than 10%, there was no obvious bottleneck between virtualized applications and the DS5020 Express.

Mixed Real-world IOPS Scalability

I/Os per second, or IOPS, is a measure of the number of operations that a storage system can perform in parallel. When a system is able to move a lot of IOPS—from disk and from cache— it will tend to be able to service more applications and users in parallel. Much like the horsepower rating for a car engine, the IOPS rating for a storage controller can be used as an indicator of the power of a storage system engine.

While IOPS out of a cache is typically a big number and can provide an indication of the speed of the front end of a storage controller, IOPS from disk is a more useful metric when determining the real-world performance of a storage system servicing a mix of business applications. For example, e-mail and interactive database applications tend to be random in nature and therefore benefit from good IOPS from disk. With that said, a mix of real-world applications tends to have random and sequential IO traffic patterns that may be serviced from disk or from cache.

ESG Lab measured IOPS performance as reported by the DS5020 Express as the number of virtual machines running mixed real-world application workloads was increased from four through sixteen. With a mix of random and sequential IO over 112 disk drives, the goal was not to record a big IOPS number; the goal with this exercise was an assessment of the scalability of the DS5020 Express as an increasing number of applications are consolidated onto a single virtualized platform. The IOPS scalability during the peak period of mixed workload activity is shown in Figure 11.

What the Numbers Mean

• IOPS varied throughput the mixed workload test with peaks occurring during the Orion small IOPs phase and towards the end as the Jetstress utility as it performed a database consistency check.
• A peak of 19,046 and a steady state of 12,688 IOPS were recorded during the four tile run.
• IOPS scaled in a near-linear fashion as mixed real-world application traffic increased from four through sixteen virtual servers.

Handling Throughput Spikes with Ease

As noticed during IOPS monitoring, peaks of throughput activity could be correlated to the periodic behavior of real-world applications. Two bursts of aggregate throughput were observed: the first during the Oracle large MBPS test, which simulates a throughput intensive OLAP application, and the second during the Jetstress database consistency check. A VMware vSphere view of mixed workload performance on one of the HS22 blades is shown in Figure 12.

What the Numbers Mean

• An aggregate throughput level of 1.21 GB/sec was recorded as mixed, real-world applications were run on 16 virtual machines sharing a single DS5020 Express storage system (605 MB/sec for one of the two physical servers is shown in Figure 12).
• As throughput intensified during the Oracle Orion OLAP test phase, bandwidth utilization for other mixed workloads operating in parallel remained steady.

Mixed Application Performance Scalability

Having looked at the IOPS and throughput ratings of the turbo-charged DS5020 Express engine, here’s where the rubber meets the road as we examine performance at the application level. The output from each of the industry standard benchmark utilities was analyzed to determine the performance scalability and responsiveness of real-world applications running in a consolidated virtual environment.

Microsoft Exchange

The Microsoft Jetstress tool was used to see how many simulated e-mail users could be supported by the DS5020 Express resources allocated for the Exchange application. The number of IOPS and response time for each database and log volume was recorded at the end of each Jetstress run. A response time goal of 20 milliseconds or less for DB reads and 5 milliseconds or less for log writes is required to pass the test. These values are defined by Microsoft as a limit beyond which end-users will feel that their e-mail system is acting slowly.

ESG used the following IBM guidelines from an IBM report describing the results of an IBM System Storage DS4800 Mailbox Jetstress Analysis report to interpret the results:

In an enterprise Exchange 2007 environment, performance is usually designed around a 0.5 IOPS user profile, which is equivalent to a very heavy Exchange user. While disk performance varies, generally you should calculate based on a one hundred IOPS per disk metric, which is a conservative starting point, and tune from there for your specific environment.[8]

Microsoft Jetstress logs were used to determine the number of IOPS and response times as the number of active virtual machines was increased from four through sixteen.[9] Based on a 0.5 IOPS user profile, the number of IOPS was used to calculate the number of supported Exchange users. The number of supported mailboxes as the number of virtual machines was increased from four to sixteen is shown in Figure 13, Figure 14 and Table 2.

What the Numbers Mean

• The single tile mixed application test supported 2,466 Exchange users with an average DB disk response time of 14 milliseconds.
• Performance scaled in a near-linear fashion to 8,680 users while the DS5020 Express was busy processing and servicing other applications concurrently.
• As the number of simulated e-mail users was increased, the DS5020 Express provided excellent response times that are well within Microsoft’s guidelines. For example, the Microsoft guideline for a database read volume is 20 milliseconds as shown by the dotted line in Figure 14.
• The four tile test, which produced 4,340 IOPS over 32 database drives, delivered 135 IOPS per drive—well above the conservative IBM guideline of 100 IOPS per drive.

Oracle Orion

The Oracle Orion utility was used to measure small transfer (8 KB) IOPS and response time and large transfer (1 MB) throughput. The small results are used to predict the performance and scalability of response time sensitive interactive database applications (e.g. OLTP). The large results are used to predict the performance of throughput intensive database mining and decision support systems (DSS).

ESG used the following guidelines from presentations presented at Oracle OpenWorld in November 2007 to interpret the results:

Target 5-10 millisecond for response time critical IO. Start by assuming 30 IOPS per disk for OLTP and 20 MB/sec per disk in DSS. This is way below the theoretical value but allows for media repair etc.[10]

For new or non-existing applications, use business rules or data model transaction profiles flow to understand “what is a transaction,” and then extrapolate for transactions per second or hour. Optionally you can use the numbers we have seen in our consulting gigs. Note that these are just guideline values. Use the following as basic guidelines for OLTP:

Low transaction system – 1,000 IOPS or 200MBytes/s

Medium transaction system – 5,000 IOPS or 600 Mbytes/s

High-end transaction system – 10,000 IOPS or 1Gbytes/s <- almost rarely achievable and usually TPC-C type workloads[11]

The results for the four tile Orion test are summarized in Table 3. A sample Orion report is shown in the Appendix.

What the Numbers Mean

• The four tile test achieved a grand total of 3,539 small IOPS and 433 large MBPS while the system was simultaneously running a mix of real-world application workloads.
• Using Oracle’s back of the envelope sizing guidelines, this level of IO activity is significantly higher than a typical “low transaction system” and nearly represents a “medium transaction system.”
• The total number of small IOPS processed during the busy four tile test yielded an excellent rate of 222 small IOPS per drive, which dwarfs the extremely conservative Oracle planning guideline of 30 IOPS per drive.
• Orion reported an average latency of 5.43 milliseconds for the small IOPs workload. Given the Oracle guidance of 5 to 10 milliseconds, ESG Lab believes that these are excellent results—especially given the mix of IO intensive workloads that were being serviced by the DS5020 Express in parallel.

Web Server and Scan/Read

Performance results as reported by the Iometer utility for the web server and scan/read workloads executing within virtual machines during the four tile test are listed in Table 4.

What the Numbers Mean

• Given the cache friendly, read-only nature of web server IO traffic, ESG Lab believes that these results indicate that the DS5020 Express has the horsepower required to service tens of thousands of simultaneous page requests.
• ESG Lab believes that a file system workload would produce results that are approximately similar to the web server workload used for this test.
• Each of the four scan/read streams sustained at least 90 MB/sec of throughput for the entire duration of the mixed workload test. A stream of this magnitude could service the data needs of a number of simultaneous backup streams, a very aggressive scan and index job, or a throughput intensive database table scan—with no perceivable performance impact on applications that are running parallel.

Much like the electrical system in your home, figuring out how many appliances you can run in parallel before blowing a fuse is not a function of the number of wires behind the walls. What matters more is the design of the circuits used to distribute the right amount of power to appliances when needed. ESG Lab testing indicates that the DS5020 Express engine delivers the right amount of power to virtualized applications when needed.

DS3950 Express Performance Analysis

The DS3950 Express supports up to four FC host interfaces as compared to the DS5020 Express, which supports up to eight FC host interfaces. Otherwise, the components and architecture of the DS3950 Express are exactly the same as the DS5020 Express. ESG Lab tested the DS5020 Express with only four active FC host interfaces connected to the IBM BladeCenter HS22 servers with a goal of analyzing the performance difference between the DS5020 Express and the DS3950 Express. The results are shown in Figure 15 and Table 5.

What the Numbers Mean

• The simulated DS3950 Express with four active host paths delivered 78.82% of the throughput and 92.17% of the IOPS from cache compared to the DS5020 Express with eight active paths.
• The mostly random ESG Lab mixed workload performed roughly the same. This is due to the fact that the most important performance consideration for mostly random business application workloads is the number of disk drives operating in parallel. In this case, the same number of drives was tested (112).

ESG Lab Validation Highlights

• 3.1 GB/sec of aggregate throughput was sustained during characterization testing within a VMware vSphere environment.
• Mixed real-world application workloads running simultaneously within sixteen virtual machines deployed over two IBM BladeCenter HS22 servers provided the performance needed to concurrently support:
  • 8,680 mailboxes using the Microsoft Exchange Jetstress utility
  • and 3,593 small database IOs per second using the Oracle Orion utility
  • and 433 MB/sec of throughput for large OLAP Oracle Orion operations
  • and 3,317 simulated web server IOPs
  • and 374 MB/sec of throughput for simulated backup/scan/index jobs
  • with the predictably fast response times and scalability
• Excellent IOPs per drive were recorded (e.g., 224 for the Oracle OLTP test).
• As the number of virtual machines sharing a single DS5020 was increased, performance scaled in a near linear fashion with predictably fast response times (16 to 17 millisecond for Jetstress DB reads, 5.35 to 5.54 milliseconds for Oracle Orion small IOPS).
• The DS5020 had horsepower to spare for rebuilds and advanced functions including copy services and remote replication.

Issues to Consider

• Generally accepted best practices and predominantly default VMware and IBM System Storage settings were used during the design of this test. As expected after any benchmark of this magnitude, deep analysis of the results indicates that tuning would probably yield slighter higher absolute results. Given that the goal of this test was not to generate a big number, ESG Lab is confident that the results presented in this report meet the objective of estimating performance scalability and responsiveness as a growing number of virtual machines share a consolidated pool of DS5020 Express storage.
• For applications requiring extreme performance beyond that which is provided by FC and SATA drives, ESG Lab believes that the DS5020 Express is an ideal architecture for the selective use of solid state disk (SSD) devices. While mixed workload testing was not performed with SSD devices, ESG Lab is confident that SSD devices could be used to improve the performance of highly referenced database indexes and temp files.
• The test results/data presented in this document are based on industry-standard benchmarks deployed together in a controlled environment. Due to the many variables in each production data center environment, it is still important to perform capacity planning and testing in your own environment to validate a storage system configuration.

The Bigger Truth

Server virtualization is being deployed by a growing number of organizations to lower costs, improve resource utilization, provide non-disruptive upgrades, and increase availability. Each benefit is fundamentally enabled by de-coupling servers, applications, and data from specific physical assets. Storage virtualization takes those very same benefits and extends them from servers to the underlying storage domain—bringing IT organizations one step closer to the ideal of a completely virtualized IT infrastructure.

While the benefits of a completely virtualized infrastructure are obvious to most IT managers, performance is a real concern. Server, storage, and network administrators are looking for answers to a number of questions:

• Can we meet performance service level agreements for a mix of business-critical applications?
• Does the storage system have the horsepower to serve mixed, real-world applications?
• Can the storage system scale to accommodate future growth and consolidation?

IBM approached ESG Lab with an ambitious goal of answering these questions. A performance benchmark was designed to measure the performance capabilities of a storage system subjected to an IO intensive mix of virtualized business applications. Taking a cue from the VMmark benchmark from VMware, a “tile” concept was used during the design of this test. Each “tile” was composed of four applications, each running in its own virtual machine. The server horsepower of a pair of IBM BladeCenter HS22 servers, with an excellent published VMmark score of 24.05@17 tiles, was used to drive up to four tiles and sixteen virtual applications in parallel. ESG believes that the results of this storage-focused benchmark complement the excellent server-focused results of the IBM BladeCenter HS22 VMmark test.

IBM has more than a decade of experience delivering modular FC-attached storage systems designed to meet the cost-optimized performance demands of medium-sized organizations, mid-tier applications, remote departments, and near-line applications. The IBM DS5000 series builds on the heritage of the previous generation DS4000 series disk system with more than 87,000 systems and 511 petabytes shipped to date. The engine under the hood of DS5000 Series has been turbo-charged to meet the real-world performance demands of virtualized applications. With twice the host bandwidth and three times the internal bandwidth of the previous generation DS4700, the DS5020 Express is designed to deliver the high performance, low latency, and balanced scalability needed to meet the demanding performance needs of a mix of real-world applications sharing a consolidated infrastructure.

ESG Lab testing began with a confirmation that the DS5020 Express test bed can deliver up to 3.1 GB/sec of raw aggregate throughput in a VMware vSphere environment. This result was an early indicator that the IBM DS5020 Express has the internal bandwidth and processing power needed to serve a mix of real-world application workloads. The results of the mixed workload tests were even more impressive. A single DS5020 Express simultaneously supported 8,680 simulated Exchange users and 4,144 Oracle Orion small database IOs per second and 433 MB/sec of throughput for large OLAP Oracle Orion operations and 4,551 simulated web server IOPs and 374 MB/sec of throughput for bandwidth intensive streams of read traffic—all while delivering predictably fast response times. Testing on the DS5020 Express with the same number of drives and less host connections indicates that the IBM System Storage DS3950 Express delivers similar levels of performance for the mix of applications tested by ESG Lab.

ESG Lab is pleased to report that the combination of IBM System Storage DS5020 Express, IBM BladeCenter HS22 servers and QLogic 8 Gb Fibre Channel switch and expansion card infrastructure delivers the performance needed to meet the needs of a mix of real-world business applications running within a VMware vSphere enabled virtual infrastructure.

Appendix

This is an example of the output created by the Jetstress utility. It shows the performance for one of four Jetstress tests running in parallel. Specifically, this report was created by the Jetstress utility running on a virtual machine within the fourth tile of the four tile test.

Microsoft Exchange Server Jetstress

Performance Test Result Report

Test Summary

Database Sizing and Throughput

Jetstress System Parameters

Disk Subsystem Performance

Host System Performance

Test Log10/8/2009 12:41:13 PM — Jetstress testing begins …

10/8/2009 12:41:31 PM — Prepare testing begins …

10/8/2009 12:41:33 PM — Attaching databases …

10/8/2009 12:41:33 PM — Prepare testing ends.

10/13/2009 3:41:47 PM — Jetstress testing begins …

10/13/2009 3:41:47 PM — Prepare testing begins …

10/13/2009 3:41:48 PM — Attaching databases …

10/13/2009 3:41:48 PM — Prepare testing ends.

10/13/2009 3:41:48 PM — Dispatching transactions begins …

10/13/2009 3:41:48 PM — Database cache settings: (minimum: 32.0 MB, maximum: 256.0 MB)

10/13/2009 3:41:48 PM — Database flush thresholds: (start: 2.6 MB, stop: 5.1 MB)

10/13/2009 3:41:49 PM — Database read latency thresholds: (average: 0.02 seconds/read, maximum: 0.05 seconds/read).

10/13/2009 3:41:49 PM — Log write latency thresholds: (average: 0.01 seconds/write, maximum: 0.05 seconds/write).

10/13/2009 3:41:50 PM — Operation mix: Sessions 32, Inserts 40%, Deletes 30%, Replaces 5%, Reads 25%, Lazy Commits 55%.

10/13/2009 3:41:50 PM — Performance logging begins (interval: 15000 ms).

10/13/2009 3:41:50 PM — Attaining prerequisites:

10/13/2009 3:43:56 PM — \MSExchange Database(JetstressWin)\Database Cache Size, Last: 242573300.0 (lower bound: 241591900.0, upper bound: none)

10/13/2009 5:43:56 PM — Performance logging ends.

10/13/2009 5:43:56 PM — JetInterop batch transaction stats: 103218.

10/13/2009 5:43:57 PM — Dispatching transactions ends.

10/13/2009 5:43:57 PM — Shutting down databases …

10/13/2009 5:43:57 PM — Instance2924.1 (complete)

10/13/2009 5:43:58 PM — Performance logging begins (interval: 30000 ms).

10/13/2009 5:43:58 PM — Verifying database checksums …

10/13/2009 5:57:56 PM — E: (19% processed)

10/13/2009 5:57:56 PM — Verifying log checksums …

10/13/2009 5:57:56 PM — F:\ (0 logs passed)

10/13/2009 5:57:56 PM — C:\Downloads\Jetstress\Logs\Performance_2009_10_13_15_41_49.blg has 488 samples.

10/13/2009 5:57:56 PM — Creating test report …

10/13/2009 5:57:57 PM — Volume E: has 0.0165 for Avg. Disk sec/Read.

10/13/2009 5:57:57 PM — Volume F: has 0.0004 for Avg. Disk sec/Write.

10/13/2009 5:57:57 PM — Volume F: has 0.0000 for Avg. Disk sec/Read.

10/13/2009 5:57:57 PM — Test has 0 Maximum Database Page Fault Stalls/sec.

10/13/2009 5:57:57 PM — Test has 0 Database Page Fault Stalls/sec samples higher than 0.

10/13/2009 5:57:57 PM — C:\Downloads\Jetstress\Logs\Performance_2009_10_13_15_41_49.xml has 479 samples queried.

This is an example of the output created by the Oracle Orion utility. It shows the performance for one of eight Orion tests running in parallel. Specifically, this report was created by the Orion utility running on a virtual machine within the fourth tile of the four tile test.

ORION VERSION 10.2.0.1.0

Commandline:

-run advanced -testname vmware -num_disks 5 -size_small 8 -size_large 1024 -type rand -simulate raid0 -write 30 -duration 150 -matrix basic

This maps to this test:

Test: vmware

Small IO size: 8 KB

Large IO size: 1024 KB

IO Types: Small Random IOs, Large Random IOs

Simulated Array Type: RAID 0

Stripe Depth: 1024 KB

Write: 30%

Cache Size: Not Entered

Duration for each Data Point: 150 seconds

Small Columns:, 0

Large Columns:, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Total Data Points: 36

Name: \\.\f: Size: 1924071424

1 FILEs found.

Maximum Large MBPS=108.27 @ Small=0 and Large=9

Maximum Small IOPS=882 @ Small=25 and Large=0

Minimum Small Latency=5.42 @ Small=1 and Large=0

This is an example of the output created by the Iometer utility after a web server test run. This example shows the performance of the four web server tests which ran in parallel during the mixed workload four tile test.

This is an example of the output created by the Iometer utility after a scan/read test run. It shows the performance of the four scan/read tests which ran in parallel during the mixed workload four tile test.

The following excerpts were extracted from the IBM DS5020 Storage System Profile Summary.

PROFILE FOR STORAGE SUBSYSTEM: ESG_DS5020 (Fri Oct 02 05:53:29 PDT 2009)

SUMMARY——————————

Number of controllers: 2

High performance tier controllers: Enabled

Number of arrays: 24

RAID 6: Enabled

Total number of logical drives used: 25

Number of standard logical drives: 24

Number of access logical drives: 1

Total number of logical drives allowed: 1024

Drive Limit Management:

Number of drive slots discovered: 112

Number of drive slots allowed: 112

FlashCopy Logical Drives: Enabled

Number of flashcopies used: 0

Number of flashcopies allowed: 2

Number of flashcopies allowed per base logical drive: 2

Remote Logical Drive Mirroring: Disabled/Deactivated

Number of mirrors used: 0

Number of mirrors allowed: 0

VolumeCopy: Disabled

Number of copies used: 0

Number of copies allowed: 0

Number of drives: 112

Mixed drive types: Enabled

Current media type(s): Hard Disk Drive (112)

Current interface type(s): Fibre (112)

Total hot spare drives: 0

Standby: 0

In use: 0

Drive Security: Disabled

Security key identifier: None

Storage Partitioning: Enabled

Number of partitions used: 2

Number of partitions allowed: 128

Number of logical drives allowed per partition: 256

Access logical drive: LUN 31,31,31 (see Mappings section for details)

Default host OS: DEFAULT (Host OS index 0)

Current configuration

Firmware version: 07.60.08.00

NVSRAM version: N1814D20R1060V08

EMW version: 10.60.G5.05

AMW version: 10.60.G5.05

NVSRAM configured for batteries: Yes

Start cache flushing at (in percentage): 80

Stop cache flushing at (in percentage): 80

Cache block size (in KB): 16

Media scan frequency (in days): Disabled

Failover alert delay (in minutes): 5

Feature enable identifier: 3030303934203030313235204A93DE36

Feature pack: DS5020 Model 20, 24, 28

Feature pack submodel ID: 121

Storage Subsystem world-wide identifier (ID): 60080E500017B6BA000000004A93DE34

CONTROLLERS——————————

Number of controllers: 2

Controller in Enclosure 85, Slot A

Status: Online

Current configuration

Firmware version: 07.60.08.00

Appware version: 07.60.08.00

Bootware version: 07.60.08.00

NVSRAM version: N1814D20R1060V08

Replacement part number: 37781-03

Model name: 4988

Board ID: 4988

Submodel ID: 121

Product ID: 1814 FAStT

Revision: 1060

Replacement part number: 37781-03

Part number: 37781-03

Serial number: SQ91100094

Vendor: IBM

Date of manufacture: June 2, 2009

Trunking supported: No

Data Cache

Total present: 1709 MB

Total used: 1709 MB

Processor cache:

Total present: 339 MB

Cache Backup Device

Status: Optimal

Type: USB flash drive

Location: Controller A, Connector USB 1

Capacity: 1,960 MB

Product ID: eUSB

Part number: Not Available

Serial number: 200902190239A7D8

Revision level: 8715

Manufacturer: SMART

Date of manufacture: Not available

Host Interface Board

Status: Optimal

Location: Slot 1

Type: Fibre channel

Number of ports: 2

Board ID: 0902

Replacement part number: L2-25043-03

Part number: PN L2-25043-03

Serial number: SN SQ91100387

Vendor: VN LSI

Date of manufacture: June 1, 2009

Date/Time: Fri Oct 02 05:54:13 PDT 2009

Associated Logical Drives (* = Preferred Owner):

BK_01*, BK_04*, JS_01*, JS_03*, JS_2L*, JS_4L*, OR_02*, OR_04*, OS_01*, OS_03*,

WB_02*, WB_03*

STANDARD LOGICAL DRIVES——————————

SUMMARY

Number of standard logical drives: 24

See other Logical Drives sub-tabs for premium feature information.

NAME STATUS CAPACITY RAID LEVEL ARRAY MEDIA TYPE INTERFACE TYPE

BK_01 Optimal 557.791 GB 10 BK_01 Hard Disk Drive Fibre channel

BK_02 Optimal 557.793 GB 10 BK_02 Hard Disk Drive Fibre channel

BK_03 Optimal 557.793 GB 10 BK_03 Hard Disk Drive Fibre channel

BK_04 Optimal 557.793 GB 10 BK_04 Hard Disk Drive Fibre channel

JS_01 Optimal 1.089 TB 10 JS_01 Hard Disk Drive Fibre channel

JS_02 Optimal 1.089 TB 10 JS_02 Hard Disk Drive Fibre channel

JS_03 Optimal 1.089 TB 10 JS_03 Hard Disk Drive Fibre channel

JS_04 Optimal 1.089 TB 10 JS_04 Hard Disk Drive Fibre channel

JS_1L Optimal 557.793 GB 10 JS_1L Hard Disk Drive Fibre channel

JS_2L Optimal 557.793 GB 10 JS_2L Hard Disk Drive Fibre channel

JS_3L Optimal 557.793 GB 10 JS_3L Hard Disk Drive Fibre channel

JS_4L Optimal 557.793 GB 10 JS_4L Hard Disk Drive Fibre channel

OR_01 Optimal 557.793 GB 10 OR_01 Hard Disk Drive Fibre channel

OR_02 Optimal 557.793 GB 10 OR_02 Hard Disk Drive Fibre channel

OR_03 Optimal 557.793 GB 10 OR_03 Hard Disk Drive Fibre channel

OR_04 Optimal 557.793 GB 10 OR_04 Hard Disk Drive Fibre channel

OS_01 Optimal 836.689 GB 5 0S_1 Hard Disk Drive Fibre channel

OS_02 Optimal 836.689 GB 5 OS_02 Hard Disk Drive Fibre channel

OS_03 Optimal 836.689 GB 5 OS_03 Hard Disk Drive Fibre channel

OS_04 Optimal 836.689 GB 5 OS_04 Hard Disk Drive Fibre channel

WB_01 Optimal 557.793 GB 10 WB_01 Hard Disk Drive Fibre channel

WB_02 Optimal 557.793 GB 10 WB_02 Hard Disk Drive Fibre channel

WB_03 Optimal 557.793 GB 10 WB_03 Hard Disk Drive Fibre channel

WB_04 Optimal 557.793 GB 10 WB_04 Hard Disk Drive Fibre channel

DETAILS

Logical Drive name: BK_01

Logical Drive status: Optimal

Capacity: 557.791 GB

Logical Drive ID: 60:08:0e:50:00:17:b6:ba:00:00:1a:5e:4a:c0:a8:24

Subsystem ID (SSID): 12

Associated array: BK_01

RAID level: 10

Secure: No

Media type: Hard Disk Drive

Interface type: Fibre channel

Enclosure loss protection: No

Preferred owner: Controller in slot A

Current owner: Controller in slot A

Segment size: 512 KB

Capacity reserved for future segment size changes: Yes

Maximum future segment size: 2,048 KB

Modification priority: High

Read cache: Enabled

Write cache: Enabled

Write cache without batteries: Disabled

Write cache with mirroring: Enabled

Flush write cache after (in seconds): 10.00

Dynamic cache read prefetch: Enabled

Enable background media scan: Enabled

Media scan with redundancy check: Disabled

Pre-Read redundancy check: Disabled

MAPPINGS (Storage Partitioning – Enabled (2 of 128 used))——————-

Logical Drive Name LUN Controller Accessible by Logical Drive status

BK_01 4 A Host Group VMware_01 Optimal

BK_03 10 B Host Group VMware_01 Optimal

JS_01 2 A Host Group VMware_01 Optimal

JS_03 8 A Host Group VMware_01 Optimal

JS_2L 3 A Host Group VMware_01 Optimal

JS_4L 9 A Host Group VMware_01 Optimal

OR_02 1 A Host Group VMware_01 Optimal

OR_04 7 A Host Group VMware_01 Optimal

OS_01 0 A Host Group VMware_01 Optimal

OS_03 6 A Host Group VMware_01 Optimal

WB_02 5 A Host Group VMware_01 Optimal

WB_04 11 B Host Group VMware_01 Optimal

BK_02 4 B Host Group VMware_02 Optimal

BK_04 10 A Host Group VMware_02 Optimal

JS_02 2 B Host Group VMware_02 Optimal

JS_04 8 B Host Group VMware_02 Optimal

JS_1L 3 B Host Group VMware_02 Optimal

JS_3L 9 B Host Group VMware_02 Optimal

OR_01 1 B Host Group VMware_02 Optimal

OR_03 7 B Host Group VMware_02 Optimal

OS_02 0 B Host Group VMware_02 Optimal

OS_04 6 B Host Group VMware_02 Optimal

WB_01 5 B Host Group VMware_02 Optimal

WB_03 11 A Host Group VMware_02 Optimal

Access Logical Drive 31 A,B Host deimos Optimal

Access Logical Drive 31 A,B Host phobos Optimal

Access Logical Drive 31 A,B Storage Subsystem Optimal

[2] The full disclosure for the IBM BladeCenter HS22 report is available at http://www.vmware.com/files/pdf/vmmark/VMmark-IBM-2009-06-30-HS22.pdf. To learn more about VMmark, including a full list of published results, go to http://www.vmware.com/products/vmmark/

[3] The IBM System Storage DS4700 is the previous generation of the IBM System Storage DS5020 examined by ESG Lab in this report.

[4] Web server Iometer (www.sourceforge.net/projects/iometer) workload definitions are included in a results file excerpt as Figure 13.

[5] For more detail, see the Appendix.

[6] The configuration and methodology that was used during characterization testing is described in the Appendix.

[7] For more on the characterization configuration and methodology please see the Appendix.

[8] IBM System Storage DS4800 Exchange Server 2007 15,000 Mailbox JetStress Analysis, David Hartman and David West, November 2007, http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101123

[9] A sample JetStress log is included in the Appendix as Figure 11.

[10] Current trends in Database Performance, Andrew Holdsworth, Oracle OpenWorld, Nov 2007, http://www.oracle.com/technology/deploy/performance/pdf/PerfTrends_Holdsworth.pdf

[11] Back of the Envelope Database Storage Design, Nitin Vengurlekar, RAC/ASM Development, Oracle Open World, Nov 2007, http://www.oracle.com/technology/products/database/asm/pdf/back%20of%20the%20env%20by%20nitin%20oow%202007.pdf

[12] See Figure 13 for workload details.