Testing was designed to demonstrate how EMC ProSphere intuitively provides end-to-end resource management for storage environments. Also of interest were ease of use and implementation as well as agent-less host, SAN, and array data collection.

Please click here to read the full report.

Background

Organizations need to be able to respond to rapidly changing market dynamics, customer requirements, and competitive threats. Many are building out private cloud infrastructures in order to achieve the requisite level of agility. One of the foundational technologies of a private cloud is server virtualization, so it shouldn’t be a surprise to learn that increasing the use of server virtualization technology has been the top IT priority for the last two years in a row according to ESG research.[1] This, in turn, is driving increased growth in networked storage environments, which are critical to enabling virtualized deployments to be highly mobile and available. In fact, ESG research indicates that the increased use of SANs was the number one impact server virtualization has on storage environments.

This rapid growth in server virtualization and SAN deployment is affecting the management of these increasingly complex and abstracted environments. In most cases, organizations end up changing management tools and processes. And this goes beyond just the server environment to include storage and network environments (see Figure 1).[2]

As organizations continue to evolve their environments and deploy highly virtualized cloud infrastructures in production, easy to deploy, simple to use management tools that provide end-to-end configuration and performance visibility will become increasingly critical.

EMC ProSphere

EMC ProSphere is an end-to-end storage resource management application designed for quick, easy, and cost-effective deployment to enable a rapid return on investment. With ProSphere, storage professionals can create a detailed view of their storage infrastructure from the host, through the SAN, to the backend storage environment.

ProSphere is designed to be deployed in a VMware environment as a virtual appliance or vApp. For ease of deployment, it’s packaged as an OVF template the can be downloaded from the EMC support website. To further simplify and reduce the management burden, it leverages an agent-less discovery and data ingest paradigm.  ProSphere gets its performance data, alerts, and end-to-end visibility using standard protocols like SNMP and SMI- S for switches and arrays and WMI, SSH, or vCenter for physical and virtual hosts.

• ProSphere is designed to be easily deployed as a virtual appliance in a VMware environment.
• The discovery and data ingest processes are agent-less, leveraging SMI-S, SNMP and manufacturer APIs.
• ProSphere can be integrated with Microsoft AD and the LDAP protocol.
• Customers with active EMC ControlCenter support contracts can use ProSphere at no cost.
• EMC ControlCenter performance data can be imported into ProSphere.
• Administrators can manage multiple ProSphere instances without logging into each site separately.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of ProSphere at an EMC facility in Hopkinton, MA. Testing was designed to demonstrate how EMC ProSphere intuitively provides end-to-end resource management for storage environments. Also of interest were ease of use and implementation as well as agent-less host, SAN, and array data collection.

The configuration used for ESG Lab testing is shown in Figure 3. On the left side of the figure are the three VMware virtual machines that made up the EMC ProSphere operating environment. These VMs consist of the Historical Database, Discovery Appliance, and the Storage Resource Manager. On the right side of the figure are the test bed objects that were discovered and monitored during the ESG Lab testing. EMC VMAX and CLARiiON arrays were discovered, as were Cisco and Brocade SAN switches. The host environment consisted of physical and virtual Windows servers as well as Solaris 9 and Solaris 10 servers. A LAN with multiple VLANs provided connectivity between the ProSphere environment and the objects being monitored.

Getting Started

Getting started with EMC ProSphere begins with obtaining and installing the OVF template packaged code to bring the environment to the initial login screen. This was accomplished by downloading the ProSphere application software from the EMC web support portal[3] and installing the application into an existing VMware environment.

ESG Lab Testing

ESG Lab began testing ProSphere by copying the downloaded OVF formatted application code to the test VMware vSphere environment. ESG Lab used the “Deploy OVF Template” option to select and install the ProSphere application. Figure 4 shows the summary of entries including hostnames, networks, and other common system configuration settings that were selected as part of the template-driven install process. The settings were confirmed and the “Finish Install” tab was selected. The ProSphere installation process completed in approximately 30 minutes and resulted in the creation of a virtual application with three virtual machines.

ESG Lab used the vSphere client to verify the creation of the application. Figure 5 shows the ProSphere application (ESG-ProSphere) and the three associated virtual machines (Discovery Appliance, Historical Database, and Storage Resource Manager). “Power on” was selected at the virtual application level to start the three virtual machines in the correct order.

Next, ESG Lab performed an upgrade of the ProSphere application using VMware Studio as shown in Figure 6.

In preparation for the ProSphere vApp upgrade, ESG Lab configured a CD/DVD drive to the VMs. This enabled access to the ISO formatted upgrade image. VMware Studio was then launched from a browser for each VM to conduct the upgrade.

Finally, ESG Lab connected to the ProSphere vApp using the browser-based GUI to validate login access and confirm proper operation. Figure 7 shows the Administration view from the ProSphere user interface. From this view, user account and security certificates can be added and managed. The Administration view also enables integration configuration for LDAP and Microsoft Active Directory.

Configuration and Discovery

Configuration is the process of setting up ProSphere to ingest data for performance analysis and storage environment alerting. Discovery is the process of identifying and ingesting performance and operating metrics as well as configuration data for the storage environment components or objects including hosts, switches, and storage arrays. With ProSphere, this process does not require an agent on each discrete object. It enables the administrator to create custom end-to-end views of the storage environment.

ESG Lab Testing

ESG Lab launched the ProSphere user interface and selected the Discovery icon to start the discovery configuration process. The first step in setting up each discovery job was defining its access credentials. Access credentials are a set of properties that define how ProSphere will connect to an object on the network.  These property sets contain permission settings for security and define the connection type used for data discovery.  Any access credential can be designated as global. Standard global access credentials can be defined and used for similar object types to streamline the discovery process by allowing a discovery job to automatically find and pick an appropriate, previously defined global credential.  This process separates the creation and management of access credentials from that of discovery jobs.  Server administrators can manage their credentials within the ProSphere environment while storage teams can then use those credentials to manage discovery jobs and meet security requirements for access management.

As shown in Figure 8, the Access Credentials tab was selected from within the discovery view. ESG Lab leveraged this configuration tab to set up discovery permission for a number of objects in the environment. Discovery access was defined for EMC storage arrays; Brocade and Cisco SAN switches; and Windows, UNIX, and VMware hosts.

Next, ESG Lab configured a number of different discovery jobs. As shown in Figure 9, ESG Lab configured a job to discover an EMC storage array. First, the scope of the job was set by selecting the object type and where on the network to look. In this case, the hostname of the server running the EMC Array SMI-S was selected, along with the object type of array.  Then the previously-defined access credentials were selected for the new discovery job. The last step was defining the discovery schedule.

It should be noted that a range of IP addresses can also be used for discovery to help simplify the process in large dynamic infrastructures such as rapidly growing virtual environments. This option also allows for excludes to be set for objects that a user does not want to include in discovery.

Figure 10 shows the results of a successful discovery job for host type objects. Here, ESG Lab selected the “Discovered Host” tab from the objects list view. Also available are switches, arrays, and fabric level object views.

Finally, ESG Lab configured groups of objects. As shown on the left of Figure 11, ESG Lab configured three groups (ESG Lab Windows Servers, ESG Lab ESX, and ESG Lab Physical Hosts). The groups were used to create more manageable views. Large environments could have hundreds or thousands of servers, making the objects list view difficult to use when searching for a specific object of interest. Groups can also be used to view objects with inter-dependencies such as a group of servers or the server, switch, and storage that support a specific application.

Three group classifications are available in ProSphere. The three groups are defined as system groups, simple groups, and smart groups. System groups are automatically created by the application to group objects by type. Simple groups can be created and used for a collection of objects that rarely change.  Objects must be manually added to simple groups.  Smart groups can be used to dynamically scan and group objects based on user-defined parameters. With smart groups, objects can be properly grouped for analysis based on business requirements such as mission-critical application or utilization for bill back purposes.

Monitoring, Alerting, Troubleshooting, and Analysis

Performance monitoring is the ability to select, view, and analyze detailed information for objects in the storage infrastructure. ProSphere can create end-to-end topology views and detailed performance charts. Alerting is the ability to monitor SAN and storage objects in the infrastructure by setting thresholds and notification. Combined, performance monitoring and alerting help accelerate the identification and remediation of performance bottlenecks and component failures.

ESG Lab Testing

ESG Lab explored the performance capabilities of ProSphere by selecting a virtual machine in the storage environment and analyzing the supporting components from the host through the storage. Figure 12 shows a topology view from the virtual machine on the left side of the figure all the way through the SAN in the middle to the storage hosting the VM datastore on the far right. The high level overview was created by selecting specific objects of interest from the host object list. ESG Lab used a drill-down process from this view to analyze detailed information of each component. Hosts, switches, and arrays in the topology view can be expanded into a detailed view by clicking on the desired icon.

In addition to simply selecting items from the object lists, ProSphere has sophisticated search capabilities. ESG Lab used the intelligent filtering feature of the search functionality to quickly and easily set views into different objects without having to scroll through the entire list to find a specific object.

Intelligent search functionality becomes more important as the environment grows organically or through the federation of multiple ProSphere sites. Geographically separated ProSphere implementations can be federated through synchronization, a process that creates a synchronized data set between sites. The search feature can retrieve data in a federated environment eliminating the need to log in to each site and enabling users in different locations to perform analysis and troubleshooting of remote environments.

Next, ESG Lab reviewed the performance metrics of the VM selected in the topology view. As shown in Figure 13, ESG was able to analyze and review resource utilization, response times, and performance characteristics for the virtual machine. Similar views are provided for physical hosts as well.

As shown in Figure 14, ESG Lab also monitored performance data on the EMC array containing the datastore for the selected VM. ESG Lab noted a number of storage performance metrics (e.g., response time, IO, and throughput) that can be used to quickly analyze the health of the supporting array.

Lastly, ESG Lab used the alerts tab to configure thresholds and then monitor storage and switch metrics in the test environment. As shown in Figure 15, utilization levels were set to flag warnings and critical conditions when a defined percent was exceeded. Alert levels can be set for the following metrics:

• Switch port link utilization
• Symmetrix port utilization
• Symmetrix host director utilization
• CLARiiON SP utilization
• CLARiiON SP % dirty pages

Setting up and monitoring thresholds allows for easy, at-a-glance identification of performance issues in the storage environment. A quick review of the alerts page will show if a switch or array component in the environments has exceeded defined parameters. ProShpere can then be used to view storage component mapping and identify and analyze physical hosts or virtual machines that might be impacted by changes in SAN performance.

ESG Lab Validation Highlights

• The installation process for ProSphere was quick, easy, and intuitive. ESG Lab downloaded the application images from EMC’s website, moved them to the VMware environment, and easily navigated through the OVF template install process, which took about 30 minutes.
• ESG Lab validated the efficiency of the agent-less discovery process driven by ProSphere. Without installing a single agent, ESG Lab was able to easily set up and discover over 100 physical and virtual hosts, four storage area networks, and six storage arrays. This was accomplished by leveraging SMI-S and SNMP for storage and switch objects and WMI, SSH, and vCenter connectivity for physical and virtual hosts.
• With ProSphere, ESG Lab confirmed that EMC can offer a comprehensive topology view from the host to the LUN on the supporting array, enabling end-to-end visibility of resources for the storage or system administrator.
• ESG Lab confirmed the ability to set up alerting for switch and storage components in the monitored environment. Thresholds can be set that notify the administrator when critical or warning conditions occur in the environment, allowing corrective action to be quickly taken to remediate the condition.

Issues to Consider

• It should be noted that currently, the historical database cannot be pruned. Best practice should dictate monitoring disk capacity for this application component. Also, to avoid corruption, the historical database requires an orderly shutdown. EMC recommends that ProSphere be started and stopped from the vApp level to maintain database consistency.
• Special attention should be paid to best practice recommendations when using VMware snapshots in federated ProSphere environments. If one site in a synchronized environment is rolled back to a snapshot, an error will be returned when users try to access discovered object detail. To avoid this, each site must be rolled back to a corresponding snapshot.
• Since EMC ProSphere was in its first GA release at the time of the validation, ESG expects the application to gain new features over time and hopes to see mixed object group views at the dashboard level.

The Bigger Truth

In an effort to respond more quickly to changing business needs, IT is transitioning to highly virtualized and cloud environments. To ensure a successful transition, storage administrators need to leverage management software that was specifically built for these new data center paradigms. This software should include comprehensive management of the networked storage environment and provide visibility into the virtual server environment, as these previously siloed technology domains are now more tightly integrated and interdependent.

ESG Lab validated that EMC ProSphere’s efficient architecture enables it to be deployed easily, accelerating the time to deliver value for organizations transitioning to cloud environments. It is able to provide end-to-end views and the performance information required to ensure applications will run in an optimized environment while meeting demanding SLAs.

ESG Lab believes EMC has made it easy for clients currently running ControlCenter to test and then migrate to ProSphere thanks to its ability to ingest ControlCenter performance data. EMC has also allowed ControlCenter clients with active maintenance contracts to deploy ProSphere at no cost.

As organizations evolve their highly virtualized infrastructures into scalable cloud solutions, they will need management solutions that can adapt to these dynamic and agile environments. By designing and architecting management software for the Cloud Computing Era, EMC enabled ProSphere to effectively manage and optimize cloud storage environments.

Appendix

[2] Source: ESG Research Report, The Evolution of Server Virtualization, November 2010.

[3] https://powerlink.emc.com

Why? Because in IT we LOVE to treat the symptom of our ills, never the cause.

We buy infrastructure to support business applications, right? Wrong. We buy infrastructure to support data. Data supports applications. Applications don’t care about infrastructure, they care about data. Users don’t care about infrastructure either, they care about data. No one, other than us of course, cares about infrastructure. Sad, but true.

We buy infrastructure to support data. Storage doesn’t support applications. It houses and delivers data to those applications. Servers do execute applications, but only once that application has been fed the data it requires.

Networks let us share the results of an application done chewing on its data–or act as a transport to get the requisite data from where it is, to where it is needed.

Virtual infrastructure is cool–but it’s not the real issue. It’s treating the symptom and not the cause of our IT issues.

If we had no data to worry about, we’d have no infrastructure issues to deal with. Virtual or physical, it wouldn’t matter.

The cause of all of our ills is data. Lots and lots of data. Lots and lots of the EXACT SAME data.

Get rid of the data, get rid of the headache. How? Virtualize it.

In simple terms, virtualization allows one physical thing to appear to be many logical things. This is true whether we’re talking about a server or a data set. The fundamental value proposition of virtualization is that ONE is easier to deal with than MANY. It’s as simple as that.

The concept of data virtualization is not new. Snapshots have been around for a long, long time. A snapshot is a virtual representation of a physical data set that can be used as if it were “real”–but takes up only a tiny fraction of the infrastructure, management, and footprint a real data set requires.

Data Domain made a zillion Canadian Loonies (note: neither the US Dollar nor the Euro seem stable enough for me to use any longer) by virtualizing ONE specific data set in your organization–the backup data set. Why? Because backup is 98% identical copies of the exact same data–over and over and over. Data Domain figured out that if they virtualized that data set, there was a double win. Win One was that you didn’t have to change any of your processes. You could keep doing backup the exact same way, fooling your backup system into thinking you made no changes. (Instead of treating the cause of the disease [stop backing up the same stuff over and over], they allowed you to treat the symptom–backup takes too long.) Win Two was that your backup window became manageable–and you didn’t need to keep buying tapes! Hooray!

The bottom line is that there are two types of data in our organizations: PRODUCTION data and COPIES of PRODUCTION data.

We take copies of production data, then take copies of those copies, and we use them in all of our other business functions within IT. We use them for protection (backup, DR). We use them for Business Intelligence (data warehousing, analytics). We use them for Business Sustaining (test/dev). We use them for Business Execution (CRM, Marketing). Blah, blah, blah.

Each one of those business functions has its OWN SILO of infrastructure (virtual or real) with its OWN storage housing its own copies of the exact same data!!! Each has its own processes, specialists, and outrageously redundant expenses. All in support of a different “application” using the exact same data. Smells wrong, doesn’t it?

I’m not saying that you shouldn’t perform these business functions. On the contrary, I’m saying you should enable yourself to do even MORE of these types of intelligent business functions. What I am suggesting is that you might consider the implications of treating your COPIED production data silos a bit more intelligently.

I’m not going to give you the answers here–as you are smarter than I am–but I want you to think about how you could gain efficiencies across the board if you used the concept of a “single system of record” for your copied data (I got that from Julie Lockner, database queen, and I love it). I’m not even talking about your production world–leave that alone. Just imagine what could happen if there were one single “master” repository for production copied data–that fed everything else–virtually.

You could have one way of snapping data, instead of 87. You could have one way of protecting data. One way of accessing data. One single data “silo” for infrastructure, and so on. The possibilities are endless.

Get yourselves out of the mode of treating the symptom and start thinking about how you would treat the cause–data.

Only in IT do we repeatedly treat the symptom, and I think it’s because the symptom is easy to see, and usually easier to “hide” (because fixing the symptom doesn’t really fix anything, it only helps us hide the problem).

True story. Last year, my 9 year old had a cough–for months. He couldn’t stop. We took him everywhere, to every specialist. No one could figure it out. At each step, they gave him stuff to try to get him to stop coughing. Medicines to relax his throat, to coat his throat, to control reflux, etc. They ALL treated the cough. Finally, we brought him to a local island witch doctor (she prefers to be called a holistic healer). Nancy, the witch doctor, did all these weird “eastern” medicine things, pulling at him, poking him, etc. Then, she took some potions in bottles (newt eyeballs I think) and had him hold them one at a time in his hand and try to hold his arm up while she pushed. After a few times, she literally said, ” He has an adrenal gland issue, probably stress-related, that is causing his immune system to not be able to combat this.” She gave us some potion (looked like sugar to me) and told us to give it to him once that night, once the next morning. Then, wait 24 hours.

I swear to god, he was cured. I’m talking 7 weeks of non-stop coughing. I’m talking about having to pull him out of school because he was distracting the other kids. Gone.

I’m fine with you thinking it’s a coincidence. I don’t care. (I’m a believer in all that goofy stuff now, by the way. No offense, but western medicine is what, 50 years old? There are a lot of Chinese that have been somehow surviving for thousands of years doing this stuff.) The bottom line is she looked for the cause and didn’t bother treating the symptom. And he stopped driving me nuts. Yay.

You can read Steve’s other blog entries at The Bigger Truth.

The new architecture is pretty impressive, especially when compared to its predecessor. ProSphere will be much easier to install, deploy, and get value from on day one–especially now that it uses agentless discovery and can load on a virtual machine! Other examples of its enhanced functionality include:

• A complete, end-to-end view of virtualized infrastructure. From VMs to storage, including innovative EMC storage technologies like virtual provisioning and fully automated storage tiering (FAST).
• Performance monitoring. ProSphere’s performance module is designed to monitor storage performance across the whole virtual infrastructure.
• Tight integration with other management and domain tools. ProSphere is tightly integrated with virtual management consoles like VMware vCenter to extract detailed VM information and it is also connected to storage array software.

With numerous announcements this week focused on cloud computing, it’s good to see EMC making an investment in solutions to help organizations better manage their storage and storage network in highly virtualized or cloud environments. This initial launch will be followed up with other releases later this year and into next to round out its functionality. For more details on the software, check out the whitepaper on EMC’s ProSphere site.

The good news for existing ECC customers is that if you are on a maintenance contract, the software will be sent to you for free. You should be able to install it yourself in less than an hour and it only takes about 10 or steps. Definitely worth the effort, especially if you are increasing your user of virtualization technologies and building out cloud storage environments to support the new cloud computing era.

Read more of Bob’s blog entries at Data Center Continuum.

However, the more I thought about it, the more I thought it was a pretty cool idea–especially if this discount will have a trickle-down effect on street pricing. Organizations constantly struggle with data management–in fact the number 2 IT, initiative for 2011 according to ESG Research was “managing data growth.” Couple that with IT organizations trying to keep down their operational costs and this offer should be really appealing to customers. IBM cites the declining cost of storage hardware (per TB–not total) as the impetus for this change. This move allows organizations to lower the cost of the overall solution–both hardware and software. It’s always good to be able to save a buck where you can.

So, good for IBM, and I look forward to seeing how this strategy plays out. If you are looking at buying a new solution, would 35% lower storage management software costs be enough to get you to change vendors? or at least make you consider changing vendors? At the very least IBM customers should be happy, and if you bought TPC in the last 30 days, I hope you saved your receipt!

You can read Bob’s other blog entries at Data Center Continuum.

• “Single-click” end-to-end provisioning for the Vblock. EMC claims this is five times faster and 80% more efficient than other methods. Ideally, this provisioning capability will be integrated with a self-service provisioning portal and service catalog.
• An operations module that will provide rapid problem identification and resolution across the entire Vblock (compute, network, and storage). As organizations deploy mission critical applications on these converged infrastructures, it will be critical to have the ability to quickly find and isolate faults. The operations module will provide that capability.
• Expanded support for unified storage arrays. UIM 2.1 will support the Vblock Series 300. The ability to provide options of scale, performance, and automation will be important as organizations of all sizes look to deploy solutions on the Vblock.

EMC Ionix UIM/Provisioning 2.1 is now generally available and UIM/Operations 2.1 is available as of May 13^th. Support for Vblock Series 300 will be available in June 2011.

UIM provides tremendous value in conjunction with Vblock for simplified management and provisioning of infrastructure services. But organizations also need to consider the implications of deploying converged infrastructure with unified management. Specifically organizations should also think about:

• The impact on people and process. With the UIM’s single-click provisioning, organizations will not require separate teams and processes for compute, network, and storage environments. At a high level, this should be viewed as a good thing. As these environments continue to grow at an astronomical pace, it will be difficult to scale teams; while technology specialists will still be required for some time to come, IT generalists who understand the business and all aspects of the environment will become more prevalent. Organizations should consider cross-training its IT staff for maximum efficiency.
• Cost vs. business process improvement and agility. Although IT is under constant pressure to reduce costs, there may come a time when improving business processes and delivering more responsive and flexible services will trump financial concerns. ESG research has shown that over the last three years, reducing operational expenses is still the number one justification for IT investments; however, the gap between it and the number two justification, business process improvement, is closing. In fact, we could see business process improvement claiming the top spot next year.

For more on ESG’s analysis of UIM 2.1, check out the full ESG Brief on this topic.

You can read Bob’s other blog entries at Data Center Continuum.

I didn’t see Dave’s keynote but I did hear a lot of post-presentation commentary. Much of the networking crowd was aghast. How could Donatelli present an Interop keynote devoid of any technical vision? How could he disrespect the networking king?

Yes, Dave did eschew vision but he talked about something just as important == the networking business. Cisco has always faced competition but not like EMC did in storage or IBM did with servers. With the acquisition of 3Com and EDS combined with its global distribution channel, HP is now ready to play the role of Avis to Cisco’s Hertz. The networking cognoscenti may not like Donatelli’s “in your face” style, but it will love when it sees a more competitive marketplace.

Aside from Donatelli’s keynote, HP did back up its statements with a few announcements that demonstrate its seriousness in the marketplace. For example, HP announced a number of high-end boxes for campus networks (HP calls these FlexCampus as part of its FlexNetwork strategy). Why the campus? Because everyone is ignoring the campus in favor of the data center, allowing HP to sneak in the enterprise network back door. Furthermore, HP announced new network management software as well. This is significant because network management has always been a weakness in Cisco’s armor.

We love to come to Interop and be dazzled by the latest networking speeds-and-feeds, and this year’s event is as good as any. Tour the show floor and you can see great demos of OpenFlow, IPv6, SPB, and 802.11N. Nevertheless, we have to remember that all of this science exists for one reason–so vendors can sell equipment and make money. Dave Donatelli is not a networking guy and his “new sheriff in town” speech certainly ruffled some packet-based feathers. That said, HP’s intent is to shake up the networking business. If it can accomplish this goal, it will be good news for networking innovation and competition henceforth.

You can read Jon’s other blog entries at Insecure About Security.

Their diverse environments provided attendees with two different views and routes for overcoming challenges. More importantly, though, it showed that regardless of your server virtualization technology and choice of management tools, it was possible to build out and effectively manage a highly virtualized environment.

Read more of Bob’s blog entries at Data Center Continuum.