ESG Lab experts spend an enormous amount of time and effort validating technology and with the onslaught server virtualization they have really rolled up their sleeves creating a storage benchmark methodology. Brian Garrett, Vice President of ESG Lab, goes through the methodology in a recent Lab Validation, IBM System Storage DS5020/DS3950 Express and IBM BladeCenter HS22: Real-World Mixed Workload Performance in VMware Environments.
Here is a snippet from the Lab Validation that walks through the methodology:
A Mixed Real-world Storage Benchmark Methodology
While VMmark is well suited for understanding the performance of a mix of applications running on a single server, it was not designed to assess what happens when a mix of applications is run on multiple servers sharing a single storage system. VMmark tends to stress server internals more than it does the storage system. The methodology presented in the balance of this report was designed to stress the storage system more than the servers. Taking a cue from the VMmark methodology, a tile-based concept was used. As shown in Figure 5, each tile is composed of a mixture of four application workloads. Two physical servers, each configured with eight virtual machines, were used to measure performance as the number of active tiles was increased from one to four.
Figure 5. ESG Lab Tile-Based Storage Benchmarking
The difference between the server-focused VMmark benchmarking and storage-focused ESG Lab benchmarking is shown in Figure 6. Note how VMmark testing is performed with a single server, often attached to multiple storage systems. As a matter of fact, the IBM BladeCenter HS22 VMmark results presented earlier in this report were achieved with a pair of IBM System Storage DS4700 arrays. In other words, when vendors publish VMmark results, they make sure there is plenty of storage available so they can record the highest VMmark server score. This provides IT managers with a fair comparison of the performance capabilities of competitive server technologies.
ESG Lab storage-focused benchmarking uses a different approach. Instead of testing with a single server and more than enough storage, multiple servers are attached to a single storage system. Rather than running application level benchmarks which stress the CPU and memory of the server, lower level industry standard benchmarks are used with a goal of measuring the maximum mixed workload capabilities of a single storage system.
Figure 6. Server-focused VMmark vs. Storage-focused ESG Lab Benchmarking
Mixed Workloads
Industry standard benchmarks were used to emulate the IO activity of four common business application workloads:
- E-Mail: The Microsoft Jetstress utility was used to generate e-mail traffic. Similar to the Microsoft LoadSimm utility used in the VMmark benchmark, Jetstress simulates the activity of typical Microsoft Exchange users as they send and read e-mails, make appointments, and manage to-do lists. The Jetstress utility is however a more light-weight utility than LoadSimm. Using the underlying Jet Engine database, Jetstress was designed to focus on storage performance.
- Database: The Orion utility from Oracle was used to generate database traffic. Much like Jetstress, Orion is a lightweight tool that is ideally suited for measuring storage performance. Orion was designed to help administrators understand the performance capabilities of a storage system, either to uncover performance issues or to size a new database installation without having to create and run an Oracle database. Orion is typically used to measure two types of database activity: response-time sensitive online transaction processing (OLTP) and bandwidth sensitive online analytic processing (OLAP).
- Web Server: The industry standard Iometer utility was used to generate web server traffic. The IO definition was composed of random reads of various block sizes. The web server Iometer profile used for this test was originally distributed by Intel, the author of Iometer. Iometer has since become an open source project. Iometer tests were performed on Windows physical drives running over VMware raw mapped devices.
- Scan/read: The Iometer utility was used to generate a single stream of read traffic. Operations that tend to generate this type of large block sequential traffic include scan and index operations, long running data base queries, backup operations, bulk data uploads, and copies. One 256 KB sequential read workload was included in each tile to add a throughput intensive component to the predominantly random IO profile of interactive e-mail, database, and web server applications. As most experienced database and storage administrators have learned, a throughput intensive burst in IO traffic can drag down the performance for interactive applications, causing performance problems for end-users. Adding a few streams of throughput intensive scan/read traffic was used to determine whether interactive performance would remain predictably responsive as the amount of mixed IO utilization increased.
Each of the four workloads ran in parallel, with the Jetstress e-mail test taking the longest to complete (approximately three hours).
Read more of Mark’s blog entries at Liquefying IT.
Share
