Storage Systems Performance Testing
Client Overview Our client is one of the world s leading providers of mid-range and high-end storage systems, servers, software and services. Our client applications include end-to-end virtualization, massive consolidation, data replication and archival solutions which are widely used across industries including Telecom, Aviation and Banking among others. Our client s solution allows companies to scale up and scale down data storage capacities thereby ensuring dynamicity as required by businesses. Our client s products, services and solutions are employed by the world s leading enterprises, including more than 70 percent of the Fortune 100 and more than 80 percent of the Fortune Global 100. Engagement situation & challenges Our client s product portfolio was expanding with continuous addition of new versions of the storage class products. Each product had multiple yearly releases introducing new and improved features/ functionality. For storage products, performance is the most important aspect, which influences the purchasing decision. Additionally data integrity, reliability and availability became the important features of the storage systems for ensuring disaster recovery and business continuity planning. In our client s case, all these improvements and dynamic nature of releases round the year have led to the critical requirement of best practices and performance benchmarks being made available to the end customers to enable optimal use of the storage products. This has mandated intensive testing of the storage products by qualified test engineers, who would be responsible for churning out performance papers for different configurations of each product release. These performance papers would also be sent across to various product councils and statutory performance bodies for product certification as a quality product adhering to industry standards.
Our client was facing a challenge to meet the aggressive product release schedules and wanted to engage expert resources to produce performance test papers without compromising the quality. Our client was looking for a partner who had prior experience with storage arrays like SAN and NAS and who also has a lab setup with the necessary infrastructure to support these highly sophisticated storage products. Thereby the requirement was to build a technically competent engineering team which could plan and execute various activities of this technical operation and was also capable of creating performance briefs adhering to industry standards. Our client wanted to establish a versatile team which can do a complete end-toend testing of their various mid-range, high-range and storage sub-systems while simultaneously gaining cost advantage from this engagement. Key engagement challenges The key challenges in this engagement were: Performing intensive tests of storage sub-systems to determine the maximum throughput / IOPS which forms the baseline input for any application level testing. This testing is very critical as this is the first level of testing for any storage sub-system. Determining the right combination/design for each storage sub-system which includes different types of disk drives like SATA, SAS, etc was a major challenge as there are more than 100 different combinations to choose from. Analyzing and testing the product to ensure data integrity and availability, which are the prime requirements of any data storage product to determine the reliability of storage devices. Testing of enterprise storages which are peta-bytes in capacity is another key challenge. Because it takes more than a week time to prepare the test bed for exchange testing on enterprise class storage with SATA 7200K drives.
Xoriant s Solution Xoriant s Storage and Computing Practice has domain experts with extensive capabilities, expertise and experience addressing the needs of performance, interoperability, automation and support for a host of clients. This made Xoriant a partner of choice for providing complete end to end solution for our client ranging from infrastructure requirements (lab setup: storage, server and switch) to technical operations to white paper creation. systems technologies as well as Xoriant team had a series of discussions with the client product architects and engineers for an in-depth understanding of the product suite and created papers documenting product knowledge which necessary infrastructure and a remote VPN connection to connect to the client labs in the Silicon Valley using high-speed Internet.
Each application has diverse storage product needs, where some applications require endless hours of constant data acquisition, while others experience peak bursts of small block I/O. The best storage device for one application is almost never the right storage device for another. This necessitated exhaustive tests to be performed by the Xoriant team. Our team prepared extensive test plans comprising storage design, dynamic provisioning configuration, RAID configuration, LUN creation, zoning and server configuration for each of the products in the suite. Different benchmark tools were analyzed to determine right tools for testing depending on the application. For any exchange application testing we decided to use Jetstress and for testing NAS appliances we used Ozone and SPECSFS. Metrics were defined and measured for each of the storage products. The following are some of the key variables recorded over the duration of a peak processing period or test run: Processor: Percentage processor time Physical Disk: average disk queue (total, read and write by disk/storage array) Physical Disk: disk bytes/seconds (total, read and write by storage array) Performance testing configurations Operating Systems Windows 2000, Windows 2008 Linux, Solaris, AIX, HP-UX Servers Switches Storage IBM, HP, Sun, Intel Platform Brocade, Qlogic & Cisco NAS platform based SAN Advanced Modular Subsystems Universal Storage Platforms
After rounds of various kinds of tests, papers were written encompassing the test results, best practices and solution for respective product which were published on various websites for general view. Additionally these papers were pushed to statutory bodies for expert views. These papers are now used by our client s customers to determine the best product fit for their application requirements. For instance, a white paper for Exchange Solution Review Program enables our client s customers to decide which storage fits the best for their organization for deploying Microsoft Exchange. Conceptual diagram
Xoriant key contributions Obtained the product performance measures by executing intensive tests on storage products by modifying configuration file of the testing tool (as required) to measure key performance metrics like IOPS (Input/Output Operations per Second), CPU busy, read/write latency, response time, etc Designed storage systems with permutations and combinations of different types of disk drives (SSD, SATA, FC and SAS) to test the compatibility of the storage products. Additionally the system was designed so that the hardware components and the CPU stays busy throughout the test run to test system s performance threshold.
Performed drive comparison (72GB FC, 146GB FC, 300GB FC, 450GB SAS, 500GB SATA, 750GB SATA) tests, RAID level backend comparison (2+2, 3+1, 4+4, 7+1) tests, etc. to determine the versatility of any storage array with different types of disk drives Utilized sophisticated and most updated benchmark tools for exhaustive testing of different applications of the system, e.g. ORION for Oracle, Jetstress for Microsoft Exchange, etc. Used data patterns that challenge components to test for data corruption and data redundancy, ensuring data integrity and availability. Employed static as well as dynamic provisioning and other virtualizations techniques to test enterprise class of storage which enables using the entire capacity of the storage array more efficiently. Performed benchmark scalability test (with RAID 10, RAID 5, and RAID 6), frontend scalability test to verify the full capacity of any storage array. Tools and Technologies Jetstress 2.1/3.0 Iozone Vdbench Orion IOMeter SPEC SFS97/2008 Fileop NetBench Key Benefits Enabled the client engineering team to focus on new product releases by offloading non-core activities like performance testing and white paper creation to the xoriant team Achieved a cost advantage to the tune of 30% due to offshore model of project execution Enabled 24x7 utilization of client s lab infrastructure as the offshore team operated during client s non-operative hours through remote connectivity
Contributed in the sales cycle by analyzing and fine tuning client s product performance and determining if the product configurations are equipped to handle customer workloads and throughput goals Provided the most important marketing push for client s various products by publishing more than 30 white papers for each application for both enterprise and modular class products in the first year of the engagement. Facilitated delivery of several more certified configurations for our client in given time as compared to competition Client Testimonial The Xoriant team has done exceedingly well beyond anyone s expectations here by learning not only our product line quickly, but also assimilating our customer configurations, as well as specialized tools to come up with optimized tests, certifications and targeted white papers, to assist our sales/marketing teams in specific customer situations. Due to the Indian time zone, the Xoriant teams were able to help us operate our labs around the clock, thus delivering lab results much faster than we have ever done. Hats off to the team effort!! Senior VP Technical Operations