Technical Operations Enterprise Lab for a Large Storage Management Client

Transcription

1 Technical Operations Enterprise Lab for a Large Storage Management Client

2 Background Xoriant s client is a large global storage management company with development centers, laboratories and sales offices all over the world. Our client s customer-facing technical operations units help the sales effort as well as customer implementation efforts for their storage management products. With the continuing proliferation of storage area network (SAN) and network attached storage (NAS) on the one hand; server and virtualization technologies on the other hand; combined with other storage-oriented software products such as databases, systems, enterprise class business solutions and document/content management solutions, our client was facing an increasing demand to conceive, plan, execute, test and tune numerous configurations of hardware and software products along with their storage management offerings. Instead of incurring the overhead of hiring experts of different skills who are not likely to be busy all the time, our client was looking for a partner who would help plan and execute various activities of this technical operations lab by creating an offsite facility and bringing in the right expertise for the appropriate requirements. In order to leverage offshore cost benefits while retaining the flexibility to serve global clients, our client was looking for dual-shore solutions provided by a vendor who not only had the technical and domain expertise, but who could also provide near 24x7 support from the US as well as India (in a totally orthogonal time zone). Xoriant was chosen as the vendor to help out in this activity. Environment Description Our client s Technical Operations organization oversees the Enterprise Lab functions to ensure that their storage is compatible with a wide range of heterogeneous products and consistently reflects the client s high standards for quality. As an integral component of our client s storage development process, Technical Operations function manages Enterprise Lab sites as they carry out essential, quality-focused activities, such as: Interoperability verification Performance management and testing Software acceptance Vendor qualification and testing Proof-of-concept and enterprise solutions demonstrations

3 The Enterprise Lab resources include multiple equipment inventories to support these activities, comprising configurations from our client, and from our client s key partners. In addition, these activities involve varied skill sets ranging from storage hardware, firmware, and storage management configurations to match the infrastructure and application software. Xoriant has been working with the US location of this client from After establishing our partnership in the US operations, we established a dedicated center in India for our Client in Currently Xoriant engineers work with several of Client s location managers in the following areas: Interoperability verification Performance management and testing Test automation Proof of concept demos for pre-sales situations. Xoriant Participation 1. Interoperability Lab The interoperability lab (ILAB) performs testing of all SAN and NAS devices with the latest hardware and software configurations. In interoperability testing, each combination of equipment is characterized by a 5 Tuple. In a given combination, the host runs one of the OS as specified in the OS block. There could be one of the MultiPath software running on the host with one host bus adapter (HBA) installed. The host is connected to a fiber channel (FC) switch. The FC switch is in turn connected to the storage.

4 HOST os Solaris 8/9/10 Windows 2K SP 4/3/2/1, Windows 2003 (31 bit, 64 bit) Linux: RH AS HBA s QLogic - all brands Emulex - all brands SUN HBA's - all brands Multipath Software VxVM MPx10 HDLM Device Manager FC Switches McData: 4500/6064/6140/i10K/4400/4700 Cisco:9509/9216/9140/9513 Brocade:3800/3850/3900/12K/24K/48K QLogic:5400/5600 Various Storage Devices

5 Xoriant engineers certify various specified configurations. This interoperability certification is performed in the following steps. The team of SAN engineers uses remote VPN connection to connect to the client labs in the Silicon Valley, CA using high-speed Internet. Once connected to the lab, the team picks up a specific configuration to be tested for interoperability from the database. The team then reserves the equipment needed for the test using the web based GUI provided by our client. The team then studies the configuration by analyzing various components and interconnects them using optical patch panels and the web based GUI of the optical patch panels. Once the setup of the equipment is completed, standard interoperability tests are carried out. The results are verified and if the tests are successful then the client database is updated with corresponding entry as a certified configuration. If the tests fail then the configuration is further investigated, troubleshooting is carried out and the whole cycle is repeated until the configuration is certified.

6 Customer's Lab Harware Environment Power Terminal Windows based Computers Linux based Computers Internet Firewall / VPN Router Lab Local Area Network 2. Test Automation Automation team members have been involved in developing components for automating the integration test phases for a variety of data storage configurations. The team is responsible for developing test automation scripts that can automate the manual test process followed for Interoperability Lab testing, thereby reducing the time taken for testing/certification and improving the efficiency.

7 The work was performed Perl and Shell Scripts, using the storage domain concepts. 3. Performance Measurement Lab The Performance Laboratory serves as an important link in our client s Technical Operations Support group for performance measurement and problem analysis. Our performance team s goal is to analyze and measure the performance of all client products, both stand alone and in specific configurations provided by the presales teams involved in various sales cycles. In a typical performance measurement cycle, the presales engineers typically determine to what extent client products are expected to perform and whether they are equipped to handle customer workloads and throughput goals. Many times, our team has had to consult with various people in our client s global organization to understand their individual experiences in working under similar circumstances. In addition to these consulting sessions, the performance team had to perform its own study and analysis or the environment. The following testing steps were taken by the performance team: Understanding the architecture of the subsystem whose performance is to be analyzed. Determining the capacity of storage products by performing standard test process. Fine-tuning the subsystems to get better throughput. Collecting raw performance data and analyzing and interpreting it. Escalating unresolved problems to the development team the last resort

8 Performance Testing Configurations Operating Systems Windows 2000, Windows 2008 Linux, Solaris, AIX, HP-UX Servers Switches Storage IBM, HP, Sun, Intel Platform Brocade, Qlogic & Cisco NAS platform based SAN Advanced Modular Subsystems Universal Storage Platforms Tools used for Performance Testing File system benchmark tool Iozone I/O measurement and characterization tool Iometer Microsoft exchange server performance and stability testing tool Jetstress Disk and tape I/O workload generator and performance reporter Vdbench Storage I/O performance measuring utility SQLIO Performance data interpretation graph sample The graph below shows the transaction per second, RT values observed for 8 kb, 100% Random Read operation. Queue depth varies from 64 to 512 Only one LUN is carved out from One Raid Group 4 numbers of LUNs used Number of outstanding I/O scales from 32 to 512 Test was done using IO meter

9 It can be seen that response time is increasing proportionately when application outstanding I/O increase KB Block, 100% Random Read, 4 LUN 8KB Block, 100% Random Read, 4 LUN # of Outstanding I/Os # of Outstanding I/Os Transactions per second is limited by either Application OutstandingIO Queue or HBA QD value, Which ever is lower. Thus both of these acts as limiting factor towards achieving better throughput. 4. Solutions Demo Team Our Solution Demo team supported the SAN marketing group. The approach in providing the solution demo was as follows: During this engagement, the Flash demo development team in India typically prepares multimedia presentations and demos to highlight various features of our client products for pre-sales situations. The development is done with the help of Flash, and other software packages such as LightWave 3D. The creative team interacts with the client demo team through STARboard and video conferencing.

10 Typically, these demos are developed and tested using the demo lab at our client location in the Silicon Valley. Certain hardware specific features and functionalities relevant to the targeted prospect/customer of our client are showcased in these demos. In some cases, remote demonstrations are also carried out for certain customer specific requirements. New Hardware/software Release or Upgrade Requirement from Product Manager for a Demo Client Requirement Gathering and Training Raise a Request for Change Solution Demo Team Build the Demo Knowledge Transwer to Team Members

11 Benefits to our client A well-planned approach to effective partnering With storage management has helped our client reap substantial long term business on the upswing, technical benefits: the operations labs were becoming Ability to deliver several more certified a limiting factor in our sales configurations in given time as compared to competition, with reduced overall cost of success. With timely and flexible operation assistance from Xoriant, we were Clearing of all Interoperability Lab pending able to significantly backlog test increase requests and get current on sales/customer requests. our technical sales support Ability to enhance Interoperability Lab s scope bandwidth and respond to many of operation with availability of additional manpower doing innovative projects more customer situations, thus Ability to measure and tune the performance supporting our sales force in of standalone client products as well as that of sales increasingly situation specific complex sales configuration, thus helping the sales cycles. situations. This model is going Round the clock operation helps faster to be a precursor to our future turnaround for Proof of Concept Demos for our client s prospects/customers. collaborative sales support Ability for the sales teams to show quick and efforts. Senior VP Technical sales situation specific demos to their Operations. prospects/customers, substantially increasing the chances of success.