Cost effective methods of test environment management. Prabhu Meruga Director - Solution Engineering 16 th July SCQAA Irvine, CA

Transcription

1 Cost effective methods of test environment management Prabhu Meruga Director - Solution Engineering 16 th July SCQAA Irvine, CA 2013

2 Agenda Basic complexity Dynamic needs for test environments Traditional test environment challenges Cost impact of not having dedicated test environments Solution Leverage Cloud based test environments Who is using it? Who provides such infrastructure? Types of cloud infrastructure Is the cloud infrastructure secure Cost comparison of On premise Vs Cloud based test environments Steps to move into Cloud What kind of testing can be done on Cloud? Case Study Illustrative shift of test environment landscape Before and after cloud what is the change in testing process? Benefits Summary 2

3 Basic complexity!! Enterprise Information systems are increasingly becoming dynamic, complex, distributed and component-based. In-house test environments are proving to be increasingly expensive and ineffective when handling these complexities With consumer base accessing the end user applications in different browsers, devices and operating systems, certifying and testing the applications in similar production like environments needs huge investments 3

4 How do I relate to this? Let s review Conventional Application development scenario in enterprises Multiple applications in enterprise with parallel releases, various development life cycles Need for end to end testing of all the applications Interim releases, technology upgrades 2013 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Application 1 (Agile) Release 1 Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Regre ssion Release 2 Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Dev Test AT PT Regre ssion Application 2 (waterfall) Release 1 Dev Test AT PT Regression Effort required to setup so many test environments Utilization levels of each test environment Maintenance of test environments Automation Testing Performance Testing 4

5 Traditional test environment - Challenges Test environments end up being different than production environments in terms of Operating Systems, Patch Level synchronization, Software Versions, Application Configuration, Multiple devices (in case of Mobile testing). Challenges Lack of Test s management process Loosely coupled Asset control mechanisms Lack of change control mechanisms Increase demands of effort, process and cost Test environments are seen as expensive and providing little real business value. Impact Most test servers run at less than 10 percent utilization. Approximately 30 % - 58%percent of defects are caused by wrongly configured test environments. Testing backlog is often very long and the single largest factor in the delay of new application deployments. Compromising steps Testing executed in shared environments Partially functioning test environments Production Issues Increased rework costs 5

6 Cost Impact due to lack of dedicated testing infrastructure On one side the cost of procuring hardware for enterprises is increasing while the other side it provides an opportunity for optimization. Of the overall hardware cost 40% is attributed to setup testing infrastructure Industry Cost of Inadequate testing infrastructure Aerospace Bn USD 54.5 Bn USD Automotive 1.7 Bn USD 377 Bn USD Potential Cost reduction from infrastructure improvements Financial Services 3.3 Bn USD 1.5 Bn USD Statistics Source : National Institute of Standards and Technology; Infrastructure here refers to servers, storage and testing tools 6

7 Solution Leverage cloud based test environments 7

8 Solution Leverage cloud based test environment (IaaS) Cloud-based testing environment This refers to use a selected cloud infrastructure (or platform) as a base to form a test bed equipped with diverse and scalable computing resources, system infrastructures, and licensed tools, which are allocated using autoprovision based on static/dynamic requests. Both virtual and physical computing resources can be included and deployed inside. Who is using it in the market? Few real time applications working on cloud..and many more How is the shift happening in the industry for cloud adoption? 8

9 Who provides the cloud infrastructure? 9

10 Types of cloud infrastructure Definition Public Private Hybrid When the services are rendered over a network that is open for public use When the services and infrastructure are maintained on a private network Benefits Basic Average High Risk High Medium Low Includes a variety of public and private options with multiple providers Cost Pay as you go Pay as you go Pay as you go Security High (needs to be managed) High High Scalability Best Fair Good Maintenance Easy Medium complexity High complexity Effort required to setup What can be setup In few hours 2-4 business days 5-7 business days Typical Usage E-Commerce, Payment gateways, Intranet and Secured applications 10

11 Is infrastructure on cloud secure? Security Challenges Security mitigation Access Control and Identity management Ongoing compliance concerns Co-mingling of other customers data Security Standards & Certification Compliance Secure Access Hardware Security Built-in firewalls VPN option Unique users 11

12 Cost comparison hypothetical analysis On Premise infrastructure Cloud based infrastructure Source : bluemountainlabs 12

13 CO$T Making the business case Physical test Infrastructure Cloud based test infrastructure Initial procurement cost IT Support staff cost Infrastructure maintenance cost 13

14 Steps to move into cloud based test infra Step 1 Kick off pilot cloud involving non critical pieces staging/it test environments Start with shifting of storage servers Move low critical applications from IT Step 2 Focus on moving the entire test infrastructure Pilot with few mission critical applications Setup a Virtual private cloud and assess the security Step 3 Evolve into running the infrastructure as a service Focus on service and cost part of it 14

15 Where can cloud based test infrastructure help? Functional testing Regular Manual Testing Regression Testing Web Services Testing Product Testing Cross browser testing Non Functional testing Load Testing Stress Testing Endurance Testing Vulnerability Testing Penetration Testing Functional Automation testing Custom framework setup Industry standard tools Automated regression testing Mobile Applications testing 15

16 Case study : Cloud based performance testing Cloud based Framework for High Load Performance Testing, Functional Testing, Test Management, Vulnerability Scanning & App monitoring

17 Eg: Leverage cloud based infrastructure for load generation in performance testing Load Generation over the cloud 17

18 Illustrative test environment landscape shift Coding Testing UAT Prod Development Unit Testing System Testing Automation Performance UAT Prod Shift to Cloud based test Infrastructure Coding Testing UAT Prod Development Unit Testing System Testing Automation Performance UAT Prod Cloud Management Console leverage cloud infrastructure for test environments Tightly coupled environment instances 18

19 Before and after cloud environment. Let s talk Setup 1 time activity. Typical duration (few days to a week based on the complexity) Select the appropriate partner for infrastructure setup. (Amazon, Google or Microsoft) Wrap up and apply the security policies as applicable for your Organization Cloud Management console logins for restricted user access (development, QA and Configuration teams) Deployment process Iterative!! As it was earlier Plan the deployments when needed No restriction on the number of deployments Usability the way environments are used Remains same u will find no change in the way environment is used You can use all the commands that you use in the physical environment You can deploy all the licenses for your custom package software Utilization and Cost Pay as you go cost model. Cost effective especially in the case of performance testing where the execution is done when needed. Multiple load generators can be setup run time as needed No worries about re setup as the same is available with a minimal cost can be rebuilt very swiftly since the components are known Share or shift the environments very easily with other priority projects within the enterprise 19

20 Benefits summary Scalable and on-demand cloud environments Enterprise IT collaboration Advanced networks Total Cost of ownership Accelerated testing cycles Create multiple test environment instances on-demand Create isolated test environments for a broad range of OS, database, browser, and application combinations Run them concurrently to create parallel work streams Capture the entire state (memory, network settings, and disk) of a multi-machine configuration and save it as a template Perform destructive tests, install intrusive tools, and get back to a "clean slate" easily Enable collaboration for multiple product and release teams Create product and release specific projects and assign groups of users and resources Enable easy collaboration including secure, limited access for contract or off-shore test teams Share bug snapshots with remote development and testing teams Recreate customer use cases with virtual data centers Create custom networks, routing and security policies Replicate customer use cases including clustering and fail-over configurations, N-tier secure configurations, and shared resource configurations in the cloud Pay-for-what-you-use model Eliminates vendor commitment overheads Accelerates 'go to market by minimizing overall lead time in infrastructure provisioning Shifts spending from Capex to Opex Ensures security and standards compliance Provisions hardware only for the required duration 20

21 Thank You! 2013 CSS Corp The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.

22 Appendix 2013 CSS Corp 22

23 Define Transform Manage Evolution of cloud based infrastructure maturity Standardize: roll out standard environment process for synchronization, new setup, sharing and deployment of applications Continuous Improvement: Review the maturity and expand the infrastructure for development and production needs as well Dedicated resources : Assign dedicated environment personnel Pilot implementation : Kick off test execution and collect the metrics Assessment : Review the ease of deployments, maintenance and security of applications on the cloud infrastructure Define the objective End to End testing, functional automation, Performance testing, Mobile testing Identify the Infrastructure requirements Storage Capacity, frequency of use, Service providers Security Policies : Identify the security, regulation challenges with the internal IT risk and security team 23

24 24

25 25

26 26

27 References : IBM Amazon Microsoft Bluemountainlabs National Institute of standards and technology (USA) 2013 CSS Corp 27