Guest Presentation Test Environment Management

Transcription

1 Guest Presentation Test Environment Management The Balance of Control & Agility By: Steven Iannuzzelli 1 Copyright ANZTB

2 Agenda Our Services and Scope Why Test Environment Management CBA Context Environment Management Challenges Environment Management Frameworks Reporting & Insights to Drive Continuous Improvement The Results

3 Enterprise Testing and Assurance.our customers and services EML / CommSec Retail Banking Institutional and Global Markets Finance, HR, Risk, Marketing, Business and Corporate Banking Funds Mgt and Insurance Enterprise Services (IT, BackOffice) Project Engagements active testing projects Test Management Services +750 Test Professionals Performance Testing / Test Automation / Tools Support Enterprise Testing and Assurance Demand and Resource Management Process Mgt, Measurement & Continuous Improvement Environment & Release Management Services

4 Why Test Environment Management? Significant enterprise demand and reliance on integrated test environments Up to % of testing time is often lost in environment related issues Significant improvements can be achieved by implementing basic ITIL practices to improve availability as test environment processes are often informal compared to production A dedicated Test Environment Management capability should be considered as a means for implementing process improvements and taking accountability to increase availability and maintain production-likeness of your test environments. The effective management and accountability for process improvement benefits realisation can result in large savings for the project portfolio

5 CBA Context Our integrated test environments that must support concurrent projects Four identical SIT/E2E functional test environments are used simultaneously by up to 500 testers. Managed Funds & GAM Vendor A No. Of applications ~9 MIS/ Statements/ Decision Systems Vendor A No. of applications ~20 ERP Vendor B No. of applications ~28 Core Financial Serv (Wintel/Unix) Vendor A No. of applications ~62 Payments Vendor B No of application ~52 Core Wealth Management Vendor C No. of applications ~43 Strategic In-house No. of applications ~20 Integrated T Environments # Application Instances T0 Prod Support 216 T1 Failover (NetBank) 108 T2 SIT/E2E 216 T3 PerfTest 137 T4 SIT/E2E 216 T5 SIT/E2E 216 T7 CoreMigration 24 Total # Application Instances 1133 Imaging & Workflow Vendor C No. of application ~14 Internet/Intranet Vendor C No. of application ~30 RCU / Cards Bureau Vendor D Commercial lending Vendor A No. of applications ~29 Business Inteligence Vendor A No. of applications ~ 101 Core Financial Serv Mainframe Vendor C No. of applications ~22 Market Systems Vendor B No of application ~93

6 Test Environment Resources - Landscape The diagram illustrates the technology stacks which supports the applications that reside within the core integrated test environments. Test environment management involves forward planning and incident management of all components of the stack. Development / Testing Environment Technology Stack 6+ Vendors 6+ Vendors Over 350+ apps split between multiple vendors DB2 Oracle SQL Server 5+ Vendors ZOS Windows Unix 5+ Vendors 1 Core, & 5+ Supporting Vendors Mainframe Midrange Wintel

7 Test Environment Management Challenges Test Environment Availability is impacted by a number of factors, namely code deployment issues, configuration errors, server and network downtime. Change controls need to balance speedy delivery, with environment stability. Planning and stakeholder expectations need to reflect realistic availability levels We can t expect same performance as production. Key challenges Sufficient Environments Increasing project demand Engagement and expectation management Environment Provisioning & Evolution Introduction of new applications across multiple integrated test environments Maintenance and upgrades of existing applications Cost of provisioning and environment management Environment Stability Agility and speed Vs. Control and stability Complexity of applying code to multiple environments Enterprise Services Service Development ET&A - Test Environment Services

8 Enterprise Testing and Assurance Strategic Framework Create the Finest customer experience delivered by the best people and services underpinned by superior processes and agile technology. Giving our customers what they want, when they need it. Providing services which are flexible and responsive to our customers needs. Transforming our organisation with an engaged, diverse team focussed on customers and continual development. Achieving better outcomes for customers through increased standardisation, reuse and cost optimisation. Ensuring a consistent experience (common process, templates, estimates) Integrated Process (SDLC/CMDB) Consistent metrics and Tailored processes to meet customer needs Risk based decision making On demand services Knowledge reuse Diverse teams Build a Quality Culture Embrace organisational change management Customer centricity Leadership development Develop trusting Continued process improvement Engagement efficiency Building process excellence capability Maximise reuse Optimized resource reporting Applying PE Principles relationships utilisation (Living Zero, getting it right first time)

9 Use standard ITIL practices to define your environment management framework. Input Application Development Team (Code, Application Environment Configuration, Test Data) Process Demand Management Request for support for Environment Test data, Configuration parameters Change Management Request for Enhancement or IT fix change Risk and Impact Assessment 9 Release Management Request for deployment of changes Fixes, Enhancements and Maintenance Build - System Administration (Upgrade/Maintenance) and Server Backup Automation of Test Environment Management Tool Management Governance Team Structure (Roles and Responsibilities) Forums and Committees Data Management Tools Management Metrics Environment Setup Knowledge Management Status Reporting Output Consistent and Accurate Test Environment Data synchronization with production environment Early defect detection - Production failure scenarios detected at Testing External failure cost reduction Lesser post-production defects Reduction in test environment downtime reduction in release cycle time

10 Define and document your services, processes, engagement model, metrics, SLA s and produce a service catalogue 1 Organization Applications Inventory Resources Inventory (including Skills and Team structure) Database Inventory 2 Process Process Flow and handoffs with other teams(application Development and Support) Nature & type of Services Rendered and Activities Performed Historical data of requests for Test Environment Change and Support 3 Automation Tools Inventory Process flows that have been automated Integration of Test Environment elements and Interface with other entities 4 Metrics Metrics currently tracked for measuring performance effectiveness Reporting of metrics Customer/Stakeholder details and frequency of reporting Key Issues and Challenges

11

12 Reporting & Insights to Drive Continuous Improvement No. of Env, Data & Deployment defects, per 100 execution Environment, Data & Deployment defects by External Vendors Jan Feb Mar Apr May June Defects / 100 test cases Jan Feb Mar Apr May Jun Gen-i Vendor A HCL Vendor C HP Vendor E IBM Vendor B SAP Vendor D TCS Vendor F 300 Release Raised Defects by Defect type Environment, Data & Deployment defects by CBA Internal Teams Jan Feb Mar Apr May June Data Environment Deployment Configuration 0 Jan Feb Mar Apr May Jun CBA - DBA CBA - Deployment CBA - Development CBA- IFW CBA - Operations CBA - Testing

13 Detailed metrics on productivity impacts, used to drive continuous improvements Lost Testing Hours by Root-Cause / by service provider) Detailed metrics by severity and repeat failed changes

14 Track and report process improvement initiatives to CIO and steering groups Completed Work in progress Jan 2010 Jun 2010 Jun 2010 Dec 2010 Jan 2011 Oct 2011 Non 2011 Jun 2012 Standardised change management process & increase applications in Test environments CAB. Improved process in capturing project environment requirements. Review estimation process gaps and cost recovery processes Investigate build downtime reduction - dynamic environment switching Increase reporting coverage by environment to further drive process improvements Increase use of automated tools and services to facilitate testing requirements. Review and formalise acceptable SLA s and SLO s for internal and vendor managed applications. Support services and scope review with cost rationalisation Enable continuous improvement by increasing detail in reporting. Increase scope in automated shakedown tests. Start of day and end of day test automation analysis Live monitoring of services and accounts to proactively maintain environments. Virtualization of application & infrastructure activities. Review resources across test environments to identify sharing opportunities Increase documentation detail of core systems Change management tool Automation

15 Realisation of Virtualisation & Cloud Benefits Investment Split & Benefit Analysis Mths Physical Servers Virtual Servers Total Op Ex % % % % % now % Investment Benefits Months Investment Enablers Increased capacity for reduced cost Procurement & provisioning 27 days reduced to 1hr Real Estate (data centres, racks) Overhead cost reduction Elastic resource utilisation Snapshot builds Availability increases Duplicate stacks CPU follows demand 20mins Speed of recovery Reduce change impacts Demand driven environment integration Automation tools & services ITKO LISA / ACDC

16 Self Service Information

17 Test Environment Services Tool

18 Booking Analysis & Summary

19 Asset Management & Asset Detail