Testing Trends - Focusing on the Right Test Tools and Industrializing Their Usage March 11, 2015

Transcription

1 Testing Trends - Focusing on the Right Test Tools and Industrializing Their Usage March 11, 2015

2 Brought to you by Vivit Testing, Quality and Application Lifecycle Management Special Interest Group (TQA-SIG) and the South Africa Chapter Leaders: Bernard P. Szymczak Jr, Andreas Birk, Robert Linton, Christopher J. Scharer, Damian Versaci

3 Hosted by Bernard P. Szymczak Jr Lead IT Systems Engineer at Progressive Insurance Vivit TQA-SIG Leader

4 Today s Presenter Mihai Grigorescu Accenture Global HP Alliance Testing Lead Vivit South Africa Leader

5 Housekeeping This LIVE session is being recorded The recording will be available on BrightTALK immediately after this session Q&A: Please type questions in the Questions Box below the presentation screen Additional information available for you behind the Attachment button and later on the Vivit website

6 AGENDA Intro Accenture testing Testing trends what the analysts say Test Architecture and how TMMi fits into the picture Testing ecosystem, choosing the right tools sets ALM industrialization Q & A

7 Intro Accenture testing SAP Test tools implementation and support partner HP SaaS SAP Sol Man to QC integration partner

8 Intro Accenture testing

9 Testing trends what the analysts say Fig.1 Source: IDC, Doc # / July Worldwide and U.S. Discrete Testing Services Forecast Fig.2 Source: HP/SOGETI, World Quality Report Testing Spend Worldwide Discrete (Standalone) Enterprise Application Testing Services market was $16.7B in 2013 and growing 15% annually to $33.7B in 2018 (Fig.1) U.S. Discrete (Standalone) Enterprise Application Testing Services market was $8B in 2013 and growing 13% annually to $15B in 2018 (see IDC report for graph) Budget spend % per projects is continuously increasing (Fig.2)

10 Testing trends what the analysts say Drive down costs Support for complex infrastructure testing (compatibility, interoperability) Drive productivity gains (e.g. more time for staff to pursue other initiatives) Test staff augmentation Standardize testing tools and processes Leverage third-party tool expertise, talent and skills Automate regression testing (to support complex business processes across multiple Help with test data management services Standardize internal QA environment across projects As insurance against potential liability resulting from insufficiently tested products by in- Leverage fully managed application lifecycle testing services Leverage fully managed specialized application test frameworks Support for localization country specific readiness testing Use turn-key test solutions for performance test or test automation Support for vertical solution testing (e.g., banking, telecom) Other Source: IDC, Doc # / July Worldwide and U.S. Discrete Testing Services Forecast Drivers for using 3 rd party services While drivers are multi-faceted, 31% of all respondents indicate that the primary driver for using 3rd party testing services centers on driving down costs. 37% of public sector organizations indicate the primary driver for using 3rd party testing services is support for complex infrastructure testing, with the second most important being driving down costs. The top three characteristics buyers indicated were most significant when selecting a provider for application testing services were: reasonable costs (35%), high level of responsiveness/rapid deployment (20%) and cloud testing experience (20%)

11 Testing trends what the analysts say Fig.1 Source: HP/SOGETI, World Quality Report Fig.2 Source: HP/SOGETI, World Quality Report Challenges and Outsourcing trends There is common trend of outsourcing testing services to specialized vendors, with a drop from 51% of companies using internal resources in 2012 to 30% in 2014 (Fig.1) In Agile environments, hybrids persist (i.e. Scrumfall), no real Agile environments, no specific methodologies nor adequate tools utilized, with difficulties in executing automation (Fig.2)

12 Testing trends what the analysts say There is an increase in app delivery which will reach approximately 120 releases per annum per app by 2020 Companies are already planning for upgrading to HP ALM 12.x Source: HP, March 2014 Source: VIVIT webcast poll, June 2014

13 Testing trends what the analysts say Source: HP, SOGETI - World Quality Report, The main testing drivers for companies are: reduced time to market, reduced costs, better resource management (Fig. 1) Companies are investing in TaaS (Testing as a Service), centralization, as well as managed services and outsourcing their testing capabilities (Fig. 2) Source: HP, SOGETI - World Quality Report, Conclusion: we believe it is paramount to ensure that test tools are optimized / industrialized within any test organization in order to support the main testing drivers and their respective processes adequately.

14 Testing trends what the analysts say Conclusions key take away s Digital transformation / business is here. One s QA methodologies and tools need to cater digital/taas/mobile testing. Virtualization (network, services etc.), testing in the cloud are all elements to consider and invest in. Applications are more fluid, delivered faster, which implies, leaner development & testing. Agile will take even more precedence than before, consequently develop competencies and centralized capabilities. More automation (non-traditional) + DevOps are required for faster, organized release to production and for bridging the gab between dev teams and operations. With increase in overall testing spend, investing (more) in tools adoption is the next logical inherent step. Security testing is still undermined (Sony knows for sure), companies need to invest in proper enterprise IT security governance frameworks and conduct more testing in this space. Investments in skills and tools on one side, as well as change management on the other are the crucial elements in winning the testing battle. With commoditization of testing, one needs to invest into more creative cost efficient TaaS models, more agile service based, output driven models, cross-shared and cross-skilled resources vs. legacy, traditional waterfall processes, dedicated resources and restrictive licensing models.

15 Test architecture and how TMMi fits into the picture What is a Test Architect? What a test architect IS Testing expert: a test architect MUST come from a testing background, with experience in all delivery models Advisor: solution architecture, estimations, test planning, metrics definition (and implementation), sales proposals Fixer: get into challenged projects, produce short term/quick wins and long term improvement plans What a test architect ISN T Test Manager Fire-fighter for daily operational issues Delivery person (project wise) Testing evangelist: deliver papers, presentations, training, provide technical leadership and strategic direction for their testing organization Implementer: Setup of TCoE / Test Factory, execute Test Assessments (Internal, External & TMMi), Enterprise Test Strategy (Definition/Setup), Test Optimization & Improvement, Non- Functional Testing, Standard and Emerging Technologies / Test Tools Great article on the topic here:

16 Test architecture and how TMMi fits into the picture How TMMi fits in the picture of testing tools Test Maturity Model integration (TMMi) Test Capability Models (based on TMMi) Test Tools Maturity Model (based on TMMi)

17 Test architecture and how TMMi fits into the picture How TMMi fits in the picture of testing tools TMMi has 5 levels A TCoE will aim at a level 2 to 3, while test factories, between level 3 and 5 Out of experience, we know that the perceived maturity level is higher than the actual one

18 POLL 1

19 POLL 2

20 Test architecture and how TMMi fits into the picture How TMMi fits in the picture of testing tools Accenture based Test capabilities models (based on TMMi)

21 Test architecture and how TMMi fits into the picture How TMMi fits in the picture of testing tools

22 Level 1 Level 2 Level 3 Level 4 Level 5 Test architecture and how TMMi fits into the picture How TMMi fits in the picture of testing tools Ad-hoc usage Basic defect management Defect Management Test Management Basic reporting Defect Management Test Management Requirement Management Advanced reporting (Project dashboards) Project templates KPI Scorecard Some automation & performance Defect Management Test Management Requirement Management Planning (release management) Automation, Performance Basic Mobile, Security Advanced reporting (Dashboards across programs) Scorecards Tool QA governance Basic CI Defect Management Test Management Requirement Management Planning (release management) Libraries and cross project asset sharing and reuse Automation, Performance, Mobile, Security Complete reporting capabilities Executive Scorecards Tool QA governance Seamless integration to 3 rd party tools (CI) Low Testing Tool Usage- Capability Maturity Level High

23 Testing ecosystem and focusing on the right tools Define (Test Architect) what tools to use in the test organization based on needs and long term requirements Ensure that all testing tools are integrated for a streamlined testing process Maximize tools ROI via: a) Tools industrialization b) Strategic partnerships with tools vendors, for band discounts

24 POLL 3

25 POLL 4

26 POLL 5

27 POLL 6

28 ALM industrialization Test Tool Map Shift Left Tools Strategy Quick Wins Tools Team HP ALM Industrialization Reporting Upgrade versions Integrations Upgrade editions Accelerators

29 ALM industrialization Test Tool Map Set up a test tools roadmap vs. your value chain / Software Development Life Cycle model Ensure that all testing areas are covered (Manual, Automation, Performance, Mobile, Sec., TDM, TEM

30 ALM industrialization Tools Strategy Define and implement an overall test tool strategy for the next 3-5 years The test tool strategy needs to include capital and operating expenses (including software, hardware and services costs) in order to provide the most efficient, cost effective solution for the organization Test tools strategy: The enterprise wide approach which describes the solution architecture for the deployment of all integrated test tools suites: manual, automation, performance, mobile & security, incl. additional tools such as requirements accelerators, COTS and in-house grown (internal) software utilized in the testing effort.

31 ALM industrialization Tools Team Set up a dedicated test tools team to handle all test technologies in the organization Establish a test tool service catalogue & use the tools as bridging mechanisms into other business areas

32 ALM industrialization Upgrade versions Upgrade to the latest HP QC/ALM versions to make sure your organization gets the best value out of the tool Make sure that your current QC version is still under support HP ALM / QC Vendor Support Time-Table Version 9.x 10.x Support End Date Nov 2010 Aug 2013 Jan 2015 TBA TBA TBA TBA ALM/QC 11.0 will be out of support in Jan 2015 (latest communication from HP): What extended support entails: Clients can still buy support beyond the standard EOL (End of Life) product, e.g. 31 Jan 2015 in this case, however: It requires a CSP Customer Support Plan, for max of 1 150% of list price Availability of support resources: best effort (when available) Support type: no product fixes/enhancements requests etc. Why consider upgrading editions: Avoidance of extended maintenance costs Continuation of vendor support after QC/ALM 11.0 end of support (Jan 2015) Decreased risks of troubleshooting issues and receiving product fixes/patches etc. License assignment per domain/business area (for all versions >11.5), enabling the possibility of producing a charge back model (to business) for recuperating software license costs (best suitable for medium to large installations/test capabilities) Less disruption: no additional required upgrades (from older versions to more recent ones) in the next 3-4 years due to end of support. Access to newest ALM features & functionalities

33 ALM industrialization Upgrade editions If not already on Premier, consider the upgrade in Editions, in order to maximize the ROI on the test tools

34 ALM industrialization Accelerators Utilize accelerators, either tools or other reusable assets in order to achieve maximum efficiency with the organization test tools sets. SAP ERP QC Project Templates, Predictive Analytics, E-approve, Business Intelligence, Operations accelerator, Requirements Management aiding tools, SAP/Automation accelerators etc.

35 ALM industrialization Integrations Ensure that there are proper integration flows between test tools and that redundant / duplicate work is not executed. Examples: AGM <-> ALM, Jira <-> ALM, Sol Man <-> ALM, ALM <-> PPM/XS, TFS <->ALM etc.

36 ALM industrialization Reporting Define and implement an adequate test metrics framework Use ALM s innate reporting capabilities (document generator, graphs and dashboards, KPI scorecards, cross project reporting), Business Views, in conjunction with 3 rd parties business intelligence apps; publish QC/ALM graphs to Share Point Site (SPS)

37 ALM industrialization Quick Wins Use the test capability to produce quick wins list that can be used with immediate effect in order to improve the test processes (via the tools) Example below: defect management process, consistent project template Training: ART (Adoption Readiness Tool, other practical hands-on classes.

38 ALM industrialization Shift Left Use ALM to drive Shift Left transformation into the organization Use simplified version of Shift Left (from a defect mgmt. perspective) improve quality via defect root cause analysis and improved management Shift Left Pushing quality upstream in the SDLC by enabling the testing capability with methodologies and tools Knowledge sharing capability development (training, coaching) Earlier defect detection, inspections, walkthroughs, JADs Methodologies + Tools

39 Q & A MIHAI.GRIGORESCU@ACCENTURE.COM

40 Upcoming Vivit SIG Webinar Best Practices in IT Architecture March 18, 2015 at 10:00 am EDT / 15:00 pm CET Brought to you by the Vivit Business Service Management Special Interest Group For more information and registration go to:

41 Vivit Board of Directors election Vote now on the 2015 Vivit Board of Directors election and the proposed Bylaw changes:

42 HP Discover Las Vegas 2015 June 2 4, 2015 at The Venetian Resort in Las Vegas. All members can Register Now via the Vivit link to save $300. Copyright 2014 Vivit Worldwide

43 Vivit s Deep Dive Sessions Deep Dive Sessions will be offered on Monday, June 1 st from 1:00 5:00 pm. Find more information on website under the training section. Copyright 2014 Vivit Worldwide

44 Thank you

45