DevOps Unify Your Tests Or Fail. Kevin Surace CEO Appvance Inc

Transcription

1 DevOps Unify Your Tests Or Fail Kevin Surace CEO Appvance Inc

2 Global 5000 CIO s have 3 nearly consistent top priorities 1) Security 2) DevOps 3) Cloud

3 AGILE / DEVOPS / CI / CD CATCH ON IN SILICON VALLEY VELOCITY WITH QUALITY By late 1990 s Internet leaders standardize on this internally Builds occur multiple times a day Automated tests occur at every build, many times a day Thousands of tests run simultaneously and complete in minutes Unit, Functional, DB, Compatibility, Security, DDOS, Performance, Load etc.

4 DEVOPS Development & Operations Teams Come Together (& QA) They strive to deliver RAPID sw development, test, deployment & enhancements Agile Continuous Integration Continuous Improvement Continuous Delivery Cloud

5 OLD VS NEW OLD Coding IT Environment QA Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Coding IT Environment QA NEW Release Release Release Release Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10

6 BUILD & RELEASE EVERY HOUR { Build complete at 1PM Tests kicked off at 1:05PM 1:05 to 1:15 - Functional, security, compatibility tests run in massive parallel 1:15 to 1:30 - Performance, load, database, DDOS penetration tests run after or on parallel environment 1:30 results in repository { Build complete at 2PM Tests kicked off at 2:05PM 2:05 to 2:15 - Functional, security, compatibility tests run in massive parallel 2:15 to 2:30 - Performance, load, database, DDOS penetration tests run after or on parallel environment 2:30 results in repository

7 PRESSURE FOR VELOCITY IS BUILDING Leading Enterprises have Moved to DevOps with Faster and Faster Release Cycles Google - over 10,000 code pushes per day Amazon - over 100,000 pushes per day Netflix - over 250 pushes per day Traditional Enterprises have been challenged to do even 1-2 updates per month - but that is changing Cisco, Apple and other traditional Enterprises have been retooling to focus on Agile and DevOps in order to compete with these fast-paced businesses - but they have all hit the same well - the speed of test

8 IMPACT ON QA IS ENORMOUS

9 WHAT GOT YOU HERE WON T GET YOU THERE

10 ORIGIN

11 Winston W. Royce, Managing the development of large software systems Proc. IEEE WESCON, Aug 1970 Winston W. Royce ( ) American leader computer scientist & Engineer of software development in the second half of the 20th century. B.Sc. in physics, M.Sc & Ph.D. in aeronautical engineering, all from California Institute of Technology (Caltech). Worked in the field of software systems engineering since Since the 1970s The director of Lockheed software technology center in Austin, Texas. The first to describe the Waterfall model as an example of a doomed method. Received the AIAA information systems award in 1975.

12 Royce s Worst Nightmare

13 Core: Solid Testing Testing phase should be the most important phase of the project development Royce recommendations for good testing: Plan the tests before product coding in the design phase. Use autonomous & detached test group of testing specialists & not program developers to test your product Test every logic path in your product at least once. Test not only the product functionality but all other aspects.

14

15 GOOGLE TESTING BLOG Tests create a feedback loop that informs the developer whether the product is working or not. The ideal feedback loop has several properties: It's fast. No developer wants to wait hours or days to find out if their change works. Sometimes the change does not work - nobody is perfect - and the feedback loop needs to run multiple times. A faster feedback loop leads to faster fixes. If the loop is fast enough, developers may even run tests before checking in a change. It's reliable. No developer wants to spend hours debugging a test, only to find out it was a flaky test. Flaky tests reduce the developer's trust in the test, and as a result flaky tests are often ignored, even when they find real product issues. It isolates failures. To fix a bug, developers need to find the specific lines of code causing the bug. When a product contains millions of lines of codes, and the bug could be anywhere, it's like trying to find a needle in a haystack.

16 Agile & DevOps: All Impediments Must Be Removed

17 AGILE AND DEVOPS AND CLOUD ARE CHANGING THE WAY ALL ENTERPRISES WILL FUNCTION IN THE FUTURE Software Development Lifecycle Processes Tools Reporting

18 NEW TEST AUTOMATION IS CRITICAL TO DEVOPS SUCCESS SURVEY OF 200 CIO S BY VANSON BOURNE - SEPTEMBER 2015 Developing a strategy for DevOps is a top priority for our organization 91% agree or strongly agree Addressing integrated testing is a critical aspect of our DevOps plans 91% agree or strongly agree Ideal to automate testing into one platform 83% agree or strongly agree The number of tools required to perform complete testing is slowing our time-to-market 68% agree or strongly agree Increasing velocity and productivity are top priorities in our evaluation of DevOps solutions 87% agree or strongly agree There are gaps in our current approach to testing 66% agree or strongly agree A unified platform that combines all the testing we need to do without scripting would greatly improve our time to market 86% agree or strongly agree Ensuring confidence in the quality of our releases is critical to the success of our DevOps efforts 89% agree or strongly agree We are currently evaluating DevOps testing tools to replace our current set 76% agree or strongly agree Our budget for DevOps has increased over the last 12 months 86% agree or strongly agree Our budget for DevOps will increase for the upcoming calendar year (2016) 85% agree or strongly agree Writing scripts for testing impacts our testing time and quality 74% agree or strongly agree

19 WATERFALL SW DEVELOPMENT - MONTHS TO YEARS

20 SILOED QA FUNCTIONS KEEP ENTERPRISES FROM ACHIEVING DEVOPS VELOCITY & QUALITY Operations APM DDOS Penetration Database Tests Application Penetration Load Testing Performance Testing Compatibility Testing Functional Testing Unit Testing Development

21 THE LEGACY TOOLS CREATE BARRIERS TO TRUE DEVOPS Weeks or months to create tests Tests are created and run in 5-10 different tools Scripting requires knowledge of 3-5 script languages hours (or more) to run basic tests Days to modify tests Result: Skip many tests to improve velocity Dev QA Ops

22 AGILE / CI / CD / DEVOPS

23 DEVOPS PULLS IT ALL TOGETHER INTO ONE SILO Requires new Dev Tools Ops Tools Test Tools Application Compatibility Performance DDOS Functional Database Development Load Unit Operations APM Penetration Testing Tests Testing

24 AUTOMATED UNIFIED TESTING DELIVERS ON DEVOPS Minutes to hours to create tests < 1 hour to run all tests Hundreds of tests run in parallel Minutes to modify tests Result: Run functional, performance, security etc. at every build for higher quality and velocity Reduce test development and runtimes by 10X

25 A COMPLETE RETHINK OF QA TOOLS IS NEEDED Agile Disrupts Testing As We Know It With Continuous Integration and Delivery, Existing Testing Practices Won t Cut It Tear Down the Walls and Make Testers An Integral Part of Development Testing Source: Forrester Report: Navigating The Agile Testing Tool Landscape

26 LEGACY TEST AUTOMATION TOOLS CAN T KEEP UP WITH AGILE REQUIREMENTS Dated Testing Tools ( ) The tools from HP, CA and others have failed to evolve over significantly over the past 20 years: Heavy reliance on scripting & coding Modern Test Tools (2000+) Point Solutions don t begin to solve the DevOps bottleneck They exacerbate the problem with more silo s Difficult to use and maintain Challenging to integrate into a modern lifecycle Vendors + A dozen open-source tools (unsupported) Basic unit/functional test and one pre-release

27 LEGACY TOOLS REQUIRE UNACCEPTABLE DEVOPS TRADEOFFS Velocity Quality

28 DEVOPS UNIFIED TEST AUTOMATION PLATFORM (UTAP) Web Apps Mobile Apps Citrix Windows Apps Java Thick Client Apps (Oracle) API testing (ie REST) SQL and NoSQL IOT

29 UNIFIED TESTING FORCES CORRECT TEST METHODOLOGY Rigor without additional Effort

30 SUPPORTING THE ENTIRE DEVOPS LIFECYCLE Development Unit Testing Acceptance Testing Sanity Testing Continuous Integration Unit Testing Acceptance Testing Incremental Integration Testing Database Testing QA Environment Smoke Testing Integration Testing Functional Testing Usability Testing Compatibility Testing Install/Uninstall Testing Regression Testing Security/App Penetration Database Testing Staging Environment Performance Testing Stress Testing Load Testing Soak Testing End-to-end Testing System Testing DDOS DDOS App Penetration

31 INTEGRATION DevOps Automated Test

32 OPEN PLATFORMS ARE KEY USE EXISTING SCRIPTS - OR SKIP CODING ALTOGETHER Selenium Selenium RC JMeter Visual Basic.NET C# Sahi HTTP Archive (HAR) PHP Java Python JUnit Perl Jruby Jython Groovy SoapUI QTP LoadRunner

33 ADVANCED RECORDING CAPABILITY IS TRANSFORMATIVE Ultramodern recording rapidly creates non-brittle easy to edit scripts UX level, API level or both together

34 CONTAINERIZATION Requires full and constant testing Automated test occurs at multiple levels and surface issues Accelerates move away from VM s

35

36 AUTOMATED TESTING REQUIRED FOR IOT SUCCESS REST: Standard API for IOT applications MQTT: a protocol for collecting device data and communicating it to servers (D2S) XMPP: a protocol best for connecting devices to people, a special case of the D2S pattern, since people are connected to the servers DDS: a fast bus for integrating intelligent machines (D2D) AMQP: a queuing system designed to connect servers to each other (S2S) IOT Database Tests for Scalability

37 AUTOMATED DATABASE TESTING Validate: Data mapping ACID properties Data integrity Schema accuracy Database performance Performance under load Server performance Get complete data on database server performance, including CPU, memory, I/O, storage and other metrics to help them rapidly identify areas for improvement and optimize their database structure, setup, schema, caching strategies, federated database configurations, DBMS and overall performance.

38

39 STANDARD WHITE HAT TESTING App Penetration Testing today is basically a CYA activity Done once a year to check for any obvious holes Misses the entire point of Agile Quality with Velocity OWASP App Penetration Website Under Test Test findable pages

40 DEVOPS SECURITY TESTING App Penetration uses the same Use Cases Creates Rigor - Runs at every single build Requires no expertise to run Test reruns at any step in any use case, testing deep and hidden pages Separately tests for openings during a DDOS attack by creating one Uncovers 80% more potential issues than standard White Hat testing OWASP App Penetration Suite Website Under Test Test ALL pages, even hidden ones, even at overload

41 VELOCITY WITH QUALITY - CREATING TESTS Hours Hours Scripting Compatibility Penetration Load Database Performance Functional Unit 0 Legacy Create 80 Hours UTAP Create

42 VELOCITY WITH QUALITY - RUNNING TESTS Hours Hours Running Automated Tests Hours Compatibility Penetration Load Database Performance Functional Unit 0 Legacy Run UTAP Run

43 DEVOPS QA CHECKLIST Reduce writing and rewriting of use cases (write once, use for all test types) Run all tests in parallel Run all test types at each build (functional, performance, security, compatibility ) Embrace new record/script technology that makes scripting 20x faster Import Legacy Scripts no need to rewrite (transition FAST!) Test from UX and from API at same time, Every Time Eliminate all point-solution siloed tools What got you here won t get you there. A new generation of automated testing is required. Target a 10X productivity enhancement goal. Velocity with Quality. It IS Doable. Unified Test Automation Platform

44 Unified Test Automation for DevOps us at