"End-to-End Testing in an Enterprise Agile Environment"

Transcription

1 W15 Track 5/6/2009 3:00:00 PM "End-to-End Testing in an Enterprise Agile Environment" Presented by: Billie Bell Intuit, Inc. Presented at: 330 Corporate Way, Suite 300, Orange Park, FL

2 Billie Bell Billie J. Bell, E2E QA Program Manager for Intuit, developed the end-to-end (E2E) QA Program Manager role after recognizing the benefits of engaging QA teams across Intuit's primarily agile project teams in understanding how cross-project integration testing influences achievement of overall business objectives. Using the rollout of specific company-wide initiatives provides a pragmatic context and motivation for development of new processes and promotes an E2E quality mindset beyond QA. Billie s recent accomplishments include coordinating the integration of 30+ applications with a PCI compliance platform, a consolidated entitlement and software license management platform infrastructure integrating desktop and Web service applications, and a 10-layer subscription and billing SAAS initiative integrating 3rd party and internally-developed applications. Including her eight years at Intuit, Billie brings more than 20 years of experience in enterprise financial software development in QA management, product management, and program management roles in both Fortune 500 and start-up software development companies.

3 Business Demands End-to-End Testing in an Enterprise Agile Environment Billie Bell Intuit, Inc. Agile Development Complex Technology 1 This presentation discusses the theory and practice of large scale integration testing Context An enterprise working to reach middle ground between agile and processbased methodologies Best practice tip Some practical examples of mechanisms Intuit has found to be effective appear throughout the presentation 2 1

4 Three forces in today s business environment interact to create challenges for quality leaders Accelerated Business Demands Agile Development Complex Technology 3 Despite our best efforts, results can sometimes come across as grim headlines v Accelerated Business Demands Agile Development Complex Technology 4 2

5 We expect headlines that reflect our intentions to deliver a high quality release on time v Accelerated Business Demands Agile Development Complex Technology 5 Today s challenges bring opportunities to take a fresh look at quality leadership 6 3

6 Quality leaders are uniquely capable to set the course for success in today s business setting Community organizers for quality Visibility promoters 7 As a community organizer for quality you can build the foundation for end-to-end mindset 8 4

7 Start with your own vision 9 Develop shared vision 10 5

8 Lead process change across the organization 11 A quality leader s enhanced role should also include promoting visibility Help pqa and other teams see their applications in the context of the enterprise Help the organization stay one step ahead by acknowledging and managing risk 12 6

9 Promoting visibility into business rationale is an element of the new role QA leaders play Business context and relative priority result in more effective testing v Accelerated Business Demands More of the business transaction ti is outside a single application Agile Development Complex Technology Agile best practices emphasize connection with the customer QA is a business discipline, not just a set of tasks that occur in serial fashion after development 13 Best practices to promote visibility bring practicality to the quality community Best practice tip Some examples of ways to promote visibility Weekly 1-hour Spotlight Sessions for overview presentation by business process experts Enterprise and system architects 2-day workshops for more complex, new systems Cross-project test case reviews 14 7

10 Quality leaders have the perspective to lead the organization in seeing the risks ahead 15 Risk assessment and management by a cross-team network promotes accountability Use risk planning sessions to 1. Identify risks 2. Identify indicators 3. Plan mitigation 4. Identify response 5. Monitor risks 16 8

11 Best practices used to manage risk include collaboration and analysis Best practice tip Ways to uncover risks Pre-assessment questionnaire Defect analysis from previous projects Risk workshops 17 With a collective understanding of where we are, we can examine how we measure quality 18 9

12 Extend definition of quality Beyond a single application Across release cycles 19 Order System High Level Architectural Diagram Web applications Order Sources blank Payment blank Processing and Accounting Call center Call Center App Legacy applications Processor 20 10

13 Order System High Level Architectural Diagram Web applications blank blank Call center Call Center App Legacy applications Processor 21 The Purchase use case is architecturally nine layers deep Processor Call Center App 22 11

14 The Update customer information use case spans five layers Business Use Case Web App A Purchase Update customer Info Process 1 Process 2 Web App B Web App C Cancel Bill subscription Web Order Processing Order Source IVR Call Centers, Mail and Fax orders Processor Call Center App Order Engine Credit Card Authorization Settlement Financial Accounting 3 rd Party CC Processor Customer Information 23 The Cancel use case is 5 layers deep Processor Call Center App 24 12

15 Bill Subscription use case is executed outside the application Processor Call Center App 25 Realities of today s enterprise technology can leave our traditional quality monitors lacking 26 13

16 When it comes to code coverage, the whole is not the sum of its parts Component Coverage Component 1 80% Component 2 80% Component 3 80% 80% x 80% 64% x 80% 80% 64% 51% Process Coverage 27 Payoff for integration analysis from business and technical view is more robust test cases Best practice tip Integration analysis Identify systems and system-to-system interfaces Identify and high-level business transactions that are dependent on these interfaces or trigger their execution Analyze the use of data across multiple systems to track data flow and reconciliation Simplify and communicate all of the above Use UML diagrams to illustrate functional touchpoints Summarize and conceptualize the wealth of information Communicate the big picture, provide maps to details Reporting test execution results by functional use case 28 14

17 Onboarding applications post release extend accountability for quality End-to-end testing becomes more like no-end-in-sight testing 29 Onboarding a new applications to our model may appear as a new box on the diagram Web App A Process 1 Process 2 Web App B New Web App Web Order Processing Order Source IVR Call Centers, Mail and Fax orders sor Proces Call Center App Order Engine Credit Card Authorization Settlement Financial Accounting 3 rd Party CC Processor Customer Information 30 15

18 But in reality, the picture is multi-dimensional Considerations Backward compatibility Persistent test infrastructure Technical support mechanisms Web App A Process 1 Process 2 New New Web App Web App New Web App Web Order Processing 31 Test automation and easy integration test mechanisms ensure quality post release Best practice tip Automated regression tests Automated smoke tests in pre-production environments on a frequent basis Autolab to run build verification regression tests Can be run on demand by development engineers Test data mechanisms Baseline test data sets for key data repositories Tools that verify data synchronization Dynamic data reservation systems to avoid data collisions Examples: Test credit cards, license numbers 32 16

19 Success in building an enterprise-wide quality community requires collaboration and inclusion 33 Enlist all stakeholders The less obvious the obvious 34 17

20 QA teams can leverage new best practices used by their development partners Certain agile best practices provide clear benefits to quality for the project at hand Test-driven development can result in better unit tests Retrospectives result in more frequent lessonslearned Other best practices evolve to provide even greater benefit to end-to-end testing Business use cases created by system and enterprise architects Imbedded developers from upstream projects 35 Business process experts are key contributors throughout the process Involve business process matter experts in all phases Where you find them can differ in every situation Be flexible in your expectations 36 18

21 In summary, an effective E2E test strategy is based on shared vision and credible results Quality leaders as activists who Create a broad quality community Enable the organization to maintain sight of the big picture Quality defined as a continuum Outward-focused in addition to being inward-focused Extending beyond the release cycle Operationally inclusive of all stakeholders and methodologies Leveraging agile development practices Finding more ways to gain from business process expertise across the company 37 No single tool or methodology guarantees success for an end-to-end test strategy It is a strategy based on a combination of people, tools, mechanisms, and best practices It works best when it is emergent and organic, but the seed has to be planted and nurtured In many ways the best practices are the same as traditional; what sets it apart is the focus on the end-to-end business process It s about people more than tools It s about mindset more than skill set 38 19