Strategies for a Successful E2E Systems Integration Test. Fiona Charles Let s Test May 9, 2012

Strategies for a Successful E2E Systems Integration Test Fiona Charles Let s Test May 9, 2012

This session Describes key project management strategies I have used to manage large- scale Systems Integration Tests Offers practical tips for implementing those strategies on your next SIT The focus is primarily on large- scale integrations, but the strategies and tips can be scaled for any size of test. Strategies for a Successful E2E SIT 2

1 Project management 2 Applications 3 People 4 Test readiness 5 Information & communication 6 Environments & data 7 Questions & discussion Strategies for a Successful E2E SIT 3

A Typical Systems Integration Project Merchandising systems New POS Accounting systems New Loyalty systems Point of Sale registers & Controllers Supply Chain systems Management Information systems Strategies for a Successful E2E SIT 4

The Test Manager s Job Develop a test strategy that addresses the major risks within the time available Build the team you need for success Build the right controlled data tests and plan Manage testing and evaluation Ensure acceptance by stakeholders Sounds just like any other test - right??? Strategies for a Successful E2E SIT 5

The test management challenge is at least as great as the testing challenge A project in itself Always highly visible, and often controversial Requirements for success often misunderstood Your job is to hold to the big picture E2E Systems Integration Testing requires the skills of an experienced test project manager Strategies for a Successful E2E SIT 6

It s a Test Project Management job Much of the complexity in managing a Systems Integration Test comes from the sheer number of things involved: Applications Platforms Environments Data Application readiness dates Business processes or functions People Managing these is a logistical, rather than test, challenge. Strategies for a Successful E2E SIT 8

The numbers can seem overwhelming! Strategies for a Successful E2E SIT 9

But it s a nightmare only if you don t manage it Divide and conquer Have a strategy for each type of thing, and a project plan that says when you are going to deal with it Keep your priorities current Don t confuse the urgent with the merely important Delegate where you can Use project management tools and techniques Strategies for a Successful E2E SIT 10

Be sure to level- set in your own head Large Vehicles Need More Room And then in everyone else s head Strategies for a Successful E2E SIT 11

Make sure you are positioned correctly Senior role, working on equal footing with: Project managers Architects No barriers to information Pulse of the program When each project is delivering (really) Risks, issues, delays, individual test strategies, etc. Authority SIT entry criteria Requirements to make SIT work Reporting to the overall Project/Program Manager Strategies for a Successful E2E SIT 12

The applications will determine test scope Start with the Architecture/integration diagrams Use whatever you can find If they don t exist, draw your own Validate what you use Keep an open mind about the test boundaries Received belief may be wrong Focus on whole system outcomes Identify all incoming data that could influence your results Ask what the end point should really be What s the last application or report the critical data impacts? Who needs to sign off? You may need to include unchanged systems Strategies for a Successful E2E SIT 14

System Context Diagram APP Teller (Authenticated) APP APP APP Perform CSR Transactions: Retrieve Client Profile Perform Client Search Withdraw Funds, Receive Cash, Deposit Funds (PDA), Negotiate Cheque/Money Order, Transfer between accounts, Purchase Drafts/ Money Orders, Get a VISA cash advance, Make VISA payment, Reverse VISA Payment Reverse VISA Cash Advance, Purchase Foreign Exchange Purchase Traveler Cheques or gift cheques, Make Bill Payment, Authenticate Client, Deposit Funds (Current Account) Perform Close Day and Extended hour Processes Batch Off Switch Workstation Query Client Session Update Bank book Insert Client Session Detail, Query Client/Transaction Session, Retrieve end of day data by branch Journal Get Traveler Cheque and Currency Txn Details (batch interface) APP Check SECAF Entitlements Teller Application Post GL Entries (batch interface) APP Credit, Debit, DDA Inquiry, Bank book updates Acct Details, Acct History Current Account - Cash and merchant envelope transactions. (batch interface) APP Retrieve Client Profile Search Client Retrieve Account List Account Details Negotiate/Accept System priced Deals Authenticate Client (PIN Verification), Perform CIS Authorization Perform CIS Inquiry(2,4,5,6 or 7), Retrieve Client Information, CIS Limits and Account List and Balances (Uses CIS 11 as primary path if the user has a client card) Retrieve Transit List APP Get Fee Get Currency List Get Denominations (XML Files from Web Server) VISA Cash Advance, Make VISA Payment, VISA Cash Advance Reversals, VISA Payment Reversal APP Existing CTP Objects for: Retrieve Client Profile Perform Client Search Retrieve Account List and Balances (if no client card or for a full account list) Retrieve & Update Offer & Opportunities APP APP APP Extract for EDW APP General Ledger Strategies for a Successful E2E SIT 15

You will be dealing with LOTS of people Strategies for a Successful E2E SIT 17

The rest of the project management team Project or Program Manager Architects Project Control People associated with each application, e.g., Project leads Project test leads and testers Programmers/Architects Management People outside the project who will want information QA, auditors, business sponsors, acceptors Your own test team Strategies for a Successful E2E SIT 18

Make sure you have a strategy for each group The rest of the project management team People associated with each application Management QA, auditors, business sponsors, acceptors Build relationships - lunch Status reports, regular status meetings Test working group or forum get input & buy- in for strategy SIT progress broadcasts 1 on 1 s Occasional status/strategy presentations Approach them know who they are and what they need Strategies for a Successful E2E SIT 19

Success depends on building relationships Be aware of cultural differences Different technology groups may have very different processes Different groups may have varied risk tolerances: e.g., front- line business vs. finance or audit Distributed teams in different countries Understand what each group wants/needs from your tests Work to get early buy- in for everything significant in your plan Develop your test strategy in a lightweight medium, and walk it around ( socialize it) If you have to publish a detailed document, try to get consensus before you write it Strategies for a Successful E2E SIT 20

Staff your own team appropriately Strategies for a Successful E2E SIT 21

You won t have time to be hands- on! Strong second- in- command Day- to- day SIT test team management Environment and data manager Work with all participating teams Small team of seasoned testers Design tests; prepare test materials Toolsmith(s) Any automation you can do Strategies for a Successful E2E SIT 22

Establish the ground rules (Entry & exit criteria) Environments ready, connected and tested Coordinated data Project- level tests complete, e.g., System and usability test Performance Near- neighbor interfaces All must- fix bugs, and all integration bugs of any severity, resolved Teams assigned (and set up in defect tracking system) Communicate requirements and get early buy- in Strategies for a Successful E2E SIT 24

Track readiness Your own team s deliverables Test materials (test cases and scenarios) Environments Data Initial setup of defect tracking system System readiness All participating systems Readiness to meet SIT entry criteria Report on readiness weekly using an easy- to- understand graphic Strategies for a Successful E2E SIT 25

A simple timeline for system readiness Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan 4 11 18 25 1 8 15 22 29 6 13 20 27 3 10 17 24 1 8 15 22 29 5 12 19 26 2 9 16 23 30 7 14 21 28 4 11 18 25 2 9 16 23 30 6 13 20 27 Program E2E SIT Test Incremental integration & pre-test Partial E2E Test with Main App v0.4, then full test with v0.9 (complete) E2E Regression Test with Main App v1.0 (final fixes) Proj 1 App 0 Main App v0.4 Main App v0.7 Main App v0.9 Main App v1.0 (final) Proj 1 App 1 Proj 1 App 2 R6 Proj 1 App 2 R7 Proj 2 App1 Proj 1 App 3 Proj 2 App 2 Proj 3 App 1 Proj 3 App 2 Proj 3 App 3 Proj 3 App 4 Proj 3 App 5 Proj 3 Special Infrastructure Proj 4 all other Business Apps Proj 4 Enablement Proj 4 Billing Proj 6 App 1 Proj 5 App 1 Proj 5 App 2 Corp Billing GL Accounting NOT YET PLANNED Proj 6 App 5 Proj 6 App 2 Proj 6 App 6 Proj 6 App 3 Proj 6 App 4 Proj 6 App 7 Proj 6 App 8 Legend: Completed - Ready for E2E Planned date Date at risk Late - new date not yet available Not yet planned Assumptions 1) Main App v0.9 will be full-featured and stable. 2) Project 3 Apps 2 & 3 will be E2E-ready no later than 12-Oct. 3) Starting E2E test without Project 3 Apps 4 & 5 will be feasible and will provide E2E test value. Risks 1) Required modifications to GL Journalling and back-end financial accounting have not yet been finalized or planned. (Change Request #29) Strategies for a Successful E2E SIT 26

Verify readiness Schedule a handover meeting with each team before scheduled entry to SIT Ask to see evidence (give advance notice of what that should be) Give teams a standard checklist to populate Review the checklist and evidence with each team when you meet Strategies for a Successful E2E SIT 27

Readiness Documentation Examples Test requirement Criteria Sample Documentation System/functional test Evidence of testing covering the functionality delivered in the release Test cases & traceability matrix Business acceptor sign- off on test coverage IDs & detail for outstanding bugs Exploratory test documentation (charters, logs. Etc.) IDs & detail for outstanding bugs Regression test of systems currently in production and modified for this project Completed regression test covering critical production functionality. Evidence that the required testing has been done (risk assessment, test completion checklist/report, etc.) Sign- off by required IT and/or business acceptors. Strategies for a Successful E2E SIT 28

Readiness Documentation Examples Test requirement Criteria Documentation UAT (or Agile Acceptance Test) All project test levels Completion of (User) Acceptance Test covering business acceptance criteria. Outstanding bugs: No outstanding must- fix Review outstanding bugs with impacts on downstream data or function Business acceptor sign- off OR Conditional acceptance by business acceptors, pending final acceptance at the end of SIT. Summary report showing current status of all bugs logged during testing Detailed bug reports for outstanding bugs, with analysis of downstream impacts, documented workarounds, and resolution plans Strategies for a Successful E2E SIT 29

Keep yourself informed Take advantage of all the communication channels Make sure you are in the key program/project level status meeting Ask regularly for updates on system readiness Review the program/project risk and issue logs for anything that might affect your test Chat to project managers and test leads Strategies for a Successful E2E SIT 31

Build your network Strategies for a Successful E2E SIT 32

Outward communication Make regular presentations to all the stakeholder groups SIT strategy Progress reports Set up a notice- board to post daily status (you ll need this during E2E SIT execution) Intranet, etc. Set up a test question and answer forum for your team s questions and share it on the notice- board Strategies for a Successful E2E SIT 33

Strategies for a Successful E2E SIT 35

Some of your most difficult issues are here Multiple platforms (hardware, OS, DBMS) Other infrastructure, like batch schedulers and message transports Excusive use for your test of adequate test environments Data, coordinated across participating applications Configuration management Co- ordinated defect management You need an Environment/Data Manager! Strategies for a Successful E2E SIT 36

Environment/Data Manager Inventory all applications and infrastructure elements Work with you and SIT test team on data strategy Work with each team to develop plans Environment plan Data plan Develop configuration management process Administer configuration management process during E2E test execution Strategies for a Successful E2E SIT 37

Test Environments Virtualize if you can but you may not be able to Physical environments may not exist for every application There may not be connectivity where you need it You may have to negotiate for exclusive use If setup/purchase is required, there will be long lead times Start early! (Get that Environment Manager on it!) Strategies for a Successful E2E SIT 38

Coordinated Test Bed Data This can be very difficult to achieve A showstopper if you don t have it You need a strategy and a plan Start early! (Get that Environment Manager on it!) Strategies for a Successful E2E SIT 39

Configuration Management Can be complex, with Production fixes and other projects going on at the same time as yours You need an exact picture at all times A system solution across platforms will probably not work Design a simple logical release process: Start with an inventory of all the systems, platforms, infrastructure elements participating in the test Names, versions, etc. Logical Release 1 is the starting inventory Any time ANYTHING changes: application for defect resolution, infrastructure upgrade, etc., update the inventory, and you re now on Release 2 Strategies for a Successful E2E SIT 40

Defect Management Essential to have one defect management system for the entire integration Set it up so you can track by application /project/ vendor Set up a SWAT team to handle defect diagnosis and assignment Neutral lead, e.g., overall Project Architect Representative from each team Make the Lead a regular attendee at your daily checkpoint during test execution Strategies for a Successful E2E SIT 41

Use project management to go from this Strategies for a Successful E2E SIT 42

To something like this Strategies for a Successful E2E SIT 43

Questions & discussion Strategies for a Successful E2E SIT 44

Fiona Charles fiona.charles@quality- intelligence.com www.quality- intelligence.com Twitter: @FionaCCharles Images and text Fiona Charles 2012