"Decoupled System Interface Testing at FedEx"

Transcription

1 T20 Concurrent Class 10/3/2013 3:00:00 PM "Decoupled System Interface Testing at FedEx" Presented by: Chris Reites FedEx Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL

2 Chris Reites FedEx Services As a Technical Principal in software quality assurance for FedEx Services, Chris Reites has experience providing cutting edge, best-practice processes for the development and testing of large-scale, complex, global software systems. Chris has been working for FedEx in IT for fifteen years. Prior to joining the software testing organization within FedEx, Chris was a software developer for several key applications within the FedEx billing system.

3 Decoupled Testing Overview Chris Reites Technical Principal FedEx Services STARWEST Conference 10/03/2013 FedEx Testing What We re Dealing With Application Testing and Certification Responsible for Test Planning, Design, Execution, and Validation Key Shipping Products (Desktop devices, etc ) Backend Rating, Revenue, Tracking, and Invoicing Systems Major releases annually, as well as weekly exception and emergency loads Hundreds of applications involved. 2 1

4 Types of Testing We Do Quality Assurance Approaches Functional Testing End-to-End Testing Definition: Evaluates the compliance of a system or component with specified functional requirements. Includes New Features, Regression, Integration and System tests. Coordinate with Revenue Testing for impacted products. Definition: Functional tests designed to validate specific outcomes within Revenue systems. Starts with entering shipments (via INET, WSVC, CAFE, FXRS, etc.) and addition of selected scans, flows through servers and intermediate systems, and concludes with validation of results on invoices and in accounts receivables. Performance Testing Vulnerability Testing Regional Testing Definition: Evaluation of a system s or component s compliance with specified performance requirements. Includes Volume testing, Load / Stress testing, Failover / Recovery Testing and Disaster Recovery. Definition: Utilizing security scanning tools and educated vulnerability testing through manual human intervention techniques, providing scanning and penetration testing for supported applications during regular releases. Definition: Includes functional testing performed on behalf of the Regions by SQA Testing groups and Language Translation Testing performed by Marketing or User groups within the Regions. Production Checkout Definition: Validates core software functionality in production after regularly scheduled software loads and regularly scheduled data updates. Supports Corporate Loads, Dotcom Loads, Exceptions Loads & some Emergency loads. Certification Services CSP Certification Label Certification (ensures operational excellence) Definition: Consulting with third-party software providers to integrate FedEx technology into their applications and validate that their applications meet FedEx requirements from a brand, revenue and operational perspective. Definition: Validation of Express labels submitted by automation clients, CSP providers, WebServices customers and FXRS customers who modify labels in production. Automation clients submit all labels for certification. 3 Potential Hazards in Large System Testing Dependency to have all code ready at the same time Interfaces were critical yet not well documented or understood Interface changes coming late into Code/Test phases Interface issues caused Ripple Effect throughout system Impacts our Speed To Market and causes a lack of flexibility in testing 4 2

5 Decoupled Testing Concepts and Potential Concept Provides the ability to test target systems independently by removing dependencies on other external systems. Divide and conquer Reduce defect fix/validation cycle time Mitigate Risk when introducing software changes comprehensive regression test Reduce validation dependency on End-to- End cycles Devices Corp. Load End to End Testing Automation Front End Process Back End Process Edit & Rating Corp. Load End to End Testing enhanced by Decoupled Test Revenue Invoicing Settlement Adoption in Automation Systems Reduced dependency on revenue test cycles Mini Revenue cycles within automation systems using Decoupled Test Tool Pilot mini Decoupled Testing cycle in one shipping device FY12 Q1 Mini Decoupled Testing cycles in multiple shipping devices for Jan12 corporate load. Devices Automation Front End Process Back End Process Edit & Rating Revenue Invoicing Settlement 5 What Makes Up Decoupled Testing? Decoupled Testing provides the ability to test target systems independently by removing dependencies on other external systems. 4 Core Functions: Interface Data Capture: supports the collection and storage of interface data Interface Data Compare: supports on-demand, field-level comparisons of interface data Interface Data Injection: supports the ability to replay previously captured interface data into the target system Interface Simulation: supports the virtualization of responses from backend system interfaces that are synchronous in nature 6 3

6 Coupled Testing Inputs End to End Corporate Load Testing Shipment Sys Regions Apps Freight Rev Rating ecommerce Revenue Back End Ground Rev Rating Corp App Outputs GL Entries Invoice A/R Express Rev Rating FCIS Sys Accounting Order Cash Input Output Testing Researcher Expected Jump 7 Coupled Testing GL Entries Inputs Shipment Sys Regions Apps Corp App Freight Rev Rating ecommerce Revenue Back End Ground Rev Rating Outputs Invoice A/R Express Rev Rating FCIS Sys Accounting Order Cash Input Output Testing Researcher Expected Jump 8 4

7 Decoupled Testing End to End Testing Inputs ecommerce Front End Shipment Sys Freight Rev Rating Revenue Back End Corp App Outputs GL Entries Invoice A/R ecommerce Back End Order Express Rev Rating FCIS Sys Accounting Cash Input Output Testing Researcher Expected evolve evolve evolve evolve 9 Decoupled Testing S Testing Researcher Actual Before Actual After ecommerce Front End A Expected Inputs A Ouputs Express Rev Rating B B Revenue Back End C 10 5

8 11 Revenue Systems Example: From Serial to Parallel Processing Common Test Data Design New test data created for both system entry points and downstream injection points simultaneously Regression test data injected with previously captured results Positive Effects: Reduce idle time (waiting for successful endto-end execution) Increase the test coverage utilizing data comparison analysis Early detection of issues Quicker validation of fixes Shipping Input Automation Compare outputs against previous results Test Data Revenue Input Revenue Inject data directly into backend system Compare outputs against previous results 12 6

9 Interface Simulation Interface Simulation is the ability to virtualize responses from backend systems. Simulated Interfaces remove backend complexity from testing environments and provide stable, predictable behavior to make system testing easier and more available. Response times of virtual interfaces can be varied to simulate latency or systems under load 13 Test Strategy Objectives Reduce Defect Resolution Time From weeks to days or hours Reduce Validation Time By providing data analysis solutions By automating validation processes Improve Code Quality coming into Integration Testing By discovering and resolving defects earlier By more quickly fixing and revalidating defects Increase Quality Assurance Capacity By providing environments on demand By providing a self service set of test solutions 14 7

10 # of Defects # of Defects 9/19/2013 Common Goals And Success Metrics Defect Removal Cost Reduction The measure of identifying and fixing defects earlier in the testing process. Normal Dev/Test Phase DTT Dev/Test Phase Agility and Time-to-Market The measure of time required to launch a project/feature through newly enabled testing processes. Support Resource Cost Reduction / Reallocation The measure of resource reduction due to functionality being delivered. 15 Defect Removal - Reduction in Cost The measure of identifying defects earlier in the testing process. Metrics above reveal the following: A definite shift in time in the number of defects found and closed in earlier cycles of testing 16 8

11 # Test Cases Shipped and Validated Relative to Traditional Method Duration of Testing Relative To Traditional Method* Duration of Testing Relative To Traditional Method* % Of Test Cases Tested Per Day Relative to Traditional Method 9/19/2013 Decoupled Testing Speed to Market Testing Process for Certification of a Key Customer Shipping Platform % % 38.00% Traditional Decoupled Testing v1 Decoupled Testing v2 Testing Method * Duration includes shipping/execution and validation 17 Decoupled Testing Resource Cost Reduction Testing Process for Certification of a Key Customer Shipping Platform Validated Baseline 100.0% 100.0% 8.5% 100.0% 147.0% 147.0% 1750% 3268% Traditional % Traditional Decoupled v1 Decoupled v2 Testing Method 67.00% Decoupled Testing v % Decoupled Testing v2 100% Traditional Decoupled v1 Decoupled v2 Testing Method For each test case: field level validation on two transactions Around 200 field comparisons per test case Testing Method 18 9

12 Case Study: FedEx Delivery Manager 19 FedEx Delivery Manager Decoupled Testing Business Challenge Parallel development and simultaneous delivery of multiple FedEx applications impacted by the project prevented integration testing prior to Integration Testing. Goals Provide development teams access to key backend systems during Unit Testing in order to identify defects early, and have the ability to inject transactions into various parts of the system to break dependency of needing all systems ready at same time

13 Results Utilizing DTT, FedEx Delivery Manager early discovered 75% of defects and validated 68% of test cases for the Back-End Systems Delivery Manager Portal started having success Utilizing injection for Back-End testing, FedEx identified: 11% of total defects during shakeout 73% of total revenue defects 82% of total credit card defects 63% of total Shipment Event Processing defects 21 Why All Companies Should Be Thinking About Decoupled Testing As applications and systems become larger and more complex, traditional end to end testing becomes un-scalable. With the increase in 3 rd party service providers and service oriented architectures, the ability to decouple those dependencies for testing is critical. The longer you wait to begin decoupling the testing of your systems, the harder it is to do. There is probably a LOT of low hanging fruit available to start with

14 Where to Begin Involvement and buy-in of internal business counterparts is critical Identify testing dependencies that cause the most issues (unavailability, late delivery, high associated costs, etc ) Find the quick wins to build confidence and momentum Partner with application developers and find ways to share the decoupled testing tools 23 What s Next? User-driven, integrated data management system Minimal development on interface rollout High-level reliability & instrumentation Failure tolerant (re-connect) Standard FedEx logging (monitor) Unattended operation Scalable without incurring major licensing cost Large Capacity Service Oriented Architecture for Integration with other Systems Rules-driven business logic Synergistic Test Management System 24 12

15 QUESTIONS? 25 13