Automated Testing of Web Services with SOAPUI. Abstract

Transcription

1 Presented to VANQ March 25, 2009 Automated Testing of Web Services with SOAPUI Ritchie Bros. Auctioneers Paul Vasak Chris Clarke Abstract Ritchie Bros. Auctioneers is the world s largest industrial auctioneer, with 40 auction sites worldwide. It sells used and unused industrial equipment at unreserved auctions. The auction business is supported by a busy interactive website ( where customers can search for equipment, buy it online at an auction that is broadcasted on the website, pay for the purchase and get a shipping quote. RBA is replacing the website with one that is designed and developed with modern technologies. The Unix-based website will communicate with internal backend systems, such as Oracle, via web services. This presentation will show what is involved in testing the web services and how the testing has been automated using the SOAPUI testing tool combined with an Excel-based data-driven technique developed in house. 1

2 About RBA Ritchie Bros. Auctioneers, since 1958 World's largest industrial auctioneer Sells used and unused industrial equipment at unreserved auctions In 2009 Conducted 195 industrial auctions at 40 sites at 14 countries Received 336,000 bidder registrations Of which 98,000 were successful bidders Sold 289,000 lots Orlando pictures 2

3 RBA Website - Demo Auctions Calendar 3. Search Equipment - "grader", photos 4. Member Services - Sign Up 5. rbauctionbid - Bidding online in real time 6. Online Payment Service > Make a Payment Now 7. Shipping Service > search "grader" > Get Shipping Estimate 3

4 RBA Website - Interactions 25,000 to 50,000 sessions / day Common user interactions Searches for equipment Registrations for bidding at auctions Online bidding during auctions Online payments Shipping quotes Project - New Website Problem: It is very difficult to implement further enhancements and changes to the current website. Solution: A complete website rewrite and new hardware infrastructure. 4

5 System What Information is Exchanged 5

6 How is Information Exchanged What are Web Services Application programming interfaces (APIs) that can be accessed over the Internet, and executed on a remote system hosting the requested services Remote procedure calls over HTTP Operations and data coded in XML messages Synchronous - calling process waits for reply before continuing e.g. Call GetExchangeRates (Date, Curr1, Curr2, outrates()) Asynchronous - calling process continues and listens for a reply e.g. Call RequestExchangeRates (Date, Curr1, Curr2) Listen ReceiveExchangeRates (outrates()) see 6

7 RBA Web Services Category RBA Web Service Type Originator Customers RbaCreateWebCustomerWS asynch w eb Customers RbaFetchCustomerWS synch w eb Customers RbaUpdateCustomerWS asynch w eb Customers RbaFetchCustomerForInternetBidderWS synch w eb Web activity RbaCreateInteractionWS asynch w eb Lists RbaFetchDaily ExchangeRatesWS synch w eb Diagnostics RbaLogWS synch O Testing What Does the PM Want Project Manager: Your task is to test the web services. Do you have test cases? How long will it take you? 7

8 The Real Question How will we know when the web services are good enough for deployment to production? Acceptance Criteria Stakeholders Persons who have interest in the project Business users Management Project Manager Developers Operations Support... 8

9 Requirements Ideally, get perfect requirements, convert them to test cases, and off you go... But, if you don't have them, use simple requirements analysis techniques to obtain some customer requirements Capturing Wild Requirements For Testing Trevor Atkins, 2008, FURPS classification of software quality attributes F U R P S FURPS is an acronym representing a model for classifying software quality attributes (functional & non-functional requirements): Functionality - Feature set, Capabilities, Generality, Security Usability - Human factors, Aesthetics, Consistency, Documentation Reliability - Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure Performance - Speed, Efficiency, Resource consumption, Throughput, Response time Supportability - Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Installability, Localizability, Portability 9

10 Quality Attributes for RBA Web Services Functionality Do the web services correctly store and retrieve data, as specified? Do the web services have sufficient error handling capabilities? Performance Response time Does the response time meet the published requirements? Throughput How many transactions per minute can the system process? Reliability Recoverability - In the event of a component failure, does the system recover from the failure and maintains data integrity? Supportability Monitoring How can Operations monitor the web services and detect and diagnose problems? Maintainability How can system components be replaced in a fully operational system and what outage (if any) is required? Usability Consistency - Is the design of the web services consistent across the set of services? Specification of RBA Web Services For example, RbaFetchDailyExchangeRatesWS Call This synchronous call is used to fetch exchange rate information. Initiating this web service will provide a daily exchange rate. The following inputs should be provided to the web service: ConversionDate: Date for which the exchange rate is required (YYYY-MM-DD) FromCurrency: From currency code ToCurrency: To currency code (refer to Currency LOV for valid values) 10

11 Sample Request for RbaFetchDailyExchangeRatesWS <soap:envelope xmlns:soap=" <soap:body xmlns:ns1=" RatesInput"> <ns1:fetchdailyexchangeratesinput> <ns1:conversiondate> </ns1:conversiondate> <ns1:fromcurrency>usd</ns1:fromcurrency> <ns1:tocurrency>cad</ns1:tocurrency> </ns1:fetchdailyexchangeratesinput> </soap:body> </soap:envelope> Sample Output from RbaFetchDailyExchangeRatesWS <outputvariable><part xmlns:xsi=" name="payload"><fetchdailyexchangeratesoutput xmlns=" put"> <FetchDailyExchangeRates> <FromCurrency>USD</FromCurrency> <ToCurrency>CAD</ToCurrency> <ConversionDate> T00:00: :00 </ConversionDate> <ConversionRate>1.23</ConversionRate> </FetchDailyExchangeRates> <ReturnStatus/> <ErrorMessage/> </FetchDailyExchangeRatesOutput> </part></outputvariable> 11

12 Manual Functional Testing Testing web services using Oracle Enterprise Manager Web-based interface called "BPEL Console All messages are XML Manual testing involves hand-crafting XML copying / pasting XML examples executing individual calls See Testing the ESB Services Demo - Basic test with Fetch Exchange Rates Using Oracle BPEL Console... Fetch an exchange rate 12

13 13

14 Demo - Update an Address Using Oracle BPEL Console... Fetch account, identify Site ID, Site Use ID, assoc. OVN (Oracle version number) Requires XML template stored somewhere to copy / paste document to window Update account: paste in numbers, modify data 14

15 Limitations of Manual Testing Very time and effort intensive Oracle does not remember your data -> use Notepad copy/paste Very error prone: esp. with Oracle record version number (OVN) changing with each update Questioning the applicability of testing async processes, as it is all executed within the same system (SOA), excluding the network BUT - Finds Bugs!! Automated Functional Testing with SOAPUI SOAPUI = tool for Web Service Testing Provides framework for building SOAP test cases, with built-in organization and storage Features: Import WSDL to automatically generate test cases based on web service definitions Organization into Test Suites / Test Cases / Test Steps Parameterization and loops to allow for more dynamic tests 15

16 Demo - Fetch Exchange Rates using SOAPUI 1. Import WSDL (web service definition) for FetchExchangeRates to SOAPUI 2. Press the Run button 3. Observe results or errors 16

17 17

18 Demo - Update Account using SOAPUI Fetch account Use parameters to copy ids into Update Demo fetch / update / catch chain Screenshots: WSDL Import Property copy tool Create listener for callback Custom Test Automation Technique Data-driven tests from Excel Using Excel as a data source Test cases stored with Web service definitions (XML) Multiple test cases created easily (clone & modify) Formulas allow cases to be linked without programming Results are saved automatically Smoke test (basic test cases for each service) MySQL database JDBC connection to MySQL for logging & post analysis Logs all transactions, inputs and output, timestamps Complete record of all XML to pass to developers More useful for performance tests reporting 18

19 "Smoke Test" Workbook File SmokeTest xls One sheet for each web service XML for input and output Any number of test cases, inputs easily cloned Outputs stored automatically, timestamped Calculated result PASS or FAIL Test case ID, name and description for reporting 19

20 Test Case Linking and Unique Values CreateWebCustomer Formulas to generate unique values, e.g. address FetchCustomer WebAccount linked to a cell in CreateWebCustomer FetchByRBNumber Error handling test case - incorrect CustomerNumber UpdateCustomer Linked to FetchCustomer 20

21 Summarize Results of Test Runs Quick summary of all test workbooks in a folder Workbook SummarizeTestResults.xls with macro code and MyMenu Pivot table of Web services and results Find workbooks with FAIL results 21

22 More Examples of Functional Tests File \UpdateCustomer\PersonalAccountWebProfile xls 7 test cases for FirstAndLastName Data type validation for DateOfBirth Missing input values in Validation All linked from CreateCustomer and FetchCustomer Executed many times - regression 22

23 Performance Requirements Expected throughput on the SOA server (all transactions) Expected response time from individual web services Regular vs peak times Recovery from failure Load Test Environment Separate computers - load generator - listener for async callback Simplified listener - logs all messages from the load generator - listens to and logs all messages from Oracle SOA - does not do any analysis System monitors - CPU, memory, disk i/o 23

24 Load Test Execution Load Test Results <WOI Load Test Details xls> Response time - all web services - individual web services Error analysis - Error message count - Async callback not received -> all messages tagged with unique id, db used - Timeout on call -> direct: load test logs -> indirect: times increasing to ceiling -> indirect: significant drop in throughput 24

25 Summary of Load Test Runs <OWS Load Test Summary.xls> 38 Performance tests frequent Smoke tests over a period of 7 months 25

26 Results & Benefits For system quality Confirmed functionality through regression testing Found software bugs - not many, but difficult ones to find Found performance bottlenecks Simple and convenient Smoke test to identify errors in new builds Additionally, for the PM, the continuous performance testing helped the technical team by: providing useful information to developers and DBAs helping to troubleshoot complex issues enforcing a disciplined deployment procedure 26

27 Conclusion Continuous performance testing is an important component of the testing effort SOAPUI is a great and very flexible tool for testing web services RBA custom data-driven testing technique helped with regression testing and performance testing Thank You Paul Vasak Chris Clarke pvasak@rbauction.com cclarke@rbauction.com 27