Wind River Test Management for Android. Ean Hou Project Manager, Solution and Service

Transcription

1 Wind River Test Management for Android Ean Hou Project Manager, Solution and Service

2 Android Software Validation/Test Automation Challenges Volume More apps in more markets More SKUs across converging device classes Increasing Android fragmentation User Experience ROI of Software quality Maintaining quality over time Perception and reality of security challenges Complexity Constant Android innovation New protocols and extensions Shift from touch to voice MMI and cloud options HTML 5 applications Speed (TTM) 6 months or less 2 nd generation expectations Efficiency gains expectations Turnkey choices but no test or documentation 2

3 Android Market Context : Complexity Community contribution-based operating system Growing in complexity Google s releases are STABLE but NOT BUG FREE Android Market offers a large number of applications that may add additional ERRORS To start, we're jointly announcing today that new devices from these partners will receive the latest Android updates for 18 months after first launch if the hardware allows. Hugo Barra, Product Management Director, Android at Google Android update policies create challenges Efficient Test Automation is key to enable Android software quality 3

4 Android Market Context : Time Market dynamics - increased competition Average handset development cycle just over 4 months First Jellybean devices out <2 months after release Efficient Test Automation is key to enable Android software quality 4

5 Life Without Test Management for Android Open Source Test Suites Test Results Package Test Programs Test Results Compliance Test Suites Test Results GUI Test Scripts (UxTDK) Test Results In-House Testing Test Results Multiple Testers Different Result Formats Confusion 5

6 Life With Test Management for Android TM Library TM Library TM Library TM Library TM Library Open Source Test Suites Test Results Package Test Programs Test Results Android Compliance Test Suites Test Results Test Management Data Base GUI Test Scripts (UxTDK) Test Results In-House Testing Test Results Homogeneous Results Clarity 6

7 Wind River Test Management for Android Key Concepts Integrates testing efforts in one application Many Tests One button to press One report to read Seamless 3 Rd party Tests/Tools integration Out-of-the-box integration with requirements management, defect tracking Comprehensive CLI fits with any existing test infrastructure Test Case Reusability Apply identical tests to different targets with minimal changes Advanced Reporting Capabilities All results and artifacts stored in one centralized database Easy to generate reports Save up to 30% on test execution time Use thousands of out-of-the-box, pre-built automated tests Shorten time-to-market by automating & reusing existing tests 7

8 Architecture User Interface Web or CLI TM Server Send Chunk Send Logs Send Status Virtual Lab Manager DMEngine Use TM s built-in VLM to find, reserve and schedule tests to run on devices. Device Service Python Chunk Runner Board Objects Device Board object controls and communicates with the target. DB Reporting Tool Script Script Test scripts Test scripts make calls to the device through the board class 8

9 ANDROID DEVICE LIBRARIES APPLICATION APPLICATION FRAMEWORK LINUX KERNEL Communication Channel (Including Power Supply Control) RUN-TIME DRIVERS Test Execution Daemon (TED) Wind River Test Management for Android Wind River UX Test Development Kit (UXTDK) for Android TETware Monkey Tool Stress Tests Instrumentation Tests Standard Android Compatibility Test Suite (CTS) Wind River Hardware Abstraction Layer (HAL) Suite Audio Battery Bluetooth Alarm Browser Calculator Calendar Wind River Android Device Characterization Suite (ADCS) Bluetooth Camera Graphics Flashlight Framebuffer GPS Application Tests Camera Clock Contacts Location Media SMS Lights Power Mgmt. RIL Media Player Messaging Phone Settings Stability Transfer WiFi Optimized CTS Sensors Touchscreen Vibrator Camera Keyboard RTC WiFi Open Source Tests/Tools Wind River Test Suites/Tools LTP IOZone IPerf 9

10 Test Assets Test Suite / Tool Android Layer No. of Test Cases Wind River UXTDK Android Applications / Ux 196 Performance Tests Android System 27 Stress Stability Tests Android Applications / System 38 Localization Android Applications / System 35 Android Application/UX layer Wind River UX Test Development Kit (UXTDK) Wind River ADCS Android Application Framework 369 App Stress Tests (using Monkey) Android Applications / System 8 Update Tests System 3 Battery Tests Android Applications/System 2 USB Mass Storage Tests Android Application Framework 5 2D/3D OpenGL Android Applications/System 31 Stagefright Android Multimedia Framework 129 Dalvik Tests Android Dalvik Virtual Machine 87 Other Instrum. Tests Android Application Framework 651 Android Middleware layer Wind River Android Device Characterization Suite Core Tests (Instrumentation) Android Application Framework 232 CTS (optimized execution) Android Application Framework 16,865 CDD (Compatibility Definition Doc) Android Application Framework 255 Wind River HAL (TETware) Linux Device Driver / Android HAL 2,326 LTP Linux/Core 998 IOZone Linux/Core 8 IPerf Linux/Core 1 Android Hardware Abstraction layer Wind River Hardware Abstraction Layer Tests Android CTS 16,865 (ICS) Standard Linux Tests 1,007 Wind River authored Tests 3,266 Integrated Open Source Tests 1,130 10

11 Test Script Concepts Test Scripts encapsulate the knowledge to execute tests and collect their results: Test Script Test cases Parameters Test Results Test scripts implement the logic needed to call (or wrap) other test cases and test suites. A test script can call a test suite that runs hundreds of test cases. Custom test scripts can also stand alone (i.e. we don t necessarily need to wrap something else). Parameterization increases the number of tests one script can perform 11

12 Test Scripts Reusability Abstraction layer makes test scripts re-usable across multiple target types. Test Scripts don t communicate directly with the target, instead they go through a board class. By replacing the board class, our test scripts now apply to new targets. Test the Android system on multiple targets, with minimal or no changes. 12

13 Requirement Management: Excel, IBM, Rational, DOORS Build Automation: IBM Buildforge, Jenkins, Electric Cloud Central repository to store manual and automated test cases Flexible and Powerful Framework Dashboard views Virtual lab manager Reports generator Easy and clear interpretation of individual test case results Easily integrated with existing build and test infrastructure Designed for embedded Systems & mobile devices Defect Tracking: JIRA, Rational, ClearQuest Test results parsed for each test case and each step within test case STDOUT for each test case step captured individually 13

14 Test Execution Comprehensive Test Automation Easy Test Plan creation with test filtering Simply select a specific test or test group One button click executes entire test suite Configuration Management support, for SVN and other CM tools Easy to use interfaces Graphical User Interface (GUI) Easy to use interface web browser interface makes TM server accessible from virtually anywhere Command Line Interface (CLI) Very useful for running tests regularly from a cron job 14

15 Test Results Centralized Database Collects and stores test results and test artifacts Test logs, stdout and stderr files, screenshots, etc All results stored in a single database for easy report generation The stored results can also include performance metrics Exporting Results Test Results can be exported using CSV/XML formats 15

16 Test Results Testrun Details Artifacts Profiler Data Detailed Logs Test Results and Performance Metrics 16

17 Reports Various reporting engines can be used to generate reports, since all the results/artifacts are stored in the database. Reports types: 1. Final Report: listing overall test results for one test run. 2. Failure Report: listing only the failed tests. 3. Comparative Report: between multiple test runs. 4. Performance Report: to show performance metrics over time. 5. Detailed Test Script: containing all keyvals and artifacts for a specific test case 6. Profilers Report: showing data being captured by runtime profilers 17

18 Customer Testimonials "To conduct software testing for Android, it would have required our teams to spend extra time and resources during the testing process. Instead, we're leveraging Wind River Test Management for Android's out-of-the-box tests for performance, stability and compliance. Wind River not only delivered an automated testing solution that fit our needs but also came to the table with vast Android expertise. Ultimately, we will be able to increase product quality and meet tight time-to-market deadlines." Hiromi Watanabe, general manager of SoC Software Platform Division at Renesas Electronics Corporation. "In these two years, Leadcore has made a focused effort on Android smartphone solutions for TD/GGE multi-mode. We were in search of a trusted partner with Android experience who could help us navigate through the complexities of Android development and testing, including Android compliance. Wind River has the proven Android expertise and deep knowledge of Chinese operator requirements. Being able to leverage the combination of both was essential for our success. Sun Yuwang, president of Leadcore. 18

19 Summary Advanced Reporting Capabilities Complete Automated Testing Solution - Standard final report - Failures report - Comparative report - Performance report - Detailed Script report - Profiler Report Easy-to-Use Web-Based GUI Test plan creation wizrards Out-of-the-box Tests Profiler Mechanism - Tests for Performance, Reliability and Compliance - Layered testing from low-level to user interface - Reusability, same tests on different targets - Integrated open source test Supports testing multiple devices in parallel Powerful Comprehensive Customizable 19

20 Wind River UX Test Development Kit for Android Overview

21 UXTDK for Android What is UXTDK? Innovative solution to automate manual testing A pre-built set of changeresitant, reusable tests ready to run! An Eclipse-based authoring environment to eaisly create test scripts Click! Type! Gesture! Look! Verify! Targets can run on a physical device or emulator UXTDK Test are fully managed and run automatically using Wind River Test Management 21

22 Current practice : Record and Replay Record: test steps are captured by the automation tool Replay: recorded test steps are executed on the AUT Automation tool compares actual results to expected set Intel Software Professionals Conference - Intel Confidential 22

23 UXTDK Tests : Maintainability The case against record & replay Recording-based tests suffer from many limitations Poor stability: They break if anything changes Poor portability: can only be used on the version they were recorded on Poor understandability: recordings are not easy to annotate with explanations of why/what they do Poor debugability: recordings break but are not easily diagnosed - where did they go wrong and why Poor applicability: recordings usually have to be started in very specific conditions, or they break 23

24 A very simple test public class TestCalculator extends TDKTest public void BasicCalculatorTest () { initfortest("calculate and check 10+(23-45)=*67/8 + (9*0.1)="); TDKCalculatorApp pcalc = TDKCalculatorApp.getAttachedCalculator(this); pcalc.doexpression("10+(23-45)=*67/8 + (9*0.1)=", true, "-99.6"); pcalc.disconnect(); } 24

25 Pre-built UXTDK Android Test Assets UxTDK comes with hundreds of abstract tests for standard Android applications: Included tests normally run for standard Android (maintained by Windriver) Available tests can be adjusted for specialized targets, under contract Each test can be run in many different initial states, many with parameters that turn each test into many tests.

26 How UXTDK works TDK Agent talks to OS TDK Agent s actions look human for target device 26

27 UxTDK Test Layer Architecture 27

28 UXTDK: Award winning technology 28

29