Software Testing ETS 200

Transcription

1 Software Testing ETS Introduction, Chapters 1, 3 Prof. Per Runeson

2 Example: Volvo City Safety

3 Select an app

4 Test the app Wifi: on/off Bluetooth: on/off 3G: on/off SD-card: size/brand Language: sv/en/ty/fr/cn How many combinations? Talk pairwise!

5 Test complexity Example 40 variables, 2 levels -> combinations 3 tests/second -> 3000 years of test!

6 What is software testing? The process of evaluating a program or a system Verifying that the product is right Validating that the right product is developed Evaluate quality Find defects = The testing paradox

7 Testing Takes Time Reports estimate that regression testing consumes as much as 80 percent of the testing budget [1] and can consume up to 50 percent of the cost of software maintenance [2], [3]. Chittimalli and Harrold, 2009 Chittimalli, P, & Harrold, M 2009, 'Recomputing coverage information to assist regression testing', IEEE Transactions On Software Engineering, 35, 4, pp

8 Testing not prioritized by students Emerging results from 40 software engineering students show that students overestimate their testing time threefold, and 50% of them test as little as 4% of their time, or less. Beller, Gousios, Zaidman, 2015 M. Beller, G. Gousios, and A. Zaidman. How (much) do developers test? 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16-24, 2015, Volume 2, pages IEEE Computer Society, 2015.

9 Levels of Testing in the development process Requirements Acceptance test High-level design System Detailed design Integration test Code Unit test See figures 1.7 and 1.8

10 Agile methods increase testing [

11 Product Lines increase testing [Pohl05]

12 Quality attributes ISO 9126

13 Aim of the course (from KA) The objective of the course is to give basic and advanced knowledge and skills within testing for large-scale development of systems completely or partly based on software. The course gives practical skills in methods and techniques for software testing. The course also aims to teach the student to search and assess knowledge at the research front in software testing

14 Goals: Knowledge & understanding For a passing grade the student must be able to define basic concepts and principles within software testing give an account of the most common techniques for software testing be able to describe the relation between the software testing process and other processes in the product lifecycle be able to describe the relation between verification and the requirements of a software system in terms of functionality and quality be able to report an in-depth area of own choise, and demonstrate theoretical depth or practical application, related to the state-of-art of the area.

15 Goals: Skills and abilities For a passing grade the student must be able to motivate the choise of test strategy for a software system be able to analyse and create test cases for white box and black box testing be able to perform an inspection be able to apply a systematic approach for software testing

16 Goals: Judgement and approach For a passing grade the student must understand the complexity of the task to test a software system and be aware of the costs and impact of different testing activities during the development of a software product. be able to value and synthesize the information given in published scientific articles about software testing.

17 During the break Send me an or write a post-it note: I would like to learn about testing

18 Lectures Theory + discussions Cover the basic parts of software testing 1. Introduction, unit testing 2. White-box testing 3. Black-box testing 4. Reliability, Inspections 5. Lifecycle, documentation 6. Organization, tools 7. Metrics, TMM Overview Technical Technical / Managerial Managerial E c o n o m i c s

19 Lab sessions (Th 8, 13 or 15) Preparation, Execution, Report (1+2 and 4) 1. White-box testing 2. Black-box testing 3. Debugging 4. Inspection and estimation Report via to & Subject line: labx and your student ID. Sign up via course web

20 Project Learn a specific area of software testing (research/evaluation/practical) Collect and summarize research information Critical thinking beyond the written information Present information in structured writing Form project groups and sign-up via course web (by L2 Th 19/1) More info in lecture 2 Choose project topic by Fr 20/1

21 Examination Written exam based on the book, articles, lectures, projects and lab sessions Lab sessions (approved) Project (F / G / VG); VG gives 5 extra exam points for those who pass. Exam Final grade points

22 Literature NEW BOOK this year E-book via LUBsearch /book/ / articles + peer student reports

23 Teachers and staff Per Runeson course resp, lecturer, exam Alma Orucevic Alagic, Lab 1-2 Sardar Muhammad Sulaman, Lab 3-4, Projects Hussan Munir, Projects Lena Ohlsson administration Det går inte att visa den här bilden just nu.

24 Student Representatives C: TBD D: TBD X: TBD

25 Types of Testing Level of detail acceptance system integration unit portability maintainability efficiency usability reliability functionality white box black box Accessibility Characteristics

26 Definitions (Ch 1) Failure is an event, error is an erroneous state of the software caused by a fault, caused by a human error Human error mistake by engineer Fault / Defect / Bug anomaly in the software Error faulty state Failure failed function, caused by an error Debugging / Fault localization localizing, repairing, retesting

27 Definitions (cont) A test case consists of: A set of inputs Expected outputs (Execution conditions) Additional attributes: Purpose Scope Origin Priority

28 Definitions (cont) A test oracle is the principle or mechanism by which you recognize a problem It provides the expected result for a test, e.g. Specification document Formula Computer program Person In many cases it is very hard to find an oracle Even the customer and end user might not be able to tell which is the correct behaviour

29 Unit testing definitions Unit testing refers to testing program units in isolation Naik and Tripathy

30 Unit testing definitions a test, executed by the developer in a laboratory environment, that should demonstrate that the program meets the requirements set in the design specification. Tim Koomen and Martin Pol

31 Unit testing definitions unit testing tests individual software components or a collection of components. Testers define the input domain for the units in question and ignore the rest of the system. Unit testing sometimes requires the construction of throwaway driver code and stubs and is often performed in a debugger. James Whittaker

32 Unit testing in practice Testing the smallest separate module in the system Some stress that it s the smallest specified module Technically oriented, with in/out parameters. In a scaffolding environment or in an almost-complete system environment 22 focus software testing A Survey of Unit Testing Practices Companies participated in a survey to define unit testing and evaluate their strengths and weaknesses at applying it. Others can use the survey to judge and improve their own practices. Per Runeson, Lund University Unit testing is testing of individual units or groups of related units. 1 You know the definition by the book, but what does it mean to you? What are a company s typical strengths and weaknesses when applying unit testing? Per Beremark and I surveyed unit testing practices on the basis of focus group discussions in a software process improvement network (SPIN) and launched a questionnaire to validate the results. I aimed to go beyond standard terminology definitions and investigate what practitioners refer to when they talk about unit testing. Based on this common understanding, I also investigated unit testing practices strengths and weaknesses. The survey revealed a consistent view of unit testing s scope, but participants didn t agree on whether the test environment is an isolated harness or a partial software system. Furthermore, unit testing is clearly a developer issue, both practically and strategically. Neither test management nor quality management seem to impact unit testing strategies or practices. Unit tests are structural, or white-box based, but developers rarely measure their completeness in terms of structural coverage. Most of the companies I surveyed desired unit test automation but had trouble spreading good practices across companies. This survey is an indication of unit testing in several companies. You can use the questionnaire at your own company to clarify what you mean by unit testing, to identify the strengths and weaknesses of your unit testing practices, and to compare with other organizations to improve those practices. The survey SPIN-syd is a noncommercial network focused on software process improvement issues. It comprises representatives from 50 companies with software as a major part of their business. The companies range from consultancy firms with one employee to regional branches of multinational companies with hundreds of developers. The network represents various application domains with a focus on embedded systems. Lund University researchers and PhD students also belong to the network. The network has a monthly three-hour meeting and occasionally launches working groups around specific themes. Typically, 10 to 15 companies are represented at each monthly meeting, depending on the topic and the companies workloads. We conducted the survey at two meetings. During the first meeting, we held a focus group discussion about unit testing. Participants included 17 representatives from 12 companies, the moderator (a software quality manager), and the secretary (me). Table 1 lists the participating companies characteristics. IEEE SOFTWARE Published by the IEEE Computer Society /06/$ IEEE

33 Static unit testing = reviews (lecture 4)

34 Dynamic unit testing Test driver Results Call and pass input parameters Output parameters Unit Under Test Call Acknowledge Call Acknowledge Print Stub Stub Print Figure 3.2 Dynamic unit test environment.

35 Sources for test case selection Specifications (requirements, functional) Source code Input and output domains Operational profile (usage) Fault models

36 Techniques for test case selection Control flow testing Data flow testing Domain testing Functional program testing White box Black box

37 juinit unit test framework Extend TestCase class with Java test code Assert output vs expected output If you haven t used it download and try

38 This week Read course program Lecture 2 Thu (E:B) Lab Sign up for lab session == registration Prepare lab 1 (assignments) Check course web (often) Project Read project description ( Projects in Software Testing ) Decide subject (deadline: Friday 20/1) Sign up for project group Get the textbook and articles!

39 Project Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc. Practical: Evaluate a testing technique/tool or use a testing technique to verify a system or design. The main focus can be either on evaluating the technique/tool or on performing testing of a system or a prototype.

40 Project important dates Decide on a subject and sign up: Friday 20/1 (this week!) Outline: Tuesday 31/1 Report: Friday 24/2 Presentation: Thursday 2/3 (Update report: Friday 17/3)