2 What is Software testing? Examination of a software unit, several integrated software units or an entire software package by running it. execution based on test cases expectation reveal faults as failures Failure incorrect execution of the system usually consequence of a fault Fault/defect/bug result of a human error
3 Objectives of testing To find defects before they cause a production system to fail. To bring the tested software, after correction of the identified defects and retesting, to an acceptable level of quality. To perform the required tests efficiently and effectively, within budgetary and scheduling limitation. To compile a record of software errors for use in error prevention (by corrective and preventive actions)
4 Software Testing Process Test Planning Test Design Test Implementation Test Execution Planning include completion criteria (coverage goal) Design - approaches for test case selection to achieve coverage goal Implementation - find for test cases input/output data state before/after test procedure Execution run tests Result verification pass or fail? Coverage? Test Library Management maintain relationships keeping track, etc. Results Verification Test Library Management
5 What is a Test Case? Test Case is a pair of <input, expected outcome> For state-less systems (e.g. a compiler) Test cases are very simple Outcome depends solely on the current input For state-oriented (e.g. ATM) Test cases are not that simple. A test case may consist of a sequences of <input, expected outcome> The outcome depends both on the current state of the system and the current input ATM example: < check balance, $ >, < withdraw, amount? >, < $200.00, $ >, < check balance, $ > Various ways input may be specified
6 Expected Outcome An outcome of program execution may include Value produced by the program State Change A sequence of values which must be interpreted together for the outcome to be valid A test oracle is a mechanism that verifies the correctness of program outputs Generate expected results for the test inputs Compare the expected results with the actual results of execution of the IUT
7 Levels of Testing Unit testing Individual program units, such as procedure, methods in isolation Integration testing Modules are assembled to construct larger subsystem and tested System testing Includes wide spectrum of testing such as functionality, and load Acceptance testing Customer s expectations from the system
8 Levels of Testing New test cases are not designed Test are selected, prioritized and executed To ensure that nothing is broken in the new version of the software
9 When to automate testing? (1) The benefits of test automation need to be greater than the (expensive!) costs of automation. General rule of thumb: it is expected that tests will have to be run many times regression testing configuration testing conformance testing agile development process capacity / stress testing performance measurements
10 When to automate testing (2) Automated testing is especially beneficial if the tests need to be re-executed quickly Frequent recompiles Large number of tests Using an agile development process An automated test can be duplicated to create many instances for capacity / stress testing.
11 Example: Test-First process in XP
12 When NOT to automate Initial functional testing Automated testing is more likely to find bugs introduced by changes to code or the execution environment, rather than in new functionality. Automated test scripts may not be ready for first software release. Situations requiring human judgment to determine if system is functioning correctly.
13 Types of Testing Tools Test Planning and Management Create/maintain test plans integrate with project plan Maintain links to Requirements/Specification generate Requirements Test Matrix Reports and Metrics on test case execution Tracking of history/status of test cases defect tracking
14 Types of Testing Tools Test Design & Implementation Automatic creation of test cases Based on test design approaches graph based data flow analysis logic based... Very few concrete usable tools Random test data generator Stubs/Mocks
15 Types of Testing Tools Test Execution Test Drivers and Execution Frameworks Run test scripts and report result e.g. JUnit Runtime test execution assistance memory leak checkers comparators
16 Types of Testing Tools Test Performance assessment Analysis of the effectiveness of test cases for extend of system covered Coverage analyzers report on various levels of coverage Analysis of the effectiveness of test cases for bug detection mutation testing
18 Types of Test Tools Capture and Replay For user interface testing, one approach to automating tests is, after the system is working, record the input supplied by the user, and capture the system responses. When the next version of the software needs to be tested, play back the recorded user input and check if the same responses are detected as are stored in the capture file. Benefits: relatively simple approach, easy to do Drawbacks: very difficult to maintain specific to one environment
19 Tool support at different levels Unit testing Tools such as JUnit Integration testing Stubs, mocks System testing Security, performance, load testers Regression testing Test Management tools (e.g. defect tracking,...)
20 What do we need to do automated testing? Test script Test Case Specification Actions to send to system under test (SUT). Responses expected from SUT. How to determine whether a test was successful or not? Test execution system Mechanism to read test script, and connect test case to SUT. Directed by a test controller.
21 Test Architecture (1) Includes defining the set of Points of Control and Observation (PCOs) Test controller Test script Test Execution System PCO SUT A PCO could be a particular method to call a device interface a network port etc.
22 Test Architecture (2) The test architecture will affect the test script because it may be significant as to which PCO is used for an action or response. Test controller Test controller m m PCO PCO 1 PCO 2 SUT SUT
23 Potential PCOs Determining the PCOs of an application can be a challenge. Potential PCOs: Direct method call (e.g. JUnit) User input / output Data file input / output Network ports / interfaces Windows registry / configuration files Log files Temporary files or network ports Pipes / shared memory
24 Potential PCOs (2) 3 rd party component interfaces: Lookup facilities: network: Domain Name Service (DNS), Lightweight Directory Access Protocol (LDAP), etc. local / server: database lookup, Java Naming and Directory Interface (JNDI), etc. Calls to: remote methods (e.g. RPC) Operating System For the purposes of security testing, all of these PCOs could be a point of attack.
25 Distributed Test Architecture (1) May require several local test controllers and a master test controller Master Test controller PCO Local Test controller SUT Component 1 Local Test controller PCO SUT Component 2
26 Distributed Test Architecture (2) Issues with distributed testing: Establishing connections at PCOs Synchronization Where are pass/fail decisions made? Communication among test controllers
27 Choosing a test architecture User Browser Web Server mouse clicks / keyboard HTTP / HTML SQL Data base
28 Choosing a Test Architecture Testing from the user s point of view: Need a test tool to simulate mouse events, or keyboard input Need to be able to recognize correct web pages Small web page changes might require large changes to test scripts. Testing without the browser: Test script would send HTTP commands to web server, and check HTTP messages or HTML pages that are returned. Easier to do, but not quite as realistic.
29 Test Scripts What should the format of a test script be? tool dependent? a standard test language? a programming language?
30 Test Script Development Creating test scripts follows a parallel development process, including: requirements creation debugging configuration management maintenance documentation Result: they are expensive to create and maintain
31 Making the automation decision (1) Will the user interface of the application be stable or not? To what extent are oracles available? To what extent are you looking for delayed-fuse bugs (memory leaks, wild pointers, etc.)? Does your management expect to recover its investment in automation within a certain period of time? How long is that period and how easily can you influence these expectations? Are you testing your own company s code or the code of a client? Does the client want (is the client willing to pay for) reusable test cases or will it be satisfied with bug reports and status reports? Do you expect this product to sell through multiple versions?
32 Making the automation decision (2) Do you anticipate that the product will be stable when released, or do you expect to have to test Release N.01, N.02, N.03 and other bug fix releases on an urgent basis after shipment? Do you anticipate that the product will be translated to other languages? Will it be recompiled or re-linked after translation (do you need to do a full test of the program after translation)? How many translations and localizations? Does your organization make several products that can be tested in similar ways? Is there an opportunity for amortizing the cost of tool development across several projects?
33 Making the automation decision (3) How varied are the configurations (combinations of operating system version, hardware, and drivers) in your market? (To what extent do you need to test compatibility with them?) What level of source control has been applied to the code under test? To what extent can old, defective code accidentally come back into a build? How frequently do you receive new builds of the software? Are new builds well tested (integration tests) by the developers before they get to the tester?
34 Making the automation decision (4) To what extent have the programming staff used custom controls? How likely is it that the next version of your testing tool will have changes in its command syntax and command set? What are the logging/reporting capabilities of your tool? Do you have to build these in?
35 Making the automation decision (5) To what extent does the tool make it easy for you to recover from errors (in the product under test), prepare the product for further testing, and re-synchronize the product and the test (get them operating at the same state in the same program). In general, what kind of functionality will you have to add to the tool to make it usable? Is the quality of your product driven primarily by regulatory or liability considerations or by market forces (competition)? Is your organization subject to a legal requirement that test cases be demonstrable?
36 Making the automation decision (6) Will you have to be able to trace test cases back to customer requirements and to show that each requirement has associated test cases? Is your company subject to audits or inspections by organizations that prefer to see extensive regression testing? If you are doing custom programming, is there a contract that specifies the acceptance tests? Can you automate these and use them as regression tests? What are the skills of your current staff?
37 Making the automation decision (7) Do you have to make it possible for non-programmers to create automated test cases? To what extent are cooperative programmers available within the programming team to provide automation support such as event logs, more unique or informative error messages, and hooks for making function calls below the UI level? What kinds of tests are really hard in your application? How would automation make these tests easier to conduct?
38 Suggested reading Henk Coetzee, Best Practices in Software Test Automation, (2005) on line at C. Kaner, Architectures of Test Automation. (2000). On line at: C. Kaner, Improving the maintainability of automated test suites, Software QA, Vol. 4, No. 4 (1997). On line at: J. Bach, Test automation snake oil, Proceedings of 14th Int l conference on Testing Computer Software (revised 1999). On line at:
1 Software Testing Testing types Software Testing 2 References Software Testing http://www.testingeducation.org/bbst/ IBM testing course available through the Academic Initiative: Principles of Software
SOFTWARE TESTING TRAINING COURSES CONTENTS 1 Unit I Description Objectves Duration Contents Software Testing Fundamentals and Best Practices This training course will give basic understanding on software
Automating Security Testing Mark Fallon Senior Release Manager Oracle Some Ground Rules There are no silver bullets You can not test security into a product Testing however, can help discover a large percentage
GLOBAL JOURNAL OF ENGINEERING SCIENCE AND RESEARCHES A LITERATURE SURVEY ON DESIGN AND ANALYSIS OF WEB AUTOMATION TESTING FRAMEWORK - SELENIUM Revathi. K *1 and Prof. Janani.V 2 PG Scholar, Dept of CSE,
Course Objectives: This course is designed to train the fresher's, intermediate and professionals on testing with the concepts of manual testing and Automation with Selenium. The main focus is, once the
CS 451 Software Engineering Winter 2009 Yuanfang Cai Room 104, University Crossings 215.895.0298 firstname.lastname@example.org 1 Testing Process Testing Testing only reveals the presence of defects Does not identify
FSW QA Testing Levels Definitions 1. Overview This document is used to help determine the amount and quality of testing (or its scope) that is planned for or has been performed on a project. This analysis
TORRY HARRIS BUSINESS SOLUTIONS Unit Testing webmethods Integrations using JUnit Practicing TDD for EAI projects Ganapathi Nanjappa 4/28/2010 2010 Torry Harris Business Solutions. All rights reserved Page
T-76.5613 Software Testing and Quality Assurance Lecture 2, 13.9.2006 Basic Testing Concepts and Terminology Juha Itkonen SoberIT Contents Realities and principles of Testing terminology and basic concepts
Software testing cmsc435-1 Objectives To discuss the distinctions between validation testing and defect testing To describe the principles of system and component testing To describe strategies for generating
Latest Trends in Testing Ajay K Chhokra Introduction Software Testing is the last phase in software development lifecycle which has high impact on the quality of the final product delivered to the customer.
International Journal of Advanced Engineering Research and Science (IJAERS) Vol-2, Issue-11, Nov- 2015] Survey on Automation Testing Tools for Mobile Applications Dr.S.Gunasekaran 1, V. Bargavi 2 1 Department
16 Example of Standard API System Call Implementation Typically, a number associated with each system call System call interface maintains a table indexed according to these numbers The system call interface
Software Automated Testing Keyword Data Driven Framework Selenium Robot Best Practices Agenda ² Automation Engineering Introduction ² Keyword Data Driven ² How to build a Test Automa7on Framework ² Selenium
ibaan ERP 5.2a Configuration Guide for ibaan ERP Windows Client A publication of: Baan Development B.V. P.O.Box 143 3770 AC Barneveld The Netherlands Printed in the Netherlands Baan Development B.V. 2002.
Chapter 8 Software Testing Summary 1 Topics covered Development testing Test-driven development Release testing User testing 2 Program testing Testing is intended to show that a program does what it is
Module : Introduction to Team Foundation Server Administration 2 Application Lifecycle Management & Team Foundation Server 200 ALM, or Application Lifecycle Management, refers to the end-to-end process
Smarter Balanced Assessment Consortium Recommendation Smarter Balanced Quality Assurance Approach Recommendation for the Smarter Balanced Assessment Consortium 20 July 2012 Summary When this document was
Requirements for Test Automation PNSQC 99 Douglas Hoffman Software Quality Methods, LLC. 24646 Heather Heights Place Saratoga, California 95070-9710 Phone 408-741-4830 Fax 408-867-4550 email@example.com
Testhouse Training Portfolio TABLE OF CONTENTS Table of Contents... 1 HP LoadRunner 4 Days... 2 ALM Quality Center 11-2 Days... 7 HP QTP Training Course 2 Days... 10 QTP/ALM Intensive Training Course 4
Test What You ve Built About Your Presenter IBM i Professional for 16 Years. Primary Focus is IBM i Engineering / Programming Well Versed in 2E. Well Versed in RPG (All Flavors) Well Versed in CM Products
SDLA-312 ISA Security Compliance Institute Security Development Lifecycle Assurance - Security Development Lifecycle Assessment v3.0 Lifecycle Phases Number Phase Name Description PH1 Security Management
Agile Test Automation James Bach, Satisfice, Inc. James@satisfice.com www.satisfice.com Examples of Agile Automation CD test system (300% improvement in CD package testing throughput in two weeks) Auction
A Guide To Evaluating a Bug Tracking System White Paper By Stephen Blair, MetaQuest Software Published: October, 2004 Abstract Evaluating a bug tracking system requires that you understand how specific
Load Testing with JMeter Presented by Matthew Stout - firstname.lastname@example.org JMeter Overview Java application for load testing and measuring performance Originally for web applications but has grown to support lots
Systems Analysis and Design in a Changing World, Fourth Edition Learning Objectives Describe implementation and support activities Choose an appropriate approach to program development Describe various
SOFTWARE TESTING Yogesh Singh MM CAMBRIDGE UNIVERSITY PRESS Contents List of Figures List of Tables Preface Acknowledgements xi xv xxi xxiii 1. Introduction 1 1.1 Some Software Failures 1 1.1.1 The Explosion
U.S. Navy Automated Software Testing Application of Standards to the Automated Test and Re-Test (ATRT) Effort Object Management Group (OMG) Technical Meeting June 2007 Approved for public release; distribution
MCSA Security + Certification Program 12 credit hours 270 hours to complete certifications Tuition: $4500 Information technology positions are high-demand occupations that support virtually all industries.
Web Application Testing Web Performance Testing Objectives of Performance Testing Evaluate runtime compliance to performance requirements Check different properties such as throughput (bits/sec, packets/sec)
Software Testing Rajat Kumar Bal Introduction In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch
Number 5.0 Policy Owner Information Security and Technology Policy Application Development Effective 01/01/2014 Last Revision 12/30/2013 Department of Innovation and Technology 5. Application Development
Ce document a été téléchargé depuis le site de Precilog. - Services de test SOA, - Intégration de solutions de test. 01 39 20 13 55 email@example.com www.precilog.com End to End Process Testing & Validation:
Znode Multifront - Installation Guide Version 6.2 1 System Requirements To install Multifront you need to have familiarity with Internet Information Services (IIS), Microsoft.NET Framework and SQL Server
Chap 1. Software Quality Management Part 1.1 Quality Assurance and Standards Part 1.2 Software Review and Inspection Part 1.3 Software Measurement and Metrics 1 Part 1.1 Quality Assurance and Standards
Course 50414B: Microsoft Windows PowerShell v2 For Administrators Course Details Course Outline Module 1: Introduction to PowerShell the Basics This module explains how to install and configure PowerShell.
Bringing Value to the Organization with Performance Testing Michael Lawler NueVista Group 1 Today s Agenda Explore the benefits of a properly performed performance test Understand the basic elements of
COMMONWEALTH OF PENNSYLVANIA DEPARTMENT S OF PUBLIC WELFARE, INSURANCE, AND AGING INFORMATION TECHNOLOGY STANDARD Name Of Standard: Defect Management and Reporting Domain: Application Domain Date Issued:
Understanding Computers Today and Tomorrow 12 th Edition Chapter 13: Program Development and Programming Languages Learning Objectives Understand the differences between structured programming, object-oriented
CHAPTER 2 About Network Data Collector The Network Data Collector is a telnet and SNMP-based data collector for Cisco devices which is used by customers to collect data for Net Audits. It provides a robust
Q SQA Labs Value Assured QUALITY ASSURANCE TESTING TOOLS QUALITY ASSURANCE TESTING TOOLS Quality Assurance refers to the steps taken to make sure that a company s products or services are of sufficiently
Oracle Insurance Policy Administration System Quality Assurance Testing Methodology An Oracle White Paper August 2008 Oracle Insurance Policy Administration System Quality Assurance Testing Methodology
17 th International Magnetic Measurement Workshop Toward the Next Generation Magnetic Measurement System Jerzy M. Nogiec Barcelona, Genesis With Fermilab s conventional magnet measurement system approaching
SysPatrol Server Security Monitor User Manual Version 2.2 Sep 2013 www.flexense.com www.syspatrol.com 1 Product Overview SysPatrol is a server security monitoring solution allowing one to monitor one or
www. routinebot.com AKS-Labs - Page 2 - It s often said that First Impression is the last impression and software applications are no exception to that rule. There is little doubt that the user interface
Software Engineering, Lecture 4 Decomposition into suitable parts Cross cutting concerns Design patterns I will also give an example scenario that you are supposed to analyse and make synthesis from The
Technical white paper Getting started with API testing Test all layers of your composite applications, not just the GUI Table of contents Executive summary... 3 Introduction... 3 Who should read this document?...
8 The one-of-a-kind bridges the gap between traffic data and end-use applications by providing a cost-effective way to get automated and immediate conversions between incompatible data formats and databases.
Achieving business benefits through automated software testing By Dr. Mike Bartley, Founder and CEO, TVS (firstname.lastname@example.org) 1 Introduction During my experience of test automation I have seen
Page 1 SECTION 4 TESTING & QUALITY CONTROL TESTING METHODOLOGY & THE TESTING LIFECYCLE The stages of the Testing Life Cycle are: Requirements Analysis, Planning, Test Case Development, Test Environment
Levels of Software Testing There are different levels during the process of Testing. In this chapter a brief description is provided about these levels. Levels of testing include the different methodologies
Oracle Database Client Quick Installation Guide 10g Release 1 (10.1.0.2.0) for Windows Part No. B13691-01 March 2004 This guide describes how to quickly install Oracle Database Client on Windows systems.
Release Notes for Version 1.5.207 Created: March 9, 2015 Table of Contents What s New... 3 Fixes... 3 System Requirements... 3 Stonesoft Appliances... 3 Build Version... 4 Product Binary Checksums... 4
Content of 6 Months Software Testing Training at EH1-Infotech Module 1: Introduction to Software Testing Basics of S/W testing Module 2: SQA Basics Testing introduction and terminology Verification and
Web Applications Testing Automated testing and verification JP Galeotti, Alessandra Gorla Why are Web applications different Web 1.0: Static content Client and Server side execution Different components
DiskBoss File & Disk Manager File Integrity Monitor Version 2.0 Dec 2011 www.flexense.com email@example.com 1 Product Overview DiskBoss is an automated, rule-based file and disk manager allowing one to
Formal Software Testing Terri Grenda, CSTE IV&V Testing Solutions, LLC www.ivvts.com Scope of Testing Find defects early Remove defects prior to production Identify Risks Unbiased opinion When Should Testing
In this Lecture you will Learn: Implementation Chapter 19 About tools used in software implementation How to draw component diagrams How to draw deployment diagrams The tasks involved in testing a system
Fuzzing is a software testing technique that introduces invalid, malformed, or random data to parts of a computer system, such as files, network packets, environment variables, or memory. How the tested
Testing, Debugging, and Verification Testing, Part II Moa Johansson 10 November 2014 TDV: Testing /GU 141110 1 / 42 Admin Make sure you are registered for the course. Otherwise your marks cannot be recorded.
Wipro & Experitest Co-webinar: Perfect Your Mobile App with Load Testing and Test Automation June 2015 Speakers Guy Arieli CTO Experitest Sudheer Mohan Director - Mobility Certification & Automation Wipro
Fuzzing in Microsoft and FuzzGuru framework OWASP-IL May-2007 John Neystadt firstname.lastname@example.org Lead Program Manager Forefront Edge, Microsoft Agenda Overview Introduction to Fuzzing FuzzGuru Architecture
A Fresh Graduate s Guide to Software Development Tools and Technologies Chapter 3 Testing CHAPTER AUTHORS Michael Atmadja Zhang Shuai Richard PREVIOUS CONTRIBUTORS : Ang Jin Juan Gabriel; Chen Shenglong
Software Testing System, Acceptance and Regression Testing Objectives Distinguish system and acceptance testing o How and why they differ from each other and from unit and integration testing Understand
ISTQB Certified Tester Foundation Level Version 2015 American Copyright Notice This document may be copied in its entirety, or extracts made, if the source is acknowledged. #1 When test cases are designed
Open Source and Commercial Performance Testing Tools Palla Vinod Kumar Accenture Delivery Center for Technology in India Accenture, its logo, and High Performance Delivered are trademarks of Accenture.
Selenium Online Training Course Content Faculty: Real time and certified (Includes theoretical as well as practical sessions) Introduction to Automation What is automation testing Advantages of Automation
Exploratory Testing in an Agile Context A guide to using Exploratory Testing on Agile software development teams. Elisabeth Hendrickson 2 Exploratory Testing. So you bang on the keyboard randomly, right?
Department of Veterans Affairs Open Source Electronic Health Record (EHR) Services Web Application Automated Testing Framework (WAATF) Software Design Document (SDD) Version 1.0 September 2013 Contract:
Note: This material is for Evaluators reference only. Caters to answers of CSTE Mock Test - Part III paper. 1. Independence is important in testing is mostly due to the fact that (Ans: C) a. Developers
TEST AUTOMATION FRAMEWORK Twister Topics Quick introduction Use cases High Level Description Benefits Next steps Twister How to get Twister is an open source test automation framework. The code, user guide
Development Best Practices 0 Toad Toad for Oracle v.9.6 Configurations for Oracle Standard Basic Toad Features + Team Coding + PL/SQL Profiler + PL/SQL Debugging + Knowledge Xpert PL/SQL and DBA Toad for
Intro to SOA Regression Testing: A Hands-on Approach Mamoon Yunus, CTO Forum Systems Learn SOA Regression Testing techniques through automated data sources and recording base-line tests. I. INTRODUCTION
Planning the Migration of Enterprise Applications to the Cloud A Guide to Your Migration Options: Private and Public Clouds, Application Evaluation Criteria, and Application Migration Best Practices Introduction
International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 4, Number 5 (2014), pp. 445-452 International Research Publications House http://www. irphouse.com /ijict.htm Finding
coursemonster.com/uk HP LoadRunner: Essentials 11 View training dates» Overview This five-day course introduces students to LoadRunner 11. The course covers topics for the VuGen, the Controller, and Analysis