Test automation: Open source compared to commercial software

Size: px

Start display at page:

Download "Test automation: Open source compared to commercial software"

Beverley Long
8 years ago
Views:

1 Test automation: Open source compared to commercial software Hall 1 Booth 1145 Folie 1

2 Agenda 1. ISTEC company profile 2. Software Quality Assurance Our services 3. Customer scenario 4. Customer requirements 5. Tool comparison criteria 6. Lessons learned 7. Conclusion 8. Announcement 9. Questions Folie 2

3 1. ISTEC Company Profile Independend software and system house Since 1982 successful on market 60 regular employees Development of customer specific solutions from the identification of problems up to the system installation Offices at Ettlingen (Karlsruhe) and Leinfelden (Stuttgart) Folie 3

specific solutions from the identification of problems up to the system

4 2. Software Quality Assurance Our services Folie 4

5 3. Customer scenario Development of a new national electronic ID Card system Over 20 involved sub systems Frequent releases by suppliers functional test case scenarios 10 load test case scenarios Environment with distributed sub systems and various operating systems Highly complex use cases Huge failure costs High demands on software quality Short response time on issues Folie 5

case scenarios Environment with distributed sub systems and various operating systems Highly

6 4. Customer requirements on test automation Platform independency using Java Detailed multi staged Reporting High performance for regression testing Run functional test cases Run load test cases Distributed sub systems with various operating systems Short response time on issues Central test execution control Folie 6

functional test cases Run load test cases Distributed sub systems with various

7 5. Tool comparison criterias - overview + Some inhouse Know-how + No license costs + No dependencies on suppliers + Functionality? Reporting? Code quality? Documentation? Professional support? Upward compatibility? Demand on hardware Folie 7

Functionality? Reporting? Code quality? Documentation?

8 5. Tool comparison criterias - reporting Commercial products generally offer more reporting capabilities and interfaces: Requirement specification tools Issue Tracking Systems Progress monitoring Systems More flexible multi stage reporting with result analysis Folie 8

Requirement specification tools Issue Tracking Systems Progress

9 5. Tool comparison criterias code quality Tools for static code analysis can be helpful for estimating: Time and development effort for customizing The risks of higher costs Upwards compatibility Reusability There are several free tools available for Java: Checkstyle Metrics Sonar Folie 9

customizing The risks of higher costs Upwards compatibility Reusability

10 5. Tool comparison criterias code quality Example results from analyzing JMeter with Metrics Metric Acceptable JMeter Cyclomatic complexity Nested block depth Folie 10

11 5. Tool comparison criterias documentation Lack of technical documentation can lead to significantly higher costs for customizing Lack of user documentation can delay the beginning of the testing phase by extending the learning curve Both increases the risk of running into technical show stoppers Commercial solutions generally offer a much better documentation quality Folie 11

of the testing phase by extending the learning curve Both increases the risk of running into

12 5. Tool comparison criterias professional support Commercial providers can generally offer professional support for their products Improves the planning certainty Shortens the learning curve Training courses available Folie 12

support for their products Improves the planning

13 5. Tool comparison criterias upwards compatibility A bad software architecture can compromise upwards compatibility, making it impossible to update to a newer version: Cyclic dependencies No strict separation of core and individual functionality code: Custom Core v1 Custom Core v2 Custom Custom Core v1 Core v2 Folie 13

version: Cyclic dependencies No strict separation of core and individual

14 5. Tool comparison criterias demand on hardware Most open source tools for test automation do have a higher demand of memory and processor capacities then commercial solutions since they are not as well optimized: Higher project costs due to higher demand of hardware Performance and memory leaks can turn out to be a showstopper when running load test scenarios Comparing memory and processor consumption in a load test scenario Consumption ISTEC Test-Center JMeter Difference Memory 370 MB 790 MB +113% Generated load 480 users 200 users +240% Folie 14

Performance and memory leaks can turn out to be a showstopper when running load test scenarios Comparing memory and processor consumption

15 6. Lessons learned Take your time to identify your demands! Also analyse the secondary characteristics of eligible tools since they can be essential to the success of the project: Code quality Availability of support, documentation and reporting Architecture and upwards compatibility Hardware consumption and performance There s not only black and white! Commercial leightweight products should be considered as well and can turn out to be the best compromise, offering the advantages of both worlds Low license costs and supplier dependency High flexibility Folie 15

Availability of support, documentation and reporting Architecture and upwards compatibility Hardware consumption and performance There s not

16 7. Conclusion Open Source Commercial light weight product Commercial heavy weight product License costs Supplier Dependency + o - Reporting o o + Code quality o + + Support Documentation o + + Upward compatibility Hardware consumption o Folie 16

Dependency + o - Reporting o o + Code quality o + + Support - + +

17 8. Announcement Live demo at the automotive testing expo on stand 1145 ISTEC Test-Center ISTEC Hardware-Simulator Folie 17

18 9. Questions?? Contact? ISTEC GmbH Michel Max-Lang-Str 56/ Leinfelden-Echterdingen Phone Hall 1 Stand 1145 Folie 18

Leinfelden-Echterdingen Phone +49 711 160 570

ISTEC.MIP Measurement Data Integration Platform

ISTEC.MIP Measurement Data Integration Platform Dr.-Ing. Carsten Booth 1145 Folie 1 ISTEC Company Profile Independend software and system house Since 1982 successful on market 60 regular employees Development