How to Handle Test Tool Dependencies in an Open Source Project?

Size: px

Start display at page:

Download "How to Handle Test Tool Dependencies in an Open Source Project?"

Leon Peters
7 years ago
Views:

1 Produced by STeP-IN SUMMIT Hosted by 5 th International Conference on Software Testing How to Handle Test Tool Dependencies in an Open Source Project? M.S. Krishna Rao Sun Microsystems India Pvt. Ltd krishnarao.ms@sun.com Copyright: STeP-IN Forum and Quality Solutions for Information Technology Pvt. Ltd. Published with permission for restricted use in STeP-IN SUMMIT in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during STeP-IN SUMMIT as per the applicable conditions. Practices Experience Knowledge Automation

2 How to Handle Test Tool Dependencies in an Open Source Project? M.S. Krishna Rao Sun Microsystems India Pvt. Ltd Open Source - Test Environment Code contributors supply a set of unit tests Some contributors may supply only tests Open source infrastructure should also provide: Test framework and/or an execution environment A set of example unit test suites and/or functional test suites Result: Easier, intuitive, user-friendly test environment (development+execution) = better community Has its own set of challenges and issues 2 Page 1 of 10

3 Test Tool Dependencies A Challenge Test tool plays a major role in defining a test environment To provide a good test environment in an open source context, dependencies on the test tool should be handled For Example: Test tool in the test environment could have been proprietary and/or commercial Test frameworks and suites could have been built using proprietary and/or commercial test tools Due to dependencies Unable to facilitate easier, intuitive, user-friendly test environment How to handle test tool dependencies in an open source project? 3 Few Approaches Open source the test tool Convert the test suites Customize an open test tool Develop tests using an open test tool 4 Page 2 of 10

4 Approach#1 Open Source The Test Tool Open source the test tool in use According to a study, community is usually flexible to adopt new test tools When: If you have a proprietary, generic test tool 5 Approach#1 Open Source The Test Tool Points to Ponder Cost Support to the Community Licensing issues Third-party dependencies Source code readiness Documentation 6 Page 3 of 10

5 Approach#1 Open Source The Test Tool Budget Constraints - Cost Effectiveness Put out binary first - If required open source later based on the community response Risk: Redistribution scenarios could be badly affected Open Source in Low Profile - Promote open source of the main product and not the test tool Risk: Occasional attention may be required 7 Approach#1 Open Source The Test Tool Advantages Disadvantages Advantages: Test tool enhancements with the help of community Test suite contributions Integrate into internal test suites Internal QA team can support the community Disadvantages: Most demanding approach in terms of costs and efforts Continuous need of expert resources in house Success depends on the learning curve and usability of the test tool Difficult to predict acceptance, adoption, response and future contributions 8 Page 4 of 10

6 Approach#2 Conversion of Test Suites Convert the test suites to be compatible with an existing open source test tool Popular, open, stable test tools already exist Choose the tool that best suits your requirements Convert test suites into its format and publish 9 Approach#2 Conversion of Test Suites Points to Ponder Test tool selection Test suite selection Test suites conversion Effect on the internal quality team Licensing issues on test suites 10 Page 5 of 10

7 Approach#2 Conversion of Test Suites Advantages Disadvantages Advantages: You can predict the response by the community You have to manage only the open sourced main product No need to introduce another test tool into the market Disadvantages: Test conversions could be repetitive and expensive Automated test converters have their own maintenance cycles Often converters do not reduce manual intervention to zero level Both tools have to be used in the internal test cycles? Open test tool expertise has to be developed in-house to help the community 11 Approach#3 Customize an Open Test Tool Customize an existing open source test tool to your requirements For Example: Customize open test tool to understand the current internal test suite formats 12 Page 6 of 10

8 Approach#3 Customize an Open Test Tool Points to Ponder Test tool selection (Suitability, Extensibility, Complexity) Customization and code re-use Community Response Inertia from Community 13 Approach#3 Customize an Open Test Tool Advantages Disadvantages Advantages: You can predict the response by the community There is no need for test suite conversions You can incrementally introduce the functionality as necessary Disadvantages: Test tool selection could be very difficult considering suitability, extensibility, complexity Depends solely on extensibility of the open test tool Expensive Learning curve - In depth knowledge of both tools is required Stability time for customized features If open source tool grows too fast, integration of the customized features could prove to be a bottleneck 14 Page 7 of 10

9 Approach#4 Develop Tests Using Open Test Tool Product is under development? Plan test development with an existing open source test tool Points to Ponder: Test tool selection Burden on QA Team Support Model 15 Approach#4 Develop Tests Using an Open Test Tool Advantages - Disadvantages Advantages: Readiness to open source the main product along with test environment Disadvantages: Support contract could become mandatory Dependencies on the features of paid version could prove to be a blocker for open source again. 16 Page 8 of 10

10 Important Areas or Issues Tool Evaluation Licensing Issues Intellectual Properties and Patents Quality Criteria Community Readiness 17 Tangible Benefits Successful open source endeavor with test environment Growth of community around the original product Save a lot of over-head costs and time to the organization 18 Page 9 of 10

11 Q and A Thank You! Please send your feedback/comments/queries to: krishnarao.ms@sun.com 19 Page 10 of 10

STeP-IN SUMMIT 2007. International Conference On Software Testing

STeP-IN SUMMIT 2007. International Conference On Software Testing STeP-IN SUMMIT 2007 International Conference On Software Testing Implementation of Continuous Integration and Automated testing of a large scale web based application for a European by Amandweep Suresh