White Paper Outsourcing of Embedded Software Testing

Transcription

1 Test and Verification Solutions White Paper Outsourcing of Embedded Software Testing

2 INTRODUCTION This document describes the issues faced when outsourcing the testing of embedded software and the methodology applied by Test and Verification Solutions (TVS) when providing this service. It describes the potential business benefits and how various aspects of outsourcing of software testing should be considered in respect of individual circumstances. CURRENT INDUSTRY CHALLENGES Notwithstanding the impact of the 2008 recession the embedded systems industry has experienced continued growth over a number of years. The UK electronics industry is worth 23 billion a year and is the fifth largest globally. The UK is widely regarded as the leading European nation for electronics design. The UK sector employs approximately 25,000 people across 11,500 companies and is the leading destination for overseas investment in to the European Union. More foreign companies set up European headquarters in the UK than anywhere else (source; Department for Business Enterprise and Regulatory Reform). However, consistent pressures have occupied the minds of industry leaders and project teams which can best be summarised as time to market, increasing complexity (multicore, software scope, multiple new technologies), increasing quality demands and underlying this is an insufficient number of skilled personnel in western Europe and the USA. Selecting the optimum architecture is a formidable challenge, getting it wrong can result in five years or more lost market share (consider the Symbian history). In addition avoiding re spins and product recalls is critical in any product lifecycle. Finally, time to market is more crucial than it ever was, with the window of opportunity available to gain a foothold in any market becoming ever shorter. Outsourcing certain aspects of your product development can make a meaningful contribution in avoiding this PROJECT CRITICAL PATH Key aspects of any project in the embedded systems market are design, development and testing. Determining what product will be competitive and then defining the requirements for that product are critical. Delivering on those findings requires a development strategy and a robust methodology for testing that ensures quality objectives are met. Added to this is an incessant need to deliver against aggressive deadlines set by company leaders. The TVS methodology for outsourcing of an embedded software testing function focuses on improving the quality of software delivered to production, shortening the development time and reducing cost. KEY ISSUES WHEN OUTSOURCING EMBEDDED SOFTWARE TESTING Management must have a clear focus on why they are outsourcing a software testing function. These can be best summarised as follows;» To accelerate the maturation of new products;» Agility and cost savings; the ability to utilise additional low cost resource at peak times» Concentrate on core competence; capitalise on your inherent IP and knowledge and use a 3 rd party for everything else

3 » Better software through additional resources; the use of outside talent and expertise will allow an organisation to develop better products over the same period than its competitors. MANAGED SERVICE MODEL (MSM) The MSM addresses the key factors in the implementation of an outsourced software testing capability (see APPENDIX 1 for details of the MSM). Discovery: this is an initial consultancy session which determines the nature of the MSM in connection with a client project. It consider a number of key issues including the product roadmap, historical product issues, product stability, client personnel, customer profile, project timeframe as well as specifics around development methodology, language, tools etc. Proposal: this details the approach to be taken by TVS for the implementation of its software testing services. It will address a number of key areas including;» Location; where is the work to be carried out and on what equipment» Personnel; team structure, ramp up plan, on site/off site/off shore, handover plan» Communication; how technical and management level issues are communicated» Knowledge Management; project information, Q&A s» Planning; project planning, resource allocation» Reporting; process, project status, variance, resolution» Deliverables; specification, review process, acceptance process» Infrastructure; requirements for the project» Automation; does the project suit itself to test automation? Cost case» Service levels; define service levels, variance allowance, turn around time» Escalation; detailed procedures CONTINUOUS IMPROVEMENT MODEL (CIM) TVS will ensure the continuous improvement of the service they provide. Regular senior management reviews of the service delivery will provide feedback from both sides on the engagement. The reviews will identify areas of concern where remedial action is required and areas of best practice which should be expanded across the project. The reviews will always identify a list of action points that will be tracked for progress and reviewed at subsequent senior management reviews. TVS will use its knowledge and experience from other client engagements to improve the service delivery to the client. CLIENT READINESS An organisations readiness to outsource its software testing needs to be assessed. Bad existing internal processes will contribute to a failed outsourcing effort. Formal measurement such as Capability Maturity Model Integration (CMMI) exist but the experience of TVS on client engagements is that most organisations will not want to base their decision on such formal methods. The minimum set of processes that your organisation will require to outsource software testing is:

4 » Requirements elicitation» Change tracking and management» Project planning, monitoring and control» Configuration management» Build, test and release automation» Validation and verification processes BUSINESS CASE Development of a business case can focus on a number of areas associated with software testing outsourcing. These can be described as follows; Reduced Time to Market: utilising TVS will free developers to focus on developing, not testing. The reduced time to market needs to be balanced against the additional costs of retaining TVS services. Test Automation: the automation of unit testing in particular will reduce the need for manual effort. It will also facilitate greater levels of testing and testing earlier in the development cycle which will uncover bugs earlier and in greater quantity. Client management will need to form a view based on historical data to determine savings as a result. Cost Reduction: taking advantage of low cost off shore expertise can provide a like for like cost reduction. Improved Company Focus: Testing is arguably not a core competency. Personnel resources should be focused on core skills. Quality Improvement: The TVS testing methodology will result in quality improvements. Client management may need to embark on a pilot project with TVS in order to understand and attach a monetary value to this. Cost Variablisation: The conversion of fixed costs to variable costs. SUMMARY The embedded systems industry is a powerful force in the UK. Foreign organisations invest in the UK to capitalise on our infrastructure and skills. However competing in the current market place is immensely challenging and organisations should focus on their core skills and utilise expert resource and low cost off shore expertise for software testing. Doing so will provide cost, time to market and quality benefits.

5 APPENDIX 1: MANAGED SERVICE MODEL (MSM) The MSM addresses the key factors in the implementation of an outsourced software testing capability. Discovery: this is an initial consultancy session which determines the nature of the MSM in connection with a client project. It consider a number of key issues including the product roadmap, historical product issues, product stability, client personnel, project timeframe as well as specifics around development methodology, language, tools etc. Proposal: this details the approach to be taken by TVS for the implementation of it s software testing services. It will address a number of key areas including; Location;» where the work is to be carried out and on what equipment Personnel; details all aspects of the team involved» A team structure for the resources including a single point of project management responsibility from TVS. This person will be known as the TVS Team Leader.» A ramp up plan for the resources that ensures the resources are productive as quickly as possible and that also ensures minimal disruption to the client project team.» Agreement on offsite work where agreed the associated operational issues.» A process for the replacement of TVS project staff.» The potential for off shore staffing. TVS can advise of the pros and cons of using an offshore capability and will resource and manage such a facility if appropriate. Communication;» The TVS team leader and will be responsible for all management level communication with the client.» Statement on how technical level communication is channeled and managed. Knowledge Management;» TVS will ensure that knowledge is properly managed within the TVS project team.» TVS will define a process by which TVS will first try to answer technical questions from within the team and the management of any resulting resolution issues.» Agreement on how questions and answers will be stored within a knowledge management system to ensure the question can be answered within the TVS team should it arise again.» The knowledge management system will take the form of a document (e.g. spreadsheet, Word document or Twiki page) to be held in an agreed location.» The knowledge management system will become a deliverable at the end of the project. Planning;» A project planning process by which the TVS team leader can agree with the client a list of expected project deliverables and a schedule for their delivery.

6 » TVS will also detail the planned allocation of project resources required to meet that project delivery schedule and the dependencies on the client to ensure their delivery. Reporting;» A project reporting process by which the TVS Team Leader can report progress against plan, highlight variance from plan and the proposed remedial actions and also highlight any dependencies from TVS on the client. Deliverables;» A specification process by which the TVS will write specifications of the agreed deliverables for review by the client.» A review process by which both parties can agree the content of those specifications.» A process by which TVS can declare a deliverable delivered.» And an acceptance process by which the client can either accept the deliverable or request changes.» Where required an interim delivery process by which interim deliveries of a deliverable can be delivered for early client feedback. Infrastructure;» A clear definition of the infrastructure provided by the client and required by TVS in order to perform the tasks identified for them on the project. Automation; This offers significant potential for productivity and quality improvements, but is not always appropriate and if implemented badly or in the wrong circumstances can have a negative effect. A number of factors must be considered;» Does the product roadmap justify the initial cost of automation?» Is the existing software sufficiently stable?» Complexity of the test environment. Complex the testing environments require higher levels of maintenance of the automated capability.» Frequency of test runs.» Ease of automating the pass/fail criteria. Service levels; The MSM agreement will define expected service level agreements from both sides. This will at least include the following;» The amount of variance allowed from the proposed schedule for the deliverables.» The expected turn around time for deliverables provided by TVS for client review.» The maximum downtime allowed in the client infrastructure identified above before the agreed delivery schedule is affected. Escalation;» Escalation procedures by which either side can escalate an issue (personnel, technical, management or other) to a defined senior point of contact at the client and TVS for resolution.