The problems of outsourcing application development and testing

Transcription

1 The problems of outsourcing application development and testing Micro Focus Research results October Micro Focus - Testing - research results

2 Demographics 590 senior IT decision-makers in organisations with mainframes in their infrastructure were interviewed in August and September 2013, split in the following ways... in organisations of 500+ employees a range of private sector organisations a range of American, European, and Asian countries IT & software Financial services Business & professional services Retail, distribution & transport Manufacturing employees employees More than 3000 employees Media, leisure & entertainment Other commercial USA UK France China Australia Brazil Germany Sweden Japan Figure 1: Analysis of size of organisations organisation Figure 2: Within which sector is your organisation? Figure 3: Analysis of location of respondents organisation 2 Micro Focus - Testing - research results

3 Four areas of interest: 1: Extent of outsourcing 2: Problems with outsourcing partners 3: The problem of defining requirements 4: The impact of change requests 3 Micro Focus - Testing - research results

4 1: Extent of outsourcing 4 Micro Focus - Testing - research results

5 Use of outsourced vendors is growing Over time, organisations are outsourcing greater percentages of their application testing/development 56% 54% to the extent that in two years time, organisations predict that the majority of application testing and development will be performed externally 43% 43% 47% 49% The benefits that organisations see from using partners for application development and testing must be substantial enough to warrant their increased use Why are organisations increasing their reliance on outsourced vendors? Two years ago Today In two years time Testing Development Figure 4: What percentage of application development/testing is outsourced over time 5 Micro Focus - Testing - research results

6 Outsourcing results in many benefits Quicker than doing it ourselves Leverage specific skills of others to do new things, possibly in new markets Cheaper than doing it ourselves Leverage maturity and capability of others to increase product quality Allows us to more effectively deploy across multiple devices/platforms Shares the risk with others We lack the skills to do this in house We have struggled to find the expertise in the employment market 38% 33% 29% 24% 17% 51% 50% 56% An average organisation cites three of these benefits with half or more saying that it is cheaper and quicker than doing it themselves It is not just about cost and speed, though. Many point specifically to leveraging skills and lacking the skills to develop these applications themselves suggesting that use of these vendors enables organisations to deploy apps that they would not be otherwise able to develop or test These projects are not insubstantial, either Figure 5: 6 The percentage that rate each option as one of their top three reasons for working with outsourced development/testing partners Micro Focus - Testing - research results

7 Outsourced projects are substantial Most say that their outsourced projects require 2-5 man years of effort 64% working out to an average of around three man years for each application development/testing project Organisations are rarely outsourcing small little projects. These development and testing programmes require significant investment in time which could partially explain why many organisations outsource this to get products deployed more quickly 15% 18% 1% These projects are cross-platform, too 0-1 man year of effort 2-5 man years of effort 6-10 man years of effort More than 10 man years of effort 7 Micro Focus - Testing - research results Figure 6: What is the average size of each one of your individual outsourced application development and testing projects?

8 Outsourced projects extend across platforms Cloud applications 44% 50% Projects are already involving different platforms Customer facing applications Mainframe applications Desktop applications Internal business applications Mobile applications 44% 48% 43% 45% 41% 45% 41% 45% 40% 48% and are increasingly likely to in two years time Application development/testing that is outsourced is rarely likely to revolve around a single platform. Outsourced vendors are helping organisations to deploy their applications across a wide range of different platform reflecting organisation s desires to launch on more than one platform At present In two years time Figure 7: 8 What percentage of your current/future outsourced applications development and/or testing projects involve the following applications? Micro Focus - Testing - research results

9 2: Problems with outsourcing partners 9 Micro Focus - Testing - research results

10 Outsourced projects encounter problems Almost a quarter fail to deliver on requirements, and a third run into time and service-level issues Although organisations are increasingly relying on vendors to develop and test applications, a substantial number of them are failing What percentage of your outsourced application development and testing projects have failed to totally deliver on the final requirement? 23% Are many organisations so focused on speed and cost of deployment that they are not doing their due diligence and ensuring that development and testing projects meet the organisation s needs? What percentage of your outsourced application development and testing projects have run into time and service-level issues? 31% Why do some projects fail? Figure 8: Average percentage of projects that organisations experience problems with 10 Micro Focus - Testing - research results

11 Many reasons for failure Too many changes occurring to the requirements during the project 55% The majority say that too many changes is why projects fail A lack of provider skills to correctly interpret or manage the requirements Insufficient testing, in either scope, depth or duration 40% 47% but there are a range of problems that organisations struggle with Notably, more organisations blame the vendor s lack of skills (47%) rather than a lack of skills on their part (31%) A lack of provider skills in developing quality software A lack of our skills or experience on our part to clearly describe the system requirements 34% 31% This implies that organisations are more likely to see the problems originating with the vendor rather than with them Poor project management at either or both parties 24% What is the impact when projects go wrong? Figure 9: 11 What is usually the key reason your outsourced application development or testing projects overrun or fail to deliver on the final requirement? (500 respondents) Micro Focus - Testing - research results

12 Outsourced projects create negative issues Only 16% have never experienced any negative impact from their outsourced projects The overall cost of developing, testing and deploying new software 47% meaning that five in every six of these organisations have suffered negatively The most likely problems revolve around being slow to market and costing too much, yet many say that they use outsourcing partners in order to cut cost and time! Are vendors not living up to expectations? Releasing customer products to market on time Maintaining a consistent level of quality, security, and user experience Protecting your company's IP The longer-term ability for you to stay close to the evolution of your business systems Your company's public image or reputation 39% 34% 29% 26% 25% Your company's revenues 12% Are these problem projects causing senior IT decision-makers problems? Our outsourced app development and testing projects have not had any negative impact on our company 16% 12 Figure 10: Have any of your outsourced application development and testing projects created any negative issues related to any of the following?

13 Some outsourced projects are terrible! Unmanageable 28% 57% describe at least one of their outsourced projects pejoratively An embarrassment 26% Though many describe some of their projects as embarrassing, one in six describe some of their projects as being total failures A nightmare A total failure 16% 16% Far from being awkward inconveniences, many problems being experienced with vendors are resulting in substantial difficulties within the organisation itself None of the above 43% How severe are these problems? Figure 11: Could any of your outsourced application development and testing projects be described in any of the following terms? 13

14 Slow outsourced projects put ITDMs at risk A third have had their jobs threatened by outsourced applications being slow to deliver Ultimately, IT decision-makers are accountable for the failures of these projects. Because of this, 31% feel that their livelihoods have been in danger due to applications developed/tested outside of the company not arriving in a timely manner 31% If organisations are looking to use these outsourced development and testing partners even more, will IT decisionmakers be under even greater risk? 69% Is part of the problem due to the way that organisations define their development/testing requirements? Yes No 14 Figure 12: Have the delays on the delivery of any of your outsourced application development and testing projects ever threatened your job?

15 3: The problem of defining requirements 15

16 Few use dedicated tools Spreadsheets (e.g. Excel documents) 76% Less than half use a dedicated tool to capture and share project requirements Instead, most rely on common office applications like spreadsheets and word processors Written documents (e.g. Word documents) 73% Given that the numbers of outsourced projects are increasing, why are so few using specialist software to help them? Dedicated requirements management tools 48% Does the lack of use of such software suggest why so many projects run into problems? Figure 13: What tools do you use to capture and share the requirements of an application development or testing project? 16

17 Projects receive input from many stakeholders Only 15% do not capture and share project requirements with other stakeholders Given that the vast majority collaborate on the requirements of their outsourced development/testing projects with many interested partners, why are so many relying on older technologies to capture project needs? 15% Can the project changes that naturally arise in dialogue be captured in spreadsheets and word-processed documents? 85% Can organisations visualise what the finished application will look like? Yes No 17 Figure 14: Do you capture and share project requirements with a variety of stakeholders including your outsourced development or test providers?

18 Mocking up application appearance A third use tools which cannot visualise how the final product will look 33% 3% This is troubling, as it means one in three organisations are unable to visually demonstrate to outsourced vendors how they would like their applications to look and behave 64% Is it unsurprising that so many projects experience problems if organisations are unable to effectively declare what they want from their application development projects? Yes No Don't know How do vendors react to this? Figure 15: Do the tools you use to capture and share project requirements show how the final product will look visually, through storyboarding and simulation? 18

19 Vendors expect organisations to struggle Only a third say that vendors need organisations to perfect their projects requirements 11% 3% This means that two-thirds are working with partners who do not demand that projects are clearly defined before they commence work 32% Should vendors not be assisting their customers to ensure that they can declare their needs accurately, so that the applications they develop/test are suitable for their customers needs? 53% It is a prerequisite Does this result in delivered applications requiring additional work? It is expected but not demanded It is thought of as a 'nice-to-have' It is not a requirement at all 19 Figure 16: What level of expectation does the outsourced application development/testing vendor put on you and your team to get project requirements right first time?

20 Delivered applications are rarely complete 19% 17% 18% 16% 9% 6% 5% 4% 2% 2% 1% 1% 0% 1-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90-99% 100% Figure 17: What percentage of your outsourced application development and testing projects require additional work by you after final delivery from your provider? Just 2% say that none of the projects require additional work upon final delivery. On average, 31% of projects need the organisation to do more work on them Is the organisation to blame for this due to not clearly defining their needs? Or is it the vendor s fault for not ensuring that their client s needs were clearly defined at the start of the project? 20

21 4: The impact of change requests 21

22 Organisations struggling with variability, variety and change Only a third are able to manage this very well 5% meaning that the majority struggle with change requests, timeline problems, and the general process of getting application projects delivered 37% Despite this, few are ensuring that their application development/testing projects are clearly defined before work commences 58% Very well all possibilities are factored in to ensure we manage variability, variety and change to our advantage and are not hampered by unnecessary financial and timeline problems How often are changes to project specifications being requested? Adequately negotiations are not totally definitive but it is impossible to imagine and guard against every unnecessary financial and timeline problem Poorly we are always hampered by unexpected change requests and the related financial and timeline problems 22 Figure 18: When considering your application development or testing outsourcing contracts, how well are you able to manage variability, variety and change?

23 Organisations make frequent change requests Multiple times a week 6% Almost half make change requests once a fortnight or more frequently About once a week 20% About once a fortnight 21% Yet few say that they are able to handle variability, variety and change well About once a month About once a quarter 16% 27% Given that project needs are not clearly defined, how many of these change requests will be contradictory, superfluous or irrelevant? Less frequently 10% Never 0% Are these change requests generating additional costs? Figure 19: On average, how frequently do you change the specification of work being performed by your application development or testing outsourcing vendors? 23

24 Change requests incur unpredicted costs Only 4% say that this is not true Unpredicted costs occur/arise with every change request 9% and over half say that there are hidden costs attached to half or the majority of change requests Given the frequency of change requests and the problem of clearly defining needs at the start of the project, these costs are likely to be substantial Unpredicted costs occur/arise with the majority of change requests Unpredicted costs occur/arise with about half of change requests 32% 30% Many organisations work with external partners to save money; these costs are likely to contradict the decision to outsource Unpredicted costs occur/arise with a minority of change requests 25% Are these unpredicted costs at least reasonable? Unpredicted costs never occur/arise 4% Figure 20: How likely is that that change requests incur unpredicted costs? 24

25 Unpredicted costs feel like vendors looking for additional profit 19% 3% Organisations have mixed feelings about these additional costs 32% 47% Only around a third feel satisfied with the cost. Many are more likely to feel that costs attached to change requests are just another way for vendors to make a profit They are usually reasonable but occasionally seem like changes for the sake of making additional profit Requirements change and we need to pay for them Are vendors choosing to rarely help organisations to clearly define their needs in order to make additional profit from change requests? Many of them seem like the outsourcing vendors looking for additional profit They all seem like the outsourcing vendors are looking to make additional profit Figure 21: How would you describe the majority of these unpredicted costs? (568 respondents) 25

26 In summary Use of outsourced vendors on application development/testing projects is increasing as organisations believe those vendors can deliver applications cheaply and quickly Applications are increasingly likely to stretch across different platforms Organisations are likely to experience problems on these outsourced projects They make too many change requests and feel that outsourced partners lacks the necessary skills These problems result in testing/development projects being more expensive and slower to market than predicted and have the possibility of risking senior IT decision-maker s jobs Few organisations use dedicated tools to define their project needs, and few can visualise how they want their applications to look As a result of poor planning, most applications need additional work post-delivery Organisations do not handle change requests to projects very well but they make frequent change requests and feel that the costs incurred from doing so are unreasonable 26

27 The problems of outsourcing application development and testing Racepoint Micro Focus Research results October